Confused about this situation?

Well it sounds to me like you are making the smart decision by saying no. Its great that you respect his wife enough not to do something you know she is uncomfortable with. If he got mad at you thats not your fault, that is something he has to work through on his own. You need to do what is best for you and your husband. Best of luck to you and I hope it all works out.