Am I wrong for not going?

Yes you were wrong. The problem is with your wife not your children. As far as your wife is concerned it seems like she is feeling unloved and neglected and the only way to catch your attention is to "nag" at you, but it seems that doesnt even work since you walk away from her instead of engaging in a conversation or loving her in return. We women are complicated, sometimes we nag you just to get your attention. Net time she nags ask her how she can make things better and see if you can help make that happen.