Should I feel embarrassed?

Me and my husband just bought a house for 250K. I am in the military and he is a gov employee. This is a very high end house for what military people, more specifically the people I work with see people in my career field buying. This guy I work with makes me feel so stupid because he's a hater basically. We can afford the payments and I think it's a good investment for the future, and no we will not be moving away from this area for another 5-7 years at least but probably longer. How should I handle the negative reactions of people when they see how much money we have spent on our home. I know its really none of there business but sometimes the haters make me feel kind of dumb.