No, that he uses it to answer charges of elitism in the "You have many houses" angle is a pretty trivial complaint. This is, in fact, quite likely one of the few places it fits.
I don't get it. How is the fact that he doesn't know how many houses he owns justified by the fact that he was POW? It just doesn't follow. At all.