I'd agree that it is a fantasy of people acting much more nobly in times past. I think they get a pass because, while they do bring up the negative aspects of the older society from time to time, it's sort of been pushed to the side. Really, in the first season it started out kind of throwing the misogyny and bigotry of those times in your face (for instance, the first shows plot largely revolves around two of the servants trying to get an invalid, Mr. Bates, fired, and everyone, especially the head butler Mr. Carson, constantly questions whether he can do his job as a Valet), but as the show progressed they watered down the negative aspects more and more in order to maintain the likability of the characters. The conservative, Tory aristocrats on the show are all remarkably tolerant and remarkably adaptable to change, more than I think it would be reasonable to expect of people living in that time. Now, of course, progressives always do have Tom Branson, the Irish Socialist radical chaufer who runs off with the Lords daughter. But Branson actually wasn't the well liked among the fan base, although maybe that's changed with the events in the third season, which result in him being aritocratized.
I always liked him. He is proud and likes getting what he wants. We will see if his position changes him.