Clinton was talking on the Daily Show about how Washington used to be a place where people would move their families to while Congress was in session, and it made for a more collegial atmosphere & better relationships. He regretted that we lost that, and cited it as a cause of much of the rancor & partisanship we see today.
Leading the free world is no small thing, and it's not for everyone. It is certainly not for people who are unwilling to make sacrifices like a 5-day work week, no matter how far they have to travel.