There is no increase in the average, but you mistakenly believe that there would be. This is your underlying error.
On a hot day (on every day, in fact) the ground and the ocean are heated by the sun. The ground and the ocean then heat the cooler/colder atmosphere. This causes convection, i.e. the atmosphere at ground/sea level increases in temperature, expands and rises. This rising air cools as it rises and stops rising when it achieves ambient temperature.
This is part of the defining properties of the troposphere. This does not occur in the stratosphere, nor is there any weather at all, one level up.
When you roll up the car windows, you are preventing that particular warming air from rising (and preventing cooler air from descending to take its place). The now warmer air remains inside the car, continuing to be heated and its temperature continues to increase (up to a point). Meanwhile, the air
outside the car is being heated in exactly the same way, but rises immediately, with cool air descending to take its place, i.e. convection.
Returning to your "average temperature," if you were to suddenly roll down the windows, the car would immediately cool as the much warmer air rises out of the car. That warmer air certainly heats up the cooler air outside the car as it rises. The descending cooler air cools the interior of the car. Yes, there is a much greater quantity of air outside the car compared to the inside of the car, but the same quantity of thermal energy is simply being redistributed and the average temperature remains unaffected. gfm7175's example of apples in baskets is brilliant and spot-on, but would be even more relevant in this case if instead of baskets, it involved one basket and one canyon, and the measure were average apple per volume, which would remain the same.
Nonetheless, I think we should stick with the original apples/basket distribution scenario.