https://www.loc.gov/classroom-materials/elections/right-to-vote/the-founders-and-the-vote/
At first the states were allowed to decide who could vote
The Founders and the Vote
In the Declaration of Independence, Thomas Jefferson wrote, "Governments are instituted among Men, deriving their just Powers from the Consent of the Governed."
James Madison, President of the United States
But how would Americans consent to be governed? Who should vote? How should they vote? The founders wrestled with these questions. They wondered about the rights of minorities. In their day, that meant worrying if the rights of property owners would be overrun by the votes of those who did not own land. James Madison described the problem this way:
The right of suffrage is a fundamental Article in Republican Constitutions. The regulation of it is, at the same time, a task of peculiar delicacy. Allow the right [to vote] exclusively to property [owners], and the rights of persons may be oppressed... . Extend it equally to all, and the rights of property [owners] ...may be overruled by a majority without property....
Eventually, the framers of the Constitution left details of voting to the states. In Article I Section 4, the Constitution says:
The times, places and manner of holding elections for Senators and Representatives, shall be prescribed in each state by the legislature thereof; but the Congress may at any time by law make or alter such regulations.
Unfortunately, leaving election control to individual states led to unfair voting practices in the U.S. At first, white men with property were the only Americans routinely permitted to vote. President Andrew Jackson, champion of frontiersmen, helped advance the political rights of those who did not own property. By about 1860, most white men without property were enfranchised. But African Americans, women, Native Americans, non-English speakers, and citizens between the ages of 18 and 21 had to fight for the right to vote in this country.