Manly? Top of the list, Andrew Jackson. He participated in at least three duels, and in one let his opponent shoot him before in turn shooting him dead. He was also a general with his most famous battle being that at New Orleans in 1815. You don't get much tougher than Jackson as a president.
Zachary Taylor is another. He has a long list of military battles he fought in often directly in the middle of combat.
Ulysses S. Grant. Obviously a general. He also loved to play poker, drink whiskey, and smoke cigars. He was also a family man and had a fairly large family. In civilian life he was a poor businessman but he never gave up trying and always looked out for his family.
Herbert Hoover. He was a mining engineer and accustomed to roughing it in some pretty remote mining locations. During the Boxer Rebellion in China he offered his services to the USMC as a guide with intimate knowledge of the area they were operating in and often was right in the thick of battle as a result... as a civilian... During the Prohibition years he often visited foreign embassies in Washington as President to imbibe legally... He was the epitome of say what you mean and do the right thing.
Teddy Roosevelt's image was mostly that. He was far more a rich dandy than the rugged frontiersman persona he put on in public.
LBJ was a crude and vulgar ahole who, arguably, also is the only case of stolen valor having awarded himself the Silver Star for bravery in combat, yet never having been in combat. The closest he came was a base tour in the SWPA in the Pacific during WW 2.
JFK is an odd case. There are things that would qualify him like military service but then others that are disqualifying like being a philander. Real men don't sleep with every hot floozy they meet.
Real men are about work hard, play hard, take risks, be tough as nails, and say and do what you mean. No weasels allowed. Being good with the ladies is a bonus but it doesn't define true manliness.