The courts have allowed regulations ("infringements") throughout our history. Whether you agree with the court decisions is irrelevant because their decisions determine the law.
Laws prohibiting violent felons, minors, or any other criteria are also infringements--are those not allowed?
Felons lose their rights per Amendment 5. The mentally defunct too, are deprived of their rights via Due Process. Minors are not capable of understanding their rights, thus they cannot exercise them. That's why they cannot consent.