Yes, the birth story is a minor part of only two gospels and differ in detail, particularly whether Mary and Joseph were from Nazareth or Bethlehem. Elsewhere, in the other 25 books of the New Testament, the birth story is not even mentioned to the best of my recollection.
I get the impression that Mary is actually arguably the most important part of the birth story. The immaculate conception and the virgin birth are the keystone for the
subsequent Trinitarian christology.
But in the grand scheme of things, Christianity is premised on the death and resurrection of Jesus, making Easter really the foundational event of the Christian tradition.