While I tend to agree, there is no evidence either way except what has been detailed in the Gospels. Since the authors of the Gospels went to so much trouble to detail the early life of Jesus, whether fabricated, exaggerated or not, it piques my curiosity that there is not a single word about his life between age 12 and 33ish. It appears his ministry began about 3 years before his execution. So what was he doing before then? Just building carts and tables with Joseph? The Christ doing nothing at all when, according to the Bible, he displayed spiritual precociousness at an early age? My question why wasn’t he doing this all of his life? Why did he virtually pop up out of nowhere and begin recruiting his apostles and starting his ministry at about age 30?
As you allude, one reason is because his early life is entirely fabricated and the authors didn’t fabricate his teens and twenties. OTOH, IMHO, there’s also the possibility that he left the region during a majority of that time, say late teens or so for a decade or so until he returned and began his ministry.