Who said they couldn't leave? Buchanan certainly never said such a thing. It wasn't until after the CSA attacked America that it was suddenly called an unlawful rebellion.
Now you're trying to rewrite history.
There was no attack.
To answer your question, Lincoln.