It is. It is exactly that right that allows you to remain silent.
You’d have to say, “I plead the 5th”, which is not silence.
You are confusing the general 5th Amendment right, with Miranda rights which do derive from the 5th Amendment but are not exactly the same thing. Under Miranda you have the right to remain silent when in police custody.