So, do you call people other than "black people" the n-word? If you just reserve it for people of color, then yes. It IS a racist term. Context also matters. If it's used in anger towards a person of color and no one else, then yes, it IS a racist term. Even if you are just using it for the shock value to shut your opponent up, it's still a racist term.
The term "white nigger" has a history going back over 150 years. In the 1840s and 1850s, Irish Americans were referred to by that term. Robert Byrd, former Klan member AND 60 year elected member of the Democratic party used it twice in a speech in 2001.
I noticed you avoided the question about a black person calling another black person by the terms that offends you. That tends to be the way things are done.
Context does not matter. That's the excuse used by those that have the hypocritical mindset of it's only racist when whites use it. You're claiming the TERM is racist then saying it's not based on who says it.