The trouble with using the "fascist" label as an attack on your political opponents comes from widely divergent views on exactly what a "fascist" might stand for.
This public confusion stems from at least 2 factors:
First, fascist governments themselves often differed on their exact philosophies. While Mussolini could well be considered the first fascist leader, his version of fascism differed in several aspects from the best known fascist leader, Adolf Hitler. Further complicating matters is the fact that both Hitler and Mussolini held wildly conflicting and contradictory ideas at different times of their lives. In their youth, both were passionate communist activists. Years later, they would torture and persecute communist sympathizers. Assembling an easy definition of what "fascism" stands for as an ideology is not a simple task given the contradictory policies of the few historical examples of fascist nations.
Secondly, modern political culture obscures and reshapes the positions of the fascists to fit its own needs. Pundits of all ideological stripes are quick to paint fascism as something supported by their opponents. Every ideology attempts to take a different moral away from the storyesque rise and fall of fascism.
To a communist or socialist, the fascists were murderers and tools of the bourgeouise. Because Hitler did not abolish private property and persecuted communists (despite having been one himself in youth), economic leftists are quick to paint him as their ideological opposite; conveniently ignoring the massive government involvement in the German economy and the organization of industries by the state in a hybrid economic model called corporatism.
To a libertarian, the fascists represent the absolute opposite side of the political spectrum; the most blind supporters of a powerful central government that regulated virtually every aspect of the lives of its citizens. Libertarians are quick to intentionally overlook Hitler's defense of private property and hostility toward communists, two traits which they also share.
"Fascist" is a powerful political attack, and we must expect that we will continue to see it hurled in the political arena for quite some time. No one wants to be connected to the horrors of the Holocaust and the evil perpetrated by Mussolini and Hitler, even through shared positions.
But it is also an unfair one. Every self-identified Communist government in the history of the world has at one time or another been involved in the wholesale slaughter, starvation, or persecution of elements of its population. But it would be a childish ploy for me to say that because modern liberals share communist positions about nationalized healthcare and domestic issues, they support the slaughter done in the name of communism. We should make more of an effort to avoid the intellectual laziness of attempting to discredit an ideology by highlighting one of its controversial leaders.
Modern Liberals and Communists should not be forced to defend Stalin's persecution of his countrymen.
Free marketeers and libertarians should not be compelled to defend the legacy of Augusto Pinochet, a butcher and ultracapitalist.
In summary I don't take issue with the use of "fascist" as a political attack, so long as it is employed with some degree of objectivity and understanding of the contradiction in terms it represents.