“A spectre is haunting Europe, the spectre of Communism.” When I first read these opening lines to Marx and Engels’ Communist Manifesto, sometime in the mid-1990s, I was a wound-up but somewhat naive lefty university student. But it was apparent then, even to me, that the spectre of communism was not haunting anyone much any more. Capitalism, it seemed, had won. I was too young to have witnessed the ideological onslaught of British Thatcherism in the early ’80s, and how it flowed through to Australia via the ALP, but her central message – “There is no alternative” – was the background noise to my generation, whose first real political memory was the fall of the Berlin Wall and the collapse of the USSR.

A lot has changed since then. In the decade since the 2008 global financial crisis, and in the context of a rapidly escalating climate emergency, millions of young people have begun to challenge the legitimacy of a capitalist system that has singularly failed to offer them a secure future. And socialism, once considered a relic of history, has once again come to signify for large numbers the possibility of a real alternative to the status quo.

In the US, signs pointing to the rising popularity of socialism have been evident for some years. But this did not seem to impact too much on the minds of the defenders of the system until Bernie Sanders romped home in the New Hampshire primary and the Nevada caucus last month. Over about a week, MSNBC, the Fox News of the Democrat establishment, and Republican never-Trumpers completely melted down. Chris Matthews, one of their most rabid talking heads, raved about Sanders’ triumph in Nevada, comparing it to the Nazi conquest of France. Full-blown panic set in as the months-long strategy of hoping the Sanders campaign would just disappear turned to dust. The day after Nevada, the liberal establishment press ran a barrage of hysterical op-ed pieces bleating about the perils of socialism and demanding that Sanders be stopped at any cost.

The spectre is well and truly haunting them again. For socialists like me, the establishment’s fear, and the fact that large numbers of people in the US are looking for something better than the neoliberal consensus that has dominated US politics for so long, are both extremely heartening. But satisfying as it is to see the incredulity on the faces of entitled cable news pundit airheads, the rising popularity of socialism demands more than just cheerleading.

Socialism, after all, has been popular before. For much of the last century, millions of workers in Western societies like Australia considered themselves socialists. Labor Party leaders openly embraced the term. Here the Communist Party commanded the loyalty of tens of thousands and had a broad influence in the unions and many working class communities. And yet, by the last decades of the 20th century, both of these currents of socialism had exhausted themselves – here and around the world. Communism had become synonymous with authoritarianism, bureaucratic rule and the suppression of individual freedoms. Democratic socialism – or social democracy – had been abandoned by parties like the ALP that embraced market capitalism and were increasingly indistinguishable from their right wing opponents.

We can’t let socialism fail again. The planet can’t afford it, and neither can the people on it. There are a lot of things that go into making sure of that, but one of them is looking at the socialisms of the past and why they were ultimately unsuccessful.

The two main wings of 20th century socialism emerged out of the First World War and the Russian Revolution. On one side were the social democrats who supported their own side in the war. On the other side were the internationalists, led by the Russian Bolsheviks, who had opposed the war and in 1917 led a revolution that overthrew first the tsar and then the capitalist provisional government, establishing the world’s first state based on a self-organised workers’ democracy.

In the years that followed, revolutionary uprisings across Europe were beaten back with the connivance of the social democrats, who, it turned out, opposed workers rising up against capitalist rule with the same enthusiasm that they had backed parliamentary resolutions sending workers to their deaths in the trenches.

This vicious betrayal left the Russian Revolution isolated, and within a few years the fledgling workers’ democracy was overturned from within as a new bureaucratic ruling class, led by Joseph Stalin, took power and proceeded to rebuild the Russian state by brutally exploiting the working class and suppressing all the democratic and emancipatory aspects of the 1917 revolution. The revolutionaries who resisted Stalin – led by Leon Trotsky – were physically exterminated in Russia and driven to the margins of political life in the rest of the world by Communist parties that were thoroughly under Stalin’s thumb.

This led to a situation in which, by the 1930s, the two main organisational and ideological representatives of socialism – Stalinism and reformist social democracy – had virtually nothing in common with the socialism that had dominated the working class movement less than two decades earlier and during the revolutionary period at the end of the war.

Prior to World War One, the socialist movement – which was organised into enormous socialist parties – was united by a view of socialism that was about ending the rule of the capitalist class and instituting a new society based on the collective, democratic rule of the working class. Authoritarianism was as anathema to it as was the idea that socialism was simply about tinkering with the system to make it more bearable for workers while leaving the basic structure of capitalist rule intact.

During the war, it became clear, first to a few socialists grouped around Lenin, and eventually to many millions, that socialism would not come about simply by a process of slow decline of the capitalist system. Power would have to be wrenched from the grasp of the capitalist class, which would stop at nothing to prevent an end to their system.

But when the revolutionary movement failed, triumphant Stalinism and reformism combined to transform totally the popular meaning of socialism. For the reformists – who in following decades, and especially during and after World War Two, took the reins of parliamentary governments in Western countries (including Australia), socialism was about managed capitalism – the bosses could still make money and exploit their workers, but social democratic governments would fight for decent pensions and unemployment benefits, introduce public health systems and legislate to protect workers’ rights – all worthwhile and in some cases life-saving reforms, but a distant cry from the idea of socialism as the fundamental transformation of society.

The Stalinist version of socialism was in some ways much more far reaching. In the USSR and the Stalinist states that were built in its image after World War Two, private ownership of industry was abolished. In theory, the resources of society were owned in common. But in reality, “common ownership” simply meant, just as in the West, control of the means of creating wealth by a tiny minority – only in this case the minority was the top state bureaucrats rather than private capitalists. The truth of this was revealed when the USSR finally fell, and the top apparatchiks of the Soviet regime transformed into the multi-billionaire oligarchs of the new Russian capitalism.

The collapse of Stalinism in 1989-91 demonstrated once and for all that the so-called communist regimes were, just as much as the West, fundamentally hostile to the interests of the people who lived under them. The collapse of social democratic reformism was less spectacular, but no less assured. Essential to the idea of managing capitalism in such a way as to improve the lot of workers is a commitment to managing capitalism. That means, because capitalism is a system premised on the exploitation of the working class for profit, being prepared to do whatever it takes to maintain profitability.

During the long boom after World War Two, there was enough meat on the bones of the system for reformist leaders to be able to straddle the divide – granting real reforms that improved workers’ lives while at the same time keeping the capitalists happy by ensuring their profits kept coming in. This all collapsed with the onset of economic crisis in the 1970s, and reformist parties all around the world – in Australia it was Gough Whitlam’s government – were faced with a choice of defending workers’ incomes, entitlements and workplace rights, or savaging those rights to prop up capitalist profitability. Without exception, they chose the latter. From the 1975 Terrigal conference, at which the ALP first voted to give its support to private enterprise and profitability, it was a steep and steady slide into the full embrace of neoliberalism that characterised Labor from the 1980s and 1990s up until today.

Throughout the 20th century there were innumerable movements of rebellion and resistance – from workers’ strikes and uprisings to national rebellions against imperialism to struggles for women’s rights, against racism and so on. Those movements almost always tended towards a socialist world view; socialism, after all, is nothing if not the expression of the ideals of solidarity and struggle.

And in many instances, the fact that these movements expressed real collective struggles for liberation meant that, in spite of the Stalinist and/or reformist framework that dominated, the movements carried out in their name were in fact truly heroic. Think of the communists in Australia who built a mighty movement of the unemployed during the Great Depression, or fought for Aboriginal rights, or helped organise the epic 1968 mass strike that smashed the anti-union penal powers. Even the ALP had its moments – like the campaign by Labor leader Doc Evatt against Menzies’ attempt to ban the Communist Party in the 1950s, or the struggle of the Labor left against the Vietnam War.

But the constraints were real and debilitating. Inside the movements, reformism and Stalinism derailed people’s radical instincts and encouraged (in many times enforced) rotten compromises and illusions in politicians who inevitably sold struggles short. More fundamentally though, the reformist and Stalinist organisations that dominated the socialist movement didn’t overturn capitalism. In fact, at crucial moments, they gave it the breathing space to survive and in the end crush all opposition.

Today we face a world capitalist system more rapacious, more violent, more indifferent to human suffering or indeed to the future of the human species than at any time in history. The revival of the idea of socialism – the hope for a fundamentally different world, one organised to secure the existence and enhance the lives of all of us, rather than protect the profits of a tiny few – is long overdue. But we need to talk seriously about the meaning of socialism, and about what will be necessary to achieve it. We can’t afford to fail again.