James Packer’s Barangaroo casino project – 60 storeys of steel and glass stuffed full of exclusive penthouse suites, signature restaurants, swimming pools, luxury sundecks and VIP-only high-roller gambling – is set to cast a long shadow across an area that was once a Sydney working class heartland.

Barangaroo, the waterfront area where the development is to take place, has a history at odds with Packer’s playground for the rich. The area is named after the proud Cammeragal woman who fought against the invasion of her land by the First Fleet marines.

Barangaroo is the site of the “Hungry Mile”, the stretch of Hicksons Road where, in the words of wharfie poet Ernest Antony, workers tramped “in their legions on the mornings dark and cold, to beg the right to slave for bread from Sydney’s lords of gold”. It is also where many of the struggles that forged the powerful wharf unions were fought.

The waterside workers of Sydney had a reputation for political sophistication and militancy. They forged unions that would put fear in the hearts of the rich, and stood side by side with those campaigning against militarism, the Vietnam War and apartheid and for countless other causes.

Packer’s power as Australia’s third richest man and the political connections built up by his empire go a long way to explaining the enthusiasm his every move receives from both sides of parliament. When “good will” like that doesn’t work, there’s always the more forceful approach: he is under investigation from the ACCC over a meeting on his private yacht with the head of rival casino operators Echo Entertainment. Packer reportedly told them that he’d trample on Echo’s turf in Brisbane unless it ‘‘behaved’’ itself ‘‘vis-a-vis Sydney’’.

Lend Lease, the building company redeveloping the site where the casino will stand, has found that the fighting history of the area has not been forgotten. Workers have struck for and won improved conditions on the site, notably as part of a national campaign in 2012 that included determined picketing.

Alongside that struggle there have been repeated disputes over safety, with various hazards still buried in the ground from the area’s years as a working port. Brian Parker, state secretary of the CFMEU, told the media that workers were finding asbestos “all over the place”. The union has enforced safety precautions, and even had to warn the company that asbestos fibres could blow into the nearby Lend Lease childcare centre. The centre was later closed unexpectedly – though the company repeatedly claims it was “totally unrelated”.

Try as they might to turn Sydney into a playground for the rich and powerful, Packer and his like can’t totally erase the proud working class history written through decades of struggle and solidarity. It’s a history still being written today, even in the shadow of VIP casinos and luxury penthouses.