When immigration minister Peter Dutton mistakenly sent a text message to a female journalist, Samantha Maiden, describing her as a “mad fucking witch”, the condemnations and demands for his resignation flowed thick and fast.

Protesters gathered outside Dutton’s electoral office in Brisbane proclaiming a “hex on sexism”. Penny Wong put out a statement calling on the prime minister to respond in what she described as “one of Malcolm Turnbull’s biggest tests so far”.

A change.org petition demanding Dutton’s resignation quickly got over 1,000 signatures, a call endorsed by Greens leader Richard Di Natale.

Of course, there is no denying Dutton is a sexist prick – the casual misogyny pervading Tony Abbott’s inner circle is anything but a state secret. But is a stupid sexist text message really his biggest crime?

Listen to these words:

“I felt like a piece of meat, these men continued to laugh they made me feel very uncomfortable and scared … the look in their eyes were they were undressing me, I was so scared. I was crying and shouting at them to please not make me take my clothes off, they kept insisting that I have to do what they tell me, it’s the new rule.”

This is the firsthand account of an Iranian refugee who was strip-searched on 11 December last year by male security guards in the Nauru Regional Processing Centre. She describes how the male guards from Australian firm Wilson Security were “laughing and leering” as they told her “you have to open your legs and hands so we can scan you with the wand”.

Peter Dutton is responsible for this abuse.

In a Senate committee hearing last year, the Department of Immigration confirmed that it has received 14 reports of sexual assault, 213 of physical assault and 798 of abusive behaviour in the offshore processing centre on Manus Island.

These cases compound the trauma many refugees have already suffered, and contribute to hundreds of incidents of self-harm and suicide attempts of those held in indefinite detention.

The most shocking example of the contempt shown by Dutton for women in detention came after the rape of a 23-year-old Somali refugee, Abyan, in July last year. Abyan became pregnant as a result of the assault. She was later found unconscious in her room as a result of the psychological trauma she was suffering. On 1 September she reported her pregnancy to the guards, and her lawyers began their appeal for her to be taken to Australia for an abortion.

It took over a month to get a response from Dutton. When Abyan was eventually brought to Australia for treatment, she was almost immediately deported after not being given adequate medical or psychological support. Dutton’s response to this torturous series of events was to issue a media release titled “Misleading Statements”, in which he accused refugee advocates of lying for political gain.

Where are the Labor Party’s demands for action when it comes to the ongoing treatment of refugees? They seek no change to the border policies that create these 21st century concentration camps in the first place.

Michelle Groeneveld from Save the Children described conditions on Nauru as a “boiling pot of despair”. Dutton is responsible for these horrors. So sure, call out casual sexism, but we should rage longer and harder against his ongoing crimes against refugees.