A few weeks ago, I turned down the offer of a freelance assignment: an article on the implications of Brexit for UK employment law. Well, the day job is pretty busy, and frankly the subject matter didn’t float my boat. Because I’m pretty sure a ‘leave’ vote in June’s referendum would have relatively little impact on workers’ rights in the UK – at least for the foreseeable future.

I wasn’t persuaded from that view by the publication, a few days later, of a 17-page TUC report on the issue. “Many rights accumulated over decades, including paid annual leave, time off for antenatal appointments and fair treatment for part-time workers, are ‘used every day by millions of workers’”, the Guardian reported the TUC as saying. But if the UK votes to leave the EU, “no one can say what will happen to these rights”.

A 17-page history of EU-derived workplace rights – some of them key, some of them fairly marginal to many workers’ lives – just to say “no one can say what will happen to these rights”. Well, yes, no one can say for certain. But, if we apply our minds to it – rather than just follow orders from Stronger In campaign HQ to scaremonger about our specialist area – we can probably say that the chances of Brexit leading directly to any significant erosion of those workplace rights are actually pretty slim, for several reasons.

Firstly, it would be a bold (or simply daft) Tory government that decided to cut the hardly over-generous statutory entitlement to paid holiday – just four weeks plus bank holidays – or to blatantly roll-back the scope of anti-discrimination law, or to slash (paper) maternity rights. These rights are now well-entrenched and, in most cases, extremely popular. So, even someone as tactically inept as George Osborne would surely see that “Vote Tory, get less holiday” is not a great campaign slogan.

Sure, some EU-derived rights are much less widely used, so would be less well protected by public support. For example, hardly anyone takes unpaid parental leave, for the simple reason that it is unpaid (though I did). But post-Brexit Tory governments would have many more pressing things to do with their parliamentary time than abolish or amend the largely forgotten parental leave provisions in the Maternity & Parental Leave Regulations 1999, at least for some considerable time. And who can say for sure that, 15 or 20 years from now, the EU itself won’t row back on such rights? Answer: no one.

The EU isn’t some kind of permanent, bleedin’ heart liberal institution. It’s made up of elected governments and politicians. Put more right-wing politicians in, and you get more right-wing (and fewer left-wing) policies out. Unfortunate, but that’s the way it is.

However, the principal reason Brexit would most likely have limited impact on workers’ rights in the UK is that Tory governments – including those aided and abetted by opportunistic if short-sighted Liberal Democrats – simply don’t need Brexit to erode and even destroy such rights. They manage perfectly well as things are.

The outcome of June’s referendum won’t make any difference to whether employment tribunal (ET) fees are abolished or lowered. Yet, by reducing the number of ET cases by 65%, those fees have given a green light to rogue or merely unscrupulous employers to mistreat and abuse their workers, in the almost certain knowledge that they will face no legal consequences. Rights on paper are worthless if they cannot be effectively enforced. And that applies to EU-derived workplace rights just as much as it does to domestically-generated rights. If proof of this were needed, just look at what’s happening with those partly EU-derived maternity rights.

Those justice-denying fees are arguably the single most important current issue when considering the threat to workers’ rights in the UK, yet this weekend Will Hutton failed even to mention them when writing in the Observer that “leaving the European Union – so suspending the array of protections it offers in the workplace – would erode what remains of British worker rights and entitlements.” That is a remarkable oversight. For what use is the ‘protection against detriment’ set out in Regulation 19 of the Maternity & Parental Leave Regulations 1999 if you cannot afford (or are simply not willing to risk) the £1,200 fees to pursue an ET claim against your rogue (former) employer?

A vote to remain in the EU won’t reverse the Coalition government’s erosion of the legal protection against unfair dismissal, or bring back legal aid for employment advice. Nor will it improve the currently pathetic enforcement of the national minimum wage, or reverse the shocking Tory assault on trade unions (and Labour Party funding) via the Trade Union Bill (which Will Hutton rightly describes as “one of the most pernicious pieces of proposed legislation in recent history”). And, against that background, it’s far from clear how it will help combat the growth in zero-hours contracts, bogus self-employment and other forms of precarious employment that has blighted so many workers’ lives.

None of which is to say that the EU-derived rights set out in the TUC’s report are unimportant, or that anyone should vote for Brexit (I certainly won’t be doing so). Apart from anything else, having a floor of basic workplace rights across the EU is crucial to ensuring a level playing field for British businesses (and their workers), whether the UK is in or out. And the European Court of Justice has in the past played an important role in driving forward and enhancing key workplace rights, not least that to equal pay.

Nor is it to deny that there might well be some impact of Brexit on workers’ rights. Darren Newman notes that Brexit could “raise some issues at the geeky margins [of employment law]” – what a wonderful phrase that is – and Michael Reed of the Free Representation Unit similarly suggests that discrimination law “might well drift at the margins”. As for geeky things like TUPE, David Whincup suggests – in his must-read blog earlier this month – that “any attempt to re-write TUPE would inevitably end up looking pretty much like TUPE”.

But to my mind the argument that Brexit would be “disastrous for rights at work” is thin and unconvincing, if not confected. Right now, the key battles over workers’ rights are wholly domestic. And they are mostly being lost.