The controversial Operation Fortitude, that was to include visa checks by the Australian Border Force across Melbourne's CBD, has been cancelled.

Victoria Police released a statement about 2.45pm, saying Operation Fortitude would not go ahead this weekend.

"We understand there has been a high level of community interest and concern which has been taken into consideration when making this decision. Victoria Police's priority is the safety and wellbeing of the whole community and we will continue to work with our partner agencies to achieve this."

It followed widespread condemnation from the public.

Protesters quickly took to the streets of Melbourne after the federal government announced that Australian Border Force would target potential visa fraudsters in the heart of the city.

About 50 people took over the steps of Flinders Street station early this afternoon and attempted to shut down the busy intersection of Flinders and Swanston streets, halting about 20 trams.

They waved placards and shouted slogans, describing the Border Force as "Nazi Gestapo" officers and chanting "no to racism, no to hate this is not a police state".

It came after the announcement that officers from the new paramilitary unit would join transport officials and police to target crime among people "travelling to, from and around the CBD", in a measure dubbed Operation Fortitude.

The union representing Australian Border Force workers welcomed the news that Operation Fortitude would not go ahead.

The CPSU has welcomed the cancellation, saying the operation would have potentially comprised the safety of their members carrying out their work.

CPSU National Secretary Nadine Flood said Australian Border Force employees had contacted the union expressing concern about how the event had been "politicised".

"While Border Force staff  have been involved in these types of operations before, they have never been publicised in this way. They were deeply concerned at the suggestion they would be stopping all people on the street, which is not how their work has been done in the past".

Ms Flood said staff are usually advised not to wear their uniforms in public, due to the nature of their work.

"We are calling on the Federal Government to stop cynically exploiting the work of the Australian Border Force for its own political ends, potentially putting these officers at risk."

The Andrews Government supported the last-minute cancellation.

Police Minister Wade Noonan said in a statement that the government had been notified of the operation this month and that it was intended to be a "standard police operation"

"We fully support the decision by Victoria Police to cancel the operation after the unfortunate and inappropriate characterisation by the Australian Border Force today."

Australian Border Force commissioner Roman Quaedvlieg conceded the initial press release announcing Operation Fortitude was a "clumsily worded" statement, released at the lower levels of the organisation.

He said there was never any intention for the Border Force to go out and proactively seek immigration breaches in Melbourne, and that ABF officers were to be present in a supporting role.

Mr Quaedvlieg rejected any suggestion Immigration Minister Peter Dutton or his office were involved.

Liberty Victoria president George Georgiou said there had been absolutely no need for the operation, which would have been a terrible waste of resources.

"It is not meaningful policy it is simply a show of force," Mr Georgiou said.

"We are concerned about the impact it has on the whole community.

"It is a great concern that officers will engage in racial profiling to stop and interrogate people."

- with Nicole Hasham, Liam Mannix and Richard Willingham