The demonstrators, many carrying British or English national flags, marched through Crowborough in East Sussex, where 27 male asylum seekers last week moved into the town’s training camp.
The camp has become the latest flashpoint as the UK government struggles to house asylum seekers, after popular opposition forced it to scale back use of hotels.
“Enough is enough. Crowborough says no. Deportation, no accommodation,” the protesters chanted, accompanied by drums and monitored by a small group of police officers.
“I’ve got six young children, and I’m not happy with it,” Shana Maria, a local resident said at the protest.
“The protection of my kids is paramount. We don’t know nothing about them.”
“We’re a small island, and we can’t afford to have too many migrants,” said fellow Crowborough resident, Harry Stepney.
“We haven’t got enough capacity, our NHS is filling up, it’s taking hours for English people to get doctor’s appointments.”
“I don’t blame the migrants, I blame our weak, faux-authoritarian government, which is imposing this on us,” said another local resident Keith Brown.
The Home Office plans to use the site to accommodate more than 500 single, male “illegal migrants” as it bids to end its controversial use of hotels for asylum seekers.
Sunday was just the latest of months of protests against the move, and the local authority has said it is considering legal action.
Interior Minister Shabana Mahmood argues that using military sites is vital to speeding up the closure of asylum hotels.
On a visit to Crowborough on Thursday, Mahmood vowed to defend any legal challenge “vigorously”.
“I understand the strength of feeling but I do also believe that getting out of asylum hotels is absolutely crucial for the country,” she told UK media.
On Thursday, the Government said 197 hotels were being used to house asylum seekers as of Jan 5, down from more than 400 at its peak under the previous government.
Police said no arrests were made at the protest, which was “peaceful”.
James Collis, Chief Superintendent of Sussex Police, said the local force was aware that housing asylum seekers at the camp has caused “concern” for residents and that “our police presence has been increased.”
Crowborough is one of two military sites the government announced last year would house asylum seekers, along with Cameron Barracks in Inverness in the Highlands.
The asylum seekers will be housed at the site for a maximum of three months while their asylum claims are being processed before being deported if their claim is rejected, the government said.