Seafood Scotland said up to a quarter of jobs in larger factories are unfilled, with fears the situation will get worse in the new year.
James Withers, chief executive of Scotland Food & Drink, said one business recently passed up £15 million worth of contracts because they did not have the staff to fulfil orders.
Meanwhile, he estimated Scotland's food and drink exports to the EU will have dropped by between £150 and £200m this year.
It follows the first 12 months of the post-Brexit trade deal between the UK and the EU, which came into force in January 2021.
Mr Withers said labour shortages due to Covid and Brexit are a "huge problem" across the food and drink manufacturing sector.
He told The Scotsman: "I think around 1.3 million people from overseas left the UK during the pandemic and went home.
"The problem is we are now stuck with a post-Brexit immigration policy which makes it very difficult for them to come back.
"At best, it's ineffective, and at worst, it's deemed as hostile to the very people we need to bring back."
Mr Withers said about 60,000 people work in Scotland's food manufacturing sector, adding: "Our estimate at the moment is that we are at least 10,000 to 11,000 staff short.
"Part of the challenge is that these aren't in Scotland's major cities.
"They are often in rural locations where food businesses actually employ more people than the population of the town they are in."
Mr Withers said there is a "desperate need to get a UK immigration policy that is effective and recognises what are structural gaps in the supply chain".
Donna Fordyce, chief executive of Seafood Scotland, said the industry is facing a "really challenging" situation as a result of Brexit and the pandemic.
She said labour shortages are now a big issue.
Before Brexit and Covid, she said, about 52 per cent of the workforce in Scotland's seafood industry was Eastern European. In the north east, it was around 70 per cent.
Ms Fordyce said the new immigration policy had "really stopped any new workforce coming in to replenish" those who leave.
Vacancies in some larger factories can be up to 25 per cent.
At one point, she said, a factory of 350 employees had 100 vacancies.
"Lines have been reduced," Ms Fordyce told The Scotsman. "They are not producing the volumes that they need."
She added: "People are turning away new business. It is really, really affecting the industry."
Ms Fordyce said there are concerns that some workers from Eastern Europe may not return to Scotland after travelling home for Christmas and New Year.
She said: "So that's a real fear as well, that actually there's going to be a bigger shortage after Christmas and New Year, because there is that risk."
She wants to see a review of the "shortage occupation list" to give seafood processing staff an easier route into the UK.
This could be on a transitional basis, she said, to allow firms to work towards introducing automation, which will take time and cost money.
Mr Withers said the youth mobility scheme visa could also be extended to EU countries.
Ms Fordyce said Brexit means "continual changes" for the industry, with new import checks set to come into force during January.
Both she and Mr Withers remember the chaotic scenes at the start of 2021 when the new post-Brexit rules first applied.
Mr Withers described the past 12 months as "torrid", adding: "No-one is going to mourn the passing of 2021.
"I think the challenge is, going into 2022, we are left with the legacy of what was frankly the worst possible Brexit at the worst possible time.
"And we have the spectre of Covid, a hospitality effective shutdown."
While labour shortages are one of the "painful legacies" of Brexit, Mr Withers said, the other is "just the simple fact that the EU is the most important market for Scottish food exports".
He said: "About two-thirds of all food exports that left Scotland going globally went to the European Union.
"And that has now become a much more complex, much more costly, and a much higher risk market to do business with.
"And that's largely because of the level of red tape, complex paperwork and cost that goes along with it."
Mr Withers said bigger businesses, such as large salmon companies and the whisky industry, have largely managed to adapt.
However, smaller businesses have been "really hammered".
UK food and drink exports to the EU for the first three quarters of 2021 were down by £1.3 billion compared to the year before.
Asked about the impact on Scotland, Mr Withers said: "My instinct is, the way things are going, by the time we get to the end of this year and when we look back, that our export sales to the European Union will be down somewhere between £150m and £200m."
A UK Government spokeswoman said: “Our points-based immigration system is delivering on the people’s priorities of getting businesses to invest in the domestic workforce while attracting those with the skills we need.
“There are many roles in food, drink and food processing sectors which are eligible under the points-based system, and we are closely monitoring labour supply and working with sector leaders to understand how we can best ease particular pinch points.
“We remain open to negotiating youth mobility arrangements with other countries within the EU and further afield, but these must be reciprocal.”