Thousands of construction workers in Birmingham could be at risk from coronavirus as the government resists growing pressure to halt building work.