Campaigners say the dilapidated state of a former town hall and courthouse could put city centre regeneration plans at risk.