Overlooking the famous Downing Street home of the British prime minister is a grand Victorian building that many visitors give but a passing glance.