When a demolition team began work at a derelict Victorian guest house they all agreed that something didn’t feel quite right