Brilliant deduction. The castle fell through the cosmic cracks, and took anyone who was within its walls and grounds at the time with it.

It is connected to other collections of things that have fallen here. We've had folk wander here before from them. I am not sure the road is open now; some major disruption occurred, and even without such the way does not open easily or swiftly.