Hey, it's a manor anyway. You can always find a reason to go see "the big manor that stand out compared to the rest of the village.
Once after the party arrived somewhere, we just get to the tavern to get the basic information about the place and who rulled it. Then i just said i wanted to pay my respect the important person who must live in the castle.
That's how i justified getting straight to the big castle instead of spending too much time turning arround.