I imagine this hideout high up in the mountains with treacherous cliff sides below and above. This location is hard to get to and even harder to get in. As a DM I recommend being generous with allowing the players to sneak into the place. There are opportunities to sneak inside from all sorts of angles.
The place was designed to work as both a bandit hideout as well as a friendly tavern. It could even double as a lair for the players themselves.