"Often I send the men not suitable for the whorehouses here."

He means the ones he's been hiring to guard the doors. Sometimes he gets the measure of a person ever so slightly wrong. Some of them he's dealt with himself, the truly vicious, the too arrogant, but others - those who know how to behave but don't suit the surroundings, well. He'd found new use for them elsewhere.

"Just look like you don't know how to keep your hands to yourself. That's usually the way of them."

Vaguely dark, as he takes up his drink again.

"Can I ask why first?"