Even those folks wouldn't be able to exorcise it, actually! The Drillmaster is something that feeds off your self-doubt and uncertainty, so it's there because you're there. You're largely meant to just get out of its way and navigate the house amid the constant danger of encountering it.

You can make the Drillmaster disappear out of your path if you face/acknowledge whatever weakness (true or self-perceived) you have that has generated the most criticism or anguish — but even then, you're not killing it, as much as... temporarily getting rid of it. It'll still be there for the next house entrant to encounter.