He hasn't asked himself why he stayed, if he's honest. As Malcolm said, he could have gone elsewhere. He has rationalised it as Raylan needing the help, not minding the company. He has turned Malcolm into a project in his mind. He has done everything except allow himself to ask what he wants, not because he is not a selfish man, but because he can't be thrown out of somewhere he doesn't let himself belong to in the first place. He will always have the choice to walk away if he never settles down, even if he wants to stay.

"It is." He doesn't feel the need to say anything more than that, having let Malcolm scale enough of his walls already. He will get defensive if Malcolm breaches any further into his guarded fortress.