He loves you. Even though Raylan hadn't talked to Neal since that fight, he knew that he still couldn't say anything. Wasn't right for him to go blowing those things up.

"Without his direct sayin' so, goin' home is just flaunting the fact that you haven't kicked me out yourself. Yeah, he was mad on your behalf and whether that's right or wrong it between you two but. It's about respect. A man's pride." A man's heart.

"If we're really a family here, no one can run around undoing things that we say. It's not fair to him." Never mind that it wasn't fair to Raylan either, but he was a big boy. He could handle the unfairness of it all, and shoulder his share and then some.