"He would have known." In Raylan's case, probably later rather then sooner, but he's supposed to be a US Marshal or something despite being a dumbass, and it would have come out anyway one way or another.

"Broken men are easier to love." Although who he is referring to there is left deliberately ambiguous. Aren't all the men in 1306 broken in some fashion?