That's not the first time he's been called a ghost. Doc can only imagine the reaction when he has to tell Raylan the whole story. He's already left out a lot of details, as much as he can get away with that'll still let what he has said so far make sense.

"Would you like to tell me a story of yours? Either one that makes sense or does not." He is willing to listen and reserve judgement. They can hang around a little while longer before Raylan drags him off somewhere more habitable.