In the scene, Sam is skeptical about the person being referred to, prompting him to question whether this individual is indeed the one in question. He then examines the man's appearance, noting that the choice of attire is somewhat better than expected, which gives him enough confidence to make a bold statement about the guy's past.
Despite the man's unkempt look, Sam recognizes that there's more to him than meets the eye. He acknowledges that this person has a troubled history, having spent more time in jail than he has, and he believes there's an unexpected toughness beneath the surface of the seemingly nerdy exterior.