When he reincarnated, his mother and father were strangers to him. He had already lived a life before his death and had lost the people he cared for that former life, so didn't want to be emotionally attached to them and have any reason to do so.
Especially since his new family could be accidentally killed or intentionally beaten to death by one of their masters. The loss of his previous family was still fresh and he didn't want to go through something like that so soon. Besides, it wasn't as if he could easily replace what he lost for some strangers he just met.
Over time, he grew closer to them. But they only became acquaintances who lived with and suffered with him in their humble abode. He found commonality in their identities as slaves that made him sympathize and empathize with them.
However, he didn't become closer to them beyond that. This was intentional, and he did it so that he wouldn't have to experience what his father was currently experiencing in this memory.
