It was all in his head, that his father is haunting him. My understand is that as a serial killer son, he was being "brain-washed" by everyone that he'll be like his Dad. However, when he decided he wants to be with Ji Won, he tells his Dad to leave, and the Dad did. There was another scene he mentioned before that Ji Won had to be around him so that he won't see his Dad again. It just means he can't afford to lose her, with her, he can be a normal person.
This "seeing his father" representation was also used in Born Again. @SilverMoonTea can fill you in on that LOL.