Family MESS!!!!!(PLEASE READ ALL)?

Background story about me so when i was 21 i met a beautiful women at university we fell in love and after 2 years together we decided to get married, soon after the marriage she was expecting, it was the best day of my life 9 months later our son was born. we lived a happy life about two years later i found out that she had a brief affair. i confronted her and she admitted it i kick her out the house after a few months of thinking of our marriage i decided that i could not forgive and i filed for divorce, we both decided to have joint custody of our son. A week after our divorce my ex wife started dating a thug she later fell pregnant by him and had twin daughters and a year later they had a baby son. he was violent to her and abused her. My ex wife boyfriend was later arrested for violence and was sentence to prison for 2 months after he got out of prison he abandon the family and his children. Now here is the REAL problems every time I visit my son i look at the other children and feel sorry for them. I take my son to two holidays a year and give him everything. but his halts siblings don't really have much. I buy them things when i visit my son so they don't feel left out but i scared that they are going to see me as a father figure and call me dad which i don't want i only want to be a father to my son only. I feel so guilty that those children had such a useless father but I dont want to have a close bond with them, I want them to see me as a family friend but the more i see them the more they will see me as their dad AM I WRONG FOR FEELING THIS WAY!