Should I move closer for my dad?

I am 19 years old and I am the oldest of four. My mom died 4 years ago. The man who raised me is not my real dad, but I respect what Ryan has done for me over the years. My mom married Ryan when I was 6. But they had been together a while. And they already had my sister Chantelle. Now I have two siblings, Chantelle (14) and Cody (13). Ryan re-married a year and a half ago. I did not see eye to eye with his wife. We hate each other and when I turned 18 I moved away. I see my siblings on the weekends when they come to stay with me. Ryan's wife (Mary) has two daughters 13 and 10. Mary is annoyed that I wont take her kids on the weekends too. I have spoken to Ryan who said that Chantelle and Cody want to live with me and that they are not accepting Mary or her kids. He wants me to move closer and re-join the family because I am not really involved with the family. When I had my siblings at the weekend they told me that they want to live with me and that they miss our mom. We talk about her when they stay and they confided that they know Ryan is not my dad and they wish we were a proper family again (mom, Ryan and us). I have a job where I live and I have an apartment. I would hate to move, but I am conflicted. I don't think Ryan and I will ever be close, but he has said no to me having Chantelle and Cody. Please offer me some advice!