Why couldn't the man and mother marry in the story Love Must Not Be Forgotten?

I've read the story a couple times, but the reason the mother and guy cannot marry isn't clear to me. Is it because he is married? Have they just not admitted their love to each other? Or what else?