Fighting cats? Always fight female?

I have 3 cats. 2 males and a female. One of the males had babies with the female and we kept one. The other male is their baby. They don't fight a lot, but I was wondering why they will only fight the female. The father will never fight the other male (it's baby). I was wondering why they never fight each other, but never hesitate to mess with her?