When I receive a phone call my ring tone volume goes up. How do I stop it?

I have a Samsung Galaxy Note 4 and I silence(no vibrate) my phone and keep it in my desk, no one touches it the whole day. The first phone call changes the ringer to vibrate, and every phone call increases the volume until it's at max ringer. I want a ring tone for the hours I'm not working so making my ringtone "none" every time would get tedious. So if anyone know how to stop this, I would be eternally grateful.