That would make sense if you’re coming from using the Astro mixamp on the Xbox, since that was already giving you Dolby Headphone surround processing (As opposed to Atmos). But for me, coming from plain old stereo, the bump is noticeable!
You might need to toggle the Atmos setting on and off each time you restart your Xbox. Although I’ve only had this problem once myself, I’ve seen this reported in a few places online.
Glad you’re having fun and that I could help!