If Caavo keeps switching over to your Xbox One, it is probably because two things are happening. On your Caavo, you probably have Xbox One device configured for Auto-Switch and on your Xbox One, you probably have auto updates turned on.
When your auto-updates run, they turn the Xbox One system on which toggles your Caavo to switch inputs. You have two options to correct this behavior.
Option 1: Disable auto updates (both games and system) on your Xbox One.
Option 2: Disable Auto-Switch option on Caavo. Caavo will no longer be able to automatically switch to Xbox when powered on in this case. To do this, click your CAAVO button and navigate to Settings>Devices>Microsoft Xbox One>Control Settings>Auto-Switch Settings and uncheck the Power On option.