The solution for this might sound more complex than it is, the Bitpool for the Bluetooth Audio Agent by default is 35, this seems to be too low if you also use a mouse and keyboard at the same time. In my case only a mouse. Raising it to a higher value solved this problem for me.

I raised it to 50 and it worked perfectly!

You can edit the preference file with a plist editor in this file: ~/Library/Preferences/com.apple.BluetoothAudioAgent

Or, more simple, you can run the following in the program Terminal located at /Applications/Utilities/Terminal.app

At the command line enter the following command:

defaults write com.apple.BluetoothAudioAgent "Apple Bitpool Min (editable)" 50

After doing this you might need to reconnect your device or reboot your Mac to apply the change. In my case turning off the bluetooth adapter and enabling it again was enough.

I tested this with a Sennheiser PX 210 BT and it works perfectly!