Send keyboard shortcuts to guest and not host after switching workspaces on Linux

I have several workspaces inside an XFCE (though this also happened with Ciamon). Each workspace is used exclusively for a VM in fullscreeen mode. My problem is that whenever I switch to a different workspace using CTRL+ALT+right or left arrow keys, the fullscreen VM is not working as it should.

The problem is that key combinations, such as ALT+TAB are sent to the host, not the VM, forcing me to either minimize the VM and opening it again or using CTRL+F twice to have VirtualBox send the combination keys to the VM.

Does anyone has a solution? It gets very aoying, I constantly switch between different VMs.