Networking error in my host VM form the moment I connected the BBB

I am cross-developing some software for beaglebone, using Linux VM (over Win10 host).
When I first connected my BBB, I connected it through the USB cable and used the ethernet via USB connection to connect to the beagle (, but it kept doing connectivity issues, so I now have a permanent IP address and I connected via ethernet cable.
Yet, I get my VM (using Oracle VirtualBox) keep getting network errors:
Activation of network failed
And I feel it influences my in-VM-browsing, making it slower and sometimes unavailable.
Does any one experienced this issue? a way to solve it?