ODrive keeps loosing connection vis USB with RasPi

I have a Raspberry Pi 4 Model B which is connected to ODrive controller.
The Raspberry Pi runs on Ubuntu MATE 20.04.
The communication between the Raspberry Pi and ODrive controller is via USB.
The ODrive controller is powered by a 2200 mAh 42 V LiPo battery and the Raspberry Pi is powered by a USB powerbank (5V, 2.4A) .

The connections are shown in the figure below.

With the setup shown above, I tried running the ODrive tool on Raspberry PI.
ODrive connects and I see no problem.
But when I try to go to the CLOSED_LOOP_CONTROL mode, I loose connection.

Further, i thought that since the two devices are powered by different power sources, I should have a common ground.
Thus I connected the GND pin on ODrive controller to the GND pin on Raspberry Pi.

The connection setup for that is shown below.

With the setup shown above, I again tried the odrivetool.
This time the connection does not drop when i set the axis state for both axes to CLOSED_LOOP_CONTROL mode but again tend to see connection loss while trying to set the velocity for the motors.

What could be the issue and how do i fix this ?

This is fairly common. Do you have ferrite rings on the motor wires? Adding them tends to make a big difference. Ferrite Ring ESD-R-28C-1 — ODrive