USB reliability when in AXIS_STATE_CLOSED_LOOP_CONTROL

#1

Hi, thanks for an awesome piece of hardware!

Unfortunately I am experiencing USB connectivity issues but only when the motor is driving, or even just holding a position with no load on it.

For example:
USB is stable until I enter command:
odrv0.axis0.requested_state = AXIS_STATE_CLOSED_LOOP_CONTROL
At which point the Jetson TX2 will repeatedly disconnect and reconnect (increasing the device number each time) until it gives up. Device fails to enumerate.
Also tested on two windows machines which seem more resilient but both have lost or failed to connect completely at times.

oDrivetool 0.4.11
Hardware V3.5-48 & V3.6-56
Firmware v0.4.10
440W motor using Hall effect sensors with the capacitor mod done as advised.
Have seen this happen on multiple oDrives, on multiple controlling machines and with 2 different 48v PSUs.

At first I thought the drive signal to the motor was coupling onto the 3v3 and 5v rails (probed on J4) but after probing with an as short a ground wire as possible I saw that there is only ~10mV of the signal present. Still it is only when this noise is present that the USB becomes flakey.

I have been having this problem intermittently for a while but kept ignoring it or thinking I had fixed it. It is now preventing me from having a reliable setup. Have spent a few hours reading through others USB issues but none sound quite the same.

Any suggestions greatly appreciated.

Config dump:
user_config_loaded = True (bool)
vbus_voltage = 48.26491928100586 (float)
system_stats:
min_stack_space_comms = 12956 (int)
usb: …
uptime = 363715 (int)
min_stack_space_startup = 612 (int)
min_stack_space_usb = 2768 (int)
min_stack_space_axis0 = 7828 (int)
min_heap_space = 4216 (int)
min_stack_space_usb_irq = 1784 (int)
min_stack_space_axis1 = 7828 (int)
i2c: …
min_stack_space_uart = 3932 (int)
brake_resistor_armed = True (bool)
axis0:
watchdog_feed()
requested_state = 0 (int)
current_state = 1 (int)
lockin_state = 0 (int)
motor: …
loop_counter = 2896952 (int)
error = 0x0000 (int)
controller: …
step_dir_active = False (bool)
trap_traj: …
sensorless_estimator: …
encoder: …
config: …
save_configuration()
test_function(delta: int)
test_property = 0 (int)
can:
TxMailboxCompleteCallbackCnt = 0 (int)
unexpected_errors = 0 (int)
node_id = 0 (int)
received_msg_cnt = 0 (int)
received_ack = 0 (int)
TxMailboxAbortCallbackCnt = 0 (int)
unhandled_messages = 0 (int)
get_adc_voltage(gpio: int)
fw_version_minor = 4 (int)
reboot()
fw_version_revision = 10 (int)
axis1:
watchdog_feed()
requested_state = 0 (int)
current_state = 1 (int)
lockin_state = 0 (int)
motor: …
loop_counter = 2897017 (int)
error = 0x0000 (int)
controller: …
step_dir_active = False (bool)
trap_traj: …
sensorless_estimator: …
encoder: …
config: …
fw_version_unreleased = 0 (int)
get_oscilloscope_val(index: int)
serial_number = 336431503536 (int)
erase_configuration()
hw_version_variant = 56 (int)
fw_version_major = 0 (int)
hw_version_major = 3 (int)
enter_dfu_mode()
hw_version_minor = 6 (int)
config:
gpio1_pwm_mapping: …
gpio4_pwm_mapping: …
brake_resistance = 2.0 (float)
gpio2_pwm_mapping: …
enable_i2c_instead_of_can = False (bool)
gpio4_analog_mapping: …
enable_ascii_protocol_on_usb = True (bool)
gpio3_pwm_mapping: …
gpio3_analog_mapping: …
dc_bus_overvoltage_trip_level = 59.92000198364258 (float)
enable_uart = True (bool)
dc_bus_undervoltage_trip_level = 8.0 (float)

#2

Hi @Junglist,
This is almost certainly a grounding issue. What’s likely happening is that the ground connection is poor (or has high impedance at high frequencies), and some current is going via the USB ground. Please make sure that:

  • Motor and battery -ve are both connected to chassis or mains earth.
  • ODrive has a good heavy ground connection to the battery
  • Odrive supply and ground return must go directly to the battery, not through the same distribution terminal that feeds the Jetson
  • Ideally the motor wires and the power supply wires should be twisted, to keep the inductance down, and avoid magnetic interference with nearby circuits.

This is pretty common on embedded systems where the motor controller and the USB host are fed from the same floating DC supply. If you can’t solve the USB issue, try CAN bus, which is specifically designed for this sort of problem.

Or else, try a USB isolator. I used one on my old robot which was using RoboClaw controllers with a Jetson, it solved all the USB issues.

#3

Hi Towen,

Thanks so much for that. Makes a lot of sense.
I am using a mains powered PSU as the robot will eventually have a generator on board.
I shall improve -ve connection to the PSU tomorrow and retest.

1 Like
#4

A generator? :]
Sounds like an interesting project!

#5

In the video it is powered from two 12v AGM batteries with an inverter, run time <40 minutes so not a long term solution but has allowed me to start playing with ROS navigation.
I have developed a steered wheel and am very close to having a 4WD, 4 wheel steered robot, just this USB issue holding me back now.

1 Like
#6

Very nice - but what’s the inverter for? Wouldn’t it be better (and more efficient) to run from a 48V Lithium battery?

#7

I agree entirely!
However I already had the AGMs and with the long term power source going to be a generator (plus with already having the mains powered PSUs for everything on it) then decided to go for the inverter as was way cheaper than a decent capacity 48V battery.

#8

Did some tests - no luck so far.
The cable length from PSU to ODrive is 2m long and was originally 1.5mm2 copper. Testing done just holding the motor position with no load.
Increased -ve to 3mm2 - no difference.
Increased to 10mm2 (with a few strands cut off one end just to fit in the connector) - no difference.
I’m definitely not prepared to go larger than 10mm2 so will get a USB isolator.

I would try CAN but having not used it before and it resulting in a fair amount of software changes then I’ll try stick to USB first.

Will get one of these unless you think it is worth spending more. It looks to be suffice and a lot of the more expensive ones I saw still appeared to use the ADuM3160.

#9

Interesting that it’s still a problem - did you make sure the PSU ground is connected to earth? How is the Jetson being powered?

That USB isolator should do the trick, but beware that it kind of sweeps the problem under the rug, i.e. you could still run into other EMC related issues in future until you fix the grounding

#10

Just double checked - the PSU is definitely earthed.
The Jetson is powered using the mains PSU that comes with the dev kit.

Both the 48V PSU for the oDrive and the Jetson are plugged into the same 25m extension lead.
I did try putting them on separate 25m leads but it made no difference.

So this is where my understanding beaks down - both the 48V PSU and the Jetson PSU have their 0V isolated from mains earth. So why does the PSUs themselves being earthed make any difference? Is it not just the impedance back to the PSU from the oDrive that matters?

#11

Sorry, when I said PSU ground, I meant the 0V output. THAT should be connected to mains earth.
Otherwise, the only path to mains earth might be via the USB.

Basically, you don’t want the grounds (0V) on the two boards (Jetson and ODrive) to be different, otherwise USB will have problems.
USB is designed to either connect to a device whose ground reference is mains earth OR, power the device directly from the bus.

In the embedded system I would connect each of the ODrive 0V and the Jetson 0V to the vehicle chassis.

#12

I think I finally understand now. Please correct me if I am (still) missing something:
So mains earth in this scenario is literally just serving as a connection point rather than using its property as earth, so that the 0V from two different PSUs are connected together. I had not realised that the 0V connection that the USB cable was providing between the two was not suffice for reliable comms.

All that said the problem persists… I did the following:
Connected 48V PSU 0V to mains earth via a very short 1.5mm2 cable.
Connected Jetson 0V to mains earth via short 1.5mm2 cable.
Resistance from Jetson 0V to 48V PSU 0V is now 0.19 Ohms.
Resistance from oDrive 0V to 48V PSU 0V is now 0.01 Ohms.

I shall now do as you suggested and connect both the 0V directly to the metal chassis and retest.

#13

So due to shortage of time tonight I just connected Jetson 0V directly to the 48V PSU’s 0V with 6mm2 of copper. Resistance smaller than I can measure.
48V PSU’s 0V to oDrive 0V is still 3mm2 of copper. Resistance 0.01 Ohms.
Problem persists.
At this point I’m doubtful that connecting to the chassis is going to suddenly make it work, but I appreciate what you have suggested about 0V connectivity is good practice, and so I shall do it anyway.

For my understanding - Regarding separately powered USB hubs (I am NOT currently using one):
The one I own has a plastic mains earth pin on the 5V adaptor. So how is it that the USB hub’s 0V to host computer 0V connectivity is any different from that of my original setup? i.e. The oDrive supplying its own power instead of using the power provided from the Jetson’s USB port.

#14

Hmm. Sorry you’ve not got it working so far

The other best-practice approach is to ensure that everything has only one ground connection and there are no loops, so that current cannot flow around the ground plane except past the central earth point. The USB isolator will help here.
And yes, you are right that the case of the powered USB hub is similar to your original setup (If your PSU had a floating output before we grounded it)
The powered USB hub will have a floating, isolated supply, which behaves like a battery. In that case the USB host PC still provides the ground reference and there are no current sources trying to challenge that ground.
But something like a laser printer will need to be grounded via the mains, because it does not have an isolated supply… Unless perhaps it does (doubtful, because it has high voltage internally), or perhaps it isolates the USB connection.

Btw apart from the PSU wires, do you have any other unusually-long wires, e.g. to the motor?
A wiring diagram and/or photo of your setup would be useful.

Also, out of interest, try lowering your current loop bandwidth. It may be that oscillations in current are the source of this noise.
ie odrv0.axis0.motor.config.current_loop_bandwidth

#15

Thanks for explaining and your help, am learning a lot!

Sorry not had much time to spend on this today. Couldn’t find current_loop_bandwidth, I presumed it has been renamed to current_control_bandwidth. Was set to 100, so set to 50 but made no difference. Can’t say I even know what this control does and couldn’t find it documented.

Cable from PSU to oDrive is 2m, and the USB cable is the one that came with it ~1.5m.
I did have them side by side so split them but no difference.
I probably should have mentioned before that the output of the 48V PSU is a triangle wave ~100mv pk-pk at ~25kHz. The frequency appeared to increase as the load on it increases. I assumed the oDrive should be able to handle that.
I have tomorrow off work so I’ll get a picture and more info then.

#16

I agree that it sounds like a grounding issue. But since you’ve explored that side quite a bit, maybe some other options are worth trying.

Other users have had succes trying different USB cables.

It could also be electricity being dumped back into your PSU or unstable frequencies, maybe the odrive resonates with the 25khz switching. Try adding a large capacitor between the + and - of your psu near your ODrive, or, alternatively, try replacing your psu/inverter with a battery to see if the problem persists.

You are sure the odrive doesn’t reset when it loses connection right?

#17

EDIT: Found issue, see post after photos.

Thanks for your suggestions.
I think I have now proved that the issue is my 48V PSU but I am confused as to why.
I have 2 different models and the issue happens with both but they are both Meanwell so I guess the insides are very similar.

Battery test:
I don’t have a 48V battery but do have 2x 12V AGM, series output 25V.
Using those to power oDrive (with 0V connected to Jetson 0V, which is still powered from mains) then the issue disappeared.
I was not happy that this definitely proved it with only being 25V. I happened to have a very cheap and noisy 70W boost PSU. It’s output is quite rough, at 48V out (and powered from batteries) it has 6v pk-pk ripple at 6Hz.
Despite this noise, the issue was not present.

I just checked the ripple frequency on the problematic 48V PSU and I had remembered it wrong. Is actually only @ 100Hz, 100mV and it stays the same when I enter AXIS_STATE_CLOSED_LOOP_CONTROL.
So now I am confused how this can be the issue - if this noise is constant (and not as bad as the cheap PSU) then why is the USB stable in IDLE but not when holding position?

I thought the purpose of the resistor on the AUX pins was to dump power there, instead of into the power supply?
I am using the supplied 50W 2Ohm resistor. Is this suffice for using the oDrive with a 440W and eventually a 500W motor simultaneously? From my initial tests I will not be using even 300W total most of the time.

Answers to other questions:
I tried different USB cables, one was shielded, no difference.
Am pretty sure the oDrive is not rebooting in this process. I only say this because if it reboots then it reverts back to IDLE and in that state the USB behaves. I have had to cut the power to oDrive between each test to force it back into IDLE ready for the next test.
Also swapped out the oDrive for another (again) - just as sanity check but issue persists.

#19

EDIT: Thought the below was true but seems even the short 4mm2 cable is intermittent. It works the majority of the time but then will fail constantly for a few hours, so it must be right on the edge of working. I am now waiting for some large 48V tolerant capacitors and the USB isolator to arrive.

Problem is the 2m cable between the 48V PSU and the oDrive.
I moved the PSU closer and used a shorter + larger cable and the issue disappeared.
I think the intermittent issue I had seen before on the bench was the grounding issue.

I don’t really want to mount the PSU so close to the oDrive so I did some experimenting with lengths and conductor sizes:
image
I am reluctant to use even more copper for what is only going to be 12A at most.
So surely the issue here can not be voltage drop. Even just 1.5mm2 should have been suffice and the motor is only holding position with no load at the moment, so <1A in this state.

So what is the issue with the long cable? EMI?