USB reliability when in AXIS_STATE_CLOSED_LOOP_CONTROL

#11

Sorry, when I said PSU ground, I meant the 0V output. THAT should be connected to mains earth.
Otherwise, the only path to mains earth might be via the USB.

Basically, you don’t want the grounds (0V) on the two boards (Jetson and ODrive) to be different, otherwise USB will have problems.
USB is designed to either connect to a device whose ground reference is mains earth OR, power the device directly from the bus.

In the embedded system I would connect each of the ODrive 0V and the Jetson 0V to the vehicle chassis.

#12

I think I finally understand now. Please correct me if I am (still) missing something:
So mains earth in this scenario is literally just serving as a connection point rather than using its property as earth, so that the 0V from two different PSUs are connected together. I had not realised that the 0V connection that the USB cable was providing between the two was not suffice for reliable comms.

All that said the problem persists… I did the following:
Connected 48V PSU 0V to mains earth via a very short 1.5mm2 cable.
Connected Jetson 0V to mains earth via short 1.5mm2 cable.
Resistance from Jetson 0V to 48V PSU 0V is now 0.19 Ohms.
Resistance from oDrive 0V to 48V PSU 0V is now 0.01 Ohms.

I shall now do as you suggested and connect both the 0V directly to the metal chassis and retest.

#13

So due to shortage of time tonight I just connected Jetson 0V directly to the 48V PSU’s 0V with 6mm2 of copper. Resistance smaller than I can measure.
48V PSU’s 0V to oDrive 0V is still 3mm2 of copper. Resistance 0.01 Ohms.
Problem persists.
At this point I’m doubtful that connecting to the chassis is going to suddenly make it work, but I appreciate what you have suggested about 0V connectivity is good practice, and so I shall do it anyway.

For my understanding - Regarding separately powered USB hubs (I am NOT currently using one):
The one I own has a plastic mains earth pin on the 5V adaptor. So how is it that the USB hub’s 0V to host computer 0V connectivity is any different from that of my original setup? i.e. The oDrive supplying its own power instead of using the power provided from the Jetson’s USB port.

#14

Hmm. Sorry you’ve not got it working so far

The other best-practice approach is to ensure that everything has only one ground connection and there are no loops, so that current cannot flow around the ground plane except past the central earth point. The USB isolator will help here.
And yes, you are right that the case of the powered USB hub is similar to your original setup (If your PSU had a floating output before we grounded it)
The powered USB hub will have a floating, isolated supply, which behaves like a battery. In that case the USB host PC still provides the ground reference and there are no current sources trying to challenge that ground.
But something like a laser printer will need to be grounded via the mains, because it does not have an isolated supply… Unless perhaps it does (doubtful, because it has high voltage internally), or perhaps it isolates the USB connection.

Btw apart from the PSU wires, do you have any other unusually-long wires, e.g. to the motor?
A wiring diagram and/or photo of your setup would be useful.

Also, out of interest, try lowering your current loop bandwidth. It may be that oscillations in current are the source of this noise.
ie odrv0.axis0.motor.config.current_loop_bandwidth

#15

Thanks for explaining and your help, am learning a lot!

Sorry not had much time to spend on this today. Couldn’t find current_loop_bandwidth, I presumed it has been renamed to current_control_bandwidth. Was set to 100, so set to 50 but made no difference. Can’t say I even know what this control does and couldn’t find it documented.

Cable from PSU to oDrive is 2m, and the USB cable is the one that came with it ~1.5m.
I did have them side by side so split them but no difference.
I probably should have mentioned before that the output of the 48V PSU is a triangle wave ~100mv pk-pk at ~25kHz. The frequency appeared to increase as the load on it increases. I assumed the oDrive should be able to handle that.
I have tomorrow off work so I’ll get a picture and more info then.

#16

I agree that it sounds like a grounding issue. But since you’ve explored that side quite a bit, maybe some other options are worth trying.

Other users have had succes trying different USB cables.

It could also be electricity being dumped back into your PSU or unstable frequencies, maybe the odrive resonates with the 25khz switching. Try adding a large capacitor between the + and - of your psu near your ODrive, or, alternatively, try replacing your psu/inverter with a battery to see if the problem persists.

You are sure the odrive doesn’t reset when it loses connection right?

#17

EDIT: Found issue, see post after photos.

Thanks for your suggestions.
I think I have now proved that the issue is my 48V PSU but I am confused as to why.
I have 2 different models and the issue happens with both but they are both Meanwell so I guess the insides are very similar.

Battery test:
I don’t have a 48V battery but do have 2x 12V AGM, series output 25V.
Using those to power oDrive (with 0V connected to Jetson 0V, which is still powered from mains) then the issue disappeared.
I was not happy that this definitely proved it with only being 25V. I happened to have a very cheap and noisy 70W boost PSU. It’s output is quite rough, at 48V out (and powered from batteries) it has 6v pk-pk ripple at 6Hz.
Despite this noise, the issue was not present.

I just checked the ripple frequency on the problematic 48V PSU and I had remembered it wrong. Is actually only @ 100Hz, 100mV and it stays the same when I enter AXIS_STATE_CLOSED_LOOP_CONTROL.
So now I am confused how this can be the issue - if this noise is constant (and not as bad as the cheap PSU) then why is the USB stable in IDLE but not when holding position?

I thought the purpose of the resistor on the AUX pins was to dump power there, instead of into the power supply?
I am using the supplied 50W 2Ohm resistor. Is this suffice for using the oDrive with a 440W and eventually a 500W motor simultaneously? From my initial tests I will not be using even 300W total most of the time.

Answers to other questions:
I tried different USB cables, one was shielded, no difference.
Am pretty sure the oDrive is not rebooting in this process. I only say this because if it reboots then it reverts back to IDLE and in that state the USB behaves. I have had to cut the power to oDrive between each test to force it back into IDLE ready for the next test.
Also swapped out the oDrive for another (again) - just as sanity check but issue persists.

#19

EDIT: Thought the below was true but seems even the short 4mm2 cable is intermittent. It works the majority of the time but then will fail constantly for a few hours, so it must be right on the edge of working. I am now waiting for some large 48V tolerant capacitors and the USB isolator to arrive.

Problem is the 2m cable between the 48V PSU and the oDrive.
I moved the PSU closer and used a shorter + larger cable and the issue disappeared.
I think the intermittent issue I had seen before on the bench was the grounding issue.

I don’t really want to mount the PSU so close to the oDrive so I did some experimenting with lengths and conductor sizes:
image
I am reluctant to use even more copper for what is only going to be 12A at most.
So surely the issue here can not be voltage drop. Even just 1.5mm2 should have been suffice and the motor is only holding position with no load at the moment, so <1A in this state.

So what is the issue with the long cable? EMI?

#20

Put a 3300uF capacitor on the input power connector to the oDrive - still fails.
Used a USB isolator - still fails.

While the long cable appeared to exacerbate the issue, it did not seem to be causing it.
So seems the logical culprit is the Meanwell PSUs…

Guess I will have to bite the bullet and buy a different brand PSU. Am quite reluctant to do this so will try once again to find noise on both Jetson and oDrive first.

#21

@Junglist I have one of these near my odrive to ensure power quality. I’m not sure if it will solve your problem, but it did for me. Be carefull though, they store quite a lot of energy, which is useful, but also dangerous if shorted.

#22

And I thought 3300uF was big!
Think if I add all my 63V caps together I can get close to that. Will give it a try tomorrow, thanks

#23

When using the USB isolator, did you try disconnecting the ground link that you put between the ODrive and the Jetson?

It seems that mean-well has a lot of noise and may even be resonating with the odrive’s switching frequency (although adding caps should damp and shift that). Maybe with the frequencies involved, the ESR of the caps is too high.

The issue with the long cable is inductance and transmission line effects (although 2 metres should be fine for this tbh).
But again, if you are only holding position then yiu are not drawing hardly any current so any issues with wiring inductance should be minimal.
This is a really weird one tbh.

I suppose it’s possible that your odrive has faulty caps on the board… Can your multimeter measure the DC Bus capacitance? Also maybe measure the decoupling caps on the 5V and 3.3V pins on the board.
If you have a scope, see if you have any high frequency noise coupled onto these supply pins.

#24

@Riewert
I soldered 4x 3300uF caps in parallel so total 13200uF and put as close to the oDrive connector as I could but made no difference unfortunately. Thanks for the suggestion though.

#25

@towen
I hadn’t disconnected the ground link it but just did and still failed.

What confuses me is that my cheap low power PSU running off the battery had a much more coarse output than the Meanwell but that worked fine. So would make sense what you are saying about the specific out frequency of the Meanwell resonating with the oDrive’s.

My multimeter can’t measure capacitance unfortunately. However I have a few oDrives now so as the occasional sanity check I have been swapping out the one I’m testing. So seems fairly safe to say that the on board caps aren’t to blame.

I have a scope so will probe the 3V3 and 5V rails, plus the USB lines to see if I can find the offending noise.
Just noticed on schematic that the USB interface on U2 is powered by 3V3 instead of 5V. I guess this can’t help the SNR much?!

Thanks for your suggestions.

#26

One other thing you can try, is disconnecting the feet from the body. There might be some capacitance between the motor and the frame.

Also, you really should be able to read out some errors, the odrives have a lot of failsafes against resetting. They throw errors if over- or undervoltages occur, and shut down the mosfets, preventing resets. Getting those error codes would really help debugging your issues.

It might also be the usb driver on your Jetson is fried or overly sensitive. Have you tried running the code on a seperate laptop instead of your jetson? Laptops might have beefier 5V rails that can absorb more spikes than your jetson (and help identify the weakest link).

#27

So as I understand it, the ODrive is NOT resetting (it continues to maintain position and does not log any error) but ODriveTool (or whatever you are using to control it) is dropping out; i.e. Linux says in dmesg, usb device disconnected, and then odrivetool says Oh No odrv0 disappeared! - is that correct?

As I say I have had very similar USB issues with Roboclaw controllers on a Jetson, and since the things at that point had no watchdog feature, it was a serious problem. It was instantly solved with a USB isolator, (and they updated the firmware to add a watchdog after some pestering) but I am perplexed as to why @junglist is having such trouble. The 5V rail of the Jetson USB should not be a problem if using an isolator.

At this point, I would consider an alternative communication method. I have had a lot of success with CAN bus personally - once you have a transceiver set up and working in socketcan in Linux, it works perfectly

1 Like
#28

@Riewert
towen has summarized correctly - oDrive does not reboot and there are no errors.
My laptop (and desktop) are more resilient to the issue than the Jetson so I have monitored for errors with that.

#29

@towen
Correct.

Think I have just proved the issue is related to my steering motor specifically:
It only just occurred to me this morning that I should try the same test with my hub motor, which worked fine, no USB issue, even without using USB isolator.
I swapped the axis around that each motor was using, recalibrated and repeated test - the issue follows the steering motor.
I replicated the above on a separate assembly - so different oDrive and different steering motor.
With the hub motor holding position, I can force it as hard as I can to make it fight me, and this still would not create the issue.

So I guess this rules out my PSU. The hub motor is rated to 500W while the steering motor is 440W.

The steering motor is physically right next to the oDrive. Photo below.
I thought that the fact that powering from the battery test worked fine proved that this close proximity was not an issue. Unfortunately the wires on the steering motor are so short that I can’t move much further away without soldering extensions on it.
Perhaps I should have put the oDrive in a metal box…

I will keep trying to resolve USB for now but that failing that will move to CAN.

#30

20191116_113548
20191116_113548.jpg2851×2138 1.95 MB

#31

Can you show a picture of the inside, specifically of the mounting screws for the ODrive and the junctionbox? Maybe metal screws are allowing energy from your steering motor to penetrate your junctionbox.