Thanks for explaining and your help, am learning a lot!
Sorry not had much time to spend on this today. Couldn’t find current_loop_bandwidth, I presumed it has been renamed to current_control_bandwidth. Was set to 100, so set to 50 but made no difference. Can’t say I even know what this control does and couldn’t find it documented.
Cable from PSU to oDrive is 2m, and the USB cable is the one that came with it ~1.5m.
I did have them side by side so split them but no difference.
I probably should have mentioned before that the output of the 48V PSU is a triangle wave ~100mv pk-pk at ~25kHz. The frequency appeared to increase as the load on it increases. I assumed the oDrive should be able to handle that.
I have tomorrow off work so I’ll get a picture and more info then.
I agree that it sounds like a grounding issue. But since you’ve explored that side quite a bit, maybe some other options are worth trying.
Other users have had succes trying different USB cables.
It could also be electricity being dumped back into your PSU or unstable frequencies, maybe the odrive resonates with the 25khz switching. Try adding a large capacitor between the + and - of your psu near your ODrive, or, alternatively, try replacing your psu/inverter with a battery to see if the problem persists.
You are sure the odrive doesn’t reset when it loses connection right?
Thanks for your suggestions.
I think I have now proved that the issue is my 48V PSU but I am confused as to why.
I have 2 different models and the issue happens with both but they are both Meanwell so I guess the insides are very similar.
Battery test:
I don’t have a 48V battery but do have 2x 12V AGM, series output 25V.
Using those to power oDrive (with 0V connected to Jetson 0V, which is still powered from mains) then the issue disappeared.
I was not happy that this definitely proved it with only being 25V. I happened to have a very cheap and noisy 70W boost PSU. It’s output is quite rough, at 48V out (and powered from batteries) it has 6v pk-pk ripple at 6Hz.
Despite this noise, the issue was not present.
I just checked the ripple frequency on the problematic 48V PSU and I had remembered it wrong. Is actually only @ 100Hz, 100mV and it stays the same when I enter AXIS_STATE_CLOSED_LOOP_CONTROL.
So now I am confused how this can be the issue - if this noise is constant (and not as bad as the cheap PSU) then why is the USB stable in IDLE but not when holding position?
I thought the purpose of the resistor on the AUX pins was to dump power there, instead of into the power supply?
I am using the supplied 50W 2Ohm resistor. Is this suffice for using the oDrive with a 440W and eventually a 500W motor simultaneously? From my initial tests I will not be using even 300W total most of the time.
Answers to other questions:
I tried different USB cables, one was shielded, no difference.
Am pretty sure the oDrive is not rebooting in this process. I only say this because if it reboots then it reverts back to IDLE and in that state the USB behaves. I have had to cut the power to oDrive between each test to force it back into IDLE ready for the next test.
Also swapped out the oDrive for another (again) - just as sanity check but issue persists.
EDIT: Thought the below was true but seems even the short 4mm2 cable is intermittent. It works the majority of the time but then will fail constantly for a few hours, so it must be right on the edge of working. I am now waiting for some large 48V tolerant capacitors and the USB isolator to arrive.
Problem is the 2m cable between the 48V PSU and the oDrive.
I moved the PSU closer and used a shorter + larger cable and the issue disappeared.
I think the intermittent issue I had seen before on the bench was the grounding issue.
I don’t really want to mount the PSU so close to the oDrive so I did some experimenting with lengths and conductor sizes:
I am reluctant to use even more copper for what is only going to be 12A at most.
So surely the issue here can not be voltage drop. Even just 1.5mm2 should have been suffice and the motor is only holding position with no load at the moment, so <1A in this state.
Put a 3300uF capacitor on the input power connector to the oDrive - still fails.
Used a USB isolator - still fails.
While the long cable appeared to exacerbate the issue, it did not seem to be causing it.
So seems the logical culprit is the Meanwell PSUs…
Guess I will have to bite the bullet and buy a different brand PSU. Am quite reluctant to do this so will try once again to find noise on both Jetson and oDrive first.
@Junglist I have one of these near my odrive to ensure power quality. I’m not sure if it will solve your problem, but it did for me. Be carefull though, they store quite a lot of energy, which is useful, but also dangerous if shorted.
When using the USB isolator, did you try disconnecting the ground link that you put between the ODrive and the Jetson?
It seems that mean-well has a lot of noise and may even be resonating with the odrive’s switching frequency (although adding caps should damp and shift that). Maybe with the frequencies involved, the ESR of the caps is too high.
The issue with the long cable is inductance and transmission line effects (although 2 metres should be fine for this tbh).
But again, if you are only holding position then yiu are not drawing hardly any current so any issues with wiring inductance should be minimal.
This is a really weird one tbh.
I suppose it’s possible that your odrive has faulty caps on the board… Can your multimeter measure the DC Bus capacitance? Also maybe measure the decoupling caps on the 5V and 3.3V pins on the board.
If you have a scope, see if you have any high frequency noise coupled onto these supply pins.
@Riewert
I soldered 4x 3300uF caps in parallel so total 13200uF and put as close to the oDrive connector as I could but made no difference unfortunately. Thanks for the suggestion though.
@towen
I hadn’t disconnected the ground link it but just did and still failed.
What confuses me is that my cheap low power PSU running off the battery had a much more coarse output than the Meanwell but that worked fine. So would make sense what you are saying about the specific out frequency of the Meanwell resonating with the oDrive’s.
My multimeter can’t measure capacitance unfortunately. However I have a few oDrives now so as the occasional sanity check I have been swapping out the one I’m testing. So seems fairly safe to say that the on board caps aren’t to blame.
I have a scope so will probe the 3V3 and 5V rails, plus the USB lines to see if I can find the offending noise.
Just noticed on schematic that the USB interface on U2 is powered by 3V3 instead of 5V. I guess this can’t help the SNR much?!
One other thing you can try, is disconnecting the feet from the body. There might be some capacitance between the motor and the frame.
Also, you really should be able to read out some errors, the odrives have a lot of failsafes against resetting. They throw errors if over- or undervoltages occur, and shut down the mosfets, preventing resets. Getting those error codes would really help debugging your issues.
It might also be the usb driver on your Jetson is fried or overly sensitive. Have you tried running the code on a seperate laptop instead of your jetson? Laptops might have beefier 5V rails that can absorb more spikes than your jetson (and help identify the weakest link).
So as I understand it, the ODrive is NOT resetting (it continues to maintain position and does not log any error) but ODriveTool (or whatever you are using to control it) is dropping out; i.e. Linux says in dmesg, usb device disconnected, and then odrivetool says Oh No odrv0 disappeared! - is that correct?
As I say I have had very similar USB issues with Roboclaw controllers on a Jetson, and since the things at that point had no watchdog feature, it was a serious problem. It was instantly solved with a USB isolator, (and they updated the firmware to add a watchdog after some pestering) but I am perplexed as to why @junglist is having such trouble. The 5V rail of the Jetson USB should not be a problem if using an isolator.
At this point, I would consider an alternative communication method. I have had a lot of success with CAN bus personally - once you have a transceiver set up and working in socketcan in Linux, it works perfectly
@Riewert
towen has summarized correctly - oDrive does not reboot and there are no errors.
My laptop (and desktop) are more resilient to the issue than the Jetson so I have monitored for errors with that.
Think I have just proved the issue is related to my steering motor specifically:
It only just occurred to me this morning that I should try the same test with my hub motor, which worked fine, no USB issue, even without using USB isolator.
I swapped the axis around that each motor was using, recalibrated and repeated test - the issue follows the steering motor.
I replicated the above on a separate assembly - so different oDrive and different steering motor.
With the hub motor holding position, I can force it as hard as I can to make it fight me, and this still would not create the issue.
So I guess this rules out my PSU. The hub motor is rated to 500W while the steering motor is 440W.
The steering motor is physically right next to the oDrive. Photo below.
I thought that the fact that powering from the battery test worked fine proved that this close proximity was not an issue. Unfortunately the wires on the steering motor are so short that I can’t move much further away without soldering extensions on it.
Perhaps I should have put the oDrive in a metal box…
I will keep trying to resolve USB for now but that failing that will move to CAN.
Can you show a picture of the inside, specifically of the mounting screws for the ODrive and the junctionbox? Maybe metal screws are allowing energy from your steering motor to penetrate your junctionbox.
Not so much the screws letting energy in by penetrating the plastic, but more like the plastic itself being completely transparent to EMI and near field magnetic interference from the motor.
USB is wired as a differential pair so it should be immune to near-field magnetics, unless the tracks on the PCB take different routes or have different lengths
You could try putting a steel or iron plate between the motor and the plastic box?
Finally got chance to work on this today. Seem to have gone full circle again and have seen the problem occur when just using the hub motor.
I bolted a 1.2mm thick steel sheet to the gearbox to try shield the oDrive as suggested. I tested it was grounded by the rest of the frame, which is connected to 0V (of PSU and Jetson) and mains earth. Picture below.
Unfortunately this hasn’t made any difference. Tried it with USB isolator and without.
I started probing for noise and found a 40uS waveform across C29 (5V Reg output) only when in AXIS_STATE_CLOSED_LOOP_CONTROL.
So I added a lot of extra decoupling which did attenuate it but did not get of it completely. Problem persists.
I probed TP1 and TP2 (USB_DM and DP) with an as short an earth lead as possible (see picture). I can see noise here and only when it fails. 20uS waveform at ~1V pk-pk with 2V spikes.
The noise waveform is not identical on DM and DP. The PCB traces do change their distance from each other and no effort appears to have been made to separate them from other traces. I have no idea how critical this is for USB.
After seeing the noise on the USB signal for the steering motor, I wanted to see it for the hub motor too. There is noise there but it is not as high amplitude, it did cause the USB to drop offline though a couple of times, which I have seen before intermittently throughout the project but had put down to the bad grounding issue. So seems the problem here is not just related to the steering motor.
I will put pictures below of the noise I am seeing on the USB_DM. If it’s not too much effort - would either of you able to probe it with a largish motor 400-500W to see if you see any noise?
This 40uS and 20uS noise is not present on the 48V into the oDrive.
Ok, so you have the probe on one of the USB data wires.
It’s possible that the signal you are looking at is normal USB communication. The AC coupling on your scope might be tricking you into thinking that it’s not a digital signal…?
The USB wires are differential so to see the actual data without noise over the top you would need to subtract one from the other. I guess that’s not a storage oscilloscope though, so you might not be able to do that…
You say you see the noise on the probe ONLY during the failure condition.?
Does that mean the noise starts immediately as soon as the USB drops out, and continues until you reset the drive?
If so, I wonder if it could be suffering from interrupt overload or something like that, e.g. some GPIO pin configured as an interrupt is picking up noise and spamming the microcontroller with interrupts - this could be starving the low-priority tasks i.e. USB.
20uS would equate to 50kHz. I think the PWM frequency is 25kHz, so you may be seeing spikes at both the rising and falling edges of the PWM cycle.
Just to be sure, you did get your ODrives from the shop on this site, not from Amazon/eBay/Aliexpress/BangGood? There are a lot of very dodgy-looking clones appearing.
I have a fairly big motor I can try it on - I have a scrapped Turnigy Rotomax 50cc that I picked up to play about with. I also have a Jetson TX2, so can try that too.
I’ll let you know how I get on.
There are a lot of very dodgy-looking clones appearing.