We have 6 ODrive S1 connected to a 24V power supply, and split in 4 + 2 configuration. Each subgroup is having each own CANBus network (we are working on connecting all to the same CANBus network, but for now they’re separated). Each of the 4 ODrive S1 is connected to a hubwheel, and uses encoders, hall sensors, and thermistor pins to control the wheel, max current is set to 10A, and the limit of the power supply is lower than the S1 max so it can’t be due to overcurrent. We are using pin 30 (G12) as digital input to emulate an Estop, connected to a NC switch. Yesterday, while our robot was working fine the group of 4 ODrive S1 stopped responding and after further inspection the LEDs in none of them responding when connected the USB-C adapter, and an overcurrent kernel message pops up on Linux when the ODrive is connected. We measure T8-T31 and it seems to be around 8V, T10-T31 is 1.8V and T9-T31 is 2V. That seems to be on all 4 ESCs, and I guess it means that something is fried. Any idea what could have caused that issue ? We didn’t change anything on the wiring and everything was working fine for more than 6 months. The second group of Odrives is still working fine
Hi! Sorry to hear you had this issue. Thanks for the detailed information on the wiring and setup, that makes it a lot easier to diagnose!
Unfortunately, this is a pretty straightforward cause: since G12 isn’t an isolated input, connecting them all together caused a big ground loop between all your ODrives. Nominal switching transients during motor control can cause the ground to swing up to +/- 2-5V between your ODrives, which then causes an elevated voltage at any non-isolated logic pins that are connected between the ODrives – CAN is fine since it’s designed to operate with large ground differentials, but the G12 pin can only handle about -0.3V - 5.5V input range. So likely what happened here is that over time, the large voltage transients at the pin during normal operation degraded/killed some of the internal microcontroller circuitry, failing it short – which is why you’re getting those USB overcurrent errors (and no LEDs are coming on). These sort of issues can present as slow failures, so even if everything works at first, that’s not a guarantee it’s not going to fail in the future.
To avoid this, you would’ve needed to add either an external digital isolator to each S1, or (strongly recommended) use the specific ISOLATED_IO input domains – that’s G05/G06/G07/G08 – of which G07 is specifically earmarked as an “ENABLE” input. You’d need to power the ISO_VDD/ISO_GND externally from an e.g. 3.3V or 5V supply, but doing so would break the ground loop and avoid this issue happening.
This isn’t covered by our warranty, but I know it’s pretty disappointing to have happen – if you want to shoot us an email at info@odriverobotics.com, I’d be happy to see if we could give a good-faith discount on some replacement units.
Hi solomondg,
thank you for the thorough answer. We were actually using G07 in the beginning but switched to G12 for less wiring. We will reach out regarding ordering replacements units. But before installing the new units we wanted to check with you an alternative causes of the error.
- Our main concern is if it was indeed the E-stop cabling creating ground loops, how come the second group wasn’t damaged as well, since they’re also on the same E-stop cabling ? Or maybe they’re damaged but not enough to completely fail ?
- We weren’t using the brake resistor for the 4-group and I guess adding that would also minimize the risk of something similar happening again?
- We were thinking that since the 2 groups have different CANBus cabling if the “error”, maybe over-voltage, traveled from one ESC to the next via CANBus, and in that case a CANBus isolator before each ESC could solve the issue. Or that’s highly unlikely based on your experience ?
Thanks again for your help!
Our main concern is if it was indeed the E-stop cabling creating ground loops, how come the second group wasn’t damaged as well, since they’re also on the same E-stop cabling ? Or maybe they’re damaged but not enough to completely fail ?
Since the switching transients are very heavily influenced by the wiring and particular current and load profile, it can be a bit nondeterministic. If you can share a sketch of your overall wiring, I’d be happy to take a closer look.
We weren’t using the brake resistor for the 4-group and I guess adding that would also minimize the risk of something similar happening again?
No, that shouldn’t have anything to do with it – it’s strictly the voltage differential on the DC-, not any voltage rise on DC+ that would do this.
We were thinking that since the 2 groups have different CANBus cabling if the “error”, maybe over-voltage, traveled from one ESC to the next via CANBus, and in that case a CANBus isolator before each ESC could solve the issue. Or that’s highly unlikely based on your experience ?
Definitely not via CAN bus – CAN is extremely robust, and can survive up to +/- 58V differential voltage. So this was definitely caused by the e-stop pin.
That’s good to know. I noticed than on Group2 we’re using G03 which is supposed to be an input pin, so maybe that’s what “saved” those ESCs ? Also group one was set in DIGITAL_PULL_UP mode while Group 2 on Digital_Mode but I guess that wouldn’t affect it as much.
The encoder/hall pins (as well as SPI MOSI) have a bit more robust input filtering than SCK/MISO/nCS since the latter are generally intended as output-only. That’s likely the contributing factor. But I wouldn’t rely on that at all – you should definitely be using the isolated pins.
Ah yeah, we changed it right away afterwards. We ended up connecting the E-stop to the enable pin on the power supply, so cut the power at Odrive in case something happens. Thanks again for your help!