ODrive stops responding over UART/ASCII

Hey!

I’ve been using an ODrive paired with an ESP32 for an electric skateboard project, however, I’m encountering an issue where the ODrive stops responding to ASCII commands over UART_A. The ODrive is still running, the motors will continue spinning (WDT fixed the runaway board problem :grimacing:), and the USB interface remains functional/normal. The UART interface remains unresponsive until I power cycle the ODrive or send odrv0.reboot() using the USB interface. This issue doesn’t occur very frequently (once or twice a day), and I can’t intentionally recreate it.

Hoping someone can give some suggestions for further troubleshooting or shed some light on how I can solve this issue. Let me know if there’s anything else (information, pictures, etc) that would help. Thanks!

I’ve included some more details below:

Using the USB interface I can still talk to the ODrive and it appears to be fully functional with no reported errors (dump_errors(odrv0) is all green, except for the watchdog timeouts).

In [2]: dump_errors(odrv0)
system: no error
axis0
  axis: Error(s):
    AXIS_ERROR_WATCHDOG_TIMER_EXPIRED
  motor: no error
  sensorless_estimator: no error
  encoder: no error
  controller: no error
axis1
  axis: Error(s):
    AXIS_ERROR_WATCHDOG_TIMER_EXPIRED
  motor: no error
  sensorless_estimator: no error
  encoder: no error
  controller: no error

As far as I can tell the ESP doesn’t accidentally disable UART. If I compare the result of “odrv0” before and after a reboot I get essentially the same result (besides WDT errors, uptime, vbus, n_evt). Note, some of the settings were truncated to “…”, but I didn’t think those specific ones would have anything relevant (“axis0.config”, etc). GPIO settings, UART settings, etc are all identical between an unresponsive ODrive and a responsive one.

The ESP32 detects the loss of communication (UART request timeout) and waits for the ODrive to respond by polling “r serial_number” every second. If I reboot the ODrive over USB using “odrv0.reboot()” the ODrive reappears over UART and the ESP resumes communication. Restarting the ESP but leaving the ODrive powered up does not resolve the issue. I’m taking this as “evidence” that UART messages are in fact being sent and the ESP isn’t at fault.
I don’t have an oscilloscope/logic analyzer at my house. But I can get dumps of the UART (or anything else of interest) if necessary in about a week.

During operation the ODrive is routinely polled using the following commands: “r vbus_voltage\n”, “r ibus\n”, “f 0\n”, and “f 1\n” (Every ~1ms). With occasional requests for “r axis#.current_state”, and writes to “w axis#.requested_state” and current setpoint writes with “c # ___” (At most every 10ms, closer to every 100ms). There doesn’t seem to be a pattern to which command will timeout.

I initially thought this issue was interference from the hub motors as I was originally using fairly long jumper wires/a breadboard. However, I still encounter this issue with the ESP32 on a small perf board connected directly to the ODrive J3 header (wire lengths < 2cm). In addition, I’ve had the issue occur while the ODrive was sitting idle (still being actively talked to, but the motors were off).

I implemented a really limited subset of the ODrive ASCII library as it wasn’t available for my platform (MicroPython). It works 99.99% of the time, so I don’t think it was an implementation issue (it’s kind of hard to mess up serial.write(…)/serial.read(…) :stuck_out_tongue: )

I was going to work around this issue by resetting the ODrive using the RST pin, but I see that was removed in an earlier hardware version :pensive:. Kind of stuck pulling the battery out since sending “sr\n” doesn’t do anything (the UART interface isn’t working), and I won’t have anything connected to the USB interface during use.

Here is the full “odrv0” dump while the ODrive is unresponsive to UART commands:

In [1]: odrv0
Out[1]:
axis0:
  acim_estimator: ...
  config: ...
  controller: ...
  current_state: 1 (uint8)
  encoder: ...
  error: 2048 (uint32)
  is_homed: False (bool)
  last_drv_fault: 0 (uint32)
  max_endstop: ...
  mechanical_brake: ...
  min_endstop: ...
  motor: ...
  requested_state: 0 (uint8)
  sensorless_estimator: ...
  step_dir_active: False (bool)
  steps: 0 (int64)
  task_times: ...
  trap_traj: ...
  watchdog_feed(obj: object_ref)
axis1:
  acim_estimator: ...
  config: ...
  controller: ...
  current_state: 1 (uint8)
  encoder: ...
  error: 2048 (uint32)
  is_homed: False (bool)
  last_drv_fault: 0 (uint32)
  max_endstop: ...
  mechanical_brake: ...
  min_endstop: ...
  motor: ...
  requested_state: 0 (uint8)
  sensorless_estimator: ...
  step_dir_active: False (bool)
  steps: 0 (int64)
  task_times: ...
  trap_traj: ...
  watchdog_feed(obj: object_ref)
brake_resistor_armed: True (bool)
brake_resistor_current: 0.0 (float)
brake_resistor_saturated: False (bool)
can:
  config: ...
  error: 0 (uint8)
clear_errors(obj: object_ref)
config:
  brake_resistance: 2.0 (float)
  dc_bus_overvoltage_ramp_end: 59.92000198364258 (float)
  dc_bus_overvoltage_ramp_start: 59.92000198364258 (float)
  dc_bus_overvoltage_trip_level: 59.92000198364258 (float)
  dc_bus_undervoltage_trip_level: 8.0 (float)
  dc_max_negative_current: -10.0 (float)
  dc_max_positive_current: inf (float)
  enable_brake_resistor: True (bool)
  enable_can_a: True (bool)
  enable_dc_bus_overvoltage_ramp: False (bool)
  enable_i2c_a: False (bool)
  enable_uart_a: True (bool)
  enable_uart_b: False (bool)
  enable_uart_c: False (bool)
  error_gpio_pin: 0 (uint32)
  gpio10_mode: 0 (uint8)
  gpio11_mode: 0 (uint8)
  gpio12_mode: 0 (uint8)
  gpio13_mode: 0 (uint8)
  gpio14_mode: 0 (uint8)
  gpio15_mode: 7 (uint8)
  gpio16_mode: 7 (uint8)
  gpio1_mode: 4 (uint8)
  gpio1_pwm_mapping: ...
  gpio2_mode: 4 (uint8)
  gpio2_pwm_mapping: ...
  gpio3_analog_mapping: ...
  gpio3_mode: 3 (uint8)
  gpio3_pwm_mapping: ...
  gpio4_analog_mapping: ...
  gpio4_mode: 3 (uint8)
  gpio4_pwm_mapping: ...
  gpio5_mode: 3 (uint8)
  gpio6_mode: 0 (uint8)
  gpio7_mode: 0 (uint8)
  gpio8_mode: 0 (uint8)
  gpio9_mode: 0 (uint8)
  max_regen_current: 6.0 (float)
  uart0_protocol: 3 (uint8)
  uart1_protocol: 3 (uint8)
  uart2_protocol: 3 (uint8)
  uart_a_baudrate: 115200 (uint32)
  uart_b_baudrate: 115200 (uint32)
  uart_c_baudrate: 115200 (uint32)
  usb_cdc_protocol: 3 (uint8)
enter_dfu_mode(obj: object_ref)
erase_configuration(obj: object_ref)
error: 0 (uint8)
fw_version_major: 0 (uint8)
fw_version_minor: 5 (uint8)
fw_version_revision: 4 (uint8)
fw_version_unreleased: 1 (uint8)
get_adc_voltage(obj: object_ref, gpio: uint32) -> voltage: float
get_dma_status(obj: object_ref, stream_num: uint8) -> status: uint32
get_drv_fault(obj: object_ref) -> drv_fault: uint64
get_gpio_states(obj: object_ref) -> status: uint32
get_interrupt_status(obj: object_ref, irqn: int32) -> status: uint32
hw_version_major: 3 (uint8)
hw_version_minor: 6 (uint8)
hw_version_variant: 56 (uint8)
ibus: 0.0 (float)
ibus_report_filter_k: 1.0 (float)
misconfigured: False (bool)
n_evt_control_loop: 8374489 (uint32)
n_evt_sampling: 8374493 (uint32)
oscilloscope:
  get_val(obj: object_ref, index: uint32) -> val: float
  size: 4096 (uint32)
otp_valid: True (bool)
reboot(obj: object_ref)
save_configuration(obj: object_ref) -> success: bool
serial_number: 56557042545977 (uint64)
system_stats:
  i2c: ...
  max_stack_usage_analog: 316 (uint32)
  max_stack_usage_axis: 564 (uint32)
  max_stack_usage_can: 348 (uint32)
  max_stack_usage_startup: 532 (uint32)
  max_stack_usage_uart: 2128 (uint32)
  max_stack_usage_usb: 420 (uint32)
  min_heap_space: 47432 (uint32)
  prio_analog: -2 (int32)
  prio_axis: 3 (int32)
  prio_can: 0 (int32)
  prio_startup: 0 (int32)
  prio_uart: 0 (int32)
  prio_usb: 0 (int32)
  stack_size_analog: 1024 (uint32)
  stack_size_axis: 2048 (uint32)
  stack_size_can: 1024 (uint32)
  stack_size_startup: 2048 (uint32)
  stack_size_uart: 4096 (uint32)
  stack_size_usb: 4096 (uint32)
  uptime: 1046929 (uint32)
  usb: ...
task_timers_armed: False (bool)
task_times:
  control_loop_checks: ...
  control_loop_misc: ...
  dc_calib_wait: ...
  sampling: ...
test_function(obj: object_ref, delta: int32) -> cnt: int32
test_property: 0 (uint32)
user_config_loaded: 30262 (uint32)
vbus_voltage: 46.412696838378906 (float)

Unfortunately this is an issue with the uart ascii communication that we are still trying to figure out, if you can afford to reduce the read write frequency at all this “might” help.
All ODrives should have an nRST pin somewhere on the board, which version are you using?

1 Like

Thanks Nicholas, that’s really unfortunate to hear! Guess I should have gone with the Native protocol like I had initially planned :thinking:

I’m using a v3.6 and I’ve since located the RST pin on the 3rd header. It was hiding under my hall sensor wires.

Is there a bug ticket or an issue report that I can follow?

Thanks for your help!

1 Like