ERROR_CONTROL_DEADLINE_MISSED above certain speed

We have two motors connected to M0 and M1, with unfiltered hall sensor input. They were set up according to https://docs.odriverobotics.com/hoverboard.html. The working config json is below. ODrive 3.5 48V with firmware 0.4.7.

config
{
  "config": {
    "brake_resistance": 2,
    "enable_uart": true,
    "enable_i2c_instead_of_can": false,
    "enable_ascii_protocol_on_usb": true,
    "dc_bus_undervoltage_trip_level": 8,
    "dc_bus_overvoltage_trip_level": 51.840003967285156
  },
  "axis0": {
    "config": {
      "startup_motor_calibration": false,
      "startup_encoder_index_search": false,
      "startup_encoder_offset_calibration": false,
      "startup_closed_loop_control": true,
      "startup_sensorless_control": false,
      "enable_step_dir": false,
      "counts_per_step": 2,
      "step_gpio_pin": 1,
      "dir_gpio_pin": 2,
      "ramp_up_time": 0.4000000059604645,
      "ramp_up_distance": 12.566370964050293,
      "spin_up_current": 10,
      "spin_up_acceleration": 400,
      "spin_up_target_vel": 400
    },
    "motor": {
      "config": {
        "pre_calibrated": true,
        "pole_pairs": 8,
        "calibration_current": 30,
        "resistance_calib_max_voltage": 4,
        "phase_inductance": 0.00002497009700164199,
        "phase_resistance": 0.03291689604520798,
        "direction": -1,
        "motor_type": 0,
        "current_lim": 10,
        "requested_current_range": 60,
        "current_control_bandwidth": 1000
      }
    },
    "controller": {
      "config": {
        "control_mode": 2,
        "pos_gain": 1,
        "vel_gain": 0.019999999552965164,
        "vel_integrator_gain": 0.10000000149011612,
        "vel_limit": 1000,
        "vel_limit_tolerance": 1.2000000476837158,
        "vel_ramp_rate": 10000,
        "setpoints_in_cpr": false
      }
    },
    "encoder": {
      "config": {
        "mode": 1,
        "use_index": false,
        "pre_calibrated": true,
        "idx_search_speed": 10,
        "zero_count_on_find_idx": true,
        "cpr": 48,
        "offset": -23,
        "offset_float": -0.4937344193458557,
        "bandwidth": 100,
        "calib_range": 0.03999999910593033
      }
    },
    "sensorless_estimator": {
      "config": {
        "observer_gain": 1000,
        "pll_bandwidth": 1000,
        "pm_flux_linkage": 0.0015800000401213765
      }
    },
    "trap_traj": {
      "config": {
        "vel_limit": 20000,
        "accel_limit": 5000,
        "decel_limit": 5000,
        "A_per_css": 0
      }
    }
  },
  "axis1": {
    "config": {
      "startup_motor_calibration": false,
      "startup_encoder_index_search": false,
      "startup_encoder_offset_calibration": false,
      "startup_closed_loop_control": true,
      "startup_sensorless_control": false,
      "enable_step_dir": false,
      "counts_per_step": 2,
      "step_gpio_pin": 7,
      "dir_gpio_pin": 8,
      "ramp_up_time": 0.4000000059604645,
      "ramp_up_distance": 12.566370964050293,
      "spin_up_current": 10,
      "spin_up_acceleration": 400,
      "spin_up_target_vel": 400
    },
    "motor": {
      "config": {
        "pre_calibrated": true,
        "pole_pairs": 8,
        "calibration_current": 30,
        "resistance_calib_max_voltage": 4,
        "phase_inductance": 0.000024654920707689598,
        "phase_resistance": 0.032729052007198334,
        "direction": -1,
        "motor_type": 0,
        "current_lim": 10,
        "requested_current_range": 60,
        "current_control_bandwidth": 100
      }
    },
    "controller": {
      "config": {
        "control_mode": 2,
        "pos_gain": 1,
        "vel_gain": 0.019999999552965164,
        "vel_integrator_gain": 0.10000000149011612,
        "vel_limit": 1000,
        "vel_limit_tolerance": 1.2000000476837158,
        "vel_ramp_rate": 10000,
        "setpoints_in_cpr": false
      }
    },
    "encoder": {
      "config": {
        "mode": 1,
        "use_index": false,
        "pre_calibrated": true,
        "idx_search_speed": 10,
        "zero_count_on_find_idx": true,
        "cpr": 48,
        "offset": -23,
        "offset_float": -0.4839375615119934,
        "bandwidth": 100,
        "calib_range": 0.03999999910593033
      }
    },
    "sensorless_estimator": {
      "config": {
        "observer_gain": 1000,
        "pll_bandwidth": 1000,
        "pm_flux_linkage": 0.0015800000401213765
      }
    },
    "trap_traj": {
      "config": {
        "vel_limit": 20000,
        "accel_limit": 5000,
        "decel_limit": 5000,
        "A_per_css": 0
      }
    }
  }
}

M1 runs well in CTRL_MODE_VELOCITY_CONTROL with odrv0.axis0.controller.vel_setpoint = 1000.

M0 however will work at low speeds (around 50) but fail at higher speeds or after some time at lower speed, with motor error 0x10 / ERROR_CONTROL_DEADLINE_MISSED.

M1 generates the same error when it reaches higher speeds in CTRL_MODE_POSITION_CONTROL (e.g. when running to a posiiton setpoint far away).

My question is what does this error really mean (why would the control loop of one motor be so much slower that it misses the deadline) and how can I fix it?

Edit: I just noticed that axis0’s current_control_bandwidth was 1000 instead of 100. With 100, M0 will fail at speed 300 instead of 50.

I built and flashed the firmware from the current devel branch. Interestingly, axis0 will get ERROR_CONTROL_DEADLINE_MISSED even if only axis1 spins. This time it also throws ERROR_BRAKE_CURRENT_OUT_OF_RANGE.

        Reconnected to ODrive 386537773437 as odrv0
In [9]: dump_errors(odrv0)
Axis0:
  axis: no error
  motor: no error
  encoder: no error
  controller: no error
Axis1:
  axis: no error
  motor: no error
  encoder: no error
  controller: no error

In [10]: odrv0.axis1.controller.vel_setpoint = 1000

In [11]: dump_errors(odrv0)
Axis0:
  axis: Error(s):
    ERROR_BRAKE_RESISTOR_DISARMED
    ERROR_MOTOR_DISARMED
  motor: Error(s):
    ERROR_CONTROL_DEADLINE_MISSED
    ERROR_BRAKE_CURRENT_OUT_OF_RANGE
  encoder: no error
  controller: no error
Axis1:
  axis: Error(s):
    ERROR_BRAKE_RESISTOR_DISARMED
    ERROR_MOTOR_DISARMED
  motor: Error(s):
    ERROR_BRAKE_CURRENT_OUT_OF_RANGE
  encoder: no error
  controller: no error

I’m still having this issue. I noticed that when axis1 is in idle state, axis0 will not get the deadline missed error at reasonable speeds.
I saw the The motor timing diagram and to me it sounds like the M1 control thread just doesn’t finish in time if M0 control takes too long.
Is there a way to reduce PWM frequency or increase control processing speed?

I flashed the 0.4.6 firmware release and the issue is completely gone. It must be a bug in the 0.4.7 firmware, probably introduced by increasing the PWM frequency? It would be nice if you could (help me) fix it as the new firmware has some nice features like velocity ramping which I would like to use.

Usually that error is caused by noise on the index pulse line of the encoders. It gets picked up and triggers an interrupt on repeat.

Wouldn’t that give me (at least sometimes) an illegal hall state in hall encoder mode?

Not if it’s only the index pulse, no.

There’s no index in hall encoder mode though?

:thinking: Good point. One of UVW is plugged into Z, isn’t it? I’m not sure then, I’ve never used the Hall mode.

OK so I reverted the 24kHz PWM feature merge from the devel branch (code is at https://github.com/lalten/ODrive/tree/revert-24k) and things look good. I’m do get some illegal hall states so I guess there must be some noise, but in the newer firmware I can ignore them, which is nice.
The only drawback is that the motors are quite noisy at 8kHz, but I can live with that.