Odrive data mixing between two connected drives?

Looking for input from python/odrive experts on whether there is a possibility of mixing the statistics being logged by a pi connected to two odrives (each controlling one axis of a 2-axis project - call the two axes “ROLL” and “TILT”):

Physically, these are two separate odrive (pro) controllers, connected to an industrial-grade USB hub, then through a USB signal isolator, then to a single USB port on the pi. Motors and encoders are identical on the two axes, but their ranges of motion and behaviors are different.

We had an incident where we cooked a motor and melted the surrounding plastic parts on the TILT axis. But the most obvious record of that (railed at max torque for 25 seconds) shows up in the logs as the ROLL axis. It can’t, however, be a full swap of the two axes datasets (or control) from the very beginning, as most of the time the data, control, physical behavior, etc. align with the intended assignments.
Initialization and initial assignment of the two axes is by serial number:


roll_addr = "347436783330"  #  odrive SN 
tilt_addr = "345136613330"  #  odrive SN 

...

def setup_odrive_pros():
    global roll_drive, roll_axis, tilt_drive, tilt_axis
    # establish communication with odrives.  Set them idle, then check for errors.
    err = 0
    roll_drive = odrive.find_any(serial_number=roll_addr)
    roll_axis = roll_drive.axis0
    roll_axis.requested_state = AxisState.IDLE  # set axes IDLE
    tilt_drive = odrive.find_any(serial_number=tilt_addr)
    tilt_axis = tilt_drive.axis0
    tilt_axis.requested_state = AxisState.IDLE  # set axes IDLE
    # disable all axes' watchdog timers for now
    roll_axis.config.enable_watchdog = False
    tilt_axis.config.enable_watchdog = False    
...

Logging (and control) is by the objects initialized above:

def log_stats_check_errs(status, step_idx, header=False):
    global roll_drive, tilt_drive, roll_axis, tilt_axis, cycle_count, cycle_start_t, errs

    if header:  # only stuff this one in when asked (likely first call)
        logging.info('DATA1, c, elapsed(s), status, (idx), errs, '
                    '(R), r_pos, r_dest, r_vel, r_vset, r_mpwr, r_teff, r_ibus, r_vbus, '
                    '(T), t_pos, t_dest, t_vel, t_vset, t_mpwr, t_teff, t_ibus, t_vbus,')
    # fetch data values
    data_t = time.time()
    r_pos = roll_axis.pos_vel_mapper.pos_rel
    r_dest = roll_axis.controller.input_pos
    r_vel = roll_axis.pos_vel_mapper.vel
    r_vset = roll_axis.controller.vel_setpoint
    r_teff = roll_axis.controller.effective_torque_setpoint
    r_ibus = roll_drive.ibus
    r_vbus = roll_drive.vbus_voltage
    t_pos = tilt_axis.pos_vel_mapper.pos_rel
    t_dest = tilt_axis.controller.input_pos
    t_vel = tilt_axis.pos_vel_mapper.vel
    t_vset = tilt_axis.controller.vel_setpoint
    t_teff = tilt_axis.controller.effective_torque_setpoint
    t_ibus = tilt_drive.ibus
    t_vbus = tilt_drive.vbus_voltage
    if RT_FW_v_064_OR_OLDER:
        r_mpwr = roll_axis.controller.mechanical_power
        t_mpwr = tilt_axis.controller.mechanical_power
    else:  # compatability with odrive fw 0.6.5 ++
        r_mpwr = roll_axis.motor.mechanical_power
        t_mpwr = tilt_axis.motor.mechanical_power

    # check for errors
    _rep = check_for_odrive_errs_pro()  # check for drive errors
    if len(_rep) > 0:  # if any errors found...
        logging.info('Drive Errors found! ' + ', '.join(_rep))  # log them
        errs += 2
        status = 'drive_error'

    # compile the useful bits into a return object
    latest = Status_var()  # create status var
    latest.status = status
    latest.data_t = data_t
    latest.elapsed = data_t - cycle_start_t
    latest.r_dest = r_dest
    latest.r_pos = r_pos
    latest.r_vel = r_vel
    latest.t_dest = t_dest
    latest.t_pos = t_pos
    latest.t_vel = t_vel

    # now log the data, with (new) status
    logging.info('DATA1, {}, {:.3f}, {}, {}, {:.1f}, '
                 '(R), {:.3f}, {:.3f}, {:.3f}, {:.3f}, {:.3f}, {:.4f}, {:.2f}, {:.2f}, '
                 '(T), {:.3f}, {:.3f}, {:.3f}, {:.3f}, {:.3f}, {:.4f}, {:.2f}, {:.2f},'
                 .format(cycle_count, latest.elapsed, status, step_idx, errs,
                         r_pos, r_dest, r_vel, r_vset, r_mpwr, r_teff, r_ibus, r_vbus,
                         t_pos, t_dest, t_vel, t_vset, t_mpwr, t_teff, t_ibus, t_vbus))

    return latest
...

Is there something about the subtleties of python and/or the USB connection that I’m missing, whereby the reads from roll_drive and roll_axis could actually be fetching data from the tilt odrive, and only some of the time (in blocks)??

Thanks for your brain time

Hi,

I work with a nearly identical setup on a daily basis and I have never seen odrive objects switch mid code. One of the major differences is the use of global variables for the <odrv> and <axis> objects, this can really complicate the debugging process.
Also, I would suggest having a USB isolator for each individual ODrive, rather than just isolating the hub from the pi.

Have you seen any other instances in the logs that suggest the axes may be intermittently switching?
Did the TILT axis ODrive crash/reboot when this happened?
How did the TILT axis logs compare during this 25 sec max torque? Were they zero/undefined? Were reasonable values for what you would expect from the ROLL axis?