Unresponsive USB after about 20 secs

i’m using a standard ubuntu desktop, both to compile the firmware and to send command on USB. it’s probably a 16.04, i can check but it’s likely to be off topic. I have not tried piloting the odrive from windows since it’s not likely to be use case in the future (but i can if it helps troubleshoot the issue)

To send command on USB, i simply send text commands to /dev/tty.ACM0 from the system terminal.

echo -en “p 0 9999 0 0\n” > /dev/ttyACM0

then i merely used a bash loop to repeatedly send such commands every 3 seconds or so.

I can try to reproduce the behavior using a python script instead of shell commands.

=> would that help ?

thanks

1 Like

Ah okay cool. Yeah please do try from the python. It uses pyusb drivers instead of the CDC drivers. You can have a look at test_bulk.py to see how it sends the commands. I’m keen to see if this is more stable, thanks for testing!

i finally managed to get a moment to try this out.

i modified the code of test_bulk so that instead of prompting user for a command i would provide the input directly from script

steps = 9999
while running:
  time.sleep(0.1)
  try:
      #command = input("Enter ODrive command:\n")
      command = "p 0 "+str(steps)+" 0 0"
      dev.send(command)
      steps = -1 * steps
      time.sleep(3.0)

the good news is that my turnigy 5065 236 has been spinning back and forth for about 5minutes now. however, it’s kinda weird because the system draw very little current (at 24V: 0.03A when the motor does not spin vs 0.17A when it does). I suspect my colleague has reflashed the board with default parameters that are designed for small motors (to prevent burning such motors of a new user if i remember right)

I will let the system turning back and forth for the entire night

oskar, would you by any chance know of a given set of parameter for my motors that are likely to work better (and yes I will go through the motor params procedure in the README section later).

Thanks

nope… it stopped spinning after about 10min.
And the noise of the motor is weird, it looks like it does not receive enough power (same noise my son’s remote controlled toy car makes when it’s very low on batteries). but i’m operating it with a power supply connected to main power (Rohde & Schwarz DC POWER SUPPLY NGSM 32/10)

besides, once the motor no longer moves, if i try to move the motor by hand as the script continues to run, the motors will try to spin but will not spin back. so it looks like the USB communication does work but the board does not provide enough power for the motor to spin.

i tried with 18V but observed same behavior.

oskar, would you have a quick an easy motor params for my turnigy aerodrive 5065 236 by any chance ?

We did recently find a bug that causes intermittent deadlock on the USB communication thread. This would make the ODrive unresponsive on USB until you reboot it.
I just released a patch update fw v2.1, which is now pushed to master. See if you can flash this, and if that fixes your communication issues?

However the issue you describe about the weird noise in the motor I’m not sure is related. Could you possibly record a video showing what’s going on? I am also happy to do live support over Skype if you prefer. I’m also in the ODrive Discord chat:

just updated to commit a7b11f8c6db30af55302b8b0343bd9b601579208
(see the other thread to fix the compilation error, which is gone)

unfortunately i observe the same behavior. after about maybe 20 to 40 commands sent through USB, the motor no longer moves. To be more precise, i modified the script to show USB errors (see my modification to test_bulk.py below)
the
‘ODriveBulkDevice’ object has no attribute ‘recieve’

seem to happen all the time, regardless of the motor actually moving or not.

So it really looks like it is not really related to the USB communication channel, but more something about the board itself. Besides, it the last working command execution (or maybe the 2 latest commands), the motor makes a slightly different sound , as if running out of battery (even if i connect it to a power supply) or as if it would spin slower. it’s hard to say but the sound is clearly different.

maybe I should adapt default motors parameters (i did not touch that part). could it be related ?

I’ll shoot and post a video of the behavior soon.

root@MTBD00694:~/ODriveFirmware/Firmware# cat ../tools/test_bulk2.py 
#! /usr/bin/env python3

import argparse

def parse_args():
  parser = argparse.ArgumentParser(description='Talk to a ODrive board over USB bulk channel.')
  return parser.parse_args()

if __name__ == '__main__':
  # parse args before other imports
  args = parse_args()

import sys
import time
import threading
from odrive import usbbulk

running = True
ready   = False

def main(args):
  global running
  print("ODrive USB Bulk Communications")
  print("---------------------------------------------------------------------")
  print("USAGE:")
  print("\tPOSITION_CONTROL:\n\t\tp MOTOR_NUMBER POSITION VELOCITY CURRENT")
  print("\tVELOCITY_CONTROL:\n\t\tv MOTOR_NUMBER VELOCITY CURRENT")
  print("\tCURRENT_CONTROL:\n\t\tc MOTOR_NUMBER CURRENT")
  print("---------------------------------------------------------------------")
  # query device
  dev = usbbulk.poll_odrive_bulk_device(printer=print)
  print (dev.info())
  print (dev.init())
  # thread
  thread = threading.Thread(target=recieve_thread, args=[dev])
  thread.start()
  steps = 9999
  sent_cmds = 0
  while running:
    time.sleep(0.1)
    try:
      #command = input("Enter ODrive command:\n")
      command = "p 0 "+str(steps)+" 0 0"
      dev.send(command)
      steps = -1 * steps
      sent_cmds += 1
      print(str(sent_cmds))
      time.sleep(3.0)
      
    except:
      print("there was an error in the main loop...")
      running = False

def recieve_thread(dev):
  global ready
  while running:
    time.sleep(0.1)
    try:
      message = dev.recieve(dev.recieve_max())
      message_ascii = bytes(message).decode('ascii')
      print(message_ascii, end='')
      if "ODrive Firmware" in message_ascii:
        ready = True
    except Exception as e:
      print("there was an error in the receive_thread() function: "+str(e))
      pass

if __name__ == "__main__":
   main(args)
root@MTBD00694:~/ODriveFirmware/Firmware#

Here’s a video with the behavior.

look in particular at about 00:12 and 2:30
(unfortunately someone called me on my mobile while I was shooting the video with it so the recording stopped before the behavior would repeat a 3rd time)

it’s so frustrating :cry:

How can I move forward? Should I touch the default motor parameters? (reminder I have mounted a turnigy aerodrive 5065 236)

Hi @alexisdal,
Sorry that it’s so frustrating for you, I assure you we will make sure you get it working.
Thanks for the video, this really helps. I agree with you that it seems it’s not a communication problem, since nothing changes. I can also see on the power supply current draw that the motor is trying to go, but “runs out of power” like you say.

This makes it really look like the encoder is slipping: either mechanically, or electrical noise is making false counts.
Can you please check very carefully that the coupling between the motor and encoder is good? Otherwise, we need to check with an oscilloscope the signal integrity of the encoder pulses.

Thanks for your patience!
Cheers,
Oskar

no problem i’m determined to go further.

I belive the mechanical coupling should be ok because if i remember well

  • on one side the encoder’s shaft is tapped (it is not entirely round, there is a flat area, the profile is like a D). so with two screws, I doubt it can easily slip
  • on the motor side, it’s shaft is round (profile is a circle and not a D). I beleive we made a small hole with a drill so that the screw would physically lock the shaft and could not slip. But i’m unsure. will check asap

However, it was 3D printed, I had to drill it to make sure the size of the shaft would be ok. Beside, depending on how you turn the screws the shafts may be not perfectly aligned (i did try to make a nice alignment though. it’s probably not perfect)

i will try to change the encoder too.
the one I have comes from aliexpress
https://www.aliexpress.com/item/Free-Shipping-Encoder-600-P-R-5V-24V-Incremental-Rotary-AB-2-Phase-6mm-Shaft-Coupling/32641634967.html
I ordered two units just in case.
and two “more serious” encoders from CUI
Digital Encoder Radial CUI AMT10x
490-AMT102-V
http://www.mouser.fr/ProductDetail/CUI/AMT102-V/?qs=sGAEpiMZZMvy8cVzszrmR8e0mUKZ7grDKaNMqTNrY3iTjPe3k%2bpUaQ%3D%3D

This one seems to be much better (much more compact). but i just didn’t know how to mount them and didn’t know for cabling either.

i have never had to use an oscilloscope to check the functionality of the encoder.
What pins should i connect to the oscilloscope then?

by the way, the resistor is
https://www.digikey.com/product-detail/en/te-connectivity-passive-product/HSA50R47J/A102181-ND/2056131
(don’t know if that can have an influence as well?)
I may have another unit (in general i order by 2 to be able to swap them in case of failure)
I must have another physically looking resistor with other values. must check first what we have in drawers.

hope that helps

Hi,
Thanks for all that detailed information.
You can order a metal coupling for cheap from Aliexpress: Link. Of course the shipping takes quite a while, but they do a better job than a 3D printed one.

Let’s try with a verified or changed encoder coupling, or even swapping encoders, first, before we go into checking with the Oscilloscope.
The AMT10x encoders use normal 2.54 male pins, you can use standard female jumper cables to interface.

i think i nailed the problem.
the mechanical connection between the motor’s shaft and the coupling was slipping after a while. Unlike what i remembered the shaft was not tapped/drilled, so the screw was slipping on the shaft. so basically the odrive would most likely send micro-impulsions to the motor, but would observe no movement => it probably explains the behavior.

I tapped the motor’s shaft with a power tool and now it no longer slips even if i brutally force it.

About 250 commands have been sent now (and counting). i’ll let it roll for en entire night, maybe the entire weekend (do you think there is a risk of fire? the cables are cold/normal).

Do you think you could change the software so that the odrive returns an error code in case something’s wrong? or is it bad idea? (my guess is that most users wouldn’t read error code no?)

Ok great! I can’t guarantee anything about fire risk. Personally I wouldn’t be too scared to leave it overnight; especially if everything is left on a non-flammable surface, like stone floor or concrete.

Cheers,
Oskar

it’s on a desk. it continued the test all night long and today as well.
over 26 000 commands.
no particular problem
no heat anywhere (not on the cables, not on the resistance, not on the motor (i dont dare touching the board coz i’m scared of making shorts)

i think i’ll keep it turning all weekend.

i’m building trust with ordive now :smiley:

next week i have a 2 hours time windows on monday afternoon. i’ll use it to follow the procedure and compile a low_leve.c with values pushing the limits (instead of defaults values where only 0.200A are consumed). then will let it spin back and forth like this for an entire week
Then i’ll build a 2 motors jig to simulate some GCODE moving both motors to random XY locations…

  • add some physical load/weight on the motors shaft with pulleys and continue the testing…
1 Like

330 000 commands and still going without any apparent issue.
the motor has been spinning back and forth 24/7 for about 12 days now.

i hope i’ll move on to something more serious soon

2 Likes

Thanks for taking the time to do this endurance test. Just for reference, can you tell us what git commit you were running? The command to find out is git describe --always. Thanks.

root@MTBD00694:~/ODriveFirmware# git describe --always
a7b11f8

393 000+ commands and still spinning. i did not have the time to change the testing environment (compiling fw for more suited motor parameters + adding weight/pulleys on the shaft. So i let it spin back and forth.

i find it very stable.

I can observe that while spinning in one direction (clockwise) the stop is a quick cut. However, in the other direction (counter clockwise), as it’s about to stop the movement, it continues the movement for i’d say maybe 10 degrees. But it’s been like this ever since i started playing with odrive and that motor + it’s systematic. I’m guessing that once I use more suited params (basically with more torque), the behavior should vanish?

=> Do you want a video of the behavior?

Thank you.
Yeah it makes sense that for un-tuned parameters you will get some overshoot, and yes it’s normal no have slight asymmetry, there are many sources of small disturbances. My guess is that the motor cogging torque is the contributor here.
No need for a video, just good to know that the communication seems to be rock solid.

I was doing something else on another PC and suddenly the motor stopped spinning

apparently it’s a software issue on the PC sending commands (according to the last error message). it overall made 520 000+ commands 24/7 for over 19days. The test did not try to close/open the USB communication for each command but merely reuse an already opened com port + there wasn’t someone or a video analysis performed to actually check that the motor did spin for every single command.

all in all, this is stable enough for me :smiley:

there was an error in the receive_thread() function: ‘ODriveBulkDevice’ object has no attribute ‘recieve’
there was an error in the receive_thread() function: ‘ODriveBulkDevice’ object has no attribute ‘recieve’
there was an error in the receive_thread() function: ‘ODriveBulkDevice’ object has no attribute ‘recieve’
524557
there was an error in the receive_thread() function: ‘ODriveBulkDevice’ object has no attribute ‘recieve’
there was an error in the receive_thread() function: ‘ODriveBulkDevice’ object has no attribute ‘recieve’
there was an error in the receive_thread() function: ‘ODriveBulkDevice’ object has no attribute ‘recieve’
there was an error in the receive_thread() function: ‘ODriveBulkDevice’ object has no attribute ‘recieve’
there was an error in the receive_thread() function: ‘ODriveBulkDevice’ object has no attribute ‘recieve’
there was an error in the receive_thread() function: ‘ODriveBulkDevice’ object has no attribute ‘recieve’
there was an error in the receive_thread() function: ‘ODriveBulkDevice’ object has no attribute ‘recieve’
there was an error in the receive_thread() function: ‘ODriveBulkDevice’ object has no attribute ‘recieve’
there was an error in the receive_thread() function: ‘ODriveBulkDevice’ object has no attribute ‘recieve’
there was an error in the receive_thread() function: ‘ODriveBulkDevice’ object has no attribute ‘recieve’
there was an error in the receive_thread() function: ‘ODriveBulkDevice’ object has no attribute ‘recieve’
there was an error in the receive_thread() function: ‘ODriveBulkDevice’ object has no attribute ‘recieve’
there was an error in the receive_thread() function: ‘ODriveBulkDevice’ object has no attribute ‘recieve’
there was an error in the receive_thread() function: ‘ODriveBulkDevice’ object has no attribute ‘recieve’
there was an error in the receive_thread() function: ‘ODriveBulkDevice’ object has no attribute ‘recieve’
there was an error in the receive_thread() function: ‘ODriveBulkDevice’ object has no attribute ‘recieve’
there was an error in the receive_thread() function: ‘ODriveBulkDevice’ object has no attribute ‘recieve’
there was an error in the receive_thread() function: ‘ODriveBulkDevice’ object has no attribute ‘recieve’
there was an error in the receive_thread() function: ‘ODriveBulkDevice’ object has no attribute ‘recieve’
there was an error in the receive_thread() function: ‘ODriveBulkDevice’ object has no attribute ‘recieve’
there was an error in the receive_thread() function: ‘ODriveBulkDevice’ object has no attribute ‘recieve’
there was an error in the receive_thread() function: ‘ODriveBulkDevice’ object has no attribute ‘recieve’
there was an error in the receive_thread() function: ‘ODriveBulkDevice’ object has no attribute ‘recieve’
there was an error in the receive_thread() function: ‘ODriveBulkDevice’ object has no attribute ‘recieve’
there was an error in the receive_thread() function: ‘ODriveBulkDevice’ object has no attribute ‘recieve’
there was an error in the receive_thread() function: ‘ODriveBulkDevice’ object has no attribute ‘recieve’
there was an error in the receive_thread() function: ‘ODriveBulkDevice’ object has no attribute ‘recieve’
there was an error in the receive_thread() function: ‘ODriveBulkDevice’ object has no attribute ‘recieve’
there was an error in the receive_thread() function: ‘ODriveBulkDevice’ object has no attribute ‘recieve’
there was an error in the receive_thread() function: ‘ODriveBulkDevice’ object has no attribute ‘recieve’
there was an error in the receive_thread() function: ‘ODriveBulkDevice’ object has no attribute ‘recieve’
there was an error in the main loop…
there was an error in the receive_thread() function: ‘ODriveBulkDevice’ object has no attribute ‘recieve’
root@MTBD00694:~/ODriveFirmware/Firmware#

1 Like

I’m moving the continuation of @alexisdal’s usb reliability issues from Where are the trajectories? to here, to keep that thread about trajectories.

sometimes the USB connection breaks (i’m using the builtin python odrive lib) while the boards continue to operate (motors are still holding their positions

It looks like you ran the 24/7 tests using the ASCII protocol (the one with the p 0 1000 0 0 commands), right? If nothing else changed in your test setup (same PC, same OS, same cables and hubs, same motors, same current and velocity limits, same actual max current and max velocity) this would allow us to deduce that you current USB issues are not triggered by any of the following: STM USB code, electrical issues or PC-side kernel.

If any of the mentioned parameters changed, you might try to use a shorter USB cable to reduce the likelihood of electrical issues.

In any case, we might be able to rule out a bit more of the USB stack if you connect to the odrive with find_any(consider_usb=False, consider_serial=True, ...) and then configure the firmware accordingly (CONFIG_USB_PROTOCOL=native-stream).

sometimes USB continues to operate but motors no longer move

This sounds like a DRV fault. Can you check the error code of both motors? What’s your current_limit? Does it still occur if you disable M0?
Most recent tests indicate that the DRV fault triggers as a function of the M0 I_q setpoint and the M0 electrical phase.

Edit:
I finally took some time to do some burn-in tests on USB to track down the issue to here:


Changing the priority to osPriorityAboveNormal fixes at least the issue I’m observing. Before the mean-time-to-failure was around 100’000 read operations and now I’m appoaching 5M-values without any hick-up.