I’m running a setup for durability testing of a component. ODrive runs a motor to repeat a rotation task in sessions of 1000 cycles of motion. After 3 sessions of such repeating tests (i.e., 3000 successful movements), ODrive gave an error I cannot interpret:
16777216 is 1000000 in hex, and translates to MOTOR_ERROR_SYSTEM_LEVEL.
What does it mean?
Thank you in advance.
Edit: I tried re-running the setup, and the same error occurred, this time after approximately 2600 repetition of the movement (after 2 entire sessions of 1000, plus ~60% of the following).
I’d share the dump, but after the last crash I could not reconnect to the ODrive and had to reboot it. Also, the logged data you see there actually show you all errors and I’m sure they are extracted correctly. A dump would match exactly the same output.
Anyway, I think this actually might make sense.
I have 2 ODrives connected to a machine running Ubuntu.
I run 2 separate python programs to control them at the same time.
After some crashes, and knowing this never happened with 1 ODrive only, I think that could be the problem.
Are there known issues when connecting 2 ODrive at the same time? Or better, to have 2 ODrive controlled by 2 different processes?
Is there a good way to do it?
How are you connecting to the ODrives? Through the official python package (based on the libfibre backend) or through a custom implementation? libfibre claims all matching USB interfaces so I would expect the second script you start not to work at all but to fail with “couldn’t claim interface”.
This is a limitation that we’d like to fix at some point but it may take a while.
I use the odrive library and use the command odrive.find_any.
I tested the following successfully (though I admit sometimes the “could not claim USB interface” was a bit inconsistent in setting things up):
Open Spyder (a Python API), and run my first script containing:
import odrive
odrive.find_any()
Connect the second ODrive, open a second instance of Spyder (spyder --new-instance, or something like that) and run my second script containing also:
import odrive
odrive.find_any()
To clarify things, I tried also using find_any(serial_number = “X”). Both worked with some hick-ups.
Both ODrives successfully ran for a few hours until the error of which in the original post happened.
If this is not supposed to work, or you’re suprised it does, would you please let me know how I could do this instead? E.g., maybe running 2 scripts on 2 different docker instances (forgive me if the jargon is wrong, I am not too familiar with it) assigning them different hardware interfaces, including 1 USB port for each?
Or virtual machines? Or running Spyder instances on separate cores?
Yes I’m somewhat surprised this works, maybe Spyder opens some shared backend process so the scripts actually ends up running in the same process, but I’m not familiar with Spyder so this is speculation.
Is it 100% necessary to run two separate scripts? Could you maybe run one script with two separate threads? Or a “hub” script that passes through commands from the per-ODrive scripts through some inter-process communication method of your choice?
Otherwise yeah two docker containers should work but it seems like an overly heavy workaround.
Could indeed be that even a new instance of Spyder is somewhat sharing the backend with others. Or perhaps your library actually can get “fooled” by connecting a new ODrive via USB after the first Python script started, and starting a new one.
Threading could be an option, but it would still require some redesign of my software, which at the moment is more like a script. I could turn it into a parameterised function and run multiple threads, but then I’d have to find also a way to manage and monitor the threads separately.
Never done neither threading nor Docker business. Both seem having good potential for different reasons. Have you ever tried something similar with ODrive? Do you perhaps have an open source project to share that I could start with?
I don’t have any example to share but starting a thread in Python is as simple as threading.Thread(target = [some function], args = [tuple with args]).start().
You can even import your script from odrivetool and start the thread from the interactive console. Operations on ODrive objects are thread-safe.
So a priori, without knowing your script, I think this would be much easier and much less room for breakage than going through docker.
@Samuel, I tried multithreading, but what happened is that the first thread basically took too much resources, and the second shrieked and collapsed as soon as it started going into torque control.
Multithreading, to my understanding, is restricted, in Python, to a single core.
I tried also multuprocessing, which allows multiple cores, but I ran into the same problem I had before: the first process takes over all USB ports, and leaves no chance for other processes in other cores.
So, I’m now trying to work it out with Docker… Never worked with it before, so if you happen to have any knowledge on how to set it up to:
Access 1 USB port (for ODrive)
Run on 1 core
Run Anaconda for Python and ODrive
Set up a volume to share data with host computer
I’d appreciate your input. I guess having such container for ODrive could turn useful also for you.
@Samuel I’ve been spending more time trying to solve this, using Docker.
I currently have a Docker container with Anaconda and all dependencies correctly installed. I’m almost there, but when I’ve run odrivetool from inside the container, I got:
Device permissions are not set up. Please exit odrivetool and run the following command:
At the point of running udevadm control --reload-rules, I got this error: Failed to send reload request: No such file or directory
Is there any specific thing I should do for udevadm to be able to reload rules from inside the container? Any idea why it complains?
Thank you.
Edit: I found out that with service udev restart I could get away with that specific error, but even then, even if I open odrivetool, and the message inviting me to update rules disappeared, I cannot see ODrive…