Hi, I’d like to propose using google protobuf for a communication protocol. I would be happy to help implement the same and look forward to all your views on the subject.
When I read the documentation for the communication protocol, it seems very opaque and obscure to me. I also do not quite get the reason for using the JSON files in the protocol. For every release, the protocol should be fixed, not something that keeps changing. I have not gotten into too much detail in the current communication protocol. Could someone give the thought process behind the current implementation?
It also looks like there would be decent overhead for the current method of communication. In some places, I’m seeing that it takes around 7s to download the file, initially?
What if we define all the fields and the data it should contain, perhaps make a table of “registers” in the odrive board that store the data and we read/write from it? Using protobuf, we should be able to access classes in the form of data.type1.typea.
Lets use a fixed, stream-based simple communication algorithm? Put this in a protobuf file, it supports python and c++ so making a library should be easy.
Protobuf also seems to have features for working with earlier versions of protocol and implementation on embedded devices.
Also, lets have a single communication protocol. For human-readable interaction, we can have a library, like the odrivetool.
Also, why not have the Odrive board emulate a serial over USB device? This way, it can be treated as a serial device rather than sending raw USB commands. Is there a large speed difference in the 2 methods?
FWIW, JSON is only used in the endpoint schema. It’s also not actually required that we download the schema - so long as the checksum provided in the packets matches the firmware’s, the device won’t know any different.
The JSON file enables the interactive odrivetool shell to work with any firmware version with zero configuration. But you don’t have to read the JSON file. As @metanoic said, you can simply assume your favorite endpoint layout and the device will ignore it if it’s wrong (because of a CRC mismatch). You could implement this transparently within the python library.
Protobuf’s are just a way to serialize and deserialize data, not a complete RPC protocol. It will only help with one layer of the protocol, namely to turn an language-native object into a packet of raw bytes and the other way around. You still need to write your own code to dispatch the messages, interpret them on the other side and then return a response.
The ODrive does connect both as a Virtual COM Port and a vendor specific device. The VCOM port runs a different protocol, but you can change this by setting <odrv>.config.enable_ascii_protocol_on_usb = False. The reason for accessing the USB device directly is that sometimes we had random halt conditions and the low level access is required to reset the USB device. I don’t think that there’s a major speed difference.
I am personally in favor of a single protocol on all interfaces but for some people it’s less intimidating to directly interact with the device using the ASCII protocol, rather than using a library. That’s why we kept it around.
Anyway, the current “native” protocol (aka Fibre v0.0.1) was developed to meet the basic user experience that we wanted without spending too much time on it. It was always the expectation that a lot of it will change in the future and this is also why it’s poorly documented. However I’m actually working on a major overhaul to get many of the shortcomings resolved. If you’re interested, I can elaborate on the new design or you can check out https://github.com/samuelsadok/fibre (devel branch) in a few days.
Apologies for the late reply, I was caught up in other work.
@metanoic@Samuel Thank you for the clarification, I should look into the protocol in more detail. Personally I think the JSON file makes things more complicated and slow. There could be a simple (unchanging) command to get the firmware version and the tool could adjust it’s messages based on that. For example, we’d just choose some .csv file corresponding to firmware version that contains the details. Or with protobuf, there is a .proto file which contains the various data parameters and their types - This is directly back-compatible, so it would work with all previous firmware versions (should, anyway).
I’m aware that protobuf would require additional layers to serialize and decode the messages, but it would be trivial in my opinion to add that: just put a header, length and crc before and after. The advantages that I saw were that one did not need to worry about decoding the message, back compatibility with older protocol versions and the ability to directly access the data in the form of classes: class.data1.value=<x>.
As for the ascii protocol, isn’t it easier to open up a tool like odrivetool and use that to interact with the device? Ascii messages aren’t really robust.
@Samuel it’s great to hear about Fibre, i checked out the documentation and am afraid it might be a bit over my head - But the goals seems absolutely amazing! I come from a robotics background and have extensively used Dynamixel servos, which operate on a UART bus. With dynamixels, we use a basic communciation protocol that’s worked very well for me over the past few years. My experience is unfortunately limited to such simpler protocols. But so far, I’ve used them successfully over various interfaces including Wi-fi. I was basing my thoughts on this dynamixel protocol. Seems simple and works well. What are your thoughts?
Could you also elaborate on the new design and how it might be different from a simple protocol like the one I linked to above?
PS: I use ROS in my robotics projects and look forward to making a ROS driver for the Odrive. I’ve got some results on a driver I created for the earlier V3.x firmware, having troubles with the new v4.x firmware. Since the dynamixel protocol works well with ROS, matlab and a host of other applications I see it as a good design, which can be made better/easier with integration of protobuf.
Yea that’s basically the direction I’m hoping things go (functionality and handshake wise - I hold no opinion on protobuff vs fibre vs whatver) and is how I’m modeling my c# client. Device and schema are separate, and the schema is stored and keyed by the crc, a separate tool generates the schema DLL’s from the schema archive and the consumer can just pass in whatever schema reference they want to deal with.
There is an additional property/endpoint being added (assuming this gets approved) to return the checksum of the endpoint list, which can be used to check the client’s version of the remote object against the real version and avoid fetching the endpoints/schema entirely (e.g. if it doesn’t match, you shouldn’t all willy-nilly get writing to properties as their locations may have changed between versions, or change behavior to accommodate the remote object’s version). This thing in particular was requested by @madcowswe so I’m certain it will be implemented fairly soon in some fashion or another.
@metanoic That’s great to hear! I’m not sure what is meant by Device and Schema, that line went over my head. Does schema refer to the list of functions in a protocol version? What is meant by the schema DLLs?
@madcowswe I read through your post on RPC - it seems that nanopb would fit the bill quite well. I’m not sure about gRPC - it looks like gRPC is geared towards use in web applications, like front/backend or data centers. There is no support for USB or UART as a transport mechanism. From this thread, it also seems gRPC would be too heavy to deploy on an embedded system.
I feel it might be better not to have an RPC system like this on the board. This would allow us to say, connect multiple boards on the same UART or CANbus easily from what I’ve understood so far. Looks like for RPC, one would need a separate communication channel for each server (like a unique port number or USB ID). What are your thoughts?
The run-time transmission of the JSON should theoretically be near instantaneous, it’s just that the data exchange in general is inefficient in Fibre v0.1.0. For something like an interactive shell I think the shell should not require any prior knowledge about the device’s features. Otherwise you slow down rapid prototyping and quickly get into versioning problems. For actual production applications Fibre will have a code generator that takes an interface definition as input.
What do you mean by open up? odrivetool is based on a module that any Python application can use to control ODrives. There is currently no official client side C++ library, but there will be.
Personally I prefer RPC protocols over message based protocols like Dynamixel (or ROS for that matter). I think it’s important to have a communication framework that seamlessly integrates with the programming language. RPC protocols tend to do that to some extent. Certainly you can get the same job done with message based protocols but you will need boilerplate code that scales linearly with the project complexity.
The future of Fibre is completely network agnostic, that means you can connect two devices over multiple channels or relay messages across multiple Fibre nodes.