I got a crash course in CANBus (OpenCAN) communications.
The application that I inherited did not work. OK, it exhibited some activity however there were a few flaws in the implementation:
The CANBus interface code was implemented as if CANBus was a serial link from the USB CANBus node attached to the PC to whatever remote device was being communicated with at the time. CANBus is a broadcast network. Because communications with a CANBus device was not "reliable" due to treating it as a serial link device with only that device connected, every single damn command was sent three times to the device. Why? Because sometimes another device would send a broadcast packet and this would appear on the bus in between the command to the remote device and the remote device's response. Therefore it wasn't working and the command was just sent three times and assumed to work. Yep, assumed to work...
Because the CANBus interface code was implemented as if it was serial link to a specific device, multiple devices operating simultaneously was impossible. This kind of worked when there was a single module being controlled, however the intention of the project was to have up to four identical modules, each with their own set of motors and sensor modules, therefore it was impossible to scale it.
The diagnostic and test code for the connected devices used different interface code to that used in the operating environment. Getting something working and configured in the diagnostic and test interface did not mean anything would work in the operating environment.
I/O modules were treated as polled devices rather than devices that broadcast state change. As a result, sensor triggers were only responded to when the I/O module was specifically polled for the current I/O state. The I/O state was maintained as a copy in the application itself and therefore was inevitably out of step with whatever was happening on the I/O device itself. The I/O device was, of course, asked three times in a row for the state... For those that don't know, most CANBus I/O modules are asynchronous and while they can be polled, their normal operation is to send a broadcast, or direct message to another device, on state change of their I/O signals. As a result, when an input on the I/O module was triggered, this would be missed until the control application happened to poll the device for the current status. This did not help operational efficiency or accuracy at all...
After fighting the existing code I threw every single part of it away and started from scratch. The project was already late at this point, but it was the only way to proceed and there was absolutely nothing of use in the previous application. The replacement application was fully multi-threaded, the CANBus interface processed and dispatched all received CANBus messages to whatever internal handler was interested in it which allowed a responsive state machine to operate and for this to be scaled to multiple modules.
The CANBus devices were proven to be reliable, previously it was reported that these devices were not where the only problem was the utterly inappropriate attempt at communicating with them as they were a singly connected serial device.
The project did continue to struggle however that was due to the systems that we were interfacing with not adhering to the stated method of operation that they were very loudly stated to adhere to. This changed a little after I connected to the on-site PLCs, downloaded the code from them, effectively reverse engineered it and then demonstrated that different sites had different, and incompatible, modes of operation...