List of relevant technologies
REST, Request/Response and Serializations
REST is an architectural style that is not dependent on any other protocol. For the web case HTTP/HTTPS is a common choice. Just like protocol independence, REST doesn't depend on a specific language. Each call is independent from the other and each call contains the necessary information to complete successfully, making REST a successful choice for designing stateless services. Because of the above, REST can be quite flexible on term of the underlying protocol/technology choice: is used in multiple places and is very common in IoT or when interacting with cloud based services. An alternative to HTTP for REST is CoAP, a very lightweight protocol that supports the HTTP methods and error codes represented in a binary form instead of the HTTP textual representation. Unlike SOAP, REST supports multiple format including JSON, XML and YAML. However, with the rise of HTML5, JSON seems to be the preferred choice while implementing REST via HTTP(S).
Indeed, JSON is much simpler than XML. JSON has a much smaller grammar and maps more directly onto the data structures used in modern programming languages. Compared to XML, JSON is more compact and less convoluted which makes it more readable for a Human. It needs less resources to be processed/transferred because of the smaller size and its simplicity compared to XML.
JSON Integrates well with most programming languages due to the mapping between JSON data types and programming languages data types. Efficient libraries exist to deal with JSON (json-c for C as an example). JSON schema standard was developed to allow the validation of JSON instances.
In an IoT enviroment, or in general when the bandwidth is limited, an alternative to JSON is offered by Google Protocol Buffers. It is a binary serialization toolkit that let the user to describe a message structure in a .proto file, and provides extensible generators for different languages that produce serializers and deserializers for the message described. The focus of Google Protocol Buffers is to increase the bandwith available reducing the size of a message in a strong typed language environment. When the goal is to increase the CPU performance (even despite the size of the message), Google's Flatbuffers is an interesting alternative.
Broker-based and Publish/Subscribe communications
Aside from the client-server approach of HTTP and CoAP solutions, protocols and middlewares like DBus and MQTT rely on a different kind of architecture, where a central broker is used to gather and distribute all the data.
MQTT is a protocol that relies on a publish/subscribe message exchange pattern: clients are peers that connect to the broker and subscribe and publish data to topics, implementing remotely the usage of the observer pattern. When a client publish a message to a topic, the broker forwards the message to all the other clients that are subscribed to the same topic. MQTT has support for persistent messages stored on the broker.
MQTT typically need a small code footprint, so it is ideal if processor or memory resources are limited. Having a very small packet overhead, it is a good choice if bandwidth is low. It uses three different QOS levels (at most once, at least once, exactly once), making it useful also on unreliable networks.
Typical MQTT implementations work on top of TCP/IP, optionally using SSL/TLS. MQTT over websocket is possible (making the browser to be an MQTT client). Authentication is based on username and password.
In the IT world, MQTT is a very wide spread protocol. Client libraries in most of the programming languages as well as many different kind of brokers are available publicly.
On non-TCP/IP networks (e.g. Zigbee) is used MQTT-SN (protocol for sensor network).
Even if being natually publish/subscribe, request/response message exchange pattern are possible. MQTT5 adds in the protocol the possibility to send a response topic when publishing a message, while with MQTT3 this logic is demanded to the user.
The payload for an MQTT message is a free-form byte buffer, this meaning that MQTT protocol do not inspect neither handle specific serializations in the user payload. A different approach is taken from DBus.
D-Bus is a message bus system. Like MQTT, relies on a broker (called usually D-Bus daemon) gathering and dispatching messages. The messages are described in D-Bus interface files (XML format). The D-Bus code generator helps to generate marshallers/unmarshallers for such messages, that can then be used for RPC over D-Bus.
The broker has access to the interface files, making it possible to configure the access and the usage of specific interfaces.
The message bus is built on top of a general one-to-one message passing framework, which can be used by any two apps to communicate directly (without going through the message bus daemon). Typically unix domain sockets are used for communicating applications within the same host, while TCP/IP is also suitable.
Autonomous Driving: ROS and Apollo's CyberRT
A trend emerging in the last years in the automotive industry are the Advanced Driver-Assistance Systems (ADAS), that should help the driver during the driving at different levels, up to the complete automated driving. Such systems today rely on protocols specifically developed or borrowed from the robotics industry. The most notable example is ROS.
ROS (Robotics operating systems) is an opensource initiative that provides a developer environment for creating robot applications. The ROS architecture is based on nodes (processes) that perform local or distributed computation. The nodes exchange messages each other in a publish/subscribe fashion. The protocols used for such communication is commonly referred in the community as ROS1 and - its successor - ROS2.
In ROS1, the broker is substituted by the concept of Master. Is is a process that provided service registration and lookup, however does not dispatch the messages between the clients, that communicate directly after having found each other via the Master, that offers indeed a discovery service. A Node that provides a service interacts with the Master via http/xml (xmlrpc) advertising the topics on which he would receive messages, as well as the specific data format. Other clients can query via xmlrpc the master for finding such client based on the topic and the data format used. Then, a binary protocol is used between the two clients with a publish/subscribe paradigm like in mqtt, with the difference that the underlying communication is direct and is not routed by the Master.
In ROS2, the system is completely decentralized and there is no Master. It is built on top of DDS/RTPS middleware, which handles not only discovery, but also serialization and transportation: each Node advertizes itself to other nodes on the network, that respond with the information about themselves.
Protocols other than the binary ones are available via rosbridge. It provides a JSON interface to ROS and adds support for more transport layers, such as Websockets and TCP. Modifications by LGE to rosbridge make possible to use Google Protocol Buffers instead of JSON.
Unlike ROS, Apollo Cyber RT is an opensource framework developed purposely for ADAS. Compatible with ROS up to version 3.0, apollo-ros adds shared memory support, RTPS, Google Protocol Buffers and decentralization.
Starting with version 3.5 Apollo is not compatible with ROS, Its main benefits seem to be high performance and the user level scheduler, that let the developer to tune the system according to the application and hardware resources.
Franca with CommonAPI, SOME/IP and ARA::COM
In the automotive industry, Franca is a powerful framework for definition and transformation of software interfaces. It is used for integrating software components from different suppliers, which are built based on various runtime frameworks, platforms and IPC mechanisms. The core of it is Franca IDL (Interface Definition Language), which is a textual language for specification of APIs.
If you want to generate C++ code directly from Franca interface specifications, CommonAPI C++ might be the proper solution. CommonAPI C++ is a standardized C++ API specification for the development of distributed applications which communicate via a middleware for interprocess communication. The main intention is to make the C++ interface for applications independent from the underlying IPC stack. CommonAPI was created in the context of the GENIVI initiative.
CommonAPI uses the interface description language FrancaIDL for the specification of interfaces (logical interface specification). Code generation from FrancaIDL is an integrated part of CommonAPI. The code generator for CommonAPI C++ bindings needs middleware-specific parameters (deployment parameters). These parameters are defined in Franca deployment files (*.fdepl).
CommonAPI is designed to decouple generated API from the actual IPC stack and cooperates seamlessly with Franca. It does so by providing bindings that allow to use any IPC mechanism semantically compatible to Franca; out-of-the-box are available DBus, WAMP and SOME/IP bindings.
Given the centrality of AUTOSARin the ECUs today, it is very interesting for the work group the usage of CommonAPI SOME/IP bindings.
SOME/IP is an AUTOSAR automotive/embedded communication protocol which supports remote procedure calls, event notifications and the underlying serialization/wire format. Such an automotive middleware solution can be used for control messages. It was designed from beginning on to fit devices of different sizes and different operating systems perfectly. This includes small devices like cameras, AUTOSAR devices, and up to head units or telematics devices. It was also made sure that SOME/IP supports features of the Infotainment domain as well as that of other domains in the vehicle, allowing SOME/IP to be used for MOST replacement scenarios as well as more traditional CAN scenarios.
While IT solutions often only support single middleware features (e.g. RPC or Publish/Subscribe), SOME/IP supports a wide range of middleware features:
- Serialization – transforming into and from on-wire representation (specified in the SOME/IP specification, but implemented in the CommonAPI and ARA::COM bindings) TODO .
- Remote Procedure Call (RPC) – implementing remote invocation of functions.
- Service Discovery (SD) – dynamically finding and functionality and configuring its access.
- Publish/Subscribe (Pub/Sub) – dynamically configuring which data is needed and shall be sent to the client.
- Segmentation of UDP messages – allowing the transport of large SOME/IP messages over UDP without the need of fragmentation.
CommonAPI is not the only framework that can use SOME/IP as underlying communication mechanism. Indeed, AUTOSAR Adaptive Platform offers a runtime for Adaptive Application. Such runtime, called ARA, contains its own communication middleware known as ARA::COM.
It offers bindings to SOME/IP, but bindings can be implemented also for any other technology that support publish/subscribe/event patterns. SOME/IP, however, remains the default transport layer available on the shelf for ARA::COM.
Just like CommonAPI used Franca for the definition of the software interfaces, ARA::COM uses the Service Interface Definition in an XML format (ARXML).
Part of this activity of the Generic Protocol Evaluation Project team was dedicated to the preparation of a poll to get the feedback from the automotive industry. The participants have been asked about their familiarity with the technologies taken in consideration and their importance for onboard (between ECUs) and offboard (to the Cloud) usage. Also, the participants were asked to point out with technologies they would recommend and which they are actually using.
According to the results, the technologies people are most familiar with are JSON, D-Bus, and HTTP/REST. SOME/IP, Franca IDL, MQTT and CommonAPI also are well known.
When asked about the importance of a given technology for communication between ECUs in the vehicle, SOME/IP and Franca IDL got the highest scores, followed by JSON and CommonAPI.
The scenario changes for communication outside of the vehicle (e.g. to the Cloud), where JSON and HTTP/REST are the clear winners, followed by MQTT.
The most used ones today seem to be also the most recommended: JSON, HTTP/REST and FrancaIDL. D-Bus is more used than recommended, while gRPC is more recommended than used.
Based on the answers, was possible to calculate a popularity score, shown in the following diagram:
In general the trend is quite good for the couple JSON and HTTP/REST, that have the highest scores in our overall popularity score, followed by SOME/IP and FrancaIDL. A little far away, also MQTT and CommonAPI got a fairly good result.
TODO same next paragraph for outside
SOME/IP, Franca and CommonAPI are the preferred option according to our poll when it comes to communication between ECUs. Indeed, it is already available a product that uses the three of them: the CommonAPI Franca-based generators and runtime with SOME/IP bindings. It was then of great interest in our team to understand at which level such toolkit could interoperate with an important ecosystem in the Automotive Industry represented by AUTOSAR ECUs. Out focus went in the AUTOSAR Adaptive Platform because it represents a big trend among OEMs.
When designing the current generation of connected vehicles, the automotive industry has to cope with too many choices, too much diversity, too much boiler-plate code, adaption layers, and incompatibility. There are a lot of communication protocols available, each one with different focus. During the last months, the Generic Protocol Evaluation Project Team identified SOME/IP and Franca IDL (often used together with CommonAPI C++ bindings) as two of the preferred industry options. On the other side, AUTOSAR and its ARA::COM communication middleware are very important in the automotive environment. When it comes to inter-domain communication between ECUs, messages have to be translated between CommonAPI and ARA::COM. This is not only tedious, but prone to human errors. However, both communication technologies are based on model definitions (.arxml and .fidl/.fdepl files). This opens the possibility to translate using model-to-model transformation methods that can improve software quality and reduce development time and engineering costs. In this document we present a tool that uses model-to-model transformation to achieve a compatible code generation on both sides of ARA::COM and Franca IDL. When combined with CommonAPI bindings and ARA::COM runtime, this achieves a runtime translation between the systems. Using such a tool makes it possible to have a specification in a single format (we propose Franca IDL for this), and yet to use the full advantage of both technologies on both sides of the communication.
The Franca project offers not only an interface definition language (Franca IDL), but also a framework for building model-to-model transformations. This framework is being used here to implement transformations from Adaptive AUTOSAR models to Franca IDL and vice versa. These transformations can be used as part of any current Eclipse IDE. For build automation and Continuous Integration (CI) it also useful to deploy the transformations as a command-line tool. The goal of the automatic transformations is to apply code generation by AUTOSAR-compatible code generators as well as Franca-compatible generators (e.g., CommonAPI C++) in a way that leads to transparent communication between both systems at runtime. Therefore, the tooling is based on a proper mapping between Adaptive AUTOSAR concepts and Franca IDL concepts. For example, each operation on an AUTOSAR service interface is mapped to a method in Franca IDL. The following diagram shows how the transformation tooling interacts with the code generators. The generated code on the AUTOSAR and GENIVI subsystems is using the SOME/IP protocol for communication. As the subsystems are integrated on model level, the communication is automatically compatible.
The ARA-to-Franca tool is implemented using the Xtend language, which is an extension of Java providing language features which ease the implementation of model-to-model transformations. It uses Artop (release 4.10) as an implementation of Adaptive AUTOSAR and Franca 0.13.0. Both are using the Eclipse Modeling Framework, which provides a common way of working with models.
The Franca ARA Demonstrator was built to provide proof of interoperability for a wide range of Franca IDL artifacts like message and data types and its mapping to Adaptive AUTOSAR. The setup consists of four Ethernet connected ECUs running the GENIVI, the Adaptive AUTOSAR and the Classic AUTOSAR platforms. Together, those ECUs form an Emergency Brake Assistant with visualizations on the GENIVI IVI system.
The common basis for communication between the ECUs is established using SOME/IP. However, on top of the protocol stack, different middleware such as CommonAPI and ARA::COM is used as binding to the applications. The following steps were executed to establish the communication between the applications:
- Interface definition using Franca IDL
- Transformation of the Franca IDL interface definition to an equivalent ARXML definition using the Franca model-to-model transformations
- Creation of the corresponding SOME/IP deployments for CommonAPI and ARA::COM
- Generation of the proxies and skeletons for both platforms
- Integration of the generated code into the demo applications. A clear advantage of following this approach is that there is no need to define the same interface twice in Franca IDL and AUTOSAR ARXML. This reduces errors coming from manual maintaining the service interfaces by having only one source of origin.
The results of this project consolidate the use of Franca-based descriptions of interfaces when designing complex systems by enabling a seamless integration of GENIVI APIs and components into the automotive software engineering processes used for autonomous vehicle functions (a.k.a. Adaptive AUTOSAR). The current implementation of the model transformations from Adaptive AUTOSAR to Franca IDL and vice versa is prototypical in several aspects. It is sufficient to support the requirements of the Franca ARA Demonstrator, but has to be enhanced significantly in order to be used for production development in actual automotive projects. The project demonstrates the value and relevance of the GENIVI vehicle domain interaction strategy through the delivery of tangible and useful technology (e.g. code generators) to implement the interfaces of automotive complex systems. Overall, the next step is to develop a (near) production-ready tool, which has the goal of reducing the cognitive load of developers as much as possible. This will lead to less errors and a more reliable communication between AUTOSAR and IVI systems.
The team researched the current technologies used inside and outside the Automotive Industry for model-based development and communication. We asked the opinion of some representative of the Industry, that confirmed the importance of Franca+CommonAPI+SOME/IP and AUTOSAR Adaptive software stacks in their current and future plans. Therefore, we demonstrated possible way to interconnect both environments introducing a new tool.
During CES2019 and internally in OEMs and Tier1s, there has been good feedback for such tool also in accordance of the future plans of development. AUTOSAR Adaptive platform and SOME/IP non-AUTOSAR ECUs are clearly having a trend in the near future, so would make sense to bring the tool to a production-ready level. Thanks to model-to-model transformations we could indeed provide a set of tools for easy and fast connection of domains (IVI, ASIL, DI, ADAS, Cloud, ...), either via new bindings for the Franca-based tools like CommonAPI, either as separated gateways generated by model definitions. \
For the same reason, we do also see the need extend Franca model and CommonAPI bindings to enable DDS, ROS, MQTT, REST, etc. in order to facilitate the interoperability of the traditional ECUs also with ADAS and cloud domains.
TODO: warning: may not be 100% compatibile
TODO: merge last two paragrapths somehow and franca bindings