Page tree
Skip to end of metadata
Go to start of metadata

Updated text (to be shortened)

Introduction

Five primary techniques for distributing graphics and HMI have been identified in the GENIVI Graphics Sharing and Distributed HMI project (GSHA) and we find that different solutions can be identified as belonging to those categories.

  • GPU sharing
  • Display sharing
  • Surface sharing, with sub-category Virtual Display
  • API Remoting
  • Shared state, independent rendering

This paper describes the technique that is named API Remoting.

API Remoting method

API Remoting involves taking an existing graphics API and making it accessible remotely (through a network protocol), thus enabling one system to in effect call the graphics/drawing API of another system.

In some cases an existing programming API, such as that of a local graphics library, is simply modified or extended to provide network-capability it did not have before. In other words a local programming API is translated into an RPC (Remote Procedure Call) interface. In other cases, new custom programming interfaces might be created that are specially tailored for the remote graphics use case.

Many programming interfaces are already networked interfaces, and there is also a huge amount of technologies for making local programming APIs network-capable, in other words independent of the location of the requesting and receiving software components. The sister project in GENIVI Domain-Interaction, named "Generic Protocol Evaluation" deals with analyzing some of those methods.

What we consider distinctive to Graphics Sharing through API Remoting, is that the network API in question is graphics/drawing related. In the simplest case we are dealing with specific "drawing commands" invoked by system A on system B. The actual creation of images is executed on the remote system B. Thus it's different from Surface Sharing, wherein the drawing/rendering of an image would be completed on system A, and then the image itself would be transferred.

The API in question could be low-level - consider simple functions such as "Draw Line/Box/Circle, or Fill-Area at these coordinates" - or it might be designed at a higher abstraction level.

The imperative-programming style functions like "Draw line" are typically associated with 2D-graphics whereas the frameworks associated with 3D graphics would often be less direct and instead descibe a 3D-scene using objects, textures, animations and object-relationships. This will require a more advanced processing on the receiving side that renders the graphics and it therefore involves more advanced library algorithms and of course the use of a GPU on the remote side .

Design Considerations

Various aspects should be considered when choosing the graphics sharing approach:

- Network bandwidth
- Latency affecting changes and animations
- Processing requirements and available hardware resources
- Complexity, compatibility, API versions and software upgrades.

For some cases, such as a drawing a simple Navigation map, sending drawing commands as opposed to images could be more bandwidth efficient than sending images and this is why API remoting deserves to be highlighted as a distinct approach. In parallel with this, if the processing effort to transform the instructions into graphics is significant, then the API Remoting also guarantees that this effort (in particular GPU usage) is spent on the receiving side. Finally, one should consider that programming interfaces between independent parts (in this case two separate systems) need to be in agreement to actually function. Since defining the drawing of an image requires lots of details, API Remoting is likely to lead to a larger interface on the boundary between systems than some other approaches. A large programming interface is statistically more likely to need updates and will at that point require both sides to update their software to stay compatible with each other.

<Picture>
System A <---> System B

Handheld device displays

Smart phones and similar devices are also an important way to interact with modern vehicles, and should be considered part of the extended HMI of the car. Whereas many smart phones solutions might use simply the Shared State - Independent Rendering method and the release of phone apps with their own HMI, it's also possible that some API remoting could be used in which the car can define and display an HMI on a mobile device.

This requires a framework or standard in which the car and the mobile phone agree on the distributed graphics interface. For API Remoting, the implementation of that agreed standard on the device side may still be implemented in "apps" so that it does not need to be directly implemented in the operating system of the device.

RAMSES

RAMSES is a framework for defining, storing and distributing 3D (can be also 2D) scenes for HMIs. From users perspective, RAMSES looks like a thin C++ abstraction of OpenGL commands. RAMSES offers the same feature level as OpenGL but allows higher levels constructs such as widgets to be built on top of it. In addition to basic OpenGL, RAMSES supports higher level relationships between objects such as a hierarchical scene graph, content grouping, off-screen rendering, animations, text, and compositing. All of those features follow the principle of API remoting, i.e. the commands (that define the state in the RAMSES case) can be sent to another system and the actual rendering is executed at the receiving side.

RAMSES distinguishes between content creation and content distribution; the same content (created with RAMSES) can be rendered locally, on one or more network nodes, or everywhere. RAMSES handles the distribution of the scenes, and offers an interface to control the final composition - which scene to be sent where, shown at which time, and how multiple scenes and/or videos will interact together. The control itself is subject to application logic (HMI framework, compositor, smartphone app etc.).

The data pipeline and timing are "immediate", i.e. the state of each scene is simultaneously pushed to all network participants. RAMSES provides a set of mechanisms to control framerate, timings and avoid jitter and frame drops as much as possible.

Usage in virtualized environments

Since API Remoting deals with simply commands sent over a network it should apply to virtualized environments with minimum adaption.  Whatever (virtual) network capability, the system provides will likely be transparent to the RPC system that is selected, and possibly some efficient techniques such as shared memory is employed in the implementation of the networking/RPC capability itself. As always, the impact of timing should be evaluated and system behavior may need to be confirmed with the real API Remoting framework and real HMI in place before correct behavior is fully confirmed.

Conclusions

API Remoting is an often bandwidth-efficient way to have particular graphical interaction giving the control of the graphics to one system while offloading the graphics rendering effort to the receiving side.

A possible disadvantage may be a potentially large shared API, which requires a synchronized update of both sender/controller and receiver when that API changes.

RAMSES is a framework that simplifies the programming of 3D graphics while at the same time enabling graphics to be distributed between system. The framework transparently takes care of the required distribution of data, at the right time, according to the graphical scene needs. Simplifying 3D programming serves a purpose even on one system but it also acts as a great example of a custom remote API for graphics distribution.




-----

OLD TEXT IGNORE FOR NOW


Introduction 1

<Intro, the need for unified HMI in vehicles, etc.>

Five different primary techniques for distributing graphics and HMI have been identified in the GENIVI Graphics Sharing and Distributed HMI project. This paper describes methods that  technique that was named API Remoting.

API Remoting method


API Remoting is the technique of taking an existing graphics API and making it accessible remotely, thus enabling one system to essentially call the graphics/drawing API of another system.

In some cases an existing programming API such as that of a local graphics library, is simply modified or extended to provide network-capability it did not have before.  In other words a local programming API is translated into an RPC (Remote Procedure Call) interface. In other cases, new programming interfaces might be created that are specially tailored for the remote graphics use case.

Many programming interfaces are already networked interfaces, and there is also a huge amount of technologies for making local programming APIs network-capable and independent of their location (local/remote). The sister project "Generic Protocol Evaluation" deals with analyzing some of those methods.

What we consider distinctive to Graphics Sharing through API Remoting, is that the network API in question is graphics/drawing related. We are dealing with essentially specific "drawing commands" invoked by system A on system B.  The actual creation of images is executed on the remote system B.  Thus, it's different from surface sharing wherein the drawing/rendering of an image would be completed on system A, and then transfer the image itself.

The API in question could be as low-level, as in "Draw Line/Box/Circle, or Fill Area at these coordinates". 

Apart from a very direct commandd ("Draw line"), typically associated with 2D-graphics, the remote API can also be more capable and "higher level" – in particular the defintion of 3D scene using objects, textures, animations and relationships.  This will require a more advanced processing on the receiving side that renders the graphics and it therefore involves more advanced library algorithms and of course use of a GPU on the remote side .

Various aspects should be considered when choosing graphics sharing approach:

  • Network bandwidth
  • Latency affecting changes and animations
  • Processing requirements

For some cases, such as a drawing a simple Navigation map, sending such drawing commands as opposed to images could be more bandwidth efficient than sending images, which is why API remoting deserves to be highlighted as a distinct and useful approach.  In addition to network bandwidth, there is also processing effort.  If the effort to transform the instructions into graphics is significant, then the API Remoting also guarantees that this effort (in particular GPU usage) is spent on the receiving side.

<Picture>  
System A  <---> System B


Handheld device displays

Smart phones and similar devices are an important interaction method to modern vehicles and should be considered part of the extended HMI of the car. Whereas many smart phones solutions might use simply the Shared State - Independent Rendering method, it's also likely that some API remoting could be used for the car to define and display an HMI on a mobile device. This requires a framework or standard in which the car and the mobile phone agree on the distributed graphics interface.  For API Remoting, the implementation of that agreed standard on the device side may very well be done by "apps" such that it need not be directly implemented in the operating system of the device.

RAMSES

RAMSES is a framework for defining, storing and distributing 3D (can be also 2D) scenes for HMIs. From users perspective, RAMSES looks like a thin C++ abstraction of OpenGL commands. RAMSES offers the same feature level as OpenGL but allows higher levels constructs such as widgets to be built on top of it. In addition to basic OpenGL, RAMSES supports higher level relationships between objects such as a hierarchical scene graph, content grouping, off-screen rendering, animations, text, and compositing. All of those features follow the principle of API remoting, i.e. the commands (that define the state in the RAMSES case) can be sent to another system and the actual rendering is executed at the receiving side.

RAMSES distinguishes between content creation and content distribution; the same content (created with RAMSES) can be rendered locally, on one or more network nodes, or everywhere. RAMSES handles the distribution of the scenes, and offers an interface to control the final composition - which scene to be sent where, shown at which time, and how multiple scenes and/or videos will interact together. The control itself is subject to application logic (HMI framework, compositor, smartphone app etc.).

The data pipeline and timing are "immediate", i.e. the state of each scene is simultaneously pushed to all network participants. RAMSES provides a set of mechanisms to control framerate, timings and avoid jitter and frame drops as much as possible.

Usage in virtualized environments

Since API Remoting deals with simply commands sent over a network it should apply to virtualized environments with minimum adaption necessary.  Whatever (virtual) network capability, the system provides will likely be transparent to the RPC system that is selected, and possibly some efficient techniques such as shared memory is employed in the implementation of the networking/RPC capability itself.  As always, the impact of timing should be evaluated and system behavior may need to be confirmed with the real API Remoting framework and real HMI in place before correct behavior is fully confirmed.

    

Conclusions

...




  • No labels