Page tree
Skip to end of metadata
Go to start of metadata

Distributed Graphics Control through API Remoting

Summary

Five primary techniques for distributing graphics and HMI between cooperating systems have been identified in the GENIVI Graphics Sharing and Distributed HMI project (GSHA). This describes API Remoting and briefly contrasts it with some of the other choices.

Key Characteristics

  • More bandwidth efficient (for certain cases)

  • Offloads rendering effort to receiver

  • Flexible, and the “sender” has detailed control over results

  • Potential complexity, needing synchronized software/API updates

Description

In the context of graphics-sharing, API Remoting involves taking an existing graphics API and making it accessible through a network protocol, thus enabling one system to effectively call the graphics/drawing API of another system.

In some cases an existing programming API such as that of a local graphics library, is simply extended to provide network-capability turning it into an RPC (Remote Procedure Call) interface. In other cases, new custom programming interfaces might be created that are specially tailored for distributed graphics.

Many programming interfaces are already network-capable, and there is also a huge amount of technologies for making local programming APIs independent of the location of the requesting and receiving software components. The sister project in GENIVI Domain-Interaction, named "Generic Protocol Evaluation" deals with analyzing some of those methods.

What we consider distinctive in this analysis is that the network API in question is graphics/drawing related. .

In the simplest case we are dealing with invoking graphics commands by system A on system B. The actual drawing/rendering is executed on the remote system B. Thus it's different from Surface Sharing wherein the drawing/rendering is first completed on system A, and then a finished image would be transferred. A 3D-scene can be quite complex and consist of different objects, textures, animations and relationships between them - so an efficient remote rendering API should reflect that.


A low-level and imperative-programming style API with functions like "Draw Line" is often associated with 2D-graphics while frameworks designed for 3D graphics often describe a 3D-scene using objects, textures, animations and object-relationships. The latter requires more advanced processing on the receiving side that renders the graphics and involves more advanced library algorithms and hardware (GPU) on the remote side .

Design Considerations

Various aspects should be considered when choosing the graphics sharing approach:

- Network bandwidth
- Latency affecting changes and animations
- Processing requirements on system A vs system B, and available hardware resources
- Complexity, compatibility, API versions and software upgrades

Bandwidth usage differs depending on if sending drawing commands or images.  As noted, the graphics processing demands for API Remoting are generally shifted to the receiving system B.  Finally, we should consider that a larger API between systems may require more frequent synchronized updates for systems A and B to stay compatible.

Please refer to the full whitepaper (under development) for further analysis of these aspects.

For some cases, such as a drawing a simple Navigation map, sending drawing commands could be more bandwidth efficient than sending images and this is one reason why API remoting is an important option. In addition to that, if the processing effort to transform the instructions into graphics is significant, then API Remoting also guarantees that this effort (such as GPU processing cycles) is spent on the receiving system instead of the controlling system. Finally, we should consider that programming interfaces between systems need to be in agreement to actually work. Since defining the drawing of an image requires lots of details, API Remoting is likely to lead to a larger interface on the boundary between systems than some other approaches. A large programming interface is statistically more likely to need updates and will at that point require both sides to update their software to stay compatible with each other.

<Simple RPC example picture>

Handheld device displays

Smartphones and similar devices are also an important way to interact with modern vehicles, and should be considered part of the extended HMI of the car. Many smart phones solutions might use the simple Shared State - Independent Rendering method but it's also possible that the car more directly controls the graphical view shown on a mobile  device. This requires a framework or standard in which the car and the device agree on the distributed graphics interface. The implementation of the framework might still be implemented in "apps" on the device side so that it does not need to be directly implemented into the operating system of the device. The RAMSES framework is cross-platform and has been tested on Android.

RAMSES

RAMSES is a framework for defining, storing and distributing 3D scenes for HMIs. From a user’s perspective, RAMSES looks like a thin C++ scene abstraction on top of OpenGL. RAMSES supports higher level relationships between objects such as a hierarchical scene graph, content grouping, off-screen rendering, animations, text, and compositing. All of those features follow the principle of API remoting, i.e. the commands (that define the state in the RAMSES case) can be sent to another system and the actual rendering is executed at the receiving side.

RAMSES distinguishes between content creation and content distribution; the same content (created with RAMSES) can be rendered locally, on one or more network nodes, or everywhere. RAMSES handles the distribution of the scenes, and offers an interface to control the final composition - which scene to be sent where, shown at which time, and how multiple scenes and/or videos will interact together. The control itself is subject to application logic (HMI framework, compositor, smartphone app etc.).

Usage in virtualized environments

Since API Remoting deals simply with commands sent over a network it should apply to virtualized environments with minimum adaption.  Any (virtual) network capability the system provides ought to be transparent to the RPC system that is selected. It’s possible that some efficient techniques such as shared memory (VRAM OR RAM) are employed in the implementation of the networking/RPC capability itself. As always, the impact of timing should be evaluated and system behavior may need to be confirmed with the real API Remoting framework and real HMI in place before correct behavior is fully confirmed. 

Conclusions

API Remoting is an often bandwidth-efficient way a way to have graphical interaction that gives the control of the graphics to one system while offloading the graphics rendering effort to the receiving side. Bandwidth efficiency can be better or worse than alternatives depending on the situation.  A potentially large shared API may increase the need for synchronized software updates for sending and receiving systems.

RAMSES is a framework that simplifies the programming of 3D graphics while at the same time enabling graphics to be distributed between system. The framework transparently takes care of the required distribution of data, at the right time, according to the graphical scene needs. Simplifying 3D programming serves a purpose even on one system but it also acts as a great example of a custom remote API for graphics distribution.

^^ Remove from conclusions chapter?



{eugen} first two paragraphs are is already a good description of the technology, rest could be skipped




{eugen} remove paragraph, content provided here is correct but we can skip it in order to reduce size without loosing important information

      {gunnar} OK.  Shortened – see diff.

{eugen} merge paragraphs, also I find the style quite long and unnecessary complex, suggestion:

{gunnar} Since the Tech Briefs are there to "whet your appetite", I think the reference to the sister project was useful.  But agreed, only if it fits.


In the simplest case we are dealing with invoking graphics commands by system A on system B. The actual drawing/rendering is executed on the remote system B. Thus it's different from Surface Sharing wherein the drawing/rendering is first completed on system A, and then a finished image would be transferred. Also a 3D-scene can be quite complex and consist of different objects, textures, animations and relationships between them. So efficient remote rendering API should reflect this.

{gunnar} ← your text transferred to here, with minor edits




{eugen} the point are again all correct and good, but I would see them more in the white paper which will describe and compare all sharing mechanism to each other, here I would also add one additional design consideration of the API remoting: level where the API remoting is applied: below or above the OpenGL interface of above. This is one the most important decisions to take.

Implementing the API remoting below the OpenGL will keep the compatibility to the Applications but such implementation could have some performance problem since the application is not optimized for usage of the rendering API over the network.

Implementing the API remoting above the OpenGL stack will require Application change since in this case we will not use OpenGL API from application directly, the new remoting interface has to be used. Even it looks like a big hurdle to start using this interface it might be very useful because in this case API remoting can be implemented in more efficient way and also provide new features which are required to implement new use-cases in a distributed HMI environment. 




{gunnar} I tried a shorter summary here,  and referencing the whitepaper.





{eugen}  I would remove the section completely



{gunnar} Rewritten and a big chunk stricken.



{gunnar}  I kept RAMSES section unchanged since it's been reviewed a few times already.  But it can probably shortened a bit more with some further work.





{violin} shortened the RAMSES section a bit



{gunnar}  Have not touched this (yet).  I'd like to comment on any impact of virtualization for each sharing method, but it's not the most important part.






{gunnar} Since we apparently did not agree on bandwidth efficiency, I'm compromising here and we will pick it up in the whitepaper details. The rest of the text somewhat simplified.








-----

OLD TEXT IGNORE FOR NOW


Introduction 1

<Intro, the need for unified HMI in vehicles, etc.>

Five different primary techniques for distributing graphics and HMI have been identified in the GENIVI Graphics Sharing and Distributed HMI project. This paper describes methods that  technique that was named API Remoting.

API Remoting method


API Remoting is the technique of taking an existing graphics API and making it accessible remotely, thus enabling one system to essentially call the graphics/drawing API of another system.

In some cases an existing programming API such as that of a local graphics library, is simply modified or extended to provide network-capability it did not have before.  In other words a local programming API is translated into an RPC (Remote Procedure Call) interface. In other cases, new programming interfaces might be created that are specially tailored for the remote graphics use case.

Many programming interfaces are already networked interfaces, and there is also a huge amount of technologies for making local programming APIs network-capable and independent of their location (local/remote). The sister project "Generic Protocol Evaluation" deals with analyzing some of those methods.

What we consider distinctive to Graphics Sharing through API Remoting, is that the network API in question is graphics/drawing related. We are dealing with essentially specific "drawing commands" invoked by system A on system B.  The actual creation of images is executed on the remote system B.  Thus, it's different from surface sharing wherein the drawing/rendering of an image would be completed on system A, and then transfer the image itself.

The API in question could be as low-level, as in "Draw Line/Box/Circle, or Fill Area at these coordinates". 

Apart from a very direct commandd ("Draw line"), typically associated with 2D-graphics, the remote API can also be more capable and "higher level" – in particular the defintion of 3D scene using objects, textures, animations and relationships.  This will require a more advanced processing on the receiving side that renders the graphics and it therefore involves more advanced library algorithms and of course use of a GPU on the remote side .

Various aspects should be considered when choosing graphics sharing approach:

  • Network bandwidth
  • Latency affecting changes and animations
  • Processing requirements

For some cases, such as a drawing a simple Navigation map, sending such drawing commands as opposed to images could be more bandwidth efficient than sending images, which is why API remoting deserves to be highlighted as a distinct and useful approach.  In addition to network bandwidth, there is also processing effort.  If the effort to transform the instructions into graphics is significant, then the API Remoting also guarantees that this effort (in particular GPU usage) is spent on the receiving side.

<Picture>  
System A  <---> System B


Handheld device displays

Smart phones and similar devices are an important interaction method to modern vehicles and should be considered part of the extended HMI of the car. Whereas many smart phones solutions might use simply the Shared State - Independent Rendering method, it's also likely that some API remoting could be used for the car to define and display an HMI on a mobile device. This requires a framework or standard in which the car and the mobile phone agree on the distributed graphics interface.  For API Remoting, the implementation of that agreed standard on the device side may very well be done by "apps" such that it need not be directly implemented in the operating system of the device.

RAMSES

RAMSES is a framework for defining, storing and distributing 3D (can be also 2D) scenes for HMIs. From users perspective, RAMSES looks like a thin C++ abstraction of OpenGL commands. RAMSES offers the same feature level as OpenGL but allows higher levels constructs such as widgets to be built on top of it. In addition to basic OpenGL, RAMSES supports higher level relationships between objects such as a hierarchical scene graph, content grouping, off-screen rendering, animations, text, and compositing. All of those features follow the principle of API remoting, i.e. the commands (that define the state in the RAMSES case) can be sent to another system and the actual rendering is executed at the receiving side.

RAMSES distinguishes between content creation and content distribution; the same content (created with RAMSES) can be rendered locally, on one or more network nodes, or everywhere. RAMSES handles the distribution of the scenes, and offers an interface to control the final composition - which scene to be sent where, shown at which time, and how multiple scenes and/or videos will interact together. The control itself is subject to application logic (HMI framework, compositor, smartphone app etc.).

The data pipeline and timing are "immediate", i.e. the state of each scene is simultaneously pushed to all network participants. RAMSES provides a set of mechanisms to control framerate, timings and avoid jitter and frame drops as much as possible.

Usage in virtualized environments

Since API Remoting deals with simply commands sent over a network it should apply to virtualized environments with minimum adaption necessary.  Whatever (virtual) network capability, the system provides will likely be transparent to the RPC system that is selected, and possibly some efficient techniques such as shared memory is employed in the implementation of the networking/RPC capability itself.  As always, the impact of timing should be evaluated and system behavior may need to be confirmed with the real API Remoting framework and real HMI in place before correct behavior is fully confirmed.

    

Conclusions

...

  • No labels