Page tree
Skip to end of metadata
Go to start of metadata

(star) What is this?  → Please refer to the projects overview page for a quick introduction, then for more details, the Kickoff slides and kick-off recording (see below).

Graphics Sharing and Distributed HMI - Next Call

21 June10.30 AM CET

Zoom Meeting Join Link, (register first - you can do that before to prepare) (star) Password: gsha

  • Zoom meeting number: 183-285-646

  • Agenda
    • surface sharing tech brief maturity check (Eugen, Stephen)
    • Waltham evaluation page review feedback
    • display sharing tech brief status (Eugen)
    • API remoting tech brief status (Gunnar)
    • AOB


Topics to be scheduled:

    • The graphics sharing part of SmartDeviceLink protocol (need volunteer to study & present)
    • Android IVI use case
    • Graphics sharing ideas for Qt. Examples: 
      • Wayland/Waltham + QtWayland
      • RAMSES + Qt combination
      • Possibly other Qt-unique solution 


    • Example/evidence of graphics distrubution involving other graphics / HMI tools
    • Example/evidence of QNX-based graphics sharing
    • More example/evidence of Android-based graphics sharing
    • More use cases for graphics sharing (Johan & Philippe C)

Upcoming Events

  • October Tech Summit, India (more information coming)

Previous Events

Definitions and Results

Overall (project scope)

  • Graphics Sharing
    = 1. Graphics in the form of bitmap, scene graph or drawing commands generated on one ECU, transferred for display from another ECU (or between virtual machine instances)
        2. GPU sharing in a virtualized setup
  • Distributed HMI Compositing
    = Methods and technologies to turn a multi-ECU system into what appears and acts as a single user-experience. 

The 5 Categories of Graphics Sharing technologies

(as developed by the group so far)

  • GPU sharing

    Concurrent access to the physical GPU has to be controlled by the hypervisor, hardware or other means which are implementation specific. 
    The same GPU can be used from multiple operating systems, so it is shared.

  • Display sharing
    1) The physical display can be shared across multiple operating systems. HW compositor unit composites final display buffer from HW Layers of each OS.
    This requires virtualization of the display controller hardware.  
    • Studied technologies:
      • Since this is mostly hardware specific we have not (yet) looked at it much.
      • Layer-Management is a related software abstraction, but intended a for single-system.
    2) "Virtual Display" – Full display transfer by encoding as a video stream.
    See also Detailed Description of Display Sharing
    • Technologies:
      •  Renesas Weston compositor gst-recorder plugin...
        • Renesas Weston compositor gst-recorder plugin
          • gst-recorder GStreamer plugin in Weston Compositor encodes the virtual display using H.264 and transmits. Receiver uses GStreamer to receive and decode the stream, which can then be composited as required.
          • Pros:
            • No modification required to Weston on receiver side.
            • If layer control is required the decoded stream can be transformed as required on receiver side in gfx framework of your choosing.
            • Video encoding minimises network data and bandwidth
            • Virtual display means display size can be made only as large as needed.
            • Technology has multiple applications, e.g. has been used in Waltham implementation for gfx data transfer.
          • Cons:
            • Delay of about 2-3 vsync to encode, transmit, receive, decode and composite the stream.
            • Large display containing a lot of animated objects will impact network bandwidth
          • R-Car M3 PoC in AGL demonstrator. Open source: links TBA
            • Two M3 ECUs. Navigation and mapview app encodes mapview image using gst-recorder and transfers via ethernet (udp). Receiver mapview image, decodes the data and displays moving map image between dials of cluster display.
  • Surface sharing
    Operating systems exchange graphical (bitmap) content. Then, each OS has full flexibility to use this content. In some cases, the compositor API is made available remotely, e.g. Wayland->Waltham.
  • API Remoting
    Transfer API calls, corresponding to "drawing commands", or other abstract graphics representation from one ECU to another. Commands or scene representation to be executed on the GPU of the receiving ECU.
    • Studied technologies:
      • Ramses
      • Note:  SmartDeviceLink is primarily remote APIs (car to device), but the graphics exchange is more like Surface Sharing.
  • Shared state, independent rendering
    • Each system has independent graphics systems and bitmap information. The systems only synchronize their internal state and exchange abstract data. Based on this shared data, each system independently render graphics to make it appear like they are showing the same or related graphics.

      • Example:  An appearance of synchronized map rendering could be achieved by drawing maps independently, and exchanging only the GPS position, scale of map, etc.
      • Comment: This differentiates only slightly from API Remoting in that API Remoting is active control from a primary to secondary system, i.e. "Draw this", whereas Shared state is more akin to "here is the data you need - but you control how/what to draw" (with some expectation of appearance defined ahead of time).

Project Goals

  • All project participants gain thorough understanding of available choices
  • Produce technology demonstrators, newly created or (if exists already) found and highlighted.
  • Publish hard data on learning:  Performance, resource needs.
  • Seek industry acceptance & alignment among Linux distributions
  • Separately identify and describe Hypervisor-based opportunities, how they differ, characteristics, advantages and disadvantages.
  • Seek alignment on solutions (protocols) among “closed” alternative proponents – i.e. other operating systems, commercial HMI-tools, etc.
  • Promote open standards and implementations across industry
  • Summarize and create (implementation) documentation for recommended choices


Define Graphics Sharing Categories

Publish Tech Brief - Shared State
(tick) April 2018 – DONE

"Complete" Table-based overview
24 May

Publish Tech Brief - Surface Sharing

(star) Planned publish date:  ~29 June

Version one is out here: GENIVI_ADIT_SurfaceSharing_TechBrief_201805_genivi_01.docx

Second version is here: GENIVI_ADIT_SurfaceSharing_TechBrief_201805_genivi_v2.docx

After review from Guru, version three please this one only for you review: GENIVI_ADIT_SurfaceSharing_TechBrief_201805_genivi_v3.docx

Publish Tech Brief - GPU Sharing?

  • We will first put it onto the agenda of the HV project to seek experts/volunteers among that group.

Publish Tech Brief - Display Sharing

Publish Tech Brief - API Remoting (or RAMSES?)

  • Gunnar to start a draft / intro paragraph.
  • Intention to synchronize with Ramses devs (Bernhard) later on.

Publish Project Report and/or Whitepaper on all technologies

RAMSES Open Source Release
Autumn 2018...

Use cases

Here we list some real-world functions (ideally from user perspective), to anchor the technical discussions to.

  • Navigation - map display from IVI to cluster
  • Entertainment (album art)
  • Remote "application" display
  • Telephone book (avatars/photos)
  • Video content (e.g. in cluster, when not driving)
  • Graphical information, triggered by context (people, location, ...)
  • Mirroring / Projection mode style CE-device integration and SmartDeviceLink style
  • Remote HMI, (car to CE-device)
    • "remote control" for media playback
    • controlling autonomous drive function
  • In-car camera to CE-device (e.g. security from remote location)
  • "Home screen" in a large combined HMI, the content could come from different sources.
  • RSE - Media served from one ECU to simple devices in the back seat.
  • Synchronized animation - e.g. content moving from one display to another (and consequently between ECUs)

Implementations and technologies

  • Waltham (git repo, docs)
    • Please consider reading (and correcting) Waltham evaluation.
    • first waltham plugins based on Renesas and Gstreamer for buffer transfare should be available  by 30 of March
  • QtWayland / Qt-specific ideas?
  • AvB / TSN ? – providing bandwidth & timing guarantee (only relevant in combination with other tech)
  • SmartDeviceLink (graphics transfer feature)
  • Connection to proprietary solutions - HMI-Tooling like Rightware, Altera, GL Studio, Disti, ... (TODO)

Technology overview & current options organizes and categorizes the options for different scenarios.

Asking for: 

  • Any more/other CE-device technologies (that are popular, and future-proof)?
  • ...more participation from commercial HMI tool vendors.

Evaluation Methods, Polling & Evaluation questions

  • Reading documentation & presenting to the group
  • Metrics from demonstration implementations (e.g. performance, resource usage)
  • Contacting standards organizations behind each choice for presentation/discussion?

  • Asking Questions – polling

Polling questions?

  • Does your company use <topic> for in-car embedded systems?
         Yes / No / Don’t know / (prefer not to answer?)
  • Graphics sharing
    • Between an IVI and Cluster ECU?
    • Between virtualized systems running IVI and Cluster operating system instances?
    • Between car and CE-device?
    • Between car and IT (e.g. screen dumps for debugging?)
  • Uses today vs has used?

(all direct children to this page)

  • No labels

1 Comment

  1. You can write and answer Q&A here, but we will later probably remove the comments and move the outcome to the actual Wiki page. (It makes things cleaner and clearer).