Page tree
Skip to end of metadata
Go to start of metadata

(star) What is this?  → Please refer to the projects overview page for a quick introduction, then for more details, the Kickoff slides and kick-off recording (see below).


Graphics Sharing and Distributed HMI - Next Call

Thursday, October 25, 10.30 AM CET

Zoom Meeting Join Link, (register first - you can do that before to prepare) (star) Password: gsha

  • Zoom meeting number: 183-285-646

  • Agenda (preliminary)

    • GPU Sharing followup/wrapup (main discussion to be held together with HV Project)
    • API Remoting Tech Brief - feedback and finalization
    • Tech Brief & White paper work

Backlog

Topics to be (re)scheduled:

    • API Remoting of OpenCL (Artem/EPAM).  (This is not for graphics, but it's GPU and it's API Remoting.  Might be useful.)
    • Artem requests deeper look at performance benchmarks.   (Harman QNX/Android surface-sharing  as the example to study?)
    • Provide & discuss details on Weston-remote
    • Android IVI use cases & Android's SurfaceFlinger and Wayland API's Mapping
    • Deep-dive?  Web assembly as graphics transfer technology?  (QtWebAssembly)
    • The graphics sharing part of SmartDeviceLink protocol (need volunteer to study & present)
    • Coordinate "GPU sharing" topic with the Hypervisor Project

Requests/actions:

    • (red star) Example/evidence of graphics distrubution involving other graphics / HMI tools including various "commercial HMI tools"
    • (DONE, but feel free to share more) More example/evidence of Android-based graphics sharing - (especially distributed network, since Hypervisor based is common)
    • (DONE, but feel free to share more) Example/evidence of QNX-based graphics sharing
    • (star) Use-case / requirements workstream.  Write down use cases, scenarios and requirements for graphics sharing & distributed HMI.

Upcoming Events

  • CES 2019, Las Vegas, USA. 

Previous Events




Definitions and Results

Overall (project scope)

  • Graphics Sharing
    = 1. Graphics in the form of bitmap, scene graph or drawing commands generated on one ECU, transferred for display from another ECU (or between virtual machine instances)
        2. GPU sharing in a virtualized setup
  • Distributed HMI Compositing
    = Methods and technologies to turn a multi-ECU system into what appears and acts as a single user-experience. 


The 5 Categories of Graphics Sharing technologies

(as developed by the group so far)

  • GPU sharing
    The GPU can be used from multiple operating systems, so it is shared. Concurrent access to the physical GPU has to be controlled by the hypervisor, hardware or other means which are implementation specific.

  • Display sharing
    The physical display can be shared across multiple operating systems. HW compositor unit composites final display buffer from HW Layers of each OS. This requires virtualization of the display controller hardware.

  • Surface sharing
    Operating systems exchange graphical (bitmap) content. Then, each OS has full flexibility to use this content. In some cases, the compositor API is made available remotely, e.g. Wayland->Waltham.
  • API Remoting
    Transfer API calls, corresponding to "drawing commands", or other abstract graphics representation from one ECU to another. Commands or scene representation to be executed on the GPU of the receiving ECU.
  • Shared state, independent rendering
    • Each system has independent graphics systems and bitmap information. The systems only synchronize their internal state and exchange abstract data. Based on this shared data, each system independently render graphics to make it appear like they are showing the same or related graphics.
      • Example:  An appearance of synchronized map rendering could be achieved by drawing maps independently, and exchanging only the GPS position, scale of map, etc.
      • Comment: This differs slightly from API Remoting as follows:  API Remoting implies active control from a primary to secondary system, i.e. a command exchange such as "Draw this", whereas Shared state is more akin to "here is the data you need - but you control how/what to draw" (still with the expectation that the appearance will match the "common" HMI).
    • (green star) See Shared State tech brief

Project Goals

  • All project participants gain thorough understanding of available choices
  • Produce technology demonstrators, newly created or (if exists already) found and highlighted.
  • Publish hard data on learning:  Performance, resource needs.
  • Seek industry acceptance & alignment among Linux distributions, as well as across operating systems and domains
  • Seek alignment on solutions and protocols among proponents of “closed” alternatives  – commercial HMI-tools, etc.
  • Promote open standards and implementations across industry
  • Separately identify and describe Hypervisor-based opportunities, how they differ, characteristics, advantages and disadvantages.
  • Summarize and create (implementation) documentation for recommended choices

Milestones

Define Graphics Sharing Categories
(tick)

Publish Tech Brief - Shared State
(tick) April 2018 – DONE (see page)

"Complete" Table-based overview
→ Work in progress.  Decide on completion.

Publish Tech Brief - Surface Sharing

(tick) July 2018 – DONE (see page)

Publish Tech Brief - GPU Sharing?

  • We will first put it onto the agenda of the HV project to seek experts/volunteers among that group.

Publish Tech Brief - API Remoting (or RAMSES?)

  • API Remoting working page for detailed definition and tech brief.
  • (star) Time plan: October Tech Summit

Publish Tech Brief - Display Sharing

Publish Tech Brief - Surface Sharing: Virtual Display?

  • Do we wish to publish a tech note on the Surface Sharing sub-category "Virtual Display"?
  • Two technologies known to group:
    • Surface remoting via video streaming.
    • Extending Weston to support virtual remote displays

Publish Project Report and/or Whitepaper on all technologies

  • Time plan:  Release at end-of-year 2018
  • Compare technologies - make clear use-case and constraints that lead to decision.
    • Volunteers (writing / copy-editing)
      • Eugen, Harsha, Stephen, Gunnar.


RAMSES Open Source Release
Autumn 2018...

Use cases

Here we list some real-world functions (ideally from user perspective), to anchor the technical discussions to.

  • Navigation - map display from IVI to cluster
  • Entertainment (album art)
  • Remote "application" display
  • Telephone book (avatars/photos)
  • Video content (e.g. in cluster, when not driving)
  • Graphical information, triggered by context (people, location, ...)
  • Mirroring / Projection mode style CE-device integration and SmartDeviceLink style
  • Remote HMI, (car to CE-device)
    • "remote control" for media playback
    • controlling autonomous drive function
  • In-car camera to CE-device (e.g. security from remote location)
  • "Home screen" in a large combined HMI, the content could come from different sources.
  • RSE - Media served from one ECU to simple devices in the back seat.
  • Synchronized animation - e.g. content moving from one display to another (and consequently between ECUs)

Implementations and technologies

  • Waltham (git repo, docs)
    • Please consider reading (and correcting) Waltham evaluation.
    • first waltham plugins based on Renesas and Gstreamer for buffer transfare should be available  by 30 of March
  • RAMSES
  • QtWayland / Qt-specific ideas?
  • AvB / TSN ? – providing bandwidth & timing guarantee (only relevant in combination with other tech)
  • SmartDeviceLink (The graphics transfer feature, which is now (re)using a popular name: "Projection Mode")
  • CastScreen is an open-source (Android device) screen-cast implementation
  • Connection to proprietary solutions - HMI-Tooling like Rightware, Altera, GL Studio, Disti, ... (TODO)

Technology overview & current options organizes and categorizes the options for different scenarios.

Asking for: 

  • Any more/other CE-device technologies (that are popular, and future-proof)?
  • ...more participation from commercial HMI tool vendors.

Evaluation Methods, Polling & Evaluation questions

  • Reading documentation & presenting to the group
  • Metrics from demonstration implementations (e.g. performance, resource usage)
  • Contacting standards organizations behind each choice for presentation/discussion?

  • Asking Questions – polling

Polling questions?

  • Does your company use <topic> for in-car embedded systems?
         Yes / No / Don’t know / (prefer not to answer?)
  • Graphics sharing
    • Between an IVI and Cluster ECU?
    • Between virtualized systems running IVI and Cluster operating system instances?
    • Between car and CE-device?
    • Between car and IT (e.g. screen dumps for debugging?)
  • Uses today vs has used?

(all direct children to this page)


  • No labels

1 Comment

  1. You can write and answer Q&A here, but we will later probably remove the comments and move the outcome to the actual Wiki page. (It makes things cleaner and clearer).