NOTE:  This is for 2018.   The schedule for the following AMM is here.

Schedule used at GENIVI 18th All Member Meeting ~ Munich

(green star) Complete minutes – further down the page!

(Topic name & Responsible & Timeslot)

Presentations can be found here – look for the Hypervisor Session

  • Workshop introduction and intention  9:00-9:05 5mn

    •  presented by: Gunnar (GENIVI), Sang-Bum (Perseus)

  • History of HVs 20mn presentation 9:05-9:25

    • presented by: Sang-Bum

  • Introduction to XEN project   – 9.20-->
    • by: Lars Kurth (Xen project)
  • Market Overview 20mn presentation 9:40-10:00  (->9.50-10.15)
    • presented by:Franz Walkembach (SysGo)
    • includes an intro to certification
  • HV design and implementation 10mn + 15mn presentation + discussion 10:20-10:45
    •  Introduced by: Ralph (Open Synergy)

    >>>> short break 15mn 10:45-11:00 CET

  • Requirements gathering 10mn intro + 10mn discussion 11:00-11:20 
    •  HV vendors asking OEMs/adopters/customers/etc to clarify technical requirements
    •  Introduced by: Matti (Open Synergy)

  • Virtualization for Multi-core, SoC peripheral hardware and special- purpose CPUs 10mn intro + 10mn discussion 11:20-11:40
    • Introduced by:Artem (EPAM)
  •  Audio system design with HVs 10mn intro + 10mn discussion 11:40-12:00
    •  Introduced by: Artem (EPAM)

  • Standardization of hypervisor APIs (virtio and friends) 10mn intro + 10mn discussion 12:00-12:20
    • Introduced by: Matti (Open Synergy) (virtio intro + what's needed?) + ?

  • *morning wrapup* 12:20-12:30
    • Agree on outcome, actions, topics for future project, etc.

>>>> lunch break 12:30-14:00

  • Graphics/GPU Sharing (in relation to GSHA project) 10mn intro + 15mn discussion 14:30-14:55
    • Introduced by: looking for volunteer to introduce the topic
    • Artem? Alex? (EPAM)

  • (Cyber-)Security enhancements based on virtualization 10mn intro + 20mn discussion 
    • Introduced by: Sang-Bum

    15:30 - 15:45   Break

  • Health/Debugging/Analysis/Logging (in relation to SHDA project) 5mn intro + 10mn discussion  
    •  Introduced by: Gunnar Andersson (GENIVI)

  • *afternoon wrapup* 16:15-16:30
    • Agree on outcome, actions, topics for future project, etc.


  • Terms / Nomenclature?
  • Hypervisor project organization (post-AMM)
  • How can reliability & safety be quantified?
  • System design to optimize Boot Time,
  • Boot requirements, e.g. secure boot, integrity check,
  • System design and working with HVs from users/system integrator's perspective.
  • Defining next steps + Hypervisor Project.
  • Performance comparison between open source software based hypervisors on ARM SoC 10mn intro + 10mn discussion 
    •   Introduced by: Sang-Bum

Minutes from Hypervisor Workshop at AMM Munich April 2018

(green star) The presented material (slides) can be found here – look for the Hypervisor Session

____________________ GENIVI Jeremiah C. Foster Philippe Robin ____________________ Table of Contents _________________ 2 Minutes from 18th GENIVI AMM April, 2018 .. 2.1 Hypervisor Workshop Introduction and Intention ..... 2.1.1 History of the Hypervisor ..... 2.1.2 Xen Project Automotive and embedded overview ..... 2.1.3 Sysgo AG -- Hypervisor Market Overview ..... 2.1.4 Hypervisor Design(s) ..... 2.1.5 Requirements gathering ..... 2.1.6 EPAM Virtualization for Multi-core, SoC peripheral hardware ..... 2.1.7 Security enhancement on Xen ARM ..... 2.1.8 Standardization of hypervisor APIs (virtio) ..... 2.1.9 Health/Debugging/Analysis/Logging (in relation to SHDA project) Almost full room 2 Minutes from 18th GENIVI AMM April, 2018 ========================================== 2.1 Hypervisor Workshop Introduction and Intention ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Gunnar Andersson, GENIVI - Dr. Sang-Bum Suh, CEO Perseus Co., Ltd One of the driving forces of the Hypervisor work in GENIVI is the Domain Interaction work being done in GENIVI. The Hypervisor Working Group has really kicked off after Seoul AMM in 2018. The hope is that after this workshop there is some clearly defined goals to work on. 2.1.1 History of the Hypervisor ------------------------------- Virtual Machine Monitor == Type 1 virtualization. Runs one or more virtual machines. Each virtual machine is called a guest. Type 1 is bare metal, type 2 is with operating OS inbetween. Main use case is reduction in space, weight, and power. HVs are a scheduling operating systems. Implementation of an HV is as difficult as the linux kernel although the concept seems simple In automotive we see RFQs on digital cluster and IVI on single hardware or SoC Reuse of legacy code or using existing ECUs for new projects Fastboot and secureboot, boot via RTOS for example. Another use case is combining Android / OS software in a secure environment. Slide: shared showing example of space separation in automotive. Slide: HV vs Linux containers IBM started hypervisors for migration of bank transactions to an uninterruptable services, done in the 1970s. Slide showing theory vs. practice. The theory is that HVs are simple, but in fact are quite complex. Design of a type 1 is HV is comparable to the linux kernel in complexity. Two scenarios nowadays; SoC BOM cost savings through consolidation and zero downtime. Without support of hardware you got a heavy hypervisor and low performance. Wit SoC redesign you get a thinner HV and better performance. First comes CPU/MMU Virtualization, the I/O, then PV. GPU virtualization, embedded in SoC design, ought to provide significant performance increases. History of Xen ARM hypervisor Xen on ARM was released in 2008. CPU overhead 3% after optimizations. [Video demonstration of HV on cell phone for the purpose of security improvement] 2.1.2 Xen Project Automotive and embedded overview -------------------------------------------------- - Lars Kurth, Xen Community Manager Ecosystem overview Xen capabilities and Challenges Safety certification Xen moved heavily in cloud computing, but then saw that Xen was being used in safety cases and started to be adopted for security in end computing devices Project: OpenXT = xen + open embedded,, mostly military solutions. Forked Xen and OE but the hope is that there will be a return of code back to Xen. Project: Virtuosity from Dornerworks, avionics certified DO-178, IEC 62304, ISO 26262. Went already through the ceritifcation process for a first iteration Added support for VxWorks, RTEMs, etc. Project: EPAM Fusion Project: GlobalLogic Nautilus Automotive vendors add to the community like Renesas, Bosch, LG, Adit, Samsung. - Automotive requirements vs. Xen Project Automotive requirements come from AGL white paper Eupam running a project on memory/IO bus bandwith allocation and rebalancing GPU sharing is being generalized to co-processors - Security requirements, Root of trust and secure boot needs to be supported (WIP). Hardware isolation, core functionality is there, except firewalls. - Safety requirements Fastboot is available Power management is available, more work being done EPAM, XILINX, Aggios, discussion on accessing memory - Safety certification The goal is to make it easier for Xen users to Safety Certify KSLOC about 70 thousand lines of code The future target is to be around 40K to 50K Q: How do we track progress of this work? A: Some things are happening on the mailing list, but hope is to setup a proper project infrastructure Q: What is the cost on boot time of using containers? A: While it doesn't do everything the cost is nearly zero. 2.1.3 Sysgo AG -- Hypervisor Market Overview -------------------------------------------- - Franz Walgenbach - Sysgo Market trends, main vendors, which HV types are visible with the focus on MMU vs. MPU Sysgo goal is to be the leading European OS provider for embedded. Cluster ADAS/Gateway IVI, Virtual ECU, ECU consolidation are the spaces 500 million dollar market by 2019, so relatively small 2.1.4 Hypervisor Design(s) -------------------------- - Dr. Ralphe Sasse OpenSynergy Slide: Xen design Dom0 DomU DomU Slide: KVM is part of the kernel via a kernel module. Slide: Possible VM allocation patterns - 1 core == 1 VM N cores `= 1 VM aka SMP distribution requires inter-core communication 1 VM =' 1 Supplier 1 VM `= 1 functionality 1 VM =' 1 criticality 1 VM == 1 timing (real time / best effort) Slide: KVM major difference with XEN is that there is no tool stack, otherwise design very similar to xen Sang bum: there is a large difference in booting approaches and time between kvm and xen WATCH OUT!! for an instrument cluster mmi, we are talking about a mmu for a controlsystem, we are talking about mpu 2.1.5 Requirements gathering ---------------------------- - Matti Möll Opensynergy OpenSynergy All hypervisors have formal requirements that govern the design and implementation. Android compatibility Definition Document (link in slides) Example requirements shown. Q: Is there any discussion of diagnostics and diagnostic responses? A: Discussed but perhaps not in detail. Diagnostics will depend on design, like depending on CAN design We can use the AGL white paper as a starting point, but we have a less specific goal than AGL. 2.1.6 Virtualization for Multi-core, SoC peripheral hardware ----------------------------------------------------------------- - Artem Mygaev - EPAM Intro on peripherals sharing - design options example: USBs - how do we handle dynamic devices ? Ralf: who is the owner of a usb device that pops up ? is it one VM ? is it all VMs ? Artem: how do you create a policy of what goes where ? How do you add dynamic devices, adding devices with different capabilities? Likely better to use para-virtualization. Still there is a performance hit so not so good for data intensive scenarious. Discussion on how to add a peripheral and into which domain. How do you create policies for what goes where and how it goes. Many co-processors (DSPs, GPUs, IPUs) do not have native virtualization support. There is a technology called "mediated pass-through" (HMP) where you have partial pass-through and partial emulation. Can be emulated in an HV. Emulation and mediation does the context switch. Q: Franz - GPU sharing - OpenGL virtualization can be done in the GPU or higher in the stack, do you know the performance cost? A: With mediated pass-through there is a about a 5% overhead A: artem; we think we could do better with Vulkan but we have not started on this yet - Heterogeneous multicore an example is big.LITTLE from ARM which has different cores and presents a question on what type of virtualization to use. It can be difficult to communicate in the VM which core a thread show run on. You can do a brute force attempt but that might create performance issues. Also, energy aware scheduling might be disrupted. Q: Is there are trend towards an asymmetrical approach? A: The trend is to make things more complicated On ARM side we have an enhancement called 'ARM Dynamic' which is fully scalable but still a cluster architecture. Multiple mixed cores and much more diverse. Q: Surely the system architect would be better positioned to know what the VM gets for a core A: But what about autonomous driving and other asymmetrical use cases? note: slack reclamation is a good approach but it is not scheduling discussion on multiple scheduling layers Audio system design with HVs -------------------------------------- - Artem Mygaev - EPAM There needs to be a control logic because there are policies and modes of operation around sound in the vehicle. (Analysis of pulse vs. Alsa) We should follow up on Xen changes to GENIVI Audio Manager to have a discussion on their patches and design. Artem: there are some scenarii available in GENIVI audio manager that would be worth being investigated in the context of domain interaction there are pros & cons in using GENIVI audio manager /TODO/ Jeremiah (as Community Manager) ask GENIVI audio manager people to answer Xen request on pros and cons Q: How to hand different DSP features? A: Various DSPs are a 'final stages' and go into a pass-through mode to the domain that controls the mixing. a little bit of latency is acceptable for general purpose audio, no latency is acceptable for echo cancellation Matti: linux alsa stack does not support very well the offloading Matti: recommends the reading of Android Auto CDD to know how to measure latency (not an obvious task) 2.1.7 Standardization of hypervisor APIs (virtio) ------------------------------------------- - Matti Möll Opensynergy UEFI Unified Extensible Firmware Interface Platform standardization BIOS PXE x86 only UEFI Combo of PXE, BIOS and boot loader, also has a runtime environment Challenges of virtualization in automotive, they come down to disc and network because the systems tend to be headless. There is a low amount of reusable devices. The question for standardization is where do you cut? COTS or domain specific? Strict requirements or rough consensus? vendor specific device models: eg Virtual Box drivers cloud virtualisation/ amazon uses xen modified and nvidia drivers because most cloud providers use nvidiA device virualization technologies virtual gpu - question on status: distinction between device (server) and guest (client) Lars: virtio will be brought by security people in the xen community Kevin / Virtual Open Systems ( commented on virtio performance and some teams going to PCI Q. are the device modes available somewhere ? A. no but a device model is something simple to develop automotive use case: DRM QA OPTEE will have kernel awareness soon opensynergy, greenhills are using wirtual io Q. who is using QNX with HV ? A. no one answers GHS has jumped into the discussion Epam & Opensynergy are ok to support the standard windriver and ghs are not willing to answer Nikola will try to forward the topic back to GHS management no commitment from WR though 2.1.8 Security enhancement on Xen ARM ------------------------------------- Features for secure smartphones - Service isolation - Secure boot - Secure storage - Access control How often was a DomU switched away due to interupt handling. - Profiling data, statistical profiling where that system was at a given time. Linux users can use perf. - Information from Dom0 on scheduling is also available Q: What about IO load? A: overview of denial of service discussion on how to access HSM Ralf Opensynergy how do we make an IC safe ? Ralf: we add a monitoring system to monitor the ic, the hv based architecture enables the deployment of counter measures 2.1.9 Health/Debugging/Analysis/Logging (in relation to SHDA project) -------------------------------------------------------------------- Gunnar: what do you want to measure on an hypervisor cpu load ? memory usage ? io ? Matti time spent in interrupt profiling data: statistics on data, linux is pretty great for that, hypervisor should not block the gathering of statistics Sang bum if we instrument the system for debugging, we modified the behavior it depends on the kind of systems you are debugging the work done for assessing security might not be relevant for performance profiling Artem in xen we have the profiling tools, i.e. special tools for RT to plan budget, deadline debug is trying to analyze what happens ar run time (disk for instance) how long a context switch takes in the gpu ? there is some jitter PVRtune IMG have PVR tooling, Renesas have tooling. How well does that tooling stand-up in a HV situation? PVR Tune is a graphical monitoring tool for Imagination GPUs System health monitoring Q: Does the HV understand the process modeling in the guest? A: The only feedback you can get from the kernel if it is in the HV is the ticks scheduler Windriver it is not only about gpu, the tools depend on the context PVRtune is rather for gpu level analysis There exist Xen tools and there is a rich toolset on the developer side for debugging. Q: Is it possible to get a demonstration of the tooling? A: It is kind of use case bound but we'll release some information shortly No one tool can cover all areas, no magic wand. PS5 is the hyper debugger from ARM guest introspection Matti: the interns of the kernel change every day and you can get very little info from the kernel hw debugging environment can do it: trace32 lauterbach, arm psi tools you can do real introspection with such a tool Ralf: what is at hand for an operating system that could help bringing up an HV system ? Matti: only lauterbach can help Homologation -- granting approval by certifying authority Type certification -- certification of a component


Topic Introduction

  • Informative and teaching everyone to a similar level of understanding
  • Technical neutral information, no sales pitch
  • Try to open questions for discussion
  • Define what is the remaining problem
    • Technology needs improvement?
    • Uncertainties?
    • Information/documentation/lacking understanding?
    • ....
  • Suggestion:
    • Final slide could be a list of questions (to the workshop participants)

AMM Workshop Agenda (Topic name & Responsible)

This is a workshop session spanning several hours. We expect significant participation from all attendees. Each topic should have a short introduction (maximum 10 minutes) followed by interactive discussion. Individual topics should be introduced by a variety of Hypervisor vendors and other experts, such as OpenSynergy and Perseus.

Discussion topics will include:

  • Workshop introduction and intention
    • Introduced by: Gunnar (GENIVI), Sang-Bum (Perseus)
  • History of HVs
    • Introduced by: Sang-Bum
  • Introduction to XEN project
    • by: Lars (Xen) (TBC by Sang-bum)
  • Market Overview
    • by:Franz Walkembach (SysGo)
      • includes an intro to certification
  • Requirements gathering
    • HV vendors asking OEMs/adopters/customers/etc to clarify technical requirements
    • Introduced by: Matti (Open Synergy)
  • Performance comparison between open source software based hypervisors on ARM SoC
    • Introduced by: Sang-Bum
  • HV design and implementation
  • Virtualization for Multi-core, SoC peripheral hardware and special-purpose CPUs
    • Introduced by:Artem (EPAM) + input on Samsung roadmap (Sang bum)
  • Standardization of hypervisor APIs (virtio and friends)
    • Introduced by: Open Synergy (virtio intro + what's needed?) + ?
  • How can reliability & safety be quantified?
    • Introduced by: (question)
    • - how to invite experts? TUV/similar?
  • Audio system design with HVs
    • Introduced by: Artem (EPAM)
    • Discussing GENIVI AudioManager extensions
  • Graphics/GPU Sharing (in relation to GSHA project)
    • possibly show-and-tell multiple vendors?
    • Introduced by: (question) - reaching out to Renesas ?
  • Health/Debugging/Analysis/Logging (in relation to SHDA project)
    • Introduced by: (question) 
  • (Cyber-)Security enhancements based on virtualization
    • Introduced by: Sang-Bum
  • System design and working with HVs from users/system integrator's perspective.
    • Introduced by: (question)
    • ? tier-1 or OEM leading discussion?
    • Hardware integration.  Communication issues between vendors?

AMM Workshop Agenda Topics (brainstorm and more details)

  • Requirements gathering
    • Requests from HV vendors for OEMs/adopters/customers/etc to clarify technical requirements,
      e.g. on particular driver, performance, reliability, safety.  
      • Central device driver management?
        • ^^ Lead/intro by Sang-Bum
        • i.e. HV+BSP supporting hardware register access arbitration.
        • ...?
        • Even possible without this?  What would be design patterns to use if the HV is not giving the hardware virtualization you need.
      • BSPs available vs. needed
      • Unikernel support
        • e.g. tested/demonstrated
        • any special matching needed between HV and unikernel (para-virtualization) – likely not...
    • (Maybe in particular, how is reliability & safety quantified?)
      Boils down to: Several safety related parts brought together - how to design (and evaluate) the total system. 
    • Footprint
      • Memory usage
      • # Lines of code (auditing/reliability/trust...)
  • Training / informative materials?
    • Vendor-independent
    • From users/system integrator's perspective.
    • What can be done today, what can't be done today
    • Consequences of para virt or hardware support for virtualization, and related BSP design
    • Connection between "management/sales" slides and the very detailed HV documentation.  Knowledge required in the gap here.
  • Cross-domain architectures, examples, recommendations
    • (repeated from above) Boils down to: Several safety related parts brought together - how to design (and evaluate) the total system.
    • (repeated from above) Consequences of para virt or hardware support for virtualization, and related BSP design
      • Is design even possible without virtualized hardware?  Design patterns to coordinate access if the HV is not giving the hardware virtualization you need?

    • 12 factors for service-oriented platforms.  
  • Required technology demonstrations
  • Market/Technology survey
  • Requirements survey - Questionnaire (to OEMs)
    • Results could be statistics/anonymized if needed.
    • Start work on this questions list.
  • Feature evaluation
    • Compare features
      • Separate BSPs - paravirt or hardware virt support?
    • ... or simply list "what types of features can a Hypervisor have"
    • Comparing types of HV / key characteristics?
  • Graphics/GPU Sharing (in relation to GSHA project)
    • What things are hardware dependent?
    • How to share graphics buffers?
  • Health/Debugging/Analysis/Logging (in relation to SHDA project
  • Audio...
    • Past approaches are known and can be discussed (Sang-Bum)
      • e.g. paravirt devices
      • audio mixing in operating system can handle it instead of paravirtualized drivers
      • Complete architecture idea, e.g. with GENIVI AudioManager
      • One hardware run Cluster
  • How to minimize impact on the development process when using HVs 
  • Glossary/terms - does everyone have the same understanding of terms?