Virtual Display provides a display to the system which is not necessarily linked to the real display hardware. Applications and middleware, e.g. system compositor, should be able to use this display as usual. The concrete implementation has to transfer the content of this virtual display and present this on the real display hardware. The Virtual Display concept can be considered a subcategory of Surface Sharing.
- Simple to use because it abstracts the details of display implementation for the middleware
- Doesn't provide a fine granular control over the display content but it is still good enough for a lot of use-cases
- Entire display content is handled instead of applications content
Virtual Display describes a concept which can be used to realize distributed HMI use-cases. Important characteristic of the Virtual Display concept is that the entire display content is transferred instead of transferring content from dedicated applications. The system which provides the content should be presented with a display which act likes a physical display, but is not necessarily linked to a physical display, so the middleware and applications can use it as usual. Such a display can be called a Virtual Display. The implementation of Virtual Display on the producer system should be generic enough to look like the normal display and should take care of the transferring the display content to another HMI unit or another system. This basically means a final graphical surface needs to be transferred, therefore it can be considered as a subcategory of Surface Sharing. On the receiver site the content can be handled with more flexibility. It could be directly used as content for a physical display, it could be mapped to one physical layer of the composition hardware or used as part of a composition where it is combined with other available content local to the receiver. This flexibility makes the definition and the separation between different technologies a little blurred. An important characteristic of the Virtual Display concept is that we are handling the entire display content on the producer instead of handling content from dedicated applications.
Example: virtual display in Weston
Open source wayland compositor-Weston provides an example of Virtual Display implementation . Weston can be configured to create a Virtual Display which is not linked to any physical display. Properties like resolution, timing, display name, IP-address and port can be defined in weston.ini file. From this configuration Weston will create a virtual display "area" and all content which will appear in this area will be encoded to M-JPEG video compression format and transmitted to the configured IP address. The current Weston implementation uses corresponding GStreamer plugins for this. The assignment to virtual display will happen in differently depending on the Weston shell that is used. For example, if ivi-shell is used the new virtual display can be identified by the configured display name and the corresponding wayland protocol can be used to put application content on this display. In case of xdg-shell the user can just drag the application with the mice to the corresponding area.
The Weston example doesn't provide any code for the receiver system. In the first reference implementation examples the gstreamer pipeline provided is used to decode and present the content provided by Weston and therefore should run on major Linux distributions. For a production environment it should be fairly easy to write an application to receive the content by using available API's and frameworks on the target system.
Example: virtual display in Android
Android OS has support for creation of multiple virtual displays. The number of virtual displays is limited only by hardware processing capability. Typically Virtual display is used in Android to render content on a remote display. Android provides APIs to create Virtual Displays of any given width/height and dpi. Android associates a surface with every virtual display created. Android provides interfaces for automatic capturing of the content rendered on that surface. The content rendered can be either encoded as a video of any format supported by the platform or captured as a raw RGB buffer.
Any app can use a virtual display in two ways. If the app wants to show different content on a remote display (like two views of a map), then it can render the second view in a virtual display and send its content to remote display. It can also capture its own screen by creating a virtual display and make it mirror the main display. In this case the app would be able to only capture the screen only when the app is active.
To capture contents of any app, the app needs to run as a system app, which needs special permissions.
From Android O onward, apps can be launched on any display (Virtual or physical) dynamically. Android also provides APIs to inject user inputs such as Touch/ keys etc to a specific Virtual Display so that the app running on that display can be controlled. Even 3rd party apps can be launched on virtual displays.
The most prominent use of Virtual Display for content sharing is the implementation of Android Auto Projection. The whole Android Auto projection content is rendered onto a Virtual Display. It is then encoded as a video and sent over USB / WiFi to the head unit.
One of the other interesting use case of Virtual display is ability run apps on remote displays (Like Head Unit running apps on RSEs).
When more than one display is active (Physical or Virtual), Android imposes certain limitations:
System UI: Android by default displays the System UI (which displays battery status, WiFi / connection status, notifications) only on the primary display. System UI cannot be seen on other displays. Android framework needs to be customized to work around this.
Application Focus; Android allows only one app can be in resumed state, and only one view of an app across all displays may have focus. What this means is that any user inputs go only to the app / view at any point of time. Normally this may not be a problem, as the focus can switch very fast from one app to another. However if users try to feed concurrent inputs on more than one display, this may cause an issue.
Latency: Rendering apps on remote displays increases additional latency to the user experience. For normal apps this may not be a problem, but for heavy duty gaming apps, this could pose a problem.
System Resources: Running multiple CPU / GPU intensive apps could put heavy load on system. Also the number of virtual displays that can be supported gets also limited by the video encoding capability (How many simultaneous video encoding sesssions the platform can support)
Apps with Protected Content Rendering: Media apps that require highest level of protected content does not allow rendering of content on virtual display. For e.g. Widewine DRM has 3 levels of content protection (L1, L2, L3). L1 requires that the content is decrypted / decoded and rendered only in secure world. This means the rendered content is never available on the Rich OS world. So rendering the decoded video on virtual display becomes an issue. Some of the media streaming apps (such as Netflix) support 4K / Full HD resolution video playback only with L1 level DRM protection.
Multizone Audio: Android currently does not have support for multi-zone audio. Android’s Audio routing policy distinguishes only between different types of Audio (Media, Navigation, Alarms etc). Audio Routing policy allows routing of different types of audio to different audio sinks. AudioFlinger does not know which app is routing Audio.
Multiple instances of Same app. When more than one display is available to user, the expectation is that different users launch the same app and consume different content (Like watch different youtube videos on two displays). Android does allow multi-user mode, where one instance of app can be launched per user and with different logins. However the main restriction is that Android allows only one user to be active at a time. Apps of other users are kept in pause mode. This can be overcome by customizing Android framework.
- surface sharing tech brief
Document version: (still missing the example from Virtual Display in Android)