Virtual Display provides a display to the system which is not necessarily linked to the real display hardware. Applications and middleware, e.g. system compositor, should be able to use this display as usual. The concrete implementation has to transfer the content of this virtual display and present this on the real display hardware. The Virtual Display concept can be considered a subcategory of Surface Sharing.
- Simple to use because it abstracts the details of display implementation for the middleware
- Doesn't provide a fine granular control over the display content but it is still good enough for a lot of use-cases
- Entire display content is handled instead of applications content
Virtual Display describes a concept which can be used to realize distributed HMI use-cases. Important characteristic of the Virtual Display concept is that the entire display content is transferred instead of transferring content from dedicated applications. The system which provides the content should be presented with a display which act likes a physical display, but is not necessarily linked to a physical display, so the middleware and applications can use it as usual. Such a display can be called a Virtual Display. The implementation of Virtual Display on the producer system should be generic enough to look like the normal display and should take care of the transferring the display content to another HMI unit or another system. This basically means a final graphical surface needs to be transferred, therefore it can be considered as a subcategory of Surface Sharing. On the receiver site the content can be handled with more flexibility. It could be directly used as content for a physical display, it could be mapped to one physical layer of the composition hardware or used as part of a composition where it is combined with other available content local to the receiver. This flexibility makes the definition and the separation between different technologies a little blurred. An important characteristic of the Virtual Display concept is that we are handling the entire display content on the producer instead of handling content from dedicated applications.
Example: virtual display in Weston
Open source wayland compositor-Weston provides an example of Virtual Display implementation . Weston can be configured to create a Virtual Display which is not linked to any physical display. Properties like resolution, timing, display name, IP-address and port can be defined in weston.ini file. From this configuration Weston will create a virtual display "area" and all content which will appear in this area will be encoded to M-JPEG video compression format and transmitted to the configured IP address. The current Weston implementation uses corresponding GStreamer plugins for this. The assignment to virtual display will happen in differently depending on the Weston shell that is used. For example, if ivi-shell is used the new virtual display can be identified by the configured display name and the corresponding wayland protocol can be used to put application content on this display. In case of xdg-shell the user can just drag the application with the mice to the corresponding area.
The Weston example doesn't provide any code for the receiver system. In the first reference implementation examples the gstreamer pipeline provided is used to decode and present the content provided by Weston and therefore should run on major Linux distributions. For a production environment it should be fairly easy to write an application to receive the content by using available API's and frameworks on the target system.
Example: virtual display in Android
Android OS has support for creation of multiple virtual displays. The number of virtual displays is limited only by hardware processing capability. Typically Virtual display is used in Android to render content on a remote display. Android provides APIs to create Virtual Displays of any given width/height and dpi. Android associates a surface with every virtual display created. Android provides interfaces for automatic capturing of the content rendered on that surface. The content rendered can be either encoded as a video of any format supported by the platform or captured as a raw RGB buffer.
Any app can use a virtual display in two ways. If the app wants to show different content on a remote display (like two views of a map), then it can render the second view in a virtual display and send its content to remote display. It can also capture its own screen by creating a virtual display and make it mirror the main display. In this case the app would be able to only capture the screen only when the app is active.
To capture contents of any app, the app needs to run as a system app, which needs special permissions.
From Android O onward, apps can be launched on any display (Virtual or physical) dynamically. Android also provides APIs to inject user inputs such as Touch/ keys etc to a specific Virtual Display so that the app running on that display can be controlled. Even 3rd party apps can be launched on virtual displays.
The most prominent use of Virtual Display for content sharing is the implementation of Android Auto Projection. The whole Android Auto projection content is rendered onto a Virtual Display. It is then encoded as a video and sent over USB / WiFi to the head unit.
One of the other interesting use case of Virtual display is ability run apps on remote displays (Like Head Unit running apps on RSEs).
When more than one display is active (Physical or Virtual), Android imposes certain limitations:
- Number of displays possible is limited by the CPU / GPU resource availability.
- Android was built for a single user and may experience issues with concurrent inputs from multiple displays.
- No support for the same app running concurrently on different displays
- No separate Audio channel per display
Some of the above limitations can be overcome by customizing Android framework.
AllGo Case Study:
AllGo had done a demo to see how applications from an Android based HU can be run on remote displays. The key features that was targeted for the demo was
- Allow users to launch and control app on any of the displays
- Allow multiple users to access and control different apps simultaneously.
- Switch apps from one display to another seamlessly depending upon where they are looking / sitting.
The demo setup is described in the picture above. It had one Android based HU and two RSEs running as thin Client displays running linux. RSEs are connected over ethernet.
The diagram below gives a high level architecture of how the rendering and control of applications on RSE work.
For every unique user, a dedicated Virtual Display created on the Head Unit. For every unique user, there is an associated custom launcher. Apps that are available on the launcher is also customized for the individual user.
The Multi-display server advertises about the number of unique users using ZeroConf protocol. Multi-display Client running on the RSEs can discover dynamically the users and choose a specific user. Once a specific user is selected, the associated custom launcher for the user is launched on its associated virtual display. The virtual display content is then encoded as a video and sent out to the specific RSE, which is decoded by the client and rendered. The client also captures the user input (like touch / keys) and sends it back to the Server. The server injects these inputs to the associated virtual display.
In our demo we showcased two unique user scenario, where two users were able to independently launch apps on the two RSEs
When the passenger wants to switch to another RSE (say he changed seats), he needs to simply select the same user from the other RSE. The server then automatically switches the streaming the associated virtual display content to the newly selected one.
The number of RSEs in the system can be any number. It does not need to match the number of unique users. The number of unique user can also be configured. A video of the demo ca be seen here.
- surface sharing tech brief
Document version: (still missing the example from Virtual Display in Android)