The ideal XR device and operating system

With the Vision Pro having made its way into the hands of people outside of Apple’s secretive labs, I want to reflect a bit on my conceptions of what an ideal XR device would look like, and to what extent the Apple Vision Pro comes close to this ideal.

Defining success #

I have identified three core use cases an XR device could fulfill:

1. Provide entertainment
2. Connect people
3. Increase productivity

In order for an XR device to be able to become successful, it needs to at least deliver on one of these use cases better than a readily available alternative offering. Note that these alternative offerings can be in different product categories. A pencil and paper still compete with an iPad, regardless of technological innovativeness.

Note also that success in this case refers to the amount of time spent using the device, rather than the amount of sales. If large amounts of people buy the device, but it ends up catching dust on a shelve, I would not consider it a successful product.

Analysis of the Apple Vision Pro #

In providing entertainment #

The predominant method for undergoing entertainment on the Vision Pro is remarkably different from other XR devices, as it does not allow fully immersive experiences beyond a small perimeter, and focuses heavily on having large floating 2D screens in the air.

In terms of user experience, it’s more like having a infinitely large and easily transportable personal TV, rather than an immersive gaming device. In my experience, having such a large screen feels great, but the amount of effort required to put on the headset and connect bluetooth peripherals does not weigh against the marginal benefit of a larger screen.

In addition, having tried a plethora of spatial (AR) experiences on the Meta Quest Pro, Magic Leap and HoloLens 1 and 2, these types of games quickly become a novelty as it is much easier to sit on the couch and play Subway Surfers, than it is to be physically active.

In connecting people #

This is where it falls short. The FaceTime virtual avatar implementation, while still in Beta, is silly and will not be an improvement over calling someone with regular Facetime where someone can see your real face. Seeing a “real” video stream is more important than having it be volumetric 3D. I think this technology is a dead end and efforts would best be spent elsewhere.

In increasing productivity #

Apple Vision Pro is built on iPadOS. This severely limits it in its ability to support real productivity.

In addition, connecting to a MacBook currently only supports streaming one 4K screen, which restricts its ability to be used as a portable multi-monitor setup.

Analysis of the Meta Quest 3 #

There’s not much to be said about the Meta Quest 3 as it is far behind the Apple Vision Pro in terms of seamlessness in user experience and polish.

In providing entertainment #

But, it does do one thing well: entertainment. As the Meta Quest 3 is essentially a glorified game launcher, it works well for starting up fully immersive games and experiences that transport you into a totally different interactive environment.

In connecting people #

Horizon Workrooms is not great; I once spent 45 minutes trying to join a meeting and ultimately failing.

Low-resolution with a low amount of expressiveness and emotion inside a low quality game environment is not a great experience and gets tiring after using it for a while. For some people it might work, in the same way Second Life and other social games work, but it lacks mass appeal.

In increasing productivity #

There’s virtual desktop streaming software, but this requires a good Wi-Fi connection and extensive setup on the sending device (desktop).

Every single streaming solution I have used so far has not provided an experience that was good enough to warrant carrying an extra device, setting it up for minutes, only to then struggle with lag, hard to move windows and illegible text.

Death by a thousand paper cuts #

One conclusion that can be drawn from the Meta Quest platform is that it has a lot of great ideas and gets certain parts right, but lacks in execution and suffers from the idea “death by a thousand paper cuts”.

There is a small amount of friction in each of the interactions, which results in an unpleasant user experience overall.

With VR already only providing marginal benefits over traditional computers, these paper-cuts amount to a device that does not get used beyond gaming and highly specific B2B VR training solutions.

XR device use cases #

There is no killer app. It’s simply a different paradigm for how to design computers that solve problems for people.

If it solves certain problems better than the alternative offerings, it will be successful. Otherwise it will ultimately remain a beautiful shelve decoration.

The ideal XR device #

Operating system #

• The ideal XR device in my view is a real computer. Not one that requires streaming from an external computer to perform any real productive tasks.
• The operating system of the XR device should be similar to a desktop operating system and contain the same paradigms for file handling and running processes.
• Ideally, the operating system used is Windows or macOS, as these have the widest support for professional applications, but the operating system would have to be heavily adapted in how it gets rendered and how the user interacts with it. Starting with a clean slate (i.e. a lightweight Linux distribution) and utilizing a portability layer for Windows and macOS, might thus be another viable option.

Window manager #

• The window manager of this operating system should support interacting with the windows using a mouse or trackpad, and support a multi-monitor setup.
• As the user will want a large amount of windows open at the same time (as I’m writing this I’m looking at a monitor with at least seven windows open on the same screen), so these windows should be grouped into virtual screens. These virtual screens would act similarly to how screens work on macOS. A user would get overwhelmed with having to manage 7 floating windows. Grouping them in these virtual screens would help.

Input methods #

• The XR device should support multiple input methods that can seamlessly be connected to the device. As the main way people are productive with computers is through typing (keyboard shortcuts included), this will stay the dominant way of highly productive input into this device. Voice control is a stupid idea and should be avoided at all costs. That simply does not replace Ctrl+Z, Ctrl+Shift+S etc.
• Floating keyboards are a stupid idea. They should be positioned at least on top of a real-world surface, so that the real world becomes the touch-screen. And ideally, just use a keyboard.

Gaming and interactive experiences #

• The most amazing part about XR is full creative freedom over what a player or user of the software can see. The entire world as a blank canvas. Support for fully immersive VR experiences is at the core of the unique selling point of a spatial computing device. Not for productivity, as that is still best done with keyboards, mice and other direct input methods, but for gaming and interactive experiences such as VR trainings and architectural visualization, it is a must.

Social features #

• Videoconferencing will remain a more personal way of connecting with other people via the internet. So, for social features we should focus on the aspect of utility and how useful it can be for solving certain problems.
• One of the core use cases might be collaborating on a whiteboard on a tough programming problem, or walking through a 3D BIM visualization together with a client to catch errors in the design.
• An ideal XR device should not shy away from these incredible collaboration features, but acknowledge it will not provide better social communication features than traditional video calling or social media platforms.

Conclusion #

The ideal XR device would combine a full, open operating system, with the ability to run fully immersive experiences, and have spatial collaborative apps for brainstorming, drawing on whiteboards and inspecting or viewing 3D models.

The Meta Quest 3 works for immersive experiences, but not for productivity. Its spatial apps (ShapesXR, Tilt Brush, Resolve BIM, Gravity Sketch) suffer from this gaming focus as well. Apple Vision Pro doesn’t work well for gaming, nor for productivity, so it’s in a weird spot, looking at it purely from a product perspective – whether people are going to use this on a daily basis.

XR devices and their operating systems have a long way to go, and I hope to see it go a bit more in the direction of the vision I sketched in this short heavily opiniated analysis.