A Reflection on Programming in Virtual Reality

By Jacob Strieb

Published on May 30, 2021

Background

Since I began college, I have struggled with staying focused while working in shared spaces.¹ As a result, I spend a lot of time testing techniques to limit distractions in my work environment. This is an especially engaging diversion when I have important work to do.

Last November, when my friend Logan went home for Thanksgiving, he left behind his Oculus Go—a Virtual Reality (VR) headset he bought on eBay—and gave me permission to use it in his absence. Since I had a challenging assignment for my programming language theory course due the next day,² it was only natural to get diverted by attempting to minimize external distractions with this new toy.

Given programmers’ stereotypical penchant for multi-monitor setups, I was surprised that my cursory research into VR programming environments returned no helpful results. I found a lot of vaporware claiming to “revolutionize coding” using immersive interactive experiences and three-dimensional metaphors for code, but nothing about regular programming using a headset in lieu of computer monitors.

What follows is a description of how I set up a distraction-free programming environment using the Oculus Go, and a review of my experience.

Technical Details

The headset already had Mozilla’s VR web browser “Firefox Reality” installed,³ which defaults to opening new tabs on either side of the main one, thereby allowing multiple pages to be viewed at the same time and emulating the multi-monitor setup I was after. Rather than fumbling with peripherals and installing apps on the device, I resolved to open documentation in two separate browser tabs, and stream my terminal to the third. I would type at my regular computer keyboard, and simply view it through the web browser on the VR headset.

The video below demonstrates the final setup. Note that when the headset remote turns sideways, it has been laid on the desk next to my keyboard so I can type.

There were two steps to getting fully set-up. Streaming my terminal was done using ttyd with a shared tmux session.⁴ Reading documentation was done with PDF.js,⁵ since Firefox Reality does not come with a built-in PDF reader.

Click the subsection headers below to expand for further technical details.

# Download ttyd (I use the latest x86_64 release)
wget --output-document "ttyd" "https://github.com/tsl0922/ttyd/releases/latest/download/ttyd.x86_64"

# Make the downloaded file executable
chmod +x ./ttyd

# Start a ttyd session running in the background
./ttyd --readonly --port 6969 tmux new -A -s ttyd_tmux &> /dev/null &

tmux new -A -s ttyd_tmux

http://<my local IP>:6969/

https://mozilla.github.io/pdf.js/web/viewer.html?file=https%3A%2F%2Fcors.jstrieb.workers.dev%2Fcorsproxy%2F%3Furl%3Dhttps%3A%2F%2Fjstrieb.github.io%2Fposts%2Fvr-programming%2Fassn5.pdf

Reflection

Despite practical limitations of the headset and keyboard setup, the experience left me with the distinct impression that virtual reality is the future of human-computer interaction.

Once VR technology matures, I expect knowledge workers will spend much of their time using immersive desktop environments, rather than looking at real-world screens. Eventually, busy people on planes and trains will goggle into VR workspaces, and remote workers will walk around virtual offices.

In a world replete with distractions, deep focus is an uncommon treat. By using the VR headset, it was possible to instantly shut everything out, and to attain this otherwise rare state of complete concentration.

While wearing the headset, there were no visual distractions. Checking my phone and falling into rabbit holes on the Web were both cumbersome enough to be infrequent – the path of least resistance when working was to keep hacking on the code positioned front and center, looking away only to glance at already-open documentation on either side.

In this case, the difficulty of searching the Web and navigating to new pages was a feature, not a bug. In the course of programming outside of VR, I commonly search for something simple and realize an hour later that all I have done is read “Hot Network Questions” linked on the right side of every Stack Overflow post. Whereas in my VR setup, the most accessible documentation is that which is readable from the terminal interface. Typically, this is documentation already stored offline such as man pages or Python’s help() utility. The benefit of being restricted to offline documentation cannot be overstated.⁸

The two practical limitations of programming in VR were the headset screen resolution and battery. The low resolution was tolerable for about two hours, but the battery only lasted that long, anyway. These points of friction are core limitations of VR technology in general, and will inevitably be overcome as the tech improves. They are also far less limiting in higher-end headsets, even today.⁹

The need to touch-type might also inhibit some people from programming in VR. But I don’t expect this to prevent the steady replacement of computer monitors as headsets get better, since a headset-mounted camera pointing at the keyboard would allow a user to see their hands so they don’t have to type blind. Based on my experience, I also don’t believe novel controllers or interaction modalities are necessary for VR to replace traditional monitors in work environments – using the VR remote instead of the mouse was shockingly natural, and not seeing the keyboard was a minor inconvenience at worst.

It seems obvious that there should be startups working on VR document, spreadsheet, and presentation editing software in addition to programming environments. That way, as soon as there are hardware improvements, those companies would be positioned to be first to market with products tailored for workers looking to replace their monitors with focus-enhancing VR workstations.

On the other hand, since consumer applications are increasingly made for the web, maybe the future is Google Drive, Sheets, and Slides, but optimized for VR. A headset that only runs a browser isn’t too different from a Chromebook, so I wouldn’t be surprised if Google used their web-based, cross-platform dominance to muscle into the market for VR productivity software. Either way, after this first experience working in virtual reality, I will be watching the space eagerly.

Postscript

The utterly complete focus attained while immersed in virtual reality left me eager for more opportunities to work using a VR headset. Since the initial experience in November, I have continued to occasionally author documents and write code using the headset instead of my normal computer screen.

I bought a Raspberry Pi 400,¹⁰ and have ttyd set up to run on boot so I can use it in text mode from the VR headset without connecting a monitor.

Additionally, the following applications seem to work in VR:

• VS Code entirely in the browser with “github1s” (but the lack of keyboard makes it difficult to use)¹¹
• Google Docs works well if I have the same document open on my computer. Google Sheets works less well, and Google Slides is nearly unusable since these last two both rely more heavily on the mouse than Docs

Finally, I was able to use Apache Guacamole with the graphical mode of my “CTF Collab” project to run an entire remote desktop session in the browser where the computer I connected to was running on GitHub Actions.^{12, 13} This web-based remote desktop session enabled me to run graphical applications like Ghidra from within the virtual environment without installing anything on my computer.

There are undoubtedly many additional ways in which regular productivity applications can be brought to VR, and I can’t wait to see how the field develops.