Nreal air virtual monitor

· ranguna's blog

A TLDR is available, that only includes the basic steps needed for virtual displays to work. For slightly more in-depth explanations, continue reading from the next section.

# First attempt

  1. Connect the nreal air to the computer via USB-C cable;
  2. Nothing happens :(

# Initial setup for the nreal air

# Activation

Nreal glasses cannot be used out of the box before being activated first. Activation can be done either with a PC or a device compatible with the nebula app.

# With a PC
  1. Connect the nreal air to a computer via USB-C cable;
  2. Activate the glasses by following the instructions in https://activation.nreal.ai/en/nreal-air-upgrade.html;
  3. Done!
# With the nebula app
  1. Using a smartphone from the compatibility list* or a compatible computer (like the MacBook M series);
    1. Using a smartphone:
      1. Install the Nebula app:
        • Android 12 support is dodgy. If the next steps do not work, try to either downgrade to Android 11 or download nebula version 2.3.4 (I won't link it here since I couldn't find an official source, you'll have to find it on your own cough apksos.com cough);
    2. Using a compatible computer:
      1. Install the Nebula app. You'll have to search online for the installation instructions;
  2. Connect the nreal air to your device via USB-C cable;
  3. Wait for you the glasses to be picked up by the app;
  4. Follow the onscreen instructions to activate the glasses;
  5. Done!

* There are smartphones outside the compatibility list that work. From what I could gather, the basic requirements are:

# Virtual monitor

When connecting the glasses directly to a computer, they simply act as a static monitor fixed to the center of your view. The nreal air has 3DoF, which should allow us to fix the monitor in 3D space, giving us the ability to move our head around while the monitor stays in its fixed point in space instead of following our head's movement.

As of the time of writing, 3DoF is only available in MR space with the nebula app either using a compatible android phone or using a Mac M series.

# With a PC and a compatible smartphone

The idea is to create a new virtual display on a machine (either with a dummy plug or with xrandr) and screen share the new virtual display to the nebula app in MR space.

To share the display, we'll use deskreen. Deskreen captures the display and makes it available for viewing in a web browser. The web browser can be accessed from the nebula app in MR space and can be placed anywhere under the constraints of 3DoF.

  1. Have both a smartphone from the compatibility list and a computer connected and accessible on the same network;
  2. Install deskreen on your computer:
    1. Arch Linux: yay -S deskreen;
    2. Others: https://deskreen.com/lang-en#page-top.
  3. Open deskreen on your computer;
  4. Open the nebula app on your smartphone and launch MR space;
  5. Copy the link displayed in the deskreen app from your computer and past it into the MR space web browser;
  6. You'll notice that the page fails to load, showing an SSL-related error.

It seems that the way the nebula app was done, it automatically transforms all http links to https. Although the way deskreen works seems secure, nebula webview wants to follow the https standard. The original maintainer of deskreen does not want to implement https, so we'll have to find a way to expose the deskreen server running locally through https while the service itself is being served through http (or we could fork the project and implement https ourselves, but that would require some extra steps to generating and installing certificates).

To expose the deskreen server through https, we'll use ngrok. There are many alternatives to ngrok, but I'll be using it because it's the one I'm most familiar with.

Using ngrok (or another tunneling service) also lifts the restriction of having both the computer and the smartphone connected to the same network. Although you'll still have to find a way to use inputs remotely, like a mouse and keyboard (but that's outside the scope of this post).

  1. Install ngrok:
    1. Arch Linux: yay -S ngrok;
    2. Others: https://ngrok.com/download.
  2. If deskreen is not already running on your computer, open it;
  3. Open a terminal and run ngrok http 3131 (3131 is the default port for deskreen, change it if needed);
  4. Copy the https link that is shown in the terminal;
  5. Open the copied link in the nebula app in MR space;
  6. After a few seconds, the page should show an error.

The root cause of this error is the usage of hard-coded values for the port in deskreen. There is a pull request open to fix this, but until it's merged, we'll have to use the fork with the fix:

  1. Uninstall deskreen;
  2. Clone this fork of deskreen locally;
  3. Install all the required packages:
    1. Inside the cloned repo, run yarn install && cd app/client && yarn install && cd ../..;
    2. If there is an error due to a missing command, make sure you have all the build tools installed (yarn and nodejs, possibly npm too).
  4. Run deskreen: yarn start;

Once deskreen is running, repeat the steps from the previous section:

  1. Copy the https link that is shown in the ngrok terminal;
  2. Open the copied link in the nebula app in MR space;
  3. On your computer, the deskreen app should now show a new device requesting access. Follow the onscreen instructions to allow access to one of your existing monitors or a specific app;
  4. Done!

You should now have one of your existing displays or a specific app being shared in MR space. But we want to have a new display, not an existing one.

To do this, you can either use a dummy display plug or use xrandr to create a new virtual display.

# With a dummy display plug (untested)
  1. Get a dummy display plug;
  2. Plug it into your computer;
  3. Configure the new display to have 1920x1080 resolution (same as the nreal air);
  4. When connecting to deskreen from the nebula app, in deskreen, select the new display as the one to share;
  5. Done!
# With xrandr
  1. In your computer, make sure you have a VIRTUAL1 display by running xrandr --query | grep VIRTUAL;
    1. If something pops up, continue with the next step;
    2. If not, search online on how to create a virtual display, it might be tricky if you are using nvidia. Good luck.
  2. The existing virtual display should show up with no resolution and it should display as disconnected. For example, this is my output from xrandr --query | grep VIRTUAL: 
    VIRTUAL1 disconnected (normal left inverted right x axis y axis)
    1. To connect the display, set a resolution to the virtual display by running the following:
    1xrandr --newmode 1920x1080_60.00 $(cvt 1920 1080 60 | grep Modeline | sed 's/^Modeline \".*\"\s*//g')
2xrandr --addmode VIRTUAL1 1920x1080_60.00
    1. And place the virtual display wherever you want. For example, placing it to the left of the main display (in my case eDP1):
    1xrandr --output VIRTUAL1 --mode 1920x1080_60.00 --left-of eDP1
  3. When connecting to deskreen from the nebula app, in deskreen, select the new virtual display as the one to share;
  4. Done!

On my machine using intel iGPU, VIRTUAL displays are automatically added when assigning a mode. Meaning that multiple VIRTUAL displays can be created by running the same commands multiple times changing VIRTUAL1 to VIRTUAL2/VIRTUAL3/VIRTUALn according to the result from xrandr --query.

# TLDR

# Initial setup

  1. Make sure your glasses have been activated;
  2. Install ngrok:
    1. Arch Linux: yay -S ngrok;
    2. Others: https://ngrok.com/download.
  3. Install deskreen:
    1. Clone this fork of deskreen locally;
    2. Install all the required packages:
      1. Inside the cloned repo, run yarn install && cd app/client && yarn install;
      2. If there is an error due to a missing command, make sure you have all the build tools installed (yarn and nodejs, possibly npm too).
    3. Run deskreen: yarn start;

# Connecting the glasses to the virtual display

  1. Create a virtual display using a dummy plug or with xrandr;
  2. Open a terminal and run ngrok http 3131 (3131 is the default port for deskreen, change it if needed);
  3. Copy the https link that is shown in the terminal;
  4. Open the copied link in the nebula app in MR space;
  5. In your computer, the deskreen app should now show a new device requesting access, follow the onscreen instructions to allow access to one of your existing monitors or a specific app;
  6. Done!

# Future work

# Caveats