Localizing With a Shared Environment

This document describes how to localize multiple users in a shared environment using the ARDK by having users scan a common object in their environment.

These steps assume you’ve already added code to your AR experience that uses ARDK networking APIs to create a ARNetworking session that clients can join. See Getting Started with Shared AR and Setting Up a Networked AR Session for more information on creating and joining ARNetworking sessions.

Synchronize Using the Environment

Synchronizing using the environment works as follows:

  • The client designated as the host scans a static object in the environment and successfully uploads at least one initial scan to Niantic’s AR Backend.

  • Non-host clients receive the initial scan data from Niantic’s AR Backend, scan the same object, and then use all of this to synchronize and co-locate within the shared environment.

  • All clients now share the same coordinate space (the host’s) and can share co-located Pose information so that every client can locate other clients in the (shared) world.

Note

All clients must be using the same API key (configured in the application’s ArdkAuthConfig as described in Authentication) when trying to join the same shared AR session.

Scan the Environment as Host

First, the client designated as the host should scan the environment to collect mapping information that can be shared for synchronizing with other clients. Configure and run an ARSession to collect the mapping data. You can use an ARNetworkingManager or enable IsSharedExperienceEnabled in your ARSession, as described in “Starting a Shared Experience” in Setting Up a Networked AR Session.

As soon as IsSharedExperienceEnabled is enabled, the session starts collecting mapping information. The client designated as the host will start scanning the environment and send mapping data back to Niantic’s AR Backend. The host must send at least one map back to Niantic’s AR Backend for the shared session to work, although the host will regularly send updates to Niantic’s AR Backend throughout the lifetime of the session.

To ensure that the host user can scan a good initial map, we recommend providing clear instructions in your game during this scanning process, with the following guidelines communicated to the user:

  • Find and scan a stationary object in your environment. Look for an object with many sharp features (e.g. edges with high contrast, non-symmetrical/repeating patterns) and no shiny or reflective surfaces. One example might be an object with distinct features and colors, like a shoe.

  • Stand 3+ feet away from the object you are mapping.

  • Have your device camera looking straight at the object, not from a high or low angle. Avoid looking at the floor. We want to understand as much of the 3D scene as possible, and wide, flat surfaces don’t help. You may need to position the object on a raised surface, like a table.

  • Keeping your camera looking at the object, slowly move a few steps in a circular direction around the object, both to the left and right. Avoid moving in a direction towards or away from the object. Avoid swiveling the camera (i.e. only rotating around a single point without any side-to-side or up-and-down movement).

  • Continue moving left and right until the scanning completes. This should take approximately 15 seconds, but may take up to several minutes depending on the complexity of the environment and the capabilities of your device.

The following video demonstrates what mapping and localizing can look like in-game:

If you’re testing the scanning process, make sure not to have a debugger attached, as this may negatively impact performance.

Throughout the scanning/mapping process, provide feedback to the user indicating that scanning is progressing. You can render the point cloud data from IARFrame.RawFeaturePoints using ARFeaturePointRenderer to show the user how the scan is progressing.

The host will remain in the WaitingForLocalizationData PeerState until the first scan completes. Once the first scan has completed, the host PeerState will be updated to Stable. You can communicate to the user that a successful scan has been uploaded and that other users can join the session and start to localize. If you want to display the scanned map, you can use ARMapVisualizationRenderer.

Synchronize the Environment as Non-Host Client

When other non-host clients join the session with IsSharedExperienceEnabled enabled, they will need to localize by scanning the same environment and synchronizing with the mapping data uploaded by the host.

When a client joins the session, Niantic’s AR Backend will immediately start sending it any mapping data that was uploaded by the host. You should have your user start scanning the same stationary object the host scanned, providing similar guidance as described in “Scan the Environment as Host” above. You can track the progress by checking the PeerState for the client.

  • If the PeerState is Unknown, the client has not connected to Niantic’s AR Backend yet. You can tell the user that the game is waiting for a connection to Niantic’s AR Backend.

  • If the PeerState is WaitingForLocalizationData, this means the host hasn’t completed uploading an initial map yet, or that there are network issues. You can indicate to the user that the client is waiting for the host to upload data.

  • If the PeerState is Localizing, this means the client has received mapping data from Niantic’s AR Backend. At this point the user should start scanning the same environment as the host. Use the same user guidelines as listed for the host scan process. Users should scan the same object/environment that the host scanned, at the same distance and device orientation.

  • If the PeerState is Stable, this means ARDK has synchronized the user’s scan with the downloaded data from Niantic’s AR Backend and the client is now localized in the shared environment.

  • If the PeerState is Failed, this means the host never successfully uploaded an initial map and left the session. The client should also leave the session and try to connect to a new session.

The following basic code example subscribes to PeerState updates and checks the PeerState:

using Niantic.ARDK.AR.Networking;
using Niantic.ARDK.AR.Networking.ARNetworkingEventArgs;

IARNetworking _arNetworking;

void SubscribeToPeerStateUpdates()
{
 _arNetworking.PeerStateReceived += OnPeerStateReceived;
}
void OnPeerStateReceived(PeerStateReceivedArgs args)
{
  var peer = args.Peer;
  var peerState = args.State;
  // In this example, ignore update if it's not for the local device
  if (_arNetworking.Networking.Self.Identifier != peer.Identifier)
    return;
  // Leave session (or do something else) if localization failed
  if (peerState == PeerState.Failed)
    _arNetworking.Networking.Leave();
  // ...process other PeerStates accordingly...
}

Alternatively, the up-to-date localization states of all peers can be retrieved through the IARNetworking interface.

var allPeerStates = arNetworking.LatestPeerStates;
var myPeerState = arNetworking.LocalPeerState;

See the ARNetworking sample in ARDK-examples and the AR Voyage sample on ARDK Downloads for ways to scan localization data and communicate the process to the user.

See Sharing Virtual Object Information for how to share client positions and orientations in the shared AR space once all clients are localized.