At Niantic, we are committed to augmenting reality with content that changes our perspective and improves our understanding of the world. Doing so requires great technological advancements in geospatial mapping and intelligence. Pokémon Playgrounds, which we are rolling out in nine cities around the world as an experimental feature in Pokémon GO, in collaboration with The Pokémon Company, represent an important milestone in this quest. With Pokémon Playgrounds, we’ve turned many thousands of PokéStops and Gyms around the world into persistent shared AR experiences, where Trainers can place Pokémon for other Trainers to discover and take Pokémon GO Snapshots. This feature is made possible by our Visual Positioning System (VPS), a service that aligns persistent digital content with real world locations with centimeter accuracy.
What is the Niantic Visual Positioning System (VPS)?
The camera-based augmented reality (AR) experiences in Niantic games have, until now, lacked a true sense of persistence because there hasn’t been a way to relate Pokémon to the world with the precision required to make them feel like they were part of it. While we’re able to use signals like GPS and compass data to place Pokémon on the 2D map, there is far too much error in those signals to place AR content convincingly in the 3D physical world. To place content accurately in AR, developers rely on anchors, a concept in computer vision that defines relationships between the physical characteristics of an environment and digital content. Until now, anchors have not been easy to persist across gameplay sessions, meaning players who place content in a game session could not share that content with other players or recover it in subsequent sessions. To address this challenge, Niantic developed its Visual Positioning System (VPS). VPS creates and serves anchors for real-world locations at scale. With VPS, anchors and their associated content can be easily stored and retrieved within any database solution by simply associating each digital asset in relation to its VPS anchor.
In part, it’s our Niantic players (Pokémon GO and Ingress players specifically) and their dedication to our games that have supported a large collection of scans throughout the world that allowed us to deliver a new form of AR gameplay to our users.
With VPS, AR developers can create shared, persistent, augmented perspectives of the world. Shared and persistent experiences are critical in the adoption of AR, since in order for us to exist in the world, we need to be able to maintain a shared understanding of it. As you can imagine, we can now provide incredible magic moments that will form a more meaningful connection between players, our game worlds, and the communities they inhabit.
The VPS service, in addition to these map-based anchors, contains context and insight that allow games to make interactions at these locations more connected to place than ever before. Before a player ever visits a location, we can make game design decisions based on environmental characteristics - like whether a location is a park or an art installation, or whether it primarily supports pedestrian or car traffic.
Harnessing VPS for Pokémon Playgrounds’ groundbreaking development
Niantic’s VPS plays a crucial role in Pokémon Playgrounds. Here’s an overview of how:
We can place a digital character and have a very good idea of where it is on Earth, which can lead to some pretty interesting storytelling.
We can identify areas where gameplay is more appropriate and acceptable than others. While the idea that AR content is everywhere is very compelling, that’s not how we play games. As players of games, choosing where, when, and how long we engage in gameplay is very important. VPS gives us great insight about our Pokémon Playgrounds, from the environmental characteristics (mentioned above) to its playability (the size of a play area and how conducive the space is to gameplay).
Having pre-processed these maps, we can retrieve an anchor and interpret the map at a fraction of the time and performance cost associated with interpreting the scene in real time. We only need a single frame of compressed camera data to identify precisely where the camera is in relation to a landmark.
AR development tools are also important to delivering high quality camera-based AR experiences. The VPS service not only provides anchoring capabilities for digital content, but the dataset that drives it can also be repurposed to assist in the development of AR features.
Navigating Challenges and Leveraging New Tools in Real-World AR Development
The goal of Pokémon Playgrounds was to provide Pokémon GO players with the ability to place their Pokémon in real-world scenes and share those experiences with other players. Delivering the feature required us to overcome many different challenges, including:
Efficiently collecting and storing player scans and processing those scans into map data.
Ensuring VPS would perform well under many different conditions, so that content placed in a scene under certain weather or lighting conditions could be reliably accessed by players in different conditions.
Developing a feedback loop to identify and improve locations where VPS is underperforming.
Leveraging contextual data and semantic analysis to identify safe and socially acceptable locations for the experience.
Creating a user experience that would help first-time players, especially those unfamiliar with AR, be successful.
Despite our challenges, we also found many ways to use VPS data to make our lives easier along the way. The data we use to precisely place digital content also serves as a great tool for testing AR features from the comfort of our desks. Within the Niantic SDK for Unity are a suite of remote authoring, session recording, and playback tools, which are game changers for AR development. Using remote authoring in combination with the Niantic Geospatial Browser, developers can download VPS meshes directly into a Unity project. Even better, developers can test all Niantic SDK AR interactions – depth sensing, occlusion, real-time meshing, and semantic segmentation – directly in the Unity Editor as if it were running on device. Remote authoring makes our iteration times much faster and allows us to playtest our experiences in both AR and MR – even on rainy days.
The Future of AR Gaming
We’re excited to shape the future of AR, especially in the context of Pokémon GO. VPS maps give us understanding and insight into the world around us. The more knowledge we have about specific locations, the more we can use our various AR tools to create more magical, shared experiences. In addition to all of the new fun interactions we can have Pokémon do in the context of the world around us, we can also use the insight provided by VPS to anticipate where players may want deeper engagements with AR more than others. This type of spatial understanding is not only valuable now, when using AR on mobile phones, but also key to driving headset-powered AR experiences in the near future.
In the future, there’s an opportunity to link AR content to these locations in much more contextually relevant ways. If a character has an affinity to water or light, having them respond to water fountains and street lamps is something we can support. These kinds of deep connections to the world require that we lean in to gathering more spatial understanding and intelligence about the real-world at scale. This involves our efforts to develop a Large Geospatial Model (LGM) that integrates detailed geometrical, visual, and cultural data into scene-level features.
It’s hard to state how many technical innovations were required to get to where we are now: having Pokémon simply act as if they are physical inhabitants of our world. We are so excited to have Pokémon stay where you left them because an illusion like that at world scale was not possible until now. And now that we’ve achieved that, AR play can become a lot more connected to our world, creating endless opportunities for deep immersive moments of play in locations we frequent in our communities. Pokémon can now share the world with us, connect and interpret the world in meaningful ways, and we can start to change the player perspective of the world and other players in fun and positive ways. We’re thrilled to be on this journey with you and hope you enjoy having Pokémon Playgrounds as part of your player experience.
Evie Powell, Ph. D.
Staff Engineer, Life-Long Games Researcher, and Project Lead on Pokémon Playgrounds