I joined Niantic 3 years ago because I believe that video games can be more than a couch experience and the metaverse should not be confined to your living room. It wasn’t just that Niantic is working on AR and games that attracted me; it is the combination of pairing games with the physical world and the challenge of building the underlying infrastructure that brought me here.

There are many other companies building their vision of a fully-rendered artificial metaverse experienced in a headset, but the real-world metaverse that we’re building at Niantic is different and it’s everywhere. It occupies the space you’re in right now and all of the places you go. When our vision is fully realized, you will be able to use your mobile phone or your AR glasses to see that these augmented experiences are all around you, waiting for you to discover them.

We have been working to augment reality since we launched Ingress 10 years ago. This game was the first of its kind, making real world places part of a video game at a global scale. Although the technology did not exist to create world-scale persistent visual augmented reality, we knew that many players were immersed in this alternate reality game world, even if they could not see it yet. Places like their local library became an Ingress portal, and every player experienced it this way.

With the launch of Pokémon GO a few years later, we were able to create another augmentation of the same locations where people were playing Ingress. But to Pokémon GO players, this local library was a gym that they battled to control. It became the biggest AR game, allowing its players to see their Pokémon in the real world. To this day, Trainers have shared billions of pictures from these AR sessions. However, these experiences are just an appetizer – the full AR feast is yet to be served. Blanketing the earth in fully persistent AR is an immense technical challenge, and the hardware needed to realize our full vision does not yet exist.

Niantic’s engineering team is focused on solving these technical challenges, and we have made great progress in laying the foundation for world-scale, fully persistent AR. We believe this technical foundation will be the infrastructure of the real-world metaverse, and that gaming experiences will be how most people experience it for the first time. It’s an exciting time to be a game developer.

At Niantic, our vision for the real-world metaverse is informed by four key beliefs:

1. It will start with games,
2. the map is the key piece of infrastructure,
3. it will be a constellation of worlds and,
4. persistence will bring it to life.

I believe that the social component of gaming will be more compelling in the real-world metaverse than it will be in a virtual world.

1. It will start with games

Many modern-day video game worlds are already being called metaverses. The creators of these worlds have worked around the technical limitations (immersive 3D rendering, shared game state, network latency, etc.) to create believable, shared virtual experiences. I believe this expertise makes video game creators uniquely equipped to create the real-world metaverse. This existing talent pool knows how to work around technical constraints and to focus on what’s most important for the suspension of disbelief needed to bring players into a virtual world.

As an avid gamer, I experience and enjoy some of the very best online gaming experiences. I love multiplayer games like Monster Hunter and Mario Kart because I’m able to share the worlds the game designers have created with my friends and family. Having fun with the people you care about is definitely a worthwhile pursuit, and I would be more than delighted if the people who first experience the real-world metaverse do it while playing games with their loved ones.

I believe that the social component of gaming will be more compelling in the real-world metaverse than it will be in a virtual world. While there will be tools to communicate and coordinate with your friends while you’re online (we’ve built a tool called Campfire for this purpose), you’ll get the most out of the experiences in real life, playing next to each other. And although it takes more effort to get together in the real world, the experience is worth it. I’d rather see you than your avatar.

I am a strong believer in the positive effects games have on people, and Niantic builds games that combine the positive benefits of gaming with real life. Gaming that allows you to interact, socialize, and exercise with other people leads to positive effects on mental and physical health for people who play. So building a real-world metaverse that is based on gaming not only makes good business sense, but it’s aligned with my belief that games can be a positive force in the world (and it’s why I chose to work here).

Having fun with the people you care about is definitely a worthwhile pursuit, and I would be more than delighted if the people who first experience the real-world metaverse do it while playing games with their loved ones.

2. The map is the key piece of infrastructure

Ingress and Pokémon GO pioneered immersive real-world gaming and AR experiences, but they also set the foundation for the development of a 3D map built specifically for AR. This 3D map is foundational infrastructure for both storing the virtual content and for localizing the player (determining their position and orientation). The players of these games helped to build this map, at first by nominating the interesting locations to play the games, and later by providing the imagery of these locations used to build the map.

This 3D map is crucial infrastructure for the real-world metaverse and location-based AR. Traditional maps are good for looking up addresses and getting directions, but they are not sufficient to power ground-level AR experiences. Most imagery used by traditional maps come from satellites and from cameras mounted on cars. But a true AR map needs to work everywhere, including places where satellites and cars cannot see. For our games to work wherever people play them, we need to have a detailed perspective of what a player can see while they’re playing.

We are laser-focused on building the technical infrastructure that powers a 3D map with centimeter level accuracy. We are building this 3D map using user generated content (UGC) consisting of videos and images submitted via our games and Wayfarer app. This initial version of the 3D map covers where our games are played today, but this is just the starting point. We need to expand the map to work in all of the places where people want to experience AR. We need to keep this map up to date to provide users with the most realistic and performant experience as possible. In order for the metaverse to exist in reality we will need to know what reality is at all times.

This year, we launched the Lightship Visual Positioning System (VPS) and AR map, VPS for Web, and the ability for developers to map their own locations using the Wayfarer app. For the first time, developers can use Lightship VPS to anchor persistent AR content to real-world locations. In other words, developers now have the ability to show location-specific virtual content to their users and this virtual content will now feel like part of the environment to anyone who visits the location. Developers can also add new locations to Niantic’s AR map so they can build their own AR experiences there.

3. It will be a constellation of worlds

As we continue to build out this 3D map, developers can leverage it to place their content and build their experiences. With multiple experiences overlaid on reality, you’ll be able to swap in or choose the one you like. In Ready Player One, people teleport to different planets, but we envision people being able to load in and have different experiences wherever they are in the real world. We call this concept “reality channels.’’ These “reality channels” already exist today as separate Niantic games like Pokémon GO and Pikmin Bloom.

For example, two people could be standing in the same place, and one person playing Pokémon GO will see a gym with a life-sized Gyrados, while another person sees a gigantic flower blooming in Pikmin Bloom. And in our future real-world metaverse, you can tap a button to switch reality channels. At any given location, there will be different experiences being enjoyed by different people. Real-world metaverse experiences will be as diverse as the people using them.

4. Persistence will bring it to life

While the 3D map is necessary infrastructure for rendering AR content, the AR world only comes to life when the content and the experiences are persistent. This world has to exist even when you’re not playing.

Lightship VPS powers the persistent placement of 3D assets in the real world, and it is the platform upon which the metaverse will be created. Persistence in AR will unlock different kinds of experiences that do not exist today in most games. Even in my favorite video games, the worlds are created when I start playing with my friends and then are erased from the computer’s memory when we finish. The closest thing we have are MMORPGs and worlds created in games like Minecraft, but they are limited to the digital world. Our goal is to enable this world to exist at a planet-scale, anywhere where people want to play.

We’ve already launched the tools to make this happen. Developers can use our Lightship platform to anchor their 3D assets – an AR experience – to a Wayspot. They can then develop an app to allow players to view it. When enough of these experiences are anchored, they will open up an “undiscovered world” that people can access on their smartphone or headset.

The technical challenges in creating the real world metaverse are diverse and interesting. We are still building performant systems for mapping, persistence, 3D rendering, and concurrent player support, to name a few. I’ll explore more of that in Part 2 of this series.

– Yuji Higaki, SVP of Engineering at Niantic