Since the ‘90s we’ve been captivated by the charm of virtual pets – from Tamagotchis, to Nintendogs, to Neopets. There’s something whimsical about virtual companions that are always with us, waiting for their next treat or virtual hug. Technology has come a long way since then and evolved our 2D companions into hyper-realistic ones in virtual and augmented reality, but our affection for them remains the same. 

Today virtual pets are becoming even more lifelike in Peridot, the first fully augmented reality game to use generative AI to give virtual creatures, or ‘Dots’ the smarts to react in unexpected, surprising and delightful ways to things they find in the real world – just like a real pet would do. 

For example, just as you might not know how your puppy would react to seeing a cat on the street, your Dot will react in unpredictable ways, too. Will they growl, bark, or dive after it? If you leave a plate of food sitting out, will your Dot steal a bite, or roll around in your leftovers?

Using generative AI, Dots in Peridot will likewise demonstrate unique, random, surprising or plain adorable reactions when encountering new objects, giving the impression that the Dots are learning and understanding the world around them.

Another First for Peridot 

Already the first fully AR game, Peridot is also the first to use generative AI for the sole purpose of making adorable creatures even more adorable. Generative AI in Peridot offers a glimpse into the future of gaming powered by augmented reality and artificial intelligence. We believe it is one of – if not the first – implementations of generative AI in a mobile game. 

Using previous game design techniques, our teams would have needed to manually program a limited range of reactions for Dots when they encounter various things in the world. Using generative AI, the team can dramatically increase the possible behaviors when Peridots encounter new stimuli, for example, another animal, a plate of food, or a bouquet of flowers.

How it works:

Step 1: Understanding the environment

Using Niantic’s AI algorithms for computer vision, images of the real world seen through the user’s camera are converted into accurate 3D shapes, so that Peridots can frolic in their space. Using Niantic Lightship ARDK, Dots can recognize a variety of things such as flowers, food, pets and many other real-world objects and surfaces. 

Step 2: Passing to the LLM

Those words are then passed to a large language model, in this case a customized use of Meta’s Llama 2, along with information on the Peridot’s characteristics and personality, such as whether they are outgoing or shy, their age and history. The Peridot backend then asks Llama how a small fluffy creature might respond to what it found. 

Step 3: Generating the reaction

Peridot had a unique opportunity to use AI to take advantage of a vast library of unused animations which were difficult to leverage for precise environmental conditions. Thanks to the new system leveraging generative AI, those animations are now part of a much greater number of possible reactions. It’s as if the Dot is learning, adapting and demonstrating personality like living creatures. 

For example, during the testing process, one of our team member’s Dots found an empty plate of food, and instead of simply sniffing it, much to their surprise, the Dot rolled around in it.

We see this as an early example of how game designers and producers can use generative AI to make characters in games more lifelike and interactive. More broadly, as augmented reality characters evolve, they will need to use AI not only to understand the world but also how to react to it, and to other people. 

For Peridot, we see the game continuing to be a platform for experimentation in the applications of AI and AR. We’re excited to keep pushing the boundaries and truly make these creatures come to life in our world on mobile devices with an eye on future hardware.

–The Peridot team