Apple iOS-Powered Sesame Street Yourself App Turns Kids into Their Favorite Characters via ARKit 3's New Features

The longest-running children's television show in the United States, Sesame Street, hasn't survived 50 years without leveraging new technology along the way.

The latest innovation from the Sesame Street Workshop is Sesame Street Yourself, a new iOS app that takes advantage of the new features of ARKit 3.

• Don't Miss: Apple's ARKit 3 Casts Next-Level Illusions via Meisai AR App

Available now via the Apple App Store, Sesame Street Yourself uses ARKit's multi-face tracking feature to identify and morph users into their favorite Sesame Street characters. The ad-free app also entertains children ages 3-5 with games, singing, and dancing.

Weyo

"We just thought [ARKit 3] is a fantastic, gigantic toolbox of tricks. [We wanted to see] what we [could do with] the tech for fantastic immersive interactive experiences for children," said Baz Palmer, the co-founder of Weyo (the game's developer), in an interview with Next Reality.

Palmer and his team took advantage of ARKit's feature set for both the rear and front camera. For example, on the rear camera, children can help Cookie Monster bake cookies. On the front camera, they can become Ernie and try on different outfits.

Weyo

Using the multi-face tracking technology of ARKit 3 gives parents the opportunity to play along with their children, with each face wearing a different character mask that tracks the individual as he or she moves within the camera view.

"Working with the Sesame brands, you've got these lovable characters and have a really clear demographic that will try to engage," said Palmer. "It really focused the way we want to use the technology."

Weyo

Weyo originally included elements that took advantage of the People Occlusion and Motion Capture features of ARKit 3 as well, but, according to Palmer, the team excluded those features from the final release to keep the app lighter. However, they plan to offer those experiences later as add-ons to the app.

Since the app uses ARKit 3, it will be limited to compatible iPhones and iPads, namely iPhone XS, XS Max, and XR, the 2019 editions of the iPad Air and iPad Mini, and the latest 11-inch and 12.9-inch iPad Pro models.

This isn't the Sesame Street Workshop's first foray into augmented reality. In 2013, the organization integrated the Vuforia AR platform into Big Bird's Words, an educational game that used Vuforia's computer vision to help players find words in the real world. Vuforia also supplied the AR behind a prototype AR playset.

Nonetheless, Sesame Street Yourself is the first household brand name to adopt ARKit 3, which is significant considering that ARKit 2 never really caught on. The uniqueness of People Occlusion and Motion Capture might just be the secret ingredient to getting app developers to build more innovative AR experiences.

Cover image via Weyo