Here Are All the Things in You Can Do in Augmented Reality with the Samsung Galaxy S9 & S9+

Samsung's big reveal of the Galaxy S9 and S9+ at Mobile World Congress revolved around its "reimagined" camera and augmented reality capabilities.

Between AR Emojis, Bixby, and ARCore, the Galaxy S9 and S9+ offer more native AR software features than any other device. AR is such a central component that Samsung opted to show off the device to attendees via AR using the Unpacked app (as expected).

"After all this, I'm sure you can tell that I really love this camera. Not only because it takes the best pictures, but because it makes everything in your life easier, from shopping to eating to traveling. And with AR, the S9 camera will revolutionize the way you work, play, and interact with the world around you," said Erin Willis, Senior Manager Channel Marketing for Samsung, during the AR portion of the launch event.

AR Emojis

The headlining AR feature for the Galaxy S9 and S9+ is AR Emoji, which uses a machine learning algorithm to render an animated 3D model from more than 100 facial features of the user and then mirrors expressions in real time.

Here Are All the Things in You Can Do in Augmented Reality with the Samsung Galaxy S9 & S9+ — *Image via Samsung*

In other words, this is Samsung's copy of Apple's Animoji on the iPhone X. However, Samsung packs more features and flexibility into AR Emoji.

AR Emoji matches Animoji with cartoon characters that mimic the user, but AR Emoji also creates Bitmoji-like caricatures of the user's likeness that can be adjusted and customized. So, instead of having unicorns and aliens singing "Redbone," you can conceivably have a cartoon Childish Gambino perform his own song. AR Emoji also produces a number of animated reactions that users can share.

In addition, AR Emoji will let users turn themselves into Disney and Pixar characters, including Mickey Mouse, Minnie Mouse, and the Incredibles.

Moreover, Samsung has made AR Emoji more open than Animoji. Apple technically limits Animojis to the Messages app, though screen recording provides a loophole to capture the animations for use elsewhere. AR Emojis are recorded in an AGIF file format, making them shareable with third-party apps.

So, when Elastigirl said "I think you need to be a bit more flexible," Samsung took it to heart for AR Emoji.

Bixby

Samsung's artificially-intelligent assistant, Bixby, has learned some new tricks by way of its computer vision capabilities. For instance, Bixby can translate text in real time, just like Google Translate.

For calorie counters, Bixby can also recognize foods and provide calorie counts for tracking in the Samsung Health app. See, this is what you should have done with SeeFood, Jian Yang.

Bixby is also getting into the cosmetics marketing game, adding a Makeup Mode. Users can virtually try-on makeup through the front-facing camera and purchase products directly through the app. Having this feature native to the device is a big advantage for the participating brands, namely Sephora, Cover Girl, and Amore Pacific in the US (with Sephora being available in Canada and Amore Pacific in Korea and China).

Update: A Modiface spokesperson has confirmed that the Bixby Makeup Mode is in fact based on Modiface's Beauty AR platform. The company announced the partnership on Twitter.

Samsung plans to make Bixby an AR shopping platform. For instance, according to Samsung's news release, Bixby can identify products for purchase through retailers such as Walmart and Sam's Club.

"As other companies join our platform, there'll be more opportunities to use AR to improve the way you shop," said Willis, who we've learned has an intermediate understanding of Spanish, leads a healthy lifestyle, and loves shopping for makeup (but is too busy to go to the store).

ARCore

With Google's official release of ARCore (and developer preview support), we already knew that the two previous generations of Galaxy and Note handsets would support the Android AR toolkit.

So, the Galaxy S9 and S9+ gaining similar support was a foregone conclusion. We already had Google VP Clay Bavor's word on that, too.

As the leading Android OEM in terms of handset sales, Samsung's broad support of ARCore is critical if Google intends to catch-up with Apple and ARKit.

The AR Smartphone

With all of its capabilities, the Samsung Galaxy S9 is arguably the AR smartphone. It matches or surpasses everything the iPhone X and Google Pixel can do in AR.

At the moment, there are more ARKit apps in the wild, but we expect developers to begin porting their AR apps to ARCore now that it is official. While the iPhone has CoreML and the Pixels (and now the Google Photos app) have Google Lens, Bixby offers a computer vision platform for others to leverage AR capabilities directly through the camera app.

On paper (or digital display), the Galaxy S9 and S9+ offer the most AR. We're anxious to see if real-world performance matches the feature set.