Header Banner
Next Reality Logo
Next Reality
Mobile AR News
nextreality.mark.png
Apple Snap AR Business Google Instagram | Facebook NFT HoloLens Magic Leap Hands-On Smartphone AR The Future of AR Next Reality 30 AR Glossary ARKit Dev 101 What Is AR? Mixed Reality HoloLens Dev 101 Augmented Reality Hololens How-Tos HoloLens v. Magic Leap v. Meta 2 VR v. AR v. MR

Turn the Sky into a Shared Canvas with This Fun Augmented Reality App

Sep 6, 2018 06:27 PM
Child joyfully taking a photo in a colorful, artistic outdoor setting.

While we've seen Snapchat apply sky segmentation to AR content, the makers of the Blue Sky Paint app have applied similar capabilities to create and share airborne art.

Published by AR developer Kevaid and available now on the App Store, Blue Sky Paint is similar to Similar to Google's Just a Line app, where users press the screen and move their device around to create lines in their immediate surroundings.

However, using a combination of existing features of iPhones and ARKit along with some machine learning to identify the sky, Blue Sky Paint draws only on the sky, with lines disappearing behind trees, buildings, and other visual obstructions.

According to lead creator Aidan Wolf, the app uses ARKit's camera feed to capture the environment and a machine learning algorithm to determine what is sky and what is not. For anything that is not the sky, the app applies graphic shaders as a mask, hiding the user input from view and giving the illusion of occlusion. Also, the app uses GPS to determine the user's location and the built-in compass of the iPhone for directional orientation.

"Basically, really general things that are true about the world most of the time: the sky is always up so if your phone is pointed up its likely looking at the sky, the sky is generally blue (even the clouds) while the ground is usually lit directly by the sun giving it a yellow tint," Wolf told Next Reality. "GPS and compass orientation are what's used, but we do have a re-projection algorithm which redraws the drawing to match different perspectives and distances. The fun thing about using the sky is we can get away with a lot of GPS distortion since the sky is 2.5 km off the ground."

The developers also use the orientation as part of the user interface. Looking down in the app brings up the color palette (another differentiator between the app and Just a Line).

The dataset for placing AR content also enables the app to achieve multi-user experiences and persistent content out of the box, well ahead of the similar capabilities forthcoming in ARKit 2.0 later this month or other mobile-focused AR Cloud platforms like 6D.ai and Niantic Real World. This persistence enables users to collaborate on creations, share them with others, and discover drawings left behind by others.

"The app is social from the ground up, though it may not be obvious at first. You share the sky with everyone around you so your drawings show up in real-time to other users," said Wolf. "But once there's a good AR Cloud option, we want to 'bring the drawings down' and let you draw on buildings, and so on, so it may be an AR cloud but not that kind of cloud!"

The machine-learning and sensor shortcuts to occlusion, persistence, and multi-user functionality lend the app some inconsistencies. For instance, the app may confuse a white building with the sky. Also, tracking can be inconsistent, as content can drift from its original location.

Nonetheless, the app is an impressive feat that takes a creative approach to the same issues that other AR toolkits and AR Cloud platforms are attacking with more technically advanced solutions.

Cover image via Kevaid

You already know how to use your phone. With Gadget Hacks' newsletter, we'll show you how to master it. Each week, we explore features, hidden tools, and advanced settings that give you more control over iOS and Android than most users even know exists.

Sign up for Gadget Hacks Weekly and start unlocking your phone's full potential.

Related Articles

Comments

No Comments Exist

Be the first, drop a comment!