While we've seen Snapchat apply sky segmentation to AR content, the makers of the Blue Sky Paint app have applied similar capabilities to create and share airborne art.
Published by AR developer Kevaid and available now on the App Store, Blue Sky Paint is similar to Similar to Google's Just a Line app, where users press the screen and move their device around to create lines in their immediate surroundings.
However, using a combination of existing features of iPhones and ARKit along with some machine learning to identify the sky, Blue Sky Paint draws only on the sky, with lines disappearing behind trees, buildings, and other visual obstructions.
According to lead creator Aidan Wolf, the app uses ARKit's camera feed to capture the environment and a machine learning algorithm to determine what is sky and what is not. For anything that is not the sky, the app applies graphic shaders as a mask, hiding the user input from view and giving the illusion of occlusion. Also, the app uses GPS to determine the user's location and the built-in compass of the iPhone for directional orientation.
"Basically, really general things that are true about the world most of the time: the sky is always up so if your phone is pointed up its likely looking at the sky, the sky is generally blue (even the clouds) while the ground is usually lit directly by the sun giving it a yellow tint," Wolf told Next Reality. "GPS and compass orientation are what's used, but we do have a re-projection algorithm which redraws the drawing to match different perspectives and distances. The fun thing about using the sky is we can get away with a lot of GPS distortion since the sky is 2.5 km off the ground."
The developers also use the orientation as part of the user interface. Looking down in the app brings up the color palette (another differentiator between the app and Just a Line).
The dataset for placing AR content also enables the app to achieve multi-user experiences and persistent content out of the box, well ahead of the similar capabilities forthcoming in ARKit 2.0 later this month or other mobile-focused AR Cloud platforms like 6D.ai and Niantic Real World. This persistence enables users to collaborate on creations, share them with others, and discover drawings left behind by others.
"The app is social from the ground up, though it may not be obvious at first. You share the sky with everyone around you so your drawings show up in real-time to other users," said Wolf. "But once there's a good AR Cloud option, we want to 'bring the drawings down' and let you draw on buildings, and so on, so it may be an AR cloud but not that kind of cloud!"
The machine-learning and sensor shortcuts to occlusion, persistence, and multi-user functionality lend the app some inconsistencies. For instance, the app may confuse a white building with the sky. Also, tracking can be inconsistent, as content can drift from its original location.
Nonetheless, the app is an impressive feat that takes a creative approach to the same issues that other AR toolkits and AR Cloud platforms are attacking with more technically advanced solutions.
Be the First to Comment
Share Your Thoughts