Header Banner
Next Reality Logo
Next Reality
Augmented & Mixed Reality News, Rumors & Dev Guides
AppleSnapAR BusinessGoogleInstagram | FacebookNFTHoloLensMagic LeapHands-OnSmartphone ARThe Future of ARNext Reality 30AR GlossaryARKit Dev 101What Is AR?Mixed RealityHoloLens Dev 101Augmented RealityHololens How-TosHoloLens v. Magic Leap v. Meta 2VR v. AR v. MR

Bizarre AR Experiment Serves Tweets for Everything Your iPhone Can See

Oct 26, 2017 07:15 PM
Coffee maker with digital display on a kitchen countertop.

A development duo has concocted an iPhone app that displays related tweets based on objects recognized by the device's camera.

Or Fleisher and Anastasis Germanidis compiled the experiment, dubbed Twit.AR, using Apple's CoreML machine learning platform and Google's Inception v3 model to classify objects, Twitter APIs and a filtered Swift library to retrieve relevant tweets, and ARKit to display the tweets on the device's camera view.

"Initially the app was developed as a re-imagination of what Twitter could feel like if it invaded our physical space," said Fleisher in an email to Next Reality.

During the development, we realized that since the machine learning model sometime predicted results that perhaps are linguistically connected to the object you are looking at but have varying levels of context match, it could also serve as a satirical way of interpreting the reality that surrounds us.

— Or Fleisher

The results, chronicled in a demo video, are at times fascinating, perplexing, and horrifying. Filmed from a first-person perspective on an iPad, the video simulates how the app would operate through a pair of smartglasses constantly scanning the environment. (Clearly, walking around with an iPad in front of your face isn't a realistic proposition. But, then again, smartglasses aren't exactly mainstream at the moment, either.)

Twit.AR delivered a few glimpses at potentially practical future utilities for the AR dynamic. For example, a pair of sandals returned a fashion-related tweet. A coffeepot revealed a tweet of a famous still-life portrait on the subject.

Other results were more bizarre, and demonstrated that the machine learning in this case requires more training. When the app interpreted fencing as jail bars, it displayed a quote about prison life. Upon identifying a pile of cardboard boxes, a tweet with the promise of NSFW content appeared thanks to an unfortunate typo. Thus is life on Twitter.

We love the idea of adding a data layer to our day to day life. This experiment showed us that even with inaccurate results every now and then, we get some interesting and inspiring information. It's like having a conversation with a digital entity that lives with you wherever you go.

— Or Fleisher

Interesting, but not very accurate this time.

Does the world really need the firehose of Twitter in their face constantly? Certainly not. Could it be useful in small doses? Sure, I could see how pulling up information from Twitter as needed could be helpful, or at least entertaining.

"While this experiment shows what could be, it by no means suggests that this is the way we think Twitter or any other company should use the new 'digital real estate,' [it's] more meant to portray a re-imagination of one," said Fleisher.

I think as we see more and more augmented and mixed reality practical uses in our day-to-day life, big questions will arise such as what is the meaning of privacy policies when dealing with [an] augmented version of reality? How should notifications be implemented? Silent mode? These are all relevant and interesting and are yet to be answered, but will be established as more and more apps become available.

— Or Fleisher

The pair has also begun working on modifications to Twit.AR, including improving stability and accuracy, and layering data from other platforms to provide more relevant information. They are also interested in pursuing this and other ideas via the webAR API. Similar to webVR, the platform enables AR experiences through mobile browsers without requiring a download of a dedicated app.

"Platforms like web browsers that support mixed reality content will allow bigger audiences to experience it, while not having to download a special application," said Fleisher. "A similar process could be observed in the development and implementation of the webVR API."

Cover image via Or Fleisher/YouTube

Just updated your iPhone? You'll find updated Apple Intelligence capabilities, new wallpapers, and enhancements to Calculator, PDF cropping, and Live Voicemail, among other useful features. Find out what's new and changed on your iPhone with the iOS 18.3 update.

Related Articles

Comments

No Comments Exist

Be the first, drop a comment!