Home

Bizarre AR Experiment Serves Tweets for Everything Your iPhone Can See

Oct 26, 2017 07:15 PM

A development duo has concocted an iPhone app that displays related tweets based on objects recognized by the device's camera.

Or Fleisher and Anastasis Germanidis compiled the experiment, dubbed Twit.AR, using Apple's CoreML machine learning platform and Google's Inception v3 model to classify objects, Twitter APIs and a filtered Swift library to retrieve relevant tweets, and ARKit to display the tweets on the device's camera view.

Or Fleisher/YouTube

"Initially the app was developed as a re-imagination of what Twitter could feel like if it invaded our physical space," said Fleisher in an email to Next Reality.

During the development, we realized that since the machine learning model sometime predicted results that perhaps are linguistically connected to the object you are looking at but have varying levels of context match, it could also serve as a satirical way of interpreting the reality that surrounds us.
— Or Fleisher

The results, chronicled in a demo video, are at times fascinating, perplexing, and horrifying. Filmed from a first-person perspective on an iPad, the video simulates how the app would operate through a pair of smartglasses constantly scanning the environment. (Clearly, walking around with an iPad in front of your face isn't a realistic proposition. But, then again, smartglasses aren't exactly mainstream at the moment, either.)

Twit.AR delivered a few glimpses at potentially practical future utilities for the AR dynamic. For example, a pair of sandals returned a fashion-related tweet. A coffeepot revealed a tweet of a famous still-life portrait on the subject.

Or Fleisher/YouTube

Other results were more bizarre, and demonstrated that the machine learning in this case requires more training. When the app interpreted fencing as jail bars, it displayed a quote about prison life. Upon identifying a pile of cardboard boxes, a tweet with the promise of NSFW content appeared thanks to an unfortunate typo. Thus is life on Twitter.

We love the idea of adding a data layer to our day to day life. This experiment showed us that even with inaccurate results every now and then, we get some interesting and inspiring information. It's like having a conversation with a digital entity that lives with you wherever you go.
— Or Fleisher

Interesting, but not very accurate this time.

Does the world really need the firehose of Twitter in their face constantly? Certainly not. Could it be useful in small doses? Sure, I could see how pulling up information from Twitter as needed could be helpful, or at least entertaining.

"While this experiment shows what could be, it by no means suggests that this is the way we think Twitter or any other company should use the new 'digital real estate,' [it's] more meant to portray a re-imagination of one," said Fleisher.

I think as we see more and more augmented and mixed reality practical uses in our day-to-day life, big questions will arise such as what is the meaning of privacy policies when dealing with [an] augmented version of reality? How should notifications be implemented? Silent mode? These are all relevant and interesting and are yet to be answered, but will be established as more and more apps become available.
— Or Fleisher

The pair has also begun working on modifications to Twit.AR, including improving stability and accuracy, and layering data from other platforms to provide more relevant information. They are also interested in pursuing this and other ideas via the webAR API. Similar to webVR, the platform enables AR experiences through mobile browsers without requiring a download of a dedicated app.

"Platforms like web browsers that support mixed reality content will allow bigger audiences to experience it, while not having to download a special application," said Fleisher. "A similar process could be observed in the development and implementation of the webVR API."