Augmented reality has become quite a popular term in the past few years thanks to Google Glass, but the idea is older than the first Android phone.
Do you remember the Terminator movie? The main hero’s vision mapped the nearby area and displayed additional information about objects or people.
And that, in essence, was AR.
Share your challenge with our team and we'll work with you to deliver an outstanding digital product. Create a product with Netguru.
To show you what AR really is, let me refer to the definition of an augmented reality guru - Ronald Azuma. In late 1997 he established the three main elements that define augmented reality:
Basically, augmented reality adds digital components to the real world, often by using the camera on a smartphone to enhance the experience.
It is very important that you do not confuse augmented reality with virtual reality because these technologies are just not the same.
The crucial idea of AR is to overlay digitised information on the real (offline) world e.g. by displaying info about the building you are looking at. VR uses computer-generated reality, experienced usually by wearing a headset. Sounds great, doesn’t it?
In the past this seemed like science fiction... And actually it was!
Back then you were only able to check out the possibilities of AR on special devices like the Virtual Fixture. It was heavy, uncomfortable, and had very limited features. Nowadays, AR devices are being pushed out by smartphones, which make the augmented reality experience accessible. Let’s dive deeper and see what are the types of AR with some hot use cases.
Marker-based AR is, in my opinion, the most powerful technology. It uses image-recognition algorithms to specify the position and rotation of markers. After that it displays, for example, a 3D object in the specified place.
You may ask: what can be the marker? Well, at the beginning image recognition wasn’t well developed and the marker was just a QR code. But currently there are tools that can recognize almost everything - from text to human face. Let’s see some cool examples of marker-based AR solutions.
Face filters are one of the most popular cases where AR is used. The success story of face filters began in 2015 when Snapchat introduced them in their app. People have gone crazy about them and started to heavily use them.
But how is it actually possible to show e.g. dog’s ears or tongue on a human head? The answer is face recognition algorithms and some 3D magic. Recognizing human faces is not a trivial feature, but nowadays there are tools that allow developers to create their own face filters, for example the Firebase ML Kit.
With Firebase it is possible to detect the positions of eyes, mouth, nose, and contours. Such data can then be used to place a 3D mesh with proper graphics over the camera image. Wouldn’t it be great to have your own filters in your app?
Source: Bless This Stuff
Have you ever wanted to have a tattoo, but you weren't sure if this or that one would look good on you?
Augmented Reality can help you with this. Inkhunter is another example of marker-based AR which places a virtual tattoo on your body. The first step is to use Inkhunter to draw a smile on your hand or wherever you want to have a tattoo. The smile is used as a marker in this app. Now it’s time to select a piece of art and point your smartphone’s camera at the smile you just drew.
The previous type of AR used image-recognition to place 3D objects in the real world. Image from the camera is processed to fetch information about the position and orientation of the marker.
Markerless AR is less complicated in terms of algorithms used, but more complicated when it comes to hardware sensors. It uses sensors to learn the position and orientation of the device. What sensors, you may ask? There are 3 sensors used in this type of AR:
Thanks to them it’s possible to calculate the exact rotation of each axis in 3D space. In many cases it’s also necessary to use GPS data to place 3D objects at some distance. Let’s see some examples of such apps.
IKEA Place is a very useful app for all the people who want to buy new furniture or to arrange their home from scratch. The main feature of the app is the ability to see if a given product fits your interior. Ikea claims that their products are displayed in the app with 98% size accuracy. Thanks to ARCore and Sceneform the experience is smooth and products are shown with really good details.
Source: The Verge
We all know Google Maps, they are great, but I can bet that many of you have been in a situation when you started the navigation in the middle of nowhere and it said “Head north”, so you started walking in one direction and, if you weren’t lucky enough, you had to walk back as you didn’t choose wisely.
In 2018, Google presented a new concept of a Google Maps feature. It uses maps and street view along with device sensors to show routes to the user. What is more, it can display additional info about places nearby and guide you to them.
At the beginning of July 2016 the Pokemon Go has been released. Fans all over the world started to catch’em all, causing big concerns about the safety of augmented reality users. Pokemon GO is an example of location-based AR app. It means that despite device rotation sensor it uses also GPS data to display Pokemons in proper position. The hype for Pokemon GO is history, but the technology is still hot.
Each mobile application needs a server that will manage the users as well as the data flow. In our case, we need somewhere to store information about our Pokemon objects.
At the beginning the following should be enough:
When you develop an MVP and want immediate prototypes, it’s great to choose cloud solutions instead of developing them on your own. I would recommend giving Firebase a try. It’s a really powerful NoSQL database that can be easily integrated with the Google Cloud Platform and, for example, push notifications. Moreover, Firebase has its own SDK for Android and it’s really easy to start using it.
To populate your database, just use the Pokemon API database with detailed information that can be used for future improvements such as evolution. Maybe you fancy starting up a Water Pokemon for lakes and rivers?
An algorithm for displaying augmented information using the camera view.
You can kick off with the one described in the blog post, but it requires some fine-tuning:
One of the key features of Pokemon Go is that you actually need to walk around to catch Pokemon. To pull this off, we can use Google Maps with the first-person view — really simple to set up in your app. Thanks to markers you can place the nearby Pokemons or Pokestops on your map. You can also set a custom location icon with your avatar.
The most interesting part of the UI/UX in Pokemon Go is the Pokeball throwing animation.
As you may well expect, there are as many ways to implement this as there are developers and everything depends on how accurate and sophisticated you want it to end up.
As a starting point, you might want to check out ObjectAnimator from the SDK. This tool allows you to animate a bitmap of a Pokeball. After some user interaction that involves hitting the Pokemon, just play the pre-programmed animation of catching it.
Remember! This Is Just the Beginning!
So, we went through the basics of augmented reality. You know what are the types of augmented reality and learn about some examples and how they work.
But is this technology mature and devoid of flaws? It does have a few, but not many.
Did you ever think about how the implementation of a location based augmented reality app looks in the code? You’ll have a chance to see how to build simple app that displays some objects in the real world. But let’s start with a bit of theory.
*Geodesy theory is based on the book: A. Jagielski, Geodezja I, GEODPIS , 2005.
Wikipedia explains the azimuthal angle as:
“The azimuth is the angle formed between a reference direction (North) and a line from the observer to a point of interest projected on the same plane as the reference direction orthogonal to the zenith”
What we will do is attempt to recognize the destination point by comparing the azimuth calculated from the basic properties of a right-angle triangle and the actual azimuth the device is pointing to.
Let’s list what we need to achieve this:
Now the question is how to calculate the azimuth. It turns out to be quite trivial because we will ignore the Earth’s curvature and treat it as a flat surface:
As you can see, we have a simple right-angle triangle and we can calculate the angle ᵠ between points A and B from the simple equation:
The table below presents the relationship between the angle in grads and the azimuth A(AB):
In this tutorial, I will not describe how to get the location and azimuth orientation of the device because this is very well documented and there are a lot of tutorials online. Mainly for reference, please check Sensors Overview (especially TYPEROTATION_VECTOR) and Location Services.
Once you’ve prepared data from the sensors, it’s time to implement CameraViewActivity. The first and most important thing is to implement SurfaceHolder.Callback to cast an image from the camera to our layout. SurfaceHolder.Callback implements the three methods responsible for this: surfaceChanged() surfaceCreated() surfaceDestroyed().
The next step is to create an XML layout file. This is quite simple:
Once we have the whole presentation, we need to connect it with sensor data providers. In this case, this just means implementing and initialising the proper listeners from OnLocationChangedListener and OnAzimuthChangedListener.
When we’ve gotten the current location, we can calculate the theoretical azimuth based on the theory presented earlier.
When the orientation of the phone changes, we need to calculate accuracy, compare both angles and, if the current angle is within accuracy, we can show a pointer on the screen. Of course this approach is quite trivial but it illustrates the idea.
A method that implements calculation of the azimuth between points might look like the following:
To compare the calculated and theoretical azimuth we will use the isBetween() and calculateAzimuthAccuracy() methods:
Of course, this tutorial is quite basic. My goal was to show you that it doesn’t take much to make a simple AR app. I would like to point out some things that should be considered during further implementation:
And that’s all you need to know to start your Augmented Reality journey. I really encourage you to use your imagination and creativity to deliver innovative apps.
The potential is huge and it’s our job as developers to make it happen! Good luck!
This article was first published on Nov 4, 2017