How to Develop Augmented Reality in Mobile App

The mobile augmented reality (AR) market was valued at around US$ 12.45 billion in 2021. This addressable market is expected to exceed US$ 36 billion by 2026. The opportunities for mobile AR are massive, with smartphones, tablets, and other mobile devices all included in this category.

This article presents the commercial opportunities of augmented reality mobile apps and an overview of how to build them. This is particularly relevant for business owners and compa ny decision makers considering AR for their mobile channels.

Augmented reality in mobile app development

Augmented reality is a technology that integrates virtual content with the real-world environment. This gives users a mobile experience wherein virtual objects exist alongside them in the physical world captured by their device’s camera. Basically, augmented reality layers digital elements to the user's actual surrounding environment to elevate the mobile experience.

The term AR is sometimes interchangeably used with virtual reality (VR), but these two technologies are different. VR produces computer-generated content that users experience through a headset without the visual layer of the real world.

To better understand how augmented reality typically works in mobile applications, outlined below are some of the most common types of mobile AR. These can help you conceptualize how you want to deliver a unique mobile experience to your customers.

Marker-based AR

Mobile apps with marker-based AR detects objects from real-world surroundings and superimposes virtual objects on them. The smartphone reads the visual input from the camera and then determines how virtual content are placed (i.e. marking).

When the camera loses track of the marker (e.g. when the user moves the phone around), the virtual object is lost; when the marker comes back to camera view, the virtual objects reappear. Face filters are the most popular use case for marker-based AR. Apps detect the positions of a user’s eyes, nose, mouth, and other facial markers to determine how to position the facial filters.

Location-based AR

Markers are not required for location-based AR apps. They employ GPS and other positional inputs to place virtual objects. The camera scans the physical environment and displays objects as a result of it being programmed to be in specific locations. Pokémon Go is the most prominent example of a location-based augmented reality app.

Markerless AR

Aside from location-based AR, there are other ways to position virtual objects without the use of markers. Markerless augmented reality uses multiple sensors to detect the position and orientation of a mobile device. Here are a few types of these sensors:

• Accelerometer: The sensor measures the acceleration applied to the mobile device.
• Gyroscope: It measures the angular speed around all axes in 3D space.
• Magnetometer: It measures the ambient magnetic field in all axes.

The type of augmented reality you want to apply in your product concept will affect the technology stack for the mobile app and the expertise of the development team. It’s also possible that for more advanced AR apps, you’ll need to combine both marker-based and markerless features.

How to build augmented reality mobile apps

Creating an AR application begins much like developing any other mobile app: specify your business objectives, define user personas, research your market and competitors, identify key features, and so on.

When pinning down what you want to do with your augmented reality offering, it seems that the key lies in aligning it with your app’s business goals. It’s one thing to make your AR feature cool, fun, and interactive; it’s another thing to make it work towards tangible business results.

When it comes to the specific functionalities of the actual AR feature, consult with your technology team (whether an in-house team or from a consultancy), if your product vision is best served as marker-based AR, location-based AR, other types of markerless AR, or a combination of these.

It’s their job to advise you on the most optimal technology approaches, but it’s your job to ensure that it aligns with commercial and strategic objectives. For example, the AR feature should serve a specific phase of your marketing or sales funnel.

Maximizing hardware-dependent capabilities

Once you've determined the features for your AR mobile app, the next stage entails selecting the right tools that will go into your technology stack. A critical factor for you to consider is how your augmented reality app will be able to maximize the hardware capabilities of your users’ smartphone or tablet.

When building AR mobile applications on either Android or iOS (or both), you need to be able to take full advantage of the device’s camera feature, as well as GPS and movement detection (and maybe even Bluetooth and the microphone), all of which are hardware capabilities commonly needed to make augmented reality possible.

Whether you’re building your product as a native app or a cross-platform app, consider Kotlin Multiplatform Mobile, which is a software development kit (SDK) that simplifies app development by allowing developers to write the same business logic for both iOS and Android.

Bartolomeo - an ecommerce shop built with Kotlin Multiplatform

AR-specific technology stack

Just like any other mobile app, augmented reality applications require a backend framework, a frontend framework, a database, cloud services, and testing tools for quality assurance. You probably already know a lot about these.

The unique requirement for AR apps is that they entail a robust programming architecture on the backend that enables these apps to recognize real-world environments and place virtual objects onto them. This can only be powered by AR-specific tools, most of which (but not all) are neither open-source nor free.

On making technology choices for your AR tech stack, you’ll need to focus on augmented reality software development kits (SDK), which performs the role of merging virtual content with the physical world.

Again, you should rely on your technology team to advise you on which SDK suits best for your product idea, but the ones listed below are the most commonly used.

• Google ARCore: This is a toolkit developed by Google for Android devices, which works with Java/OpenGL, Unity, and Unreal. It enables environmental understanding (e.g. detecting horizontal surfaces to place virtual objects on the floor or on shelves). Among its many AR capabilities, it also powers motion tracking and light estimation (so objects appear natural within a space).
• Vuforia: This is one of the market-leading augmented reality SDKs supporting a wide range of functions, such as text recognition, 2D and 3D object recognition, 3D geometric map construction, conversion of static augmented photos into motion films, cloud and on-device storage, among many others. It also offers its own barcode functionality called VuMarks.
• Apple ARKit: This SDK is exclusive to iOS devices, particularly the iPhone and iPad. The Apple ARKit utilizes a technology called Visual Inertial Odometry to track the environment surrounding the device, which allows it to detect its movement within a room. In addition to ARKit, developers will likely need to use it with a suite of tools from Apple, particularly RealityKit, RoomPlan, Reality Composer, and Reality Converter.

There are also many other credible SDKs you’ll likely hear about from your technology team, such as ARToolKit, Kudan, EasyAR, Wikitude, and MAXST.

In selecting the right SDK for your project, you’ll have to consider a range of factors, especially cost, OS support (Android and/or iOS), accessory support (e.g. smart glasses), cloud recognition, 3D tracking (not all SDKs allow this), Unity support (compatibility with this popular gaming engine), and many others.

Benefits of using AR in mobile apps

Some might be quick to dismiss augmented reality as a fad, a gimmick. However, as I’ll demonstrate below, not only has the benefits for businesses been tangible, but it has been measured. Augmented reality, even while still a novelty, has already proven that it can deliver business results that matter.

AR attracts curiosity and can actually be fun and engaging for customers. According to a survey from NielsenIQ, 56% of shoppers claimed that AR gives them more confidence in the quality of a product, and 61% said they would prefer to buy from retailers that offer an augmented reality experience. In another study, those who used the AR feature spent 20.7% more time on the mobile app and viewed 28% more products.

More importantly, this increased user engagement can translate into increased sales. In a 19-month study that looked at user behavior of more than 160,000 customers for about 800,000 mobile app sessions, customers who used the augmented reality features were 19.8% more likely to make a purchase than those who didn’t, indicating that AR can help businesses raise revenue.

Businesses from a wide range of industries are already deploying augmented reality into their digital platforms. Healthcare providers, for example, use AR to enhance patient experience during treatments.

Real estate developers, brokers, interior decorators, and online marketplaces use augmented reality to elevate the home buying and makeover experience. In gaming, the real world becomes an environment for online characters and avatars.

Potential of augmented reality in ecommerce apps

The specific use case for AR I’d like to particularly highlight is how it has been transforming retail, ecommerce, and the online shopping experience through Virtual Try-On (VTO). Through augmented reality, virtual try-on lets customers virtually sample or “try on” products, such as clothes, personal effects, and makeup, among many others.

With VTO, the AR technology overlays the product onto the customer’s image to see how the clothes would fit or how an accessory would look as if they were wearing them.

Using machine learning and augmented reality to create a real-life nail salon experience

Those who have experienced virtual try-on can vouch that it makes online shopping more fun and engaging, not to mention more efficient as well. It lets customers try on the latest fashion trends or outfits they may not have even considered otherwise. This is due in part to how virtual try-on allows shoppers to switch between products more seamlessly

When used strategically within an online store's sales funnel, VTO can be a fantastic tool to increase sales by moving customers more effectively from interest to an actual purchase.

This AR retail use case can boost store revenue owing to:

• Greater personalization
• Enhanced user engagement
• Location independence
• Brand differentiation

In addition, virtual try-ons have also been seen to reduce costs from fewer item returns and better inventory management. Because of these benefits, there are already a number of popular fashion retailers taking advantage of augmented reality.

For an in-depth look at the advantages and disadvantages of virtual try-on apps, I encourage you to read Netguru’s report, Virtual Try-On: Game Changer or Hype, with insights supported by UX audits of existing VTO apps and user interviews.

Relatedly, as part of a research and development initiative at Netguru, we experimented with AR to showcase furniture displayed in front of different backgrounds.

We used a cross-platform technology called Kotlin Multiplatform Mobile (KMM) to elevate both our native and cross-platform development capabilities. At the same time, we wanted to explore and experiment how augmented reality would function with KMM.

We tested Kotlin Multiplatform to demonstrate it's business case by showcasing high-value features, such as AR of the product in the highest quality possible.

Because online shopping, whether for clothes or furniture, is a highly visual experience, retailers should always consider maximizing camera-related features for their mobile apps.

However, developers know that camera features complicate cross-platform development. Kotlin Multiplatform simplifies this process of writing the same business logic for both iOS and Android, while maximizing hardware features, which augmented reality mobile applications heavily rely on.

Develop a mobile app that your customers will love using

Trends, market figures, and consumer popularity seem to indicate that augmented reality will become a more integral part of our digital experiences, whether as part of our professional or personal lives. This means that the opportunities for businesses are still nascent and will only continue to grow.

If the leap into augmented reality feels intimidating, remember that the technology should follow your business objectives. The AR feature must fit your priorities, whether to acquire new customers, funnel them into a purchase, capture their attention longer, or elevate their experience of your brand.

When you’re ready to make technology choices for your iOS and Android app, whether it already exists out in the market or you’re just about to build them, we suggest that you consider Kotlin Multiplatform Mobile, which can work with any AR toolkit. It enables apps to maximize the use of the device’s camera, location support, and movement detection — hardware capabilities necessary to make augmented reality work.