A new way of shopping

💪 Wirecard x YND: Going Strong

When we were asked by Wirecard to look into the space of next generation shopping experiences, we discovered we had to tackle many issues. For example, how much friction can you really remove from the shopping process? Can we make it feel more natural, yet faster at the same time? And of course, will customers still enjoy it? As an answer to these questions, we developed a concept of a smart vending machine that identifies customers using facial recognition technology. In this blogpost we’ll share our findings & answers to those questions.

What The Butik is All About

The concept of the Butik is based on the biometric payment technology YND developed together with Wirecard, creating an “invisible” checkout experience. This technology can easily be deployed in different environments and use cases. Using computer vision the Butik knows what products are taken out or put back by the customer and automatically processes payment for the goods.

How it Works

The technology behind the Butik has all the “smarts” on the built-in hardware. The self contained unit does all computing for facial and product recognition . In the current setup, the Butik doesn’t require an additional client application, as the customers are pre-registered. The shop application running on the Butik is designed to make the technology accessible for users at demos and trade shows. It simulates the payment process triggered once the user shut the doors. The system is designed to allow the integration with any payment stack and client application depending on the use case.

We built a minimal version equipped with one face recognition camera, one product recognition camera, four smart weight sensors and electronic locks to control access.

The current version of Butik shows the possibilities of a fully automated, computer vision-aided shopping experience. It proves that it’s possible to achieve an accurate and tamper-proof solution by using video cameras and a set of smart weighing scales.

During the development we were able to reuse the majority of the facial recognition framework and presence detection technology from the Smartbar project, so customer identification was mostly a matter of interconnecting existing components with the electromagnetic locks securing access.

Product detection was a relatively new topic for us so naturally this is where we spent the majority of our Machine Learning efforts. We built a solution using Computer Vision, augmented by other sensors (including a precise, 4 tensometer-based weighing scale) for validating results. When the customer takes a product from the Butik, it’s detected by our video-powered “brain”, which double-checks it with the input from the scale (to validate corresponding the weight difference).

This allows us to ensure the system can handle certain tampering attempts such as:

• Taking a product and returning back just the packaging (e.g. putting back an empty can)
• Detecting “external” items which don’t belong to the inventory in Butik
• Detecting when products are hidden or covered behind each other

In our showcase, the simulated payment is only processed after the customer has closed the doors, and there are multiple actions that can change the status of the virtual basket in the meantime. We’re tracking all changes in the inventory and display the current status of the basket on an embedded OLED display. This way, the customer is always aware which goods they selected and how much they’ll end up paying. To prevent the Butik from unauthorised access during a spoofing attack, we are using the anti-spoofing solution we developed recently. This uses DNN to identify if a face is real or fake. After hours and hours of testing the Butik at our office, we deployed it offsite for the first time at the Spielfeld Digital Hub. For this partnership with Wirecard & Visa Innovation Labs, we selected health & wellness products for the Butik to validate a concept of enabling retail at places such as gyms, fitness clubs, spas etc. This use case allows customers to easily pick & pay simple items such as towels and massage rollers. Having a separate counter shop with dedicated staff is not always possible and this is where Butik shines: by providing a simple shopping experience in places where retail wasn’t possible before.

What’s in Store for Butik?

The first showcase of the Butik 1.0 is currently deployed at the Visa Innovation Hub at Berlin Spielfeld. It presents the Butik as a vending machine to be deployed in lifestyle settings such as gyms or spas. For future updates, we’ll look into adding additional cameras (front and sideway), cooling feature, analytics and much more. The Butik was also deployed at the Wirecard offices in Munich. The Wirecard team will use it to showcase biometric payment at several trade shows over the year.

This post was written by Michał Dominik Kostrzewa, Head Of Special Projects. In need of some brain power? Reach out to us via hello@ynd.co with your questions about ML/AI projects.