How Amazon is revolutionizing physical retail with computer vision and AI

My name is Sean (Xiang) Ma, and I work on physical retail technologies at Amazon that enable our customers to shop better.

I’m also currently the Head of Research for Amazon Dash Cart, which is a new product that we’ve just introduced. I’m excited to talk to you about what we’ve been doing and how computer vision and AI have been changing our customers' shopping experience.

I’ve seen the emergence of lots of new applications in recent years. The advancement of transformer neural networks has greatly improved real-world applications, and that's very exciting.

Ranging from autonomous driving, retail, online, and offline, everything is being powered by computer vision. And inside Amazon, we actually have a lot of computer vision scientists that work on various problems to improve the experience of our customers.

To give you some examples, a lot of you may use Prime Video. There's a lot of computer vision in understanding content and recommending the best content to customers.

Amazon’s Alexa has also started integrating computer vision to understand how people speak; in certain environments where we have many people talking, how else do we know who's whom?

In terms of robotics, Amazon's driven by a lot of efficiencies to deliver packages to customers. And then there's also a lot of robotics running in the warehouses.

Today's talk will look at opportunities and challenges that customers are facing in physical retail. At Amazon, we always work backward from our customers’ needs, and that's how we’ve continued to be a successful company.

We also have a bunch of solutions, and I'm going to cover a few of them today. These include Amazon Dash Cart, Just Walk Out/Amazon Go, and Amazon One.

Top challenges customers face in physical retail

Amazon did a lot of customer research to understand customers’ needs and challenges.

Number one is the long checkout line. This is almost the most painful part of going to a grocery store. You want to check out fast and then you see a long line. There's no way you can skip that line, so you have to wait. And time is precious.

The second thing, I think this is especially important due to the pandemic and inflation, is that many of our customers say that they have no knowledge of how much they’ve spent. The price of products is increasing, so you end up with a really long receipt and the total amount is too much or your credit card get declined. We actually see many cases like that.

Some other challenges that are customer-facing are when you go to the store and want to buy things but they're out of stock. So how do you solve that problem? You spend a good hour or half an hour getting to the store and you’ve found nothing you can buy.

And then lastly is not being able to find the product you want. It can be challenging to find what you want when there are just so many choices, and you spend most of the time in the store looking for where the product is.

Independent research shows that innovation for consumers, especially in physical retail, is a must-have. There are some independent surveys that ask customers what their pain points are.

Number one is the long queue for payment at the checkout. The second one is difficulty locating products. Then there's when products are out of stock, and lack of product information. When you're trying to buy things, you don't know whether there's anything that's comparable or whether there's a deal for that. You want to buy the best product and also save money.

The last pain point is not being able to find an associate to help. You're in a situation where you want to get some help, and nobody's there to assist you.

These are the problems that we're trying to deal with.

How Amazon’s solutions solve retail pain points

Amazon Dash Cart

In traditional retail, we have a long checkout line, but our Dash Cart is designed so you can just walk out with the cart. You don't have to stay in the line; there's a special line for Dash Cart and you don't have to see anybody. You can just go.

Often, people have no knowledge of how much they’re spending while shopping until they go to the checkout. We have a real-time spending preview on the screen of the cart so you can see exactly how much you’re spending. If you feel you’ve spent too much you can just pull products out.

We also have a real-time location system in our cart that can tell you where the product is you’re looking for and how to find it.

For lack of product information, we have rich, real-time product information and recommendations on the screen. And not only can you see lots of information about the product, but you can also see similar products nearby that you can choose from.

If you need an associate to help, our cart is actually a smart shopping system that's integrated with store associate tools. So whenever you need help, you just click a button and the associate will know where you are because we know where the cart is, and they'll come over to help.

And, if you don't have any issues, you can just go. It’s easy and you don't have to interact with anyone. It's your own shopping experience.

In addition, we also provide coupons, deals, and shopping lists which can help you a lot with checkout.

The first generation of the Dash Cart was launched back in 2020, and it's currently running in a chain of Amazon Fresh stores.

We have a bunch of sensors and cameras embedded into the system. We have a vision system that can detect what products people buy and whether they buy or return them in real-time and get customer feedback.

We have a touchscreen so the customer can see the information and interact with the system. We also have real-time spending updates where you can see how much you’ve spent.

We also have the in-cart scale. This is very unique. You can put in an item that’s sold by weight, and it can weigh it. So you can directly purchase bananas, apples, and all the things that are challenging to buy in the traditional way.

The carts have very easy push wheels. We actually put a lot of effort into making sure the cart is easy to push. We also have a removable battery which helps the associates who are maintaining the carts, and helps the customers continue to have a good experience.

And, lastly, we have a real-time location system that can tell you in real-time where the cart is, and based on that, we can also build out knowledge of where the product is.

We have the first generation which has already been in the field for two years. It’s been pretty successful and customers love it. If you go to YouTube, you can see how many customers are saying how nice and easy it is to shop with it.

But at Amazon, we never stop improving our products. Just this year, we launched our second-generation Dash Cart which has many more features than the first generation. And we first launched this product with our partner, Wholefoods.

What are the similarities and differences between the two? We’ve kept the real-time spending because that's what the customer wants. We’ve also kept the real-time location system because the customer wants to know where the product is and where the cart is.

We’ve added an anti-glare touchscreen so that you can see things more clearly. We’ve also implemented improvements to the robustness of the cart. People will sometimes smash the shopping cart into structures in the store, so we want to make sure the cart is durable.

We’ve done a lot of measurements and designed the cart to ensure it’s weatherproof. We’ve actually tested it and it’s IP 55 waterproof. It can also stand minus 40 to 40 degrees Celsius in outdoor environments, because sometimes a cart will be pushed out and then left in the parking lot.

The cart is also shockproof, which means that if you drop it or it collides with something, the cart is still safe.

We have two scales now because we want to help the customer with the bulky items at the bottom. So now you can actually put more bulky items in the bottom and also increase the usability of the cart.

In addition, we have branding customization and the screen is also customizable for customers.

Here's the customer journey. The customer will log in with their phone, and then they can start scanning items in the shop. They can weight items while they shop, the cart will tell them how much the items are, and you can see the receipt in real-time. And when you're done, you just skip the line and push it out.

Just Walk Out/Amazon Go

The second product we’ve developed is actually a little earlier than the Dash Cart technology, and it’s called ‘Just Walk Out.’

You might be wondering what the difference is between Just Walk Out and Amazon Go. Amazon Go started as a project, then we launched the Amazon Go product in the Amazon Go store back in 2018. And since then, we’ve been learning a lot about how we can build a store that’s good enough for our customers.

Just Walk Out is basically a new offering. Not only can it serve the needs of Amazon but it can also serve the needs of those who want to implement the system into their own store and make it an autonomous checkout store.

That’s the difference. Just Walk Out is the technology that enables our customers, and Amazon Go is an owned store that’s empowered by the Just Walk Out technology.

What's the experience for our customers? We’ve put a lot of computer vision, sensor fusion, and deep learning technologies into this whole system.

If you ever go to the Amazon Go store or a Just Walk Out power store, you’d come to the entrance, scan your phone or your palm, and then you’d go inside the store and grab whatever you want. You’d put the items in the cart, your bag, or carry them in your hands, and then you’d just walk out. It’s as easy as that.

So, how does it work?

We put a lot of sensors and cameras into the system, and we also do real-time understanding of what customers have bought. Have they returned anything? Who's buying what? This seems to be a simple problem, but it's actually very hard in reality. There could be people shopping as a family, and they might hand things over to different people.

We have a lot of challenges there. But we've been making really good progress in maintaining high accuracy and keeping the system up to date. And that's what's important for our customers.

What did we do behind the scenes? There's a bunch of technologies from computer vision and AI that's powering this. We're using sensors and cameras, of course. We need to do continuous calibration of the sensor cameras to make sure they're all working as expected.

And then we have person detection, which can identify who’s buying the product. We also have object recognition, which will understand what the customer bought.

We also have post-estimation, because sometimes you buy a product and return it back to the shelf. We have analytics, which is actually something that we offer to the stores to better optimize how their products are placed.

Lastly, everybody knows that sensor fusion is the solution for increasing the robustness of the system.

Amazon One

Last but not least, this is another product that's powered by computer vision. It's called Amazon One. We developed this one the most recently, and it's available for all customers to use.

So, what is Amazon One? It's a device that can identify the customer. So if a customer comes into whatever place they want to pay and they want to identify themselves, they don't have to carry a wallet or credit card.

To go into more detail, this is a safe, easy, and convenient way of entering or exiting a store and confirming your identity. You don't have to present your ID. It's also integrated as a payment system.

You pay with your palm, and no two palms are alike, so no one else is able to pay using your details. Also, it's contactless. I think this is very important, especially since the pandemic, as people don't want to touch anything. But this isn’t driven by COVID. It's just because it’s so easy.

So, how does it work? There's some visualization. The palm actually consists of very tiny, unique features below the surface, and those are actually the features that we use to detect. Our technology processes the features within a few seconds, and the computer vision algorithm captures and encrypts the images. Encryption is important. We don't want anybody to steal the images.

Then once we have that, the customer can associate it with their Amazon account or any other account. Then the signature captured will be the way of identifying the customer.