On Dec. 1, 2017, Amazon detected several food safety concerns on a dietary supplement. They suppressed sales the same day and initiated an investigation with the manufacturer—and the product was not reinstated.

On Jan. 26, almost two months later, the product was officially recalled due to undisclosed allergens—and the side effects listed in the official recall matched Amazon’s detected feedback almost exactly.

It was just one example of how the company is using technology to deal with food safety issues that Carletta Ooton, V.P. of health, sustainability, security, and compliance, at Amazon, shared during this year’s Food Safety Summit Keynote, sponsored by eurofins.

Although Ooton didn’t talk about the company’s recent purchase of Whole Foods, she did touch on things like Amazon Fresh and Amazon Go, and the challenges the company faces when it comes to food safety and quality issues.

“We design safety into every process touch and then we audit to ensure that the process is maintained,” she says. “In the rare occasion when a [Key Performance Indicator] and a customer anecdote disagree, the customer is nearly always right. These customer signals are a rare gift we obsessively seek.”

Monitoring customer feedback for food safety issues

In fact, through a tool called Heartbeat, Amazon aggregates more than 16 million pieces of customer feedback per week, in more than 40 languages.

“These interactions include publicly available data like product reviews and detail page ‘ask an owner’ correspondence as well as internal data like customer return comments, CS chat, and machine-transcribed CS calls,” Ooton says. “Increasingly, we are also contacted through social media mentions on the over 200 Amazon accounts. The scale alone is not the issue. Our challenge is extracting relevant and actionable meaning from the interactions.”

And Amazon has developed computer learning models that help it scan customer feedback for any food safety issues.

“To do this, we rely primarily on Natural Language Processing,” Ooton says. “NLP is a broad field of computer science that focuses on applying machine learning to understanding language. One of the key differences between NLP and general keyword searching is that understanding of context, sentiment, and sentence structure.”

She shared two examples of customer complaints and how the company responded to them.

The first read, “Gave me energy for my workouts, but also wreaked havoc on my digestive tract. That was only taking 1 scoop rather than the recommended 2 scoops. I can’t imagine how sick the full serving would have made me. I had to stop taking it after 3 days.”

The system said the comment had an 87.4 percent chance of being a food safety issue, and the risk was significant enough that Amazon immediately pulled the product. It turned out the analysis was correct.

Ooton says when the judgment has high confidence, they automate the action—otherwise they rely on human processes.

The next example was, “Really excited to get these to replace high sugar energy drinks, but I can’t even describe how disgusting the taste is. I couldn’t even take 2 sips of it before feeling sick. It is a horrible prune juice taste and the after taste just does not go away. Such a bummer that I have 11 left, heading straight to the trash what a waste of $40.”

The system said that example had a 9 percent chance of being a food safety issue, and again, it was correct.

“The last review was certainly not describing a positive customer experience, and it is something other customers can evaluate as an input to their purchasing decision—but it was not a safety concern,” Ooton says.

Dealing with recalls

During the speech, she also talked about what happens when Amazon does decide to do a recall, and considering that the company handles roughly 2,000 recalls per year for both food and non-food products, it’s not a minor issue.

First, they have automated how they get information about the recalls by regularly scanning 31 regulatory websites in North America and 67 sites worldwide.

Amazon also uses complex syntax like keywords and image matching to query listings throughout regional marketplaces, not only online, but also in physical brick-and-mortar stores. Potentially involved listings are immediately removed from sale, and customer purchase orders that have not yet been fulfilled are canceled.

Like any company, the most important part of a recall for Amazon is getting word to its customers.

Ooton says impacted customers are notified through emails with instructions for return and if relevant, a refund. And past customers of recalled products and customers with canceled orders are notified through targeted email messaging, providing them with instructions and if relevant, a refund.

And they have a high success rate with those emails; specifically, Oooton says, “68.8 percent of the messages we sent were actually opened. That’s phenomenally high.”

Amazon also does Recall Inventory Quarantine, which prevents recalled items from being attached to customer orders. A virtual recall label also is added to every item belonging to a recalled product. And software tools also prevent physical inventory from moving, incoming purchase orders are blocked, and restock/stowing/trans shipments are blocked.

Amazon also removes inventory from the supply chain by returning or destroying products. However, when impacted items can easily be distinguished from unaffected inventory, only defective products are removed.

Then, the company has to decide when to reinstate a recalled product.

“Reinstatement only occurs when our FC network has been flushed of potentially impacted inventory and when the product supplier provides written confirmation the defect has been rectified and all future products are compliant,” Ooton says.

Of course, all of these methods are constantly evolving to keep up with the ever-changing consumer base.

“We started out as a simple eCommerce seller of books. We now have over half a million employees, over 300 million customers all over the world, over 100 million Prime customers,” Ooton says. “We have to invent every day.”