How does the Amazon Store know what products and offers to display to a customer? Part of the answer involves reinforcement learning, a type of machine learning where an agent gradually learns a set of policies that will enable it to maximize some reward. Lihong Li, senior principal applied scientist at Amazon Ads, is developing reinforcement learning techniques to improve outcomes for customers.
Li’s job has two aspects. The primary one is to develop scientific solutions that benefit shoppers and advertisers in the Amazon Store. But Li also considers it part of his role to help build the science community, both within and beyond his organization.
About six months after joining Amazon in the fall of 2020, he helped establish an early career scientist program in Amazon Ads. The new initiative draws a cohort of recent PhD graduates for full-time, two-year positions, working across areas such as machine learning, causal modeling, and game theory. Li was also a senior area chair in reinforcement learning for the 2021 conference on Neural Information Processing Systems (NeurIPS).
Programming for customer success
It’s relatively easy to maximize impact from advertising in, say, one visit to the Amazon Store. But Li’s work aims for positive experiences that will keep customers coming back. That involves a series of decisions over time about what to display on a web page for a given query or product.
The science question we are looking at is ‘How do we optimize decisions to improve our customers’ experience over a long range of time?’
“The science question we are looking at is ‘How do we optimize decisions to improve our customers’ experience over a long range of time?’” he explained.
Reinforcement learning departs from other types of machine learning that focus solely on predictions. An email client might predict whether a particular message is spam or not, for example, or a medical program could assign a probability to whether an MRI image relates to a particular diagnosis. But predictions alone are not enough to make decisions that then change the system.
“We need to incorporate the downstream utility with these predictions and make a decision that will optimize the utility,” Li said. He pointed to conversational systems like Amazon Alexa.
“We don’t just predict, ‘How will the customer respond at a given point of the conversation?’” Li said of the Alexa scenario. “We need to decide an actual response to engage with and assist the customer. Then the outcome of the response is fed to the learning algorithm.”
Lihong Li: Off-policy estimation in long-horizon reinforcement learning
The main challenges involved with reinforcement learning, Li said, are complexity and risk. The task of designing an algorithm becomes much more complicated when it must update itself based on decisions and their outcomes, rather than making a prediction that will not actually change the system. That’s the complexity part. Then there’s the risk.
“We make great efforts to ensure delightful customer experiences when the system takes actions autonomously,” Li said. Engineers and scientists validate a new algorithm extensively offline before it goes into production. One of the tools is off-policy reinforcement learning that uses historical data to predict future online performance.
Pairing the academic with the practical
Li got into computer science while in high school in Guangzhou, China, where he grew up. “It was mostly an academic interest,” he said. “I was fascinated by the potential of what computers could do.” He was also inspired by his older brother, Lipeng, who also is a computer scientist and encouraged Li to pursue the field based on his aptitude for math and science.
From high school, he went on to get a bachelor’s degree in computer science at Tsinghua University in Beijing, and continued to pursue his passion by obtaining a master’s degree in computing science at the University of Alberta, and a PhD in computer science at Rutgers University.
While he was in graduate school, Li had teaching assistant roles in academia and internships with industry. He said he doesn’t remember making a decision to choose either academia or industry — his career evolved naturally over time. After he earned his doctorate, he spent the next decade working as a research scientist for Yahoo!, Microsoft, and most recently Google.
I remain in industry because I think that is a great place to be for a scientist. The boundary between academia and industry is getting blurred, and wonderful results have come from industry-academia collaboration.
Li enjoys the challenges, and the opportunities to develop solutions that research within industry provides, as well as advantages such as having direct access to data and computational resources. But he notes deciding between academic and industry research is no longer an either-or decision.
“I remain in industry because I think that is a great place to be for a scientist,” he said. “At the same time, the industry is getting more and more open to collaborating with academia. The boundary between academia and industry is getting blurred, and wonderful results have come from industry-academia collaboration.” (Editor’s note: Amazon hires academics as Amazon Scholars and Visiting Academics to work on large-scale technical challenges, while continuing to teach and conduct research at their universities.)
Earlier this year, Li and colleague Yi Liu, a senior applied scientist, presented a workshop paper at KDD 2021 on bandits, a subclass of reinforcement learning problems. The paper, “A map of bandits for e-commerce” emerged from the fact that the community around bandits has exploded in recent years. The profusion of bandit algorithms and potential applications raises a practical problem for people in industry, Li said: “How do I know which algorithm from this huge universe of algorithms I should choose for the application at hand?”
The paper, he said, offers a first step toward closing that gap by mapping those algorithms for e-commerce problems. In addition to NeurIPS, he also served as senior area chair for two other leading artificial intelligence conferences, the International Conference on Machine Learning (ICML) and the International Conference on Learning Representations (ICLR).
Joining Amazon
Prior to Amazon, Li’s previous roles focused primarily on algorithmic research within distinct research organizations. Amazon’s more customer-centric, integrated approach to research and engineering appealed to him.
“I am very impressed with how Amazon organizes its science and engineering efforts,” he said. “Being a part of the product team makes collaboration much easier and also allows a scientist to understand the business problems deeply.” He added that the huge number of technical opportunities and talented people working at Amazon were also draws for him.
The types of scientists who succeed at Amazon, according to Li, are customer-obsessed and willing to dive deep on practical problems. “It’s a balance of being both inventive and pragmatic,” he said. “We want it to work well in practice, scale, and have a positive impact for customers.”
