Dr. Andrew Barto has been selected to receive a UMass Neurosciences Lifetime Achievement Award for his pioneering research into reinforcement learning. The award will be presented to him by Chancellor Subbaswamy after his lecture at the UMass Interdisciplinary Neurosciences Conference on May 28th. Dr. Barto is Professor Emeritus and former Chair of Information and Computer Sciences at UMass.  He is perhaps best known for an influential book, which he co-wrote with Richard Sutton, called “Reinforcement Learning” (MIT Press). The book, which is now in its 2nd edition is considered almost a sacred text by neuroscientists studying the neural basis learning as well as engineers and computer scientists who work on artificial intelligence.

The extent to which Dr. Barto is loved and admired by researchers around the world is obvious in the tributes that are pouring in as a result of this announcement. If you would like to add your words to this website praising Dr. Barto’s contribution, send them to ions@umass.edu.

Richard Sutton
Professor, Computing Science
University of Alberta
Andy Barto and I have written many scholarly papers together, and all of what we did that was a contribution to Neuroscience was entirely due to Andy.
As my PhD supervisor, Andy had a strong influence on how I viewed science. He taught me that there was nothing totally new under the sun, that every new twist had its historical antecedents, and that we benefited greatly from acknowledging and understanding them. He taught me to be plain in my writing and speaking, and that an audience responds positively to what they can understand.
It is not too strong to say that Andy created the burgeoning field of modern reinforcement learning. He created it partly through his research and the research of his students, but a field’s success is due to more than just its research results. As I have grown older I have gained a deeper appreciation of Andy’s contribution in setting the tone of the field. That tone is an emphasis on scholarship, on humility, and on openness—on welcoming all fields and all people for whatever contributions they can make. First and foremost, Andy is a really nice guy. And his niceness, propagated through his students and their students, set the tone for the whole field, making it not only more successful scientifically, but also a much more enjoyable place within which to work.
To Andy Barto, a true scholar and dear friend: Thank you for all that you have done.

Dr. Peter L. Strick
Detre Professor & Chair of Neurobiology
Scientific Director, University of Pittsburgh Brain Institute
I was involved in a collaborative research project with Andy in the late 90s.  Of course, I was well aware of his intellectual contributions well before that.  However, interacting with Andy was a treat. Andy has the capacity to make complex concepts accessible.  He has the kind of mind that thrives on new ideas and problem solving.
Andy would be the first to tell you that he doesn’t know much about the brain- he couldn’t tell you the difference between the caudate and the claustrum.  And yet, Andy’s contributions to neuroscience have been remarkable. His computational structures have led to major insights into the functional significance of critical circuit motifs within the central nervous system.  Any recognition Andy receives for his work is richly deserved and long overdue.

Terrence Sejnowski
Francis Crick Professor
The Salk Institute for Biological Studies
A remarkable convergence has occurred between artificial intelligence and neuroscience over the last decade.  The key to this convergence has been based on implicit learning from experience rather than writing explicit computer programs.  Andy Barto was a pioneer in developing learning algorithms inspired by biological systems long before it became popular. His foundational  research on reinforcement learning was the basis for Kenji Doya’s model of birdsong learning and Peter Dayan’s and Read Montague’s model of reward prediction error for dopamine neurons when my lab was exploring computational models of brain systems in the 1990s.  Andy was working on the BRAIN Initiative, whose goal is to develop innovative neurotechnologies, long before it was launched in 2013. Thank you Andy for all you have done for us at the frontiers of engineering and neuroscience.

Wolfram Schultz
Professor of Neuroscience
University of Cambridge
The work of Andy Barto and Rich Sutton is not only a milestone for machine learning but also a huge achievement for neuroscience. Their theory of temporal difference learning represents the next step after Pavlov’s learning experiments and greatly helped to interpret the reward signal of dopamine neurons, which, in turn, provide a biological implementation of their theoretical constructs. My understanding of reinforcement processes, and how they were represented in neurons, benefitted a lot from personal discussions with Andy, who never tired to listen to my questions and arguments, however weird they were.

Peter Dayan
Director, Max Planck Institute for Biological Cybernetics
Tübingen, Germany.
It is an enormous pleasure and honour to help celebrate Andy Barto’s huge achievements and contributions. Andy’s research across the whole spectrum of reinforcement learning (RL) has had a huge impact in very many fields, providing foundational insight as well as algorithms of wide applicability and application. His contributions are marked by a most impressive mix of deep intuition about the nature of a fundamental problem that is both important and likely to be tractable, a subtle understanding of the balance between rigour and practicality, and a clarity of communication that allows his insights and ideas to spread widely.  These qualities of his research make him a wonderful supervisor and collaborator.
The foundation for pretty much my whole career was a fateful summer in Amherst in 1990 when I was a thoroughly callow second year graduate student in Edinburgh. I even have the artificial teeth to prove it. I was planning to attend a summer school in San Diego, where Andy was going to teach, so I wrote to ask if I could spend some time with him on my way back home. Perhaps baffingly, he said yes – and so I had the unique chance not only to learn the fundaments of modern reinforcement learning (RL) from Andy, but also to join the wonderful community that he built and nurtured – including Satinder Singh (with whom I ultimately went on to write what is both Satinder’s and my worst paper), Rich Sutton (with whom I argued about definitions for our whole meeting), and even Oliver Selfridge. I worked on TD(lambda) at Amherst, but it was particularly Andy’s deep appreciation of psychological and neural data, and understanding how these fit with the engineering aspects of RL, that set the course for a huge swathe of my subsequent work.  Andy had many other impacts on me in the subsequent thirty years, not the least examining my PhD thesis and introducing me to Chris Watkins. He has even been an active academic grandfather to some of my own supervisorial children.
In RL, we know that we have to correlate different actions with their outcomes to work out what is best. I can barely quantify my fortune in having been allowed to select the option intersecting with Andy. Thanks!

Robert A. Jacobs
Professor, Brain and Cognitive Sciences, Computer Science, and Center for Visual Science
University of Rochester
Among Andy’s many assets, the one that I admire the most is his intellectual curiosity. This curiosity led him to study computer science and artificial intelligence, but also to study related fields such as psychology and neuroscience. His cross-disciplinary insights enriched research into reinforcement learning because they pointed out how solutions in one field, such as artificial intelligence, raised important questions for researchers from other fields, such as neuroscience.  When Andy was my graduate advisor, I always found his curiosity to be infectious. His lab was an exciting place to be a student due to his enthusiasm for seeking out the connections among seemingly different research domains. Indeed, his mentorship in this regard is a major reason that I earned a graduate degree in computer science but I became a university professor in a different field, namely cognitive science. In this sense, he changed the course of my professional life!

T.W. Robbins
Head of the Department of Psychology
University of Cambridge
Andy Barto’s theoretical work on reinforcement learning with Sutton has undoubtedly had a major influence on research on the neural basis of reinforcement learning and the discoveries honoured by the Brain Prize last year. It has also certainly provided an important stimulus to my own work on the functions of the central dopamine systems and the mediation of instrumental learning in humans and other animals. I am just about to invest in the recent second edition of his magnum opus!

Anthony Dickinson
Professor, Comparative Psychology
University of Cambridge
I am so pleased that his university is recognising Andy Barto’s major contribution to machine learning and cognitive science – If it wasn’t for his work with Richard Sutton, the field would be greatly impoverished. Although the development of computational reinforcement learning theory has not had a direct impact on my own thinking, there is one rather obscure 1981 paper by Sutton and Barto on “An adaptive network that constructs and uses an internal model of the world” that greatly influenced the development of our associative-cybernetic account of goal-directed action and habits and their interaction. This paper deserves greater recognition.

Paul Glimcher
Professor, Neuroscience and Physiology
New York University
When I started as an assistant professor recording from dopamine neurons Andy was one of my heroes, so it is so hard to sum up in a few words the enormous impact that that has had on me, or on our field. He, with Rich Sutton, really defined computational neuroscience – which is funny because neither he nor Rich are really neuroscientists. But Andy’s work was the epitome of clarity and innovation for thinking about how animals and machines learn. His models guided all of us as we thought hard about how dopamine might play a role in biological learning and at each step when we got lost or confused, it was Andy’s work that showed a clear path forward. When, for example, we began to be confused about why dopamine neurons seemed to respond to novel but unrewarded stimuli, it was Andy and Rich’s previous work on the novelty bonus that showed a way forward. When Wolfram’s asymmetric ramp seemed a confusing puzzle, it was just running through Andy and Rich’s work that helped us understand that this feature was hardly new to them, and a clear prediction of their models which we all had just overlooked by assuming symmetry in the prediction error. This has happened so many times that whenever I see a new and puzzling result in physiology, I always stop and ask whether this is exactly what Andy would have expected, before I get, errr, worked up. And of course Andy is famous amongst all of my generation of computational neuroscientists for his grace and charm. Every interaction with Andy, whether by email or in person, was an opportunity for him to encourage us, to make us feel that what we were doing was important and worthwhile. So, I guess to some it up, let me say again that when I started Andy was one of my heroes. Today, 25 years later Andy is still one of my heroes and when I grow up I want to be a neuroscientist just like him!

Scott Niekum
Assistant Professor, Computer Science
University of Texas, Austin
I’m sure everyone will say thus, but Andy is a researcher that was far ahead of his time. He formalized and began solving challenging reinforcement learning problems long before machine learning was even a recognized discipline. He continued working on this line of thought persistently for decades, even during periods in which reinforcement learning was very unpopular. And yet, without his foundational work, several of the greatest, most visible achievements of modern machine learning, such as AlphaGo, would not have been possible. During my PhD in the early 2010s, he was still thinking about ideas such as intrinsic motivation and meta-learning before almost anyone else took them seriously, and predictably, these have become extremely important, exciting areas of research in recent years.
As an advisor, Andy eschewed the standard academic thinking of “publish or perish” and taught his students to publish less and better. He always pushed us to dig deep until we hit a fundamental question that was on the critical path to AI — he had little appetite for flashy, incremental work. Now that I am a professor myself, I have tried to align my lab as close as possible to this philosophy and pass it down to the next generation of students.

Naoshige Uchida
Professor, Molecular and Cellular Biology
Harvard University
Reinforcement learning is not only a key idea for training computers but also has become one of the most successful and impactful theories in neuroscience. Reinforcement learning theories, in particular, the temporal difference learning algorithm, is the foundation with which to understand dopamine signals in the brain and how the animal learns from reward and punishment, more generally. Dr. Barto has made pivotal contributions to establish these learning algorithms and the field of reinforcement learning altogether. I would like to congratulate Dr. Barto for this award recognizing his seminal contributions to artificial intelligence and neuroscience.

Randall O’Reilly
Professor, Department of Psychology and Neuroscience
University of Colorado, Boulder
Andy has had a foundational impact on the field, through his prescient and comprehensive understanding of the nature of reinforcement learning. The principles he figured out in his productive collaboration with Rich Sutton have truly stood the test of time and are as central and foundational to our field as e.g., Newton or Einstein’s work was in physics. These are truly the most fundamental and remarkably accurate principles of how the brain’s dopaminergic system functions, and have led to thousands of papers — their textbook on the topic has been cited nearly 30,000 times according to Google scholar! My personal work has attempted to connect this more abstract mathematical framework to the gory details of different brain circuits, and I never fail to be amazed at how all this complex circuitry nevertheless can be so well described by the principles that Andy and Rich developed.

Yael Niv
Professor, Neuroscience Institute and Department of Psychology
Princeton University
Congratulations to Dr. Barto on this highly deserved award!! It is patently clear that Andy’s research and work in the field of reinforcement learning was transformative to not only computer science and control theory, but maybe most importantly, to neuroscience and psychology, and our understanding of learning and decision making in the brain. I vividly remember, as a young researcher, reading the Sutton and Barto “bible” at least three times, learning new things each time. That book, and the papers Andy wrote, in particular the 1995 chapter about a critic in the basal ganglia, were fundamental to all of my later research. And I am just one of many on whom Andy’s work has had this influence. The “proof” for this can be seen in the immensely popular RLDM conference, which stemmed from discussions at Andy’s retirement party at UMass back in 2012. Hundreds of people see this as their “home conference” — an amazing testament to the amount of work that has stemmed from his pioneering research.
Thank you so much for forging our path and leading the way, and for including me (and others from the psychology/neuroscience side) in this journey!

Bruno Castro da Silva
Associate Professor, Computer Science
Federal University of Rio Grande do Sul
Porto Alegre, Brazil
I first read Andy’s book on reinforcement learning in 2005. I remember being thoroughly amazed by the elegance and succinctness with which he introduced the field that he and Rich Sutton pioneered. It was difficult to imagine that two years later I would be living in a different country and working with Andy himself. Andy’s breadth of knowledge in math, psychology, statistics, neuroscience, and machine learning make him a perfect example of a Renaissance man. His ability to see the underlying themes connecting seemingly disparate ideas is uncanny. Andy’s technical skills are only surpassed by his immense generosity, thoughtfulness, and completely unnecessary humbleness. He will always be a mentor, a colleague, and a good friend. I truly cannot imagine a better role model to aspire to. Thank you, Andy, for those incredible seven years; and thank you for your contributions to neuroscience and AI: we are all better scientists because of you!

Daeyeol Lee
Professor, Neuroscience, Psychology and Psychiatry
Yale University
Andy is one of the most inspiring and generous scientists I had the fortune to interact with. His contribution to theories of reinforcement learning and its dissemination in neuroscience is immeasurable. has also set the direction of my own research during the last 2 decades.

P. Read Montague
Professor, Fralin Biomedical Research Institute and Department of Physics
Virginia Tech
It’s a privilege and an honor to pay tribute to Andy Barto’s career and contributions to our understanding of learning, control, and valuation in both artificial and natural learning systems. His work on reinforcement learning combines the best traditions of engineering, learning theory, adaptive behavior; and, we have all profited immensely from his seminal work on temporal difference learning and it’s application to behavior. This latter work, done in glove-fitting-closeness with Richard Sutton has been so influential in neurobiology and the psychology of adaptive choice that their names are forever linked as though we are describing one person; Sutton-Barto. Not only did this work provide a way to understand the power of reinforcement learning in adaptive behavior, it also provided a computational setting to understand the algorithms that evolution has discovered again-and-again for controlling the learning that guides adaptive choice in rodents, people, songbirds, bees and a collection of other creatures. The modern flowering of reinforcement learning in applied domains only serves to reify the importance of understanding the flexibility of learning styles throughout the natural world. Barto’s contribution falls in line with an august history that includes Richard Bellman and Arthur Samuel from the 1950’s through the prescient work of Harry Klopf in the 1970’s and onward to the present. Such scope in scientific contributions is rare and I believe that U. Massachusetts has made an excellent choice in selecting Andy for this lifetime achievement award. Congratulations and well earned!

John W. Donahoe
Professor Emeritus, Department of Psychological and Brain Sciences and Graduate Program in Neuroscience & Behavior
University of Massachusetts/Amherst
Some years ago, a colloquium by Andy Barto on reinforcement learning opened the possibility of simulating the biobehavioral processes that I was studying in my experimental work. At first with an undergraduate student in Computer Science and then with my doctoral student Jose’ Burgos, we developed a hybrid genetic/learning algorithm implementing a biologically informed neural-network model that instantiated both innate reinforcement (paralleling the temporal-difference concept) and learned reinforcement (paralleling the adaptive critic). This model has successfully simulated a wide range of behavioral phenomena and would not have been developed without the inspiration provided by the Barto-Sutton computational research.

Kenji Doya
Professor, Okinawa Institute of Science and Technology Graduate University
Building a machine that learns on its own by trial and error has been a dream of every engineer, but Andy’s focus on prediction was the key to a series of breakthroughs, leading to the success of AlphaGo most recently. His theory also inspired many neuroscientists and delivered today’s understanding of key roles of dopamine and the basal ganglia in reinforcement learning in the brain. I was fortunate enough to invite Andy to our Okinawa Computational Neuroscience Course back in 2005 for a three-hour lecture. Hearing how Andy and his colleagues explored different ideas and equations until finally converging onto the temporal difference learning equation was most exciting and enlightening.

Nathaniel Daw
Princeton Neuroscience Institute, Princeton University
On the first day I arrived at graduate school, I was handed a copy of a book, “Models of Information Processing in the Basal Ganglia,” the highlight of which was a pair of chapters authored by Andy. I pretty much based my PhD — and from there, my career — on those chapters. They noted a detailed and suggestive correspondence between a set of brain systems involved in reward and movement, and temporal-difference learning algorithms. This idea (and a version separately propounded by another group) was wildly influential in neuroscience and psychology. It became both the core of a predominant account of how the brain accomplishes RL, and also more generally a seed and an exemplar for a much vaster enterprise of formalizing learning, choice and movement in terms of ideas and algorithms from RL.
As this subfield of psychology and neuroscience developed, all of us — literally everybody — learned our computational theory from Andy and Rich’s 1996 textbook, whose clearheaded rethinking and exposition of the foundations made it approachable even to biologists. It provided such a compelling vision and a fertile source of ideas that it sort of anchored neuroscience in the state of RL as it existed in 1996: think like how Dante solidified the modern Italian language. I mean this as testament to its timelessness, though I do also admit to having secretly pursued a strategy of trying to track down any bits that it omitted in order to get a jump on my colleagues, a sort of meta-influence I also ultimately owe to Andy. In any case, with the second edition now published, we are bracing for a renaissance in this area as neuroscience, all at once, takes on board twenty more years of ideas. 

George Konidaris
Assistant Professor, Department of Computer Science
Brown University
I am very pleased to hear that Andy was being honored in this way.  Despite the magnitude of his contribution, in his understated way Andy has always remained fundamentally humble and has never sought prizes or awards, which is a great pity because he deserves so many.
I joined Andy’s lab as a PhD student in 2004, by which time he was, in AI and ML circles, about as famous as a researcher could be. Nevertheless, from the start he treated me as an equal partner in
research, which at the time I took for granted but which in retrospect was astoundingly kind. In his lab I found an incredibly fertile intellectual environment full of extraordinarily talented people, and that far from resting on his laurels, he was always looking ahead, still innovating, still creating, often far ahead of the rest of the field. Indeed, some of the key ideas fermenting in that lab at the time – abstraction, hierarchies, intrinsic motivation – seemed niche and obscure to outsiders, but have roared into mainstream fashion in the last few years, a decade and a half after Andy identified them as the next important questions.
Like so many before me – I count 27 graduated PhD students – I was fundamentally intellectually transformed by the experience of working with Andy. He showed all of us what it was to be a true innovator, and of the importance of charting one’s own course in both rough and following seas. Our field is still following his lead, even after he has retired, and I’m sure it will be for years to come.

James L. (Jay) McClelland
Lucie Stern Professor in the Social Sciences, Department of Psychology,
Stanford University
To me Andy Barto stands out, not only for his outstanding contributions to our understanding of reinforcement learning, but also for his unassuming personal nature. Of all the leaders in the field of computational intelligence and the neuroscience of learning and memory, Andy may be the one who has been the least likely over the years to call attention to himself. Andy, your work truly speaks for itself and I am pleased to join the many voices celebrating your contributions. Your work truly paved the way along many of the paths now being explored in reinforcement learning, and you played a key role in creating a community of scholars using RL to help us understand natural and artificial intelligence.