Posts

New job at Meta

I’m excited to announce that I started my new job at Meta as a Research Scientist on the CodeGen team in Paris, France led by Gabriel Synnaeve.

Code generation using ML got me very excited because I was always frustrated by the amount of time it was taking me to translate my ideas into code, and it became much easier in the last couple of years. I think John Carmack said in an interview that in game development, most of code written is never read by anyone because there is just too much of it. I like to imagine systems like that in which the code is generated on the fly for different use cases, optimized, tested and debugged without us directly seeing any of that. More than anything, I just want a tool that would make programming more about designing elegant systems and solutions than actually writing or debugging them.

I am particularly excited to join Meta given their strong commitment to open source AI, which I believe is crucial for ensuring democratic access to this technology. I will keep writing and publishing papers as well as releasing my code.

Last updated on Oct 24, 2024 1 min read

Talk at Imperial College London

I was invited by Panos Parpas to deliver a talk at Imperial College London, which I had the pleasure to do today. I talked about my perspective on the optimization theory and how it can be used to approach deep learning. While training deep networks is defined by nonconvex non-smooth losses, which are nearly impossible to minimize in general, the practical performance of optimization methods is nowhere as pessimistic. At the same time, smooth optimization framework seems to suggest that methods like SVRG, which do not work well in deep learning, are the best we can use. The purpose of my talk was, therefore, to outline directions where I believe useful theory can be developed and identify some promising ways of bridging theory with practice.

Last updated on Oct 9, 2024 1 min read

Leaving Samsung

Today was my last day working as a research scientist at Samsung AI Center in Cambridge, UK. It was a pleasure to be there, we wrote a few papers and one patent application (currently under review) and I had a lot of fun collaborating with people at the office. However, I want to explore new directions and I no longer want to stay in the UK. I will announce my next steps later and I’m taking a break for now to work a little bit on composin electronic music in Ableton Live, which is something I’ve been really enjoying in the last 6 months.

Last updated on Oct 9, 2024 1 min read

Research directions in optimization

In this post, I share my thoughts on some interesting research direction. Not an exhaustive list, just something I came up with in 30 minutes when responding to an email from a student.

Last updated on Sep 26, 2024 3 min read

Invited lecture at Cambridge University

Today I gave an invited lecture at the University of Cambridge in the Federated Learning class taught by Nic Lane. My presentation was structured around the optimization aspects of federated learning, with the emphasis on what theoretical aspects of a method matter when we are trying to understand how it works in practice.

Last updated on Feb 21, 2024 1 min read

Thoughts on doing PhD in optimization

In this post, I would like to share some thoughts on whether it’s a good idea to choose optimization as one’s PhD topic. The text was written as a response to a MSc student, who reached out asking this question, but my response is general enough to share it here for anyone thinking about doing a PhD.

Last updated on Feb 21, 2024 4 min read

DoWG accepted at NeurIPS

Our work on an extension of DoG with weighted gradients got accepted for presentation at NeurIPS this year! If you want to try our method, a pytorch implementation is available on github. I hope to see more papers building upon DoG, DoWG, D-Adaptation, and Prodigy, we have barely scratched the surface on what can be done, and some of these methods are already being used in practice.

Last updated on Sep 24, 2023 1 min read

ICML Outstanding Paper Award

I’m delighted to share that Aaron Defazio and I received the ICML Outstanding Paper Award for our work on D-Adaptation. The associated github repository of our paper has been quite popular and we are working hard on making extensions that will make adaptive methods even more useful for deep learning. Our first extension, Prodigy, is available on github as well and has been performing even better than D-Adaptation in our experiments. Expect more updates from us pretty soon!

Last updated on Sep 24, 2023 1 min read

I became an Action Editor for TMLR

I accepted an invitation to serve in the role of Action Editor to handle a small number of submissions for Transactions on Machine Learning Research (TMLR). I received the invitation shorty after I became an Expert Reviewer for TMLR, but I will only handle submissions as an editor from now on.

Last updated on Jul 28, 2023 1 min read

I've been selected as an Expert Reviewer for TMLR

I was informed that I got selected as an Expert Reviewer for Transactions on Machine Learning Research (TMLR). As the email “Expert Reviewers are reviewers for TMLR who have done exemplary work in evaluating TMLR submissions. They have stood out in various ways, including by writing detailed reviews, engaging deeply with authors and performing their work in a timely fashion.”

The full list of TMLR Expert Reviewers is available here.

Last updated on Sep 24, 2023 1 min read

I've been selected as an Expert Reviewer for TMLR