Abstract: 

AI systems is becoming ubiquitous as we've observed the success of products like ChatGPT, GPT4, Claude, Llama etc. However, these products, particularly the large language models (LLMs) they are based on, face significant challenges, one of which is the increasing difficulty to aligning with human behavior due to the sheer size of the models and datasets. This talk will explore recent advancements in pretraining and fine-tuning LLMs to enhance their performance across various dimensions. We will introduce an approach that employs inverse reinforcement learning and bi-level optimization to align LLMs with human behavior more effectively. Additionally, we will examine the application of zeroth-order optimization techniques for efficient fine-tuning of LLMs. The overarching goal of this talk is to demonstrate the critical role of sophisticated optimization modeling and algorithm design in advancing the capabilities of AI systems.