Large-scale machine learning

elective monographs

Deep neural networks 1000-317bDNN
Natural language processing 1000-318bNLP
Statistical machine learning 1000-317bSML

parallel programming, computer networks, algorithms and data structures

The goal of this course is to build the theoretical foundation and practical skills necessary to use machine learning algorithms and techniques at a large scale.

We will discuss the architecture of modern large-scale computing infrastructure (cloud datacenters, and AI and HPC supercomputers). We will present methods for distributing computations across these clusters and the fundamental algorithmic models used to estimate performance. Using examples of typical ML algorithms (decision trees, neural network training), we will demonstrate the theoretical and practical challenges of using them at the scale of a few to several hundred machines. Next, we will cover the challenges of training and using large-scale language models (LLM). The course will conclude by presenting the primary problems of using ML models in large-scale production environments.

- Hardware: from a GPU to a datacenter, and why the architecture matters at scale

- Parallel and distributed optimization: how to parallelize algorithms and how to reason about their performance

- Parallelizing classic ML algorithms

- LLMs introduction: motivation, transformers and scaling laws

- Parallelizing LLM training: parallelization types, bottlenecks, common memory optimizations

- Datasets and benchmarking LLMs

- Handling data: an introduction to data engineering

- ML in production: risks, rewards, common problems

- Case study: ML in computational infrastructure

- Scientific papers used during lectures

- “The Datacenter as a Computer: Designing Warehouse-Scale Machines”, Luiz André Barroso, Jimmy Clidaras, and Urs Hölzle

- “Fundamentals of Data Engineering”, Joe Reis and Matt Housley

Final score based on programming assignments, points for participation in laboratories and a written exam.