Schedule

• Course introduction
• Logistics
• Transformers

• Arman

• Attention is all you need (2017) [link]

• Transfer learning
• Pre-training
• Pre-trained transformers

• Arman

• ELMo, Deep Contextualized Representations [link]
• ULMFit, Universal Language Model Fine-tuning for Text Classification [link]
• BERT, Pre-training of Deep Bidirectional Transformers for Language Understanding [link]

• RoBERTa, A Robustly Optimized BERT Pretraining Approach [link]

• Transfer learning
• Pre-training

• Kejian
• Huangrui

• Language Models are Unsupervised Multitask Learners (GPT-2) [link]
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5) [link]

• BART, Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension [link]
• ELECTRA, Pre-training Text Encoders as Discriminators Rather Than Generators [link]

• Model Architecture and Training Objectives

• Zhangir

• UL2- Unifying Language Learning Paradigms (2022) [link]
• What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization? (2022) [link]

• Scaling Laws for Langauge Models
• Tips on choosing a project [Arman]

• Ruixiao
• Chenxi

• Scaling Laws for Neural Language Models (2020) [link]
• Training Compute-Optimal Large Language Models (2022) [link]

• Scale Efficiently - Insights from Pre-training and Fine-tuning Transformers (2021) [link]

[Tips on choosing projects]

[Projects Doc]

• LLMs
• Power of scale

• Arman
• Ziqing

• Language Models are Few-Shot Learners (GPT3; 2020) [link]
• PaLM - Scaling Language Modeling with Pathways (2022) [link]

• Prompting and Few-shot learning

• Ayla
• Hailey

• It’s Not Just Size That Matters- Small Language Models Are Also Few-Shot Learners (2020) [link]
• Making Pre-trained Language Models Better Few-shot Learners (2021) [link]

• Prompting and ICL
• Why ICL works

• Yujie
• David

• Rethinking the Role of Demonstrations- What Makes In-Context Learning Work? (2022) [link]
• Data Distributional Properties Drive Emergent In-Context Learning in Transformers (2022) [link]

• What learning algorithm is in-context learning? Investigations with linear models (2022) [link]
• Transformers as Algorithms- Generalization and Stability in In-context Learning (2023) [link]

Finalize project teams

• Instruction tuning

• Huangrui
• Hyoungseob

• Training language models to follow instructions with human feedback (2022) [link]
• Scaling Instruction-Finetuned Language Models (2022) [link]

• Parameter efficient fine-tuning

• Zhangir
• Hailey

• Towards a Unified View of Parameter-Efficient Transfer Learning (2021) [link]
• Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning (2022) [link]

• Parameter-Efficient Transfer Learning for NLP (2019) [link]
• Prefix-Tuning- Optimizing Continuous Prompts for Generation (2021) [link]

• Chain of thought reasoning
• Emergence

• Ayla
• Ziqing

• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (2022) [link]
• Emergent Abilities of Large Language Models (2022) [link]

• Efficient transformers

• Kejian
• Ruixiao

• Longformer- The Long-Document Transformer (2020) [link]
• Big Bird- Transformers for Longer Sequences (2020) [link]

• Efficient Transformers- A Survey [link]

• Efficient transformers

• David
• Yujie

• Efficiently Modeling Long Sequences with Structured State Spaces (2021) [link]
• Simplifying S4 [link]

• On the Parameterization and Initialization of Diagonal State Space Models (2022) [link]
• Hungry Hungry Hippos- Towards Language Modeling with State Space Models (2022) [link]
• FlashAttention- Fast and Memory-Efficient Exact Attention with IO-Awareness (2022) [link]

• Memory

• Arman

• Memorizing Transformers (2022) [link]

• Training Language Models with Memory Augmentation (2022) [link]

Project proposals due on 3/4

• Guest Lecture
• Scott Yih - Efficient and Scalable NLP through Retrieval-Augmented Language Models

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020) [link]
• Improving language models by retrieving from trillions of tokens (2021) [link]
• REPLUG, Retrieval-Augmented Black-Box Language Models [link]
• Retrieval-Augmented Multimodal Language Modeling [link]

• Iterative methods

• Hailey

• PEER- A Collaborative Language Model (2022) [link]

• Generating Sequences by Learning to Self-Correct (2022) [link]

• Societal and ethical considerations, bias, safety in AI/NLP

• Kejian

• On the Dangers of Stochastic Parrots- Can Language Models Be Too Big? (2020) [link]

• Self-Diagnosis and Self-Debiasing- A Proposal for Reducing Corpus-Based Bias in NLP (2021) [link]

4/1 Progress report due

• Mixture of experts and sparse models

• Ayla

• Switch Transformers- Scaling to Trillion Parameter Models (2021) [link]

• Sparse Upcycling- Training Mixture-of-Experts from Dense Checkpoints (2022) [link]

• Data
• Spurious biases
• Dataset difficulty

• Yujie

• Competency Problems- On Finding and Removing Artifacts in Language Data (2021) [link]

• Understanding Dataset Difficulty with V-Usable Information (2021) [link]

• Multi-modal models

• Zhangir

• Flamingo- a Visual Language Model for Few-Shot Learning (2022) [link]

• CM3- A Causal Masked Multimodal Model of the Internet (2022) [link]

• Decoding methods for natural language generation

• Ruixiao

• NEUROLOGIC DECODING- (Un)supervised Neural Text Generation with Predicate Logic Constraints (2021) [link]

• NEUROLOGIC A*esque Decoding- Constrained Text Generation with Lookahead Heuristics (2022) [link]

• Training LLMs with human feedback
• AI Alignment

• David

• A General Language Assistant as a Laboratory for Alignment (2021) [link]

• Learning to Summarize with Human Feedback (2020) [link]

• AI Alignment

• Ziqing

• Fine-tuning language models to find agreement among humans with diverse preferences (2022) [link]

• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback (2022) [link]

• LLMs for Code

• Huangrui

• CodeGen- An Open Large Language Model for Code with Multi-Turn Program Synthesis (2022) [link]

• InCoder- A Generative Model for Code Infilling and Synthesis (2022) [link]

• Guest Lecture
• Tim Dettmers - Making LLMs more efficient

• LLM.int8()- 8-bit Matrix Multiplication for Transformers at Scale [link]
• Efficiently Scaling Transformer Inference [link]

• Final presentations for projects

• All students

4/27 Presentations; 5/10 Final project report