1. AI agents — what they are, and how they’ll change the way we work
2. Understanding AI Agents: How They Work, Types, and Practical Applications

1. What are the key performance metrics to look out for in LLM inference?
2. What would be the important metrics to evaluate if you were to run a language model on your PC/tablet?

1. A Queueing Theoretic Perspective on Low-Latency LLM Inference with Variable Token Length
2. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
3. TensorRT-LLM

1. By making the pipeline more smooth (less pipeline bubbles), what tradeoff does PipeDream make? i.e., in what aspect is GPipe better than PipeDream?
2. What type of parallelism do you think is most widely adopted in practice? Why?

1. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
2. GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
3. Memory-Efficient Pipeline-Parallel DNN Training (PipeDream-2BW)
4. Scaling Distributed Machine Learning with the Parameter Server
5. Optimization of Collective Communication Operations in MPICH

1. How does HLAT determine its parallelism strategies and system optimizations, and how might these trade-offs evolve as model size or cluster scale increases?
2. Which stage or component of pre-training do you consider most critical in practice, and why?

1. Distributed training of large language models on AWS Trainium (SoCC'24)
2. GEMINI: Fast Failure Recovery in Distributed Training with In-Memory Checkpoints (SOSP'23)
3. Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning (OSDI'22)
4. DeepSeek-V3 Technical Report
5. s1: Simple test-time scaling

1. List one benefit and one drawback of iterative scheduling (as compared to request-level scheduling).
2. How would you decide the batch size (number of request) for an LLM inference system you are building? List all the factors you would consider.

1. ServerlessLLM: Low-Latency Serverless Inference for Large Language Models
2. Fairness in Serving Large Language Models
3. Can Scheduling Overhead Dominate LLM Inference Performance? A Study of CPU Scheduling Overhead on Two Popular LLM Inference Systems

1. List at least two reasons why GPU memory for KV cache is wasted without PagedAttention.
2. What are the tradeoffs of using larger/smaller block sizes in PagedAttention?

1. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
2. Efficiently Programming Large Language Models using SGLang
3. MemShare: Memory Efficient Inference for Large Reasoning Models through KV Cache Reuse
4. KVLINK: Accelerating Large Language Models via Efficient KV Cache Reuse

1. List two drawbacks of speculative decoding.
2. In an agentic setting where LLMs are repeatedly called, can you think of a way to speculate output tokens without using a draft model (Mq in the paper)?

1. SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification (ASPLOS '24)
2. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
3. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test
4. SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applications
5. Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recycling

1. What's the pros and cons of megakernel as compared to traditional more fine-grained GPU kernels?
2. Do you think it's feasible to build a megakernel that incorporates speculative decoding? How?

1. Mirage: A Multi-Level Superoptimizer for Tensor Programs
2. We Bought the Whole GPU, So We're Damn Well Going to Use the Whole GPU
3. ARK: GPU-driven Code Execution for Distributed Deep Learning
4. FlashDMoE: Fast Distributed MoE in a Single Kernel
5. Inside NVIDIA GPUs: Anatomy of high performance matmul kernels
6. POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inference
7. LithOS: An Operating System for Efficient Machine Learning on GPUs

1. What will happen if all requests are scheduled only based on where their matched prefix cache reside?
2. What does Preble do when a lot of requests share the same prefix (i.e., highly skewed workloads)?

1. AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving (OSDI'23)
2. Ray Serve: Scalable and Programmable Serving

1. How does DistServe designed their system to mitigate the overhead of KV cache migration?
2. Do you think DistServe can scale to a very large cluster, say 100/1k/10k GPUs for serving? Why or why not?

1. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
2. Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Serving
3. Splitwise: Efficient generative LLM inference using phase splitting
4. Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads
5. POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inference
6. Nexus:Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Serving
7. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference

1. Can DeepSpeed-Ulysses work when the number of GPUs is higher than the number of heads? Why or why not?
2. What is the latency impact for one transformer layer caused by DeepSpeed-Ulysses?

1. Sequence Parallelism: Long Sequence Training from System Perspective
2. Tensor Parallelism and Sequence Parallelism: Detailed Analysis
3. Introducing Context Parallelism
4. Ring Attention with Blockwise Transformers for Near-Infinite Context
5. DISTFLASHATTN: Distributed Memory-efficient Attention for Long-context LLMs Training
6. LoongTrain: Efficient Training of Long-Sequence LLMs with Head-Context Parallelism

1. What is the pros and cons of merged and unmerged adapter in LoRA inference?
2. Can Algorithm 2 in the paper lead to request starvation? If so, how does dLoRA solve it?

1. LoRA: Low-Rank Adaptation of Large Language Models
2. Punica: Multi-Tenant LoRA Serving
3. S-LORA: SERVING THOUSANDS OF CONCURRENT LORA ADAPTERS
4. AIBrix: Towards Scalable, Cost-Effective Large Language Model Inference Infrastructure

1. What are the three ways of dealing with KV cache when a model calls an API?
2. Why does InferCept consider running requests that are not currently calling tools when calculating GPU memory waste?

1. Toolformer: Language Models Can Teach Themselves to Use Tools
2. KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows
3. Asynchronous LLM Function Calling

1. Why can the disaggregation of attention and FFN improve the GPU utilization during MoE serving?
2. Does the ping-pong pipeline impose any inherent constraints on the model architecture? If yes, what problems will arise when such constraints are not satisfied?

1. fMoE: Fine-Grained Expert Offloading for Large Mixture-of-Experts Serving

1. What are the main advantages and risks of using this purely RL-based approach (without supervised fine-tuning) to incentivize chain-of-thought reasoning in LLMs?
2. What implications does the study results have for designing efficient large reasoning systems? Any proposal for improving request latency?

1. Resa: Transparent Reasoning Models via SAEs
2. Reinforcement Learning for Reasoning in Large Language Models with One Training Example
3. The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity
4. ARM: Adaptive Reasoning Model
5. Hierarchical Reasoning Model

1. For tool-based AI systems, do you belive we should have more autonomy (more agent like) or less autonomy (more workflow like)? Find a use case to justify your answer.
2. Can you think of some challenges of benchmarking and ensuring robustness in agents that depend on external tools?

1. Announcing the Agent2Agent Protocol (A2A)
2. Conveyor: Efficient Tool-aware LLM Serving with Tool Partial Execution
3. An LLM Compiler for Parallel Function Calling

1. How is the LLM inference (rollout) phase in RL training different from LLM serving (for end users)? List at least two differences.
2. Can you think of a way to improve the LLM inference phase in RL training?

1. FastGRPO: Accelerating Policy Optimization via Concurrency-aware Speculative Decoding and Online Draft Learning
2. SPEC-RL: Accelerating On-Policy Reinforcement Learning via Speculative Rollouts
3. StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generation
4. History Rhymes: Accelerating LLM Reinforcement Learning with RhymeRL
5. Kimi k1.5: Scaling Reinforcement Learning with LLMs

1. What's the memory-hierarchy architecture proposed in MemGPT? What are the tradeoffs this architecture introduces in terms of latency?
2. How would you redesign the memory hierachy for throughput-oriented tasks?

1. DeepSeek-OCR: Contexts Optical Compression
2. SagaLLM: Context Management, Validation, and Transaction Guarantees for Multi-Agent LLM Planning
3. Git Context Controller: Manage the Context of LLM-based Agents like Git
4. A Comprehensive Guide to Context Engineering for AI Agents
5. Recursive Language Models
6. A-Mem: Agentic Memory for LLM Agents

1. What's the pros and cons of Cognify compared to RL-based agent tuning methods?
2. How does Cognify decides the size (number of samples) of each layer? Do you think that's reasonable?

1. GEPA: REFLECTIVE PROMPT EVOLUTION CAN OUTPERFORM REINFORCEMENT LEARNING
2. Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs
3. Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?
4. Language Agents as Optimizable Graphs
5. DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines

1. How does METIS's LLM profiler's data inform the adaptation mechanism's scheduling decision at query time?
2. Consider an agent that must perform both RAG and tool use. How could the core principles of METIS be extended to create a "quality-aware scheduler" for this type of agents?

1. Quake: Adaptive Indexing for Vector Search (OSDI '25)
2. A Comprehensive Survey on Vector Database: Storage and Retrieval Technique, Challenge
3. HedraRAG: Coordinating LLM Generation and Database Retrieval in Heterogeneous RAG Serving (SOSP '25)
4. Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?
5. Seven Failure Points When Engineering a Retrieval Augmented Generation System

1. What's the trade-offs of this dual-index approach versus relying on a single, powerful multi-modal embedding for retrieval?
2. Discuss the system-level implications of the proposed design on: (a) offline indexing cost and complexity, (b) online query latency, and (c) handling new documents that have not been fully processed

1. Deep Research Agents: A Systematic Examination And Roadmap
2. Cline Autonomous Coding Agent

1. How might an agent's effective use of tools (e.g., code execution, web search) potentially obscure the evaluation of its true underlying reasoning or planning abilities?
2. How would you design an interactive evaluation for computer-use agents?

1. Evaluation and Benchmarking of LLM Agents: A Survey
2. AgentBench: Evaluating LLMs as Agents
3. Establishing Best Practices for Building Rigorous Agentic Benchmarks
4. Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation

1. What is the fundamental limitation of using human efficiency as the "gold standard" for an AI agent?
2. Can you think of a way to reduce the end-to-end CUA latency?

1. A Case for Declarative LLM-friendly Interfaces for Improved Efficiency of Computer-Use Agents
2. The Unreasonable Effectiveness of Scaling Agents for Computer Use
3. GTA1: GUI Test-time Scaling Agent

1. How does the CAMEL framework's role-playing structure mitigates the instability issue of multi-agent systems?
2. How do you think multi-agent systems can scale (to more agents/more roles)? Any key challenges you can think of?

1. Why Do Multi-Agent LLM Systems Fail?
2. ChatDev: Communicative Agents for Software Development
3. AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation

1. Can you think of some new challenges or opportunities when serving agents in a serverless way from traditional serverless services like AWS Lambda?
2. What do you think are the key requirements for agent hosting?

1. SALT: Step-level Advantage Assignment for Long-horizon Agents via Trajectory Graph
2. Azure AI Foundry
3. Strands documentation
4. AgentRL training
5. Self-Challenging Language Model Agents

1. How do stateful, action-oriented vulnerabilities render traditional LLM defenses (like standard alignment or prompt filtering) insufficient?
2. Can you think of an example where a particular agent we've introduced in the course has a vulnerability?

1. AI Agents Are Here. So Are the Threats

1. How does CacheBlend efficiently combine retrieved KV caches with the precomputed KV cache of a large text chunk without recomputing attention for the entire sequence?
2. How does Selective Recomputation decide which tokens to recompute versus which to blend?

Read the "Hints for Computer System Design" paper and summarize what you have learned over the course. Feel free to write about anything else you want to comment on the course.