DSC 291 Safety in Generative AI (Fall 2025)

• Reading / videos: [Hinton interview, Tucker Carlson-Sam Altman, Davos AI Panel (LeCun, Russell and others) ]

Reading (due before Thursday lecture):

• Required Reading: The Foundation Model (Read the intro and at least one subsection from Section 2, 3, 4, or 5.)
• Optional Reading: Sutton’s The Bitter Lesson; GPT‑2 Multitask Learners; GPT‑3 Few‑Shot Learners; Sparks of AGI; Emergent Abilities; ; Are Emergent Abilities a Mirage?

Tue 9/30 – Course Intro, Deep Learning Refresher [slides]

• Mini‑lecture: Course overview, Deep Learning Basics .
• Activities: (1) discussion on AI safety videos, (2) vibe‑coding deep learning from scratch.

Thu 10/2 – Foundation Models & Emerging Abilities [slides]

• Mini‑lecture: zero‑shot, few‑shot / in‑context learning; emergence; prompt engineering (with math + coding).
• Activities: (1) groups explain one emerging ability from readings; (2) vibe‑code a simple in‑class game; (3) groups explain one potential vulnerability.

Homework (due Saturday):

• HW1: vibe‑code a ResNet image classifier (JAX/PyTorch) for MNIST. [code example in ipynb]

Reading (due before Thursday lecture):

• Required: Illustrated Transformer, OpenAI “Spec”
• Optional: Original GPT paper, Instruction Finetuning of LLMs, DeepSeek-R1, Altman statement on Teen Safety

Tue 10/7 – LLM Fundamentals, Pretraining [ slides ]

• Mini‑lecture: LLMs, attention mechanism, pretraining, scaling laws.
• Discussion: Building LLM from scratch

Thu 10/9 – Post‑Training & Safety Alignment [slides]

• Mini‑lecture: SFT, RLHF, safety alignment.
• Student Presentation #1: reflections on the GPT “Spec”.[slides]

Homework (due Saturday):

• HW2: vibe‑code a one‑layer Transformer (JAX/PyTorch) and train on movie‑review data.[data]

Reading (due before Thursday lecture):

• Required: Denoising Diffusion Model, Towards a Generalist Agent for the Web
• Optional: CLIP Paper, Score-based Generative Model and SDE, “LLM Agent that won Gold Medal at IMO”, Codex, AlphaCode, AIMA book Chapter 2: Intelligence Agent, Task Environment and Rationality, ACL'24 Tutorial on Language Agents

Tue 10/14 – VAEs & Diffusion Models

• Mini‑lecture: VAE & diffusion architectures; training basics. [slides]
• Activity: “Can you identify AI‑generated images?” (class game).

Thu 10/16 – LLM as Agents

• Mini‑lecture: AI agent → LLM agent; capabilities & risks. [slides]
• Discussion: propose an LLM‑agent idea and share.
• Student Presentations: #2 Computer‑Use Agent [slides]; #3 Coding Agent [slides].

Homework (due Saturday):

• HW3: vibe‑code a simple diffusion model to generate 1D data points. [ipynb Notebook for Data Generation]

Reading (due before Thursday lecture):

• Required: Jailbreaking attack; prompt‑injection attack
• Optional: Adversarial attack (Goodfellow et al 2014), Adversarial prompting for LLMs, Defense by Adversarial Training, Defence by Randomized smoothing, Certified Defense via Convex Outer Polytope.

Tue 10/21 – Adversarial Attacks on Images

• Mini‑lecture: adversarial attacks on image classifiers; how/why they work. [Slides]
• Experiment: implement gradient‑descent attack in class.
• Student Presentations: #5 Randomized Smoothing [Slides].

Thu 10/23 – Jailbreaking & Prompt Injection

• Mini-Lecture: jailbreaking and prompt-injection: attacks and defenses. [Slides]
• Student Presentations: #6 Complexity of Jailbreaking attack in the wild [slides]; #7 WASP: a benchmark for realistic prompt injection attacks [slides].
• Experiment: jailbreak "TritonGPT".

Homework (due Saturday):

• HW4: implement an adversarial attack against your MNIST ResNet.

Reading (due before Thursday lecture):

• Required: Poisoning Web-Scale Training Datasets AI Model Collapses when trained on recursively generated data
• Optional: Data PoisoningPoison Frogs: Data-Poisoning Attack , Targeted Backdoor Attacks, Clean-Label Data Poisoning, LLM Data Poisoning, Universal Jailbreak Backdoors (Poisoned HF) , Poisoning During Instruction Tuning, Fine-tuning Aligned LMs Compromises Safety,Persistent Pre-Training Poisoning of LLMs, Winter Soldier: Indirect Pre-Training Poisoning.
  Model Collapse Self-Consuming Generative Models Go MAD , Model Collapse Demystified: The Case of Regression, Larger Models Amplify Model Collapse, Is Model Collapse Inevitable?

Tue 10/28 – Data Poisoning Attacks

• Mini‑lecture: threat models, feasibility, and harms of poisoning. [slides]
• Discussion: preventing data poisoning (process & technical mitigations).
• Student Presentations: #4 Adversarially Robust Training [slides]

Thu 10/30 – Model Collapse

• Mini‑lecture: data poisoning attacks (Part 2) and training on synthetic data [slides]
• Student Presentations: #8 Model Collapse [slides]; #9 Preventing Collapse. [slides]

Homework (due Saturday):

• HW5: Poison your ML models with a backdoor. [pikachu.png]

Reading (due before Thursday lecture):

• Required: Extracting Training Data from ChatGPT, Section 1 and a chosen subsection of Section 2 in Foundation Models and Fair Use, Law-Zero Initiative by Yoshua Bengio
  
• Optional: Data Privacy risks Membership Inference Attacks (2016), MIA on Large Language Models, Data Reconstruction from Diffusion Models, Extracting Training Data from LLMs, Differential Privacy: Non-Technical Primer
  Copyright How Generative AI Turns Copyright Upside Down, Carlini blog: Privacy, Copyright & Generative AI, Provable Copyright Protection, Can Copyright Be Reduced to Privacy?
  On Existential Risks The future of AI with Stuart Russell, Essay: AI is an existential-threat, just not the way you think, Essay: Failed Strategy of AI Doomers, Chain-of-Thought Monitorability
  

Tue 11/4 – Data Privacy & Copyright

• Mini‑lecture: differential privacy. [slides]
• Student Presentations: #10 Privacy attack [slides]; #11 Does differential privacy solve copyright? [slides]

Thu 11/6 – Existential Threats of AI

• Mini‑lecture: existential risk landscape. [slides]
• AI Doomers vs Boomers. Student presentations: #12 AI is an existential threat and it's coming quicker than you think. [slides]
• Discussion: What are your opinions (after learning how GenAI works)?

Homework (due Saturday):

• HW6: Membership Inference Attack.

Reading (due before Thursday lecture):

• Required: CNN-generated images (GAN / DeepFake) are surprisingly easy to spot, DetectGPT
• Optional: Are AI Text Detectors reliable? A practical examination of AI-Generated Text detectors for LLMs, GPT detectors are biased" More work on AI Image detectors: static features with linear probing or kNN works more robustly, Diffusion-Reconstruction Errors, Neighboring Pixel Relationships, Using CLIP features.

Tue 11/11 – Veterans Day

• No class.

Thu 11/13 – AI Detectors

• Mini‑lecture: distinguishing AI from human text/media. [slides]
• Student Presentation: #14 State-of-the-Art AI detectors. [slides]

Homework (due Saturday):

• HW7: Build an AI classifier, and apply your adversarial attacks to break it. [Test DeepFake Image]

Reading (due before Thursday lecture):

• Required: Paper by Kirchenbauer et al; Blog by Scott Aaronson
• Optional: "Invisible Watermarks are Provably Removable Using GenAI", Provable Robust Watermark, On the reliability of watemarks for LLMs, Watermarking in the sand, Stable Signature (Image watermark by Meta), SoK: Watemarking for AI-Generated Content, SynthID paper

Tue 11/18 – Watermarking LLMs (Part 1)

• Mini‑lecture: history of watermarking; modern approaches. [Slides]
• Discussion: pros/cons; where watermarking fits in the safety stack.

Thu 11/20 – Watermarking LLMs (Part 2)

• Mini‑lecture: distortion-free watermarks. [Slides]
• Student Presentation: #15 "On the reliability of Watermarks for LLMs"

Homework (due Saturday):

• HW8: "LLM" watermarking from scratch.

Reading (due before Thursday lecture):

• Required: Protecting Language Generation Models via Invisible Watermarking
• Optional: First place solution to watermarking attack competition, What Lies Ahead for Generative AI Watermarking (Legal / Policy angle), Provably robust atermarks for Open-Source Language Models

Tue 11/25 – Beyond AI Content Watermarking

• Mini‑lecture: Undetectable watermarks and beyond [slides].
• Student presentation: #16 "Watermarking in the Sand" [ slides ]
  #17 Watermarked detected, but so what? Technical vs regulatory concerns [slides]

Thu 11/27 – Thanksgiving

• No class.

Homework:

• No new HW assigned due to the holiday (use time for projects).

Reading:

• No readings assigned; focus on finalizing projects.

Tue 12/2 – Instructor at NeurIPS

• No class.

Thu 12/4 – Quiz.

• In-class closed-book quiz on the course materials.

Mini-Symposium on GenAI Safety (Project Presentations)

• Adam Dziedzic and Franziska giving keynote (CISPA Helmholtz Center for Information Security)
• Team presentations: 15 minutes + 5 minutes Q&A each team.

Project report

• Submit final report by the end of the day.