This course focuses on empirical research methods, with the overall goal of the course to prepare students for performing research that relies upon measurement. We will focus on empirical research in the systems, networking, and security areas. As a result, we will cover various forms of network measurement, system measurement, and Internet crawling, and we will cover a wide variety of methods for data collection, cleaning, and analysis. The course entails reading papers, engaging in discussions, and performing a group project on a topic in the area.
| Week 1 | Week 2 | Week 3 | Week 4 | Week 5 | Week 6 | Week 7 | Week 8 | Week 9 | Week 10 | Final |
|
3/31 4/2 |
4/7 4/9 |
4/14 4/16 |
4/21 4/23 |
4/28 4/30 |
5/5 5/7 |
5/12 5/14 |
5/19 5/21 |
5/26 5/28 |
6/2 6/4 | 6/8 |
| Tue 3/31 |
|
| Thu 4/2 |
David Moore, Geoffrey M. Voelker, Stefan Savage. Inferring Internet Denial of Service Activity, In Proceedings of the 10th USENIX Security Symposium, August 2001, pp. 9–22. Optional paper: David Moore, Colleen Shannon, Doug Brown, Geoffrey M. Voelker, Stefan Savage. Inferring Internet Denial-of-Service Activity, ACM Transactions on Computer Systems, May 2006 24(2):115-139. |
| Tue 4/7 |
David Moore, Colleen Shannon, Jeffery Brown. Code-Red: a case study on the spread and victims of an Internet worm, In Proceedings of the 2nd ACM SIGCOMM Workshop on Internet Measurment (IMW'02), November 2002, pp. 273–284. Optional papers: David Moore, Vern Paxson, Stefan Savage, Colleen Shannon, Stuart Staniford, Nicholas Weaver. Inside the Slammer Worm, IEEE Security and Privacy 1(4):33–39, July 2003. (If you're going to read one of the optional papers, and just one, read this short one.) David Moore, Colleen Shannon, Geoffrey M. Voelker, Stefan Savage. Internet Quarantine: Requirements for Containing Self-Propagating Code, In Proceedings of the IEEE Infocom Conference, April 2003, pp. 1901–1910. |
| Thu 4/9 |
Discussion about how to approach you measurement research projects [Slides] (see Slack)
|
| Tue 4/14 |
Yu-Chung Cheng, John Bellardo, Péter Benkö, Alex C. Snoeren, Geoffrey M. Voelker, Stefan Savage. Jigsaw: Solving the Puzzle of Enterprise 802.11 Analysis, In Proceedings of the ACM SIGCOMM Conference, September 2006, pages 39–50. Optional paper: Yu-Chung Cheng, Mikhail Afanasyev, Patrick Verkaik, Péter Benkö, Jennifer Chiang, Alex C. Snoeren, Stefan Savage, Geoffrey M. Voelker. Automating Cross-Layer Diagnosis of Enterprise Wireless Networks, In Proceedings of the ACM SIGCOMM Conference, August 2007. |
| Thu 4/16 |
Chris Kanich, Kirill Levchenko, Brandon Enright, Geoffrey M. Voelker, Stefan Savage. The Heisenbot Uncertainty Problem: Challenges in Separating Bots from Chaff, In Proceedings of the First USENIX Workshop on Large-scale Exploits and Emergent Threats (LEET), April 2008, pp. 10:1–9. Christian Kreibich, Chris Kanich, Kirill Levchenko, Brandon Enright, Geoffrey M. Voelker, Vern Paxson, Stefan Savage. On the Spam Campaign Trail, In Proceedings of the First USENIX Workshop on Large-scale Exploits and Emergent Threats (LEET), April 2008, pp. 1:1–9. |
| Tue 4/21 |
Chris Kanich, Christian Kreibich, Kirill Levchenko, Brandon Enright, Vern Paxson, Geoffrey M. Voelker, and Stefan Savage. Spamalytics: an Empirical Analysis of Spam Marketing Conversion, In Proceedings of the ACM Conference on Computer and Communications Security (CCS), October 2008, pp. 3–14. |
| Thu 4/23 |
Read one of: Tadayoshi Kohno, Yasemin Acar, Wulf Loh. Ethical Frameworks and Computer Security Trolley Problems: Foundations for Conversations, In Proceedings of the 32nd USENIX Security Symposium, August 2023, pp. 5145–5162. Florian Hantke, Sebastian Roth, Rafael Mrowczynski, Christine Utz, Ben Stock. Where Are the Red Lines? Towards Ethical Server-Side Scans in Security and Privacy Research, In Proceedings of the IEEE Symposium on Security and Privacy (S&P), May 2024.Skim (two pages): Ben Jones, Roya Ensafi, Nick Feamster, Vern Paxson, Nick Weaver. Ethical Concerns for Censorship Measurement, In Proceedings of the 2015 ACM SIGCOMM Workshop on Ethics in Networked Systems Research (NS Ethics), August 2015, pp. 17–19.Skim these stories/blogs: Stoica, Spamalytics Revisited, Bitdefender, November 2008. (just the bit at the end) |
| Tue 4/28 |
|
| Thu 4/30 |
Muhammad Ali, Piotr Sapiezynski, Miranda Bogen, Aleksandra Korolova, Alan Mislove, Aaron Rieke. Discrimination through Optimization: How Facebook's Ad Delivery Can Lead to Biased Outcomes, In Proceedings of the ACM on Human-Computer Interaction, November 2019, pp. 1–30. Guest lecture by Alan Mislove |
| Tue 5/5 |
|
| Thu 5/7 |
|
| Tue 5/12 |
Deepak Kumar, Jeff Hanbcock, Kurt Thomas and Zakir Durumeric. Understanding the Behaviors of Toxic Accounts on Reddit, In Proceedings of the ACM WWW Conference, April 2023. Optional paper: Deepak Kumar, Patrick Gage Kelley, Sunny Consolvo, Joshua Mason, Elie Bursztein, Zakir Durumeric, Kurt Thomas, Michael Bailey. Designing Toxic Content Classification for a Diversity of Perspectives, In Proceedings of the USENIX Symposium on Usable Privacy and Security, August, 2021. |
| Thu 5/14 |
|
| Tue 5/19 |
|
| Thu 5/21 |
|
| Tue 5/26 |
|
| Thu 5/28 |
|
| Tue 6/2 |
|
| Thu 6/4 |
|
| Mon 6/8 |
|