This course focuses on empirical research methods, with the overall goal of the course to prepare students for performing research that relies upon measurement. We will focus on empirical research in the systems, networking, and security areas. As a result, we will cover various forms of network measurement, system measurement, and Internet crawling, and we will cover a wide variety of methods for data collection, cleaning, and analysis. The course entails reading papers, engaging in discussions, and performing a group project on a topic in the area.
| Week 1 | Week 2 | Week 3 | Week 4 | Week 5 | Week 6 | Week 7 | Week 8 | Week 9 | Week 10 | Final |
|
1/7 1/9 |
1/14 1/16 |
1/21 1/23 |
1/28 1/30 |
2/4 2/6 |
2/11 2/13 |
2/18 2/20 |
2/25 2/27 |
3/4 3/6 |
3/11 3/13 | 3/18 |
| Tue 1/7 |
|
| Thu 1/9 |
David Moore, Geoffrey M. Voelker, Stefan Savage. Inferring Internet Denial of Service Activity, In Proceedings of the 10th USENIX Security Symposium, August 2001, pp. 9–22. Optional paper: David Moore, Colleen Shannon, Doug Brown, Geoffrey M. Voelker, Stefan Savage. Inferring Internet Denial-of-Service Activity, ACM Transactions on Computer Systems, May 2006 24(2):115-139. |
| Tue 1/14 |
David Moore, Colleen Shannon, Jeffery Brown. Code-Red: a case study on the spread and victims of an Internet worm, In Proceedings of the 2nd ACM SIGCOMM Workshop on Internet Measurment (IMW'02), November 2002, pp. 273–284. Optional papers: David Moore, Vern Paxson, Stefan Savage, Colleen Shannon, Stuart Staniford, Nicholas Weaver. Inside the Slammer Worm, IEEE Security and Privacy 1(4):33–39, July 2003. (If you're going to read one of the optional papers, and just one, read this short one.) David Moore, Colleen Shannon, Geoffrey M. Voelker, Stefan Savage. Internet Quarantine: Requirements for Containing Self-Propagating Code, In Proceedings of the IEEE Infocom Conference, April 2003, pp. 1901–1910. |
| Thu 1/16 |
Yu-Chung Cheng, John Bellardo, Péter Benkö, Alex C. Snoeren, Geoffrey M. Voelker, Stefan Savage. Jigsaw: Solving the Puzzle of Enterprise 802.11 Analysis, In Proceedings of the ACM SIGCOMM Conference, September 2006, pages 39–50. Optional paper: Yu-Chung Cheng, Mikhail Afanasyev, Patrick Verkaik, Péter Benkö, Jennifer Chiang, Alex C. Snoeren, Stefan Savage, Geoffrey M. Voelker. Automating Cross-Layer Diagnosis of Enterprise Wireless Networks, In Proceedings of the ACM SIGCOMM Conference, August 2007. |
| Tue 1/21 |
Chris Kanich, Kirill Levchenko, Brandon Enright, Geoffrey M. Voelker, Stefan Savage. The Heisenbot Uncertainty Problem: Challenges in Separating Bots from Chaff, In Proceedings of the First USENIX Workshop on Large-scale Exploits and Emergent Threats (LEET), April 2008, pp. 10:1–9. Christian Kreibich, Chris Kanich, Kirill Levchenko, Brandon Enright, Geoffrey M. Voelker, Vern Paxson, Stefan Savage. On the Spam Campaign Trail, In Proceedings of the First USENIX Workshop on Large-scale Exploits and Emergent Threats (LEET), April 2008, pp. 1:1–9. |
| Thu 1/23 |
Chris Kanich, Christian Kreibich, Kirill Levchenko, Brandon Enright, Vern Paxson, Geoffrey M. Voelker, and Stefan Savage. Spamalytics: an Empirical Analysis of Spam Marketing Conversion, In Proceedings of the ACM Conference on Computer and Communications Security (CCS), October 2008, pp. 3–14. |
| Tue 1/28 |
Read one of: Tadayoshi Kohno, Yasemin Acar, Wulf Loh. Ethical Frameworks and Computer Security Trolley Problems: Foundations for Conversations, In Proceedings of the 32nd USENIX Security Symposium, August 2023, pp. 5145–5162. Florian Hantke, Sebastian Roth, Rafael Mrowczynski, Christine Utz, Ben Stock. Where Are the Red Lines? Towards Ethical Server-Side Scans in Security and Privacy Research, In Proceedings of the IEEE Symposium on Security and Privacy (S&P), May 2024.Skim (two pages): Ben Jones, Roya Ensafi, Nick Feamster, Vern Paxson, Nick Weaver. Ethical Concerns for Censorship Measurement, In Proceedings of the 2015 ACM SIGCOMM Workshop on Ethics in Networked Systems Research (NS Ethics), August 2015, pp. 17–19.Skim these stories/blogs: Stoica, Spamalytics Revisited, Bitdefender, November 2008. (just the bit at the end) |
| Thu 1/30 |
Zakir Durumeric, Eric Wustrow, J. Alex Halderman. ZMap: Fast Internet-Wide Scanning and its Security Applications, In Proceedings of the 22nd USENIX Security Symposium, August 2013, pp. 605–620. Optional paper: Zakir Durumeric, David Adrian, Phillip Stephens, Eric Wustrow, J. Alex Halderman. Ten Years of ZMap, In Proceedings of the ACM Internet Measurement Conference, November 2024, pp. 139–148. |
| Tue 2/4 |
Zakir Durumeric, Frank Li, James Kasten, Nicholas Weaver, Johanna Amann, Jethro Beekman, Mathias Payer, David Adrian, Vern Paxson, Michael Bailey, J. Alex Halderman. The Matter of Heartbleed, In Proceedings of the ACM Internet Measurement Conference, November 2014, pp. 475–488. Skim: Frank Li, Zakir Durumeric, Jakub Czyz, Mohammad Karami, Damon McCoy, Stefan Savage, Michael Bailey, Vern Paxson. You've Got Vulnerability: Exploring Effective Vulnerability Notifications, In Proceedings of the 25th USENIX Security Symposium, August 2016, pp. 1033–1050. |
| Thu 2/6 |
Muhammad Ali, Piotr Sapiezynski, Miranda Bogen, Aleksandra Korolova, Alan Mislove, Aaron Rieke. Discrimination through Optimization: How Facebook's Ad Delivery Can Lead to Biased Outcomes, In Proceedings of the ACM on Human-Computer Interaction, November 2019, pp. 1–30. Guest lecture by Alan Mislove Optional papers: Levi Kaplan, Nicole Gerzon, Alan Mislove, and Piotr Sapiezynski. Measurement and Analysis of Implied Identity in Ad Delivery Optimization, In Proceedings of the ACM Internet Measurement Conference (IMC'22), October 2022, pp. 195–209. Muhammad Ali, Piotr Sapiezynski, Aleksandra Korolova, Alan Mislove, and Aaron Rieke. Ad Delivery Algorithms: The Hidden Arbiters of Political Messaging, In Proceedings of the ACM International Conference on Web Search and Data Mining (WSDM'21), February 2021, pp. 13–21. |
| Tue 2/11 |
Zesen Zhang, Alexander Marder, Ricky Mok, Bradley Huffaker, Matthew Luckie, kc Claffy, Aaron Schulman. Inferring Regional Access Network Topologies: Methods and Applications, In Proceedings of the ACM Internet Measurement Conference (IMC'21), November 2021, pp. 720–738. Nishant Bhaskar, Maxwell Bland, Kirill Levchenko, Aaron Schulman. Please Pay Inside: Evaluating Bluetooth-based Detection of Gas Pump Skimmers, In Proceedings of the 28th USENIX Security Symposium, August 2019, pp. 373–388. Guest lecture by Aaron Schulman |
| Thu 2/13 |
Marti Motoyama, Kirill Levchenko, Chris Kanich, Damon McCoy, Geoffrey M. Voelker, Stefan Savage. Re: CAPTCHAs — Understanding CAPTCHA-Solving from an Economic Context, In Proceedings of the 19th USENIX Security Symposium, August 2010, pp. 435–452. |
| Tue 2/18 |
Sarah Meiklejohn, Marjori Pomarole, Grant Jordan, Kirill Levchenko, Damon McCoy, Geoffrey M. Voelker, Stefan Savage. A Fistful of Bitcoins: Characterizing Payments Among Men with No Names, In Proceedings of the ACM Internet Measurement Conference (IMC'13), October 2013, pp. 127–140. Optional papers: George Kappos, Haaroon Yousaf, Rainer Stütz, Sofia Rollet, Bernhard Haslhofer, Sarah Meiklejohn. How to Peel a Million: Validating and Expanding Bitcoin Clusters, In Proceedings of the USENIX Security Symposium, August 2022, pp. 2207–2223. |
| Thu 2/20 |
Zakir Durumeric, Zane Ma, Drew Springall, Richard Barnes, Nick Sullivan, Elie Bursztein, Michael Bailey, J. Alex Halderman, Vern Paxson. The Security Impact of HTTPS Interception, In Proceedings of the Network and Distributed System Security Symposium (NDSS), February 2017. |
| Tue 2/25 |
Kirill Levchenko, Andreas Pitsillidis, Neha Chachra, Brandon Enright, Márk Félegyházi, Chris Grier, Tristan Halvorson, Chris Kanich, Christian Kreibich, He Liu, Damon McCoy, Nicholas Weaver, Vern Paxson, Geoffrey M. Voelker, Stefan Savage. Click Trajectories: End-to-End Analysis of the Spam Value Chain, In Proceedings of the IEEE Symposium on Security and Privacy, May 2011, pp. 431–446. Optional papers: Damon McCoy, Hitesh Dharmdasani, Christian Kreibich, Geoffrey M. Voelker, and Stefan Savage. Priceless: The Role of Payments in Abuse-advertised Goods, In Proceedings of the ACM Conference on Computer and Communications Security (CCS), October 2012, pp. 845–856. |
| Thu 2/27 |
S. Shunmuga Krishnan, Ramesh K. Sitaraman. Video Stream Quality Impacts Viewer Behavior: Inferring Causality Using Quasi-Experimental Designs, In Proceedings of the ACM Internet Measurement Conference (IMC), November 2012, pp. 211–224. |
| Tue 3/4 |
Devdatta Akhawe, Adrienne Porter Felt. Alice in Warningland: A Large-Scale Field Study of Browser Security Warning Effectiveness, In Proceedings of the USENIX Security Symposium, August 2013, pp. 257–272. Optional papers: Adrienne Porter Felt, Robert W. Reeder, Hazim Almuhimedi, Sunny Consolvo. Experimenting At Scale With Google Chrome’s SSL Warning, In Proceedings of the 32nd SIGCHI Conference on Human Factors in Computing Systems, April 2014, pp. 2667–2670. Adrienne Porter Felt, Alex Ainslie, Robert W. Reeder, Sunny Consolvo, Somas Thyagaraja, Alan Bettes, Helen Harris, Jeff Grimes. Improving SSL Warnings: Comprehension and Adherence, In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, April 2015, pp. 2893–2902. |
| Thu 3/6 |
Neha Chachra, Damon McCoy, Stefan Savage, Geoffrey M. Voelker. Empirically Characterizing Domain Abuse and the Revenue Impact of Blacklisting, In Proceedings of the Workshop on the Economics of Information Security (WEIS), June 2014, pp. 4:1:1–4:1:13. Optional papers: David Wang, Matthew Der, Mohammad Karami, Lawrence Saul, Damon McCoy, Stefan Savage, Geoffrey M. Voelker. Search + Seizure: The Effectiveness of Interventions on SEO Campaigns, In Proceedings of the ACM Internet Measurement Conference (IMC), November 2014, pp. 359–372.[Slides] |
| Tue 3/11 |
Christina Yeung, Tadayoshi Kohno, Franziska Roesner. Analyzing the (In)Accessibility of Online Advertisements, In Proceedings of the ACM Internet Measurement Conference (IMC), November 2024, pp. 92–106. |
| Thu 3/13 |
Le Chen, Alan Mislove, Christo Wilson. Peeking Beneath the Hood of Uber, In Proceedings of the ACM Internet Measurement Conference (IMC), November 2015, pp. 495–508. |
| Tue 3/18 |
|