Usability of Programming Languages

CSE 291A00 / 190A00, Fall 2023

Programmers of all kinds express their ideas using programming languages. Unfortunately, languages can be hard to use correctly, resulting in lengthy development times and buggy software. How can these languages be designed to make programmers as effective as possible?

In this course, we will learn techniques to analyze and improve the usability of programming languages. Students will apply these techniques to languages of current and historical interest, and in the process, expand their knowledge of different ways to design languages. This course is intended as preparation for conducting independent research on the usability of programming languages, and will include homework assignments as well as a project.



Michael Coblenz, Assistant Professor

Office hours: Fridays 3-4 PM, CSE 3246. Office hours are also available by appointment.


Shaokang Jiang, MS Student

Office hours: Mondays 9:30-10:30, CSE B270a

CSE190 vs. CSE291

I’m offering this course as a 190 (in addition to as a 291) because I believe that many undergraduates are prepared to do research in this area, and I want to encourage interested undergraduates to pursue research that interests them. Graduate courses are not necessarily any harder than undergraduate courses! However, they tend to be deeper and narrower: rather than giving an overview of a broad area, they enable you to develop expertise in a focused field of study. Compared to typical undergraduate courses, this course requires substantially more independence. Please talk to me if you are unclear about any of the expectations.

Students registered for the two courses will have the same set of assignments and deadlines. However, I reserve the right to choose different grade cutoffs for students enrolled in the two courses.

Course Learning Outcomes

Some students in the course are research-focused, and are primarily interested in learning how to do research on usability of programming languages. For you, my goal is to prepare you to do high-quality, impactful research. Others are practice-focused, and are primarily interested in future careers in the software industry. For you, I hope to provide insight and background that will help you make wise choices (which language or tools should my team use for this project?); and to read and critically analyze scientific literature in this area so you are ready to take advantage of future discoveries and innovations. Both kinds of students are valued participants in the class.

Upon completion of this course, students will be able to:

  1. Apply qualitative and quantitative research methods to obtain insights about programming language design choices;
  2. Critically analyze design questions in the context of languages and programming systems;
  3. Read and interpret research papers in the area of usability of programming languages, and summarize major findings to date.

Course Format

The course will generally be conducted in person. Please do not come to class if you may be sick. Please contact me in advance and I will provide a Zoom link that you can use. However, due to the instructor’s conference travel schdule this quarter, selected classes will be held over Zoom. You may attend in the classroom as usual or in a place of your choice.

The course is about using research methods to study the impact of programming language design decisions on people. Therefore, the course is divided into three main segments.

  • First, we will study how to use methods from human-computer interaction research to obtain insights about the effects of language design decisions on programmers and software engineers. During this time, the homework will focus on reading papers and book chapters. To encourage thorough study of the readings, the readings will be accompanied by reading responses, which must be submitted before we discuss the readings in class.
  • Understanding programming language design tradeoffs requires some formal training in the theory of programming languages. Although the theory of programming languages is not the main focus of the course, we will cover a small amount of theory to facilitate thought and discussion.
  • In the third segment of the class, we will study results of applying these and other methods to familiarize ourselves with what is already known about the usability of programming languages. During this time, you will work with a partner to conduct a preliminary research project according to your own interests. You will present your results to the class at the end of the quarter.

I have made readings optional in cases where I expect to cover the material thoroughly in class. Those readings might be used for exam study, for reference while doing homework, or to provide an alternative explanation of the material when needed.

The class will have a midterm exam covering the research methods and theory taught in the class. There is a room reserved for a final exam; in lieu of a traditional exam, students will present their results from their research projects.

Assignments will be submitted on Gradescope; grades will be tracked on Canvas.

On Reading Papers

This class will involve reading a significant number of papers. Reading an academic paper is very different from reading a textbook chapter; since some of you have not read academic papers before, here is some advice to consider.

  Textbook chapter Academic paper
Reading order Start at the beginning, finish at the end Read the whole thing, but use random access to enhance understanding and focus your reading
Comprehension expected Everything (it could be on a test) Main ideas; answer your own questions; identify areas for future study
Comprehension strategies when confused Read all previous chapters first Write down questions in the margins, continue reading, and re-visit questions after seeing more of the paper
Schedule Read the night before class, or after class, or not at all (instructor will present content in slides anyway) Read in advance and come to office hours with questions


Readings will be posted on this web site. Many of the readings will be academic papers that have been published in various venues. They are all available for free to the UCSD community. If you click a link to a reading assignment and get a page asking you to pay to download it, try again via the UCSD library’s VPN. Do not pay any fees to download the readings; feel free to ask for help instead.

Some readings may be from Research Methods in Human-Computer Interaction by Lazar, Feng, and Hochheiser. The full text of the book is available for free via the UCSD library. A copy is available on course reserves at the library for students who prefer the paper version. This book may be a useful reference as you work on your assignments and projects and as you study for the midterm exam.

The theory of programming languages is covered in Types and Programming Languages by Benjamin C. Pierce (available to UCSD students for free online through the UCSD library).

Rose Bohrer’s Human-Centered Programming Languages textbook includes a variety of useful readings. Bohrer’s textbook is designed as a general introduction to programming languages rather than as a focused introduction to usability of programming languages.

You may wonder why my PLIERS paper is not on the reading list. The answer is that the entire course is about PLIERS! You may find it useful, however, as reference material.


Assignment Weight Due Date
Project 45% As indicated per component
Theory 5% 11/13
Reading responses 30% Before lecture
Midterm Exam 20% 10/27 in class

Reading responses will be graded on a check minus/check/check plus scale. Check plus grades will be reserved for exceptionally deep work. When assigning final grades, check minus will be worth 75%, check will be worth 85%, and check plus will be worth 100%. We will drop the lowest reading response grade when computing grades.

Grades will be given according to the following scale:

  • A: grade >= 90%
  • B: 80% <= grade < 90%
  • C: 70% <= grade < 80%
  • D: 60% <= grade < 70%
  • F: grade < 60%

The instructor has discretion to assign + and - grades within those ranges and to decrease the thresholds for grades if the overall grades are lower than expected.

Academic Integrity Policy

You will complete the project in small groups (generally two or three students per group). All collaboration is permitted and encouraged within groups, with the exception of very small individual components, which will be clearly marked.

For the remaining assignments, this class uses a whiteboard discussion + reported collaboration policy. You may discuss the readings, programming assignments, and projects with your classmates, course staff, and AI tools such as ChatGPT in front of a whiteboard. In fact, such collaboration is encouraged, since it may help you deepen your understanding of the material. After your discussions, you should erase the whiteboard and write your own solutions yourself. Submitting text written by other entities (both human and AI) will be considered an academic integrity violation.

Report the names of those with whom you collaborated (except course staff) on your submission.

Attendance and Participation

There is no textbook for the class, so class attendance will be critical to your success. Copies of the lecture slides will not suffice for understanding the material. This class is about critical thinking and research methods; you will not be able to learn these if you do not attend class.


The project represents an opportunity to investigate a research question of your choice using one or more research methods discussed in class. The research question should be relevant to programming languages, broadly construed. This means that the question might pertain only to a language, or it might pertain to a combination of a programming language and environment (e.g., an IDE), or it might focus on the IDE itself.

Projects should be completed in teams of 2-3 students. If you have a project idea you are excited about, I encourage you to recruit one or two other students to help you.

Your project should include a component of data collection (from users) and an analysis of the data. If you want to build a system or tool to analyze, be sure to scope the project so that very little development work is required. I expect most projects will concern an evaluation of a tool or design that already exists. For this class, it suffices to conduct pilot studies. For these studies, you can recruit your friends or classmates, and you do not need IRB (ethics board) approval to conduct pilot studies in class projects. However, if you would like your study to be publishable, please talk to me as soon as possible about submitting an IRB protocol for your project.

I have observed in the past that students tend to initially choose questions that are too big to investigate in one study. For example: comparing functional vs. object-oriented programming would not be a good choice because there are too many differences between the two paradigms and “programming” is too vague. Instead, focus on something very specific. For example, you could look at the use of currying in functional programming. Studying currying addresses a fragment of your initial question and is specific enough that you could run a study.

If you have a pre-existing topic and would like to continue working on it, you are welcome to do so as long as you can choose a component of the work that is well-scoped. For example, do not say in your project plan “We will continue our previous work with project X and we hope to make progress.” Instead, say: “Previously, we did X. Now, we propose to continue by doing Y, which we expect to complete by the end of the quarter.” There might be lots more to do on the project beyond Y, but your proposal should concern something that you can complete during the quarter.

Once you have identified the research question, you should plan how you will gather data to address your question: what artifacts or processes will you observe? How will you study those artifacts or processes in a structured way? Then, you should have a plan for analyzing the data and drawing conclusions.

The best way to do good research is to iterate on your ideas with feedback, so they project plan can be revised and resubmitted once for a regrade. The majority of the credit will depend on the quality of your overall project: to what extent have you designed and executed a study to faithfully answer a research question about a programming language or environment?

Grading for the project, which overall represents 35% of the grade, will be broken down as follows. The proposal and plan are identical in structure; the idea is that the proposal is a low-stakes opportunity to receive feedback on the plan before executing it.

Component Weight Due Date
Team matching survey 0% 10/4
Meet with instructor to discuss project ideas 5% 10/18
Project plan 35% (revisions permitted for regrade) 10/25
Project checkpoint 15% 11/17
Project report 35% 12/8
Project presentation 10% 12/12


All assignments may be found and submitted on Gradescope.

Course Schedule

I will try to avoid unnecessary changes to the schedule. However, I will post updates as needed to respond to arising circumstances and student needs. The date of the project presentations is fixed with the registrar, so it will not move.

Due Slides
9/29 Introduction to usability: definition, methods overview. (none)   PDF
10/2 Research methods intro Andreas Stefik and Stefan Hanenberg. 2014. The Programming Language Wars: Questions and Responsibilities for the Programming Language Community. In Onward! 2014. DOI. Optional reading: Chapter 10 of “Research Methods in HCI.” Reading response PDF
10/4 (instructor at VL/HCC; class will be held on Zoom) Designing and running usability studies Brad A. Myers, Amy J. Ko, Thomas D. LaToza, YoungSeok Yoon. Programmers Are Users Too: Human-Centered Methods for Improving Programming Tools. PDF Team matching survey; Reading response PDF
10/6 (instructor at VL/HCC; Shaokang will lead discussion) Usability study example A Multimodal Study of Challenges Using Rust Reading response  
10/9 Qualitative methods: interviews and surveys Optional: Lazar, Feng and Hochheiser: Interviews and Focus Groups; Surveys PDF  
10/11 Qualitative methods: thematic analysis, grounded theory Optional: Braun and Clarke. Using Thematic Analysis in Psychology   PDF; coding practice (from Sara Kiesler’s course)
10/13 Qualitative methods example Justin Lubin and Sarah E. Chasins. 2021. How statically-typed functional programmers write code. OOPSLA. DOI Reading response PDF
10/16 Quantitative methods 1 Optional: Amy J. Ko, Thomas D. LaToza, and Margaret M. Burnett. A practical guide to controlled experiments of software engineering tools with human participants. Empirical Software Engineering. PDF   PDF
10/18 Quantitative methods 2 Read Ko et al. (above) if you haven’t already done so   PDF
10/20 (instructor at SPLASH) Shaokang leads discussion of autocomplete study   Experimental design and analysis  
10/23 (Class on Zoom: instructor at SPLASH) Cognitive Dimensions of Notations Thomas R. G. Green and Marian Petre. 1996. Usability analysis of visual programming environments: a ‘cognitive dimensions’ framework. Journal of Visual Languages & Computing 7, 2 (1996), 131–174. ScienceDirect Reading response PDF
10/25 (instructor at SPLASH; Shaokang will facilitate) Peer project feedback   Project plan  
10/27 (instructor at SPLASH) Research methods midterm      
10/30 Theory 1      
11/1 Theory 2      
11/3 Theory 3      
11/6 Theory 4      
11/8 Corpus Studies Beckman, Nels E., Duri Kim, and Jonathan Aldrich. “An empirical study of object protocols in the wild.” European Conference on Object-Oriented Programming. 2011. PDF Reading Response PDF
11/10 Veteran’s Day (no class)      
11/13 Discuss and give feedback on group projects      
11/15 Argument type checking RCT L. Prechelt and W. F. Tichy, “A controlled experiment to assess the benefits of procedure argument type checking,” TSE. Apr. 1998, IEEE Reading response PDF
11/17 Gender HCI: programming tools and gender Margaret Burnett, Anicia Peters, Charles Hill, and Noha Elarief. Finding Gender-Inclusiveness Software Issues with GenderMag: A Field Investigation. CHI 2016. PDF Project checkpoint; Reading response PDF
11/20 Rust Michael Coblenz, Michelle L. Mazurek, and Michael Hicks. 2022. Garbage collection makes Rust easier to use: a randomized controlled trial of the Bronze garbage collector. ICSE 2022. DOI Theory homework; Reading response
11/22 Natural programming Brad A. Myers, John F. Pane, and Amy J. Ko. 2004. Natural programming languages and environments. CACM. DOI Reading response
11/24 No class (Thanksgiving)      
11/27 Static vs. Dynamic types Endrikat, Hanenberg, Robbes, Stefik, “How do API documentation and static typing affect API usability?”, ICSE 2014. ACM Reading response
11/29 Technical approaches to usability: gradual typing Video: ICFP 2018 keynote address, Ron Garcia: Gradual Typing Reading response  
12/1 Program Comprehension and Code Complexity Dror G. Feitelson. From Code Complexity Metrics to Program Comprehension. CACM. ACM Reading response  
12/4 Build It, Break It, Fix It: Contesting Secure Development James Parker, Michael Hicks, Andrew Ruef, Michelle L. Mazurek, Dave Levin, Daniel Votipka, Piotr Mardziel, and Kelsey R. Fulton. Build It, Break It, Fix It: Contesting Secure Development. TOPS. ACM Reading response  
12/6 Spreadsheets Choose one of two papers: Errors in Operational Spreadsheets or Detecting and refactoring code smells in spreadsheet formulas Reading response  
12/8 PLIERS: putting it all together   Project report  
12/12, 11:30 AM - 2:30 PM Project presentations   Project presentations  

The “TBD” topics at the end of the quarter are there to allow me to adjust the content according to student needs and interests. Please contact the instructor if there’s a particular topic you’re hoping to see!

Inclusion Statement

A key aspect of inclusion is adapting to individual circumstances. If you need an extension on an assignment, please ask. We cannot offer extensions on reading assignments, but we will be reasonable with extensions on other assignments if you are reasonable with us. Possible reasons for extensions include illness, family emergencies, or anticipated scheduling problems (such as a job interview, conference attendance, or a confluence of deadlines in other classes). We will reward self-awareness of personal schedule constraints by granting extensions on a case-by-case basis.

We are committed to fostering a learning environment for this course that supports a diversity of thoughts, perspectives and experiences, and respects your identities (including race, ethnicity, heritage, gender, sex, class, sexuality, religion, ability, age, educational background, etc.).  Our goal is to create a diverse and inclusive learning environment where all students feel comfortable and can thrive. 

Our instructional staff will make a concerted effort to be welcoming and inclusive to the wide diversity of students in this course.  If there is a way we can make you feel more included please let one of the course staff know, either in person, via email, or even in a note under the door.  Our learning about diverse perspectives and identities is an ongoing process, and we welcome your perspectives and input. 

We also expect that you, as a student in this course, will honor and respect your classmates, abiding by the UCSD Principles of Community.  Please understand that others’ backgrounds, perspectives and experiences may be different than your own, and help us to build an environment where everyone is respected and feels comfortable. If you experience any sort of harassment or discrimination, please contact the instructor as soon as possible.   If you prefer to speak with someone outside of the course, please contact the Office of Prevention of Harassment and Discrimination.  

Students with Disabilities

We aim to create an environment in which all students can succeed in this course.  If you have a disability, please contact the Office for Students with Disability (OSD), which is located in University Center 202 behind Center Hall, to discuss appropriate accommodations right away.  We will work to provide you with the accommodations you need, but you must first provide a current Authorization for Accommodation (AFA) letter issued by the OSD.  You are required to present their AFA letters to Faculty (please make arrangements to contact me privately) and to the OSD Liaison in the department in advance so that accommodations may be arranged.

Basic Needs/Food Insecurities

If you are experiencing any basic needs insecurities (food, housing, financial resources), there are resources available on campus to help, including The Hub and the Triton Food Pantry.  Please visit The Hub for more information.

Course Policies

Participation Policy

Learning to think critically about programming languages’ impact on people is a key learning goal of this course. I encourage active participation in class. Ask questions. Challenge results and methods as well as the motivation of the specific research questions. Design and research are both all about tradeoffs; there are no perfect studies, so your goal should be able to see both the strengths and weaknesses of each possible approach.

Health and Well-Being Statement

Health and well-being are even more important than the content in the class. Please contact the course staff if you need an accommodation for a health- or well-being-related reason. Extensions will be granted for reasons including: personal or family mental or physical health; family emergencies; religious observances; and job-related interview travel. Please talk to the instructor or TA as early as possible so we can help you. Please do not come to class if you are sick. If you contact the instructor, we can likely arrange for you attend that day’s class via Zoom. This should only be used in case of illness, since participating in class discussions via Zoom is very challenging. Subject to Change Policy This syllabus is subject to change! I will continually re-visit the schedule and topics to align the course with student needs. I will keep the grading system as stable as possible but reserve the right to change or reschedule readings to adapt to student needs.

Letter of Recommendation Policy

To request a letter of recommendation, please contact me at least two weeks before the letter is needed. You should include in your request a description of the position or program to which you are applying; your resume or CV; and an explanation of why you are pursuing the position or program to which you are applying.

Technology Policy

The use of devices with screens in a classroom can be distracting to other students, and has been found to reduce class learning. However, I recognize that some students normally take notes on devices. Therefore, laptops, tablets, and phones are not permitted in class except for note-taking, emergencies, and for disability accommodations.

Harassment and Discrimination

Harassment and discrimination will not be tolerated either in this course or at UCSD in general. If you experience harassment or discrimination, especially in the context of this course, we hope you talk to us so that we can help. Please know that most UCSD faculty and staff, including myself, are considered “responsible employees” and are required to report any incidents of sexual violence or sexual harassment to the Office for the Prevention of Harassment and Discrimination.