CSE 291-D Latent Variable Models (Spring 2017)

Course Description

Probabilistic machine learning models are powerful, flexible tools for data analysis. This course focuses on models with latent variables, which simplify complex data sets by discovering hidden patterns in the data. By the end of the course you will be able to apply a variety of probabilistic models for text data, social networks, computational biology, and more. You will learn how to formulate probabilistic models to solve the data science tasks you care about, derive inference algorithms for these models using Bayesian inference techniques, and evaluate their performance. Topics will include topic models, stochastic blockmodels and latent space models for social networks, Bayesian inference, Markov chain Monte Carlo, mean field variational inference, and nonparametric Bayesian models. This course continues from CSE250A.

Lecture time and venue: TuTh 6:30-7:50pm YORK 4080A.

Instructor: James Foulds
Instructor email: jfoulds [at] ucsd [dot] edu
Instructor office hours: TuTh 5-6pm Atkinson Hall 4401

Teaching assistant: Steven Hill
Teaching assistant office hours: Monday 10:30 - 11:30am, lounge outside 3140
Teaching assistant email: s5hill [at] eng [dot] ucsd [dot] edu

Grading: Homeworks 25% (5 of them, 6.25% each for your best 4 homeworks; the lowest score will be dropped), Group Project 35%, Final 35%, Participation 5%

Piazza: Sign up for this course at piazza.com/ucsd/spring2017/cse291d
Poll Everywhere: PollEv.com/jamesfoulds656 . Register your account for the course at www.polleverywhere.com/register?p=6bu49-iqvv&pg=dCc8YnS&u=wJdvD6sW , by week 2 in order to get participation credits.

Prerequisites

  • Requires CSE 250A, or consent of the instructor.
  • Required knowledge: the basics of probabilistic graphical models, as covered in CSE250A. The prerequisite knowlege for CSE250A also applies to this course: elementary probability, multivariable calculus, linear algebra, and basic programming ability in a high-level language such as C, Java, or Matlab.
  • Recommended preparation for those without required knowledge: Machine Learning: A Probabilistic Perspective (Murphy, 2012) Chapter 10, or Pattern Recognition and Machine Learning (Bishop, 2006) Chapter 8.

    • Required Textbooks

  • Machine Learning: A Probabilistic Perspective (Murphy, 2012) is the primary textbook. You will need this book for course readings. Until you obtain it, the UCSD library has an electronic desk copy (requires UCSD IP address, limited to one person at a time).
  • Information Theory, Inference, and Learning Algorithms (Mackay, 2003) is a secondary textbook that we will use for some topics. There is a free PDF download of the book from author's website. You do not need to purchase this book for this course, though I do recommend it, if you are thinking about buying it.

    • Syllabus

    Lecture Summary Details Assessment Required reading
    4/4/2017TuesdayOverview, motivating examples, recap of directed graphical modelsMotivating applications - computational social science, social networks, computational biology. Recap on d-separation/Bayes ball/explaining awayBlei, David M. (2014). Build, compute, critique, repeat: Data analysis with latent variable models. Annual Review of Statistics and Its Application, Sections 1-3
    4/6/2017ThursdayBayesian inferenceMonty hall problem, frequentist vs Bayes, MAP vs MLE, full posterior vs point estimates, posterior predictiveHW1 outRead "What is Bayesian Analysis?" at bayesian.org (scroll down, and be sure to follow the link to bayesian.org/Bayes-Explained). Then read Murphy Ch 3.1 - 3.2
    4/11/2017TuesdayGenerative models for discrete dataConjugate priors, beta/Bernoulli, Dirichlet/multinomial, urn process interpretations, naive Bayes document modelMurphy Ch 3.3 - 3.5.2
    4/13/2017ThursdayExponential families and generalized linear modelsExponential families, Bayesian inference for EFs, writing distributions in EF form, GLMsMurphy Ch 9.1 - 9.3
    4/18/2017TuesdayMonte Carlo methodsImportance sampling, rejection sampling, why they fail in high dimensionsMacKay Ch 29.1 - 29.3
    4/20/2017ThursdayMarkov chain Monte CarloGibbs sampling, Metropolis-HastingsHW1 due, HW2 outMacKay Ch 29.4 - 29.6
    4/25/2017TuesdayMixture models revisitedMixtures of Gaussians/connection to K-means, mixtures of multinomials, EM for mixtures of exponential family distributions, Bayesian inferenceProject proposal dueMurphy Ch 11.1 - 11.3
    4/27/2017ThursdayLatent linear modelsFactor analysis, probabilistic PCA, ICAMurphy Ch 12.1, 12.2.4 - 12.2.5, 12.6 - 12.6.1
    5/2/2017TuesdayHidden Markov Models revisitedApplications to part of speech tagging, Factorial HMMs, input/output HMMs, direct Gibbs sampling, Forward filtering backward sampling, collapsed MCMCMurphy Ch 17.3 - 17.4.2, 17.4.5, Gao, J. and Johnson, M. (2008). A comparison of Bayesian estimators for unsupervised Hidden Markov Model POS taggers. EMNLP, Sections 1-2 (don't worry too much about the variational inference part, since we haven't covered this yet)
    5/4/2017ThursdayEvaluating unsupervised modelsLog-likelihood on held-out data, posterior predictive checks, correlation with metadata, human evaluationHW2 due, HW3 outDave Blei's notes on posterior predictive checks
    5/9/2017TuesdayMarkov random fieldsSeparation criterion, Hammersley-Clifford theorem, Ising models.Murphy Ch 19, up to 19.4.1.
    5/11/2017ThursdayStatistical relational learning and probabilistic programmingBelief propagation. Markov logic networks, probabilistic soft logic, Stan/winBUGS Ch 20, up to 20.2.2. Domingos, P., Kok, S., Lowd, D., Poon, H., Richardson, M., Singla, P. (2008). Markov Logic. In L. De Raedt, P. Frasconi, K. Kersting and S. Muggleton (eds.), Probabilistic Inductive Logic Programming, Sections 1, 4, and 7.
    5/16/2017TuesdayVariational inferenceKL-divergence and evidence lower bound, mean-field updatesProject midterm progress report dueMurphy Ch 21 up to 21.3. My notes.
    5/18/2017ThursdayVariational inference (continued)Examples - Gaussian, linear regression. VBEMHW3 due, HW4 out, HW5 out (early since due exam week)Murphy Ch 21.5
    5/23/2017TuesdayTopic models and mixed membership modelsLSA/PLSA, Genetic admixtures, LDA, EM for LDABlei, David M. (2012). Probabilistic topic models. Communications of the ACM, 55(4), 77-84.
    5/25/2017ThursdayTopic models (continued)Collapsed Gibbs sampler, variational inference. Topic modeling with LDA extensionsAsuncion, A., Welling, M., Smyth, P., & Teh, Y. W. (2009). On smoothing and inference for topic models. UAI up to Section 3
    5/30/2017TuesdaySocial network modelsP1, P2, exponential family random graph modelsGoldenberg A., Zheng, A.X., Fienberg, S.E. and Airoldi, E.M. (2010). A Survey of Statistical Network Models. Foundations and Trends® in Machine Learning: Vol. 2: No. 2 Ch 3 up to 3.6, can skip 3.3
    6/1/2017ThursdaySocial network models (continued)Stochastic blockmodels, mixed membership stochastic blockmodels, latent space modelsHW4 dueGoldenberg A., Zheng, A.X., Fienberg, S.E. and Airoldi, E.M. (2010). A Survey of Statistical Network Models. Foundations and Trends® in Machine Learning: Vol. 2: No. 2 Ch 3.8 - 3.9
    6/6/2017TuesdayModels for computational biologyProfile HMMs for protein sequence alignment. Phylogenetic models and coalescent modelsMurphy Ch 17.3.1 subsection on Profile HMMs. What is Phylogeny? article from the Tree of Life Web Project, hosted at the University of Arizona. Teh, Y. W., and D. M. Roy. (2007). Bayesian Agglomerative Clustering with Coalescents. NIPS up to Section 2.
    6/8/2017ThursdayNonparametric Bayesian modelsChinese Restaurant process, Dirichlet process, Dirichlet process mixtures, Indian buffet processGershman, S. and Blei, D.M. A tutorial on Bayesian nonparametric models. Journal of Mathematical Psychology, 56:1–12, 2012.
    6/13/2017TuesdayFinal exam 7:00p-9:59p (location: YORK 4080A)
    6/15/2017ThursdayHW 5 due, project report due

    The syllabus may be subject to change. The details column is only a guideline of the content likely to be covered, and the dates on which material is covered may shift.

    Homework, Project, S/U Policy

  • Homeworks are due at the beginning of class on the dates specified. Late homeworks will not be accepted unless an extension is approved by me in advance. Requests for extensions must be made at least three days before the due date.
  • The project will be done in groups of 2-4. Project proposals are to be sent to me by email, and approved by the deadline (4/25/2017). Projects will be assessed based on two reports: a midterm progress report (due 5/16/2017) and a final report (due 6/15/2017).
  • As per university policy, if taken S/U (Satisfactory/Unsatisfactory), the equivalent of a B- is required for an S grade (80% or higher overall).

    Academic Integrity

    UCSD and CSE's policies on academic integrity will be strictly enforced (see http://academicintegrity.ucsd.edu/ and http://cseweb.ucsd.edu/~elkan/honesty.html). In particular, all of your work must be your own. Any exceptions will result in a zero on the assessment in question, and may lead to further disciplinary action. While you may verbally discuss assignments with your peers, you may not copy or look at anyone else's written assignment work or code, or share your own solutions. Some relevant excerpts from UCSD's policies are:

  • Don't copy another student's assignment, in part or in total, and submit it as your own work.
  • Don’t copy your online quiz or assignment answers from the internet or from anyone.
  • Acknowledge and cite source material in your papers or assignments.

    • Family Educational Rights and Privacy Act (FERPA) Notice

    Please note that as per federal law I am unable to discuss grades over email. If you wish to discuss grades, please come to my office hours.

    Campus Resources

  • Campus Community Centers (units of equity, diversity, and inclusion): diversity.ucsd.edu/centers/
  • Counseling and Psychological Services (CAPS), caps.ucsd.edu/
  • Office for Students with Disabilities, disabilities.ucsd.edu/