CSE 291-D Latent Variable Models (Spring 2016)

Course Description

Probabilistic machine learning models are powerful, flexible tools for data analysis. This course focuses on models with latent variables, which simplify complex data sets by discovering hidden patterns in the data. By the end of the course you will be able to apply a variety of probabilistic models for text data, social networks, computational biology, and more. You will learn how to formulate probabilistic models to solve the data science tasks you care about, derive inference algorithms for these models using Bayesian inference techniques, and evaluate their performance. Topics will include topic models, stochastic blockmodels and latent space models for social networks, Bayesian inference, Markov chain Monte Carlo, mean field variational inference, and nonparametric Bayesian models. This course continues from CSE250A.

Lecture time and venue: TuTh 6:30-7:50pm PETER 103.

Instructor: James Foulds
Instructor email: jfoulds at ucsd dot edu
Instructor office hours: TuTh 5-6pm Atkinson Hall 4401

Teaching assistant: Long Jin
Teaching assistant office hours: Monday 4:30-5:30 EBU3B Room B275
Teaching assistant email: longjin at eng dot ucsd dot edu

Grading: Homeworks 25% (5 of them, 5% each), Group Project 35%, Final 35%, Participation 5%

Piazza: Sign up for this course at piazza.com/ucsd/spring2016/cse291d
Poll Everywhere: PollEv.com/jamesfoulds656

Prerequisites

  • Requires CSE 250A, or consent of the instructor.
  • Required knowledge: the basics of probabilistic graphical models, as covered in CSE250A. The prerequisite knowlege for CSE250A also applies to this course: elementary probability, multivariable calculus, linear algebra, and basic programming ability in a high-level language such as C, Java, or Matlab.
  • Recommended preparation for those without required knowledge: Machine Learning: A Probabilistic Perspective (Murphy, 2012) Chapter 10, or Pattern Recognition and Machine Learning (Bishop, 2006) Chapter 8.

    • Required Textbooks

  • Machine Learning: A Probabilistic Perspective (Murphy, 2012) is the primary textbook. You will need this book for course readings. Until you obtain it, the UCSD library has an electronic desk copy (requires UCSD IP address, limited to one person at a time).
  • Information Theory, Inference, and Learning Algorithms (Mackay, 2003) is a secondary textbook that we will use for some topics. There is a free PDF download of the book from author's website. You do not need to purchase this book for this course, though I do recommend it, if you are thinking about buying it.

    • Syllabus

    Lecture Summary Details Assessment Required reading
    3/29/2016TuesdayOverview, motivating examples, recap of directed graphical modelsMotivating applications - computational social science, social networks, computational biology. Recap on d-separation/Bayes ball/explaining awayBlei, David M. (2014). Build, compute, critique, repeat: Data analysis with latent variable models. Annual Review of Statistics and Its Application, Sections 1-3
    3/31/2016ThursdayBayesian inferenceMonty hall problem, frequentist vs Bayes, MAP vs MLE, full posterior vs point estimates, posterior predictiveHW1 outRead "What is Bayesian Analysis?" at bayesian.org (scroll down, and be sure to follow the link to bayesian.org/Bayes-Explained). Then read Murphy Ch 3.1 - 3.2
    4/5/2016TuesdayGenerative models for discrete dataConjugate priors, beta/Bernoulli, Dirichlet/multinomial, urn process interpretations, naive Bayes document modelMurphy Ch 3.3 - 3.5.2
    4/7/2016ThursdayExponential families and generalized linear modelsExponential families, Bayesian inference for EFs, writing distributions in EF form, GLMsMurphy Ch 9.1 - 9.3
    4/12/2016TuesdayMonte Carlo methodsImportance sampling, rejection sampling, why they fail in high dimensionsMacKay Ch 29.1 - 29.3
    4/14/2016ThursdayMarkov chain Monte CarloGibbs sampling, Metropolis-HastingsHW1 due, HW2 outMacKay Ch 29.4 - 29.6
    4/19/2016TuesdayMixture models revisitedMixtures of Gaussians/connection to K-means, mixtures of multinomials, EM for mixtures of exponential family distributions, Bayesian inferenceProject proposal dueMurphy Ch 11.1 - 11.3
    4/21/2016ThursdayLatent linear modelsFactor analysis, probabilistic PCA, ICAMurphy Ch 12.1, 12.2.4 - 12.2.5, 12.6 - 12.6.1
    4/26/2016TuesdayHidden Markov Models revisitedApplications to part of speech tagging, Factorial HMMs, input/output HMMs, direct Gibbs sampling, Forward filtering backward sampling, collapsed MCMCMurphy Ch 17.3 - 17.4.2, 17.4.5, Gao, J. and Johnson, M. (2008). A comparison of Bayesian estimators for unsupervised Hidden Markov Model POS taggers. EMNLP, Sections 1-2 (don't worry too much about the variational inference part, since we haven't covered this yet)
    4/28/2016ThursdayEvaluating unsupervised modelsLog-likelihood on held-out data, posterior predictive checks, correlation with metadata, human evaluationHW2 due, HW3 outDave Blei's notes on posterior predictive checks
    5/3/2016TuesdayMarkov random fieldsSeparation criterion, Hammersley-Clifford theorem, Ising models, belief propagationMurphy Ch 19, up to 19.4.1. Ch 20, up to 20.2.2
    5/5/2016ThursdayStatistical relational learning and probabilistic programmingMarkov logic networks, probabilistic soft logic, Stan/winBUGSDomingos, P., Kok, S., Lowd, D., Poon, H., Richardson, M., Singla, P. (2008). Markov Logic. In L. De Raedt, P. Frasconi, K. Kersting and S. Muggleton (eds.), Probabilistic Inductive Logic Programming, Sections 1, 4, and 7.
    5/10/2016TuesdayVariational inferenceEM redux (Neal and Hinton-style), KL-divergence and evidence lower bound, mean-field updatesProject midterm progress report dueMurphy Ch 21 up to 21.3. My notes.
    5/12/2016ThursdayVariational inference (continued)Examples - Gaussian, linear regression. VBEMHW3 due, HW4 outMurphy Ch 21.5
    5/17/2016TuesdayTopic models and mixed membership modelsLSA/PLSA, Genetic admixtures, LDA, EM for LDABlei, David M. (2012). Probabilistic topic models. Communications of the ACM, 55(4), 77-84.
    5/19/2016ThursdayTopic models (continued)Collapsed Gibbs sampler, variational inference. Topic modeling with LDA extensionsAsuncion, A., Welling, M., Smyth, P., & Teh, Y. W. (2009). On smoothing and inference for topic models. UAI up to Section 3
    5/24/2016TuesdaySocial network modelsP1, P2, exponential family random graph modelsGoldenberg A., Zheng, A.X., Fienberg, S.E. and Airoldi, E.M. (2010). A Survey of Statistical Network Models. Foundations and Trends® in Machine Learning: Vol. 2: No. 2 Ch 3 up to 3.6, can skip 3.3
    5/26/2016ThursdaySocial network models (continued)Stochastic blockmodels, mixed membership stochastic blockmodels, latent space modelsHW4 due, HW 5 outGoldenberg A., Zheng, A.X., Fienberg, S.E. and Airoldi, E.M. (2010). A Survey of Statistical Network Models. Foundations and Trends® in Machine Learning: Vol. 2: No. 2 Ch 3.8 - 3.9
    5/31/2016TuesdayModels for computational biologyProfile HMMs for protein sequence alignment. Phylogenetic models and coalescent modelsMurphy Ch 17.3.1 subsection on Profile HMMs. What is Phylogeny? article from the Tree of Life Web Project, hosted at the University of Arizona. Teh, Y. W., and D. M. Roy. (2007). Bayesian Agglomerative Clustering with Coalescents. NIPS up to Section 2.
    6/2/2016ThursdayNonparametric Bayesian modelsChinese Restaurant process, Dirichlet process, Dirichlet process mixtures, Indian buffet processGershman, S. and Blei, D.M. A tutorial on Bayesian nonparametric models. Journal of Mathematical Psychology, 56:1–12, 2012.
    6/7/2016TuesdayFinal exam 7:00p-9:59p PETER 103
    6/9/2016ThursdayHW 5 due, project report due

    The syllabus may be subject to change. The details column is only a guideline of the content likely to be covered, and the dates on which material is covered may shift.

    Homework and Project Policy

  • Homeworks are due at the beginning of class on the dates specified. Late homeworks will not be accepted unless an extension is approved by me in advance. Requests for extensions must be made at least three days before the due date.
  • The project will be done in groups of 2-4. Project proposals are to be sent to me by email, and approved by the deadline (4/19/2016). Projects will be assessed based on two reports: a midterm progress report (due 5/10/2016) and a final report (due 6/9/2016).

    • Academic Integrity

    UCSD and CSE's policies on academic integrity will be strictly enforced (see http://academicintegrity.ucsd.edu/ and http://cseweb.ucsd.edu/~elkan/honesty.html). In particular, all of your work must be your own. Any exceptions will result in a zero on the assessment in question, and may lead to further disciplinary action. While you may verbally discuss assignments with your peers, you may not copy or look at anyone else's written assignment work or code, or share your own solutions. Some relevant excerpts from UCSD's policies are:

  • Don't copy another student's assignment, in part or in total, and submit it as your own work.
  • Don’t copy your online quiz or assignment answers from the internet or from anyone.
  • Acknowledge and cite source material in your papers or assignments.

    • Family Educational Rights and Privacy Act (FERPA) Notice

    Please note that as per federal law I am unable to discuss grades over email. If you wish to discuss grades, please come to my office hours.

    Campus Resources

  • Campus Community Centers (units of equity, diversity, and inclusion): diversity.ucsd.edu/centers/
  • Counseling and Psychological Services (CAPS), caps.ucsd.edu/
  • Office for Students with Disabilities, disabilities.ucsd.edu/