----------------------- REVIEW 1 --------------------- PAPER: 859 TITLE: Finding Progression Stages in Time-evolving Event Sequences AUTHORS: Jaewon Yang, Julian McAuley, Jure Leskovec, Paea Le Pendu and Nigam Shah OVERALL EVALUATION: -2 (marginal) REVIEWER'S CONFIDENCE: 3 (medium) This is the best paper among the papers I reviewed: 2 (No) Recommend this paper as a poster: 1 (Yes) ----------- REVIEW ----------- This paper proposes a probabilistic graphical model for discovering common progression stages in general event sequences. It is essentially an extension to the classic Hidden Markov Model (HMM) that the states are constrained to advance in a pre-defined sequential order. Using the model, one can predict future events in a sequence, infer meaningful progression stages, and cluster sequences based on common progression patterns. The proposed approach has been evaluated on several real-world datasets including patients' medical histories, online news, and navigational traces from the Web. Overall this is a well-written paper. It might be better if in Section 3.2 the authors could show a graphical representation of the proposed model as most graphical model papers do. In Section 5.1, the performance measure seems to be very lenient to allow matching with any of the k most probable outcomes, and the relative ranking of those k most probable outcomes has been ignored. The performance scores reported in Table 2 and 3 are very low (particularly on the two Beer datasets: the accuracy is less than 3%), so it is unclear if any reliable conclusion could be drawn. In Section 7.1, the baseline, k-means on the bag-of-events vectors, is probably too weak as the bag-of-events representation does not capture the order of events; the authors should probably construct the feature vectors using the bigram-of-events. ----------------------- REVIEW 2 --------------------- PAPER: 859 TITLE: Finding Progression Stages in Time-evolving Event Sequences AUTHORS: Jaewon Yang, Julian McAuley, Jure Leskovec, Paea Le Pendu and Nigam Shah OVERALL EVALUATION: 3 (should accept) REVIEWER'S CONFIDENCE: 2 (low) This is the best paper among the papers I reviewed: 1 (Yes) Recommend this paper as a poster: 1 (Yes) ----------- REVIEW ----------- This paper presents one model to identify progression stages and classes in time-evolving event sequences. It segments data samples into disjoint classes and then partition sequences into ordered stages. This can be considered as an extension of disjoint clustering dealing with event sequences. This problem, as far as I know, is novel (May need to double check with people working on time series), though the algorithm itself follows standard EM/alternating optimization scheme. The presentation is excellent. Well motivated and clearly presented. It is great that the authors conducted extensive experiments across different domains, from product reviews, medical records and Wikipedia browsing sessions. The plus is that they also include extensive analysis rather than just accuracies numbers. One minor comment: In this paper, the proposed model assumes the stages/classes do not change as time evolves. But as mentioned in the related work, some external events might change the trend for all/subset of customers. Would the model automatically capture that effect? ----------------------- REVIEW 3 --------------------- PAPER: 859 TITLE: Finding Progression Stages in Time-evolving Event Sequences AUTHORS: Jaewon Yang, Julian McAuley, Jure Leskovec, Paea Le Pendu and Nigam Shah OVERALL EVALUATION: -4 (should reject) REVIEWER'S CONFIDENCE: 2 (low) This is the best paper among the papers I reviewed: 1 (Yes) Recommend this paper as a poster: 1 (Yes) ----------- REVIEW ----------- The paper addersses the problem of find evolving patterns of different types in a data. For example, in a dataset of patients with their symptoms, find the different patters of disease evolvement stages. The authors suggest a generative model of how events evolve from different classes. The authors perform an extensive analysis of their algorithm on several datasets including quantitative analysis. For me, the paper was hard to read and understand. It took me 5 passes to start understanding the idea of the paper. The introduction did not do a good job explaining the problem. I would suggest to move the example of Figure 1 to the beginning of hte intro and present all the challenges on this example, then how prior work would behave on this example and then explain the current method. The main issue is the paper structure, that repeats many details several times instead of giving some intuition during the writing. The intuitions appear sporadically around the paper making it hard to follow. I would suggest to re-arrange the paper in a more coherent way. I am confused regarding the novelty. I could not understand the main difference between the suggested approach and DBNs. * If the main novelty of the paper is a new model - then what is the difference between it and a DBN? I think the problem could have been easily modeled by such a model, where the classes are a variable and the transitions are temporal variables. * If the main novelty of the paper is an estimation algorithm - how does it compare of any other algorithms for estimations. I would suggest to strengthen the novelty of the paper and what it aims to achieve. Additionally, all the field of DBNs is missing from the related work. I did like the extensive work the authors did with the analysis and the qualitative examples (even wish they would appear in the beginning of the paper). In general, the approach seems valid and the experiments are well done. However, the contribution of the paper is not clear in comparison to other related works and the paper needs a restructuring to make it easier to read. ----------------------- REVIEW 4 --------------------- PAPER: 859 TITLE: Finding Progression Stages in Time-evolving Event Sequences AUTHORS: Jaewon Yang, Julian McAuley, Jure Leskovec, Paea Le Pendu and Nigam Shah OVERALL EVALUATION: 3 (should accept) REVIEWER'S CONFIDENCE: 5 (expert) This is the best paper among the papers I reviewed: 2 (No) Recommend this paper as a poster: 2 (No) ----------- REVIEW ----------- I enjoyed this paper. The authors present a nice mechanism to infer distributions over labels corresponding to "stages," and the likelihood of traveling from stage i to stage i+j. The formulation is pretty clean, the inference looks good, and the authors do a nice job of discussing the results. They also lay out a few different domains in which this type of stage mining makes sense, and present a number of empirical investigations of the outcomes of the mining.