--======== Review Reports ========-- The review report from reviewer #1: *1: Is the paper relevant to ICDM? [_] No [X] Yes *2: How innovative is the paper? [_] 6 (Very innovative) [X] 3 (Innovative) [_] -2 (Marginally) [_] -4 (Not very much) [_] -6 (Not at all) *3: How would you rate the technical quality of the paper? [_] 6 (Very high) [X] 3 (High) [_] -2 (Marginal) [_] -4 (Low) [_] -6 (Very low) *4: How is the presentation? [_] 6 (Excellent) [_] 3 (Good) [X] -2 (Marginal) [_] -4 (Below average) [_] -6 (Poor) *5: Is the paper of interest to ICDM users and practitioners? [X] 3 (Yes) [_] 2 (May be) [_] 1 (No) [_] 0 (Not applicable) *6: What is your confidence in your review of this paper? [X] 2 (High) [_] 1 (Medium) [_] 0 (Low) *7: Overall recommendation [_] 6: must accept (in top 25% of ICDM accepted papers) [X] 3: should accept (in top 80% of ICDM accepted papers) [_] -2: marginal (in bottom 20% of ICDM accepted papers) [_] -4: should reject (below acceptance bar) [_] -6: must reject (unacceptable: too weak, incomplete, or wrong) *8: Summary of the paper's main contribution and impact (This is a metareview) - see summary and individual reviews *9: Justification of your recommendation (This is a metareview) - see summary and individual reviews *10: Three strong points of this paper (please number each point) (This is a metareview) - see summary and individual reviews *11: Three weak points of this paper (please number each point) (This is a metareview) - see summary and individual reviews *12: Is this submission among the best 10% of submissions that you reviewed for ICDM'16? [_] No [X] Yes *13: Would you be able to replicate the results based on the information given in the paper? [X] No [_] Yes *14: Are the data and implementations publicly available for possible replication? [X] No [_] Yes *15: If the paper is accepted, which format would you suggest? [X] Regular Paper [_] Short Paper *16: Detailed comments for the authors This is a metareview, based on the three individual reviews and a separate reading of the paper. All reviewers agree that the paper deals with is an interesting, timely and relevant topic and proposes a technically sound method. Novelty / additions to the existing work were judged differently; I would ask the authors to explain this better. Some details of the method and the presentation were found to be in need of improvement, but this should be easy to fix. Please note that the reviewers' recommendations are mandatory changes for the paper to be published. ======================================================== The review report from reviewer #2: *1: Is the paper relevant to ICDM? [_] No [X] Yes *2: How innovative is the paper? [_] 6 (Very innovative) [_] 3 (Innovative) [X] -2 (Marginally) [_] -4 (Not very much) [_] -6 (Not at all) *3: How would you rate the technical quality of the paper? [_] 6 (Very high) [X] 3 (High) [_] -2 (Marginal) [_] -4 (Low) [_] -6 (Very low) *4: How is the presentation? [_] 6 (Excellent) [_] 3 (Good) [X] -2 (Marginal) [_] -4 (Below average) [_] -6 (Poor) *5: Is the paper of interest to ICDM users and practitioners? [_] 3 (Yes) [X] 2 (May be) [_] 1 (No) [_] 0 (Not applicable) *6: What is your confidence in your review of this paper? [_] 2 (High) [X] 1 (Medium) [_] 0 (Low) *7: Overall recommendation [_] 6: must accept (in top 25% of ICDM accepted papers) [X] 3: should accept (in top 80% of ICDM accepted papers) [_] -2: marginal (in bottom 20% of ICDM accepted papers) [_] -4: should reject (below acceptance bar) [_] -6: must reject (unacceptable: too weak, incomplete, or wrong) *8: Summary of the paper's main contribution and impact This paper models ambiguity and subjectivity present in answers or reviews of product related questions. Specifically, it presents techniques to determine how relevant a review is to a question with respect to ambiguity and subjectivity. An EM-like mixture-of-experts framework is proposed to determine relevance. Results on Amazon product data show that their techniques are better than existing techniques in terms of finding relevant reviews. *9: Justification of your recommendation I have explained justification in number 16 *10: Three strong points of this paper (please number each point) 1. The problem is interesting. The paper has technical depth. The techniques seem to be sound. 2. It produces a new dataset and claims to make it public after publishing the paper. *11: Three weak points of this paper (please number each point) 1. Limited Novelty. 2. Incremental Work. 3. Hard to follow. Examples are required. *12: Is this submission among the best 10% of submissions that you reviewed for ICDM'16? [X] No [_] Yes *13: Would you be able to replicate the results based on the information given in the paper? [X] No [_] Yes *14: Are the data and implementations publicly available for possible replication? [X] No [_] Yes *15: If the paper is accepted, which format would you suggest? [X] Regular Paper [_] Short Paper *16: Detailed comments for the authors This paper extends the framework of its earlier version. It introduces techniques to consider a situation where a question may have multiple relevant reviews instead of just one. It studies two variations of the questions- i) Binary and ii) Open ended. For Binary questions, the authors proposed models to capture relevant reviews with the positive answer. For Open-ended questions, they presented techniques to capture the ground-truth answers as much as possible. To validate the proposed methods, the authors have done the evaluation over an Amazon dataset which they collected. Even though the experiment results show the efficacy of the methods, for the below reasons, I stand with a neutral judgment. -For subjectivity, the authors have resorted to features such as reviewer's bias, expertise, etc. It is not clear how to get these values. A more concerning issue is, the paper does not seem to connect with works related to Subjectivity analysis in text data. A reasonable model should consider the presence of Subjectivity signals in the text as well. -The presentation of the paper needs further improvement. Due to the lack of running examples, it was very hard to follow the definitions and equations. In some cases, terms were used without prior explanation. For example, "weak classifier" in page 3. Overall, the structure of the paper does not seem to me well organized. ======================================================== The review report from reviewer #3: *1: Is the paper relevant to ICDM? [_] No [X] Yes *2: How innovative is the paper? [_] 6 (Very innovative) [X] 3 (Innovative) [_] -2 (Marginally) [_] -4 (Not very much) [_] -6 (Not at all) *3: How would you rate the technical quality of the paper? [_] 6 (Very high) [X] 3 (High) [_] -2 (Marginal) [_] -4 (Low) [_] -6 (Very low) *4: How is the presentation? [_] 6 (Excellent) [_] 3 (Good) [X] -2 (Marginal) [_] -4 (Below average) [_] -6 (Poor) *5: Is the paper of interest to ICDM users and practitioners? [_] 3 (Yes) [X] 2 (May be) [_] 1 (No) [_] 0 (Not applicable) *6: What is your confidence in your review of this paper? [X] 2 (High) [_] 1 (Medium) [_] 0 (Low) *7: Overall recommendation [_] 6: must accept (in top 25% of ICDM accepted papers) [X] 3: should accept (in top 80% of ICDM accepted papers) [_] -2: marginal (in bottom 20% of ICDM accepted papers) [_] -4: should reject (below acceptance bar) [_] -6: must reject (unacceptable: too weak, incomplete, or wrong) *8: Summary of the paper's main contribution and impact The paper studies the problem of sentiment question answering. The area has not been well studied sofar. The author proposed an interesting algorithm to improve existing methods. *9: Justification of your recommendation See the detailed review. *10: Three strong points of this paper (please number each point) See the detailed review. *11: Three weak points of this paper (please number each point) See the detailed review. *12: Is this submission among the best 10% of submissions that you reviewed for ICDM'16? [X] No [_] Yes *13: Would you be able to replicate the results based on the information given in the paper? [X] No [_] Yes *14: Are the data and implementations publicly available for possible replication? [X] No [_] Yes *15: If the paper is accepted, which format would you suggest? [X] Regular Paper [_] Short Paper *16: Detailed comments for the authors Sentiment based QA systems are not well studied in the community. I agree with the authors that there are a lot of needs. This paper is thus interesting in this aspect. The paper also proposed some good improvements to the existing approaches. On the negative side, I would like to see a list of example questions. With the questions, one can judge what kind of answers would be appropriate. For example, if a question is like “do people like the lens of the camera?” it is probably not appropriate to just find relevant reviews or sentences because in this case, an opinion summarization would be more appropriate. E.g., there may be 10 people who like it and 30 people who do not like it. Relevance alone may not be sufficient. The experiment results are reasonable. I think the authors missed an important area of research that is closely related to QA, although not the same, i.e., opinion search. See B. Liu's book. Sentiment analysis and opinion mining, 2012, for some references. ======================================================== The review report from reviewer #4: *1: Is the paper relevant to ICDM? [_] No [X] Yes *2: How innovative is the paper? [_] 6 (Very innovative) [_] 3 (Innovative) [X] -2 (Marginally) [_] -4 (Not very much) [_] -6 (Not at all) *3: How would you rate the technical quality of the paper? [_] 6 (Very high) [_] 3 (High) [X] -2 (Marginal) [_] -4 (Low) [_] -6 (Very low) *4: How is the presentation? [_] 6 (Excellent) [X] 3 (Good) [_] -2 (Marginal) [_] -4 (Below average) [_] -6 (Poor) *5: Is the paper of interest to ICDM users and practitioners? [_] 3 (Yes) [X] 2 (May be) [_] 1 (No) [_] 0 (Not applicable) *6: What is your confidence in your review of this paper? [X] 2 (High) [_] 1 (Medium) [_] 0 (Low) *7: Overall recommendation [_] 6: must accept (in top 25% of ICDM accepted papers) [_] 3: should accept (in top 80% of ICDM accepted papers) [X] -2: marginal (in bottom 20% of ICDM accepted papers) [_] -4: should reject (below acceptance bar) [_] -6: must reject (unacceptable: too weak, incomplete, or wrong) *8: Summary of the paper's main contribution and impact This paper extends a very recent work that aims at answering real users’ questions from Amazon.com based on related review sentences. They collected a new dataset which contains reviews, questions and the answer list for each question. The technical contribution is to naturally add answer ambiguity and user behavior into the mixture of expert model, to improve the effectiveness of retrieving question answering reviews. *9: Justification of your recommendation This paper studies a very important and practical problem, following a very recent work in 2016. The idea of the model naturally suits the dataset that they collected. I give a marginal recommendation mainly because the evaluation results are not very convincing to show the proposed model incorporating subjective information works. *10: Three strong points of this paper (please number each point) 1) Technical novelty of considering answer ambiguity and user behaviors in community question answering. Models answer ambiguity in different levels (fixed fraction v.s. distribution) and user behavior with simple expertise and bias. The model design is suitable for the dataset. 2) A new data that contains multiple answers to a question, goes beyond the one Moqa (McAuley, 2016) used. The authors further motivated their study by demonstrating that many questions have multi-user and ambiguous answers. 3) Uses AUC for evaluation instead of accuracy that used in the previous work, which better measures the performance especially when positive and negative answers are not balanced . *11: Three weak points of this paper (please number each point) 1) The final goal is to answer a user’s question with related reviews. There is still a small gap between the evaluation metrics ( AUC and accuracy@a) and the final goal. A user study or examples shown as in the previous work could give readers a better, intuitive sense of the effectiveness of this new model. 2) The objective function didn’t take the quality of real answers to a question into account. Not all of those answers should be ranked high. 3) The AUC values are not very satisfying comparing to the random selection baseline. Especially, modeling user expertise and bias demonstrates no or marginal advantages due to the sparseness of answers from the same users in the dataset. *12: Is this submission among the best 10% of submissions that you reviewed for ICDM'16? [X] No [_] Yes *13: Would you be able to replicate the results based on the information given in the paper? [X] No [_] Yes *14: Are the data and implementations publicly available for possible replication? [X] No [_] Yes *15: If the paper is accepted, which format would you suggest? [X] Regular Paper [_] Short Paper *16: Detailed comments for the authors This is overall a very interesting work. Most comments are listed above. One more question is, in future directions you mentioned to use more complex features. But more features may involve more parameters, and how to solve the possible problem of overfitting due to the sparseness of answers from the same user? ======================================================== Meta Review: The degree of controversiality was low for this paper, and it was one of the top-rated papers in my batch. Controversiality focused on whether this was a useful extension of previous work or had limited novelty (presumably because it extends previous work). I have therefore opted for an accept as a summary, which in the recommendation options I have translates to "regular paper". Note that I emphasized these points by closing the metareview with the sentence: "Please note that the reviewers' recommendations are mandatory changes for the paper to be published."