Reviews For Paper
Paper ID 1839
Title Learning Visual Clothing Style with Heterogeneous Dyads from Co-occurrence
Masked Reviewer ID: Assigned_Reviewer_13
Review:
Question 
Paper Summary. Please summarize in your own words what the paper is about. This paper proposes to learn matching styles of outfits from product images. It generates training data from Amazon "also bought" image pairs, and train a Siamese CNN model to learn a latent space of compatible styles from binary-labelled image pairs of match and mismatch. Then image ranking based on the latent space is adopted to discover instances of matching styles for a query image.
Paper Strengths. Please discuss the positive aspects of the paper. Be sure to comment on the paper's novelty, technical correctness, clarity and experimental evaluation. Notice that different papers may need different levels of evaluation: a theoretical paper may need no experiments, while a paper presenting a new approach to a known problem may require thorough comparisons to existing methods. Also, please make sure to justify your comments in great detail. For example, if you think the paper is novel, not only say so, but also explain in detail why you think this is the case. 1. To mine and learn knowledge of fashion matching styles from web data is an interesting research topic.
2. The idea of generating pairs-of-matches images from Amazon "also bought" data is smart, and using Siamese CNN as the model is very reasonable.
3. The clustering approach to find correct instances from noisily labeled data is interesting.
Paper Weaknesses. Please discuss the negative aspects of the paper: lack of novelty or clarity, technical errors, insufficient experimental evaluation, etc. Please justify your comments in great detail. If you think the paper is not novel, explain why and give a reference to prior work. Do not ask the authors to cite your own work. If you think this is essential, write it in the confidential comments to the AC. If you think there is an error in the paper, explain in detail why it is an error. If you think the experimental evaluation is insufficient, remember that theoretical results/ideas are essential to ICCV and that a theoretical paper need not have experiments. It is not ok to reject a paper because it did not outperform other existing algorithms, especially if the theory is novel and interesting. It is also not ok to ask for comparisons with unpublished papers and papers published after the ICCV deadline. Last but not least, remember to be polite and constructive. 1. The paper misses some very important details.
2. The paper has readibility issues.
See Q5 for details.
Preliminary Rating. Please rate the paper according to the following choices. Oral: these are papers whose quality is in the top 10% of the papers at ICCV. Examples include a theoretical breakthrough with no experiments; an interesting solution to a new problem; a novel solution to an existing problem with solid experiments; or an incremental paper that leads to dramatic improvements in performance. Oral/Poster: these are very strong papers, which may have one weakness that makes you unsure as to whether they should be oral or poster. Poster: these are strong papers, which have more than one weakness. For example, a well-written paper with solid experiments, but incremental; a paper on a well studied problem with solid theory, but weak experiments; or a novel paper with good experiments, but poorly written. Weak Reject: these are papers that have some promise, but they would be better off by being revised and resubmitted. Strong Reject: these are papers that have major flaws, or have been done before. Poster
Preliminary Evaluation. Please indicate to the AC, your fellow reviewers, and the authors your current opinion on the paper. Please summarize the key things you would like the authors to include in their rebuttals to facilitate your decision making. There is no need to summarize the paper. Though the work is interesting, I rate this paper between Weak Reject and Poster because of the lack of important details. The authors are expected to provide the below details in their revised version:
1) How to generate training data is the crucial part of this work, or the key contribution. However, it is unclear how the authors generate their negative data. It's easy to understand that image instances/categories which appeared in Amazon "also bought" are regarded as positive training data, but how negative data are obtained? Image instances/categories that never co-occur in "also bought" are not necessarily "dissimilar".
2) Is the setting of 1:16 between positive and negative training images also due to the sparsity of "also bought" data?
3) Section 7, the visualization of style spaces - Is the visualization based on naive sampling or stratified sampling? If it's based on stratified sampling, I can't understand why objects from the same categories are close, since you don't have such knowledge in your training data.
4) It's important to show samples of "neighbor" categories in Figure 3, e.g. what are the closest surrounding categories for the sport shoes and leather shoes? Do the make sense? Such information are important for readers to understand the performance of your approach.
5) line 542 - How to get the knowledge of which categories are consisted in an outfit?
6) Figure 6 and line 669-689: If Jeans/Shirts/Shoes are heldout categories, what are the categories used in Figure 6(a)? And why is Figure 6(a) called "on set containing all categories?"
7) line 581 - please explain why 256-dimensional embedding. I think its standard CNN output setting but please don't assume every reader has background knowledge.

The readibility problem of this paper is twofold:
1) The figures are not readable in a black-and-white printout.
2) There are several language issues, e.g. line 478 - "both, two"; line 485 - "Since, we"

Moreover, the title is misleading. The work is not about clothing style learning, but about matching styles/categories learning.
Confidence. Write “Very Confident" to stress that you are absolutely sure about your conclusions (e.g., you are an expert who works in the paper's area), “Confident” to stress that you are mostly sure about your conclusions (e.g., you are not an expert but can distinguish good work from bad work in that area), and “Not Confident” to stress that that you feel some doubt about your conclusions. In the latter case, please provide details as confidential comments to PC/AC chairs (point 7.). Confident
Final Recommendation. After reading the author's rebuttal and the discussion, please explain your final recommendation. Your explanation will be of highest importance for making acceptance decisions and for deciding between posters and orals. Include suggestions for improvement and publication alternatives, if appropriate. Though the author's rebuttal is acceptable, it doesn't solve some of my concerns, e.g. training data sampling. I'll keep my previous rating.
Final Rating. After reading the author's rebuttal, please rate the paper according to the following choices. Poster
Masked Reviewer ID: Assigned_Reviewer_19
Review:
Question 
Paper Summary. Please summarize in your own words what the paper is about. This paper proposes a novel learning framework for visual clothing style. Rather than the visual similarity, the work aims to learn the visual compatibility across different categories. In order to do that, the authors use Siamese CNN which uses pairs of images of different categories but people have bought together. It maps images to a latent style space and on that space, items that match with each other are close and items that don't match are far apart. Training Siamese CNN requires special attention of data sampling and the authors have shown that their strategic sampling is better than naive sampling. They also compare to other baseline including vanilla CNN model. Furthermore, they conduct a user survey to compare different methods.
Paper Strengths. Please discuss the positive aspects of the paper. Be sure to comment on the paper's novelty, technical correctness, clarity and experimental evaluation. Notice that different papers may need different levels of evaluation: a theoretical paper may need no experiments, while a paper presenting a new approach to a known problem may require thorough comparisons to existing methods. Also, please make sure to justify your comments in great detail. For example, if you think the paper is novel, not only say so, but also explain in detail why you think this is the case. Strengths of this paper include:
1) It is a very interesting problem formulation. There are several related work in the past that focus on fashion data but it is a novel learning framework for visual compatibility.
2) The paper is well-written.
3) Extensive experimental results including a user survey. Though the user survey results are not consistent with the quantitative numbers, the reason might be the noisy training labels.
Paper Weaknesses. Please discuss the negative aspects of the paper: lack of novelty or clarity, technical errors, insufficient experimental evaluation, etc. Please justify your comments in great detail. If you think the paper is not novel, explain why and give a reference to prior work. Do not ask the authors to cite your own work. If you think this is essential, write it in the confidential comments to the AC. If you think there is an error in the paper, explain in detail why it is an error. If you think the experimental evaluation is insufficient, remember that theoretical results/ideas are essential to ICCV and that a theoretical paper need not have experiments. It is not ok to reject a paper because it did not outperform other existing algorithms, especially if the theory is novel and interesting. It is also not ok to ask for comparisons with unpublished papers and papers published after the ICCV deadline. Last but not least, remember to be polite and constructive. Some weakness:
1) Novelty of method itself is not that high. Siamese CNNs have been used in several other related tasks and it is a common knowledge that sampling the training data is a very crucial step to train a Siamese network. The strategic sampling method simply forces the pairs to be heterogeneous dyads, a more advanced method would consider the margin and sample some hard negative examples.
2) Noise in training data. The positive and negative examples are generated by dataset collected in [1]. The link of comp(a,b) comes from whether Amazon users have cobought a &b. The training label itself is not very reliable. A good user study would be asking users to mark whether they think the training examples are matching or not. That will give a sense about how much noise in the data.
Preliminary Rating. Please rate the paper according to the following choices. Oral: these are papers whose quality is in the top 10% of the papers at ICCV. Examples include a theoretical breakthrough with no experiments; an interesting solution to a new problem; a novel solution to an existing problem with solid experiments; or an incremental paper that leads to dramatic improvements in performance. Oral/Poster: these are very strong papers, which may have one weakness that makes you unsure as to whether they should be oral or poster. Poster: these are strong papers, which have more than one weakness. For example, a well-written paper with solid experiments, but incremental; a paper on a well studied problem with solid theory, but weak experiments; or a novel paper with good experiments, but poorly written. Weak Reject: these are papers that have some promise, but they would be better off by being revised and resubmitted. Strong Reject: these are papers that have major flaws, or have been done before. Poster
Preliminary Evaluation. Please indicate to the AC, your fellow reviewers, and the authors your current opinion on the paper. Please summarize the key things you would like the authors to include in their rebuttals to facilitate your decision making. There is no need to summarize the paper. In general, I like this paper. It is an interesting problem and I enjoy reading the paper. The dataset seems to have lots of noise and this paper is a good staring point for this direction of research. I am also wondering if the dataset will be made publicly.
Confidence. Write “Very Confident" to stress that you are absolutely sure about your conclusions (e.g., you are an expert who works in the paper's area), “Confident” to stress that you are mostly sure about your conclusions (e.g., you are not an expert but can distinguish good work from bad work in that area), and “Not Confident” to stress that that you feel some doubt about your conclusions. In the latter case, please provide details as confidential comments to PC/AC chairs (point 7.). Confident
Final Recommendation. After reading the author's rebuttal and the discussion, please explain your final recommendation. Your explanation will be of highest importance for making acceptance decisions and for deciding between posters and orals. Include suggestions for improvement and publication alternatives, if appropriate. The paper is well-written and is an interesting problem setting. The authors include some of the missing details the other reviewer pointed out.
Final Rating. After reading the author's rebuttal, please rate the paper according to the following choices. Poster
Masked Reviewer ID: Assigned_Reviewer_9
Review:
Question 
Paper Summary. Please summarize in your own words what the paper is about. This paper proposes a framework for learning clothing style by co-occurring heterogeneous dyads. The system includes (1) a strategic sampling approach for training data generation, (2) utilizing co-occurrence information in Siamese network training, (3) a nearest neighbor method to overcome the noisy labels for outfit generation. Experiments over a dataset downloaded from Amazon proved the benefit of the proposed framework over the AlexNet approach and also with respect to the GoogLeNet trained without considering heterogeneity and co-occurrence between items.
Paper Strengths. Please discuss the positive aspects of the paper. Be sure to comment on the paper's novelty, technical correctness, clarity and experimental evaluation. Notice that different papers may need different levels of evaluation: a theoretical paper may need no experiments, while a paper presenting a new approach to a known problem may require thorough comparisons to existing methods. Also, please make sure to justify your comments in great detail. For example, if you think the paper is novel, not only say so, but also explain in detail why you think this is the case. The paper is well written and well structured. The schema clearly shows the main steps. The proposed approaches are plausible, sound and the experimental results show the advantages of the proposed framework.
Paper Weaknesses. Please discuss the negative aspects of the paper: lack of novelty or clarity, technical errors, insufficient experimental evaluation, etc. Please justify your comments in great detail. If you think the paper is not novel, explain why and give a reference to prior work. Do not ask the authors to cite your own work. If you think this is essential, write it in the confidential comments to the AC. If you think there is an error in the paper, explain in detail why it is an error. If you think the experimental evaluation is insufficient, remember that theoretical results/ideas are essential to ICCV and that a theoretical paper need not have experiments. It is not ok to reject a paper because it did not outperform other existing algorithms, especially if the theory is novel and interesting. It is also not ok to ask for comparisons with unpublished papers and papers published after the ICCV deadline. Last but not least, remember to be polite and constructive. The authors only utilize the co-purchase information to discover co-occurrence between items. I would suggest that the authors take the visual co-occurrence information in different categories of the same full body into consideration.
Preliminary Rating. Please rate the paper according to the following choices. Oral: these are papers whose quality is in the top 10% of the papers at ICCV. Examples include a theoretical breakthrough with no experiments; an interesting solution to a new problem; a novel solution to an existing problem with solid experiments; or an incremental paper that leads to dramatic improvements in performance. Oral/Poster: these are very strong papers, which may have one weakness that makes you unsure as to whether they should be oral or poster. Poster: these are strong papers, which have more than one weakness. For example, a well-written paper with solid experiments, but incremental; a paper on a well studied problem with solid theory, but weak experiments; or a novel paper with good experiments, but poorly written. Weak Reject: these are papers that have some promise, but they would be better off by being revised and resubmitted. Strong Reject: these are papers that have major flaws, or have been done before. Oral/Poster
Preliminary Evaluation. Please indicate to the AC, your fellow reviewers, and the authors your current opinion on the paper. Please summarize the key things you would like the authors to include in their rebuttals to facilitate your decision making. There is no need to summarize the paper. The paper is well written with good results.
Confidence. Write “Very Confident" to stress that you are absolutely sure about your conclusions (e.g., you are an expert who works in the paper's area), “Confident” to stress that you are mostly sure about your conclusions (e.g., you are not an expert but can distinguish good work from bad work in that area), and “Not Confident” to stress that that you feel some doubt about your conclusions. In the latter case, please provide details as confidential comments to PC/AC chairs (point 7.). Confident
Final Recommendation. After reading the author's rebuttal and the discussion, please explain your final recommendation. Your explanation will be of highest importance for making acceptance decisions and for deciding between posters and orals. Include suggestions for improvement and publication alternatives, if appropriate. No change in my ratings.
Final Rating. After reading the author's rebuttal, please rate the paper according to the following choices. Oral/Poster