------------------------- METAREVIEW ------------------------ Based on the reviews and scores of the reviewers, I would suggest acceptance. All the reviewers agree that the topic is of value and fit at UMAP, and all agree on the relevance of the contribution and the soundness of the methodology. However, for the publication, I agree with reviewer 2 who requested that the theoretical implications should be discussed. ----------------------- REVIEW 1 --------------------- SUBMISSION: 7606 TITLE: Avoiding Decision Fatigue with AI-Assisted Decision-Making AUTHORS: Jessica Echterhoff, Aditya Melkote, Sujen Kancherla and Julian McAuley ----------- Overall evaluation ----------- i) contributions: The paper proposes and assess the use reinforcement learning to optimize decision-making in sequential recommendation settings to reduce decision fatigue ii) relevance: the paper is very relevant for the UMAP conference, although the evaluation is mostly based on data rather than on actual user experience iii) theoretical foundations: the work is well grounded in the domain of recommendation systems and reinforcement learning; the state of the art is clearly explained and discussed iv) research methods: the research method is robust; nevertheless I would have liked a user evaluation that actually involve final users in an ecological tasks by measuring their cognitive fatigue v) evidence and analysis: the analysis is sound and the analysis convincing vi) reproducibility: the procedures are clear, not clear if the data will be made available vii) practical impact: a novel recommendation approach viii) communication effectiveness: the work is well presented ----------- Main contribution ----------- The paper proposes and assess the use reinforcement learning to optimize decision-making in sequential recommendation settings to reduce decision fatigue. ----------- Strengths ----------- The paper is clear with solid methodology and a sound analysis. The state of the art clearly framed. ----------- Limitations ----------- I think that the major limitation is that decision fatigue is not really defined or measured but just meant as synonym for “smaller number of steps”. I think that it is might not be the whole story, possibly not all the choices are equal and some relevant parts of decision fatigue might also be due to the context in which the choice is made. All these aspects (and possibly other similar ones) are not considered in the paper. I think that it is still an interesting paper but the title is quite misleading. ----------- Showstopper in content. ----------- No fatal error. ----------- Showstopper in presentation. ----------- The presentation does not have serious issues; nevertheless, I feel that the title promised more than what it actually delivers because of the shallow use of the concept of cognitive fatigue. ----------- Ethical concerns ----------- Given the very limited user study, I can't see ethical concerns ----------- Recommendation ----------- SCORE: 1 (Weak accept) ----------- Reviewer's confidence ----------- SCORE: 4 (High) ----------- Best paper ----------- SELECTION: no ----------- Demo candidate ----------- SELECTION: no ----------- LBR candidate ----------- SELECTION: no ----------------------- REVIEW 2 --------------------- SUBMISSION: 7606 TITLE: Avoiding Decision Fatigue with AI-Assisted Decision-Making AUTHORS: Jessica Echterhoff, Aditya Melkote, Sujen Kancherla and Julian McAuley ----------- Overall evaluation ----------- The paper presents a novel approach to movie recommendation by minimizing the need for decisions during the recommendation process. This is achieved by applying a Decision Minimizer Network proposed as a contribution by the authors. The network is trained by reinforcement learning that interactively adapts to user preferences and explicit feedback. The work is highly relevant for movie recommendations, and the paper addresses a current research desideratum, especially with regard to UX. The theoretical basis for decision fatigue or also called ego-depletion in psychology, is nevertheless somewhat contested. Systematic reviews found evidence for the effect, by a multi-lab replication did not find any evidence for ego depletion. Ignoring this theoretical problem, the collected data and evidence for the effectiveness of the DMN is presented in a solid fashion, comparing the approach against other state-of-the-art recommendation practices. The data made available can help others replicate similar studies and use the sequential data set. However, no real human evaluation of the final system was conducted. So it is unclear whether less decision fatigue would be the result of the reduced amount of decisions. It could be the case that these (fewer) decisions are actually harder to make. Similar work has been done using one-armed-bandit approaches or Bayesian approaches. These should be discussed. The approach does reduce the amount of decisions which is similar to uncertainty-reduction approaches. The method could be applied to a wide variety of domains and thus would have high practical applicability. Both text and figures are very well-crafted leading to clear communication. The authors also reflect on ethical implications on their research, however this section could be extended. Not only commercial actors might be interested in misusing such a system. It could e.g. also be used to identify information that manipulates people more easily (fake news, propaganda recommendation). ----------- Main contribution ----------- The main contribution are the Decision Minimizer Network and the data set for replication. Both are novel. ----------- Strengths ----------- The Data collection from "real" users and the quality of the automatic evaluation of the system are solid. The presentation is well crafted. ----------- Limitations ----------- The problem of ego-depletion should be discussion. It is unclear whether this effect really exists and to prevent further unreflected propagation of a possibly out-dated phenomenon this should be discussed in the related-work section and the limitations (Hagger et al.). The ethical discussion needs more work focusing on dual-use aspects in an AI world. The limitions should make clear that no reduction of actual fatigue was measured, only a reduction of choices. ----------- Showstopper in content. ----------- If future research shows that ego-depletion was bad science (which is possible), this work would need rephrasing. It could easily be presented as saving time as well, without having to go through the lens of ego-depletion. This would require a rewrite. ----------- Showstopper in presentation. ----------- no ----------- Ethical concerns ----------- None beyond what was mentioned above. ----------- Recommendation ----------- SCORE: 0 (Borderline) ----------- Reviewer's confidence ----------- SCORE: 5 (Expert) ----------- Best paper ----------- SELECTION: no ----------- Demo candidate ----------- SELECTION: no ----------- LBR candidate ----------- SELECTION: yes ----------------------- REVIEW 3 --------------------- SUBMISSION: 7606 TITLE: Avoiding Decision Fatigue with AI-Assisted Decision-Making AUTHORS: Jessica Echterhoff, Aditya Melkote, Sujen Kancherla and Julian McAuley ----------- Overall evaluation ----------- In the paper reinforcement learning is used to optimize decision-making in sequential recommendation settings. Decision Minimizer Network is introduced and evaluated using a hybrid (synthetic and real dataset) which outperforms heuristic next item prediction. The paper is relevant as shopping and movie websites have large catalog of products making it difficult for users to make multiple sequential decisions. The paper is well written and comprehensively covers the background and related literature (eg on decision fatigue) ----------- Main contribution ----------- Uses Decision Minimizer Network inspired by cognitive science to reduce the number of sequential decisions that humans need to make. ----------- Strengths ----------- - Creates a hybrid synthetic and real dataset - Many practical applications on product websites ----------- Limitations ----------- The paper does a good job of highlighting the limitations and risks (eg over accelerating human decisions) ----------- Recommendation ----------- SCORE: 2 (Accept) ----------- Reviewer's confidence ----------- SCORE: 3 (Medium) ----------- Best paper ----------- SELECTION: no ----------- Demo candidate ----------- SELECTION: no ----------- LBR candidate ----------- SELECTION: no