------------------------- METAREVIEW ------------------------ The paper proposes a novel approach to optimizing sequential recommender systems by integrating calibration directly into the loss function, ensuring relevance post-calibration. Extensive evaluations on four datasets show favorable relevance-calibration tradeoffs, with comprehensive ablation studies supporting the results. Despite its strengths in novelty and reproducibility, the paper's introduction lacks clarity, and some design choices and evaluation metrics need better justification. Additionally, the necessity of the re-ranking stage is questioned due to insufficient analysis of the gamma parameter. Overall, the paper's innovative approach merits acceptance, pending improvements in the introduction, figures, and justification of the re-ranking stage. ----------------------- REVIEW 1 --------------------- SUBMISSION: 1196 TITLE: Calibration-Disentangled Learning and Relevance-Prioritized Reranking for Calibrated Sequential Recommendation AUTHORS: Hyunsik Jeon, Se-eun Yoon and Julian McAuley ----------- Overall recommendation ----------- SCORE: 1 (weak accept) ----------- Relevance to CIKM ----------- SCORE: 4 (good) ----------- Originality of the Work ----------- SCORE: 4 (good) ----------- Technical Soundness ----------- SCORE: 4 (good) ----------- Quality of Presentation ----------- SCORE: 3 (fair) ----------- Impact of Ideas or Results ----------- SCORE: 3 (fair) ----------- Reproducibility of Methods ----------- SCORE: 4 (good) ----------- Detailed Comments to the Author(s) ----------- The paper focus on the beyond-accuracy optimization in the context of the sequential recommender systems. The authors propose to modify the loss function so that it would incorporate assumed future calibration-enhancing post-processing. The main idea is that if item A was selected over item B at some time point, the difference in their estimated ratings should be large enough that even after the calibration-enhancing post-processing, these items should not be swapped. In general, this is an interesting and novel idea that could be used for other beyond-accuracy metrics as well. In evaluation, the authors demonstrated a favorable relevance-calibration tradeoff for the proposed method on four different datasets. The authors also conducted a very detailed ablation study, answering some of the concerns I had while reading their proposal. It is also interesting to see that the impact of different design components (loss function, reranking) greatly varies across datasets. Regarding the weak points, the paper's introduction was not very convincing. The authors provided several statements without sufficient justification (details below). These should be either toned down, or additional supportive evidence should be provided. Also, after reading it, I did not have a clear picture of what the authors are trying to solve and how they aim to do it. It would be very helpful to summarize in plain words how (and why) is L_{CD-BPR} defined and why is the relevance priority used. The Figure 1 setting is not very convincing. The description is somewhat sketchy, and it is unclear what exactly is being compared w.r.t. KL divergence. While, in principle, the changes in long-term preferences may indeed happen, I am not convinced (by the evidence provided by the authors) that this change is substantial and rapid enough that it actually matters. This is also illustrated by the results (Figure 5), where the difference between CaliRec and CaliRec+ is almost non-existent in two of the datasets (Grocery and Steam). I tend to disagree with the author's statement in the introduction: "However, applying calibration during reranking often leads to significant degradation of accuracy because they do not consider the impacts of calibration during the training phase." Considering, e.g., the results published in the original CaliRec paper, calibration post-processing, if not weighted too strongly, can maintain the same or similar relevance (see Table 2 for \lambda = 0.2 and 0.5). I recommend toning down the statement or making it more specific. Also, the mechanism of why calibration should degrade relevance, "especially in the sequential setting", is not fully explained. ----------- Summary to support your recommendation ----------- I quite like the paper's main idea; it seems novel with potentially broader applications than those proposed in this paper. The evaluation is definitely a strong point of the paper. As for the weaknesses, the main one is the introduction, which should be reworked thoroughly. Otherwise, the paper seems acceptable and may be a nice addition to the CIKM program. ----------------------- REVIEW 2 --------------------- SUBMISSION: 1196 TITLE: Calibration-Disentangled Learning and Relevance-Prioritized Reranking for Calibrated Sequential Recommendation AUTHORS: Hyunsik Jeon, Se-eun Yoon and Julian McAuley ----------- Overall recommendation ----------- SCORE: 1 (weak accept) ----------- Relevance to CIKM ----------- SCORE: 4 (good) ----------- Originality of the Work ----------- SCORE: 3 (fair) ----------- Technical Soundness ----------- SCORE: 4 (good) ----------- Quality of Presentation ----------- SCORE: 4 (good) ----------- Impact of Ideas or Results ----------- SCORE: 3 (fair) ----------- Reproducibility of Methods ----------- SCORE: 4 (good) ----------- Detailed Comments to the Author(s) ----------- This paper presents a two-step method for calibrated sequential recommendation. The method is clear and well-motivated, and abundant experiments are provided to support the design of each module in the method. However, the explanation of some model designs is intuitive, the calibration evaluation metric is questionable, and the result representation is a little confusing. Although no fancy technique is adopted in the paper, both calibration-disentangled learning and relevance-Prioritized Reranking modules show simple-yet-effective (and also efficient) solutions for the sequential calibration problem. I think this kind of papers will inspire more research in the future. Strong points: 1. The method is described clearly and well-motivated. 2. Abundant experiments are provided to support the design of each module in the method, further convincing the model design. 3. Details about experiment settings and the source codes are provided, making the work reproducible. (It would be better if the hyperparameters used for the final experiments can be provided in the source codes.) Weak points: 1. The method design has intuitive parts that lack convincing explanations, such as why using only L_CD-BPR is insufficient and the rationale behind Property 1. 2. The definition of the Sequential miscalibration metric might lead to unfair comparisons and bias: due to the presence of alpha in Eq(2), historical items closer to the current time are given higher weight. On one hand, Sequential recommenders are more likely to emphasize more recent history, resulting in unfair comparison between this method and baselines that do not consider history; on the other hand, it might cause recency bias, which would be better to have a discussion. 3. The curves in Figure 5-7 confuses me: How to determine the points on the trade-off curves? If possible, it would be better to add a table of comparable results and significant tests to make the overall results clearer. Additional comments: 1. Some module names and terminology do not accurately convey their actual functions. For example, in Figure 4, "with and without consideration of calibration" suggests two ranking lists, but there is only one. "Calibration disentangle" should imply two parts: one only considering calibration and one only considering relevance, but Eq9 considers relevance, while Eq10 considers both simultaneously. 2. Section3.3 and the Algorithm 1 may be redundant. The overall training process can be introduced at the front of Section3, and the corresponding algorithm illustration can be removed. ----------- Summary to support your recommendation ----------- This paper presents a two-step method for calibrated sequential recommendation. The method is clear and well-motivated, and sufficient experiments are provided to support the design of each module in the method. However, the explanation of some model designs is intuitive, the calibration evaluation metric is questionable, and the result representation is a little confusing. Therefore, I give a weak accept for the paper. ----------------------- REVIEW 3 --------------------- SUBMISSION: 1196 TITLE: Calibration-Disentangled Learning and Relevance-Prioritized Reranking for Calibrated Sequential Recommendation AUTHORS: Hyunsik Jeon, Se-eun Yoon and Julian McAuley ----------- Overall recommendation ----------- SCORE: 1 (weak accept) ----------- Relevance to CIKM ----------- SCORE: 4 (good) ----------- Originality of the Work ----------- SCORE: 4 (good) ----------- Technical Soundness ----------- SCORE: 4 (good) ----------- Quality of Presentation ----------- SCORE: 4 (good) ----------- Impact of Ideas or Results ----------- SCORE: 4 (good) ----------- Reproducibility of Methods ----------- SCORE: 4 (good) ----------- Detailed Comments to the Author(s) ----------- This paper proposes LeapRec for calibrated recommendations. Specifically, this framework includes two stages: Firstly, it incorporates KL divergence-based loss function within the training of a sequential recommendation backbone; Then, it re-ranks the generated recommendations in the training phase, leveraging the relevance property introduced in the paper. WRITING AND PRESENTATION The paper is generally well-written and well-presented. Technically, it is sound. The figures help to understand the framework and motivations. The Pareto-frontiers in the plots of Figure 5 are clear. I think such plots should contain the curves solely, without adding data that can disturb viewing them (e.g., "LepRec draws better trade-off curves" in Figure 5a). The tables are well-organized and suitable for discussing the experiment results. The discussion of the results is a bit chaotic. REPRODUCIBILITY The paper is accompanied by a GitHub repository containing the codes and datasets. PROPOSED METHOD The proposed method employs two stages to generate the final recommendations. In the first stage, the authors formulate a novel pair-wise loss function. They integrate some calibration considerations into BPR by injecting the KL divergence into the idea of BPR. I appreciate such an idea, which is a strong point of the work. Somebody could raise concerns about how they combine this novel loss with BPR into the training model. Indeed, they sum the two losses and weight the proposed loss solely through a weight gamma. Have you thought of combining them through a weighted linear combination, i.e., by weighting also BPR? It could be just a matter of design choice. However, if your approach is chosen, the behavior of the training by varying the value of gamma should be better studied. Indeed, in the experimental setup, the authors fix gamma to 0.1. What happens if we increase by giving more importance to the CD-BPR loss? Is the re-ranking stage still needed? By changing the gamma value, we can probably obtain the same shape of the Pareto frontiers in Figure 5 without re-ranking the items. In the second stage, the authors develop a novel re-ranking strategy based on prioritizing relevance at the higher positions of the recommendation list. The idea is simple but effective, and I enjoy it. EXPERIMENTS The experimental setup is comprehensive. The authors employ four datasets and five baselines. How they choose the best-performing model on the validation set is unclear. The research questions are adequate, and the experiments appropriately answer them. Regarding the performance comparison (Section 4.2.1), the comparison is limited to a qualitative point of view. Indeed, the authors limit the comparison among the baselines by observing the depicted trade-off curves. For instance, the visual dominance of these curves is strongly biased by the values range reported within the axis representing the metrics. The authors could enhance this evaluation from a quantitative point of view through the Pareto-frontier Quality Indicators (see https://dl.acm.org/doi/pdf/10.1145/3604915.3610649 as an example). The ablation and the lambda hyperparameter effect studies are compelling. An additional study about the gamma hyperparameter is required, as mentioned above. ----------- Summary to support your recommendation ----------- Overall, the methods presented in this paper are notable. The paper is well-written and technically sound. Some doubts raise about the need of the re-ranking stage, due to the missing study about the gamma parameter. The considerations above led me to a weak accept score.