======================================= SUBMISSION: 36 TITLE: WildFX: A DAW-Powered Pipeline for In-the-Wild Audio FX Graph Modeling ----------------------- REVIEW 1 --------------------- SUBMISSION: 36 TITLE: WildFX: A DAW-Powered Pipeline for In-the-Wild Audio FX Graph Modeling ----------- Overall evaluation ----------- SCORE: -2 (reject) ----- TEXT: The main contribution of this paper is an open-source tool for creating and augmenting audio datasets using a Docker container running a DAW. Important strengths of this tool are its open-source nature, the proposed data structure, and the optimized workflow leveraging parallelization. The tool can be valuable for the community, especially for machine learning research. The paper is well structured and globally clear. However, I have several concerns which make me lean towards rejection. The weakness of the experimental part, combined with the unclearness of the underlying research problems this paper addresses, make me believe that this paper is not suited for the DAFx conference. A number of typos and mistakes were found and are highlighted in red in the attached pdf. I put additional remarks on elements highlighted in yellow. A significant issue is that two versions of the conclusion are present one after another. Here are more remarks: ## Introduction - In Sections 1 and 2.1, neural audio effect modeling is said to gather effect identification, parameter estimation, AFx graph estimation, style transfer. I disagree, as for instance the paper PyNeuralFx which you cite states "Neural audio effect modeling aims to use neural networks to simulate and replicate various audio effects typically achieved through traditional digital signal processing (DSP) techniques." - "music producers do not work with Python-based differentiable modules" -> It would be valuable to cite a few references that aim to make neural effects usable: RTNeural (https://urldefense.com/v3/__https://arxiv.org/abs/2106.03037__;!!Mih3wA!E9-RcfRE6zDp9_5VMxQRzIz4OjE9czQWj8CzJB3hK2UmBHvsO66sPOa6luKJrJlafKkz5vFRLsTieB2qOqI$ ) and Neutone (https://urldefense.com/v3/__https://arxiv.org/abs/2508.09126__;!!Mih3wA!E9-RcfRE6zDp9_5VMxQRzIz4OjE9czQWj8CzJB3hK2UmBHvsO66sPOa6luKJrJlafKkz5vFRLsTizRwEFBs$ ). - "Additionally, there is little evidence that fully neural AFx modules are comparable in performance to professional-grade plugins, with even simple neural baselines (i.e. learning parameters of a plugin rather than directly modeling the plugin itself) performing similarly [12, 21]." "Despite these advances, fully “black-box" neural audio effect modules often underperform compared to their traditional DSP counterparts, with even simple neural “grey-box" parameter controllers for conventional plugins showing comparable performance to end-to-end neural models [12, 21]." -> It's not clear what exactly you're criticizing. The task of modeling an audio effect using machine learning is distinct from that of creating an effect plugin. What do you mean by "performance"? Are you referring to the accuracy of the model? If so, the connection to WildFX isn't clear. Does WildFX aim to improve the modeling accuracy? - "we introduce WildFX, the first comprehensive end-to-end pipeline (to the best of our knowledge)" -> There is DawDreamer (https://urldefense.com/v3/__https://github.com/DBraun/DawDreamer__;!!Mih3wA!E9-RcfRE6zDp9_5VMxQRzIz4OjE9czQWj8CzJB3hK2UmBHvsO66sPOa6luKJrJlafKkz5vFRLsTiDG-Ebaw$ ) which was presented as a Late-Breaking Demo at the 2021 ISMIR Conference, and should be cited here. - "ensuring maximal diversity and ecological validity" "Our results highlight WildFX’s potential to bridge the gap between AI research and practical DSP demands, enabling more ecologically valid neural modeling of audio processing workflows used by professionals in the field." -> the term "ecological validity" is unclear -> the expression "bridging the gap between AI research and DSP" repeated throughout the paper seems like an overstatement; WildFX is a dataset generation tool and a data augmentation pipeline. - The introduction fails to illustrate the research problems that led to making WildFX, and how WildFX can be employed to mitigate these problems. WildFX's role as a tool for creating and augmenting datasets should be highlighted, and some claims about its potential and the "underperformance" of other approaches should be softened or backed by better arguments. ## Experiments - The setups of the experiments lack detail. The loss functions are not revealed, and learning rate scheduling should be detailed. It is unclear how the validation set is employed, or even if validation is performed at all. Is the model checkpoint used for evaluation the one that minimizes validation loss, or is it the last obtained one after 20k/3k steps? - The robustness of the experiments is very questionable. In the first experiment, only one model is trained in both setups. In the second one, one model is trained for each of the 4 settings. Given the high variance that results from training identical machine learning models but with different random seeds, training only one model in each setting is not enough to conclude anything. Moreover, the number of training steps is extremely low: 3,000 steps for the second experiment. [16] use 200,000 training steps. In my view, that invalidates the results. - Given the limited number of pages, I would recommend keeping only one experiment among both, and detailing them better as well as going further in the observation and analysis of the results. Notably, examples and illustrations of how the trained models behave would be very valuable to better understand the performance metrics of Table 4. ----------------------- REVIEW 2 --------------------- SUBMISSION: 36 TITLE: WildFX: A DAW-Powered Pipeline for In-the-Wild Audio FX Graph Modeling ----------- Overall evaluation ----------- SCORE: -2 (reject) ----- TEXT: This paper presents an engineered artefact, namely WildFX, a system for running audio effects and processing pipelines. while the introduction makes some interesting points, such as the need for researchers to be aware of pro-grade audio processing tools, it tends towards repetition as well as making significant statements about the quality of their approach compared to others, without any evidence (that comes later). The fact that this is used for generating a dataset is almost a throwaway, yet it is ought to be the major contribution of the paper, and we see the use of generated data later to present the benefits of the architecture. A lot of the Intro reads like a sales pitch. Section 2.2 basically repeats what is in the intro. In this review, i largely skipped over section 3 because a) it is the engineering detail of the system, and b) very turgid. It would be far better presented as an appendix to what is supposed to be a research paper, not an engineering blueprint. Section 4 discusses the experiments that were performed to demonstrate the fit for purpose of WildFX. Given that the purpose is to show its data generation capabilities, the data set description is lightweight. The first experiement is source separation. Although they use a very recent and highly capable SS model, they only use it to separate bass from drums. This is hardly challenging. I am rather surprised at how poor the SDR measures are. The claim is that their augmented dataset (augmented thanks to WildFX) delivers superior performance and this is due to it generating 4 times as much data for training. Yet they didn't explore what happened when other ways to generate 4x more data is used, so this really is not a rigorous result. The other experiment involves estimating mixing graphs. to quantify this, 5 metrics are used. But these are described in a very cursory manner and this reviewer is none the wiser as to what each of them does. There aren't even any references to these metrics for a reader to follow up on. The conclusions are baffling. IT is stated that the approach in this paper is worse than in [16]. Really, the paper should have presented how well [16] did. The fact that they don't attain similar perofrmance is troublesome. Then they claim their approach is state of the art. It is not clear how they get from being worse than a 2023 paper to being state of the art! The paper would be greatly improved by drastically reducing section 3 (especially if it were in an appendix) and significantly improving the experiments, as well as fully justifying the claims made. ----------------------- REVIEW 3 --------------------- SUBMISSION: 36 TITLE: WildFX: A DAW-Powered Pipeline for In-the-Wild Audio FX Graph Modeling ----------- Overall evaluation ----------- SCORE: 1 (weak accept) ----- TEXT: This work proposes an executable data generation framework for audio effects graph. The proposed framework has complete capability in modeling any possible audio processing graph, including sidechaining. As mentioned in the paper, such framework can help curating datasets for audio effects processing models and also serve as an augmentation method for other downstream tasks. Still, I have some concerns about this work, as listed below: - Although the work mentioned the gap between academic packages and commercial tools (e.g. Albeton, Logic Pro), the proposed framework is based on REAPER, which is not the most popular DAW in the commercial field. Also, the set of plug-in is also quite limited. It is understandable that academic work has difficulty to include commercial products, but then the paragraph that mentioned about the gap lose its ground in technical sense (i.e. the gap is not due to technical difficulty). - The graph generation part is unclear. Given there are some limitations in graph configuration and requires extra treatments such as inserting stub empty chains to solve, it becomes important that whether the generation pipeline intentionally included these treatments to ensure the generated graphs can cover these cases. - The experiment results do not entirely justify the effectiveness of the proposed work. One of the two experiments do not show performance improvement over the original model. - This work requires docker, which runs in root privilege. It's not proper to say that the proposed work allows full audio processing in Linux without root privileges. Minor issues: 3.4.2 : "Therefore, we need divide the whole processing graph into chunks" -> maybe it should be "we need to" ? Section 5: The third paragraph seems to be a repetition of the first one. Overall, I believe the work is helpful for future research on audio effects processing. Despite there are some concerns, I still lean toward accepting this paper. ----------------------- REVIEW 4 --------------------- SUBMISSION: 36 TITLE: WildFX: A DAW-Powered Pipeline for In-the-Wild Audio FX Graph Modeling ----------- Overall evaluation ----------- SCORE: 2 (accept) ----- TEXT: The authors present a pipeline for efficiently rendering audio effect plugins in Linux environments. Crucially, the proposed pipeline combines a Docker container (which runs REAPER using a Python interface) with Wine and yabridge to load Windows VST plugins, not just Linux ones. Further, the pipeline supports heterogeneous audio processing graphs and efficiently computes them via parallelism on the CPU. Overall, this paper makes a strong contribution to reducing the gap between academic research and the real-world use of audio effects. Many of the recent data-driven methods rely on a limited set of processors, both in terms of diversity and implementation "quality," roughly speaking. This is in large part because widely used VST plugins are very cumbersome (if possible) to use in typical machine learning environments. The authors proposed a well-engineered solution to this problem, so I recommend acceptance. That said, I still want to raise several issues, including potential weaknesses and unresolved questions in the paper. I am not asking authors to resolve all of them. Rather, I am writing them to share my thoughts and to position the paper correctly. First, the number of plugins the authors used for the experiments seems quite limited. From my experience, more than 50 free Linux VST3 plugins can be loaded even with Pedalboard (similar reports can be found in the ST-ITO paper by Christian et al., who used slightly more than 60 Linux plugins). Therefore, I expected a larger-scale demonstration of WildFX, such as a more comprehensive inclusion of Windows plugins alongside the Linux plugins. Second, the paper proposed a quite sophisticated algorithm for accelerating graph computation. Yet, it is unclear how much speedup it provided relative to simpler alternatives, e.g., project-level parallelism rather than the proposed layer-level parallelism. Third, the paper tried to reproduce (or revalidate) the blind graph estimation model. However, as the author also noted, the dataset is relatively small, so it is unclear whether the results can be extrapolated to a larger scale. Finally, perhaps one of the key research questions could be "How much gap exists between machine learning systems trained on vanilla audio processors versus the actual real-world plugins?" I believe the proposed pipeline could help answer this.