We will accept 3 types of papers:

  • survey papers,
  • meta-analyses,
  • and retrospectives.

Survey papers will mention and cluster different types of approaches, provide pros and cons, highlight good source code implementations, applications and emphasize impactful literature. We expect this type of paper to provide a detailed investigation of the techniques, and link similar themes across multiple works.

Meta-Analyses, on the other hand, are forward-looking, aimed at providing critical insights on the current state-of-affairs of a sub-field and propose new directions based on them. These are expected to be more than just an ablation study – though an empirical analysis is encouraged as it can provide for a stronger narrative. Ideally, they will seek to showcase trends that are not possible to be seen when looking at individual papers.

A meta-analysis paper is not the same as a review paper. A review paper aims to summarize and synthesize a wide range of papers in a specific subfield, with the goal of being very thorough, as well as providing some insights as to how the papers relate to each other. On the other hand, the goal of a meta-analysis paper isn’t to summarize the content of papers, but rather to discuss and analyze an interesting aspect of a set of papers (e.g. evaluation methodology, conflicting claims, etc.), or give an opinion about emerging trends. A meta-analysis paper doesn’t have to be thorough (it could discuss only a few papers) or limited to a narrow subfield (it could analyze broader trends across the ML community). They usually also include additional results, figures, or tables.

There are many kinds of meta-analysis papers. Here are a few examples:

  • Highlighting poor scientific practices in a given area (e.g. related to evaluation, scholarship, etc.). An example of a recent paper in this category is “Troubling trends in machine learning scholarship”, Lipton & Steinhardt (2018).
  • Summarizing best practices in a given subfield (e.g. hyperparameter choices, evaluation procedures, etc.)
  • Pointing out conflicting claims in related papers, along with a discussion of these claims.
  • Doing an empirical analysis of a set of related methods that have not been compared before on the same benchmark. An example of a paper in this category is “LSTM: A search space odyssey”, Greff et al. (2015).
  • Describing emerging trends in a specific area, along with a critique or analysis of those trends.
  • Discussing what research areas/ questions are the most important or promising for achieving a certain goal (e.g. for ‘understanding’ natural language, overcoming bias in ML algorithms, creating ‘artificial general intelligence’, etc.) An example of a paper in this category is “Building machines that learn and think like people”, Lake et al. (2016).
  • Giving a historical perspective on how a subfield has changed over a longer timescale.
  • Discussing how the scientific consensus has changed on a specific research question over time.

We strongly encourage junior researchers (especially from eclectic fields) to write surveys & meta-analyses (S&A).

Retrospective is written about a single paper, by that paper’s author(s), and takes the form of an informal blog post. The purpose of a retrospective is to answer the question: “What should readers of this paper know now, that is not in the original publication?” The overarching goal of retrospectives is to do better science, increase the openness and accessibility of the machine learning field, and to show that it’s okay to make mistakes.

How to submit

Submissions to the ML-RSA workshop will be handled through CMT in NeurIPS format anonymized to remove any author related information. While there is no page limit, we expect authors to be concise.

Deadline: 23rd October, 23:59 Anywhere on Earth. Accept/reject notification will be sent out by 31st October.

Final deadline to upload (contributed) paper talks: 14th November

Please see here for the LaTeX style file to be used for final submission.

Submission Policy

  • We follow double-blind reviewing system (all author-identifying information should be removed from submissions).
  • We accept submissions of work submitted elsewhere (including arxiv.org) if it doesn’t violate that venue’s own policy, or substantial extensions of works presented at other venues (not in archival proceedings).
  • We however do not accept work that has been previously published/accepted in a journal or conference proceedings (including the main NeurIPS conference).

Reviewing criteria

We expect all submissions to follow the guidelines of basic courtesy and respect, and to abide by our code of conduct.

Submissions will be evaluated on the following criteria:

  • Adherence to our code of conduct
  • Quality of discussion of limitations
  • Significance of new insights (when applicable)
  • Presentation of new results or figures (when applicable)
  • Thoroughness of literature review (when applicable)
  • Clarity of writing
  • Vulnerability and honesty in discussion, particularly if the submission is by the original author