Update 2024-11-27-workshop-report.md

_posts/2024-11-27-workshop-report.md

@@ -5,24 +5,25 @@ author_profile: false
 
 This year, the number of workshop proposals submitted was a giant increase over previous years, resulting in a very high number of rejections, so we thought it might be useful to write a blog post detailing some of the decisions. The vast majority of submitted proposals were very strong. We had to reject many very interesting and timely workshops.  We tried a number of options to be able to accept as many workshops as possible, including asking some organizers to combine forces, but in the end, we kept running up against venue-imposed space constraints. In this blog post, we’ll go over the number of submissions versus placements, the review, voting, and decision process, and the process for combining related workshops. We also cover the process for allocating to the conferences.
 
-**Submissions and Workshop places**
+**Submissions and Workshop places** <br>
 For this year’s call for proposals, we received 139 workshop proposals to be distributed over 77 workshop slots at three conferences (NAACL, ACL, and EMNLP). We have accepted 77 workshops, some of which are two proposed workshops that were combined to fill a single workshop slot, 7 are two-day workshops, 1 will be conducted entirely virtually, and two workshops are now half-day workshops rather than full-day workshops. In contrast, in 2023 there were four conferences with a total of 89 workshop slots distributed over 83 accepted workshops. Our goal as workshop chairs is to accept as many workshops into the *ACL conferences as possible, but the space constraints at conference venues mean that there is a hard upper limit to the number of workshops that can be accepted. This number of workshops is further limited by a growing set of long-standing and SIG-sponsored workshops. This means that we have had to reject many good, and even great, workshop proposals on account of space. 
 
-**The Review, Decision, and Allocation Process**
+**The Review, Decision, and Allocation Process** <br>
 Each workshop proposal was reviewed by two Workshop Chairs and the ACL Workshop Officer, henceforth Chairs, respectively, which resulted in 45-48 proposals for each person to review. The process was then followed by an independent voting process, where all Chairs read the reviews, and voted on all proposals–aside from the proposals that they had reviewed or had conflicts of interest with. Once all Chairs had voted on all proposals, we met to provide a first draft of the accept/reject decisions. During this meeting, the reviews served as a starting point for discussing each proposal which also took into account the proposal, and the vote distribution. The numerical scores from the reviewers were used in cases where we had to make decisions between pairs of workshops (e.g., when there were multiple proposals on similar topics). Even though we considered the space limitations during this first round, the first draft of decisions contained many more workshops to be accepted than there was space for – the vast majority of proposals were interesting and well-written. 
 
 This makes the process of accepting and allocating workshops a complicated puzzle, where a certain number of workshop places are available, and the goal is to accept as broad a spectrum of proposals along the lines of topic and timeliness while also attending to longstanding and continually developing areas of inquiry, e.g., linguistic annotation. There is no perfect way of making such determinations, and we took a consensus-driven approach with a broad set of considerations of which proposals to accept. For instance, among other things, we consider the novelty of the proposal, with regard to work published in the main conference and other workshop proposals; the communities around the workshop, for example, the size of the community or the importance of growing the community; the coverage in all three conferences of broad and major scientific topics; and the importance of the topic for the community, such as workshops on evaluation and linguistic annotation which have a particular importance to the NLP field. 
 
 Taking our draft of accepted workshops, we first sought to combine similar high-quality workshops (more on this below), then the Workshop Officer assigned a draft allocation, assigning as many workshops as possible to their first preference while maintaining a balance in workshop sizes and topics across the conference. This draft was discussed and edited by the remaining Chairs. Our goal was also to accept as many workshops as possible when allocating workshops, but this posed another significant challenge: 68.4% of submissions named ACL as their first choice, 17.1% named EMNLP, and only 14.6% named NAACL as their first choice location. Second choice preferences: 42.1% EMNLP, 28.9% ACL, 15.8% NAACL, and 13.2% with no preference. This distribution of preferences meant departing, in a small number of cases, from venue preferences to make space rather than rejecting workshops if there was not enough space at one of their preferred venues. Once all changes had been agreed on and all merging of workshops had been concluded, we proceeded to send out notifications for workshops.
 
-**Combining workshops**
+**Combining workshops** <br>
 Given the large number of workshop proposals submitted, there was also a high chance of multiple high-quality workshop proposals covering the same topics. Rather than accepting just one workshop on a topic and rejecting all of the others, we reached out to the organizers of workshops covering similar topics and asked if they would want to merge, understanding that there would otherwise only be space for one of the proposals. 10 workshops agreed to merge into five single workshops, and 2 workshops suggested being half-day workshops instead (which we gladly accepted). For workshops that declined to merge, only one workshop could be accepted, even though there were no meaningful differences in quality.  
 
-**Conclusions**
+**Conclusions** <br>
 The earliest conference in the cycle is always the least preferred conference: last year it was EACL, this year it’s NAACL. We encourage workshops to structure their work such that they can be conducted at any conference. As our goal is maximizing acceptance, a workshop may not be assigned to their preferred venue (though we do try to take hard constraints into account). Moving forward, we will reconsider the proposal submission timeline to add more time prior to the first conference in the cycle–but we haven’t yet investigated the broader implications of a change in timeline. Making the decisions was difficult, and we had to consider many factors beyond numerical scores to ensure a good set of workshops for the upcoming conference cycle (diversity of topics, inclusion of long-running/expected workshops, and making space for new workshops). The decision to reject workshops was, in some cases, particularly difficult as we also rejected some of the workshops proposed by the Chairs and WO. For various reasons, there is a very limited number of workshop slots this year while seeing a large increase in proposals. This is likely to change in future years, but the number of workshop slots is inherently variable and is dependent on the physical space at the conferences. 
 
 The process is quite a fraught one that is made increasingly difficult as the number of proposals increases, so we will be revisiting the process to improve the review and decision process next year. So if you have ideas and feedback, please do get in touch with the workshop chairs! 
 
+<br>
 
 1. Sheer Numbers
    - No AACL (will be later in separate process)