Materials for the EACL 2023 Ethics Tutorial: Understanding Ethics in NLP Authoring and Reviewing.
This repository archives the materials in a reusable form, related materials (ACL Stakeholder Survey) and also links to other contemporary resources. If you see things missing or needing maintenance, please file an issue or a pull request.
Authors:
- Luciana Benotti (Universidad Nacional de Córdoba)
- Karën Fort (Sorbonne Université and LORIA)
- Min-Yen Kan (National University of Singapore)
- Yulia Tsvetkov (University of Washington)
We recommend using our base materials here and adapting them to your setting. Minimally, you need to create your own shortlinks and URL codes to your version of the materials before running, as the archived materials reference our EACL 2023 instance.
For our tutorial setting in an audience participatory style and approximately 20+ participants and 4 organisers, it was important to:
- limit the number of groups so that each group could present within the session's time limit;
- designate a particular organiser to facilitate virtual participants;
- have live edit access to online documents, to allow participant leads to note take and present.
Other formats might consider:
- Lecture only: use and adapt the lecture materials from the first and seventh segments of the tutorial.
- Experiential only: use and adapt Segments 2-6, which asks participants to read and critique abstracts that bring up common ethical issues in our community.
- Student (Homework) Assignment: curate sources from the accompanying Ethics Reading List that the committee maintains, and ask participants to read and write their reflections on.
If you run a course, tutorial or other session based on these materials, we'd love to hear from you! Please get in contact with us, and we may also (with your permission) list your course or materials in the Resource segment. Also please do cite our tutorial abstract as a means of acknowledging the helpfulness of the materials. Thank you!
@inproceedings{benotti-etal-2023-understanding,
title = "Understanding Ethics in {NLP} Authoring and Reviewing",
author = {Benotti, Luciana and
Fort, Kar{\"e}n and
Kan, Min-Yen and
Tsvetkov, Yulia},
booktitle = "Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: Tutorial Abstracts",
month = may,
year = "2023",
address = "Dubrovnik, Croatia",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2023.eacl-tutorials.4",
pages = "19--24",
abstract = "With NLP research now quickly being transferred into real-world applications, it is important to be aware of and think through the consequences of our scientific investigation. Such ethical considerations are important in both authoring and reviewing. This tutorial will equip participants with basic guidelines for thinking deeply about ethical issues and review common considerations that recur in NLP research. The methodology is interactive and participatory, including case studies and working in groups. Importantly, the participants will be co-building the tutorial outcomes and will be working to create further tutorial materials to share as public outcomes.",
}
(these are internal links to the sections below)
In 2022, the ACL Ethics Committee (AEC) decided to propose to run a tutorial on ethics and its impact on ethics in both authoring and reviewing aspects for the community of CL/NLP scholars and practitioners. As part of this process, the committee chairs and Luciana Benotti (a member of the committee) put together a the proposal file and submitted it to the joint call for tutorial proposals.
The tutorial was eventually accepted to run at EACL 2023 (Dubrovnik, Croatia, 2-6 May 2023) on Saturday, 6 May as part of the 1/2-day AM events. We produced a teaser to accompany the published tutorial abstract.
The tutorial and its materials* were presented in English. It was structured as per the proposal, in seven segments, each approximately 30 minutes long. It was participatory in nature, requiring the audience to work in groups on synthetic problematic abstracts written by the proposers that represent common ethical issues experienced by ethics review chairs. This required a good facilitator-to-group ratio, to ensure all of the participants had a chance to reflect, contribute and be heard, and for the facilitators to keep the session on track.
*(Due to the sensitive nature of our EACL 2023 participants' contribution, we have scrubbed the QR coded and shortlinked materials of their presence; if any party wants to access the original document we have these archived but not in this repository nor linked to here; please contact us directly)
| Segment Topic | Led By |
|---|---|
| 1. Introduction and Foundations for Ethics | Presenters |
| 2. Case Studies: Problematic Ethical Research — First reading | Participants |
| 3. Structured Interaction / Dialogue | Presenters,Participants |
| 4. Case studies — Second reading (Rotation) | Participants |
| 5. Group Presentations Group | Leads |
| 6. Summary and Common Issues | Presenters |
| 7. Discussing and Troubleshooting Ethics and Further Resources | Presenters |
We conducted the tutorial in an active classroom style, where participants were self-organized into small groups (in our instance, 2 groups of about 10 participants each), electing leads for subsequent group presentation and worked through the exercises to record their reactions to the materials and identify the issues and conducive outcomes.
After introducing the organisers and the format and resources of the tutorial, the first segment featured two lecture segments, both focused on introductory and foundational material on ethics.
The presentation deck for the tutorial lives in a public, shared Google Slides permalink: http://bit.ly/eacl23-ethics-slides but for posterity, we have also archived forms here in this repository:
- PDF, local copy - created from the Google Slides origin deck
- Microsoft Powerpoint, local copy - created from the Google Slides origin deck
Segment 1A (Slides 6-39) covers the definition and scope of ethics; its origins; virtue, deontological and utilitarian forms of ethics; experiments and policy reactions to ethical violation (Nuremburg Trials and Code, Belmont Report). This segment was presented by Karën Fort.
Segment 1B (Slides 40-70) focuses on appropriate use and application, biases and evaluation, covering hypothetical and real instances of ethical problems in recent research (Chicken dilemma, IQ prediction, sexual orientation prediction -- the "Gaydar" paper, and how AI and NLP technology influences people both directly and indirectly. This segment was presented by Yulia Tsvetkov.
Segment 7A (Slides 86-99) reprised work from Benotti and Blackburn (2022) Ethical Consideration Sections in Natural Language Processing Papers, which examined such written ECS in NLP papers to analyze the issues covered in terms of 1) research benefits, 2) potential harms, and 3) vulnerable groups affected. It covers the differences between geographical region's reporting of such issues. This segment was presented by Luciana Benotti.
The remainder of the slides (Segment 7B) are framing materials to help introduce the tutorial, and set up the abstract critique exercise. This segment was presented by Min-Yen Kan and all presenters.
The files of the recordings are too large to fit into this repository, but the below links to the segments of the scrubbed recordings.
-
(Segment 1A) - Introduction and Foundations for Ethics - Part 1 - Karën; 22m49s
-
(Segment 1A) - Introduction and Foundations for Ethics - Part 1 (with Q/A) - Karën; 40m17s
-
(Segment 1B) - Introduction and Foundations for Ethics - Part 2 - Yulia; 27m18s
-
(Segment 1B) - Introduction and Foundations for Ethics - Part 2 (with Q/A) - Yulia; 46m47s
-
(Segment 7A) - Ethical Consideration Sections - Luciana; 17m42s
-
(Segment 7B) - Conclusion - Min; 4m54s
The hands-on activity in the tutorial starts with a 5-minute introduction of the activity and a short review of the individual abstracts, which are also described in the slides. Participants were grouped into random groups, which had to elect a scribe and a presenter. The scribe serves as a secretariat for typing in the notes from their group; and the presenter is delegated as the person in the group to present the findings to the entire tutorial audience. Each group was assigned one of the abstracts to read and critique.
The critique of the abstracts is run in two phases. In the first phase, groups discuss first to prepare a single slide for silent sharing with other groups (Segment 2). In this second phase (Segment 4), the groups could enlarge their thinking by reflecting on either additional abstracts, the single shared slide from each group, or both.
After both phases were finished, each group presented elements of their findings, with the faciltators structuring and probing for issues and clarity on different aspects.
The activity was run using a scribe document where the organizers communicated information (the abstracts below and the instructions) and solicited the audience's feedback on the overall session.
- Template Scribe Document (.pdf) - scrubbed
- Template Scribe Document (.docx) - scrubbed
- Template Scribe Document (live Google Doc) - view-only; scrubbed
However, for the group activities, we used an editable online presentation deck such that each group could create 1-3 slides (1 slide max in the first phease) in their respective subgroups.
- Template Group Slide Presentation (.pdf) -scrubbed
- Template Group Slide Presentation (live Google Slide link) - view only; scrubbed
We provide 6 abstracts below with a gloss of their topical concern. In our EACL 2023 instance, we ran only the first three, due to the smaller number of participants.
- Facial Recognition
- Social Media Dataset Collection
- Cost-prohibitive Language Models
- Language Resource Collection from Protected Groups
- Multilingual Sentiment and Crowdsourced Annotation
- Large language model use in healthcare
- The ACL Ethics Stakeholder Survey - Prior to the tutorial, the AEC also conducted a survey of our ACL stakeholders, to help priortize ethics needs of our community. This tutorial is a direct result of that mandate. To be presented live at ACL (Toronto, Canada) on 11 July 2023.
- Full report for the survey (draft): Yes, We Care (more)! Results of the 2021 Ethics and Natural Language Processing Survey
- Associated presentation slides
- The ACL Ethics Reading List - https://github.com/acl-org/ethics-reading-list - also mentioned in Slide 110
The tutorial (Slide 109) also concluded with a list of faculty courses (by our presenters, certainly there are more out there) that have also taught similar topics. For expediency we list them directly here:
- (In French and English) Karën https://members.loria.fr/KFort/teaching/nancy-tours-etc/
- (In English) Yulia https://courses.cs.washington.edu/courses/cse582/23sp/
- (In Spanish) Benotti and Alonso Alemany. Ética Práctica para Ciencia de Datos. https://sites.google.com/view/etica-practica-cd/
All materials in this repo are CC-BY-4.0
Our presenters would also like to thank the entire ACL Ethics Committee (AEC) for their support and endorsement of the process.
- Luciana Benotti, committee member (Americas) [she/her] 2021–2024
- Mark Drezde, committee member (Americas) [he/him] 2021–2024
- Karën Fort, co-chair (Europe/Africa) [she/her] 2021–2026
- Pascale Fung, committee member (Asia/Oceania) [she/her] 2021–2024
- Dirk Hovy, committee member (Europe/Africa) [he/him] 2021–2024
- Min-Yen Kan, co-chair (Asia/Oceania) [he/him] 2021–2026
- Jin-Dong Kim, committee member (Asia/Oceania) [he/him] 2021–2024
- Malvina Nissim, committee member (Europe/Africa) [she/her] 2021–2024
- Yulia Tsvetkov, co-chair (Americas) [she/her] 2021–2026