This repository contains the code and data for our CoNLL 2023 paper. We show that humans and LMs behave differently in their memorization ability, and we add a learned recency bias to the LMs to make them more similar to humans.
The notebook Behavioral analyses.ipynb contains code to load & visualize the behavioral data, vanilla LM performance, and optimized (with recency bias) performance.
The script attn_optim.py contains the code to optimize an attention bias to match behavioral data. It saves metrics (e.g. corr. with behavioral data, validation loss) during & after training, as well as the learned parameters. You will need a GPT-2 with word-level tokenization, which you can download here. (This takes ~12 minutes to run on a GTX 1080, without much optimization.)
The attention biasing is implemented with a PyTorch forward hook.
attn_optim_combos.sh will run the optimization script multiple times (random initializations) for every layer and stimulus. (This took several hours on our hardware.)
finetune_gpt2_wordlevel.py is a script that converts GPT-2 from BPE to word-level (whitespace) tokenization, as was used in this paper.
Code is licensed under the MIT license. Data is licensed under CC Attribution-NonCommercial (CC-NC).
If you use code or data from this repository, please cite the official CoNLL publication: https://aclanthology.org/2023.conll-1.5/
A version of the paper is also on arXiv: https://arxiv.org/abs/2310.06408