Changelog

Unreleased

Add RPPS pattern to the pseudonymisation-rules pipeline

v0.4.0 - 2024-24-05

Added eds_pseudo.dates_normalizer to parse ML detected dates and extract their value and format.
Support empty doc._.context field
Update EDS-NLP to v0.10.7:
- fix somes issues with jsonl loading
- more transformer overriding options
- fix out-of-memory issues (auto split transformer input depending on the available memory)
- fixes some multiprocessing deadlock issues
- add chunk sorting option to the lazy collection set_processing method
Replace gen_dataset/train.jsonl with the original fictitious templates and the dataset generation script.
Update the README with the instructions to download the public pre-trained model.
Improve packaging to add evaluation results to the model's meta field and packaged model README (for HF)

v0.3.0 - 2023-12-01

Refactoring and fixes to use edsnlp instead of spaCy.
Renamed eds_pseudonymisation to eds_pseudo and default model name to eds_pseudo_aphp.
Renamed pipelines to pipes
New scripts/train.py script to train the model

Some fixes to enable training the model:

committed the missing script infer.py
changed config default bert model to camembert-base
put config.cfg as a dependency, not params
default to cpu training
allow for missing metadata (i.e. omop's note_class_source_value)

v0.2.0 - 2023-05-04

Many fixes along the publication of our article:

Tests for the rule-based components
Code documentation and cleaning
Experiment and analysis scripts
Charts and tables in the Results page of our documentation

v0.1.0 - 2022-05-13

Inception ! 🎉

Features

spaCy project for pseudonymisation
Pseudonymisation-specific pipelines:
- pseudonymisation-rules for rule-based pseudonymisation
- pseudonymisation-dates for date detection and normalisation
- structured-data-matcher for structured data detection (eg first and last name, available in the information system)
Evaluation methodology