Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

✨ 🍻 feat(question_eval): Integrate mimoracle dataset to question eval experiments #41

Merged
merged 19 commits into from
Nov 7, 2024

Conversation

honghanhh
Copy link
Contributor

@honghanhh honghanhh commented Oct 25, 2024

Fixes

Fixes #37

Description

This PR does

  1. Extract the subset of mimoracle dataset with only meaningful sections:
    allergies|history of present illness|past medical history|discharge medications|social history|medications on admission
  2. Integrate mimoracle dataset to question eval experiments
  3. Log results - generate evaluation with 5 questions for the first 3 documents, each document with 6 above sections:
wandb: Run summary:
wandb:  mean/score/conformity 0.66667
wandb: mean/score/consistency 0.45556
wandb:    mean/score/coverage 0.11111
wandb:   sum/score/conformity 60.0
wandb:  sum/score/consistency 41
wandb:     sum/score/coverage 10
wandb: Run summary:
wandb:  mean/score/conformity 0.74444
wandb: mean/score/consistency 0.6
wandb:    mean/score/coverage 0.11111
wandb:   sum/score/conformity 67.0
wandb:  sum/score/consistency 54
wandb:     sum/score/coverage 10

Technical details

Definition of Done

  • I followed the repository practices (e.g. I used gitmojis).
  • I have added/updated tests according to my changes.
  • I have added/updated documentation according to my changes.

@honghanhh honghanhh changed the title ✨ 🍻 fix(question_eval) Integrate mimoracle dataset to question eval experiments ✨ 🍻 feat(question_eval): Integrate mimoracle dataset to question eval experiments Oct 25, 2024
Copy link
Contributor

@simonmeoni simonmeoni left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have added some comments on separating the script into smaller methods and also considerations on adding the possibility to replace quickly the dataset we want to use using simple config in hydra

@honghanhh honghanhh marked this pull request as ready for review October 30, 2024 15:52
Copy link
Contributor

@RaphGardies RaphGardies left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM for the parts I understand, I made a few minor comments

@honghanhh honghanhh merged commit 2a3b364 into main Nov 7, 2024
3 checks passed
@honghanhh honghanhh deleted the ht/mimoracle branch November 7, 2024 11:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

✨ Integrate mimoracle dataset to question eval experiments
3 participants