Deep NLP @ CIS - LMU

All

31 repositories

GlotLID
Public
Language Identification with Support for More Than 2000 Labels -- EMNLP 2023
language-detection multlingual language-detector language-recognition glot lid language-identification language-classification language-identification-toolkit low-resource-languages
Python
•
Apache License 2.0
•7•93•1•0•Updated Nov 28, 2024Nov 28, 2024
GlotWeb
Public
GlotWeb: Web Indexing for Low-Resource Languages -- under construction.
multilingual dataset glot low-resource-languages news-dataset awsome-list
Python
•
Creative Commons Zero v1.0 Universal
•0•10•4•0•Updated Nov 24, 2024Nov 24, 2024
GlotCC
Public
GlotCC Dataset and Pipline -- NeurIPS 2024
crawler multlingual corpus-linguistics glot language-identification commoncrawl common-crawl glotcc multilingual-dataset glotlid
Jupyter Notebook
•
Creative Commons Zero v1.0 Universal
•0•16•0•0•Updated Nov 1, 2024Nov 1, 2024
oscar-io
Public
Readers/Writers for GlotCC/OSCAR corpus
glotcc glotlid
Rust
•
Apache License 2.0
•1•1•0•0•Updated Oct 23, 2024Oct 23, 2024
ungoliant
Public
🕷️ The pipeline for the OSCAR/GlotCC corpus
glotcc glotlid
Rust
•
Apache License 2.0
•14•3•0•0•Updated Oct 23, 2024Oct 23, 2024
oscar-tools
Public
The original tooling for the GlotCC/OSCAR corpus rewritten in Rust
glotcc glotlid
Rust
•
Apache License 2.0
•3•1•0•0•Updated Oct 23, 2024Oct 23, 2024
2024fall-crosslingual-vlm-block-seminar
Public
Materials of 2024 Fall cross-lingual visual language models block seminar at LMU Munich.
Python
•0•2•0•0•Updated Oct 14, 2024Oct 14, 2024
cisnlp.github.io
Public
Homepage of cisnlp
SCSS
•
MIT License
•0•3•0•0•Updated Oct 14, 2024Oct 14, 2024
MEXA
Public
Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignment
multilingual evaluation embeddings evaluation-metrics cross-lingual multilingual-nlp large-language-models decoder-only
Python
•
Apache License 2.0
•0•8•0•0•Updated Oct 10, 2024Oct 10, 2024
LangSAMP
Public
LangSAMP: Language-Script Aware Multilingual Pretraining
multilingual language-embeddings continued-pretraining
Python
•0•1•0•0•Updated Sep 30, 2024Sep 30, 2024
analogical_reasoning
Public
JavaScript
•0•0•0•0•Updated Jul 28, 2024Jul 28, 2024
Transliteration-PPA
Public
Breaking the Script Barrier in Multilingual Pre-Trained Language Models with Transliteration-Based Post-Training Alignment
multilingual transliteration cross-lingual-transfer script-barrier
Python
•0•2•0•0•Updated Jul 3, 2024Jul 3, 2024
lohoravens-webpage
Public
JavaScript
•0•2•0•0•Updated Jun 28, 2024Jun 28, 2024
MaskLID
Public
MaskLID: Code-Switching Language Identification through Iterative Masking -- ACL 2024
language-identification code-mixing code-switching language-identification-toolkit code-switch language-identifier
Python
•
MIT License
•1•3•0•0•Updated Jun 11, 2024Jun 11, 2024
GlotScript
Public
Resource and Tool for Writing System Identification -- LREC 2024
unicode script language-detection iso15924 alphabet unicode-characters writing-systems writing-system glot language-identification
Python
•
MIT License
•2•13•0•0•Updated Jun 7, 2024Jun 7, 2024
Taxi1500
Public
Python
•
Apache License 2.0
•0•7•1•0•Updated May 31, 2024May 31, 2024
TransMI
Public
TransMI: A Framework to Create Strong Baselines from Multilingual Pretrained Language Models for Transliterated Data
transliteration multilinguality crosslingual-transfer tokenizer-modification
Python
•0•4•0•0•Updated May 30, 2024May 30, 2024
TransliCo
Public
TransliCo: A Contrastive Learning Framework to Address the Script Barrier in Multilingual Pretrained Language Models
transliteration multilinguality crosslingual-transfer script-barrier
Python
•0•6•0•0•Updated May 23, 2024May 23, 2024
Spatial_Schemas
Public
JavaScript
•0•1•0•0•Updated May 23, 2024May 23, 2024
XAMPLER
Public
XAMPLER: Learning to Retrieve Cross-Lingual In-Context Examples
Python
•0•2•0•0•Updated May 9, 2024May 9, 2024
Glot500
Public
Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages -- ACL 2023
multilingual nlp natural-language-processing acl dataset glot xlm multilingual-models xlm-r multilingual-nlp
Python
•
Other
•3•97•2•0•Updated Apr 20, 2024Apr 20, 2024
GlotSparse
Public
GlotSparse: Building Corpora in Under-Resourced Languages
multilingual dataset corpus-linguistics glot low-resource-languages news-dataset awsome-list
Creative Commons Zero v1.0 Universal
•0•0•0•0•Updated Apr 18, 2024Apr 18, 2024
GlotStoryBook
Public
Children StoryBooks for 180 langauges.
multilingual storybook dataset glot low-resource-languages low-resource-nlp
Jupyter Notebook
•
Creative Commons Zero v1.0 Universal
•1•3•1•0•Updated Mar 12, 2024Mar 12, 2024
mPLM-Sim
Public
mPLM-Sim: Better Cross-Lingual Similarity and Transfer in Multilingual Pretrained Language Models
Python
•
Other
•0•10•0•0•Updated Jan 19, 2024Jan 19, 2024
ColexificationNet
Public
Crosslingual Transfer Learning for Low-Resource Languages Based on Multilingual Colexification Graphs
embeddings crosslingual-transfer colexification
Jupyter Notebook
•0•4•0•0•Updated Jan 15, 2024Jan 15, 2024
ofa
Public
A Framework aims to wisely initialize unseen subword embeddings in PLMs for efficient large-scale continued pretraining
Python
•2•12•0•0•Updated Nov 26, 2023Nov 26, 2023
simalign
Public
Obtain Word Alignments using Pretrained Language Models (e.g., mBERT)
Python
•
MIT License
•48•352•5•0•Updated Nov 7, 2023Nov 7, 2023
parcoure
Public
ParCourE - Parallel Corpus Explorer
Python
•
MIT License
•0•12•0•0•Updated Dec 27, 2021Dec 27, 2021
graph-align
Public
code for EMNLP graph align paper
Python
•0•9•0•0•Updated Nov 5, 2021Nov 5, 2021
bias-in-nlp
Public
Literature overview: gender bias in natural language processing
nlp bias nlp-machine-learning gender-bias
Python
•0•10•0•0•Updated Jan 26, 2021Jan 26, 2021