Linting dictionary entries

[aecepoglu]

We would benefit from introducing a way to automatically measure "how good" a Sozluk entry is.

You can imagine this as a linter for turkish text.

An example list of criteria:

• sentence length. Long sentences are more difficult to understand
• weasel words. Weasel words don't belong in a dictionary (this is my personal opinion). Example: "Weasel words don't quite belong in a dictionary" vs "Weasel words don't belong in a dictionary"
• spell checking
• proper capitalisation
• redundant abbreviations. Orn: OSS Sinavi

Questions:

• What other things could we measure?
• How would we use such a tool? At the moment we are opening PRs to create new entries so we would want to integrate it into our CI. Later we may have a web UI to create entries though.

Lets get some more opinions in here and I think @arabacibahadir can comment on which of these are solveable and how easily.

[arabacibahadir]

I think we can automate these with HuggingFace hooks, but I've never used them before. I only know that they work like GitHub hooks.

Redundant abbreviations require different datasets, like the "ÖSS Sınavı". We would need to generate them ourselves for this. I think at this point we can lower the priority of this because it would require a lot of effort.

What we can do at first stage:

• Sentence length and the presence of complex sentences that are hard to understand
• Automatic capitalization
• Weasel words
• Spell checking

[aecepoglu]

let us

1. get a confirmation that there is scope for a feature like this
2. consider how/if to integrate this feature with other features such as tag-suggestion
3. figure out how we're going to integrate this into the rest of the system. As a CLI-tool to run during CI? As a microservice?
4. focus on the easier wins first: spell-checking, sentence length, proper capitalisation.

[aecepoglu]

From @usirin 's stream today, the answers were:

1. get a confirmation that there is scope for a feature like this

yes

2. consider how/if to integrate this feature with other features such as tag-suggestion

not discussed. Lets ignore for now

3. figure out how we're going to integrate this into the rest of the system. As a CLI-tool to run during CI? As a microservice?

Yes and yes.

4. focus on the easier wins first: spell-checking, sentence length, proper capitalisation.

ok.

---

With those in mind, we should be good to start scoping out and planning the task