CLDR-15391 Update Canadian census and Fix Cans Script Match #4208
Conversation
Important: CLDR treats macrolanguage codes as regular languages — identifying them with their most common encompassed language. For example, zh is interpreted as identical to 'cmn' (and preferred — cmn aliases to zh). So it should be expected that cr (=cwd) appears along with other languages that ISO considers encompassed by cr (eg crj). We test in other cases, and should test here, that we don't have any aliased language codes. |
| Canada CA "36,328,480" 99% "1,774,000,000,000" Mi'kmaq mic 0.0254% https://www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=9810021601 Canada 2021 Census language "Knowledge of Language"; official status from Wikipedia Languages_of_Canada | ||
| Canada CA "36,328,480" 99% "1,774,000,000,000" recognized Atikamekw atj 0.0187% https://www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=9810021601 Canada 2021 Census language "Knowledge of Language"; official status from Wikipedia Languages_of_Canada | ||
| Canada CA "36,328,480" 99% "1,774,000,000,000" Siksika bla 0.0183% https://www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=9810021601 Canada 2021 Census language "Knowledge of Language"; official status from Wikipedia Languages_of_Canada | ||
| Canada CA "36,328,480" 99% "1,774,000,000,000" Woods Cree cwd 0.0140% https://www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=9810021601 Canada 2021 Census language "Knowledge of Language"; official status from Wikipedia Languages_of_Canada |
There was a problem hiding this comment.
The unit tests should prevent aliased language codes (like cwd). We need to fix that, but in the meantime you should use cr instead of cwd.
There was a problem hiding this comment.
Indeed, this change needs work.
I'd like to understand the macrolanguage better. As I've learned, Woods Cree cwd is being aliased to Cree cr. However, Woods Cree (5,110 speakers according to CA 2021 Census) isn't even the biggest Cree dialect -- Plains Creek crk is (12,005 speakers) -- however the vast majority of Cree speakers aren't grouped (there are 87,875 Cree speakers in total. 61,000 are not matched to a constituent dialect). So perhaps Woods Cree is actually the biggest.
So back to aliasing -- personally I'm not a big fan of that because they are grouping oranges with apples -- yes they are all fruits but they are different groups. Do you have any design docs about aliasing so I can understand the background?
For another example of aliasing losing data precision, when I worked on CLDR-10478, the Macao census has different estimates for Traditional Chinese (written, any spoken dialect zh_Hant, at 98%), Simplified Chinese (written, any spoken dialect zh_Hans, at 5%) as well as Spoken Cantonese yue @ 86.2% and Spoken Mandarin cmn @ ~40%. I ended up just ignoring the Mandarin category because was being aliased back to zh.
Nonetheless, I have faith this has been thought-through a lot already, so maybe I just need to catch up to the design choices.
|
@srl295 ah this was a larger can of worms than I anticipated. So it looks like currently The original reported bug is that "und_Cans" is matching to "iu_Cans" not "cr_Cans" even though the Cree community is bigger than the Inuktitut community. However I need to figure out the macrolanguage matching to see hwo to move forward. Happy to punt this and get back to this once we are all back in late November. |
|
The handling of the 'macrolanguage' concept was introduced in BCP47 for backwards compatibility, but causes its own — more severe — compatibility problems. If 'zh' truly means 'any Chinese language', then it would be perfectly fine for an implementation to request 'zh' and for us to serve up 'yue' content in LDML. So we have a longstanding policy for macro/encompassed languages that I outlined. See also https://cldr.unicode.org/index/cldr-spec/picking-the-right-language-code. There are times where we will adjust the aliasing, where there is a strong shift one way or another. But for small changes we favor stability. The key is, just treat 'cr' exactly as if it were 'cwd', and don't use 'cwd'. |
|
Ooof that makes sense and I don't want this ticket to generate any more work than it already has so I am inclined to punt on the macrolanguage conversation. To return to the narrow origin of the ticket of 1) making |
conradarcturus
force-pushed
the
CLDR-15391-Fix-Cans-Default
branch
from
71db1c0
to
1f00d07
Compare
|
Notice: the branch changed across the force-push!
~ Your Friendly Jira-GitHub PR Checker Bot |
| <language type="crg" scripts="Latn"/> | ||
| <language type="crh" scripts="Cyrl"/> | ||
| <language type="crj" scripts="Cans"/> | ||
| <language type="crj" scripts="Latn" alt="secondary"/> | ||
| <language type="crk" scripts="Cans"/> | ||
| <language type="crk" territories="CA" alt="secondary"/> |
There was a problem hiding this comment.
This change happened because Woods Cree is not an official language of a region of Canada.
Rather, Plains Cree is. Cree is only official in the Northern Territories (NT). Unfortunately, the NT law does not specify which Cree variation. We can deduce the variation because the only Cree language present in NT is Plains Cree [crk] so I infer that is the correct match.
The main purpose of this change is to update the default language for Canadian Aborginal syllabics [Cans] from Inukitut [iu] to Cree [cr] since Cree has a larger population. Understandably, both of these languages are macrolanguages with many variations -- so its funny to include both the Cree macrolanguage along with its constituents. But I think it's better to cover both groupings because depending on the consumer they may want [cr] data or constituent data. While I was doing this I updated all of the Canadian locale data to the 2021 Census. I also added a few missing aborginal Canadian languages: Woods Cree [cwd] and Western Ojibway [ojw]. See the 2021 Census table here: https://www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=9810021601
I'll note that Cree is only official in the Northern Territories (NT). However [the NT law](https://web.archive.org/web/20090324202430/http://www.justice.gov.nt.ca/PDF/ACTS/Official_Languages.pdf) does not specify which Cree variation -- the only Cree language present in NT is Plains Cree [crk] so I infer that is the correct match.
Removed hardcoded _CA entries because they aren't necessary since the likely subtags can be derived from the population data. Also edited the cr_Cans_CA comment because it was overflowing the line and to add more context.
conradarcturus
force-pushed
the
CLDR-15391-Fix-Cans-Default
branch
from
1f00d07
to
d00b78d
Compare
|
Notice: the branch changed across the force-push!
~ Your Friendly Jira-GitHub PR Checker Bot |
The original purpose of this change is to update the default language for Canadian Aboriginal syllabics [Cans] from Inuktitut [iu] to Cree [cr] since Cree has a larger population. Understandably, both of these languages are macrolanguages with many variations -- so its funny to include both the Cree macro-language along with its constituents. But I think it's better to cover both groupings because depending on the consumer they may want [cr] data or constituent data.
While I was doing this I updated all of the Canadian locale data to the 2021 Census. I also added a few missing aboriginal Canadian languages: Woods Cree [cwd] and Western Ojibway [ojw].
See the 2021 Census table here: https://www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=9810021601
CLDR-15391
ALLOW_MANY_COMMITS=true