Withdraw text suggesting that MF1 messages sometimes can't be transformed to MF2

[macchiati]

I was looking more at conversion of MF1 to MF2, and realized something.

Take this badly formed pattern:

{num_guests, plural, offset:1 
    =0 {{host} does not give a party.}
    =1 {{host} invites {guest} to her party.}
    =2 {{host} invites {guest} and one other person to {gender_of_host, select, 
        female {her} male {his} other {their}
        } party.}
other {{host} invites {guest} and # other people to {gender_of_host, select, 
        female {her} male {his} other {their}
        } party.}}}

It is badly formed because it presumes that only one part of the message is governed by the gender_of_host. That will fail for many languages. However, we can always pull the embedded select placeholder for gender_of_host to be the topmost, duplicating the messages and replacing the embedded placeholder contents in each case by the 'resolved value'.

For example, look at the following:

{gender_of_host, select, 
  female {
    {num_guests, plural, offset:1 
      =0 {{host} does not give a party.}
      =1 {{host} invites {guest} to her party.}
      =2 {{host} invites {guest} and one other person to her party.}
      other {{host} invites {guest} and # other people to her party.}}}
  male {
    {num_guests, plural, offset:1 
      =0 {{host} does not give a party.}
      =1 {{host} invites {guest} to his party.}
      =2 {{host} invites {guest} and one other person to his party.}
      other {{host} invites {guest} and # other people to his party.}}}
  other {
    {num_guests, plural, offset:1 
      =0 {{host} does not give a party.}
      =1 {{host} invites {guest} to their party.}
      =2 {{host} invites {guest} and one other person to their party.}
      other {{host} invites {guest} and # other people to their party.}}}}

And importantly, when we dot same thing to the translated messages, they also produce the same result. (Of course, they could be retranslated to take advantage of additional freedom, but they are no worse than they were.)

This was the only real roadblock for compatibly transforming MF1 messages into MF2, so I think we can change some of our language talking about the migration.

BTW, the second example is from https://unicode-org.github.io/icu/userguide/format_parse/messages/

[aphillips]

Any message might be transformed in the way that you suggest, with a machine naively producing it. I believe @eemeli has a working one.

However, there are MF1 messages that cannot be transformed into MF2 messages--because we took the functions away. Any message that uses choice, spellout, or duration won't currently work. @mihnita could provide :icu namespaced functions to support these and that would restore compatibility. Also, note that the # shorthand doesn't exist in MF2.

Looking this morning, I don't see in the spec where we say that MF1 messages are not transformable. Can you point to text? Pull requests are better than guessing 😄

[eemeli]

Any message might be transformed in the way that you suggest, with a machine naively producing it. I believe @eemeli has a working one.

You'll find that here: https://github.com/messageformat/messageformat/tree/main/packages/mf2-icu-mf1

However, there are MF1 messages that cannot be transformed into MF2 messages--because we took the functions away. Any message that uses choice, spellout, or duration won't currently work. @mihnita could provide :icu namespaced functions to support these and that would restore compatibility. Also, note that the # shorthand doesn't exist in MF2.

Actually, there's nothing in the MF2 spec preventing choiceformat from being representable.

I think it's important to note that while the MF2 default functions do not provide sufficient functionality for representing any and all MF1 messages, the spec does absolutely support the definition of a set of functions providing such functionality, including choiceformat selection.

It sounds like the ICU could (should?) make a determination how well the ICU implementations will be backwards-compatible with MF1, separately from the MF2 spec.

[aphillips]

It sounds like the ICU could (should?) make a determination how well the ICU implementations will be backwards-compatible with MF1, separately from the MF2 spec.

One other callout: JDK java.text.MessageFormat is fully represented by the default registry.

[mihnita]

I have code transforming "concatenated selectors" (allowed but not recommended in MF1) to "top level selectors" (the only ones available in MF2).
It is currently part of Okapi.

It does the exact transformation Mark describes in the first comment / issue description.

It "normalizes" MF1 messages (producing the nested selection), but the algorithm should be easy to adapt for MF2.

This normalization fails for MF1 messages with multiple plurals offsets because there can only be one # placeholder, representing the innermost plural variable.
Not a problem for MF2 (no # mechanism), and I can't imagine a use case anyway.

---

My intention was to include things like choiceformat as custom selector in some unit tests (if at all).
And only as a proof that it can be done.

Because having it as an official (but namespaced) registration would require it to be closer to production quality, rather than the hand-wavy "this is how you do it".
And because it is not i18n / translator friendly.

[macchiati]

You're right. I searched, and there isn't any specific mention of that in the specs or the site material (what you can get to from https://github.com/unicode-org/message-format-wg). So I'll close this issue as a user-error.

I think a few steps have to be in play to make migration easy, which will be important to build momentum. I'll talk to the ICU TC about this.

• support all the functions and their operands needed to transform MF1 message patterns into MF2, including translated messages. Such transforms should support the full functionality of MF1, but can be defined in ICU. (choice is already deprecated, so it could be debated whether or not to support it.)
• provide code that does that transformation
• provide code that transforms an MF2 message in a source locale into another MF2 message in another, to provide a basis for translation (eg for plural category selection, expands or contracts the variants to account for the difference in plural categories between the source locale and a target locale.)
• consider structuring a wrapper providing parameter descriptions, sample values, which option values can be modified by translators, etc.

[macchiati]

Will delete this week.

[aphillips]

provide code that transforms an MF2 message in a source locale into another MF2 message in another

You're thinking here of selector matrices and specifically about the "explosion" (or collapse, in some cases) of selectors such as plural and ordinal. Note that the default registry text is written with this sort of thing in mind. It's also probably the case that thinking of this as "implementing the data model" and "implementing a selector API/method for obtaining per-locale key sets" would let one write a short program that does "locale X => locale Y" transforms.

consider structuring a wrapper providing parameter descriptions, sample values, which option values can be modified by translators, etc.

This is a part of the specification--one that we elected not to complete: the "function registry". Completing this work would be useful in enabling custom function plug-in as well as interfacing with translation tooling.

I note that @mihnita says that MF2 code in ICU is standalone (not code-dependent on the rest of ICU). This has a certain appeal. Having portable message tooling, registry tools, and data model tools might help kickstart adoption.

[mihnita]

I note that @mihnita says that MF2 code in ICU is standalone (not code-dependent on the rest of ICU)

The parsing and the formatting logic is 100% ICU independent.
I avoided using anything ICU, even when it would have been convenient (for example UnicodeSet)

The implementation of the registry functions depends on ICU, but they are registered at runtime.
The MF2 formatting invokes the registered functions, but does not care how they are implemented.
One should be able to write such functions on top of Android APIs, or something else.