Without callbacks we can possibly make it work by having an introspecting adapter like you added in #2766

Hello, can I take this one? My WASM demo is now running

I think short-term solutions has already been implemented by @sffc and implementing the medium term solution remains

Yes! This should be a good starter project now that you have the wasm running in your development environment. Happy to review an implementation plan or answer any questions.

Here is my findings so far, feel free to point anything out:

1. As we are currently using ICU4XDataProvider.create_compiled() which is not an efficient approach. We could have the following directory specification, if we want to use fixed_decimal with bn locale:

   |-generated
     |-fixed-decimal
       |-bn.json 
     |-segmenter
     |-date-time
   

   Now to download the locale, we could fetch to get locale data from something like:

   https://raw.githubusercontent.com/unicode-org/icu4x/main/provider/datagen/tests/data/json/decimal/symbols%401/bn.json
   

2. We need to keep track whether root data is being used or not. It should use an FFI hook to call a JavaScript function if root data is being used and display the message in our frontend. (I am slightly unsure about this one, apologies)

3. This one should be comparitively easy as we need to re-render

Thoughts and question about second point:

1. I am unsure about how js be informed, as diplomat is used to call into Rust libraries that too in a unidirectional fashion.
2. I am still learning about diplomat so will update this comment if I could find something.

It should use an FFI hook to call a JavaScript function if root data is being used and display the message in our frontend

We don't have callback support for Diplomat and are unlikely to have it soon. The introspection should be done on the JS side using the APIs from icu_capi, we can add more if needed.

We could create an adapter which wraps our DataProvider and expose an API which returns a boolean. The design I have in my mind is something like:

pub struct DynamicLocaleAdapter<P> {
    inner: P,
}

impl<P> DynamicLocaleLoaderAdapter<P> {
    pub fn new(provider: P) -> Self {
        Self { inner: provider }
    }
}

impl<P, M> DataProvider<M> for DynamicLocaleLoaderAdapter<P>
where
    P: DataProvider<M>,
    M: KeyedDataMarker,
{
    fn load(&self, req: DataRequest) -> Result<DataResponse<M>, DataError> {
        let response = self.inner.load(req);

        if let Ok(ref data_response) = response {
            if data_response.metadata.locale.is_langid_und() {
               // Do something here
            }
        }
        response
    }
}

This needs to be written in our ffi/capi/src/provider.rs. As our ultimate goal is to use in our frontend.

Thoughts regarding this?

This could work, but another approach that might be simpler would be

1. In JS, make an array or Set of the locales that have been loaded; the set starts empty
2. When a new locale is requested in the GUI, we check if data is loaded for it. If yes, we are done. If not, continue.
3. Try to load the locale bundle file from the server. If you get a 404, use the LocaleFallbacker to request the parent locale. Continue doing this until you either find the locale bundle or reach root.
4. When you finally get the bundle, add it to the data provider as a fork-by-locale provider. I think there is an FFI for this but if not we can add one.

Also, the bundle should be in postcard format, not JSON, and I think for the purposes of this demo it can contain all keys so we need to load it only one for all components.

Thanks @sffc for suggesting a much easier approach. Though I have couple of points:

1. load the locale bundle file from the server

   I am unsure about the URL to load the files from. As for the URL I could only find URL for chakma locale:

    https://storage.googleapis.com/static-493776/icu4x_2023-11-03/ccp.blob
   

   Or do I have to use icu4x-datagen tools to generate it?

2. I think there is an FFI

   Yes! We do have fork_by_locale in our FFI declarations

Yes, we have the ccp.postcard sample file. Part of this project is generating similar files for all other locales. They can likely be hosted in the same place as the WASM code.

So basically I have to expose the DatagenDriver over the FFI declaration to be able to use in JavaScript (generating them on demand). I guess currently it is not available.

The data files should be generated during the build. You shouldn't need to run Datagen in WASM.

I could do the following in package.json

  "scripts": {
    "clean": "rm -rf dist/*",
    "prebuild": "npm run generate-locale-data",
    "build": "webpack",
    "start": "webpack serve --mode development --port 12349",
    "tsc": "tsc -p",
    "generate-locale-data": "icu4x-datagen --keys all --locales en --format blob2 --out dist/my_data_blob.postcard"
  }

Now running npm run build would generate the files.

PS: Thank you for being so patient and thoughtful 🚀

@sffc Performance issue may occur just in one case:

If the user doesn't have icu_datagen installing it would take quite sometime.

Generating the files via an npm build seems like the right approach. There should not be performance issues for end users because datagen is run only when generating the WASM demo. Only ICU4X developers (and CI) should need to install icu4x-datagen.

Almost done, just a slight question.

It's being correctly formatted, but any reason why am I getting this in the console?

Also I have implemented a fallback mechanism which logs to where did it fallback. Should I submit a PR if its difficult to point to out?

This is how fallback is being printed

The LocaleFallbacker API should silence the errors and run the fallback algorithm; are you using it? I think in FFI there is also enable_locale_fallback() which you should use.

Yes, I am already using enProvider.enable_locale_fallback_with(this.fallbacker);. But I wanted to add where it fallback. Like using de-AT-u-co-phonebk should fallback to de which is working fine.

de-AT-u-co-phonebk should reach de-AT and then de with fallback.

I think the issue is with async await of javascript, here is the outline of my code:

private async init() {

        let enProvider = await this.loadLocale('und');
        let unProvider = await this.loadLocale('und');

        let fallbacker = ICU4XLocaleFallbacker.create(unProvider);
        let fallbackConfig = new ICU4XLocaleFallbackConfig();
        fallbackConfig.priority = ICU4XLocaleFallbackPriority.Language;
        let fallbackerWithConfig = fallbacker.for_config(fallbackConfig);

        this.fallbacker = fallbackerWithConfig;
        enProvider.enable_locale_fallback_with(this.fallbacker);
        this.dataProvider = enProvider;
}

public async trackLocaleFallback(fallbacker: ICU4XLocaleFallbacker, locale: ICU4XLocale): Promise < string > {
        let localeSet = new Set(localeDefault.default);
        let fallbackIterator = fallbacker.fallback_for_locale(locale);
        while(fallbackIterator.get().to_string() != 'und') {
            const fallbackLocale = fallbackIterator.get().to_string();
            if(localeSet.has(fallbackLocale)) {
                return fallbackLocale;
            }
            fallbackIterator.step();
        }
        return 'un';
}
    
    
public async loadLocale(newLocale: string): Promise < ICU4XDataProvider > {
        const icu4xLocale = ICU4XLocale.create_from_string(newLocale);
        // Gets the fallback locale from the iterator
        const fallbackLocale = await this.trackLocaleFallback(this.fallbacker, icu4xLocale);
        const newFilePath = `dist/${fallbackLocale}.postcard`;
        let newProvider = await this.createDataProviderFromBlob(newFilePath);
        await newProvider.fork_by_locale(this.dataProvider);
        this.dataProvider = newProvider;
        this.loadedLocales.add(newLocale);
        this.fallbackLocale = ICU4XLocale.create_from_string(fallbackLocale);
        return newProvider;
 }

Is it the right direction?

Also in the above image it fallbacks from de-AT-u-co-phonebk to de-AT to de.

Personally I feel the error is because it takes a while to load the file, when we get the file we update the provider and eventually update the formatter to use the new formatter.

I think the issue is with async await of javascript

@sffc Fixed, finally!!!!