[RFC] Support instantiation linking

[lum1n0us]

Instantiation linking is a necessary method for processing imports (such as functions, globals, tables, and memory) of a Wasm module with given external values. Please refer to Spec. for more details.

	loading linking(MULTI_MODULE)	instantiation linking
when to process	during loading	during instantiation
how to give what is needed	by module(by runtime)	by module,by host,(by runtime)
which objects are allowed	function,global	function,global,memory,table

It is different from the current method of loading linking, which is made possible by WAMR_BUILD_MULTI_MODULE.

Loading linking copies the way ld-linux works. It looks for a needed Wasm file in specified paths using a module name, and it looks for needed external values within the Wasm file using a field name. It then connects multiple Wasm files at the module level. As a result, every instance of one module will be linked in the exact same way. Later on, a Wasm module and its related modules will all be instantiated by the Runtime. The Runtime will select the needed items from the dependent module instances and then supply them as imports, sorted by type.

Instantiation linking, on the other hand, requires users to provide a list of external values. This list should match the types of imports a Wasm module needs. The advantage of instantiation linking is that it does not require actual Wasm modules like loading linking does. Users can gather the list of external values from Wasm modules, from instances(such as function instances, global instances, memory instances and table instances) created by the host, and from dummy Wasm instances.

Instantiation linking is more adaptable than loading linking. Because of this, it can be included within loading linking. In some ways, we could say that loading linking = dependencies loader + dependencies instantiation + instantiation linking. As a result, loading linking can be reworked using instantiation linking APIs.

necessary APIs

We must ensure that all the following functions are fully supported and align with the Spec. definitions in both wasm_c_api.h and wasm_export.h, whether through new or existing APIs.

• Identify the types of imports and exports of a Wasm module.
• Retrieve the export values from a Wasm instance.
• Use the given external values when instantiating a Wasm module.
• Create instances of functions, globals, tables, memories and/or tags from the host.
• Grow memory and table from host

To be continued...

may affected

default linking policy

Clearly, loading linking can at least make use of the new instantiation linking APIs. However, we will keep it as is and separate it within the WAMR_BUILD_MULTI_MODULE controlled blocks. The tasks to come include reworking loading linking and creating a new linking layer.

Since instantiation linking is defined by Spec., it should always be an active feature by default. We must acknowledge that users may supply their own versions for some of the Wasm module import requirements and rely on the Runtime's built-in versions for the rest.

Due to the above line, the original WAMR_BUILD_MULTI_MODULE will need to have its scope expanded to encompass all content related to loading linking. It may also be necessary to rename it to WAMR_BUILD_LOADING_LINKING.

an import list

The Runtime should connect built-in functions (registered native functions) and built-in globals (when WASM_ENABLE_LIBC_BUILTIN is on) with imports in the instantiation stage. If we strictly follow the Spec., appropriate registered native functions should be part of the list of external values given when a Wasm module is initiated. However, this could result in a very long list that may not be user-friendly for API users in certain situations, such as when WASM_ENABLE_LIBC_WASI is on. Therefore, we might not insist that the list of external values completely matches the content of a Wasm module's import section. For any missing parts, the Runtime could provide the registered native functions. Another approach is to offer APIs that assist the host in creating a complete list of imports, keeping the instantiation process as straightforward as possible. The new APIs would generate the required imports using the registered native functions and/or host functions provided by the user through arguments.

host creation

It requires to create WASMFunctionInstance, WASMGlobalInstance, WASMMemoryInstance, and WASMTableInstance directly instead of instantiating from related Wasm module section content.

The Runtime must be able to process and manage creations made by users. Typically, this means that all necessary fields for execution should be populated with appropriate values.

Reusing the current xxx_instantiate() in wasm_runtime.c to create a WASMXXXInstance appears to be feasible. Be sure that an XXXModuleInstance is not a required argument.

instances of a spawned thread

Both LIB_WASI_THREADS and THREAD_MGR share a memory model. Every spawned thread reuses its parent's WASMMemoryInstance(?WASMFunctionInstance, WASMGlobalInstance, WASMTableInstance). We can inherit WASMMemoryInstance from parents

AOT supports import memories

WAMR used to say it doesn't support import_memory_count > 0, now we are going to support it. The new aot loader can still handle those AOT files without any import memories.

AOT supports import tables

Typically, WAMR uses a separate area within a WASMTableInstance/AOTTableInstance to store elements for both imported and local tables. If it's an imported table, only the initialized value will be shared. The MULTI_MODULE feature does not extend to AOT.

Conversely, the specification mandates that elements be shared between an imported table and its source throughout their entire lifetime, not just the initialized values but also any subsequent modifications. This would require significant changes in several areas:

• There would be no need to allocate a large space for the elements of an imported table.
• A different method would be needed to access the elements of a table, which should be applicable to the interpreter, LLVM-JIT, fast-JIT, and AOT.
• The potential growth of local tables that will be exported must be taken into account.

AOT compatible

new content	previous status	new aot_loader backwards compatible
import memories	0	no extra work
import tables	w/o names	use package_version

To be continued...

Tasks

Beta Give feedback

1. Identify the types of imports and exports of a Wasm module.
2. Retrieve the export values from a Wasm instance.
3. Use the given external values when instantiating a Wasm module.
4. Create instances of functions, globals, tables, and memories from the host.
5. During the loading phase, do not resolve import items.
6. In the instantiation phase, resolve import items. Utilize wasm_native symbols and built-in libc symbols to resolve import items not provided by the user.
7. reconsider Add no_resolve to LoadArgs and wasm_runtime_resolve_symbols so one can #3790
8. Create an external value list for spectest requirement.
9. Create an external value list for wasi-test requirement
10. Support to grow memory from host
11. Support to grow table from host
12. refactor wasm_instance_new_xxx() in wasm_c_api

[lum1n0us]

• [instantiation linking] create and import WASMMemoryInstance for interp #3845
• [instantiation linking] create and import WASMMemoryInstance for aot  #3887. based on ⬆️
• [instantiation linking] Refactor wasm_runtime_instantiate() #3893. based on ⬆️ ⬆️
• [instantiation linking] Create and import WASMTableInstance #3898