Add support for Android/iOS (and react-native)

[ospfranco]

I want to add op-sqlite support for sqlite-vec. In order to do this I need to compile sqlite-vec to Android and iOS. Once that is done, bridging the functionality for react-native is 0 cost.

Takes care of #68 and #69.

Feedback is of course welcome.

[ospfranco]

ANDROID_NDK_HOME needs to be defined in case you want to compile libraries for Android.

[ospfranco]

I had to compile with OMIT_SIMD=1 make android as there is a flag added above with mcpu=m1 which is not compatible by the clang compiler android uses.

[ospfranco]

Not sure if this commands are correct, this floating unzip -d was incorrect and I had to comment it so the amalgamation was correctly downloaded.

[ospfranco]

With the current changes I got Android running on a React Native app. Tomorrow I will try to compile for iOS.

[asg017]

Hey @ospfranco , thanks for your work here, would love to get this inside op-sqlite!

Thanks for the insight on android/ios release assets as well. This is something I want to get into the official release process, compiled on Github Actions on every release.

For android, it seems like the best approach would be new sqlite-vec-$VERSION-loadable-x86_64-linux-android.tar.gz tarballs for x86_64/aarch/armv7/i686, each with their own libsqlite_vec.so file. I can work on making some github actions for that.

For iOS, would individual xcframework files for x86_64/aarch64/sim be sufficient? I haven't made these by hand before, but they look straightforward to make.

Does this approach make sense, pubishing these tarballed .so and xcframework files to github releases? Or is there some npm package convention that react native folks use for native libraries?

I think ideally most of this code will move into Github Actions, ie not in the Makefile. Ideally we can keep the make loadable command and overwrite CC/CFLAGS as needed. But I'm definitely referencing this while I try to figure out github actions for this

[ospfranco]

@asg017 yeah, that's exactly what is needed. The github actions instead of the makefile sounds good to me.

For generating the xcframework you only need a couple of commands. You can find them in this Rust guide I created. Again the process is similar, compile the library into .a for the different architectures. Then generate a fat lib for clashing archs and then finally roll them into a single xcframework.

mkdir -p simulator_fat
  lipo -create target/x86_64-apple-ios/release/$(LIB) target/aarch64-apple-ios-sim/release/$(LIB) -output simulator_fat/$(LIB)
  xcodebuild -create-xcframework -library target/aarch64-apple-ios/release/$(LIB) -headers include -library simulator_fat/$(LIB) -headers include -output $@
  cp -r $@ ../ios/$@

I will for now compile iOS and update the makefile so you have all the commands and then close this PR.

[ospfranco]

I got iOS working, but it turned out more complex than I thought. I will provide more detailed instructions later, but basically, compilation is tricky and then not only an .xcframework is needed but inside dynamic .frameworks need to be inside. As well as modifying the rpath with install_name_tool command.

[ospfranco]

Ok, here are more detailed compilation steps for iOS:

• OMIT_SIMD=1 make ios will compile the library for all the necessary iOS archs as a dylib (forget what I said about a static .a, sqlite extensions need to be dylibs)
• It is then necessary to combine x86_64 and arm64 simulators (for m1 and intel machines) into a fat lib. You might get some errors depending on which folder you run the commands, just try it creating different folders but the commands should be the same: lipo -create ./x86_64/sqlitevec ./arm64_simulator/sqlitevec -output ./sim_fat/sqlitevec
• One you have the libraries you need to create a .xcframework with .frameworks inside. You can copy the structure and the plist contents from op-sqlite. Don't forget to change the bundle ID, I just used my own com.ospfranco.sqlitevec because it needs to be referenced when loading the dylib in iOS.
• After you have the .xcframework (hand created is fine, no need to use CLI tools), you do need to use install_name_tool on each of the frameworks to change the rpath of the dylibs to point to themselves: install_name_tool -id @rpath/sqlitevec.framework/sqlitevec sqlitevec.framework/sqlitevec (run it for arm64 and x86_64_arm64_simulator)
• Afterwards you can just publish the artifacts in github releases and it will be a matter of downloading them.

[ospfranco]

I just had a report when submitting to the app store the compiled binary does not correctly match the minOS version on the Info.plist. I might have forgotten setting the min-iOS version when compiling. I've modified the makefile to add the flag:

MIN_IOS_VERSION = 8.0

IOS_ARM64_FLAGS = -target arm64-apple-ios -miphoneos-version-min=$(MIN_IOS_VERSION)
IOS_ARM64_SIM_FLAGS = -target arm64-apple-ios-simulator -miphoneos-version-min=$(MIN_IOS_VERSION)
IOS_X86_64_FLAGS = -target x86_64-apple-ios-simulator -miphoneos-version-min=$(MIN_IOS_VERSION)

@asg017 just FYI when you publish your own artifacts

[ospfranco]

The rpath was not properly set. I've written more extensively what the process is to correctly generate an .xcframework. I saw there are now pure releases of the compiled binary but that's not enough to load sqlite-vec in iOS.

Here is the full tutorial