[FIXED] How can I read a trained Tensorflow model file (.json) into Android app built with Capacitor (Vanilla JavaScript)?

Issue

EDITED

I've built a cross-platform app with Capacitor, written in Vanilla JS. It uses the following code to load a trained Tensorflow model, a simple classifier NN built in Python.

import * as tf from '@tensorflow/tfjs';
....
const model = await tf.loadLayersModel('./assets/models/model.json');

This works on the browser and I can use the model to make predictions. The model fails to load when deployed on Android.

In the Android Studio logcat I see the following error:

E  Unable to open asset URL: http://localhost/assets/models/model.json

I also added these in the AndroidManifest.xml:

<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE"/>
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />

I know these are in the right location since there are other working permissions in the same place.

So far I've looked into:

• using the @capacitor/filesystem package but I'm not sure how it can be used to replicate the model-loading behaviour of loadLayersModel.
• using file://, file:///, file:///andriod_asset, content://, and content:// in front of the path.
• using Capacitor.convertFileSrc('./assets/models/model.json')
• using different versions of the path (e.g. src/assets/..,)
• using import {fetch as fetchPolyfill} from 'whatwg-fetch'; and window.fetch = fetchPolyfill;

..but I keep getting the error above, Unable to open asset URL. When using fetchPolyfill and file:///andriod_asset/... I get Msg: Not allowed to load local resource: file:///android_asset

I checked the Capacitor Doctor but seems fine for Android:

   Capacitor Doctor

Latest Dependencies:

  @capacitor/cli: 5.5.1
  @capacitor/core: 5.5.1
  @capacitor/android: 5.5.1
  @capacitor/ios: 5.5.1

Installed Dependencies:

  @capacitor/cli: 5.4.0
  @capacitor/ios: 5.4.0
  @capacitor/core: 5.4.1
  @capacitor/android: 5.4.0

[success] Android looking great! 👌
[error] Xcode is not installed

Here is an example app that makes the issue reproducible: https://github.com/beetbox-music/cap_nn/tree/master

How can I replicate, in Android, the working behaviour of the app running on the browser?

UPDATE:

Apparently, the issue has to do with the vite.config.ts file which is generated automatically by Capacitor.

As suggested by jcesarmobile:

• I added a folder to my src folder
• named it public
• added the line publicDir: '../public' in the vite.config.ts
• ran npm run build and npx cap sync

sadly this doesn't seem to have fixed the issue yet as I still don't see the model file in the android folders and see the same error.

import { defineConfig } from 'vite';

export default defineConfig({
  root: './src',
  publicDir: '../public',
  build: {
    outDir: '../dist',
    minify: false,
    emptyOutDir: true,
  },
});

I also updated the app linked above to reflect these changes. Let me know if I've interpreted your answer incorrectly.

Solution

As discovered here, the problem was vite not copying the src/assets into dist folder.

Creating a separate public folder, putting the assets folder inside and configuring vite to use that public folder like publicDir: “../public” should fix the issue and then the vanilla code should work.

Answered By - jcesarmobile