Is @tensorflow-models/[email protected] not performed on the react native platform?

[adjh54ir]

Please make sure that this is a bug. As per our
GitHub Policy,
we only address code/doc bugs, performance issues, feature requests and
build/installation issues on GitHub. tag:bug_template

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow.js): No
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): localhost
• Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: Galaxy S21 5G(SM-G991N) or ALL
• TensorFlow.js installed from (npm or script link): yarn 1.22.22
• TensorFlow.js version (use command below): @tensorflow/tfjs(3.11.0)
• Browser version: Android OS 14
• Tensorflow.js Converter Version: 3.11.0

Describe the current behavior

The following error occurred: TypeError: Cannot read property 'includes' of undefined, js engine: hermes.

Describe the expected behavior

hello!

I have been using version @tensorflow-models/[email protected] in the React Native platform environment.
I recently performed an update to version 1.0.6, but confirmed that the update did not work and the following error occurred.
After checking the error, I determined that it was a method called in a web environment rather than a mobile environment, so I was curious to see if it was actually performed.
The error was TypeError: Cannot read property 'includes' of undefined, js engine: hermes and was confirmed to be an error caused by loading the @mediapipe/face_mesh library in the faceLandmarkDetection.SupportedModels.MediaPipeFaceMesh section.

"dependencies": {
    "react": "18.2.0",
    "react-native": "0.72.4",
    "@tensorflow-models/blazeface": "^0.1.0",
    "@tensorflow-models/face-landmarks-detection": "1.0.6",
    "@tensorflow/tfjs": "3.11.0",
    "@tensorflow/tfjs-backend-cpu": "3.11.0",
    "@tensorflow/tfjs-backend-webgl": "3.11.0",
    "@tensorflow/tfjs-converter": "^3.11.0",
    "@tensorflow/tfjs-core": "3.11.0",
    "@tensorflow/tfjs-react-native": "0.8.0",
    "@mediapipe/face_detection": "^0.4.1646425229",
    "@mediapipe/face_mesh": "^0.4.1633559619"
}

Please refer to the source code as below.

import * as faceLandmarkDetection from '@tensorflow-models/face-landmarks-detection';
useEffect(() => {
    loadModel();
}, []);


const loadModel = async () => {
    const model = faceLandmarkDetection.SupportedModels.MediaPipeFaceMesh; // error 
    console.log("model :: ", model)
    const detectorConfig = {
        runtime: 'mediapipe', // or 'tfjs'
        solutionPath: 'base/node_modules/@mediapipe/face_mesh',
    }
    console.log(detectorConfig)
    const detector: faceLandmarkDetection.FaceLandmarksDetector = await faceLandmarkDetection.createDetector(model, detectorConfig);

    console.log(detector)
}

Please reply. Thank you!

Standalone code to reproduce the issue
Provide a reproducible test case that is the bare minimum necessary to generate
the problem. If possible, please share a link to Colab/CodePen/any notebook.

Other info / logs Include any logs or source code that would be helpful to
diagnose the problem. If including tracebacks, please include the full
traceback. Large logs and files should be attached.

[shmishra99]

Hi @adjh54ir ,

I am able to use faceLandmarkDetection.createDetector with the tfjs runtime, but I am getting a TypeError: t.FaceMesh is not a constructor error while using the MediaPipe runtime. I have imported the following dependencies in my App.js:

import '@mediapipe/face_mesh';
import '@tensorflow/tfjs-core';
// Register WebGL backend.
import '@tensorflow/tfjs-backend-webgl';
import * as faceLandmarksDetection from '@tensorflow-models/face-landmarks-detection';   

Thank you!

[adjh54ir]

Hi @shmishra99,

First of all, thank you for answering my question.
I tested it by injecting the following dependencies using the suggestions you suggested.

"dependencies": {
	"react": "18.2.0",
	"react-native": "0.72.4",
	"@mediapipe/face_detection": "^0.4.1646425229",
	"@mediapipe/face_mesh": "^0.4.1633559619",
	"@tensorflow/tfjs": "4.22.0",
	"@tensorflow/tfjs-backend-webgl": "4.22.0",
	"@tensorflow/tfjs-converter": "4.22.0",
	"@tensorflow/tfjs-core": "4.22.0",
	"@tensorflow/tfjs-react-native": "0.8.0",
	"@tensorflow-models/face-detection": "^1.0.3",
	"@tensorflow-models/face-landmarks-detection": "1.0.6",
}

[ The development code is below. ]

import '@mediapipe/face_mesh';
import '@tensorflow/tfjs-core';
// Register WebGL backend.
import '@tensorflow/tfjs-backend-webgl';
import * as faceLandmarksDetection from '@tensorflow-models/face-landmarks-detection';

const StudyPureScreen = ({ route, navigation, appState }: CommonType.CommonProps) => {

useEffect(() => {
        loadModel();
    }, []);

    const loadModel = async () => {
        const model = faceLandmarksDetection.SupportedModels.MediaPipeFaceMesh; // error 
        console.log("model :: ", model)
        const detectorConfig = {
            runtime: 'tfjs', // or 'mediapipe' // <--------------- modify
            solutionPath: 'base/node_modules/@mediapipe/face_mesh',
            refineLandmarks: true
        }
        console.log(detectorConfig)
        const detector: faceLandmarksDetection.FaceLandmarksDetector = await faceLandmarksDetection.createDetector(model, detectorConfig);

        console.log(detector)
    }
}
export default StudyPureScreen();

[ Metro Console Error]

[ react-native logbox Console Error ]

I look forward to your response on this.
thank you!

[scrapecoder]

Yes, faceLandmarks is not working on the react-native side. im also getting the same includes errors.