Real-time object detection in React Native using TensorFlow
Object detection, machine learning, and deep learning all sound quite intimidating. Object detection is one of the computer vision techniques that is used to identify and locate objects in images, videos, and live streams as well.
Problem Statement
I was recently working on a React Native app in which I was required to perform Real-Time object detection using TensorFlow in React Native. In this article, I will shed light on my journey of achieving this by overcoming all the hurdles along the path. I used Tensorflow.js to get it done.
The major problem I was facing to get Real-time object detection was the camerawithTensors provided by TensorFlow, there was a lag and the recorded video which was too laggy, after research I found a solution to record a video first by using expo camera and then extract images from the video and send them to model for detection.
Main Task
Record a video then get Real-time object detection using TensorFlow in React Native in a mobile application.
Tensorflow.js
TensorFlow.js is a library for machine learning in JavaScript. TensorFlow.js lets you develop or execute ML models in JavaScript, and use ML directly in the browser client side, server side via Node.js, mobile native via React Native, desktop native via Electron, and even on IoT devices via Node.js on Raspberry Pi.
React Native
React Native is an open-source UI software framework. React Native combines the best parts of native development with React, a best-in-class JavaScript library for building user interfaces. It is used to develop applications for Android, Android TV, iOS, macOS, tvOS, Web, Windows, and UWP by enabling developers to use the React framework along with native platform capabilities
Major packages used
- React Native FFmpeg
- Expo Camera
- File System
- Tensorflow/tfjs-react-native
- Tensorflow/tfjs
Solution Steps
- Extract images from the video
- Convert images to tensor
- Send tensor to model for detection
-
1 . Extract Images From Video
I used the react native ffmpeg package which helps to extract images from videos. As the video is recorded using expo camera by the user it goes to the function which extracts all images using this command
I used the react native ffmpeg package which helps to extract images from videos. As the video is recorded using expo camera by the user it goes to the function which extracts all images using this command
Where,
1- videoURL is the video source URL
2- To output 25 images every second, named out1.png, out2.png, out3.png, etc, we use fps=25.
3- The %01d dictates that the ordinal number of each output image will be formatted using 1 digits.
4- saveFilePath is the path at which you want your pictures to be saved -
2 . Convert Images To Tensor
After extraction these images will be converted to tensor and pass for object detection to our custom TensorFlow tfjs model, Filesystem and tfjs were used for conversion
Where,
1- img is the path of the image
2- imagesTensor is the tensor -
3 . Send Tensors To Model
These tensors will be passed to the model for detection, as I was using our custom TensorFlow tfjs model for object detection, the tensor needs to be resized, and normalized in the particular size to pass to the model as an input
Conclusion
Real-time object detection relies on robust, well-annotated training data to ensure accuracy and speed. Labeling thousands of frames by hand is slow and error-prone, which is why we built Minmini, an automated image annotation tool that speeds up dataset preparation for custom detection models.
That’s a Wrap!
I’ve covered every step to get real-time object detection running with TensorFlow in a React Native app, using videos recorded by the user. Moving from a working prototype to a production app that handles lighting changes, device limits, and custom object classes is where computer vision services come in, from model training and optimization to mobile deployment.
Taking Object Detection to Production?
A demo that works on your phone is step one. We help teams train custom detection models, optimize them for mobile and edge devices, and ship them in production apps.
✅ You own 100% of your code.
✅ 100% confidential.
✅ NDA available before discussions.
