Object detection, machine learning, and deep learning all sound quite intimidating. Object detection is one of the computer vision techniques that is used to identify and locate objects in images, videos, and live streams as well.
I was recently working on a React Native project in which I was required to perform Real-Time object detection using TensorFlow in React Native. In this article, I will shed light on my journey of achieving this by overcoming all the hurdles along the path. I used Tensorflow.js to get it done.
The major problem I was facing to get Real-time object detection was the camerawithTensors provided by TensorFlow, there was a lag and the recorded video which was too laggy, after research I found a solution to record a video first by using expo camera and then extract images from the video and send them to model for detection.
Record a video then get Real-time object detection using TensorFlow in React Native in a mobile application.
Major packages used
- Extract images from the video
- Convert images to tensor
- Send tensor to model for detection
1 . Extract Images From Video
I used the react native ffmpeg package which helps to extract images from videos. As the video is recorded using expo camera by the user it goes to the function which extracts all images using this command
- videoURL is the video source URL
- To output 25 images every second, named out1.png, out2.png, out3.png, etc, we use fps=25.
- The %01d dictates that the ordinal number of each output image will be formatted using 1 digits.
- saveFilePath is the path at which you want your pictures to be saved
2 . Convert Images To Tensor
After extraction these images will be converted to tensor and pass for object detection to our custom TensorFlow tfjs model, Filesystem and tfjs were used for conversion
- img is the path of the image
- imagesTensor is the tensor
3 . Send Tensors To Model
These tensors will be passed to the model for detection, as I was using our custom TensorFlow tfjs model for object detection, the tensor needs to be resized, and normalized in the particular size to pass to the model as an input
That’s a Wrap! I have provided all the steps of how to get real-time object detection using TensorFlow in react native application on the videos recorded by the user in real-time.