My thesis topic is “Detection of small objects using Multi-baseline Stereo Vision and Online Sensor Fusion, Image Stitching and 3D Reconstruction on a UAV”. This has been a highly exciting project for me. In this I did or am doing the following:
- camera and lens calculations and selection,
- built a stereo rig using 3 cameras,
- wrote a C++ program using Linux USB Video Class (UVC) driver library and OpenCV light and smart enough to capture & save images at upto 54fps x 3 cameras = 162fps,
- added hardware trigger to get synchronized pictures for stereo vision,
- linked program with MAVLink messages to log GPS and IMU messages,
- trigger cameras using Tegra TX2 GPIO pins when GPS message gets logged to get pictures synchronized with GPS tags,
- image stitching using feature matching,
- working to build a Kalman Filter for Sensor Fusion to combine pose estimates from GPS/IMU and Feature Matching and reduce uncertainty for image pose correction,
- working on building uncertainty models for GPS, IMU and feature matching to be used in the Kalman Filter,
- transform images to ground plane and make a 2D world map
- work on camera and stereo calibration which is not as straight forward with high focal length lenses,
- generate depth maps using Semi Global Matching (SGM) online on UAV making use of CUDA enabled NVIDIA GPU,
- transform depth images using above calculated transforms, generate a 2D depth map of world on ground plane,
- convert the generated depth map to 3D by converting it into point cloud.
I have faced enough hardware and software issues and am still facing them. But I am hopeful of completing my project well and extract research outputs from it. It has been very exciting till now and its getting better.
Stereo-Rig mounted to the UAV with Tegra TX2 on-board computer:
Data collection experiments:
Real-time view of program capturing synchronous images from 3 cameras:
Program to rectify/undistort images and generate disparity maps using SGM algorithm:
Will update more later.
The Cool Robotics Blog 