Depth camera enabled with Qualcomm's Snapdragon 820

Depth cameras are critical to the world of robotics and now come in different shapes and frames. To find objects or paths in the real world, robots need to sense their surroundings. Most of these cameras use two cameras with some distance and perform a lot of processing to create a 3D world from these two 2D video captures. In most cases, the raw captured video must be passed to other modules for different purposes - one of the missing parts of most existing depth camera modules. In this article, we will introduce Qualcomm's Snapdragon 820 as a solution for real-time depth cameras without losing the essential functionality of a dual-camera system.

Qualcomm's Snapdragon 820 is one of the more powerful SoC in its family. The SoC has an octa-core ARM CPU as the application processor that works with several subsystems such as GPU, DSP, RPM, etc. The Hexagon DSP is a powerful DSP with multiple hardware threads, packet instructions, L1 and L2 caches, access to peripherals, and double data rates (DDR), such as application processors Qualcomm (APQ) and Very Long Instruction Word (VLIW). In the 820 version, 6 of this architecture is used, and in this new architecture, a Hexagon Vector eXtension (HVX) coprocessor is provided to handle 1024-bit vectors.

One feature of the Snapdragon 820 is the ability to add custom modules to the image signal processing (ISP) pipeline.

A custom module in the ISP pipeline will allow the user to process the image before passing it on to other modules in the ISP. The input to this module is in Bayer and non-Bayer formats, and the output is in Bayer format. The module has direct access to the DDR, and the processing results are available to the High-Level Operating System (HLOS) in real-time.

The Snapdragon 820 has two ISPs, and both pipelines can contain custom image processing modules. In other words, if you connect two cameras to the 820, you can customize the image processing for each camera separately. If users want to use the HVX module for both custom image processing modules, they can use 512-bit vectors. Using a custom image processing module will put Hexagon in a specific model, assigning two specific hardware threads to each custom image processing module.

Processing Architecture

We used two cameras with the same resolution, arranged in a fixed fixture for our purposes. This means that we know the degree of overlap between the two cameras. 

Using the neural network solution, we can simultaneously use the HVX module to perform the edge detection process on both camera images. The edge detection image on one camera is passed to the other camera using a feedback pointer. The two images are then processed using the SAD algorithm. The results are used in DDR for sharing with HLOS. As mentioned earlier, the camera position and alignment are fixed and can be used to find the overlap of the two captured images. If this is not the case, we need to add overlap detection modules throughout the process.

Experiment

We can see that the frames have a fixed overlap by fixing two cameras in a still frame.

Using the SAD algorithm on the two related frames and converting the numbers to grayscale by interpolation.

Using this algorithm, we can achieve full HD video at 30 FPS while the video is available to HLOS without interruption.