Method and device for stereo images processing

• US 10,356,385 B2
• Filed: 06/06/2017
• Issued: 07/16/2019
• Est. Priority Date: 06/07/2016
• Status: Active Grant

• Current Assignee: Stock Company Research and Development Center "Electronic Information Computation Systems"
• Original Assignee: Stock Company Research and Development Center "Electronic Information Computation Systems"
• Inventors: Petrichkovich, Yaroslav, Gusev, Vladimir, Lotorev, Vitaliy, Reshetnikova, Iuliia, Andreev, Aleksandr, Solokhina, Tatiana, Belyaev, Andrey, Khamukhin, Anatoly, Leontiev, Anton, Frolov, Dmitriy, Kuznetsov, Denis, Putrya, Fedor, Funkner, Aleksandr, Menshenin, Leonid
• Primary Examiner(s): Entezari, Michelle M
• Application Number:
  US15/615,507
• Publication Number:
  US 20170353708A1
• Time in Patent Office: 770 Days
• Field of Search: None
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

The present invention is related to the processing of stereo pairs (images obtained simultaneously from two cameras), namely, to methods and devices for processing stereo images. The technical result of the invention is creation of a method and a device for processing high-resolution stereo images with improved efficiency (performance) using a limited amount of memory with the help of a hardware accelerator for calculating the disparity map using a semi-global stereo matching algorithm with three-stage post processing (peak detection, interpolation and median filtering), which requires placing only a few strings (from 4 to 24 at different stages of calculating) in the local memory, which allows to obtain a high-quality disparity map in real time, and also by placing programmable cores of signal processors into the device, which prepare data for calculating the disparity map (rectification of images) and final processing of data using video analytics algorithms, and the hardware accelerator, which minimizes the transfer between external memory and the device, since the left and right images are fully loaded only once, and the results of the processing are data structures of a much smaller volume.

Claims

1. A method for stereo image processing in which:

left and right source images are loaded line by line using a DMA controller from external memory into local memory buffers;

the left and right source images are processed line by line using cores of signal processors, while their rectification is being performed, and the rectified left and right images are written to the buffers of the local memory;

a disparity map is computed, using a hardware accelerator using the rectified left and right images, and is written to the local memory buffer, wherein detecting peaks in the disparity map is carried out in parallel for all pixels in a processing window of a predetermined size, and wherein the disparity map is interpolated after detecting the peaks by median filtering on nearest valid pixels of eight directions with a programmable length of the rays according to an anisotropic filtering algorithm or a major isotropic filtering; and

the rectified left and right images are processed, using the cores of the signal processors, using the disparity map to create processed pixel-related information data structures that are written to the local memory and then unloaded with the use of the DMA controller into the external memory.

2. The method of claim 1 distinguished by the local memory buffers, in which the images and the disparity map are stored, having a size of several lines of the image, and are organized on the principle of double buffering.

3. The method of claim 1 distinguished by the fact that the rectification of the left and right source images is performed, the coordinates of the prototypes of the rectified image, pixels in the original image are calculated, and the brightness of the prototypes of the pixels is calculated by bilinear interpolation over the four neighbouring pixels.

4. The method of claim 1 distinguished by the fact that the semi global block matching algorithm (SGBM) is calculated in two directions, while calculating the matching cost function and the accumulated cost in each direction, while checking the disparity map of the left image based on the disparity map of the right image and performing the post-filtration of the disparity map, consisting of three stages:

peak detection, interpolation, and median filtration.

5. The method of claim 1 distinguished by the fact that the synchronization of the devices using the local memory is performed using hardware and an event register.

6. The method of claim 1 distinguished by the fact that the rectification of the left and right source images is performed in parallel.

7. The method of claim 1 distinguished by the fact that the rectification of the left and right source images is performed simultaneously with correction of radial and tangential distortions of the first and second orders.

8. The method of claim 1 distinguished by the fact that the disparity map is computed, while two lines of the image are processed concurrently, with intermediate results stored in the local memory only for odd lines.

9. The method of claim 1 distinguished by the fact that the median filtering of a dense disparity map is performed in several successive stages, during each of which parallel processing of pixels from several adjacent image lines is performed using a median filter group.

16. The method of claim 4 distinguished by the fact that the peak detection is performed using a computational module that is a matrix of similar computational cells controlled by a single instruction, multiple data (SIMD) flow, in which the number of cells corresponds to the number of pixels in the computational window, each cell is configured with the possibility of obtaining a pixel value corresponding to the position of this cell in the computational window, as well as the values, of neighbouring pixels on the left, right, bottom and top, with the ability to store the current value, which changes as a result of the execution of instructions and completion of the sequence of instructions for the formation of the sign of invalidation.

10. A stereo image processing device comprising several cores of signal processors and a hardware accelerator connected via an arbiter to local memory, a DMA controller, a program cache and a data cache, while the cores of the signal processors and the hardware accelerator are connected to an external system interface and are configured to exchange data therethrough under the control of an external central processing unit, and the DMA controller, the program cache and the data cache are connected to the external memory and configured to exchange data, while

the arbiter is configured to control the access of the device to the external memory;

the DMA controller is configured to transfer data between the device and the external memory, while loading the left and right source images from the external memory line by line into the local memory buffers;

the cores of the signal processors are configured to process line by line the left and right source images, while rectifying them and writing the rectified left and right images to the buffers of the local memory;

the hardware accelerator is configured to calculate the disparity map, using the rectified left and right images, and, write it to the local memory buffer, wherein the hardware accelerator comprises left and right image pipelines containing a matching cost function calculating unit and an accumulating cost calculating unit that are configured to simultaneously calculate left and right disparity maps or are combined into one device for processing the left disparity map, a unit for checking the disparity map of the left image using the disparity map of the right image, a post-processing unit of the disparity map containing a peak detection unit, an interpolation unit, a median filtering unit, as well as a control unit, and wherein the interpolation unit is configured to perform parallel processing of N pixels of N lines, wherein the maximum length of the rays in all directions is N;

the cores of the signal processors are designed to process the rectified left and right images the disparity map to create processed pixel-related information data structures and to write the data structures to the local memory; and

the DMA controller is configured to unload the data structures from the local memory to the external memory.

11. The device of claim 10 distinguished by the fact that the local memory is made as banks of the same volume and bit capacity, access to which can be obtained both by transactions to one bank with a capacity equal to the bank'"'"'s capacity and transactions to several banks simultaneously with a bit capacity equal to the total bit capacity of several banks, using two addressing modes under the control of the arbiter, configured with the possibility of arbitration to each bank independently with fixed priorities.

12. The device of claim 10 distinguished by the fact that the unit for calculating the matching cost function is made as an array of the same cells that are configured to process several lines of the image in the pipeline, while the processing time of one pixel can vary several times depending on the maximum disparity values.

13. The device of claim 10 distinguished by the fact that the calculation of the accumulated matching cost in the horizontal and vertical directions is performed by identical blocks consisting of an array of the same cells and a unit for calculating the minimum, while the number of successive processing steps varies depending on the maximum disparity value.

14. The device of claim 10 distinguished by the fact that the peak detection unit is a computational module that is a matrix of similar computational cells controlled by a single instruction, multiple data (SIMD) flow, in which the number of cells corresponds to the number of pixels in the computational window, each cell is configured with the possibility of obtaining a pixel value corresponding to the position of this cell in the computational window, as well as the values of neighbouring pixels on the left, right, bottom and top, with the ability to store the current value, which changes as a result of the execution of instructions and completion of the sequence of instructions for the formation of the sign invalidation.

15. The device of claim 10 distinguished by the fact that the median filtering unit is made as several consecutive processing steps, each of which performs parallel processing of pixels from several lines.