Generation of a disparity result with low latency

• US 7,925,077 B2
• Filed: 01/30/2009
• Issued: 04/12/2011
• Est. Priority Date: 04/15/1997
• Status: Expired due to Fees

• Current Assignee: Intel Corporation
• Original Assignee: Tyzx, Inc. (Intel Corporation)
• Inventors: Von Herzen, Brian, Baker, Henry Harlyn, Alkire, Robert Dale, Woodfill, John Iselin
• Primary Examiner(s): Johns; Andrew W
• Application Number:
  US12/362,562
• Publication Number:
  US 20090136091A1
• Time in Patent Office: 802 Days
• Field of Search: 382/106, 382/107, 382/154, 382/278, 382/303, 382/304, 348/42, 348/43, 348/46, 348/47, 708/422, 708/424
• US Class Current: 382/154
• CPC Class Codes: G01C 11/06 by comparison of two or more pictures of the same area
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: G06V 10/757 Matching configurations of points or features
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/167 Synchronising or controlling image signals
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

A powerful, scaleable, and reconfigurable image processing system and method of processing data therein is described. This general purpose, reconfigurable engine with toroidal topology, distributed memory, and wide bandwidth I/O are capable of solving real applications at real-time speeds. The reconfigurable image processing system can be optimized to efficiently perform specialized computations, such as real-time video and audio processing. This reconfigurable image processing system provides high performance via high computational density, high memory bandwidth, and high I/O bandwidth. Generally, the reconfigurable image processing system and its control structure include a homogeneous array of 16 field programmable gate arrays (FPGA) and 16 static random access memories (SRAM) arranged in a partial torus configuration. The reconfigurable image processing system also includes a PCI bus interface chip, a clock control chip, and a datapath chip. It can be implemented in a single board. It receives data from its external environment, computes correspondence, and uses the results of the correspondence computations for various post-processing industrial applications. The reconfigurable image processing system determines correspondence by using non-parametric local transforms followed by correlation. These non-parametric local transforms include the census and rank transforms. Other embodiments involve a combination of correspondence, rectification, a left-right consistency check, and the application of an interest operator.

Claims

1. A method of generating disparity results with low latency in an image processing system that processes elements of a first set of image data and a second set of image data, each having a plurality of elements, comprising:

generating, using a processor, a disparity result for a first element before all elements of the first data set and the second data set have been received.

2. The method of claim 1, further comprising recognizing a hand gesture in order to control an application by tracking a location and an orientation of a user'"'"'s hand based at least in part on the disparity result.

3. The method of claim 2, wherein the application includes providing instructions for a visual display.

4. The method of claim 1, further comprising alerting a person entering a controlled zone to a danger or redirecting hazardous activity around the person based at least in part on the disparity result.

5. The method of claim 1, further comprising detecting a position of one or more vehicles and/or one or more pedestrians for a traffic monitoring application based at least in part on the disparity result.

6. The method of claim 1, further comprising determining whether an object associated with the first element is moving by calculating a motion associated with the object as the object appears in a first image and in a second image taken at different times.

7. A system for generating disparity results with low latency in an image processing system that processes elements of a first set of image data and a second set of image data, each having a plurality of elements, comprising:

a processor configured to generate a disparity result for a first element before all elements of the first data set and the second data set have been received; and

an input interface coupled to the processor and configured to receive the first set of image data and the second set of image data; and

an output interface coupled to the processor and configured to provide the disparity result.

8. A method of generating a disparity result with low latency in a data processing system that processes elements of a first set of data and a second set of data, each having a plurality of elements, comprising:

performing a transformation on a first subset of one or more elements and a second subset of one or more elements of a first data set;

performing a transformation on a third subset of one or more elements and a fourth subset of one or more elements of a second data set;

generating a first correlation result by correlating a transformed first subset of one or more elements with a transformed third subset of one or more elements; and

generating a second correlation result, concurrently with generating the first correlation result, by correlating a transformed second set of one or more elements with a transformed fourth subset of one or more elements, and

generating a disparity result based at least in part on the first correlation result and the second correlation result.

9. The method of claim 8, wherein the first subset and the second subset contain the same elements.

10. The method of claim 8, further comprising recognizing a hand gesture in order to control an application by tracking a location and an orientation of a user'"'"'s hand based at least in part on the correlation result.

11. The method of claim 10, wherein the application includes providing instructions for a visual display.

12. The method of claim 8, further comprising alerting a person entering a controlled zone to a danger or redirecting hazardous activity around the person based at least in part on the correlation result.

13. The method of claim 8, further comprising detecting a position of one or more vehicles and/or one or more pedestrians for a traffic monitoring application based at least in part on the correlation result.

14. The method of claim 8, further comprising determining whether an object associated with the transformed first subset of one or more elements is moving by calculating a motion associated with the object as the object appears in a first image and in a second image taken at different times.

15. A system for generating a disparity result with low latency in a data processing system that processes elements of a first set of data and a second set of data, each having a plurality of elements, comprising:

a processor configured to;

perform a transformation on a first subset of one or more elements and a second subset of one or more elements of a first data set;

perform a transformation on a third subset of one or more elements and a fourth subset of one or more elements of a second data set;

generate a first correlation result by correlating a transformed first subset of one or more elements with a transformed third subset of one or more elements; and

generate a second correlation result, concurrently with generating the first correlation result, by correlating a transformed second set of one or more elements with a transformed fourth subset of one or more elements;

generate a disparity result based at least in part on the first correlation result and the second correlation result; and

an input interface coupled to the processor and configured to receive the first set of data and the second set of data; and

an output interface coupled to the processor and configured to provide the disparity result.

16. The system of claim 15, wherein an application processor is configured to recognize a hand gesture in order to control an application by tracking a location and an orientation of a user'"'"'s hand based at least in part on the disparity result.

17. The system of claim 16, wherein the application includes providing instructions for a visual display.

18. The system of claim 15, wherein an application processor is configured to alert a person entering a controlled zone to a danger or redirecting hazardous activity around the person based at least in part on the disparity result.

19. The system of claim 15, wherein an application processor is configured to detect a position of one or more vehicles and/or one or more pedestrians for a traffic monitoring application based at least in part on the disparity result.

20. The system of claim 15, wherein an application processor is configured to determine whether an object associated with the first element is moving by calculating a motion associated with the object as the object appears in a first image and in a second image taken at different times.

21. The system of claim 15, wherein the first subset and the second subset contain the same elements.

22. The system of claim 15, wherein an application processor is configured to recognize a hand gesture in order to control an application by tracking a location and an orientation of a user'"'"'s hand based at least in part on the correlation result.

23. The system of claim 22, wherein the application includes providing instructions for a visual display.

24. The system of claim 15, wherein an application processor is configured to alert a person entering a controlled zone to a danger or redirecting hazardous activity around the person based at least in part on the correlation result.

25. The system of claim 15, wherein an application processor is configured to detect a position of one or more vehicles and/or one or more pedestrians for a traffic monitoring application based at least in part on the correlation result.

26. The system of claim 15, wherein an application processor is configured to determine whether an object associated with the transformed first subset of one or more elements is moving by calculating a motion associated with the object as the object appears in a first image and in a second image taken at different times.