Data processing system and method

• US 20060013473A1
• Filed: 07/21/2005
• Published: 01/19/2006
• Est. Priority Date: 04/15/1997
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Vulcan Patents LLC
• Inventors: Baker, Henry Harlyn, Alkire, Robert Dale, Woodfill, John Iselin, Herzen, Brian Von
• Granted Patent: US 7,567,702 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G01C 11/06 by comparison of two or more pictures of the same area
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: G06V 10/757 Matching configurations of points or features
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/167 Synchronising or controlling image signals
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

A powerful, scaleable, and reconfigurable image processing system and method of processing data therein is described. This general purpose, reconfigurable engine with toroidal topology, distributed memory, and wide bandwidth I/O are capable of solving real applications at real-time speeds. The reconfigurable image processing system can be optimized to efficiently perform specialized computations, such as real-time video and audio processing. This reconfigurable image processing system provides high performance via high computational density, high memory bandwidth, and high I/O bandwidth. Generally, the reconfigurable image processing system and its control structure include a homogeneous array of 16 field programmable gate arrays (FPGA) and 16 static random access memories (SRAM) arranged in a partial torus configuration. The reconfigurable image processing system also includes a PCI bus interface chip, a clock control chip, and a datapath chip. It can be implemented in a single board. It receives data from its external environment, computes correspondence, and uses the results of the correspondence computations for various post-processing industrial applications. The reconfigurable image processing system determines correspondence by using non-parametric local transforms followed by correlation. These non-parametric local transforms include the census and rank transforms. Other embodiments involve a combination of correspondence, rectification, a left-right consistency check, and the application of an interest operator.

Claims

1. A method of generating disparity results with low latency in an image processing system that processes elements of a first set of image data and a second set of image data, each having a plurality of elements, comprising:

receiving elements of the first and second data sets, including a first element of the first data set; and

generating a disparity result for the first element before substantially all elements of the first and second data sets have been received.

2. The method of claim 1 wherein the elements of the first and second data sets represent pixel intensities, further comprising performing a left-right consistency check in which inconsistent pixels are labeled and discarded for purposes of future processing.

3. The method of claim 1 further comprising performing an interest operation in which displacements in regions which have a low degree of contrast or texture are discarded.

4. The method of claim 1 further comprising filtering selected disparities based on a population analysis.

5. The method of claim 1 further comprising determining whether an object is moving by calculating motion between a first image and a second image taken at different times.

6. The method of claim 1 wherein the image data is received from a camera, further comprising storing as an attribute of pixel information associated with one or more images depth information calculated based at least in part on the disparity result, wherein the depth information is used to compress objects in an image that are more distant to the camera to a greater degree than objects which are closer to the camera.

7. The method of claim 1 further comprising guiding a robot by determining the distance to an object based at least in part on the disparity result.

8. The method of claim 1 further comprising eliminating background information beyond a certain distance based at least in part on the disparity result.

9. The method of claim 1 further comprising creating a composite image in which an object from a first video sequence is inserted at the appropriate depth into a second video sequence based at least in part on the disparity result.

10. A method for determining the distance of image elements from an aligned first camera and second camera by generating disparity results for such image elements with low latency, comprising:

obtaining image elements of a first data set and a second data set, representing pixel intensities captured respectively by the first camera and the second camera, the first and second data sets each having a plurality of image elements, including a first image element of the first data set that is offset from each of a plurality of selected image elements of the second data set by a particular disparity;

performing transformations on such received image elements, including the first image element and each of the plurality of selected image elements;

correlating the transformed first image element with each of the plurality of transformed selected image elements to generate a correlation result for each disparity, while concurrently performing transformations on one or more image elements of each of the first and second data sets;

generating a disparity result for the first image element by selecting an optimal one of the correlation results, in accordance with predetermined optimization criteria, while concurrently correlating transformed image elements of each of the first and second data sets; and

determining the distance of the first image element from the first and second cameras, based upon the disparity result;

wherein the first and second cameras are included in a pair of eyeglasses.

11. A method of generating disparity results with low latency in a sound processing system that processes elements of a first set of sound data and a second set of sound data, each having a plurality of elements, comprising:

receiving elements of the first and second data sets, including a first element of the first data set; and

generating a disparity result for the first element before substantially all elements of the first and second data sets have been received.