Method and apparatus for processing images

• US 20010019621A1
• Filed: 04/18/2001
• Published: 09/06/2001
• Est. Priority Date: 08/28/1998
• Status: Active Grant

• Current Assignee: IMAX Corporation
• Original Assignee: Sarnoff Corporation (SRI International, Inc.)
• Inventors: Hanna, Keith James, Kumar, Rakesh, Sawhney, Harpreet Singh, Bergen, James Russell, Lubin, Jeffrey
• Granted Patent: US 6,430,304 B2
• Time in Patent Office: Days
• US Class Current: 382/107
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/15 for colour aspects of image signals
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 13/286 having separate monoscopic and stereoscopic modes
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

A method and apparatus for accurately computing parallax information as captured by imagery of a scene. The method computes the parallax information of each point in an image by computing the parallax within windows that are offset with respect to the point for which the parallax is being computed. Additionally, parallax computations are performed over multiple frames of imagery to ensure accuracy of the parallax computation and to facilitate correction of occluded imagery.

Claims

1. A method for computing image flow information from a plurality of mages comprising:

aligning a plurality of images to form an aligned image;

defining a plurality of windows, where each of said windows circumscribe an image region containing a point within said aligned image;

offsetting at least one of said windows from said point;

computing a flow estimation within each of said windows;

identifying the flow estimation having the lowest error; and

deeming said flow estimation associated with said lowest error as said flow information for said point.

2. The method of

claim 1

wherein said flow information is constrained to produce parallax information.

3. The method of

claim 1

wherein one of said windows is centered upon said point.

4. The method of

claim 1

wherein said windows have different sizes.

5. The method of

claim 1

wherein said plurality of images comprises a plurality of images and said windows are defined in said aligned images.

6. The method of

claim 1

wherein said plurality of images are tiled and pairs of tiles form said plurality of images.

7. The method of

claim 1

wherein each said plurality of images are imaged contemporaneously.

8. The method of

claim 1

further comprising the steps of:

computing a flow estimate for each of said aligned images;

identifying a flow estimate having a lowest error;

identifying, in response to said flow estimate, errant information in a first aligned image; and

repairing said errant information in said first aligned image with information from at least one other aligned image.

9. The method of

claim 1

wherein said flow estimate is constrained to form a parallax estimate.

10. The method of

claim 1

wherein said flow estimation is corrected.

11. A method for enhancing regions within a plurality of images comprising:

aligning a plurality of images to form a plurality of aligned images;

computing a flow estimation for each of said aligned images;

identifying flow estimation having the lowest error;

identifying, in response to said flow estimation, regions in a first aligned image; and

enhancing said regions in said first aligned image with information from at least one other aligned image.

12. The method of

claim 11

wherein said flow estimation is constrained to form a parallax estimation.

13. The method of

claim 11

wherein computing step further comprises:

computing an epipolar constraint for each of said aligned images; and

computing a flow field representing image changes from aligned image to aligned image.

14. The method of

claim 11

wherein said computing step further comprises the step of:

computing a temporal constraint.

15. The method of

claim 11

further comprising the steps of:

computing a flow estimation for a second aligned image; and

using the flow estimation from said second aligned image to correct a flow estimation for said first aligned image.

16. The method of

claim 11

wherein said region is caused by noise and said enhancing said step reduces said noise.

17. A method of determining image flow comprising the steps of:

aligning a plurality of pairs of images in said plurality of images to form a plurality of aligned images;

computing a flow estimation for each of said aligned images to produce a plurality of flow estimates;

weighting the flow estimates; and

compositing an image by combining the weighted flow estimates.

18. The method of

claim 17

wherein said flow estimation is constrained to produce parallax estimation.

19. The method of

claim 17

wherein said flow estimation is corrected.

20. The method of

claim 17

wherein the weighting step weights flow estimates for images over time.

21. Apparatus for generating a enhancing an image comprising:

a first imaging device for producing first images at a first resolution;

a second imaging device for producing second images at a second resolution;

an image processor coupled to said first and said second imaging devices, for using said second image to enhance said first image.

22. The apparatus of

claim 21

wherein said image processor comprises:

an image flow generator.

23. The apparatus of

claim 22

wherein said image flow generator is a parallax computer.

24. The apparatus of

claim 23

wherein said parallax computer further comprises one or more augmentation modules selected from the group consisting of:

a module for dividing the images into tiles, a depth correlator, a module which performs Just Noticeable Differences, a correspondence checker, and a blank area avoidance module.