Scalable architecture for corresponding multiple video streams at frame rate

• US 7,054,491 B2
• Filed: 11/16/2001
• Issued: 05/30/2006
• Est. Priority Date: 11/16/2001
• Status: Active Grant

• Current Assignee: STMicroelectronics Incorporated (STMicroelectronics NV)
• Original Assignee: STMicroelectronics Incorporated (STMicroelectronics NV)
• Inventors: Stein, Clifford M., Chen, George Q., McGuinness, Peter J., Ng, Kim C.
• Primary Examiner(s): MARIAM, DANIEL G
• Application Number:
  US09/993,970
• Publication Number:
  US 20030095711A1
• Time in Patent Office: 1,656 Days
• Field of Search: 382/181, 382/190, 382/201, 382/216, 382/218, 382/278, 382/294
• US Class Current: 382/209
• CPC Class Codes: G06T 7/33 using feature-based methods

Abstract

An image processing system which processes, in real time, multiple images, which are different views of the same object, of video data in order to match features in the images to support 3 dimensional motion picture production. The different images are captured by multiple cameras, processed by digital processing equipment to identify features and perform preliminary, two-view feature matching. The image data and matched feature point definitions are communicated to an adjacent camera to support at least two image matching. The matched feature point data are then transferred to a central computer, which performs a multiple-view correspondence between all of the images.

Claims

1. An image processing system comprising:

a plurality of image feature detectors, wherein each image feature detector within the plurality of image feature detectors detects a set of distinct image features in at least one of a first plurality of images, wherein the first plurality of images comprises images of a scene that were captured from different physical locations whose physical location is not known a priori;

a plurality of initial image correlators, wherein each initial image correlator within the plurality of initial image correlators is communicatively coupled with at least one of the image feature detectors within the plurality of image feature detectors, with each initial image correlator within the plurality of initial image correlators determining a respective first correspondence of distinct image features within at least two images of the first plurality of images, the plurality of initial image correlators thereby determining a plurality of correspondences;

a final image correlator, separate from the plurality of initial image correlators, for determining a final correspondence of distinct image features detected in a second plurality of images by processing the first set of distinct image features, the second set of distinct image features, and the plurality of correspondences; and

a communications interface communicatively coupling each of the plurality of initial image correlators to the final image correlator in order to communicate the first set of distinct image features, the second set of distinct image features, and the first correspondence from each of the plurality of initial image correlators to the final image correlator, and wherein each inital image correlator computes a first likely match set of distinct image features that is determined to have a maximum average strength of correspondence based at least in part on a total number of matching neighbor distinct image features for each match of the first likely match set.

2. An image processing system according to claim 1, wherein distinct image features are feature points.

3. An image processing system according to claim 1, wherein each image within the first plurality of images and within the second plurality of images is one image within a real time motion picture.

4. An image processing system according to claim 1, wherein a first image feature detector within the plurality of image feature detectors and a first initial image correlator within the plurality of initial image correlators are communicatively coupled to a first digital processor that is communicatively coupled to a first camera for capturing at least one image of the plurality of images.

5. An image processing system according to claim 4, wherein each initial image correlator within the plurality of initial image correlators and the final image correlator are separate from one another, and wherein the communications interface provides a communications channel for each correspondence determined by each initial image correlator.

6. An image processing system of claim 1, wherein each initial image correlator computes a first likely match set of distinct image features that is determined to have a maximum average strength of correspondence based at least in part on a total number of matching neighbor distinct image features for each match of the first likely match set.

7. An image processing system of claim 1, wherein the final image correlator refines the first correspondence, resulting in at least a second potential match set of image features between the at least two images, wherein the at least a second potential match set is based at least in part on a computation of reprojection error for matched distinctive image points that result from a projective reconstruction of the at least a second potential match set.

8. An image processing system of claim 7, wherein the at least a second potential match set is based at least in part on a least median of squares computation of the reprojection errors related to matched distinctive image points in the at least a second potential match set.

9. A method of matching image features between a plurality of images, the method comprising the steps of:

detecting, by a first processor, a first set of distinct image features in a first image of a first plurality of images, wherein the first plurality of images comprises images of a scene that were contemporaneously captured from different physical locations which are not known a priori;

determining, by the first processor, a first correspondence of distinct images features “

determined to have a maximum average strength of correspondence”

between the first set of distinct image features and at least a second set of distinct image features detected within at least a second image of the first plurality of images;

communicating the first set of distinct image features, the second set of distinct image features, and a plurality of correspondences of distinct image features to a second processor, wherein the plurality of correspondences comprises the first correspondence; and

determining, by the second processor, a final correspondence of distinct image features detected in a second plurality of images by processing the first set of distinct image features, the second set of distinct image features, and the plurality of correspondences, and wherein the step of determining a first correspondence comprises the step of producing a first likely match set of distinct image features that is determined to have a maximum average strength of correspondence based at least in part on a total number of matching neighbor distinct image features for each match of the first likely match set.

10. A method according to claim 9, wherein distinct image features are feature points.

11. A method according to claim 9, wherein the steps of the method are performed repeatedly to process real-time video data.

12. A method according to claim 9, wherein the step of detecting and the step of determining a first correspondence are performed in the first processor that is associated with a camera which captured the first image.

13. The method of claim 9, wherein the step of determining a first correspondence comprises the step of producing a first likely match set of distinct image features that is determined to have a maximum average strength of correspondence based at least in part on a total number of matching neighbor distinct image features for each match of the first likely match set.

14. The method of claim 9, wherein the step of determining a final correspondence comprises the step of determining a refinement of the first correspondence that results in at least a second potential match set of image features between the first image and the at least a second image, wherein the at least a second potential match set is based at feast in part on a computation of reprojection error for matched distinctive image points that results from a projective reconstruction of the at least a second potential match set.

15. The method of claim 14, wherein the at least a second potential match set is based at least in part on a least median of squares computation of the reprojection errors related to matched distinctive image points in the at least a second potential match set.

16. The method of claim 9, wherein the first plurality of images further comprises a third image, the method further comprising:

detecting, by a third processor, a third set of distinct image features in the third image;

determining, by the third processor, a second correspondence of distinct image features between the third set of distinct image features and at least the second set of distinct image features; and

communicating, as part of the plurality of correspondences of distinct image features, the second correspondence to the second processor, and

wherein the second plurality of images comprises the first image, the at least a second image, and the third image.

17. The method of claim 9, wherein the at least a second image comprises a second image and a third image, the method further comprising:

detecting, by a third processor, a third set of distinct image features in the third image, and

wherein the determining a second correspondence comprises comparing three images using a trifocal tensor method, and

wherein the communicating further comprises communicating the third set of distinct image, features and, as part of the plurality of correspondences of distinct Image features, the second correspondence to the second processor.

18. A computer readable medium including computer instructions for matching image features between a plurality of images, the computer instructions comprising instructions for:

detecting, by a first processor, a first set of distinct image features in a first image of a first plurality of images, wherein the first plurality of images comprises images of a scene that were contemporaneously captured from different physical locations which are not known a priori;

determining, by the first processor, a first correspondence of distinct images features “

determined to have a maximum average strength of correspondence”

between the first set of distinct image features and at least a second set of distinct image features detected within at least a second image of the first plurality of images;

communicating the first set of distinct image features, the second set of distinct image features, and a plurality of correspondences of distinct image features to a second processor, wherein the plurality of correspondences comprises the first correspondence; and

determining, by the second processor, a final correspondence of distinct image features detected in a second plurality of images by processing the first set of distinct image features, the second set of distinct image features, and the plurality of correspondences, and wherein the instructions for determining first correspondence comprise instructions for producing a first likely match set of distinct image features that is determined to have a maximum average strength of correspondence based at least in part on a total number of matching neighbor distinct image features for each match of the first likely match set.

19. A computer readable medium according to claim 18, wherein distinct image features are feature points.

20. A computer readable medium according to claim 18, wherein the instructions are performed repeatedly to process real-time video data.

21. A computer readable medium according to claim 18, wherein the instructions for detecting and the instructions for determining a first correspondence are executed in the first processor that is associated with a camera which captured the first image.

22. The computer readable medium according to claim 18, wherein the instructions for determining first correspondence comprise instructions for producing a first likely match set of distinct image features that is determined to have a maximum average strength of correspondence based at least in part on a total number of matching neighbor distinct image features for each match of the first likely match set.

23. The computer readable medium according to claim 18, wherein the instructions for determining a final correspondence comprises instructions for determining a refinement of the first correspondence that results in at least a second potential match set of image features between the first image and the at least a second image, wherein the at least a second potential match set is based at least in part on a computation of reprojection error for matched distinctive image points that results from a projective reconstruction of the at least a second potential match set.

24. The computer readable medium according to claim 23, wherein the second potential match set Is based at least in part on a least median of squares computation of the reprojection errors related to matched distinctive image points in the at least a second potential match set.