Scalable architecture for corresponding multiple video streams at frame rate

• US 20030095711A1
• Filed: 11/16/2001
• Published: 05/22/2003
• Est. Priority Date: 11/16/2001
• Status: Active Grant

• Current Assignee: STMicroelectronics Incorporated (STMicroelectronics NV)
• Original Assignee: STMicroelectronics Incorporated (STMicroelectronics NV)
• Inventors: Stein, Clifford M., Chen, George Q., McGuinness, Peter J., Ng, Kim C.
• Granted Patent: US 7,054,491 B2
• Time in Patent Office: Days
• US Class Current: 382/209
• CPC Class Codes: G06T 7/33 using feature-based methods

Abstract

An image processing system which processes, in real time, multiple images, which are different views of the same object, of video data in order to match features in the images to support 3 dimensional motion picture production. The different images are captured by multiple cameras, processed by digital processing equipment to identify features and perform preliminary, two-view feature matching. The image data and matched feature point definitions are communicated to an adjacent camera to support at least two image matching. The matched feature point data are then transferred to a central computer, which performs a multiple-view correspondence between all of the images.

Claims

1. An image processing system comprising:

a plurality of image feature detectors, wherein each image feature detector within the plurality of image feature detectors detects a set of distinct image features in at least one of a first plurality of images, wherein the first plurality of images comprises images of a scene that were captured from different physical locations whose physical location is not known a priori;

a plurality of initial image correlators, wherein each initial image correlator within the plurality of initial image correlators is communicatively coupled with at least one of the image feature detectors within the plurality of image feature detectors, for determining a first correspondence of distinct image features within at least two images of the first plurality of images; and

a final image correlator, communicatively coupled with each of the plurality of initial image correlators, for determining a final correspondence of distinct image features detected in a second plurality of images, including the at least two images, within the first plurality of images.

2. An image processing system according to claim 1, wherein distinct image features are feature points.

3. An image processing system according to claim 1, wherein each image within the first plurality of images and within the second plurality of images is one image within a real time motion picture.

4. An image processing system according to claim 1, wherein a first image feature detector within the plurality of image feature detectors and a first initial image correlator within the plurality of initial image correlators are communicatively coupled to a first digital processor that is communicatively coupled to a first camera for capturing at least one image of the plurality of images.

5. An image processing system according to claim 4, wherein the final image correlator is communicatively coupled to a second processor.

6. An image processing system of claim 1, wherein each initial image correlator computes a first likely match set of distinct image features that is determined to have a maximum average strength of correspondence based at least in part on a total number of matching neighbor distinct image features for each match of the first likely match set.

7. An image processing system of claim 1, wherein the final image correlator refines the first correspondence, resulting in at least a second potential match set of image features between the at least two images, wherein the at least a second potential match set is based at least in part on a computation of reprojection error for matched distinctive image points that result from a projective reconstruction of the at least a second potential match set.

8. An image processing system of claim 7, wherein the at least a second potential match set is based at least in part on a least median of squares computation of the reprojection errors related to matched distinctive image points in the at least a second potential match set.

9. A method of matching image features between a plurality of images, the method comprising the steps of:

detecting a first set of distinct image features in a first image of a first plurality of images, wherein the plurality of images comprises images of a scene that were contemporaneously captured from different physical locations which are not known a priori;

determining a first correspondence of distinct feature images between the first set of distinct image features and at least a second set of distinct image feature detected within at least a second image of the first plurality of images; and

determining a final correspondence of distinct image features detected in a second plurality of images, including the first image and the at least a second image, within the first plurality of images.

10. A method according to claim 9, wherein distinct image features are feature points.

11. A method according to claim 9, wherein the steps of the method are performed repeatedly to process real-time video data.

12. A method according to claim 9, wherein the step of detecting and the step of determining a first correspondence are performed in a first processor that is associated with a camera which captured the first image.

13. A method according to claim 12, wherein the step of determining a final correspondence is performed in a second processor.

14. The method of claim 9, wherein the step of determining a first correspondence comprises the step of producing a first likely match set of distinct image features that is determined to have a maximum average strength of correspondence based at least in part on a total number of matching neighbor distinct image features for each match of the first likely match set.

15. The method of claim 9, wherein the step of determining a final correspondence comprises the step of determining a refinement of the first correspondence that results in at least a second potential match set of image features between the first image and the at least a second image, wherein the at least a second potential match set is based at least in part on a computation of reprojection error for matched distinctive image points that results from a projective reconstruction of the at least a second potential match set.

16. The method of claim 15, wherein the at least a second potential match set is based at least in part on a least median of squares computation of the reprojection errors related to matched distinctive image points in the at least a second potential match set.

17. A computer readable medium including computer instructions for matching image features between a plurality of images, the computer instructions comprising instructions for:

detecting a first set of distinct image features in a first image of a first plurality of images, wherein the plurality of images comprises images of a scene that were contemporaneously captured from different physical locations which are not known a priori;

determining a first correspondence of distinct feature images between the first set of distinct image features and at least a second set of distinct feature images detected within at least a second image of the first plurality of images; and

determining a final correspondence of distinct image features detected in a second plurality of images, including the first image and the at least a second image, within the first plurality of images.

18. A computer readable medium according to claim 17, wherein distinct image features are feature points.

19. A computer readable medium according to claim 17, wherein the instructions are performed repeatedly to process real-time video data.

20. A computer readable medium according to claim 17, wherein the instructions for detecting and the instructions for determining a first correspondence are executed in a first processor that is associated with a camera which captured the first image.

21. A computer readable medium according to claim 20, wherein the instructions for determining a final correspondence are executed in a second processor.

22. The computer readable medium according to claim 17, wherein the instructions for determining first correspondence comprise instructions for producing a first likely match set of distinct image features that is determined to have a maximum average strength of correspondence based at least in part on a total number of matching neighbor distinct image features for each match of the first likely match set.

23. The computer readable medium according to claim 17, wherein the instructions for determining a final correspondence comprises instructions for determining a refinement of the first correspondence that results in at least a second potential match set of image features between the first image and the at least a second image, wherein the at least a second potential match set is based at least in part on a computation of reprojection error for matched distinctive image points that results from a projective reconstruction of the at least a second potential match set.

24. The computer readable medium according to claim 23, wherein the second potential match set is based at least in part on a least median of squares computation of the reprojection errors related to matched distinctive image points in the at least a second potential match set.