Image capture and viewing system and method for generating a synthesized image

• US 7,307,654 B2
• Filed: 10/31/2002
• Issued: 12/11/2007
• Est. Priority Date: 10/31/2002
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Chang, Nelson Liang An
• Primary Examiner(s): Villecco, John M.
• Application Number:
  US10/285,969
• Publication Number:
  US 20040085451A1
• Time in Patent Office: 1,867 Days
• Field of Search: 348/47, 348/187, 348/188, 348/218.1, 348/263, 348/333.11, 348/333.12, 382/294, 382/293, 382/295
• US Class Current: 348/218.1
• CPC Class Codes: G06T 15/10 Geometric effects
• CPC Class Codes: G06T 7/85 Stereo camera calibration

Abstract

An image capture and viewing system is disclosed. In one embodiment, the present invention provides an image capture system for providing an image of a subject. The image capture system includes a planar surface, a first camera, a second camera and a control module. The first camera is positioned to capture a first subject image of a subject positioned between the planar surface and the first camera. The second camera is spaced from the first camera. The second camera is positioned to capture a second subject image of the subject. The control module is configured to generate a synthesized image of the subject from a virtual view point using the relative positioning of the first camera, the second camera, and a planar surface.

Claims

1. An image capture system for providing an image of a subject comprising:

a single, fixed planar surface of a planar background;

a first camera positioned to capture a first subject image of a live subject positioned between the single, fixed planar surface and the first camera;

a second camera spaced from the first camera, the second camera positioned to capture a second subject image of the live subject located between the single, fixed planar surface and the second camera; and

a control module configured to generate a synthesized image of the live subject from a virtual viewpoint using a fixed, relative position of the first camera, the second camera, and only the single, fixed planar surface,

wherein the first camera, the second camera, and the single, fixed planar surface have the same fixed relative position during a calibration of the first camera and the second camera.

2. The image capture system of claim 1, further comprising:

a first image capture device, wherein the first subject image is captured from the first camera via the first image capture device; and

a second image capture device, wherein the second subject image is captured from the second camera via the second image capture device.

3. The image capture system of claim 2, wherein the control module is configured to generate a first coordinate set with respect to the first camera and a second coordinate set with respect to the second camera, the first coordinate set adapted to be converted into the second coordinate set.

4. The image capture system of claim 2, further comprising:

calibration settings stored within the control module, used to determine the fixed relative position of the first camera, the second camera, and the single, fixed planar surface.

5. The image capture system of claim 4, further comprising:

a calibration pattern temporarily positioned on the planar surface, the calibration pattern having a plurality of robust features;

a first calibration image of the calibration pattern generated from the first camera via the first image capture device; and

a second calibration image of the calibration pattern generated from the second camera via the second image capture device;

wherein the control module determines the calibration settings using the first calibration image, the second calibration image, and the robust features of the calibration pattern to determine the fixed relative position of the first camera, the second camera, and the single, fixed planar surface.

6. The image capture system of claim 1, wherein the first central processing unit is configured to generate the synthesized image at an individual time instance.

7. The image capture system of claim 1, wherein the first central processing unit is configured to generate the synthesized image at a plurality of consecutive time instances.

8. The image capture system of claim 1, further comprising:

a control unit in communication with the control module, adapted to allow a user to select the location of the virtual viewpoint.

9. The image capture system of claim 1, further comprising:

an image display coupled to the control module, the image display adapted to allow the user to view at least one of the synthesized image, the first subject image, or the second subject image.

10. An image capture system comprising:

a planar background having a single, fixed planar surface;

a plurality of cameras including;

a first camera positioned to capture a first subject image of a live subject positioned between the single, fixed planar surface of the planar background and the first camera;

a second camera spaced from the first camera, the second camera positioned to capture a second subject image of the subject; and

a first central processing unit coupled to each of the plurality of cameras, the first central processing unit configured to utilize a fixed relative position of the first camera, the second camera, and only the single, fixed planar surface of the planar background to convert the first subject image and the second subject image into a synthesized image of the live subject from a virtual viewpoint, wherein the virtual viewpoint is positioned between the first camera and the second camera,

wherein the first camera, the second camera, and the single, fixed planar surface have the same fixed relative position during a calibration of the first camera and the second camera.

11. The image capture system of claim 10 wherein the first central processing unit is configured to generate a first coordinate series with respect to the first camera and a second coordinate with respect to the second camera, the first coordinate adapted to be converted into the second coordinate.

12. The image capture system of claim 10, further comprising:

a calibration pattern temporarily positioned on the single, fixed planar surface of the planar background, the calibration pattern having a plurality of robust features; and

wherein the first camera is configured to capture a first calibration image of the calibration pattern and the second camera is configured to capture a second calibration image of the calibration pattern, the first central processing unit having a memory of the robust features and being adapted to utilize the first calibration image, the second calibration image, and the robust features to determine the fixed relative position of the first camera, the second camera, and the single, fixed planar surface of the planar background.

13. The image capture system of claim 10, wherein the first central processing unit is configured to generate the synthesized image at an individual time instance.

14. The image capture system of claim 10, wherein the first central processing unit is configured to generate the synthesized image at a plurality of consecutive time instances.

15. The image capture system of claim 10, further comprising:

a controller unit coupled to the first central processing unit, the controller unit adapted to allow a user to select the location of the virtual viewpoint.

16. The image capture system of claim 10, further comprising:

an image display coupled to the first central processing unit, the image display adapted to allow the user to view at least one of the synthesized image, the first subject image, or the second subject image.

17. The image capture system of claim 10, further comprising:

a network communication link coupled to the first central processing unit; and

a second central processing unit coupled to the network communication link;

a controller unit coupled to the second central processing unit, the controller unit adapted to allow a user to select the location of the virtual viewpoint; and

an image display coupled to the second central processing unit, the image display adapted to allow the user to view at least one of the synthesized image, the first subject image, or the second subject image.

18. A method of viewing synthesized images derived from actual images, the method comprising:

providing a plurality of visual characteristics for a first subject image of a live image and a second subject image of a live image, the first subject image being captured by a first camera and the second subject image being captured by a second camera spaced from the first camera, wherein the first subject image is related to the second subject image based on a predetermined spatial relationship of each respective first and second camera relative to a single, fixed planar surface of a planar background;

calibrating the first and second camera while maintaining the same fixed relative position of the first camera, second camera, and the single, fixed planar surface;

selecting a virtual viewpoint positioned between the first camera and the second camera;

generating a synthesized image by interpolating a plurality of visual characteristics of the synthesized image from the plurality of visual characteristics of the first subject image and the second subject image based on a position of the virtual viewpoint with respect to the first camera the second camera, and the single, fixed planar surface of a planar background wherein the synthesized image represents an image that would have been captured from the virtual viewpoint; and

displaying the synthesized image to a user.

19. The method of claim 18, wherein the first image and the second image provided represent an individual time instance.

20. The method of claim 19, wherein providing a plurality of visual characteristics, selecting a virtual viewpoint, generating a synthesized image, and displaying the synthesized image are repeated for a plurality of individual time instances.

21. The method of claim 18, wherein generating a synthesized image by interpolating the visual characteristics of the synthesized image includes:

determining an interpolation parameter based upon the location of the virtual viewpoint with respect to the first camera and the second camera;

deriving the visual characteristics of the synthesized image using the interpolation parameter and the visual characteristics of the first subject image and the second subject image.

22. A computer-readable medium having computer-executable instructions for performing a method of viewing synthesized images derived from actual images comprising:

providing a plurality of visual characteristics for a first subject image of a live image and a second subject image of a live image, the first subject image being captured by a first camera and the second subject image being captured by a second camera spaced from the first camera, wherein the first subject image is related to the second subject image based on a predetermined spatial relationship of each respective first and second camera relative to a single, fixed planar surface of a planar background;

calibrating the first and second camera while maintaining the same fixed relative position of the first camera, second camera and the single, fixed planar surface;

selecting a virtual viewpoint positioned between the first camera and the second camera;

generating a synthesized image by interpolating a plurality of visual characteristics of the synthesized image from the plurality of visual characteristics of the first subject image and the second subject image based on a position of the virtual viewpoint with respect to the first camera, the second camera, and the single, fixed planar surface of a planar background wherein the synthesized image represents an image that would have been captured from the virtual viewpoint; and

displaying the synthesized image to a user.

23. A computer-readable medium having computer-executable instructions for performing a method of image capture for generation of a synthesized image comprising:

directing a plurality of cameras toward a planar background, wherein the plurality of cameras includes a first camera and a second camera spaced from the first camera;

capturing a first calibration image with the first camera, the first calibration image being of a calibration pattern, the calibration pattern having a plurality of known robust features and being positioned on the planar background;

capturing a second calibration image with the second camera, the second calibration image being of the calibration pattern;

determining a spatial relationship between the first camera, the second camera, and the planar background by analyzing the first calibration image and the second calibration image with respect to the robust features of the calibration pattern;

placing a subject between the planar background and the first and second cameras;

capturing a first subject image of the subject with the first camera and a second subject image of the subject with the second camera;

comparing each point in the first subject image to a corresponding object point of the second subject image for a color discrepancy, any object points having the color discrepancy are classified as an extracted object point; and

deriving a representation of the shape and color for each of the extracted object points.

24. A method of image capture for subsequent generation of a synthesized image, the method comprising:

directing a plurality of cameras toward a planar background, wherein the plurality of cameras includes a first camera and a second camera spaced from the first camera;

capturing a first calibration image with the first camera, the first calibration image being of a calibration pattern, the calibration pattern having a plurality of known robust features and being positioned on the planar background;

capturing a second calibration image with the second camera, the second calibration image being of the calibration pattern;

determining a spatial relationship between the first camera, the second camera, and the planar background by analyzing the first calibration image and the second calibration image with respect to the robust features of the calibration pattern;

placing a subject between the planar background and the first and second cameras;

capturing a first subject image of the subject with the first camera and a second subject image of the subject with the second camera;

comparing each point in the first subject image to a corresponding object point of the second subject image for a color discrepancy, any object points having the color discrepancy are classified as an extracted object point; and

deriving a representation of the shape and color for each of the extracted object points.

25. The method of claim 24, wherein determining a spatial relationship between the first camera, the second camera, and the planar background includes:

examining the first calibration image and the second calibration image for the robust features of the calibration image;

comparing a spatial relationship of the robust features in the first calibration image to the spatial relationship of the robust features in the calibration pattern;

determining the spatial relationship of the first camera to the planar background;

comparing the spatial relationship of the robust features in the second calibration image to the spatial relationship of the robust features in the calibration pattern;

determining the spatial relationship of the second camera to the planar background.

26. The method of claim 25, wherein determining a spatial relationship between the first camera, the second camera, and the planar background further includes:

determining the relationship of the first camera to the second camera utilizing the relationship of the first camera to the planar background and the relationship of the second camera to the planar background.

27. The method of claim 24, wherein the corresponding object point of the second image is determined using the known spatial relationship between the first camera, the second camera, and the planar background.

28. The method of claim 24, further comprising:

dividing the first subject image and the second subject image over a time period into a plurality of individual time instances.

29. The method of claim 28, wherein converting the first subject image, comparing each of a plurality of object points, and deriving a representation of the position and color are completed for each of the individual time instances.

30. The method of claim 24, wherein the representation includes a shape and a color for the first image and a shape and a color of the second image.