PROCESSING IMAGE DATA FROM MULTIPLE CAMERAS FOR MOTION PICTURES

• US 20120176481A1
• Filed: 03/16/2012
• Published: 07/12/2012
• Est. Priority Date: 02/29/2008
• Status: Active Grant

• Current Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Original Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Inventors: LUKK, Howard, HENDRICKSON, Andrew
• Granted Patent: US 9,094,675 B2
• Time in Patent Office: Days
• US Class Current: 348/47
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof

Abstract

An approach is disclosed for generating stereoscopic image sequences. The approach includes receiving a plurality of image sequences, each image sequence captured from a distinct camera or other capture device in a multi-camera array. One image sequence is a primary image sequence and the other image sequences are secondary image sequences. The approach further includes generating, from the primary and at least one or more of the secondary image sequences, depth maps corresponding to images of the primary image sequence and generating, for each image in the primary image sequence, based on at least the depth maps and the corresponding images of the primary image sequence, stereoscopic images of a stereoscopic image sequence.

Claims

1. A computer-implemented method for processing image data, comprising:

receiving a plurality of image sequences, each image sequence captured from a distinct camera in a multi-camera array, wherein image sequences include a primary image sequence captured by a primary camera of the multi-camera array and one or more secondary image sequences captured by respective one or more secondary cameras of the multi-camera array;

generating, from at least the primary image sequence and one or more of the secondary image sequences, a depth map corresponding to each of a plurality of images of the primary image sequence; and

generating, for each image in the plurality of images of the primary image sequence, based on at least the depth map and the corresponding image of the primary image sequence, at least two images of a stereoscopic image sequence.

2. The method of claim 1, further comprising:

determining, based on the depth maps and the corresponding images of the primary image sequence, a set of objects depicted in the primary image sequence;

associating an object of the determined set of objects with a distinct layer in at least one of the depth maps; and

modifying an appearance of the object in the distinct layer so as to change an appearance of the object in stereoscopic images.

3. The method of claim 2, wherein generating the stereoscopic images includes filling a hole in at least a first one of the stereoscopic images using image data from one or more images of the secondary image sequences, wherein the hole corresponds to an area in the first stereoscopic image for which image data of a scene captured in the corresponding primary image is unavailable.

4. The method of claim 1, wherein at least one of the secondary cameras provides a plenoptic lens configured to capture light field data.

5. The method of claim 1, wherein the stereoscopic images include at least a left-eye image and a right-eye image, and wherein an appearance of the objects in the left-eye image and an appearance of the objects in the right-eye image are displaced from their respective positions in the corresponding image of the primary image sequence.

6. The method of claim 1, further comprising, prior to generating the depth maps:

receiving a set of images of the primary image sequence and one or more editing operations to be performed on the set of images; and

performing the one or more editing operations on the set of images.

7. The method of claim 1, further comprising, changing a viewpoint of a viewer of the stereoscopic image sequence based on at least the depth maps and the corresponding images of the primary image sequence.

8. The method of claim 1, further comprising, applying at least one graphical effect to one or more images of the primary image sequence based on data from one or more images of the secondary image sequences, wherein the graphical effect comprises at least one of enhancing a resolution, altering a zoom or a lighting, removing a defect, or increasing a frame rate of the stereoscopic image sequence.

9. The method of claim 1, wherein generating, for each image in the primary image sequence, based on at least the depth maps and the corresponding images of the primary image sequence, at least two images of a stereoscopic image sequence comprises generating an auto-stereoscopic image sequence.

10. A non-transitory computer-readable storage medium storing a program, which, when executed by a processor performs an operation for processing image data, the operation comprising:

receiving a plurality of image sequences, each image sequence captured from a distinct camera in a multi-camera array, wherein image sequences include a primary image sequence captured by a primary camera of the multi-camera array and one or more secondary image sequences captured by respective one or more secondary cameras of the multi-camera array;

generating, from at least the primary image sequence and one or more of the secondary image sequences, a depth map corresponding to each of a plurality of images of the primary image sequence; and

generating, for each image in the plurality of images of the primary image sequence, based on at least the depth map and the corresponding image of the primary image sequence, at least two images of a stereoscopic image sequence.

11. The computer-readable storage medium of claim 10, wherein the operation further comprises:

determining, based on at least the depth maps and the corresponding images of the primary image sequence, a set of objects depicted in the primary image sequence;

associating an object of the determined set of objects with a distinct layer in at least one of the depth maps; and

modifying an appearance of the object in the distinct layer so as to change an appearance of the object in stereoscopic images.

12. The computer-readable storage medium of claim 10, wherein the operation further comprises, filling a hole in at least a first one of the stereoscopic images using image data from one or more images of the secondary image sequences, wherein the hole corresponds to an area in the first stereoscopic image for which image data of a scene captured in the corresponding primary image is unavailable.

13. The computer-readable storage medium of claim 11, wherein at least one of the secondary cameras provides a plenoptic lens configured to capture light field data.

14. The computer-readable storage medium of claim 10, wherein the stereoscopic images include at least a left-eye image and a right-eye image, and wherein an appearance of the objects in the left-eye image and an appearance of the objects in the right-eye image are displaced from their respective positions in the corresponding image of the primary image sequence.

15. The computer-readable storage medium of claim 10, wherein the operation further comprises, prior to generating the depth maps:

receiving a set of images of the primary image sequence and one or more editing operations to be performed on the set of images; and

performing the one or more editing operations on the set of images.

16. The computer-readable storage medium of claim 10, wherein the operation further comprises, changing a viewpoint of a viewer of the stereoscopic image sequence based on at least the depth maps and the corresponding images of the primary image sequence.

17. The computer-readable storage medium of claim 10, wherein the operation further comprises, applying at least one graphical effect to an image of the primary image sequence based on data from one or more images of the secondary image sequences, wherein the graphical effect comprises at least one of enhancing a resolution, altering a zoom or a lighting, removing a defect, or increasing a frame rate of the stereoscopic image sequence based on data from one or more images of the secondary image sequences.

18. A system, comprising:

a processor; and

a memory, wherein the memory includes an application program configured to perform an operation, the operation comprising;

receiving a plurality of image sequences, each image sequence captured from a distinct camera in a multi-camera array, wherein image sequences include a primary image sequence captured by a primary camera of the multi-camera array and one or more secondary image sequences captured by respective one or more secondary cameras of the multi-camera array,

generating, from at least the primary image sequence and one or more of the secondary image sequences, a depth map corresponding to each of a plurality of images of the primary image sequence, and

generating, for each image in the plurality of images of the primary image sequence, based on at least the depth map and the corresponding image of the primary image sequence, at least two images of a stereoscopic image sequence.

19. The system of claim 18, wherein the operation further comprises:

determining, based on at least the depth maps and the corresponding images of the primary image sequence, a set of objects depicted in the primary image sequence;

associating an object of the determined set of objects with a distinct layer in at least one of the depth maps; and

modifying an appearance of the object in the distinct layer so as to change an appearance of the object in stereoscopic images.

20. The system of claim 18, wherein at least one of the secondary cameras provides a plenoptic lens configured to capture light field data.

21. The system of claim 18, wherein the system further comprises at least one time-of-flight device configured to capture depth information in conjunction with the primary image sequence captured by the primary camera.

22. A multi-camera array, comprising:

a primary camera configured to capture a primary image sequence of a scene; and

one or more secondary cameras each configured to capture a secondary image sequence of the scene, the secondary cameras having a sufficient depth of field to permit depth maps to be generated for images in the primary image sequence,

wherein at least the image sequences captured by the one or more secondary cameras are used to generate depth maps during post-production processing of the primary image sequence, and wherein image sequences captured by the primary camera and the depth maps are used to generate stereoscopic image sequences.

23. The multi-camera array of claim 22, wherein at least one of the secondary cameras provides a plenoptic lens configured to capture light field data.

24. The multi-camera array of claim 22, wherein the secondary cameras are generator locked with the primary camera such that the secondary cameras operate synchronously with the primary camera, and wherein time codes of the secondary cameras are synchronized with time codes of the primary camera.

25. The multi-camera array of claim 22, wherein the system further comprises at least one time-of-flight device configured to capture depth information.