SYSTEM AND METHOD FOR RAPID IMAGE SEQUENCE DEPTH ENHANCEMENT WITH AUGMENTED COMPUTER-GENERATED ELEMENTS

• US 20110164109A1
• Filed: 12/22/2010
• Published: 07/07/2011
• Est. Priority Date: 05/04/2001
• Status: Active Grant

• Current Assignee: Legend3D, Inc.
• Original Assignee: Legend3D, Inc.
• Inventors: Sandrew, Barry, BALDRIDGE, Tony
• Granted Patent: US 8,401,336 B2
• Time in Patent Office: Days
• US Class Current: 348/43
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/266 by scanning a film

Abstract

Motion picture scenes to be colorized/depth enhanced (2D->3D) are broken into separate elements, backgrounds/sets or motion/onscreen-action. Background and motion elements are combined into composite frame which becomes a visual reference database that includes data for all frame offsets used later for the computer controlled application of masks within a sequence of frames. Masks are applied to subsequent frames of motion objects based on various differentiating image processing methods, including automated mask fitting/reshaping. Colors and/or depths are automatically applied to masks throughout a scene from the composite background and to motion objects. Areas never exposed by motion or foreground objects in a series of images may be partially or fully realistically drawn or rendered and applied to the occluded areas of the background and then automatically applied throughout the images to generate of minimal artifact or artifact-free secondary viewpoints when translating foreground objects horizontally during 2D->3D conversion.

Claims

1. A system configured to modify a set of time ordered digital images comprising a computer configured to:

associate a first mask with a motion object in a first image;

copy said first mask to create a second mask associated with a second image;

import a computer-generated element associated with said first image;

obtain a depth data for said computer-generated element for said first image;

move said second mask to a location of said motion object in said second image;

reshape said second mask to fit said motion object in said second image;

import a computer-generated element associated with said second image;

obtain a depth data for said computer-generated element for said second image;

set at least one depth parameter associated with said first mask;

apply said at least one depth parameter to said second image;

generate a first pair of images for left eye viewing and right eye viewing respectively for said first image;

generate a second pair of images for left eye viewing and right eye viewing respectively for said second image;

2. The system of claim 1 wherein said computer is further configured to:

import an alpha mask associated with an object in said first image and said second image and further reshape said first mask and reshape said second mask to fit said alpha mask.

3. The system of claim 1 wherein said computer is further configured to:

import an alpha mask for said computer-generated element in said first image and said second image and further reshape said first mask and reshape said second mask to fit a border of said alpha mask for said computer-generated element.

4. The system of claim 1 wherein said computer is further configured to apply a numeric depth value for said first mask.

5. The system of claim 1 wherein said computer is further configured to apply a three-dimensional geometric depth shape to said first mask and translate, rotate or change a size of said three-dimensional geometric depth shape.

6. The system of claim 1 wherein said computer is further configured to associate a transparency mask with said first mask and assign depth to said transparency mask.

7. The system of claim 1 wherein said computer is further configured to:

display at least one area of a background that represents missing background image data where said motion object and said computer-generated element does not expose said at least one area of said background.

8. The system of claim 1 wherein said computer is further configured to:

generate said missing background image data;

utilize said missing background image data to generate said first pair of images and to generate said second pair of images when either or both said motion object and said computer-generated element is or are translated in a horizontal direction to apply depth for a left or right eye.

9. The system of claim 1 wherein said computer is further configured to perform one or more of the operations:

obtain a camera angle value,

obtain a camera location value,

estimate a camera focal length value,

estimate a camera zoom value

for said at least one image selected from said plurality of images and store said camera angle value, said camera location value, said camera focal length value and said camera zoon value in a visual reference database.

10. A system configured to modify a set of time ordered digital images comprising:

a visual reference database;

a computer coupled with said visual reference database wherein said computer is configured to create a mask based on bimodal thresholding at a mask location;

import a computer-generated element associated with a first image and obtain said computer-generated element mask edge;

adjust a mask shape of said mask by a weighted sliding scale for luminance and pattern wherein said mask shape is adjusted to match a border of an adjacent computer-generated element mask edge; and

,

enter said mask location and said mask shape into said visual reference database coupled with said computer.

11. The system of claim 10 wherein said computer is further configured to:

import an alpha mask for an object in said first image and further reshape said mask to fit said alpha mask.

12. A system configured to modify a set of time ordered digital images comprising a computer configured to:

copy a first mask associated with a first mask location related to a first image to a second mask associated with a second mask location related to a second image;

search via gradient descent within a bounding box around said second mask location in said second image for a pattern of luminance contained within said first mask in said first image or up to any computer-generated element mask edge; and

,

adjust said second mask location in said second image.

13. The system of claim 12 wherein said computer is further configured to:

calculate a weighted index for said first mask in said first image using bilinear interpolation;

create a fit box for said second mask in said second image; and

,

modify a mask shape of said second mask using a gradient descent of minimum errors based on luminance and pattern matching and using any computer-generated element mask edge to define a portion of said mask shape.

14. The system of claim 13 wherein said computer is further configured to:

set a depth parameter associated with said first mask in said first image;

apply said depth parameter to said second mask in said second image.

15. The system of claim 13 wherein said computer is further configured to:

generate a second viewpoint image through use of generated missing background data when said second mask in said second image is translated in a horizontal direction.

16. The system of claim 13 wherein said computer is further configured to:

modify said mask shape using Bezier curve animation with edge detection.

17. The system of claim 13 wherein said computer is further configured to:

modify said mask shape via polygon animation with edge detection.

18. A method for modifying a set of time ordered digital images comprising:

associating a first mask with a motion object in a first image;

copying said first mask to create a second mask associated with a second image;

importing a computer-generated element associated with said first image;

obtaining a depth for said computer-generated element for said first image;

moving said second mask to a location of said motion object in said second image;

reshaping said second mask to fit said motion object in said second image;

importing a computer-generated element associated with said second image;

obtaining a depth for said computer-generated element for said second image;

setting at least one depth parameter associated with said first mask;

applying said at least one depth parameter to said second image;

generating a first pair of images for left eye viewing and right eye viewing respectively for said first image using a computer;

generating a second pair of images for left eye viewing and right eye viewing respectively for said second image using said computer;

19. The method of claim 18 further comprising:

importing an alpha mask associated with an object in said first image and said second image and further reshaping said first mask and reshaping said second mask to fit said alpha mask.

20. The method of claim 18 further comprising:

importing an alpha mask for said computer-generated element in said first image and said second image and further reshaping said first mask and reshaping said second mask to fit a border of said alpha mask for said computer-generated element.