System and method for rapid image sequence depth enhancement with translucent elements

• US 8,897,596 B1
• Filed: 02/06/2012
• Issued: 11/25/2014
• Est. Priority Date: 05/04/2001
• Status: Expired due to Fees

• Current Assignee: Legend3D, Inc.
• Original Assignee: Legend3D, Inc.
• Inventors: Passmore, Charles, Baldridge, Tony, Sandrew, Barry
• Primary Examiner(s): Kassa, Yosef
• Application Number:
  US13/367,316
• Time in Patent Office: 1,023 Days
• Field of Search: 382/275, 382/278, 382/284, 382/291, 358/3.26, 358/3.27
• US Class Current: 382/284
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/266 by scanning a film

Abstract

Motion picture scenes to be colorized/depth enhanced (2D→3D) are broken into separate elements, backgrounds/sets or motion/onscreen-action. Background and motion elements are combined into composite frame which becomes a visual reference database that includes data for all frame offsets used later for the computer controlled application of masks within a sequence of frames. Masks are applied to subsequent frames of motion objects based on various differentiating image processing methods, including automated mask fitting/reshaping. Colors and/or depths are automatically applied to masks throughout a scene from the composite background, translucent, motion objects. Areas never exposed by motion or foreground objects in a series of images may be partially or fully realistically drawn or rendered and applied to the occluded areas of the background and then automatically applied throughout the images to generate of minimal artifact or artifact-free secondary viewpoints when translating foreground objects horizontally during 2D→3D conversion.

Claims

1. A system configured to modify a set of time ordered digital images comprising a computer configured to:

associate a first mask as a first layer in memory of a computer with a translucent object in a first image;

copy said first mask to create a second mask in said memory associated with a second image;

move said second mask to a location of said translucent object in said second image;

automatically reshape said second mask in memory with said computer to fit said translucent object in said second image based on underlying image data in said second image;

accept input for at least one depth parameter and set said at least one depth parameter associated with said first mask said memory;

apply said at least one depth parameter to said second image;

generate a first pair of images for left eye viewing and right eye viewing respectively for said first image through ray tracing two or more layers comprising said first layer wherein said first layer and said two or more layers comprise depth values that may overlap in depth;

generate a second pair of images for left eye viewing and right eye viewing respectively for said second image through said ray tracing.

2. The system of claim 1 wherein said computer is further configured to:

display a false color image that represents said translucent object in said first mask.

3. The system of claim 1 wherein said computer is further configured to:

accept a range of depth for said translucent object based on a luminance value in said first mask or an alpha of said first mask.

4. The system of claim 1 wherein said computer is further configured to:

obtain said at least one depth value based on motion of said translucent object between frames.

5. The system of claim 1 wherein said computer is further configured to:

accept at least one second depth parameter for said second mask;

tween said at least one depth parameter in any images between said first image and said second image.

6. The system of claim 1 wherein said computer is further configured to:

import an alpha mask associated with a second layer having an object in said first image and said second image and further automatically reshape said first mask and automatically reshape said second mask with said computer to fit said alpha mask.

7. The system of claim 1 wherein said computer is further configured to:

import an alpha mask associated with a third layer having a computer-generated element in said first image and said second image and further automatically reshape said first mask and automatically reshape said second mask with said computer to fit a border of said alpha mask for said computer-generated element.

8. The system of claim 1 wherein said computer is further configured to:

alter said at least one depth parameter through use of a blur radius.

9. The system of claim 1 wherein said computer is further configured to:

alter said at least one depth parameter through use of a depth line.

10. The system of claim 1 wherein said computer is further configured to apply a three-dimensional geometric depth shape to said first mask and translate, rotate or change a size of said three-dimensional geometric depth shape.

11. The system of claim 1 wherein said computer is further configured to:

display at least one area of a background that represents missing background image data wherein any layers having a motion object or a computer-generated element does not expose said at least one area of said background.

12. The system of claim 1 wherein said computer is further configured to:

generate said missing background image data;

utilize said missing background image data to generate said first pair of images and to generate said second pair of images.

13. The system of claim 1 wherein said computer is further configured to:

adjusting a shape of said first mask based on an input luminance or alpha value with said computer; and

,

entering said mask location and said mask shape into said memory of said computer.

14. The system of claim 1 wherein said computer is further configured to:

search via gradient descent within a bounding box around said second mask location in said second image for a pattern of luminance alpha value contained within said first mask in said first image to adjust said second mask location in said second image.

15. The system of claim 14 wherein said computer is further configured to:

calculate a weighted index for said first mask in said first image using bilinear interpolation;

create a fit box for said second mask in said second image; and

,

modify a mask shape of said second mask using a gradient descent of minimum errors based on luminance and pattern matching and using any translucent element mask edge to define a portion of said mask shape.

16. A system configured to modify a set of time ordered digital images comprising a computer configured to:

associate a first mask as a first layer in memory of a computer with a translucent object in a first image;

copy said first mask to create a second mask in said memory associated with a second image;

move said second mask to a location of said translucent object in said second image;

accept input for at least one depth parameter and set said at least one depth parameter associated with said first mask said memory;

apply said at least one depth parameter to said second image;

import an alpha mask associated with a second layer having an object in said first image and said second image and further automatically reshape said first mask and automatically reshape said second mask with said computer to fit said alpha mask and based on underlying image data in said second image that borders said second mask;

import an alpha mask associated with a third layer having a computer-generated element in said first image and said second image and further automatically reshape said first mask and automatically reshape said second mask with said computer to fit a border of said alpha mask for said computer-generated element and based on underlying image data in said second image that borders said second mask;

generate a first pair of images for left eye viewing and right eye viewing respectively for said first image through ray tracing two or more layers comprising said first layer wherein said first layer and said two or more layers comprise depth values that may overlap in depth;

generate a second pair of images for left eye viewing and right eye viewing respectively for said second image through said ray tracing.

17. The system of claim 16 wherein said computer is further configured to:

alter said at least one depth parameter through use of a blur radius.

18. The system of claim 16 wherein said computer is further configured to:

alter said at least one depth parameter through use of a depth line.

19. The system of claim 16 wherein said computer is further configured to apply a three-dimensional geometric depth shape to said first mask and translate, rotate or change a size of said three-dimensional geometric depth shape.

20. The system of claim 16 wherein said computer is further configured to:

display at least one area of a background that represents missing background image data wherein any layers having a motion object or a computer-generated element does not expose said at least one area of said background.