Minimal artifact image sequence depth enhancement system and method

• US 8,078,006 B1
• Filed: 11/01/2010
• Issued: 12/13/2011
• Est. Priority Date: 05/04/2001
• Status: Expired due to Fees

• Current Assignee: Legend3D, Inc.
• Original Assignee: Legend3D, Inc.
• Inventors: Sandrew, Barry, Baldridge, Tony
• Primary Examiner(s): Kassa, Yosef
• Application Number:
  US12/916,637
• Time in Patent Office: 407 Days
• Field of Search: 382/284, 382/291, 382/283, 382/294, 358/540, 358/450
• US Class Current: 382/284
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: H04N 9/43 Conversion of monochrome picture signals to colour picture signals for colour picture display

Abstract

Motion picture scenes to be colorized/depth enhanced (2D→3D) are broken into separate elements, backgrounds/sets or motion/onscreen-action. Background and motion elements are combined into composite frame which becomes a visual reference database that includes data for all frame offsets used later for the computer controlled application of masks within a sequence of frames. Masks are applied to subsequent frames of motion objects based on various differentiating image processing methods, including automated mask fitting/reshaping. Colors and/or depths are automatically applied to masks throughout a scene from the composite background and to motion objects. Areas never exposed by motion or foreground objects in a series of images may be partially or fully realistically drawn or rendered and applied to the occluded areas of the background and then automatically applied throughout the images to generate of minimal artifact or artifact-free secondary viewpoints when translating foreground objects horizontally during 2D→3D conversion.

Claims

1. A method of creating a visual reference database comprising:

associating a plurality of images comprising common background elements;

obtaining a composite background comprising said plurality of images wherein said composite background excludes a set of motion objects that move in said plurality of images in relation to said common background elements;

associating a first mask with a motion object selected from said set of motion objects in a first image selected from said plurality of images;

accessing a mask value associated with said first mask in said first image wherein said mask value is stored in a visual reference database;

accessing a camera value associated with said first image wherein said camera value is stored in said visual reference database;

copying said mask value associated with said first mask to create a second mask value associated with a second mask and further associated with a second image selected from said plurality of images;

moving said second mask value to a location of said motion object in said second image;

reshaping said second mask value automatically to fit said motion object in said second image based on said camera value using a computer; and

,

storing said second mask value in said visual reference database.

2. The method of claim 1 further comprising:

displaying at least one area that represents missing background image data in said composite background where said one or more motion objects does not expose said at least one area of said common background elements.

3. The method of claim 2 further comprising:

generating said missing background image data comprising said at least one color parameter or at least one depth parameter or, said at least one color parameter and said at least one depth parameter, for at least a portion of said at least one area where said one or more motion objects does not expose said at least one area of said common background elements.

4. The method of claim 3 further comprising:

generating a second viewpoint image using said missing background data when translating said one or more motion objects in a horizontal direction.

5. The method of claim 1 further comprising:

accessing a descriptive text value associated with said first image in said visual reference database.

6. The method of claim 1 wherein said accessing a mask value comprises one or more of the following steps:

accessing a mask location associated with said first image in said visual reference database;

accessing a mask shape associated with said first image in said visual reference database.

7. The method of claim 1 wherein said accessing a camera value comprises one or more of the following steps:

accessing a camera angle value associated with said first image in said visual reference database;

accessing a camera location value associated with said first image in said visual reference database;

accessing a camera focal length value associated with said first image in said visual reference database;

accessing a zoom value associated with said first image in said visual reference database.

8. A system configured to modify a set of time ordered digital images comprising

a visual reference database;

a computer configured to

associate a plurality of images comprising common background elements;

obtain a composite background comprising said plurality of images wherein said composite background excludes a set of motion objects that move in said plurality of images in relation to said common background elements;

associate a first mask with a motion object selected from said set of motion objects in a first image selected from said plurality of images;

access a mask value associated with said first mask in said first image wherein said mask value is stored in said visual reference database;

access a camera value associated with said first image wherein said camera value is stored in said visual reference database;

copy said mask value associated with said first mask to create a second mask value associated with a second mask and further associated with a second image selected from said plurality of images;

move said second mask value to a location of said motion object in said second image;

reshape said second mask value to fit said motion object in said second image based on said camera value; and

,

store said second mask value in said visual reference database.

9. The system of claim 8 wherein said computer is further configured to:

display at least one area that represents missing background image data in said composite background where said one or more motion objects does not expose said at least one area of said common background elements.

10. The system of claim 9 wherein said computer is further configured to:

generate said missing background image data comprising said at least one color parameter or at least one depth parameter or, said at least one color parameter and said at least one depth parameter, for at least a portion of said at least one area where said one or more motion objects does not expose said at least one area of said common background elements.

11. The system of claim 10 wherein said computer is further configured to:

generate a second viewpoint image through use of said missing background data when said one or more motion objects is translated in a horizontal direction.

12. The system of claim 8 wherein said computer is further configured to:

access a descriptive text value associated with said first image in said visual reference database.

13. The system of claim 8 wherein said computer is further configured to perform one or more of the following operations:

access a mask location associated with said first image in said visual reference database;

access a mask shape associated with said first image in said visual reference database.

14. The system of claim 8 wherein said computer is further configured to perform one or more of the following operations:

access a camera angle value associated with said first image in said visual reference database;

access a camera location value associated with said first image in said visual reference database;

access a camera focal length value associated with said first image in said visual reference database;

access a zoom value associated with said first image in said visual reference database.