Minimal artifact image sequence depth enhancement system and method

• US 8,396,328 B2
• Filed: 10/27/2010
• Issued: 03/12/2013
• Est. Priority Date: 05/04/2001
• Status: Expired due to Term

• Current Assignee: Legend3D, Inc.
• Original Assignee: Legend3D, Inc.
• Inventors: Sandrew, Barry, Baldridge, Tony
• Primary Examiner(s): Kassa, Yosef
• Application Number:
  US12/913,614
• Publication Number:
  US 20120063681A1
• Time in Patent Office: 867 Days
• Field of Search: 382/164, 382/165, 382/282, 382/278, 382/286, 358/515, 358/517, 358/518
• US Class Current: 382/286
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/266 by scanning a film
• CPC Class Codes: H04N 9/43 Conversion of monochrome picture signals to colour picture signals for colour picture display

Abstract

Motion picture scenes to be colorized/depth enhanced (2D->3D) are broken into separate elements, backgrounds/sets or motion/onscreen-action. Background and motion elements are combined into composite frame which becomes a visual reference database that includes data for all frame offsets used later for the computer controlled application of masks within a sequence of frames. Masks are applied to subsequent frames of motion objects based on various differentiating image processing methods, including automated mask fitting/reshaping. Colors and/or depths are automatically applied to masks throughout a scene from the composite background and to motion objects. Areas never exposed by motion or foreground objects in a series of images may be partially or fully realistically drawn or rendered and applied to the occluded areas of the background and then automatically applied throughout the images to generate of minimal artifact or artifact-free secondary viewpoints when translating foreground objects horizontally during 2D->3D conversion.

Claims

1. A method of modifying a set of time ordered digital images comprising:

associating a plurality of images comprising common background elements;

obtaining a composite background comprising said plurality of images wherein said composite background excludes a set of motion objects that move in said plurality of images in relation to said common background elements;

accepting at least one color parameter associated with a region within said composite background;

applying said at least one color parameter automatically to at least one image selected from said plurality of images using a computer.

2. The method of claim 1 further comprising:

displaying at least one area that represents missing background image data in said composite background where said one or more motion objects does not expose said at least one area of said common background elements.

3. The method of claim 2 further comprising:

generating said missing background image data comprising said at least one color parameter or at least one depth parameter or, said at least one color parameter and said at least one depth parameter for at least a portion of said at least one area where said one or more motion objects does not expose said at least one area of said common background elements.

4. The method of claim 3 further comprising:

generating a second viewpoint image using said missing background data when translating said one or more motion objects in a horizontal direction.

5. The method of claim 1 wherein said obtaining said composite background further comprises one or more of the following steps:

obtaining a camera angle value for said at least one image selected from said plurality of images,

obtaining a camera location value for said at least one image selected from said plurality of images,

estimating a camera focal length value for said at least one image,

estimating a zoom value for said at least one image,

blending said plurality of images together,

separating said plurality of images at scene changes.

6. A method of modifying a set of time ordered digital images comprising:

associating a plurality of images comprising common background elements;

obtaining a composite background comprising said plurality of images wherein said composite background excludes a set of motion objects that move in said plurality of images in relation to said common background elements;

associating a first mask with a motion object selected from said set of motion object in a first image selected from said plurality of images;

copying said first mask to create a second mask associated with a second image selected from said plurality of images;

moving said second mask to a location of said motion object in said second image based on underlying image data in said second image;

reshaping said second mask automatically to fit said motion object in said second image based on said underlying image data in said second image using a computer.

7. The method of claim 6 further comprising:

displaying at least one area that represents missing background image data in said composite background where said one or more motion objects does not expose said at least one area of said common background elements.

8. The method of claim 7 further comprising:

generating said missing background image data comprising said at least one color parameter or at least one depth parameter or, said at least one color parameter and said at least one depth parameter, for at least a portion of said at least one area where said one or more motion objects does not expose said at least one area of said common background elements.

9. The method of claim 8 further comprising:

generating a second viewpoint image using said missing background data when translating said one or more motion objects in a horizontal direction.

10. The method of claim 6 further comprising:

accepting at least one color parameter associated with a region within said composite background;

applying said at least one color parameter to at least one non-composited image selected from said plurality of images.

11. The method of claim 6 further comprising:

accepting at least one motion object color parameter associated with said first mask;

applying said at least one motion object color parameter to said second image.

12. The method of claim 6 further comprising:

accepting at least one depth parameter associated with a region within said composite background;

applying said at least one depth parameter to at least one non-composited image selected from said plurality of images.

13. The method of claim 6 further comprising:

accepting at least one motion object depth parameter associated with said first mask;

applying said at least one motion object depth parameter to said second image.

14. A system configured to modify a set of time ordered digital images comprising a computer configured to:

associate a plurality of images comprising common background elements;

obtain a composite background comprising said plurality of images wherein said composite background excludes a set of motion objects that move in said plurality of images in relation to said common background elements;

accept at least one color parameter associated with a region within said composite background;

apply said at least one color parameter to at least one image selected from said plurality of images.

15. The system of claim 14 wherein said computer is further configured to:

display at least one area that represents missing background image data in said composite background where said one or more motion objects does not expose said at least one area of said common background elements.

16. The system of claim 15 wherein said computer is further configured to:

generate said missing background image data comprising said at least one color parameter or at least one depth parameter or, said at least one color parameter and said at least one depth parameter for at least a portion of said at least one area where said one or more motion objects does not expose said at least one area of said common background elements.

17. The system of claim 16 wherein said computer is further configured to:

generate a second viewpoint image through use of said missing background data when said one or more motion objects is translated in a horizontal direction.

18. The system of claim 14 wherein said computer is configured to perform one or more of the following operations:

obtain a camera angle value for said at least one image selected from said plurality of images,

obtain a camera location value for said at least one image selected from said plurality of images,

estimate a camera focal length value for said at least one image,

estimate a zoom value for said at least one image,

blend said plurality of images together,

separate said plurality of images at scene changes.

19. An system configured to modify a set of time ordered digital images comprising a computer configured to:

associate a plurality of images comprising common background elements;

obtain a composite background comprising said plurality of images wherein said composite background excludes a set of motion objects that move in said plurality of images in relation to said common background elements;

associate a first mask with a motion object selected from said set of motion object in a first image selected from said plurality of images;

copy said first mask to create a second mask associated with a second image selected from said plurality of images;

move said second mask to a location of said motion object in said second image based on underlying image data in said second image;

reshape said second mask automatically to fit said motion object in said second image based on said underlying image data in said second image.

20. The system of claim 19 wherein said computer is further configured to:

display at least one area that represents missing background image data in said composite background where said one or more motion objects does not expose said at least one area of said common background elements.

21. The system of claim 20 wherein said computer is further configured to:

generate said missing background image data comprising said at least one color parameter or at least one depth parameter or, said at least one color parameter and said at least one depth parameter, for at least a portion of said at least one area where said one or more motion objects does not expose said at least one area of said common background elements.

22. The system of claim 21 wherein said computer is further configured to:

generate a second viewpoint image through use of said missing background data when said one or more motion objects is translated in a horizontal direction.

23. The system of claim 19 wherein said computer is further configured to:

accept at least one color parameter associated with a region within said composite background;

apply said at least one color parameter to at least one non-composited image selected from said plurality of images.

24. The system of claim 19 wherein said computer is further configured to:

accept at least one motion object color parameter associated with said first mask;

apply said at least one motion object color parameter to said second image.

25. The system of claim 19 wherein said computer is further configured to:

accept at least one depth parameter associated with a region within said composite background;

apply said at least one depth parameter to at least one non-composited image selected from said plurality of images.

26. The system of claim 19 wherein said computer is further configured to:

accept at least one motion object depth parameter associated with said first mask;

apply said at least one motion object depth parameter to said second image.