Minimal artifact image sequence depth enhancement system and method

• US 8,073,247 B1
• Filed: 11/01/2010
• Issued: 12/06/2011
• Est. Priority Date: 05/04/2001
• Status: Expired due to Fees

• Current Assignee: Legend3D, Inc.
• Original Assignee: Legend3D, Inc.
• Inventors: Sandrew, Barry, Baldridge, Tony
• Primary Examiner(s): Kassa, Yosef
• Application Number:
  US12/916,636
• Time in Patent Office: 400 Days
• Field of Search: 382/164, 382/167, 382/260, 382/274, 382/275, 382/283
• US Class Current: 382/164
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: H04N 9/43 Conversion of monochrome picture signals to colour picture signals for colour picture display

Abstract

Motion picture scenes to be colorized/depth enhanced (2D->3D) are broken into separate elements, backgrounds/sets or motion/onscreen-action. Background and motion elements are combined into composite frame which becomes a visual reference database that includes data for all frame offsets used later for the computer controlled application of masks within a sequence of frames. Masks are applied to subsequent frames of motion objects based on various differentiating image processing methods, including automated mask fitting/reshaping. Colors and/or depths are automatically applied to masks throughout a scene from the composite background and to motion objects. Areas never exposed by motion or foreground objects in a series of images may be partially or fully realistically drawn or rendered and applied to the occluded areas of the background and then automatically applied throughout the images to generate of minimal artifact or artifact-free secondary viewpoints when translating foreground objects horizontally during 2D->3D conversion.

Claims

1. A method of modifying a set of time ordered digital images comprising:

associating a first mask with a motion object in a first image;

copying said first mask to create a second mask associated with a second image;

moving said second mask to location of said motion object in said second image; and

,

reshaping said second mask automatically to fit said motion object in said second image using a computer.

2. The method of claim 1 further comprising:

setting at least one color parameter associated with said first mask; and

,

applying said at least one color parameter to said second image.

3. A method of modifying a set of time ordered digital images comprising:

creating a mask based on bimodal thresholding at a mask location;

adjusting a mask shape of said mask by a weighted sliding scale for luminance and pattern using a computer; and

,

entering said mask location and said mask shape into a visual reference database coupled with said computer.

4. A method of modifying a set of time ordered digital images comprising:

copying a first mask associated with a first mask location related to a first image to a second mask associated with a second mask location related to a second image;

searching via gradient descent within a bounding box around said second mask location in said second image for a pattern of luminance contained within said first mask in said first image; and

,

adjusting said second mask location in said second image using a computer.

5. The method of claim 4 further comprising:

calculating a weighted index for said first mask in said first image using bilinear interpolation;

creating a fit box for said second mask in said second image; and

,

modifying a mask shape of said second mask using a gradient descent of minimum errors based on luminance and pattern matching using a computer.

6. The method of claim 5 further comprising:

setting a depth parameter associated with said first mask in said first image;

applying said depth parameter to said second mask in said second image.

7. The method of claim 6 wherein modifying the mask shape of said second mask further comprises:

generating a second viewpoint image using generated missing background data when translating said second mask in said second image in a horizontal direction.

8. A method of modifying a set of time ordered digital images comprising:

calculating a weighted index for a first mask in a first image using bilinear interpolation;

creating a fit box for a second mask in a second image; and

,

modifying a mask shape of said second mask using a gradient descent of minimum errors based on luminance and pattern matching using a computer.

9. The method of claim 8 wherein modifying the mask shape of said second mask further comprises:

setting a depth parameter associated with said first mask in said first image;

applying said depth parameter to said second mask in said second image.

10. The method of claim 9 wherein modifying the mask shape of said second mask further comprises:

generating a second viewpoint image using generated missing background data when translating said second mask in said second image in a horizontal direction.

11. The method of claim 8 wherein modifying the mask shape of said second mask further comprises:

modifying said mask shape using Bezier curve animation with edge detection.

12. The method of claim 8 wherein modifying the mask shape of said second mask further comprises:

modifying said mask shape via polygon animation with edge detection.

13. A system configured to modify a set of time ordered digital images comprising a computer configured to:

associate a first mask with a motion object in a first image;

copy said first mask to create a second mask associated with a second image;

move said second mask to location of said motion object in said second image; and

,

reshape said second mask to fit said motion object in said second image.

14. The system of claim 13 wherein said computer is further configured to:

set at least one color parameter associated with said first mask; and

,

apply said at least one color parameter to said second image.

15. A system configured to modify a set of time ordered digital images comprising:

a visual reference database;

a computer coupled with said visual reference database wherein said computer is configured to

create a mask based on bimodal thresholding at a mask location;

adjust a mask shape of said mask by a weighted sliding scale for luminance and pattern; and

,

enter said mask location and said mask shape into said visual reference database coupled with said computer.

16. A system configured to modify a set of time ordered digital images comprising a computer configured to:

copy a first mask associated with a first mask location related to a first image to a second mask associated with a second mask location related to a second image;

search via gradient descent within a bounding box around said second mask location in said second image for a pattern of luminance contained within said first mask in said first image; and

,

adjust said second mask location in said second image.

17. The system of claim 16 wherein said computer is further configured to:

calculate a weighted index for said first mask in said first image using bilinear interpolation;

create a fit box for said second mask in said second image; and

,

modify a mask shape of said second mask using a gradient descent of minimum errors based on luminance and pattern matching.

18. The system of claim 17 wherein said computer is further configured to:

set a depth parameter associated with said first mask in said first image;

apply said depth parameter to said second mask in said second image.

19. The system of claim 18 wherein said computer is further configured to:

generate a second viewpoint image through use of generated missing background data when said second mask in said second image is translated in a horizontal direction.

20. A system configured to modify a set of time ordered digital images comprising a computer configured to:

calculate a weighted index for a first mask in a first image using bilinear interpolation;

create a fit box for a second mask in a second image; and

,

modify a mask shape of said second mask using a gradient descent of minimum errors based on luminance and pattern matching.

21. The system of claim 20 wherein said computer is further configured to:

set a depth parameter associated with said first mask in said first image;

apply said depth parameter to said second mask in said second image.

22. The system of claim 21 wherein said computer is further configured to:

generate a second viewpoint image through use of generated missing background data when said second mask in said second image is translated in a horizontal direction.

23. The system of claim 20 wherein said computer is further configured to:

modify said mask shape using Bezier curve animation with edge detection.

24. The system of claim 20 wherein said computer is further configured to:

modify said mask shape via polygon animation with edge detection.