System and method for minimal iteration workflow for image sequence depth enhancement

• US 8,385,684 B2
• Filed: 02/17/2011
• Issued: 02/26/2013
• Est. Priority Date: 05/04/2001
• Status: Expired due to Fees

• Current Assignee: Legend3D, Inc.
• Original Assignee: Legend3D, Inc.
• Inventors: Sandrew, Jared, Baldridge, Tony, Sandrew, Barry
• Primary Examiner(s): Kassa, Yosef
• Application Number:
  US13/029,862
• Publication Number:
  US 20120120054A1
• Time in Patent Office: 740 Days
• Field of Search: 382/212, 382/213, 382/274, 382/275, 382/282, 382/283, 358/517, 358/537, 358/538
• US Class Current: 382/284
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/266 by scanning a film

Abstract

Movies to be colorized/depth enhanced (2D->3D) are broken into backgrounds/sets or motion/onscreen-action. Background and motion elements are combined into composite frame which becomes a visual reference database that includes data for all frame offsets used later for the computer controlled application of masks within a sequence of frames. Masks are applied to subsequent frames of motion objects based on various differentiating image processing methods, including automated mask fitting/reshaping. Colors/depths are automatically applied with masks throughout a scene from the composite background and to motion objects. Areas never exposed by motion or foreground objects may be partially or fully realistically drawn/rendered/applied to the occluded areas and applied throughout the images to generate artifact-free secondary viewpoints during 2D->3D conversion. Iterative workflow is eliminated for simple artifact correction through real-time manipulation of images to avoid re-rendering of images and associated delays of sending work product to other workgroups for correction.

Claims

1. A system configured to modify a set of time ordered digital images comprising at least one computer configured to:

create at least one mask associated with a region a source image;

associate depth values with said at least one mask of said source image;

render a left viewpoint image of said source image;

generate a left translation map for said left viewpoint image;

render a right viewpoint image of said source image;

generate a right translation map for said right viewpoint image;

modify said at least one mask associated with said left translation map and said right translation map to create at least one modified mask; and

,

update said left viewpoint image and said right viewpoint image based on said left translation map and said right translation map respectively and said at least one modified mask.

2. The system of claim 1 wherein said render said left viewpoint and said render of said right viewpoint by said at least one computer occurs before said update of said left viewpoint image and said right viewpoint image occurs and does not occur after said update said left viewpoint image and said right viewpoint image occurs for said source image.

3. The system of claim 1 wherein said update occurs without another said render of said left viewpoint image and without another said render of said right viewpoint image.

4. The system of claim 1 wherein said left translation map and said right translation map comprises a single relative or absolute pixel-by-pixel horizontal offset or pair of offsets of each pixel in said source image to said respective pixel in said left viewpoint image and said right viewpoint image respectively.

5. The system of claim 1 wherein said left translation map and said right translation map comprise relative or absolute pixel-by-pixel horizontal mappings and are implemented as UV maps or U maps or a single map having a relative offset on a pixel-by-pixel basis that indicates a horizontal left and right offset to apply to each pixel of the source image.

6. The system of claim 1 wherein said at least one computer configured to create said at least one mask is located distally to said at least one computer configured to render said left and right viewpoint images.

7. The system of claim 1 wherein said at least one computer configured to create said at least one mask is configured to be controlled by a first laborer.

8. The system of claim 1 wherein said at least one computer configured to create said at least one mask is located distally to said at least one computer configured to generate said left and right viewpoint images.

9. The system of claim 1 wherein said at least one computer configured to render said left viewpoint image and said right viewpoint image is configured to be controlled by a second laborer.

10. The system of claim 1 wherein said update or said render or said update and said render of said left and right viewpoint images utilizes computer generated elements.

11. The system of claim 1 wherein said render or said update or said render and said update of said left and right viewpoint images utilizes generated missing background information for at least one portion of said source image that does not expose at least one area of another layer.

12. The system of claim 1 wherein said at least one computer configured to render said left and right viewpoint images generates script to allow said at least one computer configured to generate said left viewpoint image and said right viewpoint image to display a graphical user interface that differs from a rendering graphical user interface utilized on said at least one computer configured to render said left and right viewpoint images.

13. A method of modifying a set of time ordered digital images comprising at least one computer comprising:

creating at least one mask associated with a region a source image;

associating depth values with said at least one mask of said source image;

rendering a left viewpoint image of said source image;

generating a left translation map for said left viewpoint image;

rendering a right viewpoint image of said source image;

generating a right translation map for said right viewpoint image;

modifying said at least one mask associated with said left translation map and said right translation map; and

,

updating said left viewpoint image and said right viewpoint image based on said left translation map and said right translation map respectively and said at least one modified mask.

14. The method of claim 13 wherein said updating occurs without another said rendering of said left viewpoint image and without another said render of said right viewpoint image.

15. The method of claim 13 wherein rendering said left and right viewpoint images occurs once before said updating occurs and does not occur after said updating occurs for said source image.

16. The method of claim 13 wherein said generating said left translation map and said right translation map comprises including a single relative or absolute pixel-by-pixel horizontal offset or pair of offsets of each pixel in said source image to said respective pixel in said left viewpoint image and said right viewpoint image respectively.

17. The method of claim 13 wherein said rendering or said updating or both said rendering and said updating of said left and right viewpoint images comprises utilizing generated missing background information for at least one portion of said source image that does not expose at least one area of another layer.

18. The method of claim 13 wherein said rendering or said updating or both said rendering and said updating of said left and right viewpoint images comprises utilizing computer generated elements.

19. The method of claim 13 wherein said left translation map and said right translation map comprise relative or absolute pixel-by-pixel horizontal and are implemented as UV maps or U maps or a single map having a relative offset on a pixel-by-pixel basis that indicates a horizontal left and right offset to apply to each pixel of the source image.

20. The method of claim 13 further comprising generating a script to display a graphical user interface that differs from a rendering graphical user interface utilized in rendering said left and right viewpoint images.