Rapid workflow system and method for image sequence depth enhancement

• US 8,953,905 B2
• Filed: 06/07/2012
• Issued: 02/10/2015
• Est. Priority Date: 05/04/2001
• Status: Expired due to Fees

• Current Assignee: Legend3D, Inc.
• Original Assignee: Legend3D, Inc.
• Inventors: Sandrew, Jared, Baldridge, Tony, Lopez, Anthony, Tranquill, Timothy, Cesareo, Craig
• Primary Examiner(s): Kassa, Yosef
• Application Number:
  US13/490,454
• Publication Number:
  US 20120242790A1
• Time in Patent Office: 978 Days
• Field of Search: 382/213, 382/274, 382/275, 382/283, 382/284, 358/517, 358/537, 358/538, 358/452, 358/453
• US Class Current: 382/284
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/266 by scanning a film

Abstract

Movies to be colorized/depth enhanced (2D→3D) are broken into backgrounds/sets or motion/onscreen-action. Background and motion elements are combined into composite frame which becomes a visual reference database that includes data for all frame offsets used later for the computer controlled application of masks within a sequence of frames. Masks are applied to subsequent frames of motion objects based on various differentiating image processing methods, including automated mask fitting/reshaping. Colors/depths are automatically applied with masks throughout a scene from the composite background and to motion objects. Areas never exposed by motion or foreground objects may be partially or fully realistically drawn/rendered/applied to the occluded areas and applied throughout the images to generate artifact-free secondary viewpoints during 2D→3D conversion. Iterative workflow is eliminated for simple artifact correction through real-time manipulation of images to avoid re-rendering of images and associated delays of sending work product to other workgroups for correction.

Claims

1. A system configured to modify a set of time ordered digital images comprising at least one computer configured to:

obtain a source image and at least one mask associated with a region in said source image;

obtain a depth map or translation values associated with said at least one mask of said source image;

obtain a render of a left viewpoint image of said source image created with ray tracing to a left virtual camera;

obtain a render of a right viewpoint image of said source image created with ray tracing to a right virtual camera;

modify said at least one mask or an area next to said at least one mask where missing background information is located or said depth map or said translation values or any combination thereof to create at least one modified mask or modified missing background information or modified depth map or modified translation values or any combination thereof; and

,

update said left viewpoint image and said right viewpoint image based on said at least one modified mask or said modified depth map or said modified translation values or said any combination thereof without another said ray tracing to said left virtual camera and without said ray tracing to said right virtual camera.

2. The system of claim 1 wherein said depth map comprises depth information related to said at least one mask and wherein said depth map is sliced to exclude areas further than a threshold and wherein said at least one mask is reshaped to create said at least one modified depth mask.

3. The system of claim 1 wherein said depth map comprises a distance away from a pair of virtual cameras.

4. The system of claim 1 wherein said translation values comprise a left translation map and a right translation map that each contain pixel-by-pixel horizontal offsets of each pixel in said source image to said respective pixel in said left viewpoint image and said right viewpoint image respectively.

5. The system of claim 1 wherein said left translation map and said right translation map are UV maps or U maps.

6. The system of claim 1 wherein said at least one mask is created on a computer that is located distally to said at least one computer configured to update said left and right viewpoint images.

7. The system of claim 1 wherein said depth map or said translation values are created on a computer that is located distally to said at least one computer configured to update said left and right viewpoint images.

8. The system of claim 7 wherein said at least one computer configured to render said left viewpoint image and said right viewpoint image is configured to be controlled by a second laborer.

9. The system of claim 1 wherein said update or said render or said update and said render of said left and right viewpoint images utilizes computer generated elements.

10. The system of claim 1 wherein said update of said left and right viewpoint images utilizes generated missing background information for at least one portion of said source image that does not expose at least one area of another layer.

11. The system of claim 1 wherein said update of said left and right viewpoint images utilizes generated missing background information or gap fill data for at least one portion of said source image that does not expose at least one area of another layer wherein said generated missing background information or said gap fill data is applied to areas detected in said translation values that indicate an over threshold left or right shift of image data and wherein said areas are optionally blurred, optionally clamped and optionally dilated.

12. The system of claim 1 wherein said update of said left and right viewpoint images utilizes gap fill data for at least one portion of said source image that does not expose at least one area of another layer wherein said gap fill data comprises data that matches horizontal or vertical data near said missing background information in said source image.

13. The system of claim 1 wherein said update of said left and right viewpoint images utilizes gap fill data for at least one portion of said source image that does not expose at least one area of another layer wherein said gap fill data comprises blurred data from said source image.

14. The system of claim 1 wherein update of said left and right viewpoint images utilizes gap fill data for at least one portion of said source image that does not expose at least one area of another layer wherein said gap fill data comprises film grain from a region of said source image.

15. The system of claim 1 wherein said update of said left and right viewpoint images utilizes gap fill data for at least one portion of said source image that does not expose at least one area of another layer wherein said gap fill data comprises blurred data from said source image and film grain from a region of said source image.

16. The system of claim 1 wherein said update of said left and right viewpoint images utilizes gap fill data for at least one portion of said source image that does not expose at least one area of another layer wherein said gap fill data comprises film grain from a region of said source image that is applied as a function of underlying luminance.

17. The system of claim 1 wherein said update of said left and right viewpoint images utilizes modified translation values that are created by alteration of at least one translation value in a UV map or U map.

18. The system of claim 1 wherein said update of said left and right viewpoint images utilizes a modified depth map that is created by alteration of at least one depth value in said depth map.

19. The system of claim 1 wherein said update of said left and right viewpoint images utilizes a modified depth map that is created through displacement of pixels associated with said at least one mask left and right respectively by a numerical value shift in said depth map.

20. The system of claim 1 wherein said computer is further configured to display a gap with a color based on a function of the size of said gap.