Image sequence enhancement and motion picture project management system and method

• US 9,615,082 B2
• Filed: 03/14/2016
• Issued: 04/04/2017
• Est. Priority Date: 05/04/2001
• Status: Active Grant

• Current Assignee: Legend3D, Inc.
• Original Assignee: Legend3D, Inc.
• Inventors: Sandrew, Barry, Lopez, Anthony, Sandrew, Jared, Baldridge, Tony
• Primary Examiner(s): JONES, HEATHER RAE
• Application Number:
  US15/069,827
• Publication Number:
  US 20160198142A1
• Time in Patent Office: 386 Days
• Field of Search: None
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/261 with monoscopic-to-stereoscopic image conversion
• CPC Class Codes: H04N 13/266 by scanning a film

Abstract

The system enables conversion of black and white images to color images and/or two-dimensional images into three-dimensional images based on adding color and/or depth to images using masks for regions in the images, as well as reshaping of masks to cover objects that have moved and changed shape as the objects move in a sequence of images. Also, includes motion picture project management system for reviewers, coordinators and artists. Artists utilize image analysis and image enhancement and computer graphics processing for example to convert two-dimensional images into three-dimensional images or otherwise create or alter motion pictures. Enables the efficient management of projects related to motion pictures to enable enterprises to manage assets, control costs, predict budgets and profit margins, reduce archival storage and otherwise provide displays tailored to specific roles to increase worker efficiency.

Claims

1. A computer-implemented method of modifying a set of time ordered digital images using a computer having memory, comprising executing on said computer the steps of:

associating a first mask with a motion object in a first image at a first mask location;

copying said first mask to create a second mask associated with a second image;

moving said second mask to a second mask location associated with said motion object in said second image; and

,

reshaping said second mask to fit said motion object in said second image using said computer to process said set of time ordered digital images comprising at least said first image and said second image based on underlying image data in said memory of said computer that corresponds to said second image.

2. The method of claim 1 further comprising:

accepting user input of at least one threshold associated with said underlying image data;

creating said first mask based on said at least one threshold associated with said underlying image data at said first mask location; and

,

wherein said associating said first mask with said motion object comprises storing said first mask in said memory of said computer.

3. The method of claim 1 further comprising:

moving said second mask based on said underlying image data in said second image.

4. The method of claim 1 further comprising:

reshaping said second mask by searching near said second mask location in said second image based on said underlying image data contained within said first mask in said first image.

5. The method of claim 1 further comprising:

tracking underlying image data in said second image that is contained within said first mask in said first image before accepting input for manually adjusting, or before said automatically reshaping, of a mask shape of said second mask in said second image.

6. The method of claim 1 further comprising:

reshaping said second mask by searching via gradient descent within a bounding box around said second mask location in said second image based on said underlying image data contained within said first mask in said first image.

7. The method of claim 6 further comprising:

creating a fit box for said second mask in said second image; and

,

modifying a mask shape of said second mask using a gradient descent of minimum errors within said fit box based on said underlying image data.

8. The method of claim 7 further comprising modifying said mask shape using Bezier curve animation with edge detection.

9. The method of claim 7 further comprising modifying said mask shape via polygon animation with edge detection.

10. The method of claim 7 further comprising:

calculating a weighted index associated with said first mask in said first image using bilinear interpolation; and

,

calculating said minimum errors based on said weighted index associated with said second mask and based on said underlying image data.

11. The method of claim 1 wherein said moving or reshaping or both said moving and said reshaping of said second mask comprises changing a location of at least one pixel of said second mask or one or more vertices or control points of said second mask.

12. The method of claim 11 wherein said reshaping of said second mask further comprises modifying a mask shape of said second mask based on Bezier curve animation with edge detection.

13. The method of claim 11 wherein said reshaping of said second mask further comprises modifying a mask shape of said second mask based on polygon animation with edge detection.

14. The method of claim 1 further comprising:

accepting at least one depth parameter associated with said first mask in said first image;

applying said at least one depth parameter to said first image at said first mask location.

15. The method of claim 14 further comprising:

generating at least one translation map associated with said first image;

modifying said first mask associated with said at least one translation map in creating a modified mask; and

,

updating a left viewpoint image or a right viewpoint image or both based on said at least one translation map and said modified mask.

16. The method of claim 1 further comprising:

accepting user input to manually adjust a mask shape of said second mask.

17. The method of claim 1 further comprising:

importing a first alpha mask for a computer-generated element in said first image and importing a second alpha mask for said second image and further reshaping said first mask and reshaping said second mask to fit a first and second border of said first alpha mask and said second alpha mask respectively associated with said computer-generated element.

18. The method of claim 1 further comprising calculating an amount of disk space that may be utilized is required to archive a movie conversion project and signifying at least one asset that is rebuildable from other assets to avoid archival of said at least one asset.

19. The method of claim 1 wherein said underlying image data comprises a feature or an image feature or an image characteristic in said memory of said computer that is associated with one or more pixels in said second image.

20. The method of claim 1 further comprising:

utilizing said underlying image data in said first mask in determining at least one depth parameter associated with said first image; and

,

applying said at least one depth parameter to said first mask in said first image and creating a depth selection mask.

21. The method of claim 20 further comprising:

generating at least one translation map associated with said first image;

modifying said first mask associated with said at least one translation map in creating a modified mask; and

,

updating a left viewpoint image or a right viewpoint image or both based on said at least one translation map and said modified mask.

22. The method of claim 1 further comprising:

copying said first image to said memory of said computer;

creating a depth selection mask comprising said associating said first mask with said motion object, wherein said first mask is associated with at least one characteristic of the first image comprising gray scale or color, and associated with at least a portion of the first image, including

selecting a set of image features comprising said gray scale or said color, and

determining real values of said underlying image data for the set of image features associated with a plurality of pixel locations in a portion of the first image;

determining a vector associated with each of the plurality of pixel locations of the portion of the first image;

applying said depth selection mask to the vector associated with each of the plurality of pixel locations of the portion of the first image, to create a weighted displacement mask having a displacement vector associated with the plurality of pixel locations of the portion of the first image, the displacement vector having a magnitude and direction;

displacing each of the plurality of pixel locations of the portion of the first image in accordance with the magnitude of the displacement vector associated with each respective pixel to create a second viewpoint image; and

creating a stereo pair of images comprising said first image said second viewpoint image.

23. The method of claim 22 further comprising:

generating at least one translation map associated with said first image;

modifying said first mask associated with said at least one translation map in creating a modified mask; and

,

updating said first image or a left viewpoint image or said right viewpoint image or both based on said at least one translation map and said modified mask.

24. The method of claim 1 further comprising:

accessing a database coupled with said computer, wherein said database comprises

a shot table which comprises

a shot identifier associated with a plurality of images that are ordered in time and that each make up at least one shot, and

a starting frame value and an ending frame value associated with each of said at least one shot,

wherein said plurality of images are associated with a motion picture and

wherein said database further comprises metadata associated with said at least one shot or associated with regions within said plurality of images in said at least one shot, or both;

presenting a grouping tool interface coupled with said computer and said database;

presenting user interface elements;

accepting input of said metadata and accepting a selection of one or more shots from said at least one shot associated with said metadata via said user interface elements;

storing said metadata associated with said selection of said one or more shots in said shot table;

accepting a selection of metadata to obtain selected metadata to search said at least one shot;

querying said shot table with said selected metadata associated with said at least one shot or said regions within said plurality of images in said at least one shot or said both; and

,

displaying a list of shots having said selected metadata, wherein said list of shots includes at least one shot that is non-sequential in time in said motion picture with respect to another shot in said list of shots.

25. The method of claim 1 further comprising:

associating a plurality of images comprising common background elements;

obtaining a composite background comprising said plurality of images wherein said composite background excludes a set of motion objects that move in said plurality of images in relation to said common background elements;

accepting at least one color parameter associated with a region within said composite background; and

,

applying said at least one color parameter automatically to at least one image selected from said plurality of images using said computer wherein said at least one color parameter comprises one or more of hue or saturation or luminance.

26. The method of claim 25 further comprising:

displaying at least one area that represents missing background image data in said composite background where said one or more motion objects do not expose said at least one area of said common background elements.

27. The method of claim 25 further comprising limiting a range of said saturation based on a luminance input value or an underlying gray scale density in said region.

28. The method of claim 25 further comprising setting at least one color parameter associated with said region wherein said setting comprises setting at least either said saturation or said luminance within said region based on a luminance input value or an underlying gray scale density to automatically compensate for shadows and changes in luminance in said plurality of images.

29. The method of claim 1 further comprising:

creating at least one mask associated with a region of a source image;

associating depth values with said at least one mask of said source image;

rendering a left viewpoint image of said source image;

generating a left translation map corresponding to said left viewpoint image;

wherein said left translation map comprises horizontal directional offsets of one or more pixels in said left viewpoint image;

rendering a right viewpoint image of said source image;

generating a right translation map corresponding to said right viewpoint image;

wherein said right translation map comprises horizontal directional offsets of one or more pixels in said right viewpoint image;

modifying said at least one mask associated with said left translation map and said right translation map and regenerating said horizontal direction offsets in said left translation map and said right translation map; and

,

updating said left viewpoint image and said right viewpoint image based on said left translation map and said right translation map respectively and said at least one modified mask.

30. The method of claim 29 wherein said generating said left translation map and said generating said right translation map comprise including a single relative or absolute pixel-by-pixel horizontal offset or pair of offsets of each pixel in said source image to said respective pixel in said left viewpoint image and said right viewpoint image respectively.

31. The method of claim 29 wherein said left translation map and said right translation map comprise relative or absolute pixel-by-pixel horizontal mappings and are implemented as UV maps or U maps or a single map having a relative offset on a pixel-by-pixel basis that indicates a horizontal left or right offset respectively to apply to each pixel of the source image or a disparity map.

32. The method of claim 1 further comprising setting at least one color parameter associated with said first mask wherein said setting comprises limiting a range of a saturation value based on a luminance input value or an underlying gray scale density in said first mask.

33. The method of claim 1 further comprising setting at least one color parameter associated with said first mask wherein said setting comprises setting at least a saturation or a luminance value within said first mask based on a luminance input value or an underlying gray scale density to automatically compensate for shadows and changes in luminance in said underlying image data that occur between said first mask and said second mask.

34. A computer-implemented method of modifying a set of time ordered digital images using a computer having memory, comprising executing on said computer the steps of:

associating a first mask with a motion object in a first image at a first mask location;

copying said first mask to create a second mask associated with a second image;

moving said second mask to a second mask location associated with said motion object in said second image based on underlying image data in said second image; and

,

reshaping said second mask to fit said motion object in said second image using said computer to process said set of time ordered digital images comprising at least said first image and said second image based on said underlying image data in said memory of said computer that corresponds to said second image by searching near said second mask location in said second image based on said underlying image data contained within said memory of said computer that corresponds to said first mask in said first image.

35. The method of claim 34 further comprising:

accepting at least one depth parameter associated with said first mask in said first image; and

,

applying said at least one depth parameter to said first mask in said first image at said first mask location.

36. The method of claim 35 further comprising:

generating at least one translation map associated with said first image;

modifying said first mask associated with said at least one translation map in creating a modified mask; and

,

updating a left viewpoint image or a right viewpoint image or both based on said at least one translation map and said modified mask.

37. The method of claim 34 further comprising:

utilizing said underlying image data in said first mask in determining at least one depth parameter associated with said first image; and

,

applying said at least one depth parameter to said first mask in said first image.

38. The method of claim 37 further comprising:

generating at least one translation map associated with said first image;

modifying said first mask associated with said at least one translation map in creating a modified mask; and

,

updating a left viewpoint image or a right viewpoint image or both based on said at least one translation map and said modified mask.

39. The method of claim 34 further comprising:

copying said first image to said memory of said computer;

creating a depth selection mask comprising said associating said first mask with said motion object, wherein said first mask is associated with at least one characteristic of the first image comprising gray scale or color, and associated with at least a portion of the first image, including

selecting a set of image features comprising said gray scale or said color, and

determining real values of said underlying image data for the set of image features associated with a plurality of pixel locations in a portion of the first image;

determining a vector associated with each of the plurality of pixel locations of the portion of the first image;

applying said depth selection mask to the vector associated with each of the plurality of pixel locations of the portion of the first image, to create a weighted displacement mask having a displacement vector associated with the plurality of pixel locations of the portion of the first image, the displacement vector having a magnitude and direction;

displacing each of the plurality of pixel locations of the portion of the first image in accordance with the magnitude of the displacement vector associated with each respective pixel to create a second viewpoint image; and

creating a stereo pair of images comprising said first image said second viewpoint image.

40. The method of claim 39 further comprising:

generating at least one translation map associated with said first image;

modifying said first mask associated with said at least one translation map in creating a modified mask; and

,

updating said image or a left viewpoint image or said right viewpoint image or both based on said at least one translation map and said modified mask.

41. A system for modifying a set of time ordered digital images comprising:

means for associating a first mask with a motion object in a first image at a first mask location;

means for copying said first mask to create a second mask associated with a second image;

means for moving said second mask to a second mask location associated with said motion object in said second image; and

,

means for reshaping said second mask to fit said motion object in said second image using a computer having memory to process said set of time ordered digital images comprising at least said first image and said second image based on underlying image data in said memory of said computer that corresponds to said first image and said second image.

42. The system of claim 41 further comprising:

means for accepting user input of at least one threshold associated with said underlying image data; and

,

means for creating said first mask based on said at least one threshold associated with said underlying image data at said first mask location;

wherein said means for associating said first mask with said motion object comprises means for storing said first mask in said memory of said computer.

43. The system of claim 41 further comprising:

means for associating a plurality of images comprising common background elements;

means for obtaining a composite background comprising said plurality of images wherein said composite background excludes a set of motion objects that move in said plurality of images in relation to said common background elements;

means for accepting at least one color parameter associated with a region within said composite background; and

,

means for applying said at least one color parameter automatically to at least one image selected from said plurality of images using said computer wherein said at least one color parameter comprises one or more of hue or saturation or luminance.

44. The system of claim 41 further comprising means for calculating an amount of disk space that required to archive a movie conversion project and means for signifying at least one asset that is rebuildable from other assets to avoid archival of said at least one asset.