Two dimensional to three dimensional moving image converter

• US 10,015,478 B1
• Filed: 06/08/2015
• Issued: 07/03/2018
• Est. Priority Date: 06/24/2010
• Status: Active Grant

• Current Assignee: Steven Mark Hoffberg
• Original Assignee: Steven Mark Hoffberg
• Inventors: Rabin, Gregory S., Hoffberg, Steven M.
• Primary Examiner(s): Drennan, Barry
• Assistant Examiner(s): Pringle-Parker, Jason
• Application Number:
  US14/733,224
• Time in Patent Office: 1,121 Days
• Field of Search: None
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06V 10/89 using frequency domain filters, e.g. Fourier masks implemented on spatial light modulators
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/264 using the relative movement of objects in two video frames or fields
• CPC Class Codes: H04N 13/356 having separate monoscopic and stereoscopic modes

Abstract

The inventive method involves receiving as input a representation of an ordered set of two dimensional images. The ordered set of two dimensional images is analyzed to determine at least one first view of an object in at least two dimensions and at least one motion vector. The next step is analyzing the combination of the first view of the object in at least two dimensions, the motion vector, and the ordered set of two dimensional images to determine at least a second view of the object; generating a three dimensional representation of the ordered set of two dimensional images on the basis of at least the first view of the object and the second view of the object. Finally, the method involves providing as output an indicia of the three dimensional representation.

Claims

1. A method, comprising:

receiving at least one image within a stream of images, comprising a natural object having three dimensions;

automatically classifying at least one region of at least one image and identifying the natural object by pattern recognition;

automatically determining at least one depth gradient, comprising extracting a vanishing line and a vanishing point, associated with the natural object, in the at least one image, and producing a three-dimensional model of the natural object;

automatically identifying the natural object, based on at least metadata received with the stream of images;

automatically predicting, with at least one automated processor, a spatial configuration of at least one hidden surface comprising a portion of the natural object that is not visible in the at least one image, based on at least the classified at least one region, information extrinsic to the at least one image and not derived from the stream of images from a database containing information records, accessed based on at least the identification of the natural object, and the determined at least one depth gradient;

storing in a memory the three dimensional model of the natural object comprising the predicted spatial configuration, renderable according to at least one motion vector.

2. The method according to claim 1, wherein the at least one image is a color image, the at least one automated processor predicts a color of the at least one portion of the object not visible in the image, and the three dimensional model comprises a color configuration model of the natural object.

3. The method according to claim 1, wherein the at least one image is one of a series of images, which differ in at least one of time and perspective, and wherein said classifying is responsive to a plurality of said series of images.

4. The method according to claim 1, wherein the at least one image is one of a series of images, which differ in at least one of time and perspective, and wherein said predicting is responsive to a plurality of said series of images.

5. The method according to claim 1, further comprising rendering the three dimensional model as a video three dimensional image.

6. The method according to claim 1, further comprising rendering the three dimensional model as a series of three dimensional images over time rendered according to the at least one motion vector.

7. The method according to claim 1, wherein the stream of images comprises a stream of video images having a real time frame rate, and wherein the at least one automated processor completes a real-time prediction of the configuration of the at least one portion of the natural object for a first image of the stream of video images as a second image of the stream of video images is received, based on a plurality of video images comprising the first image and at least one third image preceding the second image.

8. The method according to claim 1, wherein the at least one processor comprises a single-instruction multiple data (SIMD) type processor having a plurality of processing elements which perform the same operation on multiple data values simultaneously, the at least one processor being controlled to perform at least said determining the at least one depth gradient on multiple data simultaneously.

9. The method according to claim 1, further comprising generating a series of parallax view stereo images of a video stream comprising the natural object within the natural space, based on at least the three dimensional model.

10. A method, comprising:

receiving a stream of images comprising a time sequence having a plurality of sequential images comprising at least one natural object;

automatically processing the stream of images to produce a three spatial dimensional model representing the stream of images comprising the at least one natural object;

automatically segmenting the at least one image of the stream of images into a plurality of regions having different characteristics;

automatically determining a hidden surface of the at least one natural object in the stream of images comprising at least one portion of the at least one natural object that is not visible in the stream of images;

automatically identifying the at least one natural object in the stream of images, based on at least metadata received with the stream of images;

automatically determining at least one depth gradient associated with the identified at least one natural object in the stream of images, based on at least extracting a vanishing line and a vanishing point associated with the at least one natural object;

automatically predicting, with at least one automated processor, a configuration of the hidden surface, based on at least information extrinsic to the stream of images and not derived from the stream of images from a remote database, accessed through a digital communication network, based on at least the identification of the at least one natural object, and the at least one depth gradient; and

storing, in a memory, a three dimensional model of the at least one natural object comprising the predicted configuration of the hidden surface, renderable according to at least one motion vector.

11. The method according to claim 10, wherein the at least one object comprises a foreground object having a motion relative to a camera which acquired the stream of images, said automatically predicting is further dependent on a visual pattern proximate to the hidden surface visible in the stream of images, and the at least one automated processor uses single-instruction multiple data (SIMD) type processing to perform the same operation on multiple data values simultaneously, to at least determine the at least one depth gradient.

12. A non-transitory computer readable medium, storing instructions for controlling at least one programmable automated processor to perform a method, comprising:

receiving a stream of images comprising a time sequence having a plurality of sequential images, comprising a natural object in a natural space represented in the stream of images;

automatically processing the stream of images to produce a three spatial dimensional model representing the stream of images comprising the natural object;

classifying at least one region of the natural space represented within the stream of images;

automatically determining a hidden surface of the natural object in the stream of images comprising at least one portion of the natural object that is not visible in the stream of images;

generating an identification of the natural object, based on at least metadata received with the stream of images;

determining at least one depth gradient of the natural object within the natural space represented in the stream of images, based on at least extracting a vanishing line and a vanishing point associated with the at least one natural object;

predicting a configuration of the hidden surface, based on at least the classified at least one region of the natural space represented in the stream of images and not derived from the stream of images, information extrinsic to the stream of images from a remote database, accessed based on at least the identification of the natural object, and the determined at least one depth gradient;

storing in a memory a three dimensional model of the object comprising the predicted configuration, renderable according to at least one motion vector.

13. The non-transitory computer readable medium according to claim 12, wherein the stream of images is a stream of color images, the three dimensional model comprises a color configuration model of the at least one natural object, and the instructions further control the automated processor to predict a color of the at least one portion of the natural object.

14. The non-transitory computer readable medium according to claim 12, wherein the stream of images comprise images having different perspectives over time.

15. The non-transitory computer readable medium according to claim 12, wherein the predicting is implemented using at least one neural network.

16. The non-transitory computer readable medium according to claim 12, wherein said determining at least one depth gradient comprises extracting a vanishing line and a vanishing point associated with the natural object.

17. The non-transitory computer readable medium according to claim 12, and the instructions further control the automated processor to render the three dimensional model as a stream of three dimensional images in which the natural object moves in the natural space over time.

18. The non-transitory computer readable medium according to claim 12, wherein the instructions further control the automated processor to access a remote database through a digital communication network, generate a query for the remote database based on at least the generated identification, receive from the remote database information relating to the natural object, wherein the stored three dimensional model comprises the received information relating to the natural object.

19. The non-transitory computer readable medium according to claim 12, wherein the at least one programmable automated processor comprises a single-instruction multiple data (SIMD) type processor having a plurality of processing elements which perform the same operation on multiple data values simultaneously, the at least one programmable automated processor being further controlled by the instructions to perform at least said determining the at least one depth gradient on multiple data simultaneously.

20. The non-transitory computer readable medium according to claim 19, wherein the at least one programmable automated processor is further controlled by the instructions to perform at least said predicting on multiple data simultaneously using the single-instruction multiple data (SIMD) type processor.