System and method for 3D space-dimension based image processing

• US 9,177,220 B2
• Filed: 06/24/2012
• Issued: 11/03/2015
• Est. Priority Date: 07/30/2004
• Status: Active Grant

• Current Assignee: Jungo Connectivity Ltd.
• Original Assignee: Extreme Reality Ltd.
• Inventors: Givon, Dor
• Primary Examiner(s): PHILIPPE, GIMS S
• Application Number:
  US13/531,543
• Publication Number:
  US 20120320052A1
• Time in Patent Office: 1,227 Days
• Field of Search: 348/42, 348/49, 382/103, 382/128, 382/173, 382/300, 345/473
• US Class Current: 1/1
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 2201/12 Acquisition of 3D measurements of objects
• CPC Class Codes: H04N 13/167 Synchronising or controlling image signals
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals

Abstract

An apparatus for 3D representation of image data, comprising: a structure identifier for identifying structures in motion within image data, and a skeleton insertion unit, which associates three-dimensional skeleton elements with the identified structures. The skeleton elements are able to move with the structures to provide a three-dimensional motion and structural understanding of said image data which can be projected back onto the input data. As well as individual elements, complex bodies can be modeled by complex skeletons having multiple elements. The skeleton elements themselves can be used to identify the complex objects.

Claims

1. An apparatus for image based human machine interfacing by estimating a 3D representation of at least a portion of a user, based on a set of 2D image data of the at least a portion of the user, comprising:

image processing circuits for;

(1) identifying non-rigid structures of the user within said image data; and

(2) associating three-dimensional skeleton model elements with identified structures, such that model defined spatial constraints between skeleton model elements and spatial relations between identified non-rigid structures in the 2D image data set are used to fit the model to the 2D image data set and to approximate 3D coordinates of at least one of said non-rigid structures of the user.

2. An apparatus according to claim 1, wherein said image processing circuits are further adapted to identify within said image data a complex body made up of a plurality of interrelated structures whose three-dimensional movement constraints relative to one another are defined by the three-dimensional skeleton model.

3. An apparatus according to claim 2, further adapted to analyze relative movements within a series of sets of 2D image data, thereby providing three-dimensional movement information.

4. An apparatus according to claim 2, further adapted to store a plurality of predetermined skeleton models.

5. An apparatus according to claim 1, further adapted to associate a given skeleton element with an identified structure and to adjust a size of the given skeleton element to correspond to a dimension of the identified structure.

6. An apparatus according to claim 1, further adapted to associate a given skeleton element to a given identified structure and to deform the given skeleton element a deformation corresponding to the identified structure.

7. An apparatus according to claim 1, further adapted to associate a given skeleton element to a given identified structure and to apply texture to the given skeleton element.

8. An apparatus according to claim 1, further adapted to track respective structures at a first level and move corresponding skeleton elements at a second level.

9. An apparatus according to claim 8, further adapted to track respective skeleton elements at said second level and calculate the deviation of the structure at said first level.

10. An apparatus according to claim 1, further adapted to animate said image data by applying motion to said skeleton elements.

11. An apparatus according to claim 1, further adapted to select a first viewpoint and project said three-dimensional skeleton elements onto a 2D plane associated with said first viewpoint.

12. An apparatus according to claim 11, further adapted to select a second viewpoint and project said three-dimensional skeleton elements onto a 2D plane associated with said second viewpoint.

13. An apparatus according to claim 12, further adapted to select said first and second viewpoints to provide stereoscopic vision.

14. An apparatus according to claim 1, further adapted to capture input data onto said skeleton elements, to provide three-dimensional motion capture.

15. An apparatus according to claim 1, further adapted to store data relating to said skeleton elements, to provide three-dimensional image data compression and recording.

16. An apparatus according to claim 1, further adapted to form images at a given resolution by fitting interpolated pixels at said given resolution over said skeleton elements.