Methods and systems for interactively producing shapes in three-dimensional space

• US 9,383,895 B1
• Filed: 05/03/2013
• Issued: 07/05/2016
• Est. Priority Date: 05/05/2012
• Status: Active Grant

• Current Assignee: F Vinayak, Hairong Liu, Karthik Ramani, Raja Jasti
• Original Assignee: F Vinayak, Hairong Liu, Karthik Ramani, Raja Jasti
• Inventors: Liu, Hairong, Vinayak, F., Ramani, Karthik, Jasti, Raja
• Primary Examiner(s): Lee, Ting
• Application Number:
  US13/886,691
• Time in Patent Office: 1,159 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/04102 Flexible digitiser, i.e. constructional details for allowing the whole digitising part of a device to be flexed or rolled like a sheet of paper
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/0426 tracking fingers with respect to a virtual keyboard projected or printed on the surface virtual keyboards on touch screens G06F3/04886
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2219/2016 Rotation, translation, scaling
• CPC Class Codes: G06T 2219/2021 Shape modification

Abstract

A system and method for 3D design includes defining a three-dimensional virtual interaction space visualized with a 3D camera system operable to generate three-dimensional coordinate data corresponding to physical objects within the interaction space. A physical gesture of the user within the interaction space is interpreted according to pre-determined rules, the physical gesture including a movement of a physical object. A virtual shape is generated or manipulated in response to the interpretation of the physical gesture, the virtual shape residing virtually within the interaction space. A representation of the virtual 3D shape is interactively displayed during the physical gesture.

Claims

1. A method for generating a three-dimensional shape in three-dimensional space comprising:

defining a three-dimensional interaction space in the proximity of a user;

visualizing the interaction space with a three-dimensional camera system operable to generate three-dimensional coordinate data corresponding to three-dimensional physical objects within the interaction space;

interpreting a three-dimensional physical hand gesture of the user within the interaction space according to pre-determined rules, the three-dimensional physical hand gesture including a movement of a three-dimensional physical object and wherein the interpreting is performed at low latency and includes detecting intended interactions using pattern mining, wherein the pattern mining selects relevant salient features by detecting frequent gesture patterns and discarding irrelevant features of the three-dimensional physical hand gesture, wherein the interpreting further includes applying a classification method to only the selected relevant salient features to obtain an interpretation of the three-dimensional physical hand gesture;

generating or manipulating a virtual shape based upon the obtained interpretation of the three-dimensional physical hand gesture, the virtual shape residing virtually within the interaction space; and

interactively displaying a representation of the virtual three-dimensional shape during the three-dimensional physical hand gesture.

2. The method of claim 1, further comprising interactively displaying a representation of the three-dimensional physical object during the three-dimensional physical hand gesture.

3. The method of claim 1, wherein the three-dimensional physical object is at least one hand of the user.

4. The method of claim 1, wherein the step of interpreting the three-dimensional physical hand gesture includes determining the context of the three-dimensional physical hand gesture in relation to the interaction space and/or virtual shape, wherein a different rule is applied to the same physical gesture in different contexts.

5. The method of claim 4, wherein the step of determining context includes determining the spatial proximity of a physical object to the virtual shape within the interaction space and determining the location of the physical object on the virtual surface of the virtual shape.

6. The method of claim 5, wherein the step of determining context includes determining the spatial distances using predefined thresholds between the virtual shape and the physical object in spatial proximity to the shape.

7. The method of claim 4, wherein the step of determining context includes determining whether the physical gesture is made simultaneously on one object or multiple objects in the interaction space.

8. The method of claim 4, wherein:

the physical gesture includes pointing with a finger of at least one hand of the user within the interaction space;

the context includes a determination that no virtual shape is present within the interaction space in proximity to the pointing fingers; and

the predetermined rule applied to the gesture in the context is that movement of the finger generates a three-dimensional shape.

9. The method of claim 4, wherein:

the physical gesture includes pointing with a finger of at least one hand of the user within the interaction space;

the context includes a determination that a virtual shape is present within the interaction space and that the pointing finger is in spatial proximity to a location on the virtual surface of the virtual shape; and

the predetermined rule applied to the gesture in the context is that movement of the finger modifies or manipulates the three-dimensional shape at the location.

10. The method of claim 9, wherein the pre-determined rule is that movement of the finger into the three-dimensional shape inwardly deforms the virtual surface of the shape at the location.

11. The method of claim 4, wherein:

the physical gesture includes pinching together two fingers of at least one hand of the user within the interaction space;

the context includes a determination that a virtual shape is present within the interaction space and that the pinched fingers are in spatial proximity to a location on the virtual surface of the virtual shape; and

the predetermined rule applied to the gesture in the context is that movement of the pinched fingers modifies or manipulates the three-dimensional shape at the location.

12. The method of claim 11, wherein the pre-determined rule is that movement of the finger away from the virtual surface of the three-dimensional shape outwardly deforms the virtual surface of the shape at the location.

13. The method of claim 4, wherein the context includes whether the virtual three-dimensional shape is being generated using previously stored data.

14. The method of claim 13, wherein the previously stored data includes predefined types of shapes generated by pre-determined gestures.

15. The method of claim 1, wherein the step of interpreting a gesture includes interpreting a posture of the physical object and interpreting the movement of the physical object in the pose.

16. The method of claim 1, further comprising:

comparing the virtual shape to previously generated three-dimensional shapes maintained in a data base; and

displaying a representation of a previously-generated shape determined to be a match to the virtual shape.

17. The method of claim 1, further comprising generating a machine-readable output containing data defining the virtual three-dimensional shape.

18. The method of claim 1, wherein the virtual three-dimensional shape is generated using previously stored data.

19. The method of claim 1, wherein:

the three-dimensional physical gesture of the user includes a plurality of gestures within the interaction space during a time period, the plurality of three-dimensional gestures including meaningful gestures corresponding to an intent of the user and unintentional gestures; and

the step of interpreting the three-dimensional physical gesture includes recognizing the meaningful gestures from among the plurality of gestures.

20. The method of claim 19, further comprising training a processor to recognize the meaningful gestures based on core characteristics of predefined gestures.

21. The method of claim 20, wherein:

the core characteristic of a predefined gesture is unique among all of the predefined gestures; and

the step of recognizing the physical gesture occurs before the user completes a meaningful gesture corresponding to the predefined gesture having the unique core characteristic.

22. The method of claim 20, wherein the training includes applying a random forest algorithm to detect a core characteristic from among multiple instances of a meaningful gesture.

23. The method of claim 22, wherein a core characteristic is a frequent pattern of movement of the physical object creating the gesture.

24. The method of claim 23, wherein the frequent pattern is detected in the plurality of gestures by:

applying a multi-scale sliding time window during the plurality of gestures to generate a data set of gestures; and

then applying a random space partition scheme to the data set to find gestures that include the frequent pattern.

25. The method of claim 23, wherein:

the physical object is the user'"'"'s hand(s), the gesture is the posture or pose of the user'"'"'s hand(s) and the trajectory of movement of the hand(s) and the core characteristic is independent of the physical orientation of the posture or of the trajectory.

26. The method of claim 19, wherein the class of meaningful gestures to be recognized includes the subclass of gestures that are only the posture of the physical object, the subclass of gestures that are only the trajectory of movement of the physical object and the subclass of gestures that include posture and trajectory.

27. The method of claim 1, further comprising:

maintaining a data base of predetermined shape templates;

interpreting a physical gesture to conduct a search within the data base; and

interpreting a subsequent physical gesture to generate a shape; and

selecting a shape template within the data base corresponding to the generated shape.

28. The method of claim 1, further comprising:

providing a virtual navigation feature; and

interpreting a physical gesture to select a navigation command in the navigation feature.

29. The method of claim 28, wherein the physical gesture for the navigation feature occurs in the interaction space during the generation or manipulation of a virtual shape.