Three-dimensional user interface

• US 8,166,421 B2
• Filed: 01/13/2009
• Issued: 04/24/2012
• Est. Priority Date: 01/14/2008
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: PrimeSense Ltd (Apple Inc.)
• Inventors: Magal, Oz, Guendelman, Eran, Golman, Sergio, Hendel, Ziv, Maizels, Aviad, Berliner, Tamir, Pokras, Jonathan
• Primary Examiner(s): Bashore, William
• Assistant Examiner(s): PHANTANA ANGKOOL, DAVID
• Application Number:
  US12/352,622
• Publication Number:
  US 20090183125A1
• Time in Patent Office: 1,197 Days
• Field of Search: 715/863, 715/850, 715/852
• US Class Current: 715/863
• CPC Class Codes: A61B 8/08 Detecting organic movements or changes, e.g. tumours, cysts, swellings A61B8/02 - A61B8/06 take precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

A user interface method includes defining an interaction surface containing an interaction region in space. A sequence of depth maps is captured over time of at least a part of a body of a human subject. The depth maps are processed in order to detect a direction and speed of movement of the part of the body as the part of the body passes through the interaction surface. A computer application is controlled responsively to the detected direction and speed.

Claims

1. A user interface method, comprising:

displaying an object on a display screen;

defining an interaction surface containing an interaction region in space, and mapping the interaction surface to the display screen;

capturing a sequence of depth maps over time of at least a part of a body of a human subject;

processing the depth maps in order to detect a direction and speed of movement of the part of the body and to predict a touch point of the part of the body, responsively to the movement, wherein the touch point indicates a location in the interaction surface where the part of the body penetrates the interaction surface;

controlling a computer application so as to change the displayed object on the screen responsively to the mapping and to the predicted touch point; and

wherein processing the depth maps comprises identifying, responsively to the detected movement, a collision induced by the movement with a predefined three-dimensional shape in space.

2. The method according claim 1, and comprising:

defining a visualization surface containing a visualization region in the space, such that the interaction surface is within the visualization region; and

processing the depth maps in order to identify the part of the body that is located within the visualization region,

wherein displaying the object comprises presenting on the display screen a representation of the part of the body that is located within the visualization region.

3. The method according claim 1, wherein defining the interaction surface comprises specifying dimensions of the interaction surface, wherein the interaction surface is mapped to the display screen responsively to the specified dimensions.

4. The method according to claim 1, wherein processing the depth maps comprises applying a three-dimensional connected component analysis to the depth maps in order to identify the part of the body.

5. The method according to claim 1, wherein processing the depth maps comprises identifying, responsively to the detected movement, a gesture made by the human subject.

6. The method according to claim 5, wherein identifying the gesture comprises learning the gesture during a training phase, and thereafter detecting the learned gesture in order to control the computer application.

7. The method according to claim 1, wherein processing the depth maps comprises identifying a posture of at least the part of the body, and controlling the computer application responsively to the posture.

15. The method according to claim 1, wherein defining the interaction surface comprises receiving an input from a user of the computer application, and defining the interaction surface responsively to the input.

8. User interface apparatus, comprising:

a display screen, which is configured to display an object;

a sensing device, which is configured to capture a sequence of depth maps over time of at least a part of a body of a human subject;

a processor, which is configured to define an interaction surface, which contains an interaction region in space, and to map the interaction surface to the display screen, and to process the depth maps in order to detect a direction and speed of movement of the part of the body and to predict a touch point of the part of the body,

responsively to the movement, wherein the touch point indicates a location in the interaction surface where the part of the body penetrates the interaction surface and to control a computer application so as to change the displayed object on the screen responsively to the mapping and to the predicted touch point; and

wherein processing the depth maps comprises identifying, responsively to the detected movement, a collision induced by the movement with a predefined three-dimensional shape in space.

9. The apparatus according claim 8, wherein the processor is configured to process the depth maps in order to identify the part of the body that is located within a visualization region contained within a predefined visualization surface, such that the interaction surface is within the visualization region, and to present on the display screen a representation of the part of the body that is located within the visualization region.

10. The apparatus according claim 8, wherein the processor is configured to accept a specification of dimensions of the interaction surface, and to map the interaction surface to the display screen responsively to the dimensions.

11. The apparatus according to claim 8, wherein the processor is configured to apply a three-dimensional connected component analysis to the depth maps in order to identify the part of the body.

12. The apparatus according to claim 8, wherein the processor is configured to identify, responsively to the detected movement, a gesture made by the human subject.

13. The apparatus according to claim 12, wherein the processor is configured to learn the gesture during a training phase, and thereafter to detect the learned gesture in order to control the computer application.

14. A computer software product, comprising a computer-readable medium in which program instructions are stored, which instructions, when read by a computer,

cause the computer to display an object on a display screen,

to define an interaction surface, which contains an interaction region in space, and to map the interaction surface to the display screen,

to process a sequence of depth maps created over time of at least a part of a body of a human subject in order to detect a direction and speed of movement of the part of the body and to predict a touch point of the part of the body,

responsively to the movement, wherein the touch point indicates a location in the interaction surface where the part of the body penetrates the interaction surface,

to control a computer application so as to change the displayed object on the screen responsively to the mapping and to the predicted touch point, and

wherein processing the depth maps comprises identifying, responsively to the detected movement, a collision induced by the movement with a predefined three-dimensional shape in space.

16. A user interface method, comprising:

displaying an object on a display screen;

defining, responsively to an input received from a user of the computer application, an interaction surface containing an interaction region in space for a computer application while specifying, based on the input received from the user, dimensions in space of the interaction region that correspond to an area of the display screen;

capturing a sequence of depth maps over time of at least a part of a body of a human subject;

processing the depth maps in order to detect a movement of the part of the body as the part of the body passes through the interaction surface;

controlling the computer application so as to change the displayed object on the screen responsively to the movement of the part of the body within the specified dimensions of the interaction region; and

wherein processing the depth maps comprises identifying, responsively to the detected movement, a collision induced by the movement with a predefined three-dimensional shape in space.

17. The method according to claim 16, wherein the input received from the user specifies a depth dimension of the interaction surface.

18. The method according to claim 17, wherein the input received from the user also specifies transverse dimensions of the interaction surface.

19. The method according to claim 16, wherein specifying the dimensions in space comprises defining a zoom factor that maps transverse dimensions of the interaction surface to corresponding dimensions of the computer display screen.