Systems and methods for gesture based interaction with viewpoint dependent user interfaces

• US 9,619,105 B1
• Filed: 01/30/2014
• Issued: 04/11/2017
• Est. Priority Date: 01/30/2014
• Status: Active Grant

• Current Assignee: Aquifi, Inc.
• Original Assignee: Aquifi, Inc.
• Inventors: Dal Mutto, Carlo
• Primary Examiner(s): Edwards, Carolyn R
• Assistant Examiner(s): Said, Mansour M
• Application Number:
  US14/169,124
• Time in Patent Office: 1,167 Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object

Abstract

Systems and methods are disclosed for performing three-dimensional (3D) gesture based interaction with a user interface rendered from a 3D object model based upon the viewpoint of a user. Gesture based interactive systems in accordance with many embodiments of the invention utilize a spatial model of how user interface objects, such as icons and a cursor, spatially relate to one another. The user interface model can be constructed as a 3D object model. With a 3D object model, the operating system can use depth information to render a user interface display appropriate to the requirements of a specific display technology. In many embodiments, head tracking is used to determine a viewpoint from which to render a user interface display from a 3D object model maintained by the operating system and the user interface can be updated in response to the detection of 3D gestures.

Claims

1. A real-time gesture based interactive system configured to enable gesture based interaction with user interfaces rendered from a 3D object model in a viewpoint dependent manner, comprising:

a processor;

a camera system configured to capture image data;

memory containing;

an operating system including a 3D object model that describes three dimensional spatial relationships between a set of user interface objects comprising a first user interface object and a second user interface object;

a head tracking application; and

an object tracking application;

wherein the operating system configures the processor to;

capture image data using the camera system;

detect first physical coordinates of a user'"'"'s head by processing at least a portion of the image data using the head tracking application;

determine a user viewpoint from which to render a user interface display based on the first physical coordinates of the user'"'"'s head such that a portion of the first user interface object is occluded by the second user interface object in the rendered user interface display;

determine an object location by processing at least a portion of the captured image data using the object tracking application;

map the object location to a cursor location comprising three dimensional coordinates;

render a user interface display from the 3D object model and the cursor location based upon the user viewpoint determined based on the first physical coordinates of the user'"'"'s head;

capture additional image data using the camera system;

detect second physical coordinates of the user'"'"'s head by processing at least a portion of the additional image data using the head tracking application, the second physical coordinates being different from the first physical coordinates;

determine an updated user viewpoint from which to render a user interface display based on the second physical coordinates of the user'"'"'s head, the updated user viewpoint being different from the user viewpoint such that the portion of the first user interface object is not occluded by the second user interface object in the updated user interface display;

determine an updated object location by processing at least a portion of the additional captured image data using the object tracking application;

map the updated object location to an updated cursor location comprising three dimensional coordinates; and

render an updated user interface display from the 3D object model and the updated cursor location based upon the updated user viewpoint determined based on the second physical coordinates of the user'"'"'s head and the updated object location, where the updated user interface display is rendered to simulate motion parallax based upon depth of the user interface objects and the updated cursor location in the 3D object model.

2. The system of claim 1, wherein the operating system configures the processor to map the object location to a cursor location so that the cursor is not occluded when the processor renders a user interface display.

3. The system of claim 2, wherein the operating system configures the processor to map the object location to a cursor location within the 3D object model, where the cursor location has a predetermined foreground depth.

4. The system of claim 2, wherein the operating system configures the processor to map the object location to a cursor location within an overlay on a user interface display rendered from the 3D object model based upon the user viewpoint.

5. The system of claim 1, wherein the operating system configures the processor to map the object location to a cursor location within the 3D object model.

6. The system of claim 5, wherein the operating system configures the processor to render a user interface display from the 3D object model and the cursor location based upon the user viewpoint so that user interface objects that occlude the cursor location within the 3D object model are rendered to be semi-transparent.

7. The system of claim 5, wherein the operating system configures the processor to render a user interface display from the 3D object model and the cursor location based upon the user viewpoint so that the processor does not render user interface objects that are closer to a foreground of the 3D object model than the cursor location.

8. The system of claim 7, wherein the operating system configures the processor to render a user interface display from the 3D object model and the cursor location based upon the user viewpoint so that user interface objects that are rendered are scaled based upon at least the depth of the cursor location within the 3D object model.

9. The system of claim 5, wherein the operating system configures the processor to render a user interface display from the 3D object model and the cursor location based upon the user viewpoint to include at least one visual indicator of the depth of the cursor location.

10. The system of claim 1, wherein:

the object tracking application configures the processor to detect the 3D location of a finger within at least a portion of the captured video data; and

the operating system configures the processor to map the 3D location of a finger to a cursor location within the 3D object model.

11. The system of claim 1, wherein the object tracking application configures the processor to detect pose of a human hand within at least a portion of the captured video data.

12. The system of claim 1, wherein the operating system configures the processor to determine a user viewpoint from which to render a user interface display by using the head tracking application to configure the processor to:

determine a head pose from at least a portion of the captured image data; and

map the head pose to a user viewpoint relative to a display device.

13. The system of claim 1, wherein the operating system configures the processor to determine a user viewpoint from which to render a user interface display by using the head tracking application to configure the processor to:

determine a head pose and a gaze direction from at least a portion of the captured image data; and

map the head pose and gaze direction to a user viewpoint relative to a display device.

14. The system of claim 1, wherein the operating system further configures the processor to detect a 3D gesture using the object tracking application and to modify the 3D object model in response the detected 3D gesture.

15. The system of claim 14, wherein the 3D gesture is selected from the group consisting of a static 3D gesture and a dynamic 3D gesture.

16. The system of claim 1, wherein the operating system further configures the processor to detect an object location within an interaction zone.

17. The system of claim 1, wherein the camera system comprises at least two cameras.

18. The system of claim 1, wherein the user interface display is rendered from the 3D object model, the cursor location, and the user viewpoint determined based on the first physical coordinates of the user'"'"'s head such that the cursor is occluded by at least one user interface object of the set of user interface objects in the rendered user interface display, and

wherein the updated user interface display is rendered from the 3D object model, the updated cursor location, and the updated user viewpoint determined based on the second physical coordinates of the user'"'"'s head such that the cursor is not occluded by the at least one user interface object in the updated user interface display.

19. A method for gesture based interaction with a user interfaces rendered from a 3D object model that describes three dimensional spatial relationships between a set of user interface objects comprising a first user interface object and a second user interface object in a viewpoint dependent manner, comprising:

capturing image data using a camera system;

detecting first physical coordinates of a user'"'"'s head by processing at least a portion of the image data using the processor configured by a head tracking application;

determining a user viewpoint from which to render a user interface display based on the first physical coordinates of the user'"'"'s head such that a portion of the first user interface object is occluded by the second user interface object;

determining an object location by processing at least a portion of the captured image data using a processor configured by an object tracking application;

mapping the object location to a cursor location using the processor configured by an operating system, the cursor location comprising three dimensional coordinates;

rendering a user interface display from the 3D object model and the cursor location based upon the user viewpoint determined based on the first physical coordinates of the user'"'"'s head using the processor configured by the operating system;

capturing additional image data using the camera system;

detecting second physical coordinates of the user'"'"'s head by processing at least a portion of the additional image data using the head tracking application, the second physical coordinates being different from the first physical coordinates;

determining an updated user viewpoint from which to render a user interface display based on the second physical coordinates of the user'"'"'s head, the updated user viewpoint being different from the user viewpoint such that the portion of the first user interface object is not occluded by the second user interface object;

determining an updated object location by processing at least a portion of the additional captured image data using the processor configured by the object tracking application;

mapping the updated object location to an updated cursor location using the processor configured by the operating system, the updated cursor location comprising three dimensional coordinates; and

rendering an updated user interface display from the 3D object model and the updated cursor location based upon the updated user viewpoint determined based on the second physical coordinates of the user'"'"'s head and the updated object location using the processor configured by the operating system, where the updated user interface display is rendered to simulate motion parallax based upon depth of the user interface objects and the updated cursor location in the 3D object model.