Systems and methods for implementing three-dimensional (3D) gesture based graphical user interfaces (GUI) that incorporate gesture reactive interface objects

• US 9,298,266 B2
• Filed: 08/12/2013
• Issued: 03/29/2016
• Est. Priority Date: 04/02/2013
• Status: Active Grant

• Current Assignee: KAYA Dynamics LLC
• Original Assignee: Aquifi, Inc.
• Inventors: Blackstone, Keith, Webb, Eric, Girdwood, David, Dal Mutto, Carlo
• Primary Examiner(s): Alvesteffer, Stephen
• Assistant Examiner(s): Shiberou, Mahelet
• Application Number:
  US13/965,157
• Publication Number:
  US 20140298273A1
• Time in Patent Office: 960 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042

Abstract

Systems and methods in accordance with embodiments of the invention implement three-dimensional (3D) gesture based graphical user interfaces (GUI) using gesture reactive interface objects. One embodiment includes using a computing device to render an initial user interface comprising a set of interface objects, detect a targeting 3D gesture in captured image data that identifies a targeted interface object within the user interface, change the rendering of at least the targeted interface object within the user interface in response to the targeting 3D gesture that targets the interface object, detect an interaction 3D gesture in additional captured image data that identifies a specific interaction with a targeted interface object, modify the user interface in response to the interaction with the targeted interface object identified by the interaction 3D gesture, and render the modified user interface.

Claims

1. A method of rendering a user interface on a computing device, comprising:

rendering an initial user interface comprising a set of interface objects using a computing device, where each interface object in the set of interface objects includes a graphical element that is rendered when the interface object is rendered for display and a target zone within the user interface;

detecting a targeting 3D gesture in captured image data that identifies a targeted interface object within the user interface using the computing device by;

identifying a 3D interaction zone within the captured image data that maps to the user interface;

determining the location of at least a portion of a human hand within the 3D interaction zone;

identifying a first pose of the at least a portion of a human hand corresponding to a targeting 3D gesture;

mapping the location of the at least a portion of a human hand within the 3D interaction zone to a location within the user interface;

determining that the mapped location within the user interface falls within the target zone of a specific interface object within the user interface; and

identifying the specific interface object as the targeted interface object in response to an identification of the first pose as a targeting gesture and a determination that the mapped location of at least a portion of the human hand falls with the target zone of the specific interface object;

enabling a set of one or more interaction gestures for the targeted interface object in response to the detection of the targeting 3D gesture using the computing device wherein each of the one or more interaction gestures is associated with a permitted interaction in a set of permitted interactions allowed for the targeted interface object and each permitted interaction is an action performed via the user interface to manipulate the targeted interface object;

changing the rendering of at least the targeted interface object within the user interface in response to the targeting 3D gesture that targets the interface object using the computing device;

detecting an interaction 3D gesture from the set of one or more interaction gestures for the targeted interface object in additional captured image data that identifies a specific interaction from the set of permitted interactions with the targeted interface object using the computing device, where the detection of the interaction 3D gesture comprises;

tracking the motion of at least a portion of a human hand within the 3D interaction zone in the additional captured image data;

identifying a change in pose of at least a portion of a human hand within the 3D interaction zone from the first pose to a second pose during the motion of the at least a portion of the human hand irrespective of the location of the at least a portion of a human hand within the 3D interaction zone during the motion; and

identifying the second pose of the at least a portion of the human hand as corresponding to one of the 3D interaction gestures from the set of one or more interactive gestures to control the targeted interface object, the identifying the second pose being independent of a mapping between the user interface and a location of the human hand in the 3D interaction zone;

modifying the user interface in response to the specific interaction with the targeted interface object identified by the detected interaction 3D gesture using the computing device; and

rendering the modified user interface using the computing device.

2. The method of claim 1, wherein:

the computing device is part of a real-time gesture based interactive system further comprising a display device and an image capture system;

and the method further comprises;

displaying the rendered user interfaces using the display; and

capturing the captured image data and the additional captured image data using the image capture system.

3. The method of claim 1, wherein the targeting 3D gesture is a pointing motion towards the target zone of the targeted interface.

4. The method of claim 1, wherein changing the rendering of at least the targeted interface object within the user interface in response to the 3D gesture targeting the interface object using the computing device further comprises changing the size of the graphical element of the targeted interface object.

5. The method of claim 1, wherein changing the rendering of at least the targeted interface object within the user interface in response to the 3D gesture targeting the interface object using the computing device further comprises changing the size of the graphical elements of a plurality of interface objects including the targeted interface object.

6. The method of claim 1, wherein changing the rendering of at least the targeted interface object within the user interface in response to the 3D gesture targeting the interface object using the computing device further comprises changing the size of the target zone of the targeted interface object and rendering a graphical element indicating the location of the target zone.

7. The method of claim 1, wherein changing the rendering of at least the targeted interface object within the user interface in response to the 3D gesture targeting the interface object using the computing device further comprises changing the size of the target zone of a plurality of interface objects including the targeted interface object and rendering graphical elements indicating the location of the target zone of each of the plurality of interface objects.

8. The method of claim 1, further comprising displaying an interaction element indicating the time remaining to interact with the targeted interface object in response to the targeting 3D gesture targeting the interface object for a predetermined period of time using the computing device.

9. The method of claim 8, wherein detecting an interaction 3D gesture from the set of one or more interaction gestures for the targeted interface object in the additional captured image data that identifies a specific interaction with a targeted interface object using the computing device further comprises detecting the interaction 3D gesture within a predetermined time period from the receipt of the detection of the targeting 3D gesture.

10. The method of claim 1, wherein determining that the tracked motion of at least a portion of a human hand within the 3D interaction zone corresponds to an interaction 3D gesture from the set of one or more interaction gestures for the targeted interface object further comprises identifying motion corresponding to at least one motion selected from the group consisting of:

bending and straightening of at least one finger; and

lowering and raising of at least one finger.

11. The method of claim 1, the method further comprising determining that the targeting 3D gesture targets the targeted interface object for a predetermined period of time, where the determination considers the targeting 3D gesture to be targeting the targeted interface object during any period of time in which the targeting 3D gesture does not target the interface object that is less than a hysteresis threshold.

12. The method of claim 1, further comprising:

detecting an input via a 3D gesture input modality from captured image data using the computing device; and

changing the manner in which the initial user interface is rendered in response to detection of input via a 3D gesture input modality using the computing device.

13. The method of claim 12, wherein changing the manner in which the initial user interface is rendered in response to detection of input via a 3D gesture input modality further comprises changing at least one selected from the group consisting of:

the size of the graphical elements of a plurality of interface objects;

the size of the target zones of a plurality of interface objects and rendering graphical elements showing the locations of the target zones; and

the size of the region of the user interface that is rendered.

14. The method of claim 12, wherein:

detecting an input via a 3D gesture input modality further comprises determining a distance from the display at which 3D gestures are being performed to provide 3D gesture input from the captured image data; and

changing the manner in which the initial user interface is rendered in response to detection of input via a 3D gesture input modality further comprises changing the manner in which the user interface is rendered based upon the distance from the display at which 3D gestures are being performed to provide 3D gesture input.

15. The method of claim 1, further comprising:

determining a distance of a user from a display on which the rendered user interface is being displayed using the computing device; and

changing the manner in which the initial user interface is rendered by the computing device based upon the distance of the user from the display on which the rendered user interface is being displayed.

16. The method of claim 15, wherein changing the manner in which the initial user interface is rendered based upon the distance of the user from the display on which the rendered user interface is being displayed further comprises changing at least one selected from the group consisting of:

the size of the graphical elements of a plurality of interface objects;

the size of the target zones of a plurality of interface objects and rendering graphical elements showing the locations of the target zones; and

the size of the region of the user interface that is rendered.

17. The method of claim 1, wherein the size of at least one of a rendered graphical element or a target zone of at least one interface object is determined based upon at least one of the size of a display on which the rendered user interface is displayed and the resolution of the display.

18. The method of claim 1, further comprising increasing the size of the target zone of the targeted interface object within the user interface in response to the detection of the targeting 3D gesture that targets the interface object using the computing device.

19. The method of claim 1, further comprising:

detecting an input via a 3D gesture input modality from captured image data using the computing device; and

increasing the size of the target zones of at least one interface object within the user interface in response to the detection of the input via the 3D gesture input modality using the computing device.

20. A method of rendering a user interface on a real-time gesture based interactive system comprising an image capture system including at least two cameras, an image processing system and a display device, the method comprising:

rendering an initial user interface comprising a set of interface objects using the image processing system, where each interface object comprises;

a graphical element that is rendered when the interface object is rendered for display;

a target zone that defines at least one region in the user interface in which a targeting three-dimensional (3D) gesture targets the interface object; and

a description of a set of permitted interactions;

displaying the rendered user interface using the display;

capturing image data using the image capture system;

detecting an input via a 3D gesture input modality from the captured image data using the image processing system;

changing the manner in which the initial user interface is rendered in response to detection of an input via a 3D gesture input modality using the image processing device;

displaying the rendered user interface using the display;

detecting a targeting 3D gesture that targets a targeted interface object within the user interface using the image processing system by;

identifying a 3D interaction zone within the captured image data that maps to the user interface;

determining the location of at least a portion of a human hand within the 3D interaction zone from the captured image data;

identifying a first pose of the at least a portion of a human hand within the target zone that corresponds to a targeting 3D gesture;

mapping the location of the at least a portion of a human hand within the 3D interaction zone to a location within the user interface;

determining that the mapped location within the user interface falls within the target zone of a specific interface object in the user interface;

identifying the specific interface object as the targeted interface object in response to an identification of the first pose as the targeting gesture and a determination that the mapped location of the at least a portion of the human hand falls within the target zone of the specific interface object in the user interface; and

changing the rendering of at least the targeted interface object within the user interface in response to the 3D gesture targeting the interface object using the image processing system;

displaying the user interface via the display;

capturing additional image data using the image capture system;

determining that the targeting 3D gesture targets the targeted interface object for a predetermined period of time, where the determination considers the targeting 3D gesture to be targeting the targeted interface object during any period of time in which the targeting 3D gesture does not target the targeted interface object that is less than a hysteresis threshold;

enabling a set of one or more interaction gestures for the targeted interface object in response to the detection of the targeting 3D gesture using the computing device wherein each of the one or more interaction gestures is associated with a permitted interaction in a set of permitted interactions allowed for the targeted interface object and each permitted interaction is an action performed via the user interface to manipulate the targeted interface object;

displaying an interaction element indicating the time remaining to interact with the targeted interface object in response to a determination that the targeting 3D gesture has targeted the interface object for a predetermined period of time using the image processing system;

detecting an interaction 3D gesture for the set of one or more interaction gestures in additional captured image data within a predetermined time period from the detection of the targeting 3D gesture input, where the interaction 3D gesture identifies a specific interaction with the targeted interface object using the image processing system and is detected by;

tracking the motion of at least a portion of a human hand within the 3D interaction zone;

identifying a change in pose of at least a portion of a human hand within the 3D interaction zone from the first pose to a second pose during the motion of the at least a portion of the human hand has changed and corresponds to an interaction 3D gesture from the set of one or more interaction gestures for the targeted interface object irrespective of the location of the at least a portion of a human hand within the 3D interaction zone during the motion;

identifying the second pose of the at least a portion of the human hand as corresponding to one of the 3D interaction gestures from the set of one or more interactive gestures to control the targeted interface object, the identifying the second pose being independent of a mapping between the user interface and a location of the human hand in the 3D interaction zone;

verifying that the interaction gesture is associated with a specific interaction within the set of permitted interactions for the interface object using the image processing system;

modifying the user interface in response to the specific interaction with the targeted interface object identified by the interaction 3D gesture using the image processing system;

rendering the modified user interface using the image processing system; and

displaying the rendered user interface using the display.

21. A real-time gesture based interactive system configured to display a user interface and receive three-dimensional (3D) gesture based input, comprising:

a processor;

an image capture system configured to capture image data and provide the captured image data to the processor;

memory containing;

an operating system;

an interactive application; and

a 3D gesture tracking application;

wherein the interactive application and the operating system configure the processor to;

generate and render an initial user interface comprising a set of interface objects, where each interface object includes a graphical element that is rendered when the interface object is rendered for display and a target zone that defines at least one region in the user interface in which the interface object is to be targeted; and

modify an initial user interface in response to a detected interaction with a targeted interface object and render an updated user interface; and

wherein the 3D gesture tracking application and the operating system configure the processor to;

capture image data using the image capture system;

detect a targeting 3D gesture in captured image data that identifies a targeted interface object within a user interface by;

identifying a 3D interaction zone within the captured image data that maps to the user interface;

determining the location of at least a portion of a human hand within the 3D interaction zone;

identifying a first pose of the at least a portion of a human hand that corresponds to a targeting 3D gesture;

mapping the location of the at least a portion of a human hand within the 3D interaction zone to a location within the user interface;

determining that the mapped location within the user interface falls within a target zone of a specific interface object; and

identifying the specific interface object as the targeted interface object in response to an identification of the first pose as a targeting gesture and a determination that the mapped location of at least a portion of the human hand falls with the target zone of the specific interface object;

enable a set of one or more interaction gestures for the targeted interface object in response to the detection of the targeting 3D gesture using the computing device wherein each of the one or more interaction gestures is associated with a permitted interaction in a set of permitted interactions allowed for the targeted interface object and each permitted interaction is an action performed via the user interface to manipulate the targeted interface object;

change the rendering of at least the targeted interface object within a user interface in response to detection of a targeting 3D gesture that targets the interface object;

detect an interaction 3D gesture from the set of one or more interaction gestures for the targeted interface object in captured image data that identifies a specific interaction with a targeted interface object, where the detection of the interaction 3D gesture comprises;

tracking the motion of at least a portion of a human hand within the 3D interaction zone in the additional captured image data;

identifying a change in pose of at least a portion of a human hand within the 3D interaction zone from the first pose to a second pose during the motion of the at least a portion of the human hand irrespective of the location of the at least a portion of a human hand within the 3D interaction zone during the motion; and

identifying the second pose of the at least a portion of the human hand as corresponding to one of the 3D interaction gestures from the set of one or more interactive gestures to control the targeted interface object, the identifying the second pose being independent of a mapping between the user interface and a location of the human hand in the 3D interaction zone; and

provide events corresponding to specific interactions with targeted interface objects to the interactive application.