System and method for gesture detection and feedback

• US 9,218,119 B2
• Filed: 03/25/2011
• Issued: 12/22/2015
• Est. Priority Date: 03/25/2010
• Status: Active Grant

• Current Assignee: Malikie Innovations Limited (Key Patent Innovations Limited)
• Original Assignee: Blackberry Limited
• Inventors: Johansson, Karl-Anders, Haliburton, James, Grdenfors, Dan
• Primary Examiner(s): Olson, Jason
• Assistant Examiner(s): Abebe, Sosina
• Application Number:
  US13/072,328
• Publication Number:
  US 20110234543A1
• Time in Patent Office: 1,733 Days
• Field of Search: 345/173, 345/175, 340/4.1, 715/863
• US Class Current: 1/1
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/041 Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence

Abstract

A computer device with a sensor subsystem for detecting off-surface objects, that carries out continued processing of the position and shape of objects detected in the vicinity of the device, associates these positions and shapes with predetermined gesture states, determines if the object is transitioning between gesture states and provides feedback based on the determined transition between the gesture states.

Claims

1. A computing device, comprising:

a display;

a touch screen for detecting on-surface user input;

a sensor subsystem for detecting off-surface objects; and

memory having instructions stored on it which when run on a processor, causes the processor to perform the steps of;

setting the sensor subsystem to a powered-on state, from a powered-off state, when a predetermined on-surface gesture is detected via the touch screen;

determining a compound gesture using a combination of the detected on-surface gesture and an off-surface gesture determined using the sensor subsystem by;

detecting an off-surface object using the sensor subsystem;

determining a position and a shape of said detected off-surface object;

matching the determined position with a set of reference zones and the determined shape with a set of reference poses to thereby determine an off-surface gesture state;

detecting changes in the determined position and shape of said detected off-surface object using the sensor subsystem;

matching said changes in the determined position with transitions between said reference zones and matching said changes in determined shapes with transitions between said reference poses to thereby determine a transition between off-surface gesture states; and

,

instructing a user interface to provide feedback based on the compound gesture including the determined transition between off-surface gesture states.

2. The computing device of claim 1, wherein the feedback is at least one of graphical, auditory and haptic in nature.

3. The computing device of any preceding claim, wherein the feedback is a continuous feedback.

4. The computing device of claim 3, wherein the processor performs the further step of:

determining a degree of transition between off-surface gesture states and providing the continuous feedback based on said degree of transition.

5. The computing device of claim 4, wherein the degree of transition includes at least one of:

where in between the reference zones the determined positions are present, where in between the reference poses the determined shapes are, which reference zone the determined position is closest and which reference pose the determined shape is closest.

6. The computing device of claim 1, wherein the reference zones are represented as at least one of volumes in a three-dimensional space, points in a three-dimensional space, areas in a two-dimensional space and points in a two-dimensional space.

7. The computing device of claim 1, wherein the sensor subsystem includes at least one of a single camera and two cameras in stereoscopic configuration.

8. The computing device of claim 1, wherein the set of reference zones and the set of reference poses are changeable.

9. A computer-implemented method, for operating a computing device comprising a display, a touch screen for detecting on-surface user input, sensor sub system, processor and memory, comprising the steps of:

setting the sensor subsystem to a powered-on state, from a powered-off state, when a predetermined on-surface gesture is detected via the touch screen;

determining a compound gesture using a combination of the detected on-surface gesture and an off-surface gesture determined using the sensor subsystem by;

detecting an off-surface object using the sensor subsystem;

determining a position and a shape of said detected off-surface object;

matching the determined position with a set of reference zones and the determined shape with a set of reference poses to thereby determine an off-surface gesture state;

detecting changes in the determined position and shape of said detected off-surface object using the sensor subsystem;

matching said changes in the determined position with transitions between said reference zones and matching said changes in determined shapes with transitions between said reference poses to thereby determine a transition between off-surface gesture states; and

,

instructing a user interface to provide feedback based on the compound gesture including the determined transition between off-surface gesture states.

10. A non-transitory computer-readable medium having stored on it instructions which when run on a processor cause the processor to carry out the method of claim 9.