Gesture handlers of a gesture engine

• US 9,250,788 B2
• Filed: 03/18/2009
• Issued: 02/02/2016
• Est. Priority Date: 03/18/2009
• Status: Active Grant

• Current Assignee: Valorem Consulting Group, LLC (Reply SpA)
• Original Assignee: Identifymine Incorporated
• Inventors: Whiddett, Andrew F.
• Primary Examiner(s): WIENER, ERIC A
• Application Number:
  US12/406,957
• Publication Number:
  US 20100241973A1
• Time in Patent Office: 2,512 Days
• Field of Search: 715/863
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/0382 Plural input, i.e. interface arrangements in which a plurality of input device of the same type are in communication with a PC
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

Techniques for manipulating graphic objects in response to gestures provided with a user interface device are disclosed. The technique includes indicating when a gesture associated with the graphics objects is to use a determined action. Others gestures are locked out from using the determined action with other graphic objects. Graphic objects are manipulated on the display based on the indication that the gesture associated with one of the graphics objects is to use the determined action.

Claims

1. A method for manipulating graphic objects in response to at least one of a plurality of gestures provided through a user interface device, the method comprising:

displaying, on a display of the user interface device, a hierarchical visual tree comprising a plurality of graphic objects;

associating the plurality of gestures with a plurality of gesture handlers of a gesture engine of the user interface device, each of the plurality of gesture handlers being a unique gesture handler and having a unique internal set of matching rules;

receiving a portion of one or more gestures of the plurality of gestures, at the user interface device, the one or more gestures of the plurality of gestures being input through an input device at the user interface device;

in response to receiving the portion of the one or more gestures of the plurality of gestures at the user interface device, receiving one or more gesture indications, in the gesture engine, corresponding to the one or more gestures,

in response to receiving the one or more gesture indications in the gesture engine;

indicating that the one or more gesture indications have begun to be received;

indicating, on the display, a portion of the plurality of graphic objects that may be associated with the one or more received gesture indications; and

separate from indicating the portion of the plurality of graphic objects that may be associated with the one or more received gesture indications, indicating a selection of one or more of the plurality of graphic objects, at least one of the selected one or more of the plurality of graphic objects selected from the portion of the plurality of graphic objects that may be associated with the one or more received gesture indications;

determining, through the plurality of gesture handlers, a plurality of gesture actions based on the one or more received gesture indications, by each of the plurality of gesture handlers comparing a content of each of the received one or more gesture indications to each unique internal set of matching rules of each respective gesture handler of the plurality of unique gesture handlers, one or more of the plurality of gesture actions indicating, on the display, a manipulation of one or more of the plurality of graphic objects;

wherein the determining through the plurality of gesture handlers the plurality of gesture actions comprises selecting with the at least one of the plurality of gesture handlers a gesture action from a list of action categories consisting of;

a Select, a Matrix, and a Union category;

wherein the Matrix category includes a list of actions comprising at least one of;

Move, Resize, Rotate, Swipe, Spin, and Slide/Pan,

wherein the Union category comprises a list of actions comprising at least one of;

ScatterView manipulation, and Drag and Drop, and

wherein any action in the list of actions for any of the action categories may be re-assigned to another action category;

indicating, through two or more of the plurality of gesture handlers, to other of the plurality of gesture handlers, that the two or more of the plurality of gesture handlers are to use one of the plurality of gesture actions that has been determined by the plurality of gesture handlers to manipulate, on the display, at least one of the selected one or more of the plurality of graphic objects;

indicating, through a second one of the two or more of the plurality of gesture handlers, to the other of the plurality of gesture handlers including a first one of the two or more of the plurality of gesture handlers, that the second one of the two or more of the plurality of gesture handlers is not to use one of the plurality of gesture actions that has been determined by the plurality of gesture handlers to manipulate the at least one of the selected one or more of the plurality of graphic objects, wherein the first one of the two or more of the plurality of gesture handlers is distinct from the second one of the two or more of the plurality of gesture handlers;

locking out, through the first one of the two or more of the plurality of gesture handlers, the other of the plurality of gesture handlers including the second one of the two or more of the plurality of gesture handlers, from using the determined gesture action to manipulate, on the display, other of the plurality of graphic objects, in response to the second one of the two or more of the plurality of gesture handlers providing an indication to the other of the plurality of gesture handlers that the other of the plurality of gesture handlers are not to use the determined gesture action; and

manipulating the one or more selected graphic objects on the display with the first one of the two or more of the plurality of gesture handlers, based on the indication that first one of the two or more of the plurality of gestures handlers associated with one of the plurality of gestures is to use the determined gesture actions and in response to the first one of the two or more of the plurality of gesture handlers locking out the other of the plurality of gesture handlers including the second one of the two or more of the plurality of gesture handlers;

wherein the manipulating of the one or more selected graphic objects on the display with the first one of the two or more of the plurality of gesture handlers comprises;

ranking a listing of the action categories used to manipulate a graphics object in the action category using a predetermined ranking order of the gesture actions, and

selecting the action categories with a highest ranking to manipulate one or more of the selected graphics objects;

wherein the indicating of the portion of the plurality of the graphic objects comprises;

arranging the plurality of graphics objects in an order in the hierarchical visual tree, and

indicating, with one of the plurality of gesture handlers, when one of the gestures associated with one of the plurality of graphic objects in the hierarchical visual tree uses the determined gesture action by showing corresponding adorners or hints on the display.

2. The method as recited in claim 1 wherein the locking out, through the first one of the two or more of the plurality of gesture handlers, the other of the plurality of gesture handlers from using the determined gesture action to manipulate the other of the plurality of graphic objects further comprises:

providing a lock out indication to the other of the plurality of gesture handlers to lock out other of the one or more of the plurality of gesture handlers from using the determined gesture action to manipulate the other of the plurality of graphic objects if one of the other of the plurality of gesture handlers provides an indication that a second determined action for the other one or more gestures are not part of an action category compatible with one of the plurality of determined actions.

3. The method as recited in claim 2 wherein the locking out of the other of the plurality of gesture handlers from using the one of the plurality of the determined gesture actions to manipulate the other of the plurality of graphic objects if one of the other of the plurality of gesture handlers provides an indication that the second determined action for the other of the plurality of gestures are not part of an action category compatible with the one of the plurality of determined action further comprises:

locking out the other of the plurality of gesture handlers from using the determined action with the other of the plurality of graphic objects if a second determined action for the other of the plurality of gestures are not part of an action category identical to the determined action.

4. The method as recited in claim 1 further comprising:

adding one or more additional gestures handlers that compare the content of the received one or more gesture indications to the unique internal set of matching rules of the added one or more additional gesture handlers after the one or more additional gesture handles have been installed, and associating the added one or more addition gestures handlers with one or more of the plurality of graphic objects and at least one of the plurality of gestures.

5. The method as recited in claim 1 further comprising:

receiving one or more additional gesture indications from the user interface device;

indicating a selection of the one or more graphic objects, the graphic objects being selected from the plurality of graphic objects that may be associated with the one or more received gesture indications and that are also associated with the received one or more additional gesture indications; and

determining an action based on a combination of at least one of the received gesture indications and the received additional gesture indications.

6. The method as recited in claim 1, wherein the plurality of gesture handlers include:

(i) a first set of gesture handlers that respond to a first class of gestures, and (ii) a second set of gesture handlers that respond to a second class of gestures, and wherein locking out the other of the plurality of gesture handlers from using the determined action includes;

indicating, to the second set of gesture handlers, by at least one of the first set of gesture handlers, at least one of the second set of gestures to be locked out; and

indicating, to the first set of gesture handlers, by at least one of the second set of gesture handlers, at least one of the first set of gestures to be locked out from manipulating at least one of a size, a position or an orientation of at least one of the plurality of graphics objects on the display.

7. The method of claim 1, wherein locking out, with the first one of the two or more of the plurality of gesture handlers, the other of the plurality of gesture handlers including the second one of the two or more of the plurality of gesture handlers from using the determined gesture action to manipulate the other of the plurality of graphic objects on the display in response to the second one of the two or more of the plurality of gesture handlers providing the indication to the other of the plurality of gesture handlers that the other of the plurality of gesture handlers are not to use the determined gesture action includes:

locking out with the first one of the two or more of the plurality of gesture handlers, the other of the plurality of gesture handlers including the second one of the two or more of the plurality of gesture handlers from using the determined gesture action to manipulate a size, an orientation, or a position of the plurality of graphic objects on the display in response to the second one of the two or more of the plurality of gesture handlers providing the indication to the other of the plurality of gesture handlers that the other of the plurality of gesture handlers are not to use the determined gesture action.

8. A non-transitory computer readable medium comprising computer executable instructions for manipulating, with a gesture engine that includes a plurality of gesture handlers, graphic objects in response to at least one of a plurality of gestures generated through a user interface device, the computer-executable instructions when executed by one or more processors, perform acts comprising:

displaying, on a display of the user interface device, a hierarchical visual tree comprising a plurality of graphic objects;

associating the plurality of gestures with the plurality of gesture handlers of the gesture engine of the user interface device, each of the plurality of gesture handlers being a unique gesture handler and having a unique internal set of matching rules;

receiving a portion of one or more gestures of the plurality of gestures, at the user interface device, the one or more gestures of the plurality of gestures being input through an input device at the user interface device;

in response to receiving the portion of the one or more gestures of the plurality of gestures at the user interface device, receiving one or more gesture indications, in the gesture engine, corresponding to the one or more gestures,

in response to receiving the one or more gesture indications in the gesture engine;

indicating that the one or more gesture indications have begun to be received;

indicating, on the display, a portion of the plurality of graphic objects that may be associated with the one or more received gesture indications; and

separate from indicating the portion of the plurality of graphic objects that may be associated with the one or more received gesture indications, indicating a selection of one or more of the plurality of graphic objects, at least one of the selected one or more of the plurality of graphic objects selected from the portion of the plurality of graphic objects that may be associated with the one or more received gesture indications;

determining, through the plurality of gesture handlers, a plurality of gesture actions based on the one or more received gesture indications, by each of the plurality of gesture handlers comparing a content of each of the received one or more gesture indications to each unique internal set of matching rules of each respective gesture handler of the plurality of unique gesture handlers, one or more of the plurality of gesture actions indicating, on the display, a manipulation of one or more of the plurality of graphic objects;

wherein the determining through the plurality of gesture handlers the plurality of gesture actions comprises selecting with the at least one of the plurality of gesture handlers a gesture action from a list of action categories consisting of;

a Select, a Matrix, and a Union category;

wherein the Matrix category includes a list of actions comprising at least one of;

Move, Resize, Rotate, Swipe, Spin, and Slide/Pan,

wherein the Union category comprises a list of actions comprising at least one of;

ScatterView manipulation, and Drag and Drop, and

wherein any action in the list of actions for any of the action categories may be re-assigned to another action category;

indicating, through two or more of the plurality of gesture handlers, to other of the plurality of gesture handlers, that the two or more of the plurality of gesture handlers are to use one of the plurality of gesture actions that has been determined by the plurality of gesture handlers to manipulate, on the display, at least one of the selected one or more of the plurality of graphic objects;

indicating, through a second one of the two or more of the plurality of gesture handlers, to the other of the plurality of gesture handlers including a first one of the two or more of the plurality of gesture handlers, that the second one of the two or more of the plurality of gesture handlers is not to use one of the plurality of gesture actions that has been determined by the plurality of gesture handlers to manipulate the at least one of the selected one or more of the plurality of graphic objects, wherein the first one of the two or more of the plurality of gesture handlers is distinct from the second one of the two or more of the plurality of gesture handlers;

locking out, through the first one of the two or more of the plurality of gesture handlers, the other of the plurality of gesture handlers including the second one of the two or more of the plurality of gesture handlers, from using the determined gesture action to manipulate, on the display, other of the plurality of graphic objects, in response to the second one of the two or more of the plurality of gesture handlers providing an indication to the other of the plurality of gesture handlers that the other of the plurality of gesture handlers are not to use the determined gesture action; and

manipulating the one or more selected graphic objects on the display with the first one of the two or more of the plurality of gesture handlers, based on the indication that first one of the two or more of the plurality of gestures handlers associated with one of the plurality of gestures is to use the determined gesture actions and in response to the first one of the two or more of the plurality of gesture handlers locking out the other of the plurality of gesture handlers including the second one of the two or more of the plurality of gesture handlers;

wherein the manipulating of the one or more selected graphic objects on the display with the first one of the two or more of the plurality of gesture handlers comprises;

ranking a listing of the action categories used to manipulate a graphics object in the action category using a predetermined ranking order of the gesture actions, and

selecting the action categories with a highest ranking to manipulate one or more of the selected graphics objects;

wherein the indicating of the portion of the plurality of the graphic objects comprises;

arranging the plurality of graphics objects in an order in the hierarchical visual tree, and

indicating, with one of the plurality of gesture handlers, when one of the gestures associated with one of the plurality of graphic objects in the hierarchical visual tree uses the determined gesture action by showing corresponding adorners or hints on the display.

9. The non-transitory computer readable medium as recited in claim 8, further comprising adding additional gestures handlers and associating the added addition gestures handlers with at least one of the plurality of graphics objects.

10. The non-transitory computer readable medium as recited in claim 8, further comprising indicating a gesture begins upon receiving the received gesture and indicating when the gesture ends upon receiving the additional gesture.

11. A user interface device for receiving one or more gestures of a plurality of gestures, comprising:

a display to indicate graphic objects;

a processor; and

a memory containing instructions which when executed by the processor comprise;

displaying, on the display of the user interface device, a hierarchical visual tree comprising a plurality of graphic objects;

associating the plurality of gestures with a plurality of gesture handlers of a gesture engine of the user interface device, each of the plurality of gesture handlers being a unique gesture handler and having a unique internal set of matching rules;

receiving a portion of one or more gestures of the plurality of gestures, at the user interface device, the one or more gestures of the plurality of gestures being input through an input device at the user interface device;

in response to receiving the portion of the one or more gestures of the plurality of gestures at the user interface device, receiving one or more gesture indications, in the gesture engine, corresponding to the one or more gestures,

in response to receiving the one or more gesture indications in the gesture engine;

indicating that the one or more gesture indications have begun to be received;

indicating, on the display, a portion of the plurality of graphic objects that may be associated with the one or more received gesture indications; and

separate from indicating the portion of the plurality of graphic objects that may be associated with the one or more received gesture indications, indicating a selection of one or more of the plurality of graphic objects, at least one of the selected one or more of the plurality of graphic objects selected from the portion of the plurality of graphic objects that may be associated with the one or more received gesture indications;

determining, through the plurality of gesture handlers, a plurality of gesture actions based on the one or more received gesture indications, by each of the plurality of gesture handlers comparing a content of each of the received one or more gesture indications to each unique internal set of matching rules of each respective gesture handler of the plurality of unique gesture handlers, one or more of the plurality of gesture actions indicating, on the display, a manipulation of one or more of the plurality of graphic objects;

wherein the determining through the plurality of gesture handlers the plurality of gesture actions comprises selecting with the at least one of the plurality of gesture handlers a gesture action from a list of action categories consisting of;

a Select, a Matrix, and a Union category;

wherein the Matrix category includes a list of actions comprising at least one of;

Move, Resize, Rotate, Swipe, Spin, and Slide/Pan,

wherein the Union category comprises a list of actions comprising at least one of;

ScatterView manipulation, and Drag and Drop, and

wherein any action in the list of actions for any of the action categories may be re-assigned to another action category;

indicating, through two or more of the plurality of gesture handlers, to other of the plurality of gesture handlers, that the two or more of the plurality of gesture handlers are to use one of the plurality of gesture actions that has been determined by the plurality of gesture handlers to manipulate, on the display, at least one of the selected one or more of the plurality of graphic objects;

indicating, through a second one of the two or more of the plurality of gesture handlers, to the other of the plurality of gesture handlers including a first one of the two or more of the plurality of gesture handlers, that the second one of the two or more of the plurality of gesture handlers is not to use one of the plurality of gesture actions that has been determined by the plurality of gesture handlers to manipulate the at least one of the selected one or more of the plurality of graphic objects, wherein the first one of the two or more of the plurality of gesture handlers is distinct from the second one of the two or more of the plurality of gesture handlers;

locking out, through the first one of the two or more of the plurality of gesture handlers, the other of the plurality of gesture handlers including the second one of the two or more of the plurality of gesture handlers, from using the determined gesture action to manipulate, on the display, other of the plurality of graphic objects, in response to the second one of the two or more of the plurality of gesture handlers providing an indication to the other of the plurality of gesture handlers that the other of the plurality of gesture handlers are not to use the determined gesture action; and

manipulating the one or more selected graphic objects on the display with the first one of the two or more of the plurality of gesture handlers, based on the indication that first one of the two or more of the plurality of gestures handlers associated with one of the plurality of gestures is to use the determined gesture actions and in response to the first one of the two or more of the plurality of gesture handlers locking out the other of the plurality of gesture handlers including the second one of the two or more of the plurality of gesture handlers;

wherein the manipulating of the one or more selected graphic objects on the display with the first one of the two or more of the plurality of gesture handlers comprises;

ranking a listing of the action categories used to manipulate a graphics object in the action category using a predetermined ranking order of the gesture actions, and

selecting the action categories with a highest ranking to manipulate one or more of the selected graphics objects;

wherein the indicating of the portion of the plurality of the graphic objects comprises;

arranging the plurality of graphics objects in an order in the hierarchical visual tree, and

indicating, with one of the plurality of gesture handlers, when one of the gestures associated with one of the plurality of graphic objects in the hierarchical visual tree uses the determined gesture action by showing corresponding adorners or hints on the display.