Authoring multi-finger interactions through demonstration and composition

• US 9,218,064 B1
• Filed: 03/08/2013
• Issued: 12/22/2015
• Est. Priority Date: 09/18/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Li, Yang, Lu, Hao
• Primary Examiner(s): BARRETT, RYAN S
• Application Number:
  US13/791,830
• Time in Patent Office: 1,019 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D

Abstract

A computing device comprises a processor and an authoring tool executing on the processor. The processor receives demonstration data representative of at least one demonstration of a multi-finger gesture and declaration data specifying one or more constraints for the multi-finger gesture. The processor generates, in accordance with the demonstration data and the declaration data, a module to detect the multi-finger gesture within a computer-generated user interface.

Claims

1. A method comprising:

receiving, at a processor of a computing device, first demonstration data representative of a demonstration of a first multi-finger gesture;

receiving, at the processor, second demonstration data representative of a demonstration of a second multi-finger gesture;

receiving, at the processor, declaration data specifying one or more constraints associated with one or more of the first multi-finger gesture or the second multi-finger gesture;

outputting, by the processor and in response to input indicative of selection by the user, a first graphical video clip representation of the first multi-finger gesture along a timeline;

outputting, by the processor and in response to input indicative of selection by the user, a second graphical video clip representation of the second multi-finger gesture along the timeline;

receiving input temporally arranging the first graphical video clip and the second graphical video clip along the timeline to form a compound multi-finger gesture that includes the first multi-finger gesture and the second multi-finger gesture; and

generating, by the processor and in accordance with the first and second demonstration data, the declaration data, and temporal constraints defined by the temporal arrangement of the first graphical video clip and the second graphical video clip along the timeline, a module configured to support the compound multi-finger gesture within a computer-generated user interface.

2. The method of claim 1,

wherein receiving the input temporally arranging the first graphical video clip comprises receiving input indicative of a resizing of the first graphical video clip representation within the timeline to change the temporal constraint of the first multi-finger gesture with respect to the timeline, and

wherein generating the module further comprises generating the module in accordance with the changed temporal constraint for the first multi-finger gesture.

3. The method of claim 1, further comprising:

wherein receiving declaration data comprises receiving input indicative of a shifting of the first graphical video clip representation within the timeline to change a temporal constraint associated with a starting time or an ending time for the first multi-finger gesture with respect to the timeline, and

wherein generating the module comprises generating the module in accordance with the changed starting time and ending time for the first multi-finger gesture along the timeline.

4. The method of claim 1,

wherein receiving declaration data comprises receiving input specifying one or more software callback functions associated with the first graphical video clip representation at respective points in time along the timeline;

and

wherein generating the module comprises generating the module to invoke the software callback functions at the respective points in time upon detection of the compound multi-finger gesture.

5. The method of claim 4, further comprising:

receiving, by the processor executing in a test mode, demonstration data representative of a second demonstration of the first or second multi-finger gesture; and

outputting, with the processor, an indicator for each software callback function within the module invoked in response to the demonstration data representative of the second demonstration of the first or second multi-finger gesture.

6. The method of claim 1, further comprising outputting, by the processor, the first graphical video clip representation to include a horizontal indicator along the timeline for each finger involved in the first multi-finger gesture.

7. The method of claim 1, wherein the first graphical video clip representation includes indicators that divide the first graphical video clip representation into a plurality of segments along the timeline, wherein a transition between the segments corresponds to a change in a number of fingers involved at a corresponding point in time in the first multi-finger gesture.

8. The method of claim 1, wherein generating the module comprises

computing, by the processor, a first state machine based on the first multi-finger gesture;

computing, by the processor, a second state machine based on the second multi-finger gesture;

computing, by the processor, a third state machine for the compound multi-finger gesture by merging the first state machine with the second state machine; and

generating the module in accordance with the third state machine.

9. The method of claim 8, wherein merging the first state machine and the second state machine comprises, in response to determining, by the processor, that the first graphical video clip representation of the first multi-finger gesture and the second graphical video clip representation of the second multi-finger gesture are positioned within a same track of the timeline, linearly connecting the first state machine and the second state machine to form the third state machine.

10. The method of claim 8, wherein merging the first state machine and the second state machine comprises:

in response to determining, by the processor, that the first graphical video clip representation of the first multi-finger gesture and the second graphical video clip representation of the second multi-finger gesture are positioned within different tracks of the timeline, computing the third state machine as a set of states and transitions that are a product of states of the first state machine with states of the second state machine; and

removing any of the states of the third state machine that violate any temporal constraints specified for the first multi-finger gesture and the second multi-finger gesture within the timeline.

11. The method of claim 1, wherein generating the module comprises:

computing, from the demonstration data and the declaration data, an inference model as a probabilistic finite state machine having a plurality of states and a set of probabilistic transitions between the states that are triggered by one or more events, wherein each of the states represents a current state of a plurality of fingers in the compound multi-finger gesture and the events represent a motion associated with one or more fingers; and

generating the module to be operable in accordance with the inference model.

12. The method of claim 11, wherein computing an inference module comprises computing probabilistic distribution of possible states in the state machine for each event within the demonstration data for the compound multi-finger gesture.

13. The method of claim 1, wherein the module comprises software source code.

14. A computing device comprising:

a processor; and

an authoring tool executing on the processor,

wherein the processor is operable to;

receive first demonstration data representative of at least one demonstration of a first multi-finger gesture, second demonstration data representative of at least one demonstration of a second multi-finger gesture, and declaration data specifying one or more constraints for one or more of the first multi-finger gesture or the second multi-finger gesture;

output, for display at a display device operatively coupled to the processor, a first graphical video clip representation of a first multi-finger gesture along a timeline and a second graphical video clip representation of a second multi-finger gesture along the timeline;

receive input temporally arranging the first graphical video clip and the second graphical video clip along the timeline of the user interface to form a compound multi-finger gesture; and

generate, in accordance with the first and second demonstration data, the declaration data, and temporal constraints defined by the temporal arrangement of the first graphical video clip and the second graphical video clip within the timeline of the user interface, a module to support the compound multi-finger gesture within a computer-generated user interface.

15. The computing device of claim 14, wherein the processor is operable to receive the the input temporally arranging the first graphical video clip as input indicative of a resizing or shifting of the first graphical video clip representation within the timeline to specify a temporal constraint associated with a starting time or an ending time for first the multi-finger gesture.

16. The computing device of claim 14,

wherein the processor is operable to receive input specifying one or more software callback functions associated with the graphical video clip representation at respective points in time along the timeline, and

wherein the processor is operable to generate the module to invoke the software callback functions at the respective points in time upon detection of the compound multi-finger gesture.

17. A non-transitory computer-readable storage medium comprising instructions that, when executed, cause one or more processors of a computing device to:

receive first demonstration data representative of a demonstration of a first multi-finger gesture;

receive second demonstration data representative of at least one demonstration of a second multi-finger gesture;

receive declaration data specifying one or more constraints associated with one or more of the first multi-finger gesture or the second multi-finger gesture;

output, for display, a first graphical video clip representation of the first multi-finger gesture along a timeline;

output, for display, a second graphical video clip representation of the second multi-finger gesture along the timeline;

receive input temporally arranging the first graphical video clip and the second graphical video clip along the timeline to form a compound multi-finger gesture that includes the first multi-finger gesture and the second multi-finger gesture; and

generate, in accordance with the first and second demonstration data, the declaration data and temporal constraints defined by the temporal arrangement of the first graphical video clip and the second graphical video clip within the timeline of the user interface, a module to support the compound multi-finger gesture within a computer-generated user interface.

18. The non-transitory computer-readable storage medium of claim 17,

wherein the input temporally arranging the first graphical video clip comprises a input indicative of a change to the temporal constraint for the first multi-finger gesture with respect to the timeline, and

wherein the instructions cause the one or more processors to generate the module in accordance with the changed temporal constraint for the first multi-finger gesture.

19. The non-transitory computer-readable storage medium of claim 17, wherein the instructions cause the one or more processors to:

receive the declaration data as input indicative of a resizing or shifting of the graphical video clip representation within the timeline to specify a temporal constraint associated with a starting time or an ending time for the first multi-finger gesture; and

generate the module in accordance with the changed starting time and ending time for the first multi-finger gesture along the timeline.