GENERAL SPATIAL-GESTURE GRAMMAR USER INTERFACE FOR TOUCHSCREENS, HIGH DIMENSIONAL TOUCH PAD (HDTP), FREE-SPACE CAMERA, AND OTHER USER INTERFACES

• US 20160378198A1
• Filed: 09/09/2016
• Published: 12/29/2016
• Est. Priority Date: 03/07/2011
• Status: Active Grant

• Current Assignee: NRI R&D Patent Licensing, LLC (Advanced Touchscreen and Gesture Technologies, LLC)
• Original Assignee: NRI R&D Patent Licensing, LLC (Advanced Touchscreen and Gesture Technologies, LLC)
• Inventors: Ludwig, Lester F.
• Granted Patent: US 10,073,532 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06F 40/268 Morphological analysis

Abstract

A system for a spatial-gesture user interface employing grammatical rules at various levels. Various distinct subset of the gestemes can be concatenated in space and time to construct a distinct gestures. Real-time spatial-gesture information measured by a spatial-gesture user interface is processed to at least a recognized sequence of specific gestemes and that the sequence of gestemes that the user'"'"'s execution a gesture has been completed. The specific gesture rendered by the user is recognized according to the sequence of gestemes. Many additional features are then provided from this foundation, including gesture grammars, structured-meaning gesture-lexicon, imposed interpretations, context, and the use of gesture-rendering prosody. The invention can be used to provide very general spatial-gesture grammar user interface for touchscreens, high dimensional touch pad (hdtp), free-space camera, and other user interfaces.

Claims

1. A system implementing a spatial-gesture user interface, the system comprising:

at least one processor for running at least one algorithm, the least one algorithm for

receiving an input stream of spatial measurements made over an interval of time by at least one spatial user interface sensor;

recognizing a sequence of individual gestemes from the input stream of measurements, each gesteme comprising primitive gesture segment elements;

determining from the sequence of gestemes that the user'"'"'s execution a first gesture has been completed, and determining from the sequence of gestemes that a first specific gesture is represented by the first sequence of gestemes;

determining from the sequence of gestemes that the user'"'"'s execution a second gesture has been completed, and determining from the sequence of gestemes that a second specific gesture is represented by the second sequence of gestemes;

imposing an interpretation of the first specific gesture and second specific gesture based on at least a first grammatical rule,

wherein the algorithm recognizes the first specific gesture and the second specific gesture according to the particular sequence of gestemes recognized from the input stream of measurements made over an interval of time by the at least one spatial user interface sensor, and

wherein the algorithm reports at least the interpretation as output information.

2. The system of claim 1 wherein first specific gesture and second specific gesture are regarded as a sequence of gestures.

3. The system of claim 1 wherein prosody is used as an input into the first grammatical rule.

4. The system of claim 1 wherein context is context is determined at least in part from prosody.

5. The system of claim 1 wherein a context is determined at least in part from at least one earlier-provided gesteme.

6. The system of claim 1 wherein a context is determined at least in part from at least one earlier-provided gesture.

7. The system of claim 1 wherein the first grammatical rule comprises a structured-meaning gesture-lexicon framework for interpreting the sequence of gestures.

8. The system of claim 1 wherein context is context is determined at least in part from by the sequence of gestemes.

9. The system of claim 1 wherein at least a second grammatical rule is used in the recognition of at least one of the first specific gestures and the second specific gesture from the sequence of gestemes.

10. The system of claim 8 wherein the second grammatical rule comprises a structured-meaning gesture-lexicon framework for interpreting the sequence of gestemes.

11. The system of claim 8 wherein prosody is used as an input into the second grammatical rule.

13. The system of claim 1 wherein prosody is used as an input into the first grammatical rule.

14. The system of claim 1 wherein context is determined at least in part from prosody.

15. The system of claim 1 wherein a context is determined at least in part from at least one earlier-provided gesteme.

16. The system of claim 1 wherein a context is determined at least in part from at least one earlier-provided gesture.

17. The system of claim 1 wherein the first grammatical rule comprises a structured-meaning gesture-lexicon framework for interpreting the sequence of gestures.

18. The system of claim 1 wherein context is context is determined at least in part by the sequence of gestemes.

19. The system of claim 1 wherein at least a second grammatical rule is used in the recognition of at least one of the first specific gestures and the second specific gesture from the sequence of gestemes.

20. The system of claim 8 wherein the second grammatical rule comprises a structured-meaning gesture-lexicon framework for interpreting the sequence of gestemes.

21. The system of claim 8 wherein prosody is used as an input into the second grammatical rule.

12. A system for controlling an application operating on a processor with a spatial-gesture user interface, the system comprising:

A processor for receiving a gesture information measured by a spatial-gesture user interface and for executing an algorithm for

recognizing gestemes comprising distinct primitive gesture segments;

recognizing gestures from sequences of gestemes, and from this transforming the sequences of gestemes into a sequence of gestures; and

imposing an interpretation of the first specific gesture and second specific gesture based on at least a first grammatical rule,

wherein the algorithm transforms at least two of

(a) the sequence of gestures,

(b) the imposed interpretation, and

(c) a provided context

into a particular command for use in controlling a software application; and

wherein the system transmits the particular command to the application.