Method to detect and assist user intentions with real time visual feedback based on interaction language constraints and pattern recognition of sensory features

• US 20090006101A1
• Filed: 10/24/2007
• Published: 01/01/2009
• Est. Priority Date: 06/28/2007
• Status: Active Grant

• Current Assignee: Sovereign Peak Ventures, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: Matsushita Electric Industrial Company Limited (Panasonic Holdings Corporation)
• Inventors: Rigazio, Luca, Kryze, David, Morin, Philippe
• Granted Patent: US 8,170,869 B2
• Time in Patent Office: Days
• US Class Current: 704/275
• CPC Class Codes: G06F 3/0237 using prediction or retrieval techniques
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces

Abstract

A language model back-off system can be used with a user interface employing one or more language models to constrain navigation of selectable user interface input components. A user input interpretation module receives user input and interprets the user input to determine if a selection is made of one or more user interface input components. If a selection is not made, the user input interpretation module determines whether conditions are met for backing off one or more language models employed to constrain navigation of the user interface input components. If the conditions are met, a language model back-off module backs off the one or more language models.

Claims

1. A language model back-off system for use with a user interface employing one or more language models to constrain navigation of selectable user interface input components, the system comprising:

a user input interpretation module receiving user input and interpreting the user input to determine if a selection is made of one or more user interface input components, wherein said user input interpretation module, if a selection is not made, determines whether conditions are met for backing off one or more language models employed to constrain navigation of the user interface input components; and

a language model back-off module backing off the one or more language models if the conditions are met.

2. The system of claim 1, further comprising a user interface displaying the user interface input components with a focus indicator.

3. The system of claim 2, further comprising a navigation constraint module employing the one or more language models, based on previous selections, to constrain navigation of the selectable user interface input components.

4. The system of claim 3, wherein said navigation constraint module constrains navigation of the user interface input components by controlling sizes of the components based on probabilities of the components being next selected as determined by employing the one or more language models.

5. The system of claim 3, wherein said navigation constraint module constrains navigation of the user interface input components by controlling attraction of a pointer to the components based on probabilities of the components being next selected as determined by employing the one or more language models.

6. The system of claim 1, wherein said language model back-off module backs off the one or more language models by adjusting weights of probabilities that are computed using multiple language models.

7. The system of claim 6, wherein said language model back-off module adjusts the weights to ensure that probabilities computed by employing a more highly constrained language model are given less weight while probabilities computed by employing a less highly constrained language model are given more weight.

8. The system of claim 1, wherein said user input interpretation module determines whether the conditions are met based on a time since a previous selection.

9. The system of claim 1, wherein said user input interpretation module determines whether the conditions are met by employing a probabilistic model taking into account a feature set of sensed user interaction with an input device of the user interface.

10. The system of claim 9, wherein said user input interpretation module employs a feature set including at least one of the following sensory inputs:

(a) a finger position;

(b) a click event;

(c) an acceleration derivative;

(d) a holding integral;

(e) dual finger positions and their multiple derivatives;

(f) dual click events;

(g) dual finger pressure values;

(h) input device acceleration and its integrals;

or

(i) holding sensors outputs and their derivatives.

11. A language model back-off method for use with a user interface employing one or more language models to constrain navigation of selectable user interface input components, the method comprising:

receiving user input and interpreting the user input to determine if a selection is made of one or more user interface input components;

if a selection is not made, determining whether conditions are met for backing off one or more language models employed to constrain navigation of the user interface input components; and

if the conditions are met, backing off the one or more language models.

12. The method of claim 11, further comprising displaying the user interface input components with a focus indicator.

13. The method of claim 12, further comprising employing the one or more language models, based on previous selections, to constrain navigation of the selectable user interface input components.

14. The method of claim 13, further comprising constraining navigation of the user interface input components by controlling sizes of the components based on probabilities of the components being next selected as determined by employing the one or more language models.

15. The method of claim 13, further comprising constraining navigation of the user interface input components by controlling attraction of a pointer to the components based on probabilities of the components being next selected as determined by employing the one or more language models.

16. The method of claim 11, further comprising backing off the one or more language models by adjusting weights of probabilities that are computed using multiple language models.

17. The method of claim 16, further comprising adjusting the weights to ensure that probabilities computed by employing a more highly constrained language model are given less weight while probabilities computed by employing a less highly constrained language model are given more weight.

18. The method of claim 11, further comprising determining whether the conditions are met based on a time since a previous selection.

19. The method of claim 11, further comprising determining whether the conditions are met by employing a probabilistic model taking into account a feature set of sensed user interaction with an input device of the user interface.

20. The method of claim 19, further comprising employing a feature set including at least one of the following sensory inputs:

(a) a finger position;

(b) a click event;

(c) an acceleration derivative;

(d) a holding integral;

(e) dual finger positions and their multiple derivatives;

(f) dual click events;

(g) dual finger pressure values;

(h) input device acceleration and its integrals;

or

(i) holding sensors outputs and their derivatives.

21. A language model back-off system for use with a user interface employing one or more language models to constrain navigation of selectable user interface input components, the system comprising:

a user interface displaying the user interface input components with a focus indicator;

a navigation constraint module employing the one or more language models, based on previous selections, to constrain navigation of the selectable user interface input components;

a user input interpretation module receiving user input and interpreting the user input to determine if a selection is made of the one or more user interface input components, wherein said user input interpretation module, if a selection is not made, determines whether conditions are met for backing off the one or more language models based on;

(a) a time since a previous selection; and

(b) a probabilistic model taking into account a feature set of sensed user interaction with an input device of the user interface; and

a language model back-off module backing off the one or more language models if the conditions are met by adjusting weights of probabilities that are computed using multiple language models to ensure that probabilities computed by employing a more highly constrained language model are given less weight while probabilities computed by employing a less highly constrained language model are given more weight.