GESTURE-ACTIVATED INPUT USING AUDIO RECOGNITION

• US 20120260177A1
• Filed: 09/30/2011
• Published: 10/11/2012
• Est. Priority Date: 04/08/2011
• Status: Abandoned Application

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Sehrer, Trevor
• Application Number:
  US13/251,054
• Publication Number:
  US 20120260177A1
• Time in Patent Office: Days
• US Class Current: 715/727
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/166 Editing, e.g. inserting or deleting

Abstract

In one example, a method includes, displaying, at a presence-sensitive screen of a computing device, an input field in a region of a graphical user interface (GUI). The method further includes receiving, at the presence-sensitive screen, user input including one or more gestures to select the input field, wherein the one or more gestures to select the input field include motion at a location of the presence-sensitive screen that corresponds to the region of the GUI displaying the input field. The method also includes, while the input field is selected, detecting, by the computing device, an audio signal and identifying, by the computing device, at least one input value based on the detected audio signal. The method also includes assigning, by the computing device, the at least one input value to the input field in the GUI.

Claims

1. A method comprising:

displaying, at a region of a presence-sensitive screen of a computing device, an input field;

receiving, at the region of the presence-sensitive screen of the computing device, user input comprising one or more gestures to select the input field, wherein the one or more gestures to select the input field comprise motion at the region of the presence-sensitive screen, wherein the one or more gestures comprise positioning at least one input unit at or near the location of the presence-sensitive screen, wherein the input unit is detectable by the presence-sensitive screen only when at or near the location of the presence-sensitive screen;

in response to receiving the user input, determining, by the computing device, that the input field is selected based on the input unit currently being detected by the presence-sensitive screen;

in response to determining that the input field is selected, begin receiving, by the computing device, an audio signal;

in response to determining the input field is no longer selected based on the input unit no longer being detected by the presence-sensitive screen, stop receiving, by the computing device, the audio signal;

identifying, by the computing device, at least one input value based on the received audio signal; and

assigning, by the computing device, the at least one input value to the input field.

2. The method of claim 1, wherein identifying the at least one input value based on the received audio signal further comprises:

defining, by the computing device and based on the received audio signal, digital audio data that comprises a digital representation of the received audio signal;

dividing, by the computing device, the digital audio data into one or more digital audio data segments;

selecting, by the computing device, a subset of the one or more digital audio segments;

providing, by the computing device, a grammar comprising a plurality of symbols associated with a plurality of pre-defined digital audio data segments, wherein the grammar enables the computing device to identify a symbol based on an audio data segment; and

selecting, by the computing device, one or more symbols of the grammar based on a match of the one or more selected digital audio data segments with the plurality of pre-defined digital audio data segments.

3. The method of claim 2, wherein the match indicates a degree of similarity, within a similarity interval, between the selected digital audio data segments and the plurality of pre-defined digital audio data segments.

4. The method of claim 3, wherein the degree of similarity comprises a ratio of equivalence between the selected digital audio data segments and one or more pre-defined digital audio segments of the plurality of digital audio data segments.

5. The method of claim 2, wherein the selected one or more symbols comprise one or more characters of an alphabet.

6. The method of claim 2, wherein the selected one or more symbols comprise one or more phonemes, words, or phrases of a language.

7. The method of claim 2, wherein the one or more symbols comprise one or more images, videos, or visual objects.

9. The method of claim 1, wherein assigning, by the computing device, the at least one input value to the input field further comprises:

determining, by the computing device, an input field operation based on the input value, wherein the input field operation changes a visual appearance of the input field.

10. The method of claim 1, wherein assigning, by the computing device, the at least one input value to the input field further comprises:

determining, by the computing device, an input field operation based on the input value, wherein the input field operation changes a visual appearance of the at least one input value assigned to the input field.

11. The method of claim 1, further comprising:

determining, by the computing device, that the input field is no longer selected in response to the one or more gestures no longer being detected, by the presence-sensitive screen, at or near the presence-sensitive screen.

12. The method of claim 1, wherein the presence-sensitive screen comprises a touch-sensitive screen.

13. The method of claim 1,

wherein the input value is initially displayed at a first location of the presence-sensitive screen, and wherein the method further comprises:

receiving, by the computing device, a second user input comprising one or more gestures, wherein the one or more gestures of the second user input comprise motion from the first location of the presence-sensitive screen to a second, different location of the presence-sensitive screen; and

displaying, by the presence-sensitive screen of the computing device, the at least one input value at the second location of the presence-sensitive screen.

14. The method of claim 1, wherein assigning, by the computing device, the at least one input value to the input field further comprises:

displaying, by the presence-sensitive screen of the computing device, the at least one input value at the region of the presence-sensitive screen that displays the input field.

15. The method of claim 1, wherein assigning, by the computing device, the at least one input value to the input field further comprises:

assigning the at least one input value to the input field upon determining that the input field is no longer selected.

16. The method of claim 1, wherein assigning, by the computing device, the at least one input value to the input field further comprises:

assigning the at least one input value to the input field while the input field is selected.

17. The method of claim 1, further comprising:

displaying, by a projection device of the computing device, a graphical representation of the input field at a display surface.

18. The method of claim 17, wherein the projection device comprises a pico projector.

19. The method of claim 1, further comprising:

determining, by the computing device, that the input field is no longer selected.

20. The method of claim 1, wherein the input field comprises a text box control or a free form drawing control.

21. The method of claim 1, wherein assigning, by the computing device, the at least one input value to the input field further comprises assigning the at least one input value to any location of the input field.

8. (canceled)

22. A computer-readable storage medium encoded with instructions that, when executed by one or more processors of a computing device, cause the one or more processors to perform operations comprising:

displaying, at a region of a presence-sensitive screen, an input field;

receiving, at the region of the presence-sensitive screen, user input comprising one or more gestures to select the input field, wherein the one or more gestures to select the input field comprise motion at the region of the presence-sensitive screen, wherein the one or more gestures comprise positioning at least one input unit at or near the location of the presence-sensitive screen, wherein the input unit is detectable by the presence-sensitive screen only when at or near the location of the presence-sensitive screen;

in response to receiving the user input, determining the input field is selected based on the input unit currently being detected by the presence-sensitive screen,

in response to determining that the input field is selected, begin receiving, an audio signal;

in response to determining the input field is no longer selected based on the input unit no longer being detected by the presence-sensitive screen, stop receiving the audio signal;

identifying at least one input value based on the received audio signal; and

assigning the at least one input value to the input field.

23. A computing device, comprising:

one or more processors;

an input device;

a presence-sensitive screen that is operable to display an input field at a region the presence-sensitive screen;

wherein the presence-sensitive screen is operable to receive, at the region of the presence-sensitive screen, user input comprising one or more gestures to select the input field, wherein the one or more gestures to select the input field comprise motion at a the region of the presence-sensitive screen, wherein the one or more gestures comprise positioning at least one input unit at or near the location of the presence-sensitive screen, wherein the input unit is detectable by the presence-sensitive screen only when at or near the location of the presence-sensitive screen;

an input field module operable by the one or more processors to, in response to receiving the user input, determining that the input field is selected based on the input unit currently being detected by the presence-sensitive screen;

wherein the input device is operable to begin receiving an audio signal in response to determining the input field is selected;

wherein input device is operable by the one or more processors to, in response to determining the input field is no longer being detected by the presence-sensitive screen, stop receiving the audio signal;

an audio conversion module operable by the one or more processors to identify at least one input value based on the received audio signal; and

wherein the input field module is operable by the one or more processors to assign the at least one input value to the input field.