Predictive pre-recording of audio for voice input

• US 8,195,319 B2
• Filed: 09/30/2011
• Issued: 06/05/2012
• Est. Priority Date: 03/26/2010
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Kristjansson, Trausti, Lloyd, Matthew I.
• Primary Examiner(s): ELBIN, JESSE A
• Application Number:
  US13/250,533
• Publication Number:
  US 20120022675A1
• Time in Patent Office: 249 Days
• Field of Search: 700 29- 31, 700/44, 700/47, 700/52, 700/53, 700/94, 704/275, 455/414.3, 455/414.4, 715/727, 715/728, 715/863, 702/141, 702/187
• US Class Current: 700/94
• CPC Class Codes: G06F 1/1626 with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for providing predictive pre-recording of audio for voice input. In one aspect, a method includes establishing, as input data, state data that references a state of a mobile device and sensor data that is sensed by one or more sensors of the mobile device, applying a rule or a probabilistic model to the input data, inferring, based on applying the rule or the probabilistic model to the input data, that a user of the mobile device is likely to initiate voice input, and invoking one or more functionalities of the mobile device in response to inferring that the user is likely to initiate voice input.

Claims

1. A system comprising:

one or more computers; and

a computer-readable medium coupled to the one or more computers having instructions stored thereon which, when executed by the one or more computers, cause the one or more computers to perform operations comprising;

applying a rule or a probabilistic model to state data that references a state of a mobile device and sensor data that is sensed by one or more sensors of the mobile device;

inferring, based on applying the rule or the probabilistic model to the state data and the sensor data, that a user of the mobile device is likely to initiate voice input;

commencing a background audio recording in response to inferring that the user is likely to initiate voice input; and

transmitting (i) an audio signal that encodes the background audio recording and an utterance, and (ii) duration data that indicates a duration of the background audio recording encoded in the audio signal.

2. The system of claim 1, wherein the state data comprises data that references whether a display of the mobile device is turned on or turned off, data that references whether the mobile device is operating in a locked mode or in an unlocked mode, data that references one or more applications that are executing on the mobile device, data that references whether a voice search application is executing on the mobile device, data that references whether a field selected by the user is enabled for voice text entry, or data that references whether the user is operating an input mechanism of the mobile device.

3. The system of claim 1, wherein the state data comprises data that references a current state or a recent state of the mobile device.

4. The system of claim 1, wherein the sensor data comprises data that is sensed by a keypad sensor of the mobile device, data that is sensed by a position determination sensor of the mobile device, data that is sensed by an accelerometer of the mobile device, data that is sensed by a magnetometer of the mobile device, data that is sensed by a light sensor of the mobile device, data that is sensed by a proximity sensor of the mobile device, data that is sensed by a capacitive sensor of the mobile device, or data that is sensed by a touch sensor of the mobile device.

5. The system of claim 1, wherein:

applying the rule or the probabilistic model to sensor data further comprises applying the rule or the probabilistic model to data that is sensed by a first sensor of a mobile device;

commencing the background audio recording further comprises activating a second sensor of the mobile device in response to inferring that the user is likely to initiate voice input.

6. The system of claim 5, wherein the operations further comprise:

applying an additional rule or an additional probabilistic model to the data that is sensed by the first sensor of the mobile device, and additional data that is sensed by the second sensor of the mobile device;

inferring, based on applying the additional rule or the additional probabilistic model to the data and the additional data, that the user remains likely to initiate voice input; and

commencing the background audio recording in response to inferring that the user remains likely to initiate voice input.

7. The system of claim 1, wherein:

applying the rule or the probabilistic model to the state data and the sensor data further comprises determining a score associated with the state data and the sensor data; and

inferring that the user is likely to initiate voice input further comprises determining that the score associated with the state data and the sensor data satisfies a predefined threshold.

8. The system of claim 1, wherein applying a rule or a probabilistic model to the state data and the sensor data further comprises applying a rule to the state data and the sensor data.

9. The system of claim 1, wherein applying a rule or a probabilistic model to the state data and the sensor data further comprises applying a probabilistic model to the state data and the sensor data.

10. The system of claim 1, wherein the operations further comprise:

inferring, based on applying the rule or the probabilistic model to the state data and the sensor data, that it is unclear whether the user is likely to initiate voice input;

instructing the mobile device to collect additional state data or additional sensor data in response to inferring that it is unclear whether the user is likely to initiate voice input; and

applying the rule or the probabilistic model to the state data, the sensor data, and the additional state data or the additional sensor data,

wherein inferring that the user is likely to initiate voice input is further based on applying the rule or the probabilistic model to the state data, the sensor data, and the additional state data or the additional sensor data.

11. The system of claim 1, wherein the operations further comprise generating the rule or training the probabilistic model using historical state data or historical sensor data.

12. The system of claim 1, wherein inferring that the user is likely to initiate voice input further comprises inferring, by the mobile device, that the user is likely to initiate voice input.

13. The system of claim 1, wherein inferring that the user is likely to initiate voice input further comprises inferring, by a server, that the user is likely to initiate voice input.

14. The system of claim 1, wherein the operations further comprise:

communicating the audio signal and the duration data between the mobile device and a server, wherein the duration data indicates the duration of the background audio recording encoded in the audio signal that is recorded before the user begins speaking.

15. A computer-implemented method comprising:

applying, by a mobile device, a rule or a probabilistic model to state data that references a state of the mobile device and sensor data that is sensed by one or more sensors of the mobile device;

inferring, based on applying the rule or the probabilistic model to the state data and the sensor data, that a user of the mobile device is likely to initiate voice input;

commencing, by the mobile device, a background audio recording in response to inferring that the user is likely to initiate voice input; and

transmitting, by the mobile device, (i) an audio signal that encodes the background audio recording and an utterance, and (ii) duration data that indicates a duration of the background audio recording encoded in the audio signal.

16. The method of claim 15, wherein:

applying the rule or the probabilistic model to sensor data further comprises applying the rule or the probabilistic model to data that is sensed by a first sensor of the mobile device, and

commencing the background audio recording further comprises activating a second sensor of the mobile device in response to inferring that the user is likely to initiate voice input.

17. The method of claim 16, further comprising:

applying an additional rule or an additional probabilistic model to the data that is sensed by the first sensor of the mobile device, and additional data that is sensed by the second sensor of the mobile device;

inferring, based on applying the additional rule or the additional probabilistic model to the data and the additional data, that the user remains likely to initiate voice input; and

commencing the background audio recording in response to inferring that the user remains likely to initiate voice input.

18. The method of claim 15, wherein:

applying the rule or the probabilistic model to the state data and the sensor data further comprises determining a score associated with the state data and the sensor data; and

inferring that the user is likely to initiate voice input further comprises determining that the score associated with the state data and the sensor data satisfies a predefined threshold.

19. A non-transitory computer storage medium encoded with a computer program, the program comprising instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

applying a rule or a probabilistic model to state data that references a state of a mobile device and sensor data that is sensed by one or more sensors of the mobile device;

inferring, based on applying the rule or the probabilistic model to the state data and the sensor data, that a user of the mobile device is likely to initiate voice input;

commencing a background audio recording in response to inferring that the user is likely to initiate voice input; and

transmitting (i) an audio signal that encodes the background audio recording and an utterance, and (ii) duration data that indicates a duration of the background audio recording encoded in the audio signal.