Predictive pre-recording of audio for voice input

• US 8,504,185 B2
• Filed: 07/31/2012
• Issued: 08/06/2013
• Est. Priority Date: 03/26/2010
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Kristjansson, Trausti, Lloyd, Matthew I.
• Primary Examiner(s): ELBIN, JESSE A
• Application Number:
  US13/563,504
• Publication Number:
  US 20120296655A1
• Time in Patent Office: 371 Days
• Field of Search: 700 29- 30, 700/47, 700/52, 700/94, 704/275, 702/141, 702/150, 4554142-4144
• US Class Current: 700/94
• CPC Class Codes: G06F 1/1626 with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for providing predictive pre-recording of audio for voice input. In one aspect, a method includes obtaining sensor data from one or more sensors of a mobile device while the mobile device is operating in an inactive state, determining that a user of the mobile device is interacting with the mobile device based on the sensor data, invoking voice input functionality of the mobile device in response to determining that the user of the mobile device is interacting with the mobile device, detecting a voice input, and activating the mobile device in response to detecting the voice input.

Claims

1. A system comprising:

one or more computers; and

a computer-readable medium coupled to the one or more computers having instructions stored thereon which, when executed by the one or more computers, cause the one or more computers to perform operations comprising;

obtaining sensor data from one or more sensors of a mobile device, while the mobile device is operating in a mode in which the one or more sensors of the mobile device are enabled, and audio recording functionality and additional functionality of the mobile device are both disabled;

determining that a user of the mobile device is interacting with the mobile device based on the sensor data, while the mobile device is operating in the mode in which the one or more sensors of the mobile device are enabled, and the audio recording functionality and the additional functionality of the mobile device are both disabled;

placing the mobile device in a mode in which the one or more sensors of the mobile device and the audio recording functionality of the mobile device are both enabled, and the additional functionality of the mobile device is disabled, in response to determining that the user of the mobile device is interacting with the mobile device;

detecting a voice input while the mobile device is operating in the mode in which the one or more sensors of the mobile device and the audio recording functionality of the mobile device are both enabled, and the additional functionality of the mobile device is disabled; and

determining whether to place the mobile device in a mode in which the one or more sensors of the mobile device, the audio recording functionality, and the additional functionality of the mobile device are all enabled, in response to detecting the voice input,

wherein the mobile device is operating in a mode in which the additional functionality of the mobile device is disabled when at least one of (i) a display of the mobile device is powered off, (ii) the mobile device is operating in a locked mode, and (iii) no applications are executing on the mobile device.

2. The system of claim 1, wherein obtaining the sensor data from the one or more sensors of the mobile device, while the mobile device is operating in the mode in which the one or more sensors of the mobile device are enabled, and the audio recording functionality and the additional functionality of the mobile device are both disabled comprises continuously obtaining the sensor data from the one or more sensors of the mobile device, while the mobile device is operating in the mode in which the one or more sensors of the mobile device are enabled, and the audio recording functionality and the additional functionality of the mobile device are both disabled.

3. The system of claim 1, wherein the sensor data comprises data that is sensed by a keypad sensor of the mobile device, data that is sensed by a position determination sensor of the mobile device, data that is sensed by an accelerometer of the mobile device, data that is sensed by a magnetometer of the mobile device, data that is sensed by a light sensor of the mobile device, data that is sensed by a proximity sensor of the mobile device, data that is sensed by a capacitive sensor of the mobile device, or data that is sensed by a touch sensor of the mobile device.

4. The system of claim 1, wherein the operations further comprise:

obtaining state data that references a state of the mobile device; and

determining whether the mobile device is operating in a mode in which the additional functionality of the mobile device is disabled based on the state data.

5. The system of claim 4, wherein the state data comprises data that references whether a display of the mobile device is turned on or turned off, data that references whether the mobile device is operating in a locked mode or in an unlocked mode, data that references whether an application is executing on the mobile device, or data that references whether the user is operating an input mechanism of the mobile device.

6. The system of claim 1, wherein determining that the user of the mobile device is interacting with the mobile device based on the sensor data comprises determining that the user is holding and moving the mobile device based on the sensor data.

7. The system of claim 1, wherein determining that the user of the mobile device is interacting with the mobile device based on the sensor data comprises:

applying a rule or a probabilistic model to the sensor data; and

inferring that the user of the mobile device is interacting with the mobile device based on applying the rule or the probabilistic model to the sensor data.

8. The system of claim 7, wherein the operations further comprise generating the rule or training the probabilistic model using historical sensor data.

9. The system of claim 8, wherein generating the rule or training the probabilistic model using historical sensor data comprises:

receiving historical input data collected by other mobile devices; and

training the probabilistic model using the historical input data associated with the mobile device and the historical input data collected by the other mobile devices.

10. The system of claim 1, wherein determining that the user of the mobile device is interacting with the mobile device based on the sensor data comprises:

determining a score associated with the sensor data; and

determining that the score associated with the sensor data satisfies a predefined threshold.

11. The system of claim 1, wherein placing the mobile device in a mode in which one or more sensors of the mobile device and the audio recording functionality of the mobile device are both enabled, and the additional functionality of the mobile device is disabled, in response to determining that the user of the mobile device is interacting with the mobile device comprises commencing an audio recording in response to determining that the user of the mobile device is interacting with the mobile device.

12. The system of claim 1, wherein determining whether to place the mobile device in a mode in which the one or more sensors of the mobile device, the audio recording functionality, and the additional functionality of the mobile device are all enabled comprises determining whether to operate the mobile device in an unlocked mode.

13. The system of claim 1, wherein determining whether to place the mobile device in a mode in which the one or more sensors of the mobile device, the audio recording functionality, and the additional functionality of the mobile device are all enabled comprises determining whether to invoke an application on the mobile device.

14. The system of claim 1, wherein determining whether to place the mobile device in a mode in which the one or more sensors of the mobile device, the audio recording functionality, and the additional functionality of the mobile device are all enabled comprises determining whether to invoke a functionality of the mobile device associated with the voice input.

15. A system comprising:

one or more computers; and

a computer-readable medium coupled to the one or more computers having instructions stored thereon which, when executed by the one or more computers, cause the one or more computers to perform operations comprising;

obtaining sensor data from one or more sensors of a mobile device, while the mobile device is operating in a mode in which the one or more sensors of the mobile device are enabled, and audio recording functionality and additional functionality of the mobile device are both disabled;

determining that a user of the mobile device is interacting with the mobile device based on the sensor data, while the mobile device is operating in the mode in which the one or more sensors of the mobile device are enabled, and the audio recording functionality and the additional functionality of the mobile device are both disabled;

placing the mobile device in a mode in which the one or more sensors of the mobile device and the audio recording functionality of the mobile device are both enabled, and the additional functionality of the mobile device is disabled, in response to determining that the user of the mobile device is interacting with the mobile device;

detecting a voice input while the mobile device is operating in the mode in which the one or more sensors of the mobile device and the audio recording functionality of the mobile device are both enabled, and the additional functionality of the mobile device is disabled; and

determining whether to place the mobile device in a mode in which the one or more sensors of the mobile device, the audio recording functionality, and the additional functionality of the mobile device are all enabled, in response to detecting the voice input, wherein determining whether to place the mobile device in the mode in which the one or more sensors of the mobile device, the audio recording functionality, and the additional functionality of the mobile device are all enabled comprises determining whether to power on a display of the mobile device.

16. A non-transitory computer storage medium encoded with a computer program, the program comprising instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

obtaining sensor data from one or more sensors of a mobile device, while the mobile device is operating in a mode in which the one or more sensors of the mobile device are enabled, and audio recording functionality and additional functionality of the mobile device are both disabled;

determining that a user of the mobile device is interacting with the mobile device based on the sensor data, while the mobile device is operating in the mode in which the one or more sensors of the mobile device are enabled, and the audio recording functionality and the additional functionality of the mobile device are both disabled;

placing the mobile device in a mode in which the one or more sensors of the mobile device and the audio recording functionality of the mobile device are both enabled, and the additional functionality of the mobile device is disabled, in response to determining that the user of the mobile device is interacting with the mobile device;

detecting a voice input while the mobile device is operating in the mode in which the one or more sensors of the mobile device and the audio recording functionality of the mobile device are both enabled, and the additional functionality of the mobile device is disabled; and

determining whether to place the mobile device in a mode in which the one or more sensors of the mobile device, the audio recording functionality, and the additional functionality of the mobile device are all enabled, in response to detecting the voice input,

wherein the mobile device is operating in a mode in which the additional functionality of the mobile device is disabled when at least one of (i) a display of the mobile device is powered off, (ii) the mobile device is operating in a locked mode, and (iii) no applications are executing on the mobile device.

17. A computer-implemented method comprising:

obtaining sensor data from one or more sensors of a mobile device, while the mobile device is operating in a mode in which the one or more sensors of the mobile device are enabled, and audio recording functionality and additional functionality of the mobile device are both disabled;

determining that a user of the mobile device is interacting with the mobile device based on the sensor data, while the mobile device is operating in the mode in which the one or more sensors of the mobile device are enabled, and the audio recording functionality and the additional functionality of the mobile device are both disabled;

placing the mobile device in a mode in which the one or more sensors of the mobile device and the audio recording functionality of the mobile device are both enabled, and the additional functionality of the mobile device is disabled, in response to determining that the user of the mobile device is interacting with the mobile device;

detecting a voice input while the mobile device is operating in the mode in which the one or more sensors of the mobile device and the audio recording functionality of the mobile device are both enabled, and the additional functionality of the mobile device is disabled; and

determining whether to place the mobile device in a mode in which the one or more sensors of the mobile device, the audio recording functionality, and the additional functionality of the mobile device are all enabled, in response to detecting the voice input,

wherein the mobile device is operating in a mode in which the additional functionality of the mobile device is disabled when at least one of (i) a display of the mobile device is powered off, (ii) the mobile device is operating in a locked mode, and (iii) no applications are executing on the mobile device.