Tap to Initiate a Next Action for User Requests

• US 20170139672A1
• Filed: 01/27/2017
• Published: 05/18/2017
• Est. Priority Date: 12/30/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: X Development LLC (Alphabet Inc.)
• Inventors: Li, Chun Yat Frank, Magana, Daniel Rodriguez, Teixeira, Thiago, Chen, Charles, Agarawala, Anand
• Granted Patent: US 9,798,517 B2
• Time in Patent Office: Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 16/2457 with adaptation to user needs
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/03547 Touch pads, in which fingers can move on a surface
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 9/453 Help systems
• CPC Class Codes: G06Q 30/0601 Electronic shopping [e-shopping]
• CPC Class Codes: G10L 15/00 Speech recognition G10L17/00 takes precedence
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications

Abstract

Embodiments may relate to intuitive user-interface features for a head-mountable device (HMD), in the context of a hybrid human and computer-automated response system. An illustrative method may involve a head-mountable device (HMD) that comprises a touchpad: (a) sending a speech-segment message to a hybrid response system, wherein the speech-segment message is indicative of a speech segment that is detected in audio data captured at the HMD, and wherein the speech-segment is associated with a first user-account with the hybrid response system, (b) receiving a response message that includes a response to the speech-segment message and an indication of a next action corresponding to the response to the speech-segment message, (c) displaying a screen interface that includes an indication of the response, and (d) while displaying the response, detecting a singular touch gesture and responsively initiating the at least one next action.

Claims

1. A method comprising:

detecting a first touch gesture on a touch interface of a user-facing computing device, and responsively sending a speech-segment message to a hybrid response system, wherein the speech-segment message comprises an initial query corresponding to a speech segment in audio data captured at the user-facing computing device;

receiving, at the user-facing computing device, a response message comprising both;

(a) an answer to the initial query, and (b) an indication of a next action to be cued as an anticipated response to a follow-up request related to the answer to the initial query; and

in response to receiving the response message, the user-facing computing device;

(a) displaying a first screen indicating the answer to the initial query, and

(b) while displaying the first screen interface, detecting touchpad data corresponding to a second touch gesture on the touch interface and responsively initiating and displaying a second screen interface corresponding to the next action.

2. The method of claim 1, wherein the second touch gesture comprises a single tap on the touch interface.

3. The method of claim 1, wherein initiating the next action comprises displaying a first screen interface that indicates the information related to the response.

4. The method of claim 1, wherein the indication of the next action comprises an indication of a transaction that is associated with the response message.

5. The method of claim 4, wherein the response message comprises an indication of a product or service, and wherein the transaction comprises a purchase of the product.

6. The method of claim 4, wherein the speech-segment message is associated with a first user-account, and wherein initiating the next action comprises initiating a purchase of a product or service via a second user-account that is associated with the first user-account.

7. The method of claim 6, wherein initiating the at least one next action comprises displaying the first screen interface that requests input that indicates whether or not to purchase a product or service via a second user-account associated with the first user-account.

8. The method of claim 1, wherein the speech-segment message is associated with a first user-account, and wherein the response message comprises proposed text for a message that can be sent via at least one second user-account associated with the first user-account, and wherein the indication of the next action comprises an indication to display the first screen interface that indicates message-capable applications associated with the first user-account.

9. The method of claim 8, wherein initiating the next action comprises displaying the first screen interface that indicates the one or more message-capable applications associated with the first user-account.

10. The method of claim 1, wherein the speech-segment message is associated with a first user-account, wherein the response message comprises proposed text for the message that can be sent via at least one second user-account associated with the first user-account, and wherein the indication of the next action comprises an indication to send a message that includes the proposed text via the at least one second user-account.

11. The method of claim 10, wherein initiating the next action comprises sending, via the at least one second user-account, the message that includes the proposed text.

12. A computing device comprising:

at least one microphone configured to provide an audio signal based on speech detected at the computing device;

a touch interface;

a control system configured to;

detect a first touch gesture on the touch interface and responsively send a speech-segment message to a hybrid response system, wherein the speech-segment message comprises an initial query corresponding to a speech segment in audio data captured using the at least one microphone;

receiving a response message comprising both;

(a) an answer to the initial query, and (b) an indication of a next action to be cued as an anticipated response to a follow-up request related to the answer to the initial query; and

in response to receipt of the response message;

(a) display a first screen interface indicating the answer to the initial query, and (b) while displaying the first screen interface, detect touch data corresponding to a second touch gesture on the touch interface and responsively initiate and display a second screen interface corresponding to the next action.

13. The system of claim 12, wherein the touch gesture comprises a single tap on the touch interface.

14. The system of claim 12, wherein the indication of the next action comprises information related to the response, and wherein initiating the at least one next action comprises displaying a screen interface that indicates the information related to the response.

15. The system of claim 12, wherein the indication of the next action comprises an indication of a transaction that is associated with the response, and wherein the response comprises an indication of a product or service, and wherein the transaction comprises a purchase of the product.

16. A non-transitory computer readable medium having stored therein instructions executable by at least one processor to cause the computing device of a user-facing computing device to perform functions comprising:

detecting a first touch gesture on a touch interface of a user-facing computing device, and responsively sending a speech-segment message to a hybrid response system, wherein the speech-segment message comprises an initial query corresponding to a speech segment in audio data captured at the user-facing computing device;

receiving, at the user-facing computing device, a response message comprising both;

(a) an answer to the initial query, and (b) an indication of a next action to be cued as an anticipated response to a follow-up request related to the answer to the initial query; and

in response to receiving the response message, the user-facing computing device;

(a) displaying a first screen interface indicating the answer to the initial query, and

(b) while displaying the first screen interface, detecting touchpad data corresponding to a second touch gesture on the touch interface and responsively initiating and displaying a second screen interface corresponding to the next action.

17. The non-transitory computer readable medium of claim 16, wherein the second touch gesture comprises a single tap on the touch interface.

18. A method carried out by a head-mountable device comprising:

sending a speech-segment message to a hybrid response system in response to detecting a first singular touch gesture on a touchpad of the head-mountable device (HMD),

wherein the HMD comprises a near-eye display, and wherein the touchpad is located on a side-arm of the HMD,

wherein the speech-segment message is indicative of a speech segment detected in audio data captured at the HMD by at least one microphone, wherein the speech-segment is associated with a first user-account with the hybrid response system, and

wherein the speech-segment message comprises an initial query;

receiving, at the HMD, a response message from the hybrid response system, wherein the response message comprises;

both (a) an answer to the initial query and (b) an indication of a next action, wherein the next action is an anticipated actionable follow-up request related to the answer of the initial query; and

in response to receiving the response message;

displaying, on the HMD, a first card interface that includes an indication of the answer to the initial query; and

while displaying the first card interface that indicates the answer to the initial query, detecting touchpad data corresponding to a second singular touch gesture on the touchpad and responsively initiating the next action corresponding to the anticipated follow-up request originally indicated in the response message by displaying a second card interface of a cued next action.