Tap to initiate a next action for user requests

• US 9,798,517 B2
• Filed: 01/27/2017
• Issued: 10/24/2017
• Est. Priority Date: 12/30/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: X Development LLC (Alphabet Inc.)
• Inventors: Li, Chun Yat Frank, Magana, Daniel Rodriguez, Teixeira, Thiago, Chen, Charles, Agarawala, Anand
• Primary Examiner(s): Jansen, II, Michael J
• Application Number:
  US15/417,609
• Publication Number:
  US 20170139672A1
• Time in Patent Office: 270 Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 16/2457 with adaptation to user needs
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/03547 Touch pads, in which fingers can move on a surface
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 9/453 Help systems
• CPC Class Codes: G06Q 30/0601 Electronic shopping [e-shopping]
• CPC Class Codes: G10L 15/00 Speech recognition G10L17/00 takes precedence
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications

Abstract

Embodiments may relate to intuitive user-interface features for a head-mountable device (HMD), in the context of a hybrid human and computer-automated response system. An illustrative method may involve a head-mountable device (HMD) that comprises a touchpad: (a) sending a speech-segment message to a hybrid response system, wherein the speech-segment message is indicative of a speech segment that is detected in audio data captured at the HMD, and wherein the speech-segment is associated with a first user-account with the hybrid response system, (b) receiving a response message that includes a response to the speech-segment message and an indication of a next action corresponding to the response to the speech-segment message, (c) displaying a screen interface that includes an indication of the response, and (d) while displaying the response, detecting a singular touch gesture and responsively initiating the at least one next action.

Claims

1. A method comprising:

detecting a first touch gesture on a touch interface of a client computing device, and responsively sending a speech-segment message to a hybrid response system, wherein the speech-segment message comprises an initial query corresponding to a speech segment in audio data captured at the client computing device;

receiving, at the client computing device, a response message comprising both;

(a) an answer to the initial query, and (b) an indication of a next action that is cued as a response to an anticipated follow-up request related to the answer to the initial query; and

in response to receiving the response message, the client computing device;

(a) displaying, on a graphic display associated with the client computing device, a first screen indicating the answer to the initial query, and

(b) while displaying the first screen interface, detecting touchpad data corresponding to a second touch gesture on the touch interface and responsively initiating and displaying a second screen interface corresponding to the next action.

2. The method of claim 1, wherein the second touch gesture comprises a single tap on the touch interface.

3. The method of claim 1, wherein initiating the next action comprises displaying a first screen interface that indicates the information related to the response.

4. The method of claim 1, wherein the indication of the next action comprises an indication of a transaction that is associated with the response message.

5. The method of claim 4, wherein the response message comprises an indication of a product or service, and wherein the transaction comprises a purchase of the product.

6. The method of claim 4, wherein the speech-segment message is associated with a first user-account, and wherein initiating the next action comprises initiating a purchase of a product or service via a second user-account that is associated with the first user-account.

7. The method of claim 6, wherein initiating the at least one next action comprises displaying the first screen interface that requests input that indicates whether or not to purchase a product or service via a second user-account associated with the first user-account.

8. The method of claim 1, wherein the speech-segment message is associated with a first user-account, and wherein the response message comprises proposed text for a message that can be sent via at least one second user-account associated with the first user-account, and wherein the indication of the next action comprises an indication to display the first screen interface that indicates message-capable applications associated with the first user-account.

9. The method of claim 8, wherein initiating the next action comprises displaying the first screen interface that indicates the one or more message-capable applications associated with the first user-account.

10. The method of claim 1, wherein the speech-segment message is associated with a first user-account, wherein the response message comprises proposed text for the message that can be sent via at least one second user-account associated with the first user-account, and wherein the indication of the next action comprises an indication to send a message that includes the proposed text via the at least one second user-account.

11. The method of claim 10, wherein initiating the next action comprises sending, via the at least one second user-account, the message that includes the proposed text.

12. A computing device comprising:

at least one microphone configured to provide an audio signal based on speech detected at the computing device;

a touch interface;

a control system configured to;

detect a first touch gesture on the touch interface and responsively send a speech-segment message to a hybrid response system, wherein the speech-segment message comprises an initial query corresponding to a speech segment in audio data captured using the at least one microphone;

receiving a response message comprising both;

(a) an answer to the initial query, and (b) an indication of a next action that is cued as a response to an anticipated follow-up request related to the answer to the initial query; and

in response to receipt of the response message;

(a) display, on a graphic display associated with the client computing device, a first screen interface indicating the answer to the initial query, and (b) while displaying the first screen interface, detect touch data corresponding to a second touch gesture on the touch interface and responsively initiate and display a second screen interface corresponding to the next action.

13. The system of claim 12, wherein the touch gesture comprises a single tap on the touch interface.

14. The system of claim 12, wherein the indication of the next action comprises information related to the response, and wherein initiating the at least one next action comprises displaying a screen interface that indicates the information related to the response.

15. The system of claim 12, wherein the indication of the next action comprises an indication of a transaction that is associated with the response, and wherein the response comprises an indication of a product or service, and wherein the transaction comprises a purchase of the product.

16. A non-transitory computer readable medium having stored therein instructions executable by at least one processor to cause the computing device of a client computing device to perform functions comprising:

detecting a first touch gesture on a touch interface of the client computing device, and responsively sending a speech-segment message to a hybrid response system, wherein the speech-segment message comprises an initial query corresponding to a speech segment in audio data captured at the client computing device;

receiving, at the client computing device, a response message comprising both;

(a) an answer to the initial query, and (b) an indication of a next action that is cued as a response to an anticipated follow-up request related to the answer to the initial query; and

in response to receiving the response message, the client computing device;

(a) displaying, on a graphic display associated with the client computing device, a first screen interface indicating the answer to the initial query, and

(b) while displaying the first screen interface, detecting touchpad data corresponding to a second touch gesture on the touch interface and responsively initiating and displaying a second screen interface corresponding to the next action.

17. The non-transitory computer readable medium of claim 16, wherein the second touch gesture comprises a single tap on the touch interface.

18. A method carried out by a hybrid response computing system comprising:

receiving a speech-segment message from a client computing device,

wherein the speech-segment message is indicative of a speech segment detected in audio data captured at the client computing device by at least one microphone, wherein the speech-segment is associated with a first user-account with the hybrid response computing system, and wherein the speech-segment message comprises an initial query;

in response to receipt of the speech segment message, and without use of any subsequent input from the user-facing computing device, the hybrid response computing system;

determining (a) an answer to the initial query, (b) an anticipated follow-up request related to the answer, and (c) a next action responsive to the anticipated follow-up request, wherein the next action is cued to be performed in response to detection of a singular touch gesture while the answer is displayed at the client computing device;

sending, to the client computing device, a response message that;

(a) comprises the answer to the initial query and (b) does not indicate to the user-facing computing device to display the next action;

receiving, by the hybrid response computing device, an input message indicating receipt of touchpad data corresponding to performance of a singular touch gesture on the touchpad of the client computing device while a first card interface including an indication of the answer to the initial query is displayed on a graphic display associated with the client computing device;

responsive to the input message, the hybrid response computing system initiating the cued next action corresponding to the anticipated follow-up request.