Resolving automated assistant requests that are based on image(s) and/or other sensor data

• US 10,275,651 B2
• Filed: 06/23/2017
• Issued: 04/30/2019
• Est. Priority Date: 05/16/2017
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Badr, Ibrahim, Grimsmo, Nils, Bakir, Gokhan
• Primary Examiner(s): Mistry, Oneal R
• Application Number:
  US15/631,274
• Publication Number:
  US 20180336414A1
• Time in Patent Office: 676 Days
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/90332 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/68 Food, e.g. fruit or vegetables
• CPC Class Codes: H04L 51/02 using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages

Abstract

Methods, apparatus, and computer readable media are described related to causing processing of sensor data to be performed in response to determining a request related to an environmental object that is likely captured by the sensor data. Some implementations further relate to determining whether the request is resolvable based on the processing of the sensor data. When it is determined that the request is not resolvable, a prompt is determined and provided as user interface output, where the prompt provides guidance on further input that will enable the request to be resolved. In those implementations, the further input (e.g., additional sensor data and/or the user interface input) received in response to the prompt can then be utilized to resolve the request.

Claims

1. A method implemented by one or more processors, comprising:

receiving, via an automated assistant interface of a client device, a voice input provided by a user;

determining that the voice input includes a request related to an object in an environment of the client device;

in response to determining that the voice input includes the request related to the object;

causing processing to be performed on initial sensor data captured by at least one sensor, the at least one sensor being of the client device or an additional electronic device in the environment, and the initial sensor data capturing one or more characteristics of the object;

determining, based on one or more initial attributes of the object resolved based on the processing of the initial sensor data, that the request is not resolvable based on the initial sensor data, wherein determining that the request is not resolvable based on the one or more initial attributes comprises;

determining that the one or more initial attributes fail to define the object with a degree of specificity necessary for resolving the request, wherein the degree of specificity is a target degree of classification in a classification taxonomy;

in response to determining that the request is not resolvable;

providing, for presentation to the user via the automated assistant interface of the client device, a prompt that instructs the user to capture additional sensor data or to move the object;

receiving additional sensor data, the additional sensor data captured by the client device or the additional electronic device after the prompt is presented to the user;

causing processing to be performed on the additional sensor data; and

resolving the request based on at least one additional attribute resolved based on the processing of the additional sensor data.

2. The method of claim 1, wherein the initial sensor data comprises at least one initial image captured by a camera of the client device or the additional electronic device, and further comprising:

in response to determining that the voice input includes the request related to the object;

causing the at least one initial image to be captured.

3. The method of claim 2, wherein capturing the at least one initial image occurs independent of any physical contact between the user and the client device and occurs independent of any physical contact between the user and the additional electronic device.

4. The method of claim 1, wherein resolving the request is further based on at least one of the one or more initial attributes of the object resolved based on the processing of the initial sensor data.

5. The method of claim 1, further comprising:

determining, based on the one or more initial attributes resolved based on the processing of the initial sensor data, and based on the at least one additional attribute resolved based on the processing of the additional sensor data, that the request is still not resolvable; and

in response to determining that the request is still not resolvable;

providing an additional prompt for presentation to the user via the automated assistant interface of the client device;

wherein resolving the request is further based on at least one further additional attribute resolved based on input received in response to the additional prompt.

6. The method of claim 5, wherein the additional prompt instructs the user to capture an additional image or to further move the object, and wherein the input is an image captured by a camera of the client device or the additional electronic device.

7. The method of claim 5, wherein the input is further natural language input, and wherein the further additional attribute is determined based on the further natural language input.

8. The method of claim 1, further comprising:

providing initial output for presentation to the user via the automated assistant interface of the client device, wherein the initial output is based on the initial attributes;

receiving user feedback in response to providing the initial output; and

determining the degree of specificity based on the user feedback.

9. The method of claim 1, further comprising:

determining the degree of specificity based on a learned degree of specificity associated with the one or more initial attributes of the object resolved based on the processing of the initial sensor data, the learned degree of specificity learned based on prior automated assistant interactions related to objects associated with the initial attributes.

10. The method of claim 1, further comprising:

determining the prompt based on at least one initial attribute of the one or more initial attributes of the object resolved based on the image processing of the at least one initial image.

11. The method of claim 1, wherein the request is a request for additional information related to the object, and wherein resolving the request based on the at least one additional attribute resolved based on the image processing of the at least one additional image comprises:

generating additional content based on the at least one additional attribute; and

providing, via the automated assistant interface of the client device, the additional content for presentation to the user as a response to the request.

12. The method of claim 11, wherein generating the additional content based on the additional attribute comprises:

transmitting, to an agent over one or more networks and via an application programming interface, an agent request generated based on the additional attribute; and

receiving the additional content from the agent in response to transmitting the agent request.

13. A method implemented by one or more processors, comprising:

receiving, via an automated assistant interface of a client device, a voice input provided by a user;

determining that the voice input includes a request related to an object in an environment of the client device;

in response to determining that the voice input includes the request related to the object;

causing processing to be performed on initial sensor data captured by at least one sensor, the at least one sensor being of the client device or an additional electronic device in the environment, and the initial sensor data capturing one or more characteristics of the object;

determining, based on one or more initial attributes of the object resolved based on the processing of the initial sensor data, that the request is not resolvable based on the initial sensor data, wherein determining that the request is not resolvable based on the one or more initial attributes comprises;

determining that the one or more initial attributes fail to define the object with a degree of specificity necessary for resolving the request, and wherein determining the degree of specificity comprises;

providing initial output for presentation to the user via the automated assistant interface of the client device, wherein the initial output is based on the initial attributes;

receiving user feedback in response to providing the initial output; and

determining the degree of specificity based on the user feedback;

in response to determining that the request is not resolvable;

providing, for presentation to the user via the automated assistant interface of the client device, a prompt that instructs the user to capture additional sensor data or to move the object;

receiving additional sensor data, the additional sensor data captured by the client device or the additional electronic device after the prompt is presented to the user;

causing processing to be performed on the additional sensor data; and

resolving the request based on at least one additional attribute resolved based on the processing of the additional sensor data.

14. A method implemented by one or more processors, comprising:

receiving, via an automated assistant interface of a client device, a voice input provided by a user;

determining that the voice input includes a request related to an object in an environment of the client device;

in response to determining that the voice input includes the request related to the object;

causing processing to be performed on initial sensor data captured by at least one sensor, the at least one sensor being of the client device or an additional electronic device in the environment, and the initial sensor data capturing one or more characteristics of the object;

determining, based on one or more initial attributes of the object resolved based on the processing of the initial sensor data, that the request is not resolvable based on the initial sensor data;

in response to determining that the request is not resolvable;

providing, for presentation to the user via the automated assistant interface of the client device, a prompt that instructs the user to capture additional sensor data or to move the object;

receiving additional sensor data, the additional sensor data captured by the client device or the additional electronic device after the prompt is presented to the user;

causing processing to be performed on the additional sensor data to resolve at least one additional attribute of the object;

determining, based on the one or more initial attributes resolved based on the processing of the initial sensor data, and based on the at least one additional attribute resolved based on the processing of the additional sensor data, that the request is still not resolvable;

in response to determining that the request is still not resolvable;

providing an additional prompt for presentation to the user via the automated assistant interface of the client device; and

resolving the request based on the at least one additional attribute resolved based on the processing of the additional sensor data, and based on the at least one further additional attribute resolved based on input received in response to the additional prompt.

15. The method of claim 14, wherein the additional prompt instructs the user to capture an additional image or to further move the object, and wherein the input is an image captured by a camera of the client device or the additional electronic device.

16. The method of claim 14, wherein the input is further natural language input, and wherein the further additional attribute is determined based on the further natural language input.

17. A method implemented by one or more processors, comprising:

receiving, via an automated assistant interface of a client device, a voice input provided by a user;

determining that the voice input includes a request related to an object in an environment of the client device;

in response to determining that the voice input includes the request related to the object;

causing processing to be performed on initial sensor data captured by at least one sensor, the at least one sensor being of the client device or an additional electronic device in the environment, and the initial sensor data capturing one or more characteristics of the object;

determining, based on one or more initial attributes of the object resolved based on the processing of the initial sensor data, that the request is not resolvable based on the initial sensor data, wherein determining that the request is not resolvable based on the one or more initial attributes comprises;

determining that the one or more initial attributes fail to define the object with a degree of specificity necessary for resolving the request, and wherein determining the degree of specificity comprises;

determining the degree of specificity based on a classification attribute of the one or more initial attributes of the object resolved based on the processing of the initial sensor data;

in response to determining that the request is not resolvable;

providing, for presentation to the user via the automated assistant interface of the client device, a prompt that instructs the user to capture additional sensor data or to move the object;

receiving additional sensor data, the additional sensor data captured by the client device or the additional electronic device after the prompt is presented to the user;

causing processing to be performed on the additional sensor data; and

resolving the request based on at least one additional attribute resolved based on the processing of the additional sensor data.

18. The method of claim 17, wherein the degree of specificity defines one or more fields that are specific to the classification attribute, and wherein determining that the request is not resolvable based on the one or more initial attributes comprises:

determining that one or more of the fields are undefined by the initial attributes.