Methods and systems for robotic proactive informational retrieval from ambient context

• US 8,594,845 B1
• Filed: 05/07/2012
• Issued: 11/26/2013
• Est. Priority Date: 05/06/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Gharpure, Chaitanya
• Primary Examiner(s): HOLLOWAY, JASON R
• Application Number:
  US13/466,089
• Time in Patent Office: 568 Days
• Field of Search: 700/246, 700/250, 700/253, 700/257, 700/258, 700/259, 901/1, 901/46, 901/47
• US Class Current: 700/253
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/228 of application context

Abstract

Methods and systems are provided that may help a robot provide contextual filtering of environmental information, such that the robot may be able to base its actions upon environmental data that is relevant to the device'"'"'s context.

Claims

1. A method comprising:

determining, by a robotic device that includes one or more sensory devices, a context of the robotic device;

receiving an environmental input via at least one of the sensory devices, wherein the environmental input includes first environmental data received from a first source and second environmental data received from a second source, and wherein the first source and the second source are within an environment of the robotic device;

determining that the first environmental data is relevant to the context of the robotic device and the second environmental data is not relevant to the context of the robotic device;

in response to determining that the first environmental data is relevant to the context of the robotic device, performing, by the robotic device, an output function based on the first environmental data comprising;

formulating a query based at least on the first environmental data that is relevant to the context;

based on the query, searching for relational information corresponding to the first environmental data that is relevant to the context; and

receiving the relational information corresponding to the first environmental data that is relevant to the context.

2. The method of claim 1, wherein the first environmental data comprises audio data based on speech from the first source and the second environmental data comprises audio data based on speech from the second source.

3. The method of claim 1, wherein the first environmental data comprises video data received from the first source and the second environmental data comprises audio data received from the second source.

4. The method of claim 1, wherein performing the output function based on the first environmental data further comprises initiating a video feed to a remote computing device for displaying the relational information.

5. The method of claim 4, wherein performing the output function further comprises automatically displaying the relational information on a graphic display associated with the remote computing device.

6. The method of claim 1, wherein the output function comprises an interaction by the robotic device with an object in the environment of the robotic device.

7. The method of claim 1, wherein the first environmental data includes audio data received from the first source, and wherein determining that the first environmental data is relevant to the context of the robotic device further comprises:

applying a speech-to-text process on the audio data received from the first source;

based on the applied speech-to-text process, determining text from the audio data; and

determining that the text is relevant to the context of the robotic device.

8. The method of claim 1, wherein determining that the first environmental data is relevant to the context of the robotic device comprises determining that the first environmental data includes visual environmental data, wherein the visual environmental data comprises text, and wherein the text is relevant to the context of the robotic device.

9. A non-transitory computer readable medium having stored therein instructions executable by a robotic device that includes at least one sensory device to cause the robotic device to perform functions comprising:

determining, by a robotic device that includes one or more sensory devices, a context of the robotic device;

receiving an environmental input via at least one of the sensory devices, wherein the environmental input includes first environmental data received from a first source and second environmental data received from a second source, and wherein the first source and the second source are in an environment of the robotic device;

determining that the first environmental data is relevant to the context of the robotic device and the second environmental data is not relevant to the context of the robotic device;

in response to determining that the first environmental data is relevant to the context of the robotic device, performing, by the robotic device, an output function based on the first environmental data comprising;

formulating a query based at least on the first environmental data that is relevant to the context;

based on the query, searching for relational information corresponding to the first environmental data that is relevant to the context; and

receiving the relational information corresponding to the first environmental data that is relevant to the context.

10. The non-transitory computer readable medium of claim 9, wherein the first environmental data comprises audio data based on speech from the first source and the second environmental data comprises audio data based on speech from the second source.

11. The non-transitory computer readable medium of claim 9, wherein the first environmental data comprises video data received from the first source and the second environmental data comprises audio data received from the second source.

12. The non-transitory computer readable medium of claim 9, wherein performing the output function based on the first environmental data further comprises initiating a video feed to a remote computing device and displaying the relational information on a graphic display associated with the remote computing device.

13. The non-transitory computer readable medium of claim 9, wherein the output function comprises an interaction by the robotic device with an object in the environment of the robotic device.

14. The non-transitory computer readable medium of claim 9, wherein the first environmental data includes audio data from the first source, and wherein determining that the first environmental data is relevant to the context of the robotic device further comprises:

applying a speech-to-text process on the audio data;

based on the applied speech-to-text process, determining text from the audio data received from the first source; and

determining that the text is relevant to the context of the robotic device.

15. The non-transitory computer readable medium of claim 9, wherein determining that the first environmental data is relevant to the context of the robotic device comprises determining that the first environmental data includes visual environmental data, wherein the visual environmental data comprises text, and wherein the text is relevant to the context of the robotic device.

16. A robotic device comprising:

one or more processors;

one or more sensors coupled to the one or more processors and configured to capture environmental data about an environment of the robotic device; and

memory including instructions stored thereon executable by the one or more processors to perform functions comprising;

determining a context of the robotic device;

receiving an environmental input via at least one of the sensors, wherein the environmental input includes first environmental data received from a first source and second environmental data received from a second source, and wherein the first source and the second source are in the environment of the robotic device;

determining that the first environmental data is relevant to the context of the robotic device and the second environmental data is not relevant to the context of the robotic device;

in response to determining that the first environmental data is relevant to the context of the robotic device, performing, by the robotic device, an output function based on the first environmental data comprising;

formulating a query based at least on the first environmental data that is relevant to the context;

based on the query, searching for relational information corresponding to the first environmental data that is relevant to the context; and

receiving the relational information corresponding to the first environmental data that is relevant to the context.

17. The robotic device of claim 16, wherein the one or more sensors comprise one or more acoustic sensors, wherein the one or more acoustic sensors are used to receive the environmental input, and wherein the first environmental data comprises audio data based on speech from the first source and the second environmental data comprises audio data based on speech from the second source.

18. The robotic device of claim 16, wherein the one or more sensors comprise one or more video capture devices, wherein the first environmental data comprises video data received from the first source.

19. The robotic device of claim 16, wherein performing the output function based on the environmental data comprises initiating a video feed to a remote computing device and displaying the relational information on a graphic display associated with the remote computing device.

20. The robotic device of claim 16, wherein the output function comprises an interaction by the robotic device with an object in the environment of the robotic device.