Systems and methods for determining semantic information associated with objects

• US 8,386,079 B1
• Filed: 10/28/2011
• Issued: 02/26/2013
• Est. Priority Date: 10/28/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Hickman, Ryan, Kohler, Damon
• Primary Examiner(s): Tran, Khoi
• Assistant Examiner(s): Nguyen, Bao Long T
• Application Number:
  US13/284,089
• Time in Patent Office: 487 Days
• Field of Search: None
• US Class Current: 700/246
• CPC Class Codes: G06N 3/008 based on physical entities controlled by simulated intelligence so as to replicate intelligent life forms, e.g. based on robots replicating pets or humans in their appearance or behaviour
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands

Abstract

Methods and systems for determining semantic information associated with objects are provided. An example method includes receiving information associated with an object and information associated with a contextual situation of a robotic device from the robotic device. For example, the information associated with the contextual situation may include information associated with an environment in which the robotic device is configured to operate. The method may further include performing a search of a database for information associated with the object, and receiving results of the search. The method may also include a computing system determining semantic information associated with a use of the object based on the results of the search. The use of the object may be based on the information associated with the contextual situation of the robotic device. According to the method, the semantic information may be stored as supplemental information associated with the object.

Claims

1. A method comprising:

receiving, by a computing system having a processor and a memory, information associated with an object;

performing, by the computing system, a search of a database for additional information associated with the object;

receiving results of the search, wherein the results of the search comprise videos associated with the object;

comparing, by the computing system, a robot model of a human to a human interaction with the object that occurs in a given video;

determining, by the computing system, semantic information associated with emulating the human interaction based on a mapping between the robot model and the human interaction; and

storing, by the computing system, the semantic information as supplemental information associated with the object.

2. The method of claim 1, further comprising:

receiving from a robotic device an indication of the robotic device interacting with the object;

sending the semantic information to the robotic device; and

receiving information associated with an interaction with the object by the robotic device utilizing the semantic information.

3. The method of claim 1, wherein the information associated with the object includes a three-dimensional model of the object stored in a three-dimensional object database.

4. The method of claim 3, further comprising determining a relationship between the object and at least one additional object of the object database.

5. The method of claim 1, further comprising receiving information associated with a contextual situation of a robotic device, wherein the information associated with the contextual situation includes information associated with one of a location of the robotic device, a task given to the robotic device to perform, capabilities of the robotic device, information associated with additional objects in the environment, and information associated with sensory data received by the robotic device.

6. The method of claim 1, wherein the semantic information further includes information describing one or more of:

how the object is used, why the object is used, what the object is used for, and when the object is used.

7. The method of claim 1, further comprising:

receiving from a robotic device data gathered by the robotic device and associated with a number of objects in an area;

performing object recognition on the gathered data to identify the object; and

sending the semantic information to the robotic device.

8. The method of claim 1, further comprising:

receiving from a robotic device data gathered by the robotic device and associated with a number of objects with a known availability;

performing object recognition on the gathered data to identify the object and at least one additional object;

performing a second search of a given database for information associated with the at least one additional object;

determining semantic information associated with a use of the at least one additional object based on the results of the second search; and

selecting the object or the at least one additional object in order for the robotic device to complete a task based on the semantic information associated with the use of the at least one additional object.

9. A method comprising:

receiving, by a computing system having a processor and a memory, information associated with an object;

performing, by the computing system, a search of a database for additional information associated with the object;

receiving results of the search;

analyzing, by the computing system, the results of the search to derive semantic information that is used by a robotic device in order to complete a task utilizing the object;

sending the semantic information to the robotic device;

receiving from the robotic device an indication of the robotic device attempting to complete the task utilizing the object using the semantic information;

determining a degree of utility associated with using the semantic information in order to complete the task, wherein the degree of utility includes information associated with whether the task was completed by the robotic device using the semantic information; and

storing the degree of utility as supplemental information associated with the semantic information.

10. The method of claim 9, wherein receiving results of the search comprises receiving information associated with one or more of webpages, images, videos, audio files, and three-dimensional objects.

11. The method of claim 9, wherein receiving results of the search comprises receiving textual sources of information within online discussion forums.

12. The method of claim 9, wherein receiving results of the search comprises receiving information associated with example use cases of the object, and the method further comprises:

determining from among the example use cases of the object the use case of the object for the robotic device based on the information associated with the contextual situation of the robotic device.

13. The method of claim 9, wherein receiving results of the search comprises receiving information identifying the object, and the method further comprises:

based on the information identifying the object, performing a second search on a given database for information associated with example use cases of the object; and

determining from among the example use cases of the object a use case of the object for the robotic device based on information associated with a contextual situation of the robotic device.