Multimodal object identification

• US 10,427,306 B1
• Filed: 07/06/2017
• Issued: 10/01/2019
• Est. Priority Date: 07/06/2017
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: X Development LLC (Alphabet Inc.)
• Inventors: Quinlan, Michael Joseph, Cohen, Gabriel A.
• Primary Examiner(s): Sample, Jonathan L
• Application Number:
  US15/643,138
• Time in Patent Office: 817 Days
• Field of Search: 700245-264
• CPC Class Codes: B25J 11/0005 Manipulators having means for high-level communication with users, e.g. speech generator, face recognition means
• CPC Class Codes: B25J 11/008 Manipulators for service tasks
• CPC Class Codes: B25J 13/003 by means of an audio-responsive input audible safety signals B25J19/061
• CPC Class Codes: B25J 19/026 Acoustical sensing devices
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 19/048 Monitoring; Safety
• CPC Class Codes: G05B 2219/23021 Gesture programming, camera sees hand, displays it on screen, grasp buttons
• CPC Class Codes: G05B 2219/35444 Gesture interface, controlled machine observes operator, executes commands
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback

Abstract

Methods, systems, and apparatus for receiving a command for controlling a robot, the command referencing an object, receiving sensor data for a portion of an environment of the robot, identifying, from the sensor data, a gesture of a human that indicates a spatial region located outside of the portion of the environment described by the sensor data, accessing map data indicating locations of objects within a space, searching the map data for the object, wherein the search of the map data is restricted to the spatial region, determining, based at least on searching the map data for the object referenced in the command, that the object referenced in the command is present in the spatial region, and in response to determining that the object referenced in the command is present in the spatial region, controlling the robot to perform an action with respect to the object referenced in the command.

Claims

1. A computer-implemented method comprising:

receiving a command for controlling a robot, the command referencing an object;

receiving sensor data for a portion of an environment of the robot, the sensor data being captured by a sensor of the robot;

identifying, from the sensor data, a gesture of a human that indicates a spatial region located outside of the portion of the environment described by the sensor data;

in response to identifying the gesture, accessing map data indicating locations of objects within a space, the map data being generated before receiving the command;

searching the map data for the object referenced in the command, wherein the search of the map data is restricted, based on the identified gesture, to the spatial region indicated by the gesture;

determining, based at least on searching the map data for the object referenced in the command, that the object referenced in the command is present in the spatial region indicated by the gesture; and

in response to determining that the object referenced in the command is present in the spatial region indicated by the gesture, controlling the robot to perform an action with respect to the object referenced in the command.

2. The computer-implemented method of claim 1, wherein the identified gesture is one of an arm wave, a hand gesture, or a glance.

3. The computer-implemented method of claim 1, wherein receiving the command for controlling the robot, the command referencing the object comprises:

receiving audio data captured by a microphone of the robot that corresponds to the command;

identifying, based on performing speech recognition on the audio data, one or more candidate objects that each correspond to a respective candidate transcription of at least a portion of the audio data;

accessing an inventory of one or more objects within the space; and

identifying the object from among the inventory of the one or more objects within the space based at least on comparing each candidate transcription of at least a portion of the audio data to the one or more objects of the inventory.

4. The computer-implemented method of claim 1, wherein identifying the gesture of the human that indicates the spatial region located outside of the portion of the environment described by the sensor data comprises:

determining a location of the robot within the space; and

determining the spatial region based at least on the gesture of the human and the location of the robot within the space.

5. The computer-implemented method of claim 1, wherein identifying the gesture of the human that indicates the spatial region located outside of the portion of the environment described by the sensor data comprises:

determining an orientation of the robot within the space when receiving the sensor data; and

determining the spatial region based at least on the gesture of the human and the orientation of the robot within the space.

6. The computer-implemented method of claim 1, wherein identifying the gesture of the human that indicates the spatial region located outside of the portion of the environment described by the sensor data comprises:

detecting one or more predetermined shapes from the sensor data, each of the one or more predetermined shapes corresponding to a gesture of a human;

determining one or more locations of the detected one or more predetermined shapes within the sensor data; and

determining the spatial region based at least on the one or more predetermined shapes and the one or more locations of the detected one or more predetermined shapes within the sensor data.

7. The computer-implemented method of claim 1, comprising:

receiving a second command for controlling the robot, the command referencing a second object;

receiving second sensor data for a portion of the environment of the robot, the second sensor data being captured by the sensor of the robot;

identifying, from the second sensor data, a second gesture of a human that indicates a second spatial region located outside of the portion of the environment described by the second sensor data;

searching the map data for the second object referenced in the second command, wherein the search of the map data is restricted, based on the identified second gesture, to the second spatial region indicated by the second gesture;

determining, based at least on searching the map data for the second object referenced in the second command, that the second object referenced in the second command is absent from the second spatial region indicated by the second gesture;

based at least on determining that the second object referenced in the second command is absent from the second spatial region indicated by the second gesture, searching the map data for the second object referenced in the second command, wherein the search of the map is restricted to a third spatial region;

determining, based at least on searching the map data for the second object referenced in the second command, that the second object referenced in the second command is present in the third spatial region; and

in response to determining that the second object referenced in the second command is present in the third spatial region, controlling the robot to perform a second action with respect to the second object referenced in the second command.

8. The computer-implemented method of claim 7, wherein the third spatial region is larger than the second spatial region indicated by the second gesture.

9. The computer-implemented method of claim 1, comprising:

receiving a second command for controlling the robot, the command referencing a second object;

receiving second sensor data for a portion of the environment of the robot, the second sensor data being captured by the sensor of the robot;

identifying, from the second sensor data, a second gesture of a human that indicates a second spatial region located outside of the portion of the environment described by the second sensor data;

searching the map data for the second object referenced in the second command, wherein the search of the map data is restricted, based on the identified second gesture, to the second spatial region indicated by the second gesture;

determining, based at least on searching the map data for the second object referenced in the second command, that the second object referenced in the second command is absent from the second spatial region indicated by the second gesture; and

based at least on determining that the second object referenced in the second command is absent from the second spatial region indicated by the second gesture, controlling the robot to indicate that the second object referenced in the second command is absent from the second spatial region indicated by the second gesture.

10. The computer-implemented method of claim 1, wherein controlling the robot to perform the action with respect to the object referenced in the command comprises controlling the robot to perform one of retrieving the object referenced in the command, move the object referenced in the command to a predetermined location, move the object referenced in the command to a location indicated by the command, or navigate to the location of the object referenced in the command.

11. The computer-implemented method of claim 1, comprising:

determining a location of the object referenced in the command within the spatial region, wherein the location of the object referenced in the command within the spatial region is represented by a set of coordinates within the space.

12. The computer-implemented method of claim 1, wherein the sensor data is one of image data, infrared image data, light detection and ranging (LIDAR) data, thermal image data, night vision image data, or motion data.

13. The computer-implemented method of claim 1, wherein the sensor data for the portion of the environment of the robot is image data for a field of view of a camera of the robot, the image data being captured by the camera of the robot.

14. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

receiving a command for controlling a robot, the command referencing an object;

receiving sensor data for a portion of an environment of the robot, the sensor data being captured by a sensor of the robot;

identifying, from the sensor data, a gesture of a human that indicates a spatial region located outside of the portion of the environment described by the sensor data;

in response to identifying the gesture, accessing map data indicating locations of objects within a space, the map data being generated before receiving the command;

searching the map data for the object referenced in the command, wherein the search of the map data is restricted, based on the identified gesture, to the spatial region indicated by the gesture;

determining, based at least on searching the map data for the object referenced in the command, that the object referenced in the command is present in the spatial region indicated by the gesture; and

in response to determining that the object referenced in the command is present in the spatial region indicated by the gesture, controlling the robot to perform an action with respect to the object referenced in the command.

15. The system of claim 14, wherein the identified gesture is one of an arm wave, a hand gesture, or a glance.

16. The system of claim 14, wherein receiving the command for controlling the robot, the command referencing the object comprises:

receiving audio data captured by a microphone of the robot that corresponds to the command;

identifying, based on performing speech recognition on the audio data, one or more candidate objects that each correspond to a respective candidate transcription of at least a portion of the audio data;

accessing an inventory of one or more objects within the space; and

identifying the object from among the inventory of the one or more objects within the space based at least on comparing each candidate transcription of at least a portion of the audio data to the one or more objects of the inventory.

17. The system of claim 14, wherein identifying the gesture of the human that indicates the spatial region located outside of the portion of the environment described by the sensor data comprises:

determining a location of the robot within the space; and

determining the spatial region based at least on the gesture of the human and the location of the robot within the space.

18. The system of claim 14, wherein identifying the gesture of the human that indicates the spatial region located outside of the portion of the environment described by the sensor data comprises:

determining an orientation of the robot within the space when receiving the sensor data; and

determining the spatial region based at least on the gesture of the human and the orientation of the robot within the space.

19. The system of claim 14, wherein identifying the gesture of the human that indicates the spatial region located outside of the portion of the environment described by the sensor data comprises:

detecting one or more predetermined shapes from the sensor data, each of the one or more predetermined shapes corresponding to a gesture of a human;

determining one or more locations of the detected one or more predetermined shapes within the sensor data; and

determining the spatial region based at least on the one or more predetermined shapes and the one or more locations of the detected one or more predetermined shapes within the sensor data.

20. A non-transitory computer-readable storage device storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:

receiving a command for controlling a robot, the command referencing an object;

receiving sensor data for a portion of an environment of the robot, the sensor data being captured by a sensor of the robot;

identifying, from the sensor data, a gesture of a human that indicates a spatial region located outside of the portion of the environment described by the sensor data;

in response to identifying the gesture, accessing map data indicating locations of objects within a space, the map data being generated before receiving the command;

searching the map data for the object referenced in the command, wherein the search of the map data is restricted, based on the identified gesture, to the spatial region indicated by the gesture;

determining, based at least on searching the map data for the object referenced in the command, that the object referenced in the command is present in the spatial region indicated by the gesture; and

in response to determining that the object referenced in the command is present in the spatial region indicated by the gesture, controlling the robot to perform an action with respect to the object referenced in the command.