Using observations from one or more robots to generate a spatio-temporal model that defines pose values for a plurality of objects in an environment

• US 10,127,677 B1
• Filed: 05/18/2017
• Issued: 11/13/2018
• Est. Priority Date: 08/25/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: X Development LLC (Alphabet Inc.)
• Inventors: Anderson-Sprecher, Peter Elving, Mason, Julian Mac Neille, Saboo, Rohit Ramesh
• Primary Examiner(s): Dunphy, David F
• Application Number:
  US15/599,232
• Time in Patent Office: 544 Days
• CPC Class Codes: G05D 1/024 in combination with a laser lasers per se H01S
• CPC Class Codes: G05D 1/0251 extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/344 involving models
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

Methods, apparatus, systems, and computer-readable media are provided for generating and using a spatio-temporal model that defines pose values for a plurality of objects in an environment and corresponding times associated with the pose values. Some implementations relate to using observations for one or more robots in an environment to generate a spatio-temporal model that defines pose values and corresponding times for multiple objects in the environment. In some of those implementations, the model is generated based on uncertainty measures associated with the pose values. Some implementations relate to utilizing a generated spatio-temporal model to determine the pose for each of one or more objects an environment at a target time. The pose for an object at a target time is determined based on one or more pose values for the object selected based on a corresponding measurement time, uncertainty measure, and/or source associated with the pose values.

Claims

1. A method implemented by one or more processors, comprising:

accessing a spatio-temporal model stored in one or more computer readable media,

the spatio-temporal model generated based on received observations from multiple robots operating in an environment over a period of time, and

the spatio-temporal model defining, for each of a plurality of objects in the environment, a plurality of pose values and corresponding times for the pose values;

identifying an object of the objects in the environment, and a target time;

selecting, from the spatio-temporal model, a group of pose values for the object based on proximity of the target time to the corresponding times for the pose values of the group; and

generating a pose for the object for the target time based on the selected group of the pose values.

2. The method of claim 1, further comprising:

providing the pose for the object to at least one of the multiple robots;

wherein in response to receiving the pose for the object, the at least one of the multiple robots utilizes the pose for path planning.

3. The method of claim 1, further comprising:

generating robot control commands based on the pose for the object.

4. The method of claim 1, further comprising:

identifying an area indication for the target time that defines one or more areas;

wherein selecting the group of the pose values is further based on the pose values of the group defining a position that is proximal to or within the one or more areas defined by the area indication.

5. The method of claim 1, wherein the spatio-temporal model further defines, for each of the pose values, an uncertainty measure for the pose value, and wherein selecting the group of the pose values further comprises:

selecting the group of the pose values based on the uncertainty measures associated with the pose values of the group.

6. The method of claim 1, wherein the spatio-temporal model further defines, for each of the pose values, a source identifier that indicates a corresponding source of a corresponding observation on which the corresponding pose value is based, and wherein selecting the group of the pose values further comprises:

selecting the group of the pose values based on each of the pose values of the group being associated with the same source identifier in the spatio-temporal model.

7. The method of claim 1, wherein the spatio-temporal model further defines, for each of the pose values, a source identifier that indicates a corresponding source of a corresponding observation on which the corresponding pose value is based, and wherein selecting the group of the pose values further comprises:

selecting the group of the pose values based on determining the pose value of the group are associated with multiple source identifiers in the spatio-temporal model and that a quantity of the pose values of the group satisfies a threshold.

8. A system, comprising:

one or more computer readable media storing a spatio-temporal model,

the spatio-temporal model generated based on received observations from multiple robots operating in an environment over a period of time, and

the spatio-temporal model defining, for each of a plurality of objects in the environment, a plurality of pose values and corresponding times for the pose values;

a pose value selection component comprising one or more processors and instructions that, when executed by one or more of the processors, cause one or more of the processors to;

access the spatio-temporal model stored in the computer readable media;

identify an object of the objects in the environment, and a target time;

select, from the spatio-temporal model, a group of pose values for the object based on proximity of the target time to the corresponding times for the pose values of the group; and

generate a pose for the object for the target time based on the selected group of the pose values.

9. The system of claim 8, wherein, in executing the instructions, one or more of the processors of the pose value selection component are further to:

provide the pose for the object to at least one of the multiple robots;

wherein in response to receiving the pose for the object, the at least one of the multiple robots utilizes the pose for path planning.

10. The system of claim 8, wherein, in executing the instructions, one or more of the processors of the pose value selection component are further to:

generate robot control commands based on the pose for the object.

11. The system of claim 8, wherein, in executing the instructions, one or more of the processors of the pose value selection component are further to:

identify an area indication for the target time that defines one or more areas;

wherein in selecting the group of the pose values, one or more of the processors are to select the group of the pose values based on the pose values of the group defining a position that is proximal to or within the one or more areas defined by the area indication.

12. The system of claim 8, wherein the spatio-temporal model further defines, for each of the pose values, an uncertainty measure for the pose value, and wherein in selecting the group of the pose values, one or more of the processors are to:

select the group of the pose values based on the uncertainty measures associated with the pose values of the group.

13. The system of claim 8, wherein the spatio-temporal model further defines, for each of the pose values, a source identifier that indicates a corresponding source of a corresponding observation on which the corresponding pose value is based, and wherein in selecting the group of the pose values, one or more of the processors are to:

select the group of the pose values based on each of the pose values of the group being associated with the same source identifier in the spatio-temporal model.

14. The system of claim 8, wherein the spatio-temporal model further defines, for each of the pose values, a source identifier that indicates a corresponding source of a corresponding observation on which the corresponding pose value is based, and wherein in selecting the group of the pose values, one or more of the processors are to:

select the group of the pose values based on determining the pose value of the group are associated with multiple source identifiers in the spatio-temporal model and that a quantity of the pose values of the group satisfies a threshold.

15. A system comprising:

a plurality of robots in an environment providing observations over a period of time;

a spatio-temporal model generation component, the spatio-temporal model generation component comprising one or more processors and instructions that, when executed by one or more of the processors, cause one or more of the processors to;

generate a spatio-temporal model based on the observations provided by the plurality of robots,

the spatio-temporal model defining, for each of a plurality of objects in the environment, a plurality of pose values and corresponding times for the pose values;

a pose value selection component comprising one or more additional processors and additional instructions that, when executed by one or more of the additional processors, cause one or more of the additional processors to;

access the spatio-temporal model stored in the computer readable media;

identify an object of the objects in the environment, and a target time;

select, from the spatio-temporal model, a group of pose values for the object based on proximity of the target time to the corresponding times for the pose values of the group; and

generate a pose for the object for the target time based on the selected group of the pose values.