Using observations from one or more robots to generate a spatio-temporal model that defines pose values for a plurality of objects in an environment

• US 9,691,151 B1
• Filed: 08/25/2015
• Issued: 06/27/2017
• Est. Priority Date: 08/25/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: X Development LLC (Alphabet Inc.)
• Inventors: Anderson-Sprecher, Peter Elving, Mason, Julian Mac Neille, Saboo, Rohit Ramesh
• Primary Examiner(s): Dunphy, David F
• Application Number:
  US14/834,980
• Time in Patent Office: 672 Days
• CPC Class Codes: G05D 1/024 in combination with a laser lasers per se H01S
• CPC Class Codes: G05D 1/0251 extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/344 involving models
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

Methods, apparatus, systems, and computer-readable media are provided for generating and using a spatio-temporal model that defines pose values for a plurality of objects in an environment and corresponding times associated with the pose values. Some implementations relate to using observations for one or more robots in an environment to generate a spatio-temporal model that defines pose values and corresponding times for multiple objects in the environment. In some of those implementations, the model is generated based on uncertainty measures associated with the pose values. Some implementations relate to utilizing a generated spatio-temporal model to determine the pose for each of one or more objects an environment at a target time. The pose for an object at a target time is determined based on one or more pose values for the object selected based on a corresponding measurement time, uncertainty measure, and/or source associated with the pose values.

Claims

1. A computer-implemented method, comprising:

receiving observations for multiple robots in an environment over a period of time, wherein the observations each include;

an object observation generated by a corresponding robot of the robots that defines, for a corresponding time in the period of time;

a corresponding identifier of a corresponding object of multiple objects of the environment and a measured object pose for the corresponding object, the measured object pose generated based on at least one sensor of the corresponding robot, and

a localization observation for the corresponding robot that defines, for the corresponding time;

a measured source pose for the sensor utilized to generate the measured object pose;

generating, for each of the observations, a pose value for the corresponding object of the observation with respect to a reference frame, the generating based on the measured object pose of the observation;

generating, for each of the pose values, an uncertainty measure based on both the object observation and the localization observation of the observation utilized to generate the pose value, wherein the uncertainty measures are each indicative of a degree of uncertainty in a corresponding one of the pose values;

generating, based at least in part on the uncertainty measures, a spatio-temporal model that defines the pose values and the corresponding times for a plurality of the objects of the environment;

wherein generating the spatio-temporal model based at least in part on the uncertainty measures comprises one or both of;

filtering a plurality of the pose values and the corresponding times from the spatio-temporal model based on the uncertainty measures; and

assigning the uncertainty measures to the corresponding pose values in the spatio-temporal model;

accessing the generated spatio-temporal model to determine a pose for an object of the objects of the spatio-temporal model at a target time;

identifying one or more pose values for the object in the spatio-temporal model based on proximity of the target time to the corresponding measurement times for the pose values; and

determining a pose for the object for the target time based on at least one of the identified pose values.

2. The computer-implemented method of claim 1, wherein:

the object observations each further include corresponding first sensor data related to the sensor of the robot utilized to generate the corresponding measured object pose;

the localization observations each further include corresponding second sensor data related to at least one localization sensor utilized to generate the measured source pose; and

wherein generating, for each of the pose values, the uncertainty measure based on both the object observation and the localization observation comprises;

generating the uncertainty measure based on the corresponding first sensor data and the corresponding second sensor data.

3. The computer-implemented method of claim 2, wherein the first sensor data of a given observation of the observations indicates a sensor type of the sensor and the second sensor data indicates a sensor type of the localization sensor, and wherein generating the uncertainty measure for the given observation based on the first sensor data and the second sensor data comprises:

generating the uncertainty measure based on a first error measure associated with the sensor type of the sensor and a second error measure associated with the sensor type of the localization sensor.

4. The computer-implemented method of claim 2, wherein the first sensor data indicates a first error measure associated with the sensor and the second sensor data indicates a second error measure associated with the localization sensor.

5. The computer-implemented method of claim 1, wherein:

the object observations each further include a corresponding object observation uncertainty measure associated with the corresponding measured object pose;

the localization observations each further include a corresponding localization observation uncertainty measure associated with the corresponding measured source pose; and

wherein generating, for each of the pose values, the uncertainty measure based on both the object observation and the localization observation comprises;

generating the uncertainty measure based on the corresponding object observation uncertainty measure and the corresponding localization observation uncertainty measure.

6. The computer-implemented method of claim 5, wherein the object observation uncertainty measures and the corresponding localization observation uncertainty measures are each covariance measures.

7. The computer-implemented method of claim 1, wherein the localization observation for at least one of the observations is based at least in part on a sensor reading of a sensor located external to the corresponding robot.

8. The computer-implemented method of claim 1, wherein identifying the pose values for the object in the spatio-temporal model based on proximity of the target time to the corresponding measurement times for the pose values comprises selecting only a single pose value based on the measurement time for the single pose value being more proximal to the target time than the measurement time of any other of the pose values.

9. The computer-implemented method of claim 1, wherein identifying the pose values for the object in the spatio-temporal model comprises selecting a plurality of the pose values and wherein determining the pose for the object for the target time based on the pose values comprises:

determining the pose for the object based on the selected plurality of the pose values.

10. The computer-implemented method of claim 9, wherein generating the spatio-temporal model further comprises associating each of the pose values with a source identifier, wherein each of the source identifiers indicates a corresponding source of the corresponding observation on which the corresponding pose value is based, and wherein selecting the plurality of the pose values comprises:

selecting the plurality of the pose values based on each of the selected plurality of the pose values being associated with the same source identifier in the spatio-temporal model.

11. The computer-implemented method of claim 1, wherein generating the spatio-temporal model based at least in part on the uncertainty measures comprises assigning the uncertainty measures to the corresponding pose values in the spatio-temporal model.

12. A computer-implemented method, comprising:

receiving a group of observations for multiple robots in an environment over a period of time;

wherein the observations of the group each include;

an object observation generated by a corresponding one of the robots that defines, for a corresponding time in the period of time;

a corresponding identifier of a corresponding object of multiple objects of the environment, a measured object pose for the corresponding object, the measured object pose generated based on at least one sensor of the corresponding robot, and

a localization observation for the corresponding robot that defines, for the corresponding time;

a measured source pose for the sensor utilized to generate the measured object pose;

generating, for each of the observations of the group, pose values for a corresponding object with respect to a reference frame, wherein generating a given pose value of the pose values is based on the measured object pose for the corresponding object;

generating a spatio-temporal model that defines, for each of a plurality of the objects of the environment, the pose values, the corresponding times, and the corresponding robots for each of a plurality of the observations;

accessing the generated spatio-temporal model to determine a pose for an object of the objects of the spatio-temporal model at a target time;

selecting a plurality of pose values for the object in the spatio-temporal model based on proximity of the target time to the corresponding times for the pose values; and

determining the pose for the object for the target time based on the selected pose values.

13. The computer-implemented method of claim 12, wherein generating the spatio-temporal model further comprises associating each of the pose values with a source identifier, wherein each of the source identifiers indicates a corresponding source of the corresponding observation on which the corresponding pose value is based, and wherein selecting the plurality of the pose values comprises:

selecting the plurality of the pose values based on each of the selected plurality of the pose values being associated with the same source identifier in the spatio-temporal model.

14. The computer-implemented method of claim 12, wherein generating the spatio-temporal model further comprises associating each of the pose values with a source identifier, wherein each of the source identifiers indicates a corresponding source of the corresponding observation on which the corresponding pose value is based, and wherein selecting the plurality of the pose values comprises:

selecting the plurality of the pose values based on determining the selected plurality of the pose values are associated with multiple source identifiers in the spatio-temporal model and that a quantity of the selected plurality of the pose values satisfies a threshold.

15. The computer-implemented method of claim 12, further comprising:

receiving an area indication for the target time that defines one or more areas;

wherein selecting the plurality of the pose values is based on proximity of the target time to the corresponding times for the pose values and is based on the pose values defining a position that is proximal to or within the one or more areas defined by the area indication.

16. A system comprising:

a spatio-temporal model generation system comprising one or more computing systems;

a plurality of robots in an environment providing observations to the spatio-temporal model generation system over a period of time, wherein the observations each include;

an object observation generated by a corresponding robot of the robots that defines, for a corresponding time in the period of time;

a corresponding identifier of a corresponding object of multiple objects of the environment and a measured object pose for the corresponding object, the measured object pose generated based on at least one sensor of the corresponding robot, and

a localization observation for the corresponding robot that defines, for the corresponding time;

a measured source pose for the sensor utilized to generate the measured object pose;

wherein the spatio-temporal model generation system comprises instructions that, when executed by the one or more computing systems, cause the computing systems to;

generate, for each of the observations, a pose value for the corresponding object of the observation with respect to a reference frame, the generating based on the measured object pose of the observation;

generate, for each of the pose values, an uncertainty measure based on both the object observation and the localization observation of the observation utilized to generate the pose value, wherein the uncertainty measures are each indicative of a degree of uncertainty in a corresponding one of the pose values; and

generate, based at least in part on the uncertainty measures, a spatio-temporal model that defines the pose values and the corresponding times for a plurality of the objects of the environment;

wherein generating the spatio-temporal model based at least in part on the uncertainty measures comprises one or both of;

filtering a plurality of the pose values and the corresponding times from the spatio-temporal model based on the uncertainty measures; and

assigning the uncertainty measures to the corresponding pose values in the spatio-temporal model;

a pose values selection system that comprises one or more computing systems and instructions that, when executed by the one or more computing systems, cause the computing systems to;

access the generated spatio-temporal model to determine a pose for an object of the objects of the spatio-temporal model at a target time;

identify one or more pose values for the object in the spatio-temporal model based on proximity of the target time to the corresponding times for the pose values; and

determine a pose for the object for the target time based on at least one of the identified pose values.