Using object observations of mobile robots to generate a spatio-temporal object inventory, and using the inventory to determine monitoring parameters for the mobile robots

• US 10,195,740 B2
• Filed: 09/10/2015
• Issued: 02/05/2019
• Est. Priority Date: 09/10/2015
• Status: Active Grant

• Current Assignee: Boston Dynamics, Inc. (SoftBank Group Corp.)
• Original Assignee: X Development LLC (Alphabet Inc.)
• Inventors: Anderson-Sprecher, Peter Elving
• Primary Examiner(s): Kuan, John
• Application Number:
  US14/850,909
• Publication Number:
  US 20170072563A1
• Time in Patent Office: 1,244 Days
• Field of Search: None
• CPC Class Codes: B25J 5/00 Manipulators mounted on wheels or on carriages B25J1/00 takes precedence; programme-controlled manipulators B25J9/00 ; vehicle aspects B60, B62, e.g. remote-controlled steering for motor vehicles B62D1/24; control of position of vehicles G05D1/00
• CPC Class Codes: B25J 9/1664 characterised by motion, path, trajectory planning
• CPC Class Codes: B25J 9/1694 characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
• CPC Class Codes: G01B 11/14 for measuring distance or clearance between spaced objects or spaced apertures G01B11/26 takes precedence; rangefinders G01C3/00
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: G05D 1/0297 by controlling means in a control room
• CPC Class Codes: G06K 7/10376 the interrogation device being adapted for being moveable
• CPC Class Codes: G06Q 10/06315 Needs-based resource requirements planning or analysis
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: Y10S 901/01 Mobile robot

Abstract

Methods, apparatus, systems, and computer-readable media are provided for generating a spatio-temporal object inventory based on object observations from mobile robots and determining, based on the spatio-temporal object inventory, monitoring parameters for the mobile robots for one or more future time periods. Some implementations relate to using the spatio-temporal object inventory to determine a quantity of movements of objects that occur in one or more areas of the environment when one or more particular criteria are satisfied—and using that determination to determine monitoring parameters that can be utilized to provide commands to one or more of the mobile robots that influence one or more aspects of movements of the mobile robots at future time periods when the one or more particular criteria are also satisfied.

Claims

1. A computer-implemented method, comprising:

receiving, by a system of one or more computers, object observations from a plurality of mobile robots in an environment, wherein a plurality of the object observations each include;

an object identifier for a corresponding object, wherein the object identifier is determined based on sensor data from one or more sensors of a corresponding one of the mobile robots and

a measured object pose for the object identifier that indicates an object pose of the corresponding object;

generating, by the system, a spatio-temporal object inventory based on the object identifiers, the measured object poses, and measurement times of the object observations;

wherein the spatio-temporal object inventory defines, for each of a plurality of objects indicated by the object identifiers of the object observations, one or more poses for the object and corresponding measurement times for the one or more poses;

determining, by the system and based on the spatio-temporal object inventory, a quantity of movements of the objects that occurred when one or more criteria were satisfied,

wherein the one or more criteria include a particular temporal period, and

wherein the movements of the objects include at least one of;

movements of the objects into at least one area of the environment, and movements of the objects out of the area of the environment;

determining, by the system and based on a magnitude of the quantity of the movements that occurred when the criteria were satisfied, one or more monitoring parameters for the mobile robots for one or more future time periods when the criteria are also satisfied; and

providing, by the system to one or more of the mobile robots, commands based on the monitoring parameters, wherein the commands influence one or more aspects of movements of the mobile robots at the future time periods and resultantly influence a quantity of the object observations for the area of the environment received at the future time periods,

wherein the commands influence the one or more aspects of movements of the mobile robots at the future time periods based on the future time periods each satisfying the one or more criteria, including being a reoccurrence of the particular temporal period.

2. The method of claim 1, further comprising:

determining, by the system, a current or anticipated satisfaction of the one or more criteria, including a current or anticipated reoccurrence of the particular temporal period;

wherein providing the commands is in response to determining the current or anticipated satisfaction of the one or more criteria.

3. The computer-implemented method of claim 1, wherein the particular temporal period is one of:

one or more particular days of the week, one or more particular days of the year, or one or more time periods during one or more of the particular days of the week or the days of the year.

4. The computer-implemented method of claim 1, wherein determining the movements of the objects occurred when the one or more criteria were satisfied comprises:

determining, by the system, that a movement of the movements of the objects into the area occurred during the particular temporal period based on;

determining that a first pose of the one or more poses, for a selected object of the objects, is outside of the area and has a first corresponding measurement time of the corresponding measurement times that is temporally before the particular temporal period; and

determining that a second pose of the one or more poses, for the selected object, is within the area and has a second corresponding measurement time of the corresponding measurement times that is temporally within the particular temporal period.

5. The computer-implemented method of claim 1, wherein determining the movements of the objects occurred when the one or more criteria were satisfied comprises:

determining, by the system, that a movement of the movements of the objects into the area occurred during the particular temporal period based on;

determining that no pose that is temporally before the particular temporal period is defined for a selected object of the objects in the spatio-temporal object inventory; and

determining that a pose of the one or more poses for the selected object is within the area and has a corresponding measurement time of the corresponding measurement times that is temporally within the particular temporal period.

6. The computer-implemented method of claim 1, wherein determining the movements of the objects occurred when the one or more criteria were satisfied comprises:

determining, by the system, that a movement of the movements of the objects out of the area occurred during the particular temporal period based on;

determining that a first pose of the one or more poses, for a selected object of the objects, is inside the area and has a first corresponding measurement time of the corresponding measurement times that is temporally before the particular temporal period; and

determining that a second pose of the one or more poses for the selected object is outside of the area and has a second corresponding measurement time of the corresponding measurement times that is temporally within the particular temporal period.

7. The computer-implemented method of claim 1, wherein the one or more criteria include presence of a group of one or more humans in the area and further comprising:

receiving, by the system, human presence data indicating a plurality of temporal periods when the group of one or more humans were present in the area; and

determining the movements of the objects occurred when the one or more criteria were satisfied based on determining the movements of the objects occurred during one of the temporal periods when the group was present in the area.

8. The computer-implemented method of claim 1, wherein the one or more criteria include one or more particular quantities of one or more humans are present in the area, and further comprising:

receiving, by the system, human quantity data indicating a plurality of temporal periods when the one or more particular quantities of the one or more humans were present in the area; and

determining the movements of the objects occurred when the one or more criteria were satisfied based on determining the movements of the objects occurred during one of the temporal periods when the one or more particular quantities were present in the area.

9. The computer-implemented method of claim 1, wherein the one or more criteria include inventory criteria of the environment and further comprising:

determining the movements of the objects occurred when the one or more criteria were satisfied based on determining that the movements of the objects occurred when the inventory criteria was present in the environment.

10. The computer-implemented method of claim 9, wherein the inventory criteria of the environment is inventory criteria restricted to the area of the environment.

11. The computer-implemented method of claim 1, wherein the area is a subset of the environment.

12. The computer-implemented method of claim 1, wherein the commands that influence one or more aspects of movements of the mobile robots at the future time periods based on the monitoring parameters include commands that direct the mobile robots to the area during the future time periods.

13. The computer-implemented method of claim 1, wherein the commands that influence one or more aspects of movements of the mobile robots at the future time periods based on the monitoring parameters include commands that increase the likelihood that the mobile robots will select a path, that diverts from an optimal path, and that encounters the area during the future time periods.

14. A system comprising:

one or more computing systems;

a plurality of robots in an environment providing object observations to the one or more computing systems over a period of time, wherein the object observations each include;

an object identifier for a corresponding object, wherein the object identifier is determined based on sensor data from one or more sensors of a corresponding one of the mobile robots and

a measured object pose for the object identifier that indicates an object pose of the corresponding object;

wherein the one or more computing systems comprise instructions that, when executed by the one or more computing systems, cause the one or more computing systems to;

generate a spatio-temporal object inventory based on the object identifiers, the measured object poses, and measurement times of the object observations;

wherein the spatio-temporal object inventory defines, for each of a plurality of objects indicated by the object identifiers of the object observations, one or more poses for the object and corresponding measurement times for the one or more poses;

determine, based on the spatio-temporal object inventory, a quantity of movements of the objects that occurred when one or more criteria were satisfied,

wherein the one or more criteria include a particular temporal period, and

wherein the movements of the objects include at least one of;

movements of the objects into at least one area of the environment, and movements of the objects out of the area of the environment;

determine, based on a magnitude of the quantity of the movements that occurred when the criteria were satisfied, one or more monitoring parameters for the mobile robots for one or more future time periods when the criteria are also satisfied; and

provide, to one or more of the mobile robots, commands based on the monitoring parameters, wherein the commands influence one or more aspects of movements of the mobile robots at the future time periods and resultantly influence a quantity of the object observations for the area of the environment received at the future time periods,

wherein the commands influence the one or more aspects of movements of the mobile robots at the future time periods based on the future time periods each satisfying the one or more criteria, including being a reoccurrence of the particular temporal period.

15. The system of claim 14, wherein the one or more criteria include inventory criteria of the environment and wherein the instructions further comprise instructions that, when executed by the one or more computing systems, cause the one or more computing systems to:

determine the movements of the objects occurred when the one or more criteria were satisfied based on determining that the movements of the objects occurred when the inventory criteria was present in the environment.

16. The system of claim 14, wherein the particular temporal period is one of:

one or more particular days of the week, one or more particular days of the year, or one or more time periods during one or more of the particular days of the week or the days of the year.

17. The system of claim 14, wherein the one or more criteria include presence of a group of one or more humans in the area and wherein the instructions further comprise instructions that, when executed by the one or more computing systems, cause the one or more computing systems to:

receive human presence data indicating a plurality of temporal periods when the group of one or more humans were present in the area; and

determine the movements of the objects occurred when the one or more criteria were satisfied based on determining the movements of the objects occurred during one of the temporal periods when the group was present in the area.

18. The system of claim 14, wherein the one or more criteria include one or more particular quantities of one or more humans are present in the area, and wherein the instructions further comprise instructions that, when executed by the one or more computing systems, cause the one or more computing systems to:

receive human quantity data indicating a plurality of temporal periods when the one or more particular quantities of the one or more humans were present in the area; and

determine the movements of the objects occurred when the one or more criteria were satisfied based on determining the movements of the objects occurred during one of the temporal periods when the one or more particular quantities were present in the area.