Systems, devices, and methods for the distribution and collection of multimodal data associated with robots

• US 10,471,594 B2
• Filed: 11/30/2016
• Issued: 11/12/2019
• Est. Priority Date: 12/01/2015
• Status: Active Grant

• Current Assignee: Ocado Innovation Limited (Ocado Group PLC)
• Original Assignee: Kindred Systems Inc (LifePoint Health, Inc.)
• Inventors: Bergstra, James Sterling, Gildert, Suzanne, Rose, George Samuel
• Primary Examiner(s): Sample, Jonathan L
• Application Number:
  US15/365,527
• Publication Number:
  US 20170151667A1
• Time in Patent Office: 1,077 Days
• Field of Search: 700245-264
• CPC Class Codes: B25J 9/1656 characterised by programming, planning systems for manipulators
• CPC Class Codes: G05B 2219/39354 Operation, work space observer
• CPC Class Codes: G05B 2219/40405 Master slave position control

Abstract

A dynamic representation of a robot in an environment is produced, one or more observer agent collects data, and respective values of one or more metrics for the robot are computed based at least in part on the collected data. Tasks for the robot to perform are generated. Ratings and challenge questions are generated. A server may produce a user interface and a value of a metric based on collected observer data.

Claims

1. A method of operation in a computing system, including a robot, the method comprising:

causing, by a processor, a robot to perform a task in an environment;

capturing, by a camera coupled to the processor, media data of the robot performing the task in the environment, the media data including a video feed;

generating, by the processor, a dynamic representation of the robot performing the task in the environment from the captured media data;

providing, by the processor, the dynamic representation of the robot performing the task in the environment to at least one observer agent;

collecting data from at least one observer agent responsive to the observer agent observing the dynamic representation of the robot performing the task in the environment;

computing, by the processor, a value of a metric for the robot based on the collected data;

returning, by the processor, the value of the metric; and

receiving, by the processor, a score from the observer agent, wherein the score is related to the performance of the task by the robot, and wherein the score is used to train a machine learning model.

2. The method of claim 1, wherein the media data captured by the camera includes an audio feed.

3. The method of claim 1, further comprising:

storing, by the processor, the value of the metric.

4. The method of claim 1, further comprising:

defining, by the processor, the task for the robot to perform.

5. The method of claim 1 wherein collecting data from the at least one observer agent comprises:

receiving, by the processor, information that represents one or more ratings provided by a first observer agent within the at least one observer agent at an observer interface.

6. The method of claim 1 wherein collecting data from the at least one observer agent comprises:

receiving, by the processor, one or more ratings from an automated classification computing system.

7. The method of claim 1 wherein collecting data from the at least one observer agent comprises:

generating, by the processor, a challenge question for the at least one observer agent.

8. The method of claim 1, further comprising:

distributing, by the processor, an observer interface for the at least one observer agent.

9. The method of claim 8, further comprising:

generating, by the processor, the observer interface for the at least one observer agent.

10. The method of claim 8 wherein collecting data from at least one observer agent comprises:

receiving, by the processor, a first portion of data associated with the metric from the at least one observer agent via the observer interface.

11. The method of claim 10 wherein collecting data from at least one observer agent comprises:

receiving, by the processor, at least one time series from the at least one observer agent via the observer interface.

12. The method of claim 10, further comprising:

receiving, by the processor, at least categorical data from the at least one observer agent via the observer interface.

13. The method of claim 1, further comprising:

receiving, via a network, information that represents robot instructions created at an operator interface.

14. The method of claim 1 wherein computing the value for the metric for the robot based on the collected data comprises:

aggregating, by the processor, data from a plurality of observer agents, wherein the plurality of observer agents includes the at least one observer agent; and

determining, by the processor, the metric from the aggregated data.

15. The method of claim 1, further comprising:

stopping, by the processor, autonomous control of the robot; and

causing, by the processor, the robot to restart under user control.

16. The method of claim 1, further comprising:

stopping user control of the robot; and

causing, by the processor, the robot to restart under autonomous control.

17. The method of claim 1, further comprising:

receiving, by the processor, information that represents a control history of the robot over a first period.