Automated collection and labeling of object data

• US 9,878,447 B2
• Filed: 04/10/2015
• Issued: 01/30/2018
• Est. Priority Date: 04/10/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Thibodeau, Bryan J., Revow, Michael, Jalobeanu, Mihai, Shirakyan, Grigor
• Primary Examiner(s): Mott, Adam R
• Application Number:
  US14/683,810
• Publication Number:
  US 20160297068A1
• Time in Patent Office: 1,026 Days
• CPC Class Codes: B25J 9/163 learning, adaptive, model based, rule based expert control
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 19/408 characterised by data handling or data format, e.g. reading, buffering or conversion of data
• CPC Class Codes: G05B 2219/37008 Calibration of measuring system, probe, sensor
• CPC Class Codes: G05B 2219/37009 Calibration of vision system, camera, adapt light level
• CPC Class Codes: G05B 2219/37555 Camera detects orientation, position workpiece, points of workpiece
• CPC Class Codes: G06F 18/28 Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06V 10/772 Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: Y10S 901/02 Arm motion controller
• CPC Class Codes: Y10S 901/47 Optical

Abstract

Data about a physical object in a real-world environment is automatically collected and labeled. A mechanical device is used to maneuver the object into different poses within a three-dimensional workspace in the real-world environment. While the object is in each different pose an image of the object is input from one or more sensors and data specifying the pose is input from the mechanical device. The image of the object input from each of the sensors for each different pose is labeled with the data specifying the pose and with information identifying the object. A database for the object that includes these labeled images can be generated. The labeled images can also be used to train a detector and classifier to detect and recognize the object when it is in an environment that is similar to the real-world environment.

Claims

1. A system for automatically collecting and labeling data about a physical object, comprising:

a mechanical device configured to maneuver the object within a three-dimensional (3D) workspace in a real-world environment;

one or more sensors configured to capture and output images of the object when it is within one or more prescribed regions of the workspace;

a computing device; and

a computer program having program modules executable by the computing device, the computing device being directed by the program modules of the computer program to,

calibrate the sensors to the coordinate system of the mechanical device,

for each of a plurality of different object poses within said regions of the workspace,

use the mechanical device to maneuver the object into the object pose,

input an image of the object from each of the sensors while the object is in the object pose,

input data specifying the object pose, the data specifying the pose being in the coordinate system of the mechanical device, and

label the image of the object input from each of the sensors with the data specifying the object pose and with information identifying the object, and

generate a database for the object, said database comprising the labeled image of the object input from each of the sensors for each of said different object poses.

2. The system of claim 1, wherein,

the mechanical device comprises a robotic arm comprising an end-effector configured to seize and release the object,

the action of using the mechanical device to maneuver the object into the pose comprises the actions of,

instructing the robotic arm to seize the object,

instructing the robotic arm to dispose the object onto a desired surface in a desired one of said regions of the workspace,

instructing the robotic arm to release the object, and

instructing the robotic arm to retract away from the object, and

the data specifying the pose comprises the coordinates of the pose of the end-effector at the time the object is released.

3. The system of claim 1, wherein,

the mechanical device comprises a computer numerical controlled (CNC) apparatus comprising a tool control point to which a platform is attached,

the object is disposed onto the platform,

the action of using the mechanical device to maneuver the object into the pose comprises an action of instructing the CNC apparatus to maneuver the platform into the pose, and

the data specifying the pose comprises the coordinates of the pose of the tool control point after the platform has been maneuvered into the pose.

4. The system of claim 1, wherein,

the mechanical device comprises a mobile robot comprising a platform onto which the object is disposed,

the action of using the mechanical device to maneuver the object into the pose comprises an action of instructing the mobile robot to maneuver into the pose, and

the data specifying the pose comprises the coordinates of the mobile robot after it has been maneuvered into the pose.

5. The system of claim 1, wherein the action of calibrating the sensors to the coordinate system of the mechanical device comprises the actions of:

for each of a variety of poses within said regions of the workspace,

inputting an image of a planar calibration pattern from each of the sensors while said pattern is in the pose, the metrics of said pattern being known, and

detecting feature points in the image of said pattern input from each of the sensors;

using a closed-form analytical solution to estimate the intrinsic and extrinsic parameters of each of the sensors from the feature points detected in the image of said pattern input from the sensor for each of said variety of poses; and

refining said estimated parameters using a nonlinear optimization technique based on the maximum-likelihood criterion.

6. The system of claim 1, wherein the action of calibrating the sensors to the coordinate system of the mechanical device comprises an action of using an automatic in-situ registration and calibration method to compute a coordinate system transformation function that maps coordinates between the coordinate system of the sensors and the coordinate system of the mechanical device.

7. The system of claim 1, wherein one or more of the sensors comprise a structured-light 3D scanner, and the image of the object input therefrom comprises a depth map.

8. The system of claim 1, wherein one or more of the sensors comprise a modulated-light 3D scanner, and the image of the object input therefrom comprises a depth map.

9. The system of claim 1, wherein one or more of the sensors comprise a time-of-flight camera, and the image of the object input therefrom comprises a depth map and a color image.

10. The system of claim 1, wherein one or more of the sensors comprise two or more visible light cameras that operate together as a stereo camera, and the image of the object input therefrom comprises a stereo image.

11. The system of claim 1, wherein one or more of the sensors comprise a single visible light camera, and the image of the object input therefrom comprises either a color image or a monochrome image.

12. The system of claim 1, wherein each of the sensors is static and is disposed at a different location in the environment.

13. The system of claim 1, wherein the computing device is further directed by the program modules of the computer program to:

input data specifying the location of each of the sensors relative to the location of the mechanical device, and

add said data specifying the location of each of the sensors to said database.

14. The system of claim 1, wherein,

the system further comprises one or more visible light sources,

each of the visible light sources is static and is disposed at a different location in the environment, and

the computing device is further directed by the program modules of the computer program to,

input data specifying the location of each of the visible light sources relative to the location of the mechanical device, and

add said data specifying the location of each of the visible light sources to said database.

15. The system of claim 1, wherein the action of labeling the image of the object input from each of the sensors with the data specifying the pose comprises the actions of:

using said calibration of the sensors to translate the data specifying the pose from the coordinate system of the mechanical device to the coordinate system of the sensors; and

labeling the image of the object input from each of the sensors with the data specifying the pose in both the coordinate system of the mechanical device and the coordinate system of the sensors.

16. The system of claim 1, wherein,

the system further comprises a data repository, and

the computing device is further directed by the program modules of the computer program to store said database in the data repository.

17. A system for training a detector and classifier to detect and recognize a physical object, comprising:

one or more computing devices, said computing devices being in communication with each other via a computer network whenever there is a plurality of computing devices; and

a computer program having program modules executable by the one or more computing devices, the one or more computing devices being directed by the program modules of the computer program to,

for each of a plurality of different object poses within one or more prescribed regions of a three-dimensional workspace in a real-world environment,

receive an image of the object from each of one or more sensors while the object is in the object pose, said sensors being configured to capture and output images of the object when it is within said regions of the workspace,

receive data specifying the object pose from a mechanical device configured to maneuver the object into the pose, the data specifying the pose being in the coordinate system of the mechanical device, the sensors being calibrated to the coordinate system of the mechanical device, and

label the image of the object received from each of the sensors with the data specifying the object pose, and

use the labeled image of the object received from each of the sensors for each of said different object poses to train the detector and classifier, said trained detector and classifier then being able to detect and recognize the object when it is in an environment similar to the real-world environment.

18. The system of claim 17, wherein said detector and classifier training is performed using a machine learning method comprising one of:

a logistic regression method;

or

a decision trees method;

or

a support vector machine method.

19. The system of claim 17, wherein the detector and classifier comprises one of:

a support vector machine;

or

an artificial neural network;

or

a Bayesian statistical detector and classifier.

20. A system for detecting and recognizing a physical object in a real-world environment, comprising:

a sensor configured to capture and output images of the object when it is in the real-world environment;

one or more computing devices, said computing devices being in communication with each other via a computer network whenever there is a plurality of computing devices; and

a computer program having program modules executable by the one or more computing devices, the one or more computing devices being directed by the program modules of the computer program to,

receive an image of the object from the sensor while the object is in a particular object pose in the real-world environment,

receive data specifying said particular object pose, and

use a pre-trained object detector and classifier to detect and recognize the object from the image thereof and the data specifying said particular object pose, said detector and classifier having been trained using a database comprising images of the object that were captured while the object was maneuvered by a mechanical device into a plurality of different object poses in an environment similar to the real-world environment, each of the images of the object in said database being labeled with data specifying the object pose the object was in when the image thereof was captured.