Multisensory data fusion system and method for autonomous robotic operation

• US 10,493,629 B2
• Filed: 01/05/2017
• Issued: 12/03/2019
• Est. Priority Date: 05/27/2016
• Status: Active Grant

• Current Assignee: GE Global Sourcing LLC (Westinghouse Air Brake Technologies Corporation)
• Original Assignee: GE Global Sourcing LLC (Westinghouse Air Brake Technologies Corporation)
• Inventors: Jain, Arpit, Theurer, Charles Burton, Kannan, Balajee, Sen, Shiraj, Sharma, Pramod, Li, Shuai, Liu, Shubao
• Primary Examiner(s): Harandi, Siamak
• Application Number:
  US15/399,313
• Publication Number:
  US 20170341237A1
• Time in Patent Office: 1,062 Days
• CPC Class Codes: B25J 9/1612 characterised by the hand, wrist, grip control
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G06F 18/256 of results relating to different input data, e.g. multimodal recognition
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/811 the classifiers operating on different input data, e.g. multi-modal recognition
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: H04N 13/10 Processing, recording or transmission of stereoscopic or multi-view image signals
• CPC Class Codes: H04N 13/20 Image signal generators
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: Y10S 901/01 Mobile robot
• CPC Class Codes: Y10S 901/47 Optical

Abstract

A robotic system includes one or more optical sensors configured to separately obtain two dimensional (2D) image data and three dimensional (3D) image data of a brake lever of a vehicle, a manipulator arm configured to grasp the brake lever of the vehicle, and a controller configured to compare the 2D image data with the 3D image data to identify one or more of a location or a pose of the brake lever of the vehicle. The controller is configured to control the manipulator arm to move toward, grasp, and actuate the brake lever of the vehicle based on the one or more of the location or the pose of the brake lever.

Claims

1. A robotic system comprising:

one or more optical sensors configured to separately obtain two dimensional (2D) image data and three dimensional (3D) image data of a vehicle; and

a controller configured to compare the 2D image data with the 3D image data to identify one or more of a location or a pose of a component of the vehicle;

wherein the controller is configured to identify potential objects from the 2D image data and the 3D image data, combine the potential objects, and examine the potential objects that are combined using simultaneous localization and mapping to identify the one or more of the location or the pose of the component of the vehicle.

2. The system of claim 1, wherein the controller is configured to fuse the 2D image data with the 3D image data to identify the one or more of the location or the pose of the component of the vehicle.

3. The system of claim 1, wherein the controller is configured to determine confidence parameters for the potential objects based on spatial consistency and temporal consistency of the potential objects.

4. The system of claim 1, wherein the one or more optical sensors include at least a first camera configured to obtain and provide the 2D image data to the controller and at least a different, second camera configured to obtain and provide the 3D image data to the controller.

5. The system of claim 4, wherein the at least a first camera includes a red-green-blue (RGB) camera and the at least a second camera includes one or more of a time of flight camera or a structured light sensor.

6. The system of claim 1, wherein the one or more optical sensors and the controller are disposed onboard a robotic vehicle that moves the one or more optical sensors and the controller relative to the vehicle.

7. The system of claim 1, further comprising a robotic arm, wherein the controller is configured to control the robotic arm to move toward, grasp, and actuate the component of the vehicle based on the one or more of the location or the pose of the component of the vehicle as identified from the 2D image data and the 3D image data.

8. The system of claim 1, wherein the component of the vehicle includes a brake lever.

9. The system of claim 1, wherein the controller is configured to compare the 2D image data with the 3D image data by identifying one or more shapes in the 2D image data, superimposing the 3D image data onto at least the one or more shapes in the 2D image data, extract potential objects from the 3D image data based on the 3D image data superimposed on the 2D image data, and select at least one of the potential objects as representative of the component of the vehicle.

10. The system of claim 1, wherein the 3D image data is a point cloud.

11. A robotic system comprising:

one or more optical sensors configured to separately obtain two dimensional (2D) image data and three dimensional (3D) image data of a brake lever of a vehicle;

a manipulator arm configured to grasp the brake lever of the vehicle; and

a controller configured to compare the 2D image data with the 3D image data to identify one or more of a location or a pose of the brake lever of the vehicle, wherein the controller is configured to control the manipulator arm to move toward, grasp, and actuate the brake lever of the vehicle based on the one or more of the location or the pose of the brake lever.

12. The system of claim 11, wherein the one or more optical sensors include at least a first camera configured to obtain and provide the 2D image data to the controller and at least a different, second camera configured to obtain and provide the 3D image data to the controller.

13. The system of claim 12, wherein the at least a first camera includes a red-green-blue (RGB) camera and the at least a second camera includes one or more of a time of flight camera or a structured light sensor.

14. The system of claim 11, wherein the one or more optical sensors and the controller are disposed onboard a robotic vehicle that moves the one or more optical sensors and the controller relative to the vehicle.

15. The system of claim 11, wherein the controller is configured to compare the 2D image data with the 3D image data by identifying one or more shapes in the 2D image data, superimposing the 3D image data onto at least the one or more shapes in the 2D image data, extract potential objects from the 3D image data based on the 3D image data superimposed on the 2D image data, and select at least one of the potential objects as representative of the brake lever of the vehicle.

16. The system of claim 11, wherein the 3D image data is a point cloud.

17. A method comprising:

obtaining two dimensional (2D) image data of a vehicle;

separately obtaining three dimensional (3D) image data of the vehicle;

comparing the 2D image data with the 3D image data;

identifying potential objects from the 2D image data and the 3D image data;

combining the potential objects, and examining the potential objects that are combined using simultaneous localization and mapping to identify the one or more of the location or the pose of the component of the vehicle; and

automatically controlling a robotic system to grasp and actuate the component of the vehicle to change a state of the vehicle based on the one or more of the location or the pose that is determined.

18. The method of claim 17, wherein obtaining the 2D image data is performed by at least a first camera and obtaining the 3D image data is performed by at least a different, second camera.

19. The method of claim 17, wherein the 2D image data is obtained from a red-green-blue (RGB) camera and the 3D image data is obtained from one or more of a time of flight camera or a structured light sensor.