Vision-guided electromagnetic robotic system

• US 9,259,844 B2
• Filed: 02/12/2014
• Issued: 02/16/2016
• Est. Priority Date: 02/12/2014
• Status: Active Grant

• Current Assignee: General Electric Company
• Original Assignee: General Electric Company
• Inventors: Xu, Yi, DeRose, Lynn Ann, Griffin, Weston Blaine, Mao, Ying, Tong, Xianqiao, Kannan, Balajee
• Primary Examiner(s): Patel, Jayesh A
• Application Number:
  US14/178,902
• Publication Number:
  US 20150224650A1
• Time in Patent Office: 734 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: B25J 15/0608 with magnetic holding means
• CPC Class Codes: B25J 9/1692 Calibration of manipulator
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 2219/36371 Barcode reader
• CPC Class Codes: G05B 2219/39393 Camera detects projected image, compare with reference image, position end effector
• CPC Class Codes: G05B 2219/39567 Use electromagnetic attraction to bring robot hand in contact with workpiece
• CPC Class Codes: G05B 2219/40053 Pick 3-D object from pile of objects
• CPC Class Codes: G05B 2219/40564 Recognize shape, contour of object, extract position and orientation
• CPC Class Codes: Y10S 901/02 Arm motion controller
• CPC Class Codes: Y10S 901/40 Vacuum or mangetic
• CPC Class Codes: Y10S 901/47 Optical
• CPC Class Codes: Y10S 902/02 Protects transmitted data, e.g. encryption or decryption

Abstract

Embodiments of the disclosure comprise a vision-based robotic manipulation system that identifies, singulates, and removes an individual surgical instrument from a cluttered environment. A computer-vision algorithm incorporated with the system is robust against optical challenges such as changing light conditions, specularities, and inter-reflections among various surgical instruments. The system estimates 2D pose (as opposed to a more challenging 3D pose) using a camera with normal room lighting to identify each object, including its specified data matrix barcode; and incorporates the use of a robotic arm with a compliant electromagnetic gripper to handle objects having large shape variance. The system can then relocate the objects as desired. The robot manipulator is utilized in hospital and research settings, manufacturing, and sterile environments.

Claims

1. A computer-implemented method of detecting and manipulating a plurality of instruments, the computer including a computer processor, and a camera with a barcode reading software module;

the method comprising the steps of;

creating a template image of each of the instruments by way of the camera and storing the template images in memory;

capturing an input image of the plurality of instruments;

identifying each of the instruments by using the barcode reading software module that (i) reads an identification tag from each of the instruments and (ii) localizes reference points of the identification tag, each of the identification tags encoding a unique identifier;

retrieving the template image from memory that corresponds to the identification tag;

aligning one or more edges of an instrument in the input image with edges of the instrument in the template image to compute an initial affine transformation;

computing an occupancy map that models one or more intersections between the instruments; and

synthesizing occlusion relationships that correspond to one or more hypotheses, wherein the input image is compared against at least a first hypothesis and a second hypothesis;

wherein the template image and the input image are two dimensional and utilized in combination to select a top-most instrument from a cluttered pile.

2. The computer-implemented method of claim 1, wherein each of the template images includes one or more segmented images of the instrument to create a foreground mask.

3. The computer-implemented method of claim 1, wherein the identification tag is a two-dimensional (2D) data matrix barcode.

4. The computer-implemented method of claim 3, wherein the reference points in said step of identifying are four corner points of the 2D data matrix barcode.

5. The computer-implemented method of claim 3, wherein the step of identifying includes localizing the position of a selected instrument in relation to the plurality of instruments which are remaining in a cluttered environment.

6. The computer-implemented method of claim 1, wherein the step of aligning the initial affine transformation is subsequently refined using a non-linear optimization.

7. The computer implemented method of claim 1, further comprising a step of refining the initial affine transformation using non-linear optimization by determining edges of the template image and edges of the input image to create a distance transform that computes the distance between the edges of the template image and the edges of the input image.

8. The computer-implemented method of claim 7, wherein the distance between edges of template image and edges of the input image is approximated by summing pixels at transformed template edge locations in the distance transform.

9. The computer-implemented method of claim 1, wherein the occupancy map is a single channel image and each bit of a pixel in the occupancy map is assigned to a foreground binary mask of the template image of one instrument.

10. The computer-implemented method of claim 1, wherein the step of synthesizing includes rendering each occlusion relationship in a different sequential order.

11. The computer-implemented method of claim 1, further comprising a step of picking up the top-most instrument using an electromagnetic gripper attached to a robotic arm.

12. The computer-implemented method of claim 11, wherein the electromagnetic gripper has an electromagnetic current value that corresponds to each of the instruments, the electromagnetic current value stored and referenced in a table indexed by the identification tag of each of the instruments.

13. The computer-implemented method of claim 12, wherein the electromagnetic gripper can interface with a robot controller or the computer processor.

14. The computer-implemented method of claim 13, wherein the electromagnetic current value is transferred to the computer processor or to the robot controller, alone or in combination, to automate control of the electromagnetic gripper.

15. A vision-guided robotic system for instrument singulation comprising:

a robotic arm;

a camera comprising a barcode reading software module to identify and image the plurality of instruments;

one or more instruments positioned in a clutter;

a computer processor to implement the steps of;

creating a template image of each of the instruments by way of the camera and storing the template images in memory;

capturing an input image of the plurality of instruments;

identifying each of the instruments by using the barcode reading software module that (i) reads an identification tag from each of the instruments and (ii) localizes reference points of the identification tag, each of the identification tags encoding a unique identifier;

retrieving the template image from memory that corresponds to the identification tag;

aligning one or more edges of an instrument in the input image with edges of the instrument in the template image to compute an initial affine transformation;

computing an occupancy map that models one or more intersections between the instruments; and

synthesizing occlusion relationships that correspond to one or more hypotheses, wherein the input image is compared against at least a first hypothesis and a second hypothesis;

wherein the template image and the input image are two dimensional and utilized in combination to select a specific instrument from the clutter.

16. The vision-guided robotic system of claim 15, wherein the robotic arm comprises an electromagnetic gripper.

17. The vision-guided robotic system of claim 16, wherein the electromagnetic gripper has an electromagnetic current value for each instrument that is stored in a table indexed by the identification tag of the instrument.

18. The vision-guided robotic system of claim 17, wherein the electromagnetic current value is provided to the computer processor or to the robotic arm to facilitate movement of the electromagnetic gripper.

19. The vision-guided robotic system of claim 18, wherein movement of the electromagnetic gripper is automated.

20. The vision-guided robotic system of claim 16, wherein the electromagnetic gripper comprises:

at least three passive joints along an x-axis, a y-axis, and z-axis;

each joint having a spring attached to a shaft to make the joints compliant;

the passive joints comprising a prismatic joint along the z-axis and two revolute joints along the x and y axes;

a rotary damper attached to each of the revolute joint shafts;

an electromagnet attached to a bottomside of the electromagnetic gripper, current of the electromagnet modulated by a power amplifier to generate gripping force via electromagnetic current to pick up a target instrument;

wherein the electromagnetic current for each instrument is determined empirically and stored in the processor as indexed by instrument identification tag;

an adaptor for attachment to the robotic arm; and

a load cell interfaced with a controller through an input-output (I/O) module to control the robotic arm and the electromagnetic gripper, alone or in combination.

21. The electromagnetic gripper of claim 20, wherein current of the electromagnet is modulated by a power amplifier to pick up a target instrument.

22. The electromagnetic gripper of claim 20, wherein the electromagnet interfaces with a robot controller that is capable of automating control of the robot arm and the electromagnet, alone or in combination.