Automated steering systems and methods for a robotic endoscope

• US 10,646,288 B2
• Filed: 03/22/2018
• Issued: 05/12/2020
• Est. Priority Date: 04/12/2017
• Status: Active Grant

• Current Assignee: Iemis Limited
• Original Assignee: Bio-medical Engineering (HK) Limited
• Inventors: Yeung, Chung-Kwong, Chu, Steven, Chan, Wenling, Pang, Tin Chak Johnson, Chung, Hon Chung
• Primary Examiner(s): Henderson, Ryan N
• Assistant Examiner(s): Wu, Pamela F
• Application Number:
  US15/928,843
• Publication Number:
  US 20180296281A1
• Time in Patent Office: 782 Days
• Field of Search: 600117-118, 600146, 600148-149, 600128
• CPC Class Codes: A61B 1/000096 using artificial intelligence
• CPC Class Codes: A61B 1/00097 Sensors
• CPC Class Codes: A61B 1/0016 using motor drive units
• CPC Class Codes: A61B 1/31 for the rectum, e.g. proctoscopes, sigmoidoscopes , colonoscopes
• CPC Class Codes: A61B 2034/301 for introducing or steering flexible instruments inserted into the body, e.g. catheters or endoscopes
• CPC Class Codes: A61B 2034/303 specifically adapted for manipulations within body lumens, e.g. within lumen of gut, spine, or blood vessels
• CPC Class Codes: A61B 2090/065 for measuring contact or contact pressure
• CPC Class Codes: A61B 2576/00 Medical imaging apparatus involving image processing or analysis A61B1/00009, A61B6/52 and A61B8/52 take precedence; image processing per se G06T; details of algorithms for analysing medical images G06T7/0012
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: A61B 34/32 operating autonomously
• CPC Class Codes: A61B 5/065 Determining position of the probe employing exclusively positioning means located on or in the probe, e.g. using position sensors arranged on the probe
• CPC Class Codes: G05B 2219/45118 Endoscopic, laparoscopic manipulator
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06T 2207/10068 Endoscopic image
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30028 Colon; Small intestine
• CPC Class Codes: G06T 2207/30172 Centreline of tubular or elongated structure
• CPC Class Codes: G06T 7/13 : Edge detection
• CPC Class Codes: G06T 7/174 : involving the use of two or more images
• CPC Class Codes: G06T 7/64 : of convexity or concavity
• CPC Class Codes: G06V 10/44 : Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 10/764 : using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 : using neural networks
• CPC Class Codes: G06V 2201/03 : Recognition of patterns in medical or anatomical images
• CPC Class Codes: G16H 30/40 : for processing medical images, e.g. editing

Abstract

Systems and methods for automated steering control of a robotic endoscope, e.g., a colonoscope, are provided. The control system may comprise: a) a first image sensor configured to capture a first input data stream comprising a series of two or more images of a lumen; and b) one or more processors that are individually or collectively configured to generate a steering control output signal based on an analysis of data derived from the first input data stream using a machine learning architecture, wherein the steering control output signal adapts to changes in the data of the first input data stream in real time.

Claims

1. A control system for providing an adaptive steering control output signal for steering a robotic endoscope, the control system comprising:

a) a first image sensor configured to capture a first input data stream comprising a series of two or more images of a lumen; and

b) one or more processors that are individually or collectively configured to;

i) perform automated image processing and feature extraction on one or more of the images using a combination of at least two different image processing algorithms simultaneously to generate an input data set comprising at least two different estimates of a location of a lumen center;

ii) output a steering direction based on an analysis of the input data set using an artificial neural network that has been trained using a training data set, the training data set comprises recorded image data from previous endoscopic procedures and empirical steering control instructions provided to a virtual steering mechanism by a surgeon or other skilled operator during simulated endoscopic procedures, wherein the steering direction adapts to changes in the input data set in real time, and wherein the steering direction need not be aligned with the estimated location(s) of the lumen center; and

iii) generate the adaptive steering control output signal in real time.

2. The control system of claim 1, wherein the first input data stream comprises video data.

3. The control system of claim 1, wherein the automated image processing and feature extraction further comprises a determination of the location of a wall of the lumen, a determination of the location of an obstacle, a calculation of a confidence level for determining the location of the center of the lumen, a determination of a loss of or change in the integrity of lumen center position data, or any combination thereof.

4. The control system of claim 1, wherein the center of the lumen is not visible within the one or more images, and a predicted position for the center of the lumen is calculated based on motion vectors derived from the center positions of the lumen in two or more images previously captured by the first image sensor.

5. The control system of claim 1, wherein the automated image processing comprises identifying areas of high light reflection and thereby determining the proximity of a lumen wall or an obstacle to a distal end of the robotic endoscope based on a relationship between light reflection and proximity.

6. The control system of claim 1, wherein one estimate of the location of the center of the lumen comprises:

(i) generating a plurality of vectors, where each vector is aligned in a direction that is normal to a local segment of a contour identified in an image, and (ii) counting the number of vectors that intersect a given pixel in the image.

7. The control system of claim 1, wherein the one or more processors are further configured to generate the adaptive steering control output signal to direct the robotic endoscope to search along a specified route within the lumen when a confidence level for determining the location of the center of the lumen relative to a distal end of the robotic endoscope is below a first threshold, and to repeat the steps of:

(i) performing automated image processing and feature extraction on one or more images of at least the first input data stream to identify the center position of the lumen,

(ii) calculating a new confidence level for the determination of the location of the center of the lumen, and

(iii) generating the steering control output signal to direct the robotic endoscope to search along the specified route until the new confidence level is above a second threshold.

8. The control system of claim 7, wherein the confidence level is calculated based on an intensity calculation, a moment of inertia calculation, and/or a convexity calculation for the location of the center of the lumen determined from the first input data stream.

9. The control system of claim 7, wherein the specified route is a pre-determined route for steering the distal end of the robotic endoscope selected from the group consisting of a spiral search pattern, a rectilinear grid search pattern, a circular search pattern, or any combination thereof.

10. The control system of claim 1, wherein the at least two different image processing algorithms are selected from the group consisting of a dark region method, a mucosal pattern method, a light gradient method, and an optical flow method.

11. The control system of claim 1, further comprising a second image sensor that is used in conjunction with the first image sensor to provide a stereoscopic image stream that comprises depth information about the lumen proximal to a distal end of the robotic endoscope.

12. The control system of claim 1, wherein one of the at least two different image processing algorithms comprises an optical flow method.

13. The control system of claim 1, wherein the artificial neural network is trained using a training dataset that comprises simulated image data, wherein the simulated image data comprises an adjustment to one or more properties of the recorded image data.

14. The control system of claim 1, wherein the artificial neural network comprises a convolutional neural network.

15. The control system of claim 1, wherein the artificial neural network comprises a recurrent neural network.

16. The control system of claim 1, wherein the recurrent neural network comprises a recurrent convolutional neural network.

17. A method for providing an adaptive steering control output signal for steering a robotic endoscope, the method comprising:

a) providing a first input data stream comprising a series of two or more images of a lumen;

b) performing automated image processing and feature extraction on one or more of the images using a combination of at least two different image processing algorithms simultaneously to generate an input data set comprising at least two different estimates of a location of a lumen center;

c) outputting a steering direction based on an analysis of the input data set using an artificial neural network that has been trained using a training data set, the training data set comprises recorded image data from previous endoscopic procedures and empirical steering control instructions provided to a virtual steering mechanism by a surgeon or other skilled operator during simulated endoscopic procedures, wherein the steering direction adapts to changes in the input data set in real time, and wherein the steering direction need not be aligned with the estimated location(s) of the lumen center; and

d) generating the adaptive steering control output signal in real time.

18. The method of claim 17, wherein the first input data stream comprises video data.

19. The method of claim 17, wherein the automated image processing and feature extraction further comprises a determination of the location of a wall of the lumen, a determination of the location of an obstacle, a calculation of a confidence level for determining the location of the center of the lumen, a determination of a loss of or change in the integrity of lumen center position data, or any combination thereof.

20. The method of claim 17, wherein the center of the lumen is not visible within the one or more images, and a predicted position for the center of the lumen is calculated based on motion vectors derived from the center positions of the lumen in two or more images previously captured by a first image sensor.

21. The method of claim 17, wherein the automated image processing comprises identifying areas of high light reflection and thereby determining the proximity of a lumen wall or an obstacle to the distal end of the robotic endoscope based on a relationship between light reflection and proximity.

22. The method system of claim 17, wherein one estimate of the location of the center of the lumen comprises:

(i) generating a plurality of vectors, where each vector is aligned in a direction that is normal to a local segment of a contour identified in an image, and (ii) counting the number of vectors that intersect a given pixel in the image.

23. The method of claim 17, further comprising generating a steering control output signal to direct the robotic endoscope to search along a specified route within the lumen when a confidence level for determining the location of the center of the lumen relative to the distal end of the robotic endoscope is below a first threshold, and repeating the steps of:

(i) performing automated image processing and feature extraction on one or more images of at least the first input data stream to identify the center position of the lumen,

(ii) calculating a new confidence level for the determination of the location of the center of the lumen, and

(iii) generating the steering control output signal to direct the robotic endoscope to search along the specified route until the new confidence level is above a second threshold.

24. The method of claim 23, wherein the confidence level is calculated based on an intensity calculation, a moment of inertia calculation, and/or a convexity calculation for the location of the center of the lumen determined from the first input data stream.

25. The method of claim 23, wherein the specified route is a pre-determined route for steering the distal end of the robotic endoscope selected from the group consisting of a spiral search pattern, a rectilinear grid search pattern, a circular search pattern, or any combination thereof.

26. The method of claim 17, wherein the at least two different image processing algorithms are selected from the group consisting of a dark region method, a mucosal pattern method, a light gradient method, and an optical flow method.

27. The method of claim 17, further comprising providing a second input data stream comprising a series of two or more images of the lumen, and wherein the automated image processing of step (b) is applied to both the first input data stream and the second input data stream and provides a stereoscopic image stream that comprises depth information about the lumen proximal to a distal end of the robotic endoscope.

28. The method of claim 17, wherein one of the at least two different image processing algorithms comprises an optical flow method.

29. The method of claim 17, wherein the artificial neural network is trained using a training dataset that comprises simulated image data, wherein the simulated image data comprises an adjustment to one or more properties of the recorded image data.

30. The method of claim 17, wherein the artificial neural network comprises a convolutional neural network.

31. The method of claim 17, wherein the artificial neural network comprises a recurrent neural network.

32. The method of claim 17, wherein the recurrent neural network comprises a recurrent convolutional neural network.

33. A robotic endoscope, comprising:

a) an elongated body structure comprising one or more actuation units and a steerable distal portion; and

b) a control system comprising;

i) a first image sensor configured to capture a first input data stream comprising a series of two or more images of a lumen; and

ii) one or more processors that are individually or collectively configured to;

i) perform automated image processing and feature extraction on one or more of the images using a combination of at least two different image processing algorithms simultaneously to generate an input data set comprising at least two different estimates of a location of a lumen center;

ii) output a steering direction based on an analysis of the input data set using an artificial neural network that has been trained using a training data set, the training data set comprises recorded image data from previous endoscopic procedures and empirical steering control instructions provided to a virtual steering mechanism by a surgeon or other skilled operator during simulated endoscopic procedures, wherein the steering direction adapts to changes in the input data set in real time, and wherein the steering direction need not be aligned with the estimated location(s) of the lumen center; and

iii) generate an adaptive steering control output signal to control the one or more actuation units in real time.

34. The robotic endoscope of claim 33, wherein the elongated body structure further comprises one or more wires that are driven by the one or more actuation units to steer the steerable distal portion.

35. The robotic endoscope of claim 33, wherein the elongated body structure comprises an elongated elastomeric body structure.

36. The robotic endoscope of claim 33, wherein the automated image processing and feature extraction further comprises a determination of the location of a wall of the lumen, a determination of the location of an obstacle, a calculation of a confidence level for determining the location of the center of the lumen, a determination of a loss of or change in the integrity of lumen center position data, or any combination thereof.

37. The robotic endoscope system of claim 33, wherein the control system comprises a second image sensor that is used in conjunction with the first image sensor to provide a stereoscopic image stream that comprises depth information about the lumen proximal to a distal end of the robotic endoscope.

38. The robotic endoscope of claim 33, wherein one of the at least two different image processing algorithms comprises an optical flow method.

39. The robotic endoscope of claim 33, wherein the artificial neural network is trained using a training dataset that comprises simulated image data, wherein the simulated image data comprises an adjustment to one or more properties of the recorded image data.

40. The robotic endoscope of claim 33, wherein the artificial neural network comprises a convolutional neural network.

41. The robotic endoscope of claim 33, wherein the artificial neural network comprises a recurrent neural network.

42. The robotic endoscope of claim 33, wherein the recurrent neural network comprises a recurrent convolutional neural network.