SURGICAL SIMULATOR PROVIDING LABELED DATA

• US 20190325574A1
• Filed: 04/02/2019
• Published: 10/24/2019
• Est. Priority Date: 04/20/2018
• Status: Active Grant

• Current Assignee: Verily Life Sciences LLC (Alphabet Inc.)
• Original Assignee: Verily Life Sciences LLC (Alphabet Inc.)
• Inventors: Jin, Xing, Barral, Joelle K., Yang, Lin, Habbecke, Martin, Campion, Gianni
• Granted Patent: US 11,232,556 B2
• Time in Patent Office: Days
• CPC Class Codes: A61B 2034/101 Computer-aided simulation of surgical operations
• CPC Class Codes: A61B 2034/2055 Optical tracking systems
• CPC Class Codes: G06F 18/24143 Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30004 Biomedical image processing
• CPC Class Codes: G06T 7/0012 Biomedical image inspection
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/751 Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 2201/03 Recognition of patterns in medical or anatomical images
• CPC Class Codes: G06V 2201/034 of medical instruments
• CPC Class Codes: G09B 23/28 for medicine

Abstract

A surgical simulator for simulating a surgical scenario comprises a display system, a user interface, and a controller. The controller includes one or more processors coupled to memory that stores instructions that when executed cause the system to perform operations. The operations include generating simulated surgical videos, each representative of the surgical scenario. The operations further include associating simulated ground truth data from the simulation with the simulated surgical videos. The ground truth data corresponds to context information of at least one of a simulated surgical instrument, a simulated anatomical region, a simulated surgical task, or a simulated action. The operations further include annotating features of the simulated surgical videos based, at least in part, on the simulated ground truth data for training a machine learning model.

Claims

1. A method comprising:

generating, with a surgical simulator, simulated surgical videos each representative of a simulation of a surgical scenario;

associating simulated ground truth data from the simulation with the simulated surgical videos, wherein the simulated ground truth data corresponds to context information of at least one of a simulated surgical instrument, a simulated anatomical region, or a simulated surgical task; and

annotating, with the surgical simulator, features of the simulated surgical videos based, at least in part, on the simulated ground truth data for training a machine learning model.

2. The method of claim 1, further comprising:

pre-training the machine learning model with the features annotated from the simulated surgical videos, wherein the pre-training configures the machine learning model to probabilistically identify the features from unlabeled simulated videos corresponding to the surgical scenario.

3. The method of claim 2, further comprising:

receiving, from a surgical video database, real surgical videos representative of the surgical scenario in a real environment, wherein the real surgical videos include the features annotated for the machine learning model; and

after the pre-training, training the machine learning model with the real surgical videos, wherein the training configures the machine learning model to probabilistically identify the features from unlabeled real videos corresponding to the surgical scenario.

4. The method of claim 3, further comprising:

receiving, from the surgical video database, the unlabeled real videos representative of the surgical scenario in the real environment;

identifying the features of the unlabeled real videos with the machine learning model trained with the real surgical videos; and

annotating the unlabeled real videos by labeling the features of the unlabeled real videos identified by the machine learning model.

5. The method of claim 4, further comprising:

displaying annotated surgical videos corresponding to the unlabeled real videos with the features annotated by the machine learning model superimposed on a corresponding one of the unlabeled real videos.

6. The method of claim 2, further comprising:

providing, to a refiner neural network, simulated images from the simulated surgical videos before the pre-training; and

refining the simulated surgical videos with the refiner neural network, wherein the refiner neural network adjusts the simulated images until a discriminator neural network determines the simulated images are comparable to unlabeled real images within a first threshold, wherein the features annotated from the simulated surgical videos are included after the refining, and wherein the unlabeled real images are representative of the surgical scenario in the real environment.

7. The method of claim 6, further comprising:

receiving, from a surgical video database, real surgical videos representative of the surgical scenario in a real environment, wherein the real surgical videos include the features annotated for the machine learning model; and

after the pre-training, training the machine learning model with the real surgical videos, wherein the training configures the machine learning model to probabilistically identify the features from unlabeled real videos corresponding to the surgical scenario in the real environment.

8. The method of claim 7, further comprising:

receiving the unlabeled real videos from the surgical database;

identifying the features of the unlabeled real videos with the machine learning model trained with the real surgical videos; and

annotating the unlabeled surgical videos with the machine learning model by labeling the features of the unlabeled real videos identified by the machine learning model.

9. The method of claim 1, wherein the features correspond to at least one of segments of a surgical instrument, segments of an anatomical region, motion of the surgical instrument, surgical steps of a surgical procedure, surgical technique utilization, surgical event occurrence, or separation distance between the surgical instrument and the anatomical region.

10. The method of claim 1, wherein the context information for each of the simulated surgical videos includes at least one of three-dimensional spatial boundaries of the simulated surgical instrument, three-dimensional spatial boundaries of the simulated anatomical region, motion of the simulated surgical instrument, separation distance between the simulated surgical instrument and the simulated anatomical region, orientation of the simulated surgical instrument, temporal boundaries of one or more surgical steps of the simulated surgical task from the surgical scenario, temporal boundaries of a simulated surgical complication, or spatial boundaries of the simulated surgical complication.

11. A surgical simulator for simulating a surgical scenario, the surgical simulator comprising:

a display system adapted to show simulated surgical videos to a user of the surgical simulator;

a user interface adapted to correlate a physical action of the user with a simulated action of the surgical simulator;

a controller including one or more processors coupled to memory, the display system, and the user interface, wherein the memory stores instructions that when executed by the one or more processors cause the surgical simulator to perform operations including;

generating the simulated surgical videos, each representative of a simulation of the surgical scenario;

associating simulated ground truth data from the simulation with the simulated surgical videos, wherein the simulated ground truth data corresponds to context information of at least one of a simulated surgical instrument, a simulated anatomical region, a simulated surgical task, or the simulated action; and

annotating features of the simulated surgical videos based, at least in part, on the simulated ground truth data for training a machine learning model.

12. The surgical simulator of claim 11, wherein the controller includes additional instructions that when executed by the one or more processors cause the surgical simulator to perform further operations comprising:

pre-training the machine learning model with the features annotated from the simulated surgical videos, wherein the pre-training configures the machine learning model to probabilistically identify the features from unlabeled simulated videos corresponding to the surgical scenario.

13. The surgical simulator of claim 12, further comprising:

a surgical video database coupled to the controller, wherein the surgical database includes real surgical videos representative of the surgical scenario in a real environment, wherein the real surgical videos include the features annotated for the machine learning model, and

wherein the controller includes additional instructions that when executed by the one or more processors cause the surgical simulator to perform further operations including;

after the pre-training, training the machine learning model with the real surgical videos, wherein the training configures the machine learning model to probabilistically identify the features from unlabeled real videos corresponding to the surgical scenario.

14. The surgical simulator of claim 13, wherein the controller includes additional instructions that when executed by the one or more processors cause the surgical simulator to perform further operations including:

receiving, from the surgical video database, the unlabeled real videos each representative of the surgical scenario in the real environment;

identifying the features of the unlabeled real videos with the machine learning model trained with the real surgical videos; and

annotating the unlabeled real videos with the machine learning model by labeling the features of the unlabeled real videos identified by the machine learning model.

15. The surgical simulator of claim 11, wherein the controller includes additional instructions that when executed by the one or more processors cause the surgical simulator to perform further operations including:

providing, to a refiner neural network, simulated images from the simulated surgical videos; and

refining, via the refiner neural network, the simulated images to form refined surgical videos based, at least in part, on the simulated surgical videos, wherein the features annotated from the simulated surgical videos are included in the refined surgical videos, and wherein the refiner network is trained based on an adversarial loss from a discriminator neural network that compares the simulated images to unlabeled real images representative of the surgical scenario in the real environment.

16. The surgical simulator of claim 15, wherein the controller includes additional instructions that when executed by the one or more processors cause the surgical simulator to perform further operations including:

pre-training the machine learning model with the features annotated from the refined surgical videos, wherein the pre-training configures the machine learning model to probabilistically identify the features from unlabeled simulated videos corresponding to the surgical scenario;

receiving, from the surgical video database, real surgical videos representative of the surgical scenario in the real environment, wherein the real surgical videos include the features annotated for the machine learning model; and

after the pre-training, training the machine learning model with the real surgical videos, wherein the training configures the machine learning model to probabilistically identify the features from unlabeled real videos corresponding to the surgical scenario in the real environment.

17. The surgical simulator of claim 16, wherein the controller includes additional instructions that when executed by the one or more processors cause the surgical simulator to perform further operations including:

receiving, from the surgical video database, the unlabeled real videos;

identifying the features of the unlabeled real videos with the machine learning model trained with the real surgical videos; and

annotating the unlabeled surgical videos with the machine learning model by labeling the features of the unlabeled real videos identified by the machine learning model.

18. The surgical simulator of claim 11, wherein the features correspond to at least one of segments of a surgical instrument, segments of an anatomical region, motion of the surgical instrument, surgical steps of a surgical procedure, surgical technique utilization, surgical event occurrence, or separation distance between the surgical instrument and the anatomical region.

19. The surgical simulator of claim 11, wherein the context information for each of the simulated surgical videos includes at least one of three-dimensional spatial boundaries of the simulated surgical instrument, three-dimensional spatial boundaries of the simulated anatomical region, motion of the simulated surgical instrument, separation distance between the simulated surgical instrument and the simulated anatomical region, orientation of the simulated surgical instrument, temporal boundaries of one or more surgical steps of the simulated surgical task from the surgical scenario, temporal boundaries of a simulated surgical complication, or spatial boundaries of the simulated surgical complication.

20. A non-transitory computer-readable storage medium having stored thereon instructions which, when executed by one or more processing units, cause the one or more processing units to perform operations comprising:

obtaining simulated surgical videos from a surgical video database, wherein each of the simulated surgical videos is representative of a simulation of a surgical scenario generated by a surgical simulator, wherein each of the simulated surgical videos includes simulated ground truth data corresponding to context information of at least one of a simulated surgical instrument, a simulated anatomical region, or a simulated surgical task;

annotating features of the simulated surgical videos based, at least in part, on the simulated ground truth data for training a machine learning model;

pre-training the machine learning model with the features annotated from the simulated surgical videos, wherein the pre-training configures the machine learning model to probabilistically identify the features from unlabeled videos corresponding to the surgical scenario; and

saving the machine learning model and associated parameters configured by the pre-training to a data store included in the surgical simulator.

21. The non-transitory computer-readable medium of claim 20, wherein the instructions, which when executed by the one or more processing units, cause the one or more processing units to perform further operations comprising:

receiving, from a surgical video database, real surgical videos representative of the surgical scenario in a real environment, wherein the real surgical videos include the features annotated for the machine learning model; and

after the pre-training, training the machine learning model with the real surgical videos, wherein the training configures the machine learning model to probabilistically identify the features from unlabeled real videos corresponding to the surgical scenario.

22. The non-transitory computer-readable medium of claim 20, wherein the instructions, which when executed by the one or more processing units, cause the one or more processing units to perform further operations comprising:

providing, to a refiner neural network, simulated images from the simulated surgical videos before the pre-training; and

refining the simulated surgical videos with the refiner neural network, wherein the refiner neural network adjusts the simulated images until a discriminator neural network determines that the simulated images are comparable to unlabeled real images within a first threshold, wherein the features annotated from the simulated surgical videos are included after the refining, and wherein the unlabeled real images are representative of the surgical scenario in the real environment.

23. The non-transitory computer-readable medium of claim 20, wherein the instructions, which when executed by the one or more processing units, cause the one or more processing units to perform further operations comprising:

receiving, from the surgical video database, real surgical videos representative of the surgical scenario in a real environment, wherein the real surgical videos include the features annotated for the machine learning model; and

after the pre-training, training the machine learning model with the real surgical videos, wherein the training configures the machine learning model to probabilistically identify the features from unlabeled real videos corresponding to the surgical scenario in the real environment.