System and method for surgical telementoring

• US 9,560,318 B2
• Filed: 10/05/2015
• Issued: 01/31/2017
• Est. Priority Date: 12/21/2012
• Status: Expired due to Fees

• Current Assignee: Skysurgery, LLC
• Original Assignee: Skysurgery, LLC
• Inventors: Reina, G. Anthony, L'Esperance, James Omer
• Primary Examiner(s): Gauthier, Gerald
• Application Number:
  US14/875,346
• Publication Number:
  US 20160028994A1
• Time in Patent Office: 484 Days
• Field of Search: 340/572.1, 345/537, 345/629, 345/8, 345/420, 345/423, 348/14.06, 348/14.07, 348/231.5, 348/14.08, 348/65, 348/143, 382/104, 382/128, 382/159, 382/173, 382/230, 382/237, 600/410, 600/437, 600/441, 705/14.54, 705/3, 709/219, 715/230, 715/753, 715/756, 715/716, 715/863, 250/310, 434/37
• US Class Current: 1/1
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: G06T 9/001 Model-based coding, e.g. wire frame
• CPC Class Codes: G16H 20/40 relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
• CPC Class Codes: G16H 30/20 for handling medical images, e.g. DICOM, HL7 or PACS
• CPC Class Codes: G16H 30/40 for processing medical images, e.g. editing
• CPC Class Codes: G16H 40/67 for remote operation
• CPC Class Codes: G16H 50/50 for simulation or modelling of medical disorders
• CPC Class Codes: H04N 19/25 with scene description coding, e.g. binary format for scenes [BIFS] compression
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04N 7/155 involving storage of or access to video conference sessions tracking arrangements for later retrieval of a computer conference content or participants activities H04L12/1831

Abstract

Apparatus and method for receiving and transmitting streaming live imagery data and audio signals in real time is provided. Imagery data and audio signals are acquired through a telestreamer input device and streamed to one or more remote recipients, allowing remote operators to electronically collaborate by telestrating, annotating, and sketching image overlays. Streaming video images displayed on a monitor are superimposed onto a virtual mesh projected via computer graphics. Vertices of the virtual mesh move according to a computational physics engine. Virtual tools are also superimposed onto a virtual mesh projected via computer graphics. The virtual tools interact with the virtual mesh to deliver real time, realistic modifications of the streaming image data. Recursive positioning of mesh layers and creation of a multi-layered virtual mesh enhance the realistic nature of the modified streaming image data.

Claims

1. A computer implemented method where one or more processors perform steps comprising:

receiving one or more real time images via a computer network interface, wherein the one or more real time images are aligned to at least one virtual mesh comprising a plurality of vertices and wherein the one or more illustrations are aligned to the virtual mesh;

storing said one or more real time images in a non-transitory computer readable medium;

associating one or more illustrations with the one or more real time images;

aligning the one or more illustrations with the one or more real time images; and

successively presenting on a display the one or more real time images with the one or more illustrations overlayed on the one or more real time images in accordance with said aligning.

2. The method of claim 1, further comprising:

moving a first of the one or more vertices of the at least one virtual mesh;

distorting a portion of the one or more real time images aligned with said first of the one or more vertices of the at least one virtual mesh;

presenting on the display the distorted portion of the one or more real time images.

3. The method of claim 2, wherein moving a first of the one or more vertices of the at least one virtual mesh is responsive to receiving an instruction to move a first of the one or more illustrations.

4. The method of claim 2, wherein distorting comprises stretching said portion of the one or more real time images.

5. The method of claim 2, wherein distorting comprises compressing said portion of the one or more real time images.

6. The method of claim 2, wherein distorting comprises separating a first section of said portion of the one or more real time images from a second section of said portion of the one or more real time images.

7. The method of claim 6, wherein said separated portion of the one or more real time images appears to be cut when presented on the display.

8. A method comprising:

projecting video images onto a virtual mesh configured to move according to computational model of an object presented in the video images, wherein the virtual mesh is constructed by computer graphics as a rectangle comprising equilateral triangles having interconnected vertices, and wherein movement between vertices of the virtual mesh is calculated via physics-based calculations; and

constructing a UV-map to project the video images onto the virtual mesh, wherein the projected video images are transformed as corresponding positions of the vertices of the equilateral triangles in the virtual mesh move based on the physics-based calculations.

9. The method of claim 8, wherein computer-generated, virtual tools are overlaid on the video images and manipulate the images in a realistic manner based on the computational model, wherein said virtual tools comprise three-dimensional renderings of at least one of scissors, sutures, and forceps which can be used to cut, stitch, and manipulate points within the video images and appear to react in a realistic manner in accordance with the virtual tools based on the physics-based calculations.