Automated system for medical video recording and storage

• US 10,342,410 B2
• Filed: 10/25/2017
• Issued: 07/09/2019
• Est. Priority Date: 10/26/2016
• Status: Active Grant

• Current Assignee: Virgo Surgical Video Solutions, Inc.
• Original Assignee: Virgo Surgical Video Solutions, Inc.
• Inventors: Schwartz, Matthew Zane, Strug, Ian Christopher, Guaraglia, David Oscar
• Primary Examiner(s): Nguyen, Huy T
• Application Number:
  US15/793,502
• Publication Number:
  US 20180110398A1
• Time in Patent Office: 622 Days
• CPC Class Codes: A61B 1/00006 of control signals
• CPC Class Codes: A61B 1/00009 of image signals during a use of endoscope
• CPC Class Codes: A61B 1/00016 using wireless means
• CPC Class Codes: A61B 1/0002 provided with data storages
• CPC Class Codes: A61B 1/0004 for electronic operation
• CPC Class Codes: A61B 1/00055 for alerting the user
• CPC Class Codes: G06F 2218/00 Aspects of pattern recognition specially adapted for signal processing
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/49 Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes
• CPC Class Codes: G06V 2201/031 of internal organs
• CPC Class Codes: G16H 30/20 for handling medical images, e.g. DICOM, HL7 or PACS
• CPC Class Codes: G16H 30/40 for processing medical images, e.g. editing
• CPC Class Codes: G16H 50/20 for computer-aided diagnosis, e.g. based on medical expert systems

Abstract

A method and device for automatically recording a video output signal from an endoscopy device. Video image data is received at a video capture device, and transmitted with associated metadata to a server. Prior to transmission, the video capture device analyzes the video image data to identify a recording start indicator as a function of predetermined threshold conditions. Once the indicator is identified, the device sends recording start timestamp to a backend server. The server indexes and stores the recorded video data as a function of the associated metadata in a database, and also records the recording the recording start timestamp data in the database. The device is further able to identify a recording stop indicator, and prepare recording stop timestamp data for transmission to the server.

Claims

1. A video capture device for automatically recording a video output signal from an endoscopy device during a recording session, comprising:

at least one input port configured for receiving the video output signal from the endoscopy device;

at least one processor in communication with the at least one input port,

a local storage device accessible by the first processor, and

at least one output device in communication with the processor and configured for accessing a network,

wherein the at least one processor is operable to analyze video image data of the video output signal, identify a recording start indicator for initiating the recording session as a function of predetermined threshold conditions for the video image data, and prepare and transmit recorded video data including the video image data to the network via the at least one output device upon identifying the recording start indicator,

wherein the predetermined threshold conditions are based on two or more elements selected from the group consisting of pixel values, frame-to-frame motion metrics, edge detection indicators, audio signals, neural networks, convolutional neural networks, and recurrent neural networks and

wherein the at least one processor identifies the recording start indicator to initiate the recording session only when the threshold conditions indicate that the medical device is displaying video image data internal to a patient.

2. The video capture device of claim 1, wherein the at least one processor comprises at least one central processing unit (CPU) and at least one graphics processing unit (CPU).

3. The video capture device of claim 1, wherein the endoscopy device is selected from the group consisting of arthroscopes, cystoscopes, hysteroscopes, laparoscopes, laryngoscopes, endoscopes, resectoscopes, sinuscopes, uteroscopes and colonoscopes.

4. The video capture device of claim 1, wherein the at least one processor is further operable to set a recording start timestamp and transmit the recording start timestamp to the network via the at least one output port.

5. The video capture device of claim 1, wherein the at least one processor is further operable to identify a recording stop indicator for the recording session as a function of the predetermined threshold conditions, set a recording stop timestamp and transmit the recording stop timestamp to the network via the at least one output port, and

wherein the at least one processor identifies the recording stop indicator to terminate the recording session when the threshold conditions indicate that the medical device is displaying video image data external to the patient.

6. The video capture device of claim 1, wherein the recorded video data further includes metadata associated with the video image data.

7. The video capture device of claim 6, wherein the associated metadata includes one or more elements selected from the group consisting of a calendar date, a timestamp, a location and a user identifier.

8. The video capture device of claim 7, wherein the user identifier comprises at least one of a physician identifier or a patient identifier.

9. The video capture device of claim 5, wherein:

the endoscopy device is selected from the group consisting of arthroscopes, cystoscopes, hysteroscopes, laparoscopes, laryngoscopes, endoscopes, resectoscopes, sinuscopes, uteroscopes and colonoscopes.

10. The video capture device of claim 1, wherein the processor is further capable to prepare the recorded video data by applying one or more of compression processing or encryption processing.

11. The video capture device of claim 1, further comprising at least one audio input port.

12. The video capture device of claim 1, wherein the output device comprises at least one of a physical output port or a wireless transmitter.

26. The video capture device of claim 1, wherein the predetermined threshold conditions comprise RGB pixel values.

27. The video capture device of claim 1, wherein the predetermined threshold conditions are based on at least one analysis element selected from the group consisting of pixel values, frame-to-frame motion metrics, edge detection indicators, and audio signals, and at least one neural network element selected from the group consisting of neural networks, convolutional neural networks, and recurrent neural networks, and

wherein the threshold conditions based on the at least one analysis element are adjusted by the at least one neural network element.

13. A method for automatically recording a video output signal from an endoscopy device during a recording session, comprising the steps of:

receiving video image data of the video output signal at a video capture device;

transmitting signals corresponding to the video image data and associated metadata to the server via the video capture device;

analyzing the video image data by the video capture device to identify a recording start indicator for the recording session as a function of predetermined threshold conditions, wherein the predetermined threshold conditions are based on two or more elements selected from the group consisting of pixel values, frame-to-frame motion metrics, edge detection indicators audio signals, neural networks, convolutional neural networks, and recurrent neural networks;

setting a recording start timestamp by the video capture device;

transmitting the recording start timestamp data over the network to the server via the video capture device;

indexing the recorded video data by the server as a function of the associated metadata;

storing the indexed video data for the recording session in a database accessible to the server; and

storing the recording start timestamp for the recording session in the database,

wherein the analyzing step identifies the recording start indicator to initiate the recording session only when the threshold conditions indicate that the medical device is displaying video image data internal to a patient.

14. The method of claim 13, further comprising the steps of:

receiving audio data from at least one microphone; and

transmitting signals corresponding to the audio data, associated metadata and an associated timestamp to the server via the video capture device.

15. The method of claim 13, wherein the step of transmitting signals corresponding to the video image data and associated metadata begins substantially coincidently with the step of transmitting recording start timestamp data.

16. The method of claim 13, wherein the step of transmitting signals corresponding to the video image data and associated metadata occurs continuously.

17. The method of claim 13, wherein the endoscopy device is selected from the group consisting of arthroscopes, cystoscopes, hysteroscopes, laparoscopes, laryngoscopes, endoscopes, resectoscopes, sinuscopes, uteroscopes and colonoscopes.

18. The method of claim 13, further comprising the steps of:

analyzing the video image data by the video capture device to identify a recording stop indicator as a function of the predetermined threshold conditions;

setting a recording stop timestamp for the recording session by the video capture device;

transmitting recording stop timestamp data to the server over the network via an output port of the video capture device; and

storing the recording start timestamp by the server in the database,

wherein the recording stop indicator terminates the recording session when the threshold conditions indicate that the medical device is displaying video image data external to the patient.

19. The method of claim 18, wherein the step of transmitting signals corresponding to the video image data and associated metadata terminates coincidently with the step of transmitting recording stop timestamp data.

20. The method of claim 13, wherein the associated metadata includes data indicating one or more elements selected from the group consisting of a calendar date, a timestamp, a location, and a user identifier.

21. The method of claim 20, wherein the user identifier comprises at least one of a physician identifier or a patient identifier.

22. The method of claim 18, wherein the analyzing steps determine that the medical device is either displaying video image data internal to a patient or external to the patient, as a function of the predetermined threshold conditions.

23. The method of claim 13, further comprising the steps of:

providing a network-accessible user portal by the server;

receiving one of a recording start indicator or a recording stop indicator for the recording session via the network-accessible user portal;

preparing a timestamp for the indicator received by the server; and

storing the timestamp for the server in the database.

24. The method of claim 23, wherein the recording start indicator or recording stop indicator is received by the server via a user interface of the portal by means of at least one of a keyboard device, a mouse, a foot pedal or a voice-activated device.

25. The method of claim 24, wherein a recording start indicator is received by the server, further comprising the step of:

applying a contention resolution rule to select only one of the recording start timestamp received from the video capture device or the recording start timestamp for the server, as the recording start timestamp for the recording session.

28. The method of claim 13, wherein the predetermined threshold conditions comprise RGB pixel values.

29. The method of claim 13, wherein the predetermined threshold conditions are based on at least one analysis element selected from the group consisting of pixel values, frame-to-frame motion metrics, edge detection indicators, and audio signals, and at least one neural network element selected from the group consisting of neural networks, convolutional neural networks, and recurrent neural networks, further comprising the step of

adjusting the threshold conditions based on the at least one analysis element by the at least one neural network element.