Gesture-based interface for a multi-modality medical imaging system

• US 10,368,836 B2
• Filed: 12/19/2013
• Issued: 08/06/2019
• Est. Priority Date: 12/26/2012
• Status: Active Grant

• Current Assignee: Volcano Corporation (Koninklijke Philips N.V.)
• Original Assignee: Volcano Corporation (Koninklijke Philips N.V.)
• Inventors: Merritt, Fergus, Cheline, Aaron J., Cohen, Asher
• Primary Examiner(s): Barrett, Ryan
• Application Number:
  US14/135,248
• Publication Number:
  US 20140181716A1
• Time in Patent Office: 2,056 Days
• CPC Class Codes: A61B 5/00 Measuring for diagnostic purposes radiation diagnosis A61B6/00; diagnosis by ultrasonic, sonic or infrasonic waves A61B8/00; Identification of persons
• CPC Class Codes: A61B 5/0022 Monitoring a patient using a global network, e.g. telephone networks, internet
• CPC Class Codes: A61B 5/0035 adapted for acquisition of images from more than one imaging mode, e.g. combining MRI and optical tomography
• CPC Class Codes: A61B 5/0066 Optical coherence imaging
• CPC Class Codes: A61B 5/0084 for introduction into the body, e.g. by catheters A61B5/1459 takes precedence
• CPC Class Codes: A61B 5/0095 by applying light and detecting acoustic waves, i.e. photoacoustic measurements
• CPC Class Codes: A61B 5/02007 Evaluating blood vessel condition, e.g. elasticity, compliance
• CPC Class Codes: A61B 5/021 Measuring pressure in heart or blood vessels A61B5/0205 takes precedence
• CPC Class Codes: A61B 5/029 Measuring or recording blood output from the heart, e.g. minute volume A61B8/065 takes precedence
• CPC Class Codes: A61B 5/107 Measuring physical dimensions, e.g. size of the entire body or parts thereof
• CPC Class Codes: A61B 5/4848 Monitoring or testing the effects of treatment, e.g. of medication
• CPC Class Codes: A61B 5/743 Displaying an image simultaneously with additional graphical information, e.g. symbols, charts, function plots
• CPC Class Codes: A61B 5/7435 Displaying user selection data, e.g. icons in a graphical user interface
• CPC Class Codes: A61B 5/7475 User input or interface means, e.g. keyboard, pointing device, joystick
• CPC Class Codes: A61B 5/748 Selection of a region of interest, e.g. using a graphics tablet
• CPC Class Codes: A61B 5/749 Voice-controlled interfaces
• CPC Class Codes: A61B 8/12 in body cavities or body tracts, e.g. by using catheters catheters per se A61M25/00
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G16H 30/20 : for handling medical images, e.g. DICOM, HL7 or PACS
• CPC Class Codes: G16H 30/40 : for processing medical images, e.g. editing
• CPC Class Codes: G16H 40/63 : for local operation
• CPC Class Codes: G16H 50/20 : for computer-aided diagnosis, e.g. based on medical expert systems

Abstract

Systems and methods for multi-modality data processing are provided. Some embodiments are particularly directed to interpreting gesture-based commands in a multi-modality processing system. In one embodiment, a method for interpreting user input in a medical processing system includes receiving a state designator corresponding to a mode of operation of the medical processing system, where the mode of operation includes a value representative of a modality selected from the group consisting of: IVUS, OCT, pressure, and flow. A list of active commands is generated based on the received state designator. A user input sequence is received from one or more user input devices. The medical processing system correlates the user input sequence to a command of the list of active commands, and the command is utilized to control operation of a component of the system. The list of active command may include a subset of commands common to multiple modalities.

Claims

1. A method for interpreting user input in a medical processing system, comprising:

receiving, by a gesture recognition module, a state designator corresponding to a mode of operation of the medical processing system, the medical processing system comprising the gesture recognition module and a plurality of modules associated with a plurality of modalities corresponding to medical data representative of a vessel of a patient, wherein the gesture recognition module comprises a gesture recognition engine, a gesture database, and an interface operable to communicate with the plurality of modules;

querying, by the gesture recognition engine, the gesture database of the gesture recognition module based on the state designator;

generating, by the gesture recognition engine, a list of active commands based on querying the gesture database, wherein the list of active commands is variable such that a different list of active commands is generated for a different modality of the plurality of modalities;

receiving, by the gesture recognition module, when the mode of operation corresponds to an IVUS modality, a user input sequence on an IVUS image from one or more user input devices, the user input sequence defining a first path on the IVUS image;

outputting, by the gesture recognition module, a second path on the IVUS image, the second path different from the first path and generated by the gesture recognition engine based on a vascular border of the vessel in the IVUS image, the vascular border identified by the medical processing system using a vascular detection algorithm;

correlating, by the gesture recognition engine, the second path to a single command of the list of active commands generated for the IVUS modality based on a shape of the second path and in response to a user input accepting the second path, wherein the single command identifies an image manipulation associated with the IVUS image; and

performing, by the medical processing system, the image manipulation identified by the single command.

2. The method of claim 1, wherein the image manipulation comprises a diagnostic measurement.

3. The method of claim 2, wherein the diagnostic measurement includes at least one of a length measurement, an area measurement, or a volume measurement.

4. The method of claim 1, wherein the image manipulation comprises at least one of zooming, adjusting brightness, adjusting opacity, adjusting a color mask, adjusting a resolution, interpolating, or adjusting gain.

5. The method of claim 1, wherein the list of active commands generated for the IVUS modality includes a subset of commands common to multiple modalities.

6. The method of claim 3, further comprising displaying a visual representation of the area measurement or the length measurement on a user display device prior to performing the area measurement or the length measurement.

7. The method of claim 1, wherein records of the gesture database include at least one of a command name, a command format, a correlated system status, or identifying characteristics of gesture-based commands.

8. The method of claim 1, further comprising determining a parameter of the command based on an element of the user input sequence.

9. A method for controlling a medical processing system, the method comprising:

displaying medical data on a user display device, the medical data corresponding to an active modality of the medical processing system, the active modality being selected from a plurality of modalities of the medical processing system, wherein the medical processing system comprises a gesture recognition module and a plurality of modules associated with the plurality of modalities;

querying, by a gesture recognition engine of the gesture recognition module, a gesture database of the gesture recognition module;

determining, by the gesture recognition engine, a list of active commands based on the active modality, wherein the list of active commands is variable such that a different list of active commands is generated for a different active modality;

receiving, by the gesture recognition module, when the mode of operation corresponds to an intravascular ultrasound (IVUS) modality, a user input sequence on an IVUS image from one or more user input devices, the user input sequence provided by a user in response to the displayed medical data and defining a first path on the IVUS image;

outputting, by the gesture recognition module, a second path on the IVUS image, the second path different from the first path and generated by the gesture recognition engine based on a vascular border of the vessel in the IVUS image, the vascular border identified by the medical processing system using a vascular border detection algorithm;

correlating, by the gesture recognition engine, the second path to a single command of the list of active commands generated for the IVUS modality based on a shape of the second path and in response to a user input accepting the second path, wherein the single command identifies an image manipulation associated with the IVUS image; and

performing, by the medical processing system, the image manipulation identified by the single command.

10. The method of claim 9, wherein the list of active commands associated with the IVUS modality includes a subset of commands common to at least two modalities of the plurality of modalities.

11. The method of claim 9, further comprising displaying a visual representation of the single command on the user display device prior to performing the image manipulation.

12. The method of claim 9, wherein records of the gesture database include at least one of a command name, a command format, a correlated system status, or identifying characteristics of gesture-based commands.

13. The method of claim 9, wherein the image manipulation includes at least one of a length measurement, an area measurement, a volume measurement, a zoom, a brightness adjustment, an opacity adjustment, a color mask adjustment, a resolution adjustment, an interpolation, or a gain adjustment.

14. An apparatus comprising:

an intravascular catheter or guide wire comprising a flexible elongate member sized and shaped for positioning within a vessel of a patient and a sensor disposed at the distal end of the flexible elongate member and configured to obtain medical data associated with the vessel, wherein the medical data corresponds to an intravascular ultrasound (IVUS) modality; and

a non-transitory, computer-readable storage medium that stores a plurality of instructions for execution by at least one computer processor in communication with the intravascular catheter or guide wire, wherein the instructions are for;

determining a mode of operation of a medical processing system, the medical processing system comprising a gesture recognition module and a plurality of modules associated with the group of modalities, wherein the gesture recognition module comprises a recognition engine, a gesture database, and an interface operable to communicate with the plurality of modules, the mode of operation corresponding to the IVUS modality;

querying, by the recognition engine, the gesture database;

generating, by the recognition engine, a list of active commands based on the querying the gesture database, wherein the list of active commands is variable such that a different list of active commands is generated for a different mode of operation;

receiving, by the gesture recognition module, user input sequence on an IVUS image from one or more user input devices, the user input sequence defining a first path on the IVUS image;

outputting, by the gesture recognition module, a second path on the IVUS image, the second path different from the first path and generated by the gesture recognition engine based on a vascular border of the vessel in the IVUS image, the vascular border identified by the medical processing system using a vascular border detection algorithm;

correlating, by the gesture recognition engine, the second path to a single command of the list of active commands generated for the IVUS modality based on a shape of the second path and in response to a user accepting the second path, wherein the single command identifies an image manipulation associated with the IVUS image; and

performing the image manipulation identified by the single command.

15. The apparatus of claim 14, wherein the list of active commands generated for the IVUS modality includes a subset of commands common to at least two modalities of the medical processing system.

16. The apparatus of claim 14, wherein the instructions include further instructions for displaying a visual representation of the single command on a user display device, wherein the visual representation of the single command includes one or more moveable reference points configured to allow the user to move the reference points to adjust a boundary specified by the user input sequence.

17. The apparatus of claim 14, wherein records of the gesture database include at least one of a command name, a command format, a correlated system status, or identifying characteristics of gesture-based commands.

18. The apparatus of claim 14, wherein the instructions include further instructions for specifying a parameter of the single command based on an element of the user input sequence.

19. The apparatus of claim 14, wherein the image manipulation identified by the single command includes at least one of a length measurement, an area measurement, a volume measurement, a zoom, a brightness adjustment, an opacity adjustment, a color mask adjustment, a resolution adjustment, an interpolation, or a gain adjustment.

20. The apparatus of claim 14, wherein the instructions further include instructions for correlating the second path to an area measurement command based on a determination that a movement path is longer than a threshold for an unintentional movement and exceeds a threshold tolerance for a linear path.

21. The apparatus of claim 14, wherein the instructions further include instructions for automatically adjusting the user input sequence to render a boundary characterized by a smoothed geometric path.

22. The apparatus of claim 14, wherein outputting the second path comprises modifying the first path based at least in part on a location of the vascular border within the IVUS image.

23. The apparatus of claim 14, wherein the user input sequence comprises a touchstart event, a movement path, and a touchend event.

24. The apparatus of claim 14, wherein the second path includes a straight line and is correlated, by the gesture recognition engine, to a diameter measurement command of the list of active commands.

25. The apparatus of claim 14, wherein the second path includes a curve and is correlated, by the gesture recognition engine, to an area measurement command of the list of active commands.