Device and process for manipulating real and virtual objects in three-dimensional space

• US 20050177054A1
• Filed: 02/10/2004
• Published: 08/11/2005
• Est. Priority Date: 02/10/2004
• Status: Active Grant

• Current Assignee: Sunnybrook Health Sciences Centre
• Original Assignee: Sunnybrook Health Sciences Centre
• Inventors: Yi, Dingrong, Wright, Graham Arnold, Hu, Bob Sueh-Chien
• Granted Patent: US 7,466,303 B2
• Time in Patent Office: Days
• US Class Current: 600/510
• CPC Class Codes: A61B 5/055 involving electronic [EMR] or nuclear [NMR] magnetic resonance, e.g. magnetic resonance imaging
• CPC Class Codes: A61B 5/7285 for synchronising or triggering a physiological measurement or image acquisition with a physiological event or waveform, e.g. an ECG signal A61B6/5205, A61B6/541, A61B8/5207, A61B8/543 take precedence
• CPC Class Codes: A61B 5/7475 User input or interface means, e.g. keyboard, pointing device, joystick
• CPC Class Codes: G05G 2009/04766 providing feel, e.g. indexing means, means to create counterforce
• CPC Class Codes: G05G 9/04737 with six degrees of freedom

Abstract

A device and software system with input and output capability for manipulating real and virtual objects in 3-dimensional space. The device consists of a six degree-of-freedom mechanical armature that has sensors to determine the location and orientation of a stylus and planar surface. In the input mode, manipulation of the physical armature will result in a corresponding two-dimensional, virtual image of the stylus and surface on a computer screen. The armature also has motors to automatically change the armature location and orientation in order to generate a physical representation in the real world of the location and orientation of a virtual object. The armature is built so that it maintains balance at any location and orientation to statically maintain the armature location and orientation without drifting to a null rest position.

Claims

1. An apparatus and software for manipulating real and virtual objects in three-dimensional space, comprising:

a mechanical armature, comprising;

a surface and a stylus movably connected to mechanical linkages and rotational joints so that the stylus and surface may have a location and orientation with six degrees-of-freedom;

a sensor at each rotational joint to determine the location and orientation of the mechanical linkages and rotational joints;

a motor at each rotational joint to rotate the joint and move the mechanical linkage;

a computer for receiving, sending, and processing the location and orientation information from each sensor;

an output mode whereby the motors change the location and orientation of each rotational joint to correspond to a programmed location and orientation so that the armature provides a physical representation of a virtual object;

an input mode whereby an operator moves the stylus, the sensors provide the location and orientation of each rotational joint to the computer, and the computer displays a two-dimensional representation of the armature.

2. The armature of claim 1 wherein weight balancing blocks, the motors'"'"' holding torque, and friction maintain the location and/or orientation of the stylus.

3. The armature and process of claim 1, wherein the armature and software provides image-based co-registration algorithms that can be easily validated.

4. The armature and process of claim 1, wherein the surface and stylus of the armature may be manipulated in three-dimensional space and the software provides a two-dimensional image of a scan plane, which is useful for image navigation.

5. The process of claim 4, wherein said surface manipulation in three-dimensional space can be used for image navigation based on spatial information from a 4×

4 matrix contained in a header file of each image.

6. The armature and process of claim 4, wherein said surface and stylus manipulation, and image navigation can be used for automatic manipulation of a medical device to a given location indicated by three spatial coordinates.

7. The apparatus and process of claim 6, wherein said surface manipulation and image navigation can rotate said medical device to a given orientation indicated by a 3×

3 rotation matrix.

8. The apparatus and process of claim 6, wherein said medical device includes an ultrasound transducer.

9. The apparatus and process of claim 6, wherein said surface manipulation can be used to integrate the positioning and trajectory of instruments, needles, or probes with said image guidance.

10. The apparatus and process of claim 9, wherein said surface manipulation and image navigation can be used to control and integrate the operation of two or more mechanical interventional devices each having six degrees-of-freedom.

11. The apparatus and process of claim 10, wherein said control and integration includes having multiple armatures representing multiple catheters.

12. The apparatus and process of claim 6, wherein said medical device is a magnetic resonance imager, and surface manipulation and image navigation provides magnetic resonance images that enhance the ability of a physician to distinguish between normal and abnormal tissues.

13. The apparatus and process of claim 1, wherein said armature and software provides for control of a scan plane in magnetic resonance imaging.

14. The apparatus and process of claim 13, wherein said armature and software also enables switching between guiding a magnetic resonance scan plane and an ultrasound plane in a three-dimensional acquisition mode.

15. The apparatus and process of claim 13, wherein said armature and software further provides for applications that require accurate registration of magnetic resonance imaging data with data obtained using other imaging modalities using a common coordinate system.

16. The apparatus and process of claim 13, wherein said armature and software is not reliant on operator input or influence during a magnetic resonance imaging procedure.

17. The apparatus and process of claim 13, wherein said armature and software also allows the operator to command the magnetic resonance scanner by inputting the spatial location and orientation of the scan plane relative to the object of a patient in the real world.

18. The apparatus and process of claim 13, wherein the location of anatomic structures in a human body can be accurately detected in said scan plane in said magnetic resonance images.

19. The apparatus and process of claim 13, wherein said armature and software provides for six degree-of-freedom surface manipulation and representation which is useful for both conventional clinical magnetic resonance imaging and functional magnetic resonance imaging studies.

20. The apparatus and process of claim 19, wherein said six degree-of-freedom surface manipulation and representation enables magnetic resonance imaging with the same spatial resolution and orientation in different examinations.

21. The apparatus of claim 1, wherein said mechanical armature has six degree-of-freedom surface manipulation and representation.

22. The apparatus of claim 21, wherein said mechanical armature can detect and track positional changes in a reference structure that is computationally efficient.

23. The apparatus of claim 21, wherein said mechanical armature can generate a physical representation of a two-dimensional scan plane of a magnetic resonance image relative to an object in real patient coordinates.

24. The apparatus of claim 21, wherein said mechanical armature has a planar surface that can move automatically to reflect the effects of an operator'"'"'s action on scan plane prescription.

25. The apparatus of claim 24, wherein said armature device can move in a coordinated manner through all six degrees-of-freedom required for the optimal scan plane.

26. The apparatus and process of claim 24, wherein said armature device is capable of statically maintaining its location and orientation, thereby improving spatial awareness so the operator can better appreciate the direction of the next movement.

27. The apparatus and process of claim 26, wherein said improved spatial awareness enables improved visualization by the operator of the object under investigation.

28. The apparatus and process of claim 12, wherein said armature device provides for six degree-of-freedom surface manipulation and representation whose function is independent of the magnetic resonance scanner.

29. The apparatus and process of claim 1, wherein said armature device can be used for the automatic manipulation of a medical device to a given position indicated by three spatial coordinates.

30. The apparatus and process of claim 29, wherein said mechanical armature device can also rotate said medical device to a given orientation indicated by a 3×

3 rotation matrix.

31. The apparatus and process of claim 29, wherein said medical device includes an ultrasound transducer.

32. The apparatus and process of claim 1, wherein said armature and software system provide images for interventional magnetic resonance imaging applications.

33. The apparatus and process of claim 32, wherein said interventional magnetic resonance imaging applications are used to guide and monitor minimally invasive diagnostic and therapeutic procedures.

34. The apparatus and process of claim 1, wherein said armature and software system provides integration of input and output functionality of the armature to achieve visualization and navigation of the catheter tip towards the target in an intuitive and efficient way.

35. The apparatus and process of claim 34, wherein said catheter tip can also be manually manipulated and steered towards the target.

36. The apparatus and process of claim 29, wherein said armature device under real-time computer control provides support for interventional treatment procedures for use with surgical tools and tissue manipulators.

37. The apparatus and process of claim 29, wherein said armature device under said real-time computer control is used to guide interventional devices which deliver RF, thermal, microwave or laser energy, or ionizing radiation.

38. The apparatus and process of claim 29, wherein said armature device under said integrated real-time computer control is also used to support internal illumination and imaging devices, such as catheters, endoscopes, laparoscopes, and similar instruments.

39. The apparatus and process of claim 33, wherein said interventional treatment procedures include in vivo delivery of drugs, angioplasty devices, biopsy and sampling devices, image-guided interstitial probe placement, high-temperature thermal therapy, cryotherapy, or drug therapy for tumors;

localization of non-invasive focused ultrasound probes below the tissue surface for thermal therapy; and

subcutaneous or transdural placement of biopsy needles or surgical instruments for minimally-invasive surgery.

40. The apparatus and process of claim 13, wherein said scan plane prescription and image navigation process improves detection of the passage of a contrast agent through the microcirculation of the heart in human patients.

41. The apparatus and process of claim 13, wherein said scan plane prescription and image navigation process also improves MR perfusion imaging of the brain, liver, and other solid internal body organs in human patients.

42. The apparatus and process of claim 1, wherein said software provides graphical visual information about the object being imaged, the projected display of the 2-dimensional scan plane, and the expected magnetic resonance image corresponding to that scan plane of the tissue being imaged.

43. The apparatus and process of claim 42, wherein said software also provides a user interface for the control of the magnetic resonance scanner and the six degree-of-freedom hardware, as well as the driver and algorithms that relate to the six degree-of-freedom device.

44. The apparatus and process of claim 1, wherein algorithms can be used to rotate the shaft of said motor to enable reaching the destination based on the angular position of the destination and current position of said motor, thereby eliminating the need for an expensive multi-degree motor controller.

45. The device claim 1, wherein said mechanical armature comprises:

a base removably mounted on a surface;

a first linkage connected to the base through a first rotational joint so that the first linkage is parallel to the surface and can rotate on an axis parallel to the surface;

a second linkage connected to the first linkage through a second rotational joint so that the second linkage can rotate on an axis perpendicular to the first linkage;

a third linkage connected to the second linkage;

a fourth linkage connected to the third linkage through a through a third rotational joint so that the fourth linkage can rotate on an axis perpendicular to the third linkage;

a fifth half-circle linkage connected to the fourth linkage through a fourth rotational joint;

a sixth linkage connected to the fifth linkage through fifth rotational joint and an end so that sixth linkage can rotate;

a surface connected to the sixth linkage through a sixth rotational joint so that the surface can rotate;

a stylus connected perpendicular to the surface; and

weight balancing blocks connected to a balance arm and the fourth linkage so that the stylus and surface remain static when released.

46. The device of claim 1, wherein said mechanical armature contains six or more mechanical linkages and six or more rotational joints.

47. The device of claim 45 wherein the distance between the second rotational joint and the third rotational joint is equal to the distance between the second rotational joint and the sixth rotational joint.

48. The device of claim 45 wherein the stylus has one or more ears.

49. The device of claim 45 wherein one or more of the rotational joints has a sensor.

50. The device of claim 49 wherein each sensor is connected to a computer.

51. The device of claim 50 wherein the operator manipulates the stylus or surface to a location and orientation, and software is used to evaluate the data from each sensor so that the location and orientation of the stylus or surface may be computed.

52. The device and process of claim 51 wherein the information provided by the sensors is used by the computer to display on a computer screen a two-dimensional representation of the location and orientation of the stylus or the surface.

53. The device of claim 50 wherein each rotational joint is linked to a motor so that the motor can rotate the joint, and each motor is coupled to a sensor.

54. The device of claim 53 wherein,

each motor and sensor couple is connected to the computer,

the operator programs the computer to direct each motor to rotate each joint, and

each sensor provides feedback to the computer regarding the rotation of the corresponding joint.

55. The device and process of claim 54 wherein the computer controls the location and orientation of the stylus or the surface through a series of positions and orientations.

56. The device and process of claim 5 wherein the computer moves the stylus or the surface through a series of positions and orientations.

57. The apparatus and software of claim 4, wherein the scan plane location and orientation are constrained to a pre-specified range.