Three-dimensional digital entity mesoscope system equipped with three-dimensional visual instruction functions

• US 20070184422A1
• Filed: 03/17/2005
• Published: 08/09/2007
• Est. Priority Date: 03/26/2004
• Status: Active Grant

• Current Assignee: Atsushi Takahashi
• Original Assignee: Atsushi Takahashi
• Inventors: Takahashi, Atsushi
• Granted Patent: US 7,367,809 B2
• Time in Patent Office: Days
• US Class Current: 434/262
• CPC Class Codes: A61B 2017/00221 with wireless transmission of data, e.g. by infrared radiation or radiowaves
• CPC Class Codes: A61B 2090/3616 Magnifying glass
• CPC Class Codes: A61B 2090/502 Headgear, e.g. helmet, spectacles
• CPC Class Codes: A61B 90/361 Image-producing devices, e.g. surgical cameras
• CPC Class Codes: G06Q 50/20 Education
• CPC Class Codes: G06T 2207/30004 Biomedical image processing
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G09B 23/28 for medicine
• CPC Class Codes: G16H 20/40 relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
• CPC Class Codes: G16H 50/50 for simulation or modelling of medical disorders
• CPC Class Codes: G16H 70/60 relating to pathologies
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

The present invention provides a three-dimensional digital entity magnifying glass technique-assisting and training/educational distribution system incorporating three-dimensional visual training functions by means of image composition that enables a three-dimensional visual instruction containing a depth when giving visual instructions to an HMD worn by a medical practitioner to thereby display only images of a pointing device or various kinds of instruments among image information of the instructor'"'"'s CCD camera and synchronously and compositely displaying a three-dimensional visual training image output from an image-processing apparatus that can display an after-image of the image for an arbitrary time setting into image information of the CCD camera, thereby providing equal-scaling display of image information to which an instruction and a comment by use of a visual display/instruction image are added three-dimensionally as well as an instrument actually used by the instructors (groups of lecturers) on the three-dimensional HMD of the medical practitioner.

Claims

1. A three-dimensional digital entity magnifying glass system incorporating three-dimensional visual training functions, the system used for performing assistance and training and education, relating to surgical operations and other highly precision oriented techniques performed, under direct observation, by a medical practitioner while an instructor is watching a medical practitioner'"'"'s surgical field image shared by the medical practitioner and the instructor through communication secured by use of communication means such as the internet or an intranet by utilizing three-dimensional digital entity magnifying glasses, which are worn by both the medical practitioner and the instructor and constituted of a pair of LCDs positioned directly in front of the two eyes and by three-dimensional digital entity magnifying glasses of identical standards for indicating image information from a right-side CCD camera on the right-side LCD and image information from a left-side CCD camera on the left-side LCD of automatic focusing CCD cameras equipped with a pair of synchronized zoom lenses attached toward a visual field of the naked eye from a position that correspond to a central section between the right and left pupils of the worker wearing the magnifying glass in front of the three-dimensional digital entity magnifying glass, the system comprising:

a floater image-processing apparatus for removing a background image by extracting only images of a three-dimensional pointing device which is operated by the instructor to provide visual training, or various kinds of instruments (dummies) identical to those used by the medical practitioner from a visual field image of the instructor captured by the instructor-side CCD camera on a front face of the visual field in front of the instructor-side three-dimensional digital entity magnifying glass; and

an image chroma-key composite image-processing apparatus for composing and displaying a three-dimensional visual training float image, on which no background output is performed by the image-processing apparatus on the medical practitioner'"'"'s visual field image captured by the CCD camera of the medical practitioner'"'"'s three-dimensional digital entity magnifying glass shared by the medical practitioner and the instructor,

wherein, in a minimum configuration, by utilizing the pair of synchronized three-dimensional digital entity magnifying glasses of identical standards worn by the medical practitioner and the instructor, the instructor provides visual training which is three-dimensional and produces a sense of reality in the medical practitioner'"'"'s three-dimensional visual field by utilizing images of the three-dimensional pointing device operated by the instructor or various instruments (dummies) identical to those used by the medical practitioner which images are extracted by floating image processing from visual field images of the instructor captured by the instructor'"'"'s CCD camera and chroma-key composed and displayed in the medical practitioner'"'"'s visual field image (virtual three-dimensional image) captured by the medical practitioner'"'"'s CCD camera projected onto an LCD monitor of the three-dimensional digital entity magnifying glass worn by the instructor.

2. The three-dimensional digital entity magnifying glass system incorporating three-dimensional visual training functions, comprising the chroma-key composite image-processing apparatus according to claim 1 for purposes of varying a degree of transparency of a three-dimensional visual training image of the instructor by a percentage of between 0% and 100% or, additionally, for changing a color tone thereof and displaying it compositely, so that the medical practitioner can easily judge training image and in order to prevent the medical practitioner'"'"'s visual field and an action of an surgical operation from being blocked visually as a result of overlapping of the worker'"'"'s work situation visual field image and a three-dimensional visual training image of the instructor when the floater three-dimensional visual training image with no background, output from the image-processing apparatus, of the three-dimensional pointing device operated by the instructor or various instruments (dummies) identical to those used by the medical practitioner is chroma-key composed on medical practitioner'"'"'s visual field image information in a CCD camera of the digital entity magnifying glass worn by the medical practitioner and displayed.

3. A three-dimensional digital entity magnifying glass system incorporating three-dimensional visual training functions, comprising an image position correction and follow-up apparatus for compositely displaying three-dimensionally accurate equal-scaling three-dimensional perspective images into the medical practitioner'"'"'s visual field image displayed on an LCD of a three-dimensional digital entity magnifying glass by automatically performing position correction always in respect of variations in a visual field angle of the medical practitioner with respect to a patient or to variations in postures of patients, by means of:

in an image obtained by varying a degree of transparency, by a percentage of between 0% and 100%, of image information from the medical practitioner'"'"'s three-dimensional digital entity magnifying glass CCD displayed on LCDs of both the medical practitioner and the instructor according to claims 1 and 2;

three-dimensionally superimposing an item of three-dimensional image information obtained through image processing and conversion, by changing a three-dimensional anatomical image of an item such as an arbitrary bone, a blood vessel, or nervous tissue, created beforehand from information in a tomogram such as a CT picture or an MRI picture of a patient on whom an operation is to be performed or a surgical stent image into which the anatomical three-dimensional perspective images are three-dimensionally composed together with a dissection or bone cutting position similarly by a percentage of between 0% and 100%, especially, by varying a degree of transparency or a color tone of a site or a tissue indicative of an anatomical landmark, or by applying a visual difference to the medical practitioner'"'"'s three-dimensional digital entity magnifying glass CCD image information through wireframe display etc. so as to facilitate discrimination, and onto the medical practitioner'"'"'s three-dimensional digital entity magnifying glass CCD image at an equal reduction rate by utilizing a reference triangular plane obtained through three-dimensional positional measurement by use of the right and left CCDs of the three-dimensional digital entity magnifying glass and by using, as reference points, three points on non-deformable hard tissue such as a tooth or a bone arbitrarily determined from among the medical practitioner'"'"'s three-dimensional digital entity magnifying glass CCD image information; and

by

enabling the anatomical three-dimensional image to automatically correct a change in position of, and follow, the reference point of the medical practitioner'"'"'s three-dimensional digital entity magnifying glass CCD image information as the medical practitioner'"'"'s CCD camera position changes or the patient'"'"'s posture changes.

4. A three-dimensional digital entity magnifying glass system incorporating three-dimensional visual training functions, wherein a three-dimensional anatomical image of an arbitrary item such as a bone, a blood vessel, or a nervous tissue, created beforehand from information in a tomogram such as a CT picture or an MRI picture of a patient on whom an operation is to be performed, owned at a remote location or a three-dimensional surgical stent image in which the medical practitioner or a third party (any other medical doctor) designed beforehand a position of a dissection or a bone cutting based on the anatomical three-dimensional perspective image is composed into the medical practitioner'"'"'s three-dimensional digital entity magnifying glass CCD'"'"'s image information according to claims 1 through 3 at the remote location and accurately superimposed one on the other to provide the respective composite display images showing an apparently entity-see-through situation, positions of which images are automatically corrected by the image position-correction and follow-up apparatus in real time as the medical practitioner'"'"'s CCD camera position changes and the patient'"'"'s posture changes, to distribute the images compositely displayed to a three-dimensionally correct position always to the medical practitioner and the instructor by utilizing communication means such as the internet or an intranet.

5. A three-dimensional digital entity magnifying glass system incorporating three-dimensional visual training functions, wherein by accumulating the medical practitioner'"'"'s three-dimensional digital entity magnifying glass'"'"'s medical practitioner three-dimensional image information according to claim 1 and surgical stent image information for compositely displaying three-dimensionally accurate equal-scaling three-dimensional perspective images into the medical practitioner'"'"'s visual field image according to claim 3 in a server so that each of persons wearing a plurality of synchronized three-dimensional digital magnifying glasses and sharing the image information accumulated in the server can arbitrarily switch the surgical stent image information on the medical practitioner'"'"'s three-dimensional digital entity magnifying glass'"'"'s three-dimensional image information between a display mode and a non-display mode and, further, can transmit to the server a three-dimensional visual training image obtained by utilizing the three-dimensional digital entity magnifying glass'"'"'s CCD cameras worn by them according to claims 1 and 2, and compositely display it to the medical practitioner'"'"'s three-dimensional image accumulated in the server, thereby making possible discussions through bidirectional communication on a three-dimensional moving image or a still image, by utilizing a communication line such as the internet.

6. A three-dimensional digital entity magnifying glass system incorporating three-dimensional visual training functions, wherein by providing pluralities of synchronized three-dimensional digital entity magnifying glasses worn by the medical practitioner and the instructor and the floater image-processing apparatuses according to claims 1 through 4 so that the medical practitioner and the instructor may share the images, the plurality of instructors simultaneously guides one medical practitioner three-dimensionally by using the visual training image or switches and displays visual field images of the plurality of medical practitioners as occasion demands so that one instructor provides visual instruction and guidance to the plurality of medical practitioners three-dimensionally through various communication means such as the internet or an intranet.

7. A three-dimensional digital entity magnifying glass system incorporating three-dimensional visual training functions, wherein when distributing and publishing a training situation by use of the three-dimensional visual training image according to claims 1 and 2 to a lot of students other than the medical practitioner and the instructor or to a third party institution by using live-broadcasting or video-recorded broadcasting via various kinds of communication means such as the internet, by encrypting output signals of two right and left channels output from one three-dimensional digital entity magnifying glass'"'"'s CCD camera in hardware of the three-dimensional digital entity magnifying glass and by encoding it in hardware of the other three-dimensional digital entity magnifying glass, normal image display is disabled unless a terminal having a prescribed three-dimensional digital entity magnifying glass hardware configuration is used, to provide a higher degree of security, thereby making possible protection of the patient'"'"'s privacy.

8. A three-dimensional digital entity magnifying glass system incorporating three-dimensional visual training functions, wherein by presenting conditions such as skills, experiences, and languages spoken during the course of training as well as offering monetary rewards for above-described various services to an operation assistant on the network by using the three-dimensional digital entity magnifying glass system according to claims 1 through 5 having extremely large amount of information and capable of virtually experiencing of the medical practitioner'"'"'s operating situation, to sign a contract with a registered or technician to get the staff (instructor) dispatched thereto, so that if an unexpected situation such as an error in medical treatment during a surgical operation occurs, a location of the error can be clarified based on a record of information of the three-dimensional digital entity magnifying glass system and a responsible range is also clearly defined in accordance with contents of a contract determined beforehand based on a degree of involvement between contents of each operation and each technical field of the staff and an instruction fee presented by the medical practitioner based on the contents of a contract, thereby spreading a burden of risks for compensations for errors in medical treatment and defects.

9. A three-dimensional digital entity magnifying glass system incorporating three-dimensional visual training functions, wherein it is made possible to objectively asses skills of the medical practitioner at a remote location by using the three-dimensional digital entity magnifying glass system that enables virtually experiencing an operating situation of a medical practitioner and has an extremely high degree of authenticity because this system according to claims 1 through 5 has extremely large amount of information and is of a real time type and also capable of communication only by simultaneously using three-dimensional digital entity magnifying glasses of identical standards.

10. A three-dimensional digital entity magnifying glass system incorporating three-dimensional visual training functions, wherein it is made possible to switch an instructing image compositely displayed in the medical practitioner'"'"'s visual field according to claims 1 through 5, an anatomical three-dimensional image compositely displayed in the medical practitioner'"'"'s visual field image, or a surgical stent image between the display mode and the non-display mode through switching that does not block operations of using the hand or foot involved in an action of operating by the medical practitioner in the case of his leadership or through wills of the instructor in the case of his leadership.

11. A method performed in a system incorporating three-dimensional digital entity magnifying glasses which are worn by both a medical practitioner and an instructor and constituted of a pair of LCDs installed directly in front of the two eyes and used for performing assistance, and training and education, relating to surgical operations and to other highly precision oriented techniques which are performed, under direct observation, by a medical practitioner while an instructor is watching a medical practitioners'"'"'s surgical field image shared by the medical practitioner and the instructor through communication secured by use of communication means such as the internet or an intranet by utilizing three-dimensional digital entity magnifying glasses of identical standards for indicating image information from a right-side CCD camera on the right-side LCD and image information from a left-side CCD camera on the left-side LCD of an automatic focusing CCD cameras equipped with a pair of synchronized zoom lenses which are attached toward a visual field of the naked eye from a position that nearly corresponds to a central section between the right and left pupils of the worker wearing the magnifying glass in front of the three-dimensional digital entity magnifying glass, the method comprising:

a step of floater image processing, by removing a background image by extracting only images of a three-dimensional pointing device which is operated by the instructor to provide visual instructions or various kinds of instruments (dummies) identical to those used by the medical practitioner from a visual field image of the instructor captured by the instructor-side CCD camera on a front face of the visual field in front of the instructor-side three-dimensional digital entity magnifying glass; and

a step of image chroma-key composite image processing, by composing a three-dimensional visual training float image with background output by the image-processing apparatus on the medical practitioner'"'"'s visual field image captured by the CCD camera of the medical practitioner'"'"'s three-dimensional digital entity magnifying glass shared by the medical practitioner and the instructor and displaying it,

wherein, in a minimum configuration, by utilizing the pair of synchronized three-dimensional digital entity magnifying glasses of identical standards worn by the medical practitioner and the instructor, the instructor provides visual instructions which is three-dimensional and has a sense of reality in the medical practitioner'"'"'s three-dimensional visual field by utilizing images of the three-dimensional pointing device operated by the instructor or various instruments (dummies) identical to those used by the medical practitioner which images are extracted by floating image processing from visual field images of the instructor captured by the instructor'"'"'s CCD camera and chroma-key composed and displayed in the medical practitioner'"'"'s visual field image (virtual three-dimensional images) captured by the medical practitioner'"'"'s CCD camera projected onto an LCD monitor of the three-dimensional digital entity magnifying glass worn by the instructor.