IMAGE COMPLETION SYSTEM FOR IN-IMAGE CUTOFF REGION, IMAGE PROCESSING DEVICE, AND PROGRAM THEREFOR

• US 20150145953A1
• Filed: 03/15/2013
• Published: 05/28/2015
• Est. Priority Date: 03/17/2012
• Status: Abandoned Application

• Current Assignee: Kyushu University, Waseda University (Waseda University Educational Association)
• Original Assignee: Kyushu University, Waseda University (Waseda University Educational Association)
• Inventors: Kobayashi, Yo, Hashizume, Makoto, Fujie, Masakatsu, Kawamura, Kazuya, Seno, Hiroto, Nishio, Yuya, Ieiri, Satoshi, Toyoda, Kazutaka
• Application Number:
  US14/383,776
• Publication Number:
  US 20150145953A1
• Time in Patent Office: Days
• US Class Current: 348/45
• CPC Class Codes: A61B 1/00009 of image signals during a use of endoscope
• CPC Class Codes: A61B 1/0005 combining images e.g. side-by-side, superimposed or tiled
• CPC Class Codes: A61B 1/018 for receiving instruments
• CPC Class Codes: A61B 1/313 for introducing through surgical openings, e.g. laparoscopes
• CPC Class Codes: A61B 2034/2055 Optical tracking systems
• CPC Class Codes: A61B 2090/365 augmented reality, i.e. correlating a live optical image with another image
• CPC Class Codes: A61B 90/37 Surgical systems with images on a monitor during operation
• CPC Class Codes: G02B 23/2415 Stereoscopic endoscopes
• CPC Class Codes: G02B 23/2484 Arrangements in relation to a camera or imaging device processing or control of video signals generated by an endoscope H04N23/60, H04N23/70
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/10068 Endoscopic image
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2207/30004 Biomedical image processing
• CPC Class Codes: G06T 2207/30208 Marker matrix
• CPC Class Codes: G06T 7/33 using feature-based methods

Abstract

An image completion system includes a first endoscope for obtaining an operating field image, a second endoscope for imaging the object space in a line-of-sight direction different from that of the first endoscope to obtain a completing image, a first device for measuring separating distances between a reference point and a large number of set points that are set in the object space, a second device for measuring three-dimensional positions of the endoscopes, and an image processing device for obtaining, image information on cutoff regions in the object space that are hidden on the depth sides of surgical instruments by imaging the surgical instruments in the operating field image together with the object space, and for replacing image information on the surgical instruments S in the operating field image with the obtained image information or superimposing the obtained image information onto the image information on the surgical instruments.

Claims

1-7. -7. (canceled)

8. An image completion system for an in-image cutoff region, comprising:

a main imaging device for obtaining a main image in which an object space to be monitored is imaged;

a completing-purpose imaging device for obtaining a completing image used for completing the main image by imaging the object space in a line-of-sight direction different from that of the main imaging device; and

an image processing device for completing a portion of the main image with the completing image on the basis of three-dimensional positions of set points which are set in the object space and three-dimensional positions of the main imaging device and the completing-purpose imaging device, wherein

the main imaging device and the completing-purpose imaging device are provided so as to image the object space almost simultaneously, and

the image processing device obtains, from the completing image, image information on a cutoff region in the object space that is hidden behind on a depth side of a member having a known shape by imaging the member in the main image together with the object space, on the basis of information on the three-dimensional positions that are continuously detected, and replaces image information on the member in the main image with the obtained image information or superimposes the obtained image information onto the image information on the member in the main image so as to generate a composite image in which the cutoff region is completed with the completing image.

9. The image completion system for an in-image cutoff region according to claim 8, wherein

the image processing device includes:

set point position identifying means for identifying, with respect to the set points, sets of in-screen coordinates in a screen coordinate system in the main image and sets of in-screen coordinates in a screen coordinate system in the completing image on the basis of a detection result of the three-dimensional positions;

completing image transforming means for generating, on the basis of the sets of in-screen coordinates, a transformed image in which pieces of image information on points in the completing image are moved in a screen of the completing image such that the completing image is converted into that in a line-of-sight direction of the main imaging device;

cutoff region identifying means for identifying a position of the cutoff region in the main image; and

composite image generating means for generating the composite image by replacing image information on the cutoff region with image information on a corresponding region that corresponds to the cutoff region in the transformed image or superimposing the image information on the corresponding region onto the image information on the cutoff region.

10. The image completion system for an in-image cutoff region according to claim 9, wherein the completing image transforming means generates the transformed image by moving the pieces of image information on the completing image such that the set points in the completing image match sets of in-screen coordinates of the same set points existing in the main image.

12. The image completion system for an in-image cutoff region according to claim 8, wherein at least three of the set points are set in the object space.

13. The image completion system for an in-image cutoff region according to claim 8, wherein the main imaging device and the completing-purpose imaging device are provided so as to independently move to image.

14. The image completion system for an in-image cutoff region according to claim 8, further comprising:

a distance measuring device for measuring separating distances between the set points and a predetermined reference point; and

a three-dimensional position measuring device for measuring three-dimensional positions of the main imaging device and the completing-purpose imaging device, wherein

in the image processing device, a portion of the main image is completed with the completing image on the basis of measurement results from the distance measuring device and the three-dimensional position measuring device.

11. An image processing device for performing a process to compose a main image of an object space to be monitored that is obtained by a main imaging device and a completing image of the object space that is imaged by a completing-purpose imaging device at the same time in a line-of-sight direction different from the main image so as to complete, when a member having a known shape is imaged in the main image together with the object space, image information on a cutoff region in the main image that is cut off by at least a portion of the member with image information on the completing image, the image processing device comprising:

set point position identifying means for identifying, with respect to set points which are set in the object space, sets of in-screen coordinates in a screen coordinate system of the main image and sets of in-screen coordinates in a screen coordinate system of the completing image, from three-dimensional positions of the set points and three-dimensional positions of the main imaging device and the completing-purpose imaging device;

completing image transforming means for generating, on the basis of the sets of in-screen coordinates, a transformed image in which pieces of image information on points in the completing image are moved in a screen of the completing image such that the completing image is converted into that in a line-of-sight direction of the main image;

cutoff region identifying means for identifying a position of the cutoff region in the main image; and

composite image generating means for generating a composite image in which a cutoff region in the main image is completed with the completing image by replacing image information on the cutoff region with image information on a corresponding region that corresponds to the cutoff region in the transformed image or superimposing the image information on the corresponding region onto the image information on the cutoff region.