Image processing system, and image processing method

• US 9,633,439 B2
• Filed: 06/28/2013
• Issued: 04/25/2017
• Est. Priority Date: 07/30/2012
• Status: Active Grant

• Current Assignee: National Institute of Advanced Industrial Science and Technology (Government of Japan), Kyushu University, National University Corporation Hamamatsu University School of Medicine
• Original Assignee: National Institute of Advanced Industrial Science and Technology (Government of Japan), Kagoshima University, National University Corporation Hamamatsu University School of Medicine
• Inventors: Sagawa, Ryusuke, Kawasaki, Hiroshi, Furukawa, Ryo
• Primary Examiner(s): Chen, Yu
• Application Number:
  US14/418,663
• Publication Number:
  US 20150221093A1
• Time in Patent Office: 1,397 Days
• Field of Search: None
• CPC Class Codes: G01B 11/2513 with several lines being projected in more than one direction, e.g. grids, patterns
• CPC Class Codes: G01B 11/2545 with one projection direction and several detection directions, e.g. stereo
• CPC Class Codes: G06T 1/0007 Image acquisition
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/73 using feature-based methods

Abstract

A high-density shape reconstruction is conducted in measuring animal bodies as well. An image processing system has a projection device, an imaging device, and an image processing apparatus connected to the projection device and the imaging device, wherein the projection device projects a projected pattern to an observation target, the imaging device captures the projected pattern, and the image processing apparatus performs shape reconstruction based on an input image including the projected pattern. The image processing apparatus includes a unit for fetching the input image captured by the imaging device and performing line detection for the projected pattern projected by the projection device, wherein the projected pattern is a grid pattern formed of wave lines; and a unit for performing shape reconstruction by associating intersection points of vertical and horizontal lines extracted by the line detection with the projected pattern.

Claims

1. An image processing system comprising:

a projection device for projecting a projected pattern to an observation target;

an imaging device for capturing the projected pattern; and

an image processing apparatus connected to the projection device and the imaging device, for performing shape reconstruction based on an input image including the projected pattern, the image processing apparatus including a personal computer configured to;

fetch the input image captured by the imaging device and performing line detection for the projected pattern projected by the projection device, wherein the projected pattern is a grid pattern formed of wave lines, the wave lines are wavy curves having predetermined periodicity, the grid pattern formed of the wave lines is formed of a plurality of wave lines that are arranged at predetermined intervals, the grid pattern is a set of wave lines that intersect each other in two directions, and the interval of the wave lines in one of the directions is not equal to an integral multiple of a wavelength for the wave line in the other direction; and

perform shape reconstruction by associating intersection points of vertical and horizontal lines extracted by the line detection with the projected pattern.

2. The image processing system according to claim 1, wherein the personal computer is further configured to:

reproject a patch to an image output by the projection device, wherein the patch obtained by approximating, to a tangent plane, a region around each intersection point of an input image that is captured by the imaging device;

calculate energy for stereo matching between each intersection point of the reprojected patch and a correspondence candidate for a grid point of the projected pattern projected by the projection device by employing a sum of a data term assigned to each grid point and a regularization term obtained between the grid point and an adjacent grid point, wherein the grid point is an intersection point of the wave lines in two directions of the grid pattern; and

perform shape reconstruction by associating a grid point with the projected pattern, wherein the grid point is a correspondence candidate having a minimum value of energy for stereo matching among the correspondence candidate.

3. The image processing system according to claim 1, wherein the personal computer is further configured to:

create a triangular mesh consisting of three pixel samples and calculate a depth of each sub-pixel; and

calculate, for all of the pixel samples, an error that occurs when the triangular mesh is re-projected to an output image of the projection device, minimize the error obtained, and perform linear interpolation for depths of pixels other than the pixel samples.

4. The image processing system according to claim 2, wherein the projection device includes first and second imaging devices;

and

the personal computer is further configured to select the correspondence candidate by adding a regularization term for the grid point that is obtained between the first and second imaging devices to energy for stereo matching of the correspondence candidates.

5. The image processing system according to claim 4, wherein the personal computer is further configured to employ an average to merge a depth for each pixel that is obtained, for the grid point, between the first and second imaging devices.

6. The image processing system according to claim 1, wherein the projection device includes first and second projection devices;

and

the personal computer is further configured to optimize a depth of each pixel, for grid points for which matching is obtained between a first projected pattern projected by the first projection device and a second projected pattern projected by the second projection device, wherein the grid points are intersection points of the wave lines in two directions of the grid pattern.

7. The image processing system according to claim 1, wherein the personal computer is further configured to perform shape reconstruction by calculating, for a plurality of positions around grid points being intersection points of the wave lines in two directions of the grid pattern, a difference between the projected pattern of the grid points and a result obtained through the line detection, and by employing the result as a matching cost for a correspondence candidate to associate and associating a grid point that is a minimum correspondence candidate with the projected pattern.

8. The image processing system according to claim 1, wherein, when the projected pattern is projected to the observation target, a parameter for the projected pattern is selected by comparing degrees of similarity for two arbitrary intersection points on the same epipolar line so that a degree of similarity becomes minimum.

9. An image processing method of performing shape reconstruction based on an input image including a projected pattern in an image processing apparatus connected to a projection device and an imaging device, wherein the projection device projects a projected pattern to an observation target, and the imaging device captures the projected pattern, the method comprising the steps of:

fetching, by the image processing apparatus, the input image captured by the imaging device, and performing line detection for the projected pattern projected by the projection device, wherein the projected pattern is a grid pattern formed of wave lines, the wave lines are wavy curves having predetermined periodicity, the grid pattern formed of the wave lines is formed of a plurality of wave lines that are arranged at predetermined intervals, the grid pattern is a set of wave lines that intersect each other in two directions, and the interval of the wave lines in one of the directions is not equal to an integral multiple of a wavelength for the wave line in the other direction; and

performing, by the image processing apparatus, shape reconstruction by associating intersection points of vertical and horizontal lines extracted by the line detection with the projected pattern.

10. A non-transitory computer readable storage medium having a computer program stored therein, said computer program including computer executable commands enabling an imaging device to perform shape reconstruction based on an input image including a projected pattern in an image processing apparatus connected to a projection device and the imaging device, wherein the projection device projects a projected pattern to an observation target, and the imaging device captures the projected pattern, the computer executable commands further enabling the imaging device to performing the steps of:

fetching, by the image processing apparatus, the input image captured by the imaging device, and performing line detection for the projected pattern projected by the projection device, wherein the projected pattern is a grid pattern formed of wave lines, the wave lines are wavy curves having predetermined periodicity, the grid pattern formed of the wave lines is formed of a plurality of wave lines that are arranged at predetermined intervals, the grid pattern is a set of wave lines that intersect each other in two directions, and the interval of the wave lines in one of the directions is not equal to an integral multiple of a wavelength for the wave line in the other direction; and

performing, by the image processing apparatus, shape reconstruction by associating intersection points of vertical and horizontal lines extracted by the line detection with the projected pattern.