Measurement of depth image considering time delay

• US 6,445,815 B1
• Filed: 03/12/1999
• Issued: 09/03/2002
• Est. Priority Date: 05/08/1998
• Status: Expired due to Term

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Sato, Kiyohide
• Primary Examiner(s): Johns, Andrew W.
• Assistant Examiner(s): AZARIAN, SEYED H
• Application Number:
  US09/266,859
• Time in Patent Office: 1,271 Days
• Field of Search: 382/154, 382/153, 382/173, 382/275, 382/276, 345/125, 345/419, 345/420, 345/421, 345/422, 345/423, 345/424, 345/848, 345/836, 345/156, 345/427, 356/602, 356/603, 356/389, 356/349
• US Class Current: 382/154
• CPC Class Codes: G05B 2219/32014 Augmented reality assists operator in maintenance, repair, programming, assembly, use of head mounted display with 2-D 3-D display and voice feedback, voice and gesture command
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04N 13/221 using the relative movement between cameras and objects
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 19/54 using feature points or meshes
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

An augmented reality presentation system that generates and presents a virtual image free from any latency from a real space. This system has a position/posture sensor for time-sequentially inputting viewpoint position/posture information, stereo cameras for inputting a continuous time sequence of a plurality of images, and an image processing apparatus. The image processing apparatus detects a continuous time sequence of depth images ID from the continuous time sequence of input stereo images, estimates the viewpoint position/posture of the observer at a future time at which a three-dimensional image will be presented to the observer, on the basis of changes in previous viewpoint position/posture input from the position/posture sensor, continuously warps the continuously obtained depth images to those at the estimated future viewpoint position/posture, and presents three-dimensional grayscale (or color) images generated according to the warped depth images to the observer.

Claims

1. A depth image measurement apparatus for acquiring depth information of a scene, comprising:

image input means for inputting an image of the scene at a first viewpoint;

depth image generation means for generating a first depth image from the scene image inputted at the first viewpoint by said image input means;

position/posture estimation means for estimating, based on information relating to displacement of the first viewpoint, a position and posture information at a second viewpoint viewed from a position and posture of the first viewpoint; and

warping means for warping the first depth image generated by said depth image generation means to a second depth image at the second viewpoint on the basis of the position and posture information at the second viewpoint estimated by said position/posture estimation means.

13. The apparatus according to claim 1, wherein the first viewpoint occurs at a plurality of viewpoints,

said depth image generation means generates a first depth image in correspondence with each respective first viewpoint,

said position/posture estimation means estimates viewpoint position/posture information of the second viewpoint viewed from each first viewpoint, and

said warping means warps each of the first depth image on the basis of each viewpoint position/posture information, and outputs the second depth image.

47. An apparatus according to claim 1, wherein said warping means generates at least one of the second sequence of depth images from a respective one of the first sequence of depth images by approximation method.

2. A depth image measurement apparatus for continuously acquiring depth information of a scene, comprising:

image input means for inputting a sequence of images of the scene at a first sequence of viewpoints;

depth image generation means for generating a first sequence of depth images from the scene images sequentially input at said first sequence of viewpoints by said image input means;

position/posture estimation means for estimating, based on information relating to displacement of the first viewpoint, a sequence of viewpoint position/posture information of a second sequence of viewpoints viewed from the first sequence of viewpoints; and

warping means for sequentially warping the first sequence of depth images generated by said depth image generation means to a second sequence of depth images at the second sequence of viewpoints on the basis of the viewpoint position/posture information of the second sequence of viewpoints estimated by said position/posture estimation means.

3. The apparatus according to claim 2, wherein each one of the second sequence of viewpoints is a viewpoint where said image input means has reached at a second time after an elapse of a time from a first time at which said input means input a scene image at a respective viewpoint of the first sequence of viewpoints.

4. The apparatus according to claim 3, wherein the second time is a time at which said depth image measurement apparatus outputs the second depth image.

5. The apparatus according to claim 3, wherein the second time is a time at which an apparatus connected to said depth image measurement apparatus finally uses a result of processing that uses the second depth image.

6. The apparatus according to claim 3, wherein the second time is a time at which said warping means receives the first depth image.

7. The apparatus according to claim 4, wherein a time duration between the first and second times is known in advance.

8. The apparatus according to claim 3, wherein the second time can be freely set at a time after the first time.

9. The apparatus according to claim 2, wherein each one of the second sequence of viewpoints is a viewpoint obtained by adding an offset to a respective viewpoint of said image input means, said offset relating to a position and posture of a three-dimensional sensor for sensing observer'"'"'s viewpoint.

10. The apparatus according to claim 3, wherein each one of the second sequence of viewpoints is a viewpoint obtained by adding an offset to a viewpoint position and posture of said image input means at the second time, said offset relating to a position and posture of a three-dimensional sensor for sensing observer'"'"'s viewpoint.

11. The apparatus according to claim 9, further comprising a plurality of sensors for sensing observer'"'"'s viewpoint, wherein an offset is set to each one of the sensors.

12. The apparatus according to claim 9, wherein an amount of the offset is known in advance.

14. The apparatus according to claim 2, wherein said image input means inputs images from a plurality of video cameras.

15. The apparatus according to claim 14, wherein said depth image generation means generates the first depth image by performing stereo image processing for the images input by said image input means.

16. The apparatus according to claim 2, wherein said depth image generation means generates the first depth image using an active range finder.

17. The apparatus according to claim 2, wherein said position/posture estimation means estimates the sequence of viewpoint position/posture information on the basis of the image input by said image input means.

18. The apparatus according to claim 2, wherein said position/posture estimation means extracts image coordinate positions of feature points from the images sequentially input by said image input means, and estimates the sequence of viewpoint position/posture information using the image coordinate of the feature points.

19. The apparatus according to claim 18, wherein said position/posture estimation means includes:

recovery means for recovering a time-sequential locus of a viewpoint on the basis of the images sequentially input by said image input means, and

said position/posture estimation means estimates the sequence of viewpoint position/posture information on the basis of the locus.

20. The apparatus according to claim 19, wherein said recovery means recovers the locus of the viewpoint on the basis of time-sequential information of images input by said image input means.

21. The apparatus according to claim 19, wherein said recovery means recovers the locus of the viewpoint on the basis of information of each image input by said image input means.

22. The apparatus according to claim 18, wherein said position/posture estimation means estimates the image coordinate positions of the feature points at the second viewpoint position on the basis of images sequentially input by said image input means, and

estimates the sequence of viewpoint position/posture information on the basis of the estimated image coordinate positions.

23. The apparatus according to claim 17, wherein said position/posture estimation means estimates a viewpoint position of said image input means on the basis of the image, and

outputs a viewpoint position/posture information obtained by adding an offset to the viewpoint position and posture of said image input means.

24. The apparatus according to claim 2, wherein said position/posture estimation means receives a position/posture information from a three-dimensional position/posture sensor attached to said image input means, and

estimates the viewpoint position/posture information on the basis of a signal from said three-dimensional position/posture sensor.

25. The apparatus according to claim 24, wherein said position/posture estimation means recovers a time-sequential locus of a viewpoint on the basis of the signals sequentially inputted from said three-dimensional position/posture sensor, and

estimates viewpoint on the basis of the locus.

26. The apparatus according to claim 24, wherein said position/posture estimation means estimates a position and posture of a viewpoint of said three-dimensional position/posture sensor itself on the basis of an output from said three-dimensional position/posture sensor, and

outputs a viewpoint information obtained by adding an offset to the position and posture of the viewpoint of said three-dimensional position/posture sensor.

27. The apparatus according to claim 19, wherein said position/posture estimation means specifies a function having a shape similar to a locus of the viewpoint, and

outputs as the viewpoint position/posture information a value of the function at the second time.

28. The apparatus according to claim 17, wherein said position/posture estimation means outputs as the viewpoint position/posture information a coordinate transformation matrix from each first viewpoint to a respective second viewpoint.

29. The apparatus according to claim 2, wherein said warping means includes calculation means for calculating a coordinate value and depth value of a point on the second depth image, which corresponds to a point on the first depth image, by three-dimensional coordinate transformation on the basis of the viewpoint position/posture information.

30. The apparatus according to claim 29, wherein said warping means further includes compensation means for compensating a depth value of each point on the second depth image, which cannot be calculated by said calculation means.

45. An image processing apparatus using an image measurement apparatus of claim 2.

46. An image processing method using an image measurement apparatus of claim 2.

48. An apparatus according to claim 2, wherein said warping means generates at least one of the second sequence of depth images from a respective one of the first sequence of depth images by approximation method.

31. An augmented reality presentation system comprising:

a depth image measurement apparatus for continuously acquiring depth information of a scene, comprising;

image input means for inputting a sequence of images of the scene at a first sequence of viewpoints;

depth image generation means for generating a first sequence of depth images from the scene images sequentially input at said first sequence of viewpoints by said image input means;

position/posture estimation means for estimating, based on information relating to displacement of the first sequence of viewpoints, a sequence of viewpoint position/posture information of a second sequence of viewpoints viewed from the first sequence of viewpoints; and

warping means for sequentially warping the first sequence of depth images generated by said depth image generation means to a second sequence of depth images at the second sequence of viewpoints on the basis of the viewpoint position/posture information of the second sequence of viewpoints estimated by said position/posture estimation means; and

a head mount display comprising a plurality of video cameras for inputting images in front of an observer, and a display for displaying a three-dimensional grayscale image or color image,

wherein a three-dimensional grayscale or color image generated according to the second depth image is presented to the observer.

32. The system according to claim 31, further comprising:

an optical see-through HMD;

rendering means for rendering a computer graphics figure; and

masking means for masking the computer graphics figure on the basis of the second depth image, and

wherein the computer graphics figure masked by said masking means is output to said display.

33. The system according to claim 31, further comprising:

a video see-through HMD;

rendering means for rendering a computer graphics figure; and

merging means for merging the computer graphics figure to an image input by said image input means on the basis of the second depth image, and

wherein the computer graphics figure merged by said merging means is output to said display.

34. The system according to claim 32, wherein said rendering means generates a grayscale image or color image and a depth image of the computer graphics figure, and

said masking means masks the computer graphics figure on the basis of depth ordering of the depth image of the computer graphics figure and the second depth image in units of pixels.

35. The system according to claim 33, wherein said rendering means generates a grayscale image or color image and a depth image of the computer graphics figure, and

said merging means executes occlusion processing between the computer graphics figure and an image input by said image input means on the basis of depth ordering of the depth image of the computer graphics figure and the second depth image in units of pixels.

36. The system according to claim 32, wherein the second sequence of viewpoints are a viewpoint of an observer, and

an offset is an offset from a viewpoint of one of said video cameras to a viewpoint of the observer.

37. The system according to claim 36, wherein the viewpoint of the observer includes right and left viewpoints of the observer.

38. The system according to claim 36, wherein the second time is a time at which the graphics figure is displayed on said display.

39. The system according to claim 33, wherein the second sequence of viewpoints include a viewpoint position of the video cameras corresponding to right and left viewpoints of the observer at the second time.

40. The system according to claim 39, wherein the second time is a time at which the image to be merged with the graphics figure was sensed.

49. A system according to claim 31, wherein said warping means generates at least one of the second sequence of depth images from a respective one of the first sequence of depth images by approximation method.

41. A depth image measurement method for continuously acquiring depth information of a scene, comprising:

the image input step of inputting a sequence of images of the scene from a first sequence of viewpoints;

the depth image generation step of generating a first sequence of depth images from the scene images inputted in the image input step;

the position/posture estimation step of estimating, based on information relating to displacement of the first sequence of viewpoints, a sequence of viewpoint position/posture information for a second sequence of viewpoints viewed from the first sequence of viewpoints; and

the warping step of continuously warping the first sequence of depth images generated in the depth image generation step to a second sequence of depth images at the second sequence of viewpoints on the basis of the viewpoint position/posture information of the second sequence of viewpoints estimated in the position/posture estimation step.

42. An augmented reality presentation method comprising:

the image input step of inputting a sequence of images of a scene from a first sequence of viewpoints, using a stereo camera for outputting a stereo image in front of an observer;

the depth image generation step of generating a first sequence of depth images from the scene images continuously input in the image input step;

the position/posture estimation step of estimating, based on information relating to displacement of the first sequence of viewpoints, a sequence of viewpoint position/posture information at a second sequence of viewpoints, when viewed from the first sequence of viewpoints;

the warping step of warping the first sequence of depth images continuously generated in the depth image generation step to a second sequence of depth images at the second sequence of viewpoints on the basis of the sequence of viewpoint position/posture information of the second sequence of viewpoints estimated in the position/posture estimation step;

the step of discriminating depth ordering of a virtual three-dimensional grayscale image and a real world on the-basis of the second depth image; and

the step of displaying the virtual three-dimensional grayscale image onto a head mount display to merge the grayscale image with the real world.

43. A storage medium that stores an image processing program, which is implemented on a computer and continuously presents three-dimensional images to an observer, storing:

an image input program code of inputting a sequence of images from a first sequence of viewpoints;

a depth image generation program code of generating a first sequence of depth images from the continuously input images;

a position/posture information estimation program code of estimating, based on information relating to displacement of the first sequence of viewpoints, a sequence of viewpoint position/posture information of a second sequence of viewpoints, when viewed from the first sequence of viewpoints;

a warping program code of continuously warping the continuously generated first sequence of depth images into second sequence of depth images at the second sequence of viewpoints on the basis of the viewpoint position/posture information; and

a program code of presenting to the observer three-dimensional grayscale images or color images generated according to the second depth images.

44. The medium according to claim 43, wherein the individual program codes are stored as standalone program code modules independently of other program code modules.