Image display apparatus, image display method, measurement apparatus, measurement method, information processing method, information processing apparatus, and identification method

• US 6,956,503 B2
• Filed: 09/12/2003
• Issued: 10/18/2005
• Est. Priority Date: 09/13/2002
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Eto, Daisuke, Satoh, Kiyohide, Uchiyama, Shinji, Yoshikawa, Tsuneo, Yokokohji, Yasuyoshi, Nakatsuru, Tomoyasu
• Primary Examiner(s): Nguyen, Tai T.
• Application Number:
  US10/660,637
• Publication Number:
  US 20040113818A1
• Time in Patent Office: 767 Days
• Field of Search: 340/988, 340/995, 340/905, 340/990, 340/991, 340/993, 340/435, 340/436, 340/461, 702/104, 702/150, 702/181, 702/182
• US Class Current: 340/988
• CPC Class Codes: G01C 21/365 Guidance using head up displays or projectors, e.g. virtual vehicles or arrows projected on the windscreen or on the road itself
• CPC Class Codes: G01S 5/163 Determination of attitude using inertial means G01C9/00; control of attitude G05D1/49
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30268 Vehicle interior
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G08G 1/0962 having an indicator mounted inside the vehicle, e.g. giving voice messages

Abstract

An external index detection unit (105) receives an image sensed by a camera (100), and detects an external index set in a real scene. A tachometer (110) measures the rotational velocities of two rear wheels of a vehicle. A vehicle measurement unit (120) measures the position and azimuth of the vehicle on the basis of the image coordinate position of the external index and the rotational velocities of the two rear wheels of the vehicle. A head index detection unit (135) receives an image sensed by a camera (130), and detects a head index set on the head of a passenger (132). A head measurement unit (140) measures the position of the head on the basis of the image coordinate position of the head index. An image generation unit (150) generates a display image based on the position and azimuth of the vehicle and the position of the head, so that the passenger can observe predetermined navigation information superimposed at a predetermined position of the real scene, and controls a projector (160) to project the generated image.

Claims

1. An information processing method which comprises a detection step of detecting a position of an index in a sensed image sensed by image sensing unit adapted to sense an image of a real space where the index is laid out, and a first calculation step of calculating a coordinate position of the index upon projecting the index in the real space onto the sensed image on the basis of the position of the index in the real space and a position and orientation of the image sensing unit when at least one of the position of the index in the real space and the position and orientation of the image sensing unit is obtained based on a measurement, and executes a process for determining correspondence between indices at coordinate positions with a smaller distance on the basis of coordinate positions of indices detected in the detection step and the coordinate position calculated in the first calculation step on the sensed image, comprising:

a second calculation step of calculating a value using a normal vector to an index of interest, and a visual axis vector of the image sensing unit; and

a determination step of determining, on the basis of a range of the value calculated in the second calculation step, whether or not the process for calculating the coordinate position of the index of interest on the sensed image in the first calculation step is to be executed,

wherein when it is determined in the determination step that the process for calculating the coordinate position of the index of interest on the sensed image in the first calculation step is to be executed, the process for determining correspondence between indices at coordinate positions with a smaller distance is executed based on the coordinate position of the index of interest calculated in the first calculation step and the coordinate positions of the indices detected in the detection step.

2. The method according to claim 1, wherein the second calculation step includes a step of calculating an angle the normal vector of the index of interest makes with the visual axis vector of the image sensing unit or a value based on the angle.

3. The method according to claim 2, wherein the determination step includes a step of determining that the process for calculating the coordinate position of the index of interest on the sensed image in the first calculation step is to be executed when the angle is 90°

+α

(α

≧

0).

4. The method according to claim 1, further comprising:

a correction step of correcting the position and orientation of the image sensing unit using a distance between the coordinate position calculated in the first calculation step and the coordinate position detected in the detection step of the determined corresponding indices;

a virtual space image generation step of generating an image of a virtual space using the position and orientation corrected in the correction step; and

a composition step of compositing and outputting the image of the virtual space generated in the virtual space image generation step and the sensed image sensed by the image sensing unit.

5. A program for making a computer execute an information processing method of claim 1.

6. A computer readable storage medium storing a program of claim 5.

7. An information processing apparatus which comprises image sensing unit adapted to sense an image of a real space where an index is laid out, detection unit adapted to detect a position of the index in a sensed image sensed by the image sensing unit, and first calculation unit adapted to, when at least one of a position of an index in the real space and a position and orientation of the image sensing unit is obtained based on a measurement, calculating a coordinate position of the index upon projecting the index in the real space onto the sensed image on the basis of the position of the index in the real space and the position and orientation of the image sensing unit, and executes a process for determining correspondence between indices at coordinate positions with a smaller distance on the basis of coordinate positions of indices detected by the detection unit and the coordinate position calculated by the first calculation unit, on the sensed image, comprising:

second calculation unit adapted to calculate a value using a normal vector to an index of interest, and a visual axis vector of the image sensing unit; and

determination unit adapted to determine, on the basis of a range of the value calculated by said second calculation unit, whether or not the process for calculating the coordinate position of the index of interest on the sensed image by the first calculation unit is to be executed,

wherein when said determination unit determines that the process for calculating the coordinate position of the index of interest on the sensed image by the first calculation unit is to be executed, the process for determining correspondence between indices at coordinate positions with a smaller distance is executed based on the coordinate position of the index of interest calculated by the first calculation unit and the coordinate positions of the indices detected by the detection unit.

8. An identification method of an index used to measure a position and orientation of an image sensing device for sensing an image of a real space, comprising:

a position/orientation measuring step of measuring roughly a position and orientation of the image sensing device;

an image sensing step of sensing an image of a real space including an index using the image sensing device;

a first image coordinate calculation step of calculating coordinate of the index, which is included in the image of the real space obtained in the image sensing step, in the obtained image; and

a second image coordinate calculation step of calculating a position of the index in a sensed image sensed by the image sensing device, whose position and orientation are obtained in the position/orientation measuring step,

wherein the index is identified on the basis of the coordinate of the index calculated in the first image coordinate calculation step, the coordinate of the index calculated in the second image coordinate calculation step, and relationship between a visual axis vector of the image sensing device obtained in the position/orientation measuring step and a normal vector of the index.

9. An identification method of an index used to measure a position and orientation of an object in a real space, comprising:

a position/orientation measuring step of measuring roughly a position and orientation of the object;

an image sensing step of sensing an image of the object using an image sensing device, which is fixed in position;

a first image coordinate calculation step of calculating coordinate of an index, which is included in the image of the object obtained in the image sensing step, in the obtained image; and

a second image coordinate calculation step of calculating a position of the index in a sensed image sensed by the image sensing device, on the basis of position and orientation are obtained in the position/orientation measuring step,

wherein the index is identified on the basis of the coordinate of the index calculated in the first image coordinate calculation step, the coordinate of the index calculated in the second image coordinate calculation step, and relationship between a visual axis vector of the image sensing device obtained in the position/orientation measuring step and a normal vector of the index.

10. An identification method of an index used to measure a position and orientation of an object in a real space, comprising:

a first position/orientation measuring step of measuring roughly a position and orientation of the object;

a second position/orientation measuring step of measuring roughly a position and orientation of an image sensing device which senses the object;

an image sensing step of sensing an image of the object using the image sensing device;

a first image coordinate calculation step of calculating coordinate of an index, which is included in the image of the object obtained in the image sensing step, in the obtained image; and

a second image coordinate calculation step of calculating a position of the index in a sensed image sensed by the image sensing device, on the basis of a position and orientation of the object with reference to the image sensing device, or of the image sensing device with reference to the object, which is obtained in the first and second position/orientation measuring step,

wherein the index is identified on the basis of the coordinate of the index calculated in the first image coordinate calculation step, the coordinate of the index calculated in the second image coordinate calculation step, and relationship between a visual axis vector of the image sensing device obtained in the second position/orientation measuring step, and a normal vector of the index obtained in the first position/orientation measuring step.