Image display apparatus, image display method, measurement apparatus, measurement method, information processing method, information processing apparatus, and identification method

• US 20050264433A1
• Filed: 07/14/2005
• Published: 12/01/2005
• Est. Priority Date: 09/13/2002
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Eto, Daisuke, Satoh, Kiyohide, Uchiyama, Shinji, Yoshikawa, Tsuneo, Yokokohji, Yasuyoshi, Nakatsuru, Tomoyasu
• Granted Patent: US 7,423,553 B2
• Time in Patent Office: Days
• US Class Current: 340/937
• CPC Class Codes: G01C 21/365 Guidance using head up displays or projectors, e.g. virtual vehicles or arrows projected on the windscreen or on the road itself
• CPC Class Codes: G01S 5/163 Determination of attitude using inertial means G01C9/00; control of attitude G05D1/49
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30268 Vehicle interior
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G08G 1/0962 having an indicator mounted inside the vehicle, e.g. giving voice messages

Abstract

An external index detection unit (105) receives an image sensed by a camera (100), and detects an external index set in a real scene. A tachometer (110) measures the rotational velocities of two rear wheels of a vehicle. A vehicle measurement unit (120) measures the position and azimuth of the vehicle on the basis of the image coordinate position of the external index and the rotational velocities of the two rear wheels of the vehicle. A head index detection unit (135) receives an image sensed by a camera (130), and detects a head index set on the head of a passenger (132). A head measurement unit (140) measures the position of the head on the basis of the image coordinate position of the head index. An image generation unit (150) generates a display image based on the position and azimuth of the vehicle and the position of the head, so that the passenger can observe predetermined navigation information superimposed at a predetermined position of the real scene, and controls a projector (160) to project the generated image.

Claims

1. An image display apparatus which comprises a display device that displays an image to be presented to a passenger on a vehicle on a front windshield portion of the vehicle, comprising:

vehicle measurement unit adapted to measure a position and azimuth of the vehicle on a world coordinate system;

head measurement unit adapted to measure a position of a head of the passenger on the vehicle on a coordinate system defined in the vehicle; and

control unit adapted to control the display device to display the image at a position according to the position and azimuth of the vehicle measured by said vehicle measurement unit, and the position of the head measured by said head measurement unit, on the front windshield portion.

2. The apparatus according to claim 1, wherein said vehicle measurement unit comprises external world image sensing unit adapted to sense an image of an external world of the vehicle.

3. The apparatus according to claim 2, wherein said vehicle measurement unit calculates positions of indices in the image, which are included in the image of the external world sensed by said external world image sensing unit, and measures the position and azimuth of the vehicle on the world coordinate system on the basis of the calculated position.

4. The apparatus according to claim 2, wherein said vehicle measurement unit further comprises rotational velocity measurement unit adapted to measure rotational velocities of right and left rear wheels of the vehicle, and said vehicle measurement unit calculates positions of indices in the image, which are included in the image of the external world sensed by said external world image sensing unit, and measures the position and azimuth of the vehicle on the world coordinate system on the basis of the calculated positions, and the rotational velocities measured by said rotational velocity measurement unit.

5. The apparatus according to claim 1, wherein said head measurement unit comprises head image sensing unit adapted to sense an image of the head of the passenger on the vehicle, and said head measurement unit calculates positions of indices in the image, which are included in the image of the head of the passenger sensed by said head image sensing unit, and measures the position of the head of the passenger on the vehicle on the coordinate system defined in the vehicle on the basis of the calculated positions.

6. The apparatus according to claim 1, wherein the image is an image indicating navigation information.

7. The apparatus according to claim 1, wherein said vehicle measurement unit measures the position and azimuth of the vehicle using a nonholonomic constraint.

20. A program for making a computer function as an image display apparatus of claim 1.

24. A computer readable storage medium storing a program of claim 20.

8. A measurement apparatus which is mounted on a vehicle and is used to measure a position and azimuth of the vehicle on a world coordinate system, comprising:

external world image sensing unit adapted to sense an image of an external world of the vehicle;

position calculation unit adapted to calculate positions of indices in the image, which are included in the image of the external world sensed by said external world image sensing unit; and

position/azimuth measurement unit adapted to measure the position and azimuth of the vehicle on the world coordinate system on the basis of the positions of the indices in the image calculated by said position calculation unit.

9. The apparatus according to claim 8, further comprising rotational velocity measurement unit adapted to measure rotational velocities of right and left rear wheels of the vehicle, and said position/azimuth measurement unit measures the position and azimuth of the vehicle on the world coordinate system further using the rotational velocities measured by said rotational velocity measurement unit.

21. A program for making a computer function as a measurement apparatus of claim 8.

10. A measurement apparatus which measures a position and orientation of an image sensing device or an object sensed by the image sensing device by detecting indices on an image of the object sensed by the image sensing device, comprising:

an index selection unit adapted to select indices which are to be observed on the sensed image without being occluded, wherein the position and orientation of the image sensing device or the object sensed by the image sensing device are measured on the basis of information of the selected indices.

11. The apparatus according to claim 10, wherein said index selection unit comprises:

index information holding unit adapted to hold information associated with each index; and

occlusion determination unit adapted to determine a presence/absence of occlusion of each index on the basis of the information associated with each index held by said index information holding unit, and said index selection unit selects indices which are determined to be free from occlusion.

12. The apparatus according to claim 11, wherein said index information holding unit holds at least position/orientation information of the image sensing device or the object upon sensing a template used to detect each index, and said occlusion determination unit determines the presence/absence of occlusion by comparing the position/orientation information of the image sensing device or the object held by said index information holding unit, and estimated values of a position and orientation of the image sensing device or the object.

13. The apparatus according to claim 11, wherein said index information holding unit holds at least information associated with a direction of each index, and said occlusion determination unit determines the presence/absence of occlusion by comparing the information associated with a direction of each index held by said index information holding unit with an estimated value of a direction of the image sensing device.

14. The apparatus according to claim 11, wherein said index information holding unit holds at least a position of each index, and said occlusion determination unit determines the presence/absence of occlusion on the basis of the position of each index, estimated values of a position and orientation of the image sensing device, and shape information of a real space.

15. An image display method executed by an image display apparatus which comprises a display device that displays an image to be presented to a passenger on a vehicle on a front windshield portion of the vehicle, comprising:

a vehicle measurement step of measuring a position and azimuth of the vehicle on a world coordinate system;

a head measurement step of measuring a position of a head of the passenger on the vehicle on a coordinate system defined in the vehicle; and

a control step of controlling the display device to display the image at a position according to the position and azimuth of the vehicle measured in the vehicle measurement step, and the position of the head measured in the head measurement step, on the front windshield portion.

22. A program for making a computer execute an image display method of claim 15.

16. A measurement method which is mounted on a vehicle and is used to measure a position and azimuth of the vehicle on a world coordinate system, comprising:

an external world image sensing step of sensing an image of an external world of the vehicle using an image sensing device;

a position calculation step of calculating positions of indices in the image, which are included in the image of the external world sensed in the external world image sensing step; and

a position/azimuth measurement step of measuring the position and azimuth of the vehicle on the world coordinate system on the basis of the positions of the indices in the image calculated in the position calculation step.

23. A program for making a computer execute a measurement method of claim 16.

17. A measurement method which measures a position and orientation of an image sensing device or an object sensed by the image sensing device by detecting indices on an image of the object sensed by the image sensing device, comprising:

an index selection step of selecting indices which are to be observed on the sensed image without being occluded, wherein the position and orientation of the image sensing device or the object sensed by the image sensing device are measured on the basis of information of the selected indices.

18. The method according to claim 17, wherein the index selection step comprises:

an index information holding step of holding information associated with each index in a memory; and

an occlusion determination step of determining a presence/absence of occlusion of each index on the basis of the information associated with each index held by the memory, and the index selection step includes a step of selecting indices which are determined to be free from occlusion.

19. The method according to claim 18, wherein the memory holds at least information associated with a direction of each index, and the occlusion determination step includes a step of determining the presence/absence of occlusion by comparing the information associated with a direction of each index held by the memory with an estimated value of a direction of the image sensing device.

25-34. -34. (canceled)