Image information processing system, image information processing method, image information processing program, and automobile

• US 7,672,481 B2
• Filed: 11/29/2005
• Issued: 03/02/2010
• Est. Priority Date: 11/30/2004
• Status: Active Grant

• Current Assignee: Arriver Software AB (Qualcomm, Inc.)
• Original Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Inventors: Nagaoka, Nobuharu, Kanai, Kazunori
• Primary Examiner(s): Bali; Vikkram
• Assistant Examiner(s): Brooks; Julian D
• Application Number:
  US11/289,119
• Publication Number:
  US 20060115123A1
• Time in Patent Office: 1,554 Days
• Field of Search: 382/103, 382/104, 382/106, 382/107, 382/181, 340/435, 701/300, 180/168, 180/169
• US Class Current: 382/103
• CPC Class Codes: B60K 2360/21 using cameras
• CPC Class Codes: B60K 2370/21 using cameras
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads

Abstract

An image information processing system permits stable display of image information regarding an object on an image display device even if an image area of the object recognized through an infrared camera is unstable. In the image information processing system, a first image area recognition unit recognizes a first image area associated with an object on the basis of a second variable based on the position of the object measured through the infrared camera. Further, a second image area recognition unit estimates the first image area on the basis of a first variable and a second variable based on the running mode of an automobile, and recognizes the estimated image area as a second image area. Then, an image information control unit causes a head-up display to display one or both of the first image information associated with the first image area and the second image information associated with the second image area.

Claims

1. An image information processing system for processing an image of an object shot by an infrared camera mounted in a mobile body and then displaying image information regarding the object on an image display device mounted in the mobile body, comprising:

first variable measurement means for measuring a first variable based on a running mode of the mobile body;

second variable measurement means for measuring a second variable based on a position of the object through the infrared camera;

first image area recognition means for recognizing an image area associated with the object as a first image area on the basis of the second variable measured by the second variable measurement means;

second image area recognition means for estimating an image area recognized by the first image area recognition means on the basis of the first variable measured by the first variable measurement means and the second variable measured by the second variable measurement means and then recognizing the estimated image area as a second image area;

image information control means for displaying, on the image display device, one or both of first image information associated with the first image area recognized by the first image area recognition means and second image information associated with the second image area recognized by the second image area recognition means; and

composite image area recognition means that combines the first and the second image areas recognized by the first and the second image area recognition means, respectively, and then recognizes the result as a composite image area, wherein the image information control means causes the image display device to display one of the first and the second image information associated with the first and the second image areas, or the composite image information associated with the composite image area of the first and the second image areas recognized by the composite image area recognition means, wherein

if the image information control means determines that the first image area recognized by the first image area recognition means is stable, then it causes the first image information to be displayed on the image display device, whereas, if the image information control means determines that the first image area recognized by the first image area recognition means is unstable, then it causes the second image information or both the first image information and the second image information to be displayed on the image display device, and wherein

if the first image area recognized by the first image area recognition means is unstable, while an instability of the first image area is <

a predetermined threshold, then the image information control means causes the composite information associated with the composite image area to be displayed on the image display device, and, if the first image area recognized by the first image area recognition means is unstable, while an instability of the first image area is >

the predetermined threshold, then the image information control means causes the second image information associated with the second image area recognized by the second image area recognition means to be displayed on the image display device.

2. The image information processing system according to claim 1, wherein the image information control means causes the image display device to display composite image information associated with the composite image area of the first and the second image areas if the first and the second image areas have an overlapping portion.

3. The image information processing system according to claim 1, wherein the second image area recognition means estimates a value of the second variable, which will be measured by the second variable measurement means in the future, based on the first variable measured by the first variable measurement means and the second variable measured by the second variable measurement means, and then recognizes a second image area based on the estimated value of the second variable if the second variable measured by the second variable measurement means is determined to be unstable.

4. The image information processing system according to claim 3, wherein

the second variable measurement means performs measurement, taking a part or all of the second variable as a quaternion indicating the position of the object, and

the second image area recognition means estimates the second variable by using a quaternion expressing a spatial displacement based on the first variable measured by the first variable measurement means and the second variable measured by the second variable measurement means.

5. The image information processing system according to claim 4, wherein

the first variable measurement means measures the speed of the mobile body and the angular velocity about a certain axis as first variables, and

the second image area recognition means estimates the second variable by using a quaternion that expresses a turning motion specified by the speed and the angular velocity measured by the first variable measurement means.

6. The image information processing system according to claim 1, wherein, if the second variable measured by the second variable measurement means is unstable, then the second image area recognition means recognizes the second image area based on a prior value of the second variable measured by the second variable measurement means before such unstableness.

7. The image information processing system according to claim 6, wherein the second image area recognition means determines that the second variable is unstable if the measurement of the second variable by the second variable measurement means is disabled, or a variation amount of the second variable measured by the second variable measurement means exceeds a threshold value.

8. The image information processing system according to claim 1, wherein the image information control means determines that the first image area is unstable if a difference between or a mean value of one or a plurality of representative points of the first image area recognized by the first image area recognition means and one or a plurality of representative points of the second image area recognized by the second image area recognition means exceeds a threshold value.

9. An image information processing method for processing an image of an object shot by an infrared camera mounted in a mobile body and then displaying image information regarding the object on an image display device mounted in the mobile body, comprising:

use of a programmed computer to perform the following steps;

a first variable measurement step for measuring a first variable based on a running mode of the mobile body;

a second variable measurement step for measuring a second variable based on a position of the object through the infrared camera;

a first image area recognition step for recognizing an image area associated with the object as a first image area on the basis of the second variable measured in the second variable measurement step;

a second image area recognition step for estimating an image area recognized in the first image area recognition step on the basis of the first variable measured in the first variable measurement step and the second variable measured in the second variable measurement step and then recognizing the estimated image area as a second image area;

an image information control step for causing the image display device to display one or both of first image information associated with the first image area recognized in the first image area recognition step and second image information associated with the second image area recognized in the second image area recognition step; and

a composite image area recognition step that combines the first and the second image areas, and then recognizes the result as a composite image area, wherein the image information control step causes the image display device to display one of the first and the second image information associated with the first and the second image areas, or the composite image information associated with the composite image area of the first and the second image areas,

wherein first image information is displayed on the image display device if the first image area recognized by the first image area recognition step is determined to be stable, whereas, the second image information or both the first image information and the second image information are displayed on the image display device if the first image area recognized by the first image area recognition step is determined to be unstable, and

wherein, if the first image area is unstable, while an instability of the first image area is <

a predetermined threshold, then the image information control step causes the composite information associated with the composite image area to be displayed on the image display device, and, if the first image area is unstable, while an instability of the first image area is >

the predetermined threshold, then the image information control step causes the second image information associated with the second image area to be displayed on the image display device.

10. An image information processing system for processing an image of an object shot

by an infrared camera mounted in a mobile body and then displaying image information regarding the object on an image display device mounted in the mobile body, comprising:

first variable measurement means for measuring a first variable based on a running mode of the mobile body;

second variable measurement means for measuring a second variable based on a position of the object through the infrared camera;

first image area recognition means for recognizing an image area associated with the object as a first image area based on the second variable measured by the second variable measurement means;

second image area recognition means for estimating an image area recognized by the first image area recognition means based on the first variable measured by the first variable measurement means and the second variable measured by the second variable measurement means and then recognizing the estimated image area as a second image area;

image information control means for displaying, on the image display device, one or both of first image information associated with the first image area recognized by the first image area recognition means and second image information associated with the second image area recognized by the second image area recognition means; and

a composite image area recognition means that combines the first and the second image areas recognized by the first and the second image area recognition means, respectively, and then recognizes the result as a composite image area, wherein

the image information control means determines stability of one or more of the first and the second image areas and causes the image display device to display one of the first image information and the second image information associated with the first and the second image areas, or the composite image information associated with the composite image area of the first and the second image areas recognized by the composite image area recognition means based on the determined stability of one or more of the first and the second image areas; and

wherein, if the first image area recognized by the first image area recognition means is unstable, while an instability of the first image area is <

a predetermined threshold, then the image information control means causes the composite information associated with the composite image area to be displayed on the image display device, and, if the first image area recognized by the first image area recognition means is unstable, while an instability of the first image area is >

the predetermined threshold, then the image information control means causes the second image information associated with the second image area recognized by the second image area recognition means to be displayed on the image display device.

11. The image information processing system according to claim 10, wherein the image information control means causes the image display device to display composite image information associated with the composite image area of the first and the second image areas if the first and the second image areas have an overlapping portion.

12. The image information processing system according to claim 10, wherein the second image area recognition means estimates the second variable, which will be measured by the second variable measurement means in the future, based on the first variable measured by the first variable measurement means and the second variable measured by the second variable measurement means, and then recognizes a second image area based on the estimated second variable if the second variable measured by the second variable measurement means is determined to be unstable.

13. The image information processing system according to claim 12, wherein

the second variable measurement means performs measurement, taking a part or all of the second variable as a quaternion indicating the position of the object, and

the second image area recognition means estimates the second variable by using a quaternion expressing a spatial displacement based on the first variable measured by the first variable measurement means and the second variable measured by the second variable measurement means.

14. The image information processing system according to claim 13, wherein

the first variable measurement means measures speed of the mobile body and an angular velocity about a certain axis as first variables, and

the second image area recognition means estimates the second variable by using a quaternion that expresses a turning motion specified by the speed and the angular velocity measured by the first variable measurement means.

15. The image information processing system according to claim 10, wherein, if the second variable measured by the second variable measurement means is unstable, then the second image area recognition means recognizes the second image area based on the second variable measured by the second variable measurement means before the instability.

16. The image information processing system according to claim 15, wherein the second image area recognition means determines that the second variable is unstable if the measurement of the second variable by the second variable measurement means is disabled, or a variation amount of the second variable measured by the second variable measurement means exceeds a threshold value.