Image recognition apparatus and image processing apparatus

• US 6,763,125 B2
• Filed: 03/09/2001
• Issued: 07/13/2004
• Est. Priority Date: 09/29/1999
• Status: Expired due to Fees

• Current Assignee: Fujitsu Ten Limited (DENSO Corporation)
• Original Assignee: Fujitsu Ten Limited (DENSO Corporation)
• Inventors: Ohta, Akihiro
• Primary Examiner(s): Johnson, Timothy M.
• Assistant Examiner(s): Choobin, Barry
• Application Number:
  US09/801,723
• Publication Number:
  US 20020159616A1
• Time in Patent Office: 1,222 Days
• Field of Search: 382/104, 382/266, 382/106, 382/209, 382/154, 382/165, 382/199, 382/194, 382/219, 382/274, 382/286, 382/162, 382/170, 340/907, 340/933, 340/945, 340/425.5, 356/602, 348/113, 348/414.1, 348/47, 348/143, 701/200, 701/117, 359/462, 359/464, 359/466, 396/149, 399/344
• US Class Current: 382/104
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road

Abstract

In an image recognition apparatus, a left camera and a right camera are provided so as to produce a parallax. A road surface indication recognizing section recognizes road surface indications contained in a left input image that is supplied from the left camera. An edge detecting section detects remaining edges other than the edges of the recognized road surface indications among all the edges in the left input image. A matching section extracts a reference pattern in the left input image that includes a detected edge, and conducts matching between the reference pattern and a right input image that is supplied from the right camera using a pattern matching method. Based on each matching result with the reference pattern, an object recognizing section recognizes an object in the visual fields and determines a distance from the recognized object to the image recognition apparatus.

Claims

1. An image recognition apparatus comprising:

a first camera operable to shoot in a first visual field and to provide a first input image;

a second camera operable to shoot in a second visual field and to provide a second input image;

an edge detecting section operable to detect edges in the first input image that is supplied from said first camera;

a matching section operable to extract, as a reference pattern, a portion in the first input image that includes at least part of a detected edge and to conduct matching between the reference pattern and the second input image that is supplied from said second camera using a pattern matching method;

an object recognizing section operable to recognize an object in the first and second visual fields, based on a matching result; and

a road surface indication recognizing section operable to recognize road surface indications contained in at least one input image of the first input image and the second input image,

wherein said first and second cameras are provided so as to produce a parallax; and

wherein said edge detecting section is further operable to detect remaining edges other than edges of the road surface indications among all the edges in the first input image, from the first input image, based on a road surface indication recognition result.

2. The image recognition apparatus of claim 1 wherein said road surface indication recognizing section is operable to recognize only the road surface indications in the first input image.

3. The image recognition apparatus of claim 1, wherein

wherein said first camera and said second camera are disposed on a mounting axial line that is parallel with a road surface in the first and second visual fields so as to be separated from each other by a predetermined distance,

wherein said road surface indication recognizing section is further operable to:

divide the at least one input image into a plurality of band-like regions extending in a reference direction corresponding to a direction parallel with the mounting axial line in the at least one input image,

on the band-like region basis, search for a density variation pattern in the reference direction to determine as a candidate portion for a road surface indication, a portion having one end whose density increases according to a predetermined reference increase pattern and another end whose density decreases according to a predetermined reference decrease pattern, and

compare a width of the candidate portion for a road surface indication with a predetermined reference width, and when the width of the candidate portion for a road surface indication is smaller than or equal to the reference width, recognize that the candidate portion in the at least one input image is part of a road surface indication, and

wherein the reference width is predetermined based on the width of a road surface indication that is closest to said first camera and said second camera in a displayed image of the road surface.

4. The image recognition apparatus of claim 3,

wherein the reference increase pattern is a density variation pattern of from a density corresponding to black to a density corresponding to white, and

wherein the reference decrease pattern is a density variation pattern of from the density corresponding to white to the density corresponding to black.

5. The image recognition apparatus of claim 3,

wherein the reference increase pattern is a density variation pattern of from a density corresponding to black to a density corresponding to white via a density corresponding to gray, and

wherein the reference decrease pattern is a density variation pattern of from the density corresponding to white to the density corresponding to black via the density corresponding to gray.

6. The image recognition apparatus of claim 3,

wherein the reference increase pattern is a density variation pattern of from a density corresponding to black via a density corresponding to gray to a density corresponding to white, and

wherein the reference decrease pattern is a density variation pattern of from a density corresponding to white to a density corresponding to black.

7. The image recognition apparatus of claim 3,

wherein the reference increase pattern is a density variation pattern of from a density corresponding to black to a density corresponding to white, and

wherein the reference decrease pattern is a density variation pattern of from the density corresponding to white to the density corresponding to black via the density corresponding to gray.

8. The image recognition apparatus of claim 3, wherein said road surface indication recognizing section is further operable to:

determine a number of candidate portions that have a width smaller than or equal to the reference width and that are arranged consecutively in a direction approximately perpendicular to the reference direction, and

when the number of the candidate portions is greater than or equal to a predetermined reference number, recognize that the candidate portions include parts of an image of a road surface indication in the at least one input image.

9. The image recognition apparatus of claim 8, wherein said road surface indication recognizing section is further operable to:

judge whether an approximately band-like region that consists of the candidate portions that have a width smaller than or equal to the reference width and that are arranged consecutively in the direction approximately perpendicular to the reference direction in the predetermined number or more has an inclination that is within an allowable range that is predetermined based on an inclination of road surface indications in a displayed image of the road surface, and

only when the inclination of the approximately band-like region is within the allowable range, recognize that the candidate portions constituting the approximately band-like region is a road surface indication.

10. The image recognition apparatus of claim 1, wherein said road surface indication recognizing section is further operable to search an entire area of the at least one input image for candidate portions for a road surface indication.

11. The image recognition apparatus of claim 1, wherein said road surface indication recognizing section is further operable to search, for candidate portions for a road surface indication, a prediction region of the at least one input image where a road surface indication is expected to exist.

12. The image recognition apparatus of claim 1, wherein said edge detecting section is further operable to:

judge, on a pixel basis, whether pixels in the first input image are located outside road surface indication portions therein, based on a road surface indication recognition result; and

perform, only to each of pixels in the first input image that are located outside the road surface indication portions therein, a process for judging whether the pixel constitutes an edge in the first input image.

13. An image recognition apparatus comprising:

a first camera operable to shoot in a first visual field and to provide a first input image;

a second camera operable to shoot in a second visual field and to provide a second input image;

an edge detecting section operable to detect edges in the first input image that is supplied from said first camera;

a matching section operable to extract, as a reference pattern, a portion in the first input image that includes at least part of a detected edge and to conduct matching between the reference pattern and the second input image that is supplied from said second camera using a pattern matching method; and

an object recognizing section operable to recognize an object in the first and second visual fields, based on a matching result,

wherein said first and second cameras are provided so as to produce a parallax,

wherein the first input image and the second input image each consist of a plurality of pixels,

wherein said matching section is operable to conduct matching between the reference pattern and the second input image, using color densities of the respective pixels as indices, and

wherein said matching section is further operable to correct, prior to the pattern matching, the densities of the respective pixels of at least one input image of the first input image and the second input image using an offset value commensurate with an individual, color-related difference between said first camera and said second camera.

14. The image recognition apparatus of claim 13, wherein said matching section is further operable to learn an individual, color-related difference between said first camera and said second camera, based on the first input image and the second input image.

15. An image recognition apparatus comprising:

a first camera operable to shoot in a first visual field and to provide a first input image;

a second camera operable to shoot in a second visual field and to provide a second input image;

an edge detecting section operable to detect edges in the first input image that is supplied from said first camera;

a matching section operable to extract, as a reference pattern, a portion in the first input image that includes at least part of a detected edge and to conduct matching between the reference pattern and the second input image that is supplied from said second camera using a pattern matching method; and

an object recognizing section operable to recognize an object in the first and second visual fields, based on a matching result,

wherein said first and second cameras are provided so as to produce a parallax,

wherein the first input image and the second input image each consist of a plurality of pixels,

wherein said matching section is further operable to conduct matching between the reference pattern and the second input image, using color density variation amounts of the pixels as indices,

wherein said matching section is further operable to calculate a density difference between maximum and minimum densities of the pixels in the reference pattern and only when the calculated density difference is greater than or equal to a predetermined reference density difference, use the reference pattern for the pattern matching, and

wherein the predetermined reference density difference is predetermined based on a minimum density difference of a reference pattern that enables effective matching.

16. An image recognition apparatus comprising:

a first camera operable to shoot in a first visual field and to provide a first input image;

a second camera operable to shoot in a second visual field and to provide a second input image;

an edge detecting section operable to detect edges in the first input image that is supplied from said first camera;

a matching section operable to extract, as a reference pattern, a portion in the first input image that includes at least part of a detected edge and to conduct matching between the reference pattern and the second input image that is supplied from said second camera using a pattern matching method; and

an object recognizing section operable to recognize an object in the first and second visual fields, based on a matching result,

wherein said first and second cameras are provided so as to produce a parallax,

wherein said first camera and said second camera are disposed on a predetermined mounting axial line so as to be separated from each other by a predetermined distance,

wherein in performing the pattern matching, said matching means section sets a search region as a subject of a process of the pattern matching in the second input image and compares, with a density distribution of the reference pattern, a density distribution of a portion in the second input image which portion includes at least one pixel of the search region and is congruous with the reference pattern,

wherein a width of the search region in a direction that is perpendicular to a reference direction corresponding to a direction parallel with the mounting axial line in the second input image is greater than or equal to three pixels, and

wherein a coordinate in the perpendicular direction of a center of the search region in the second input image is the same as a coordinate in the perpendicular direction of a center of the reference pattern in the first input image.

17. An image recognition apparatus comprising:

a first camera operable to shoot in a first visual field and to provide a first input image;

a second camera operable to shoot in a second visual field and to provide a second input image;

an edge detecting section operable to detect edges in the first input image that is supplied from said first camera;

a matching section operable to extract, as a reference pattern, a portion in the first input image that includes at least part of a detected edge and to conduct matching between the reference pattern and the second input image that is supplied from said second camera using a pattern matching method; and

an object recognizing section operable to recognize an object in the first and second visual fields, based on a matching result,

wherein said first and second cameras are provided so as to produce a parallax,

wherein said first camera and said second camera are disposed on a predetermined mounting axial line so as to be separated from each other by a predetermined distance,

wherein in performing the pattern matching, said matching section sets a search region as a subject of a process of the pattern matching in the second input image and compares, with a density distribution of the reference pattern, a density distribution of a portion in the second input image which portion includes at least one pixel of the search region and is congruous with the reference pattern,

wherein a width of the search region in a reference direction corresponding to a direction parallel with the mounting axial line in the second input image is a width commensurate with an upper limit parallax that is determined based on an arrangement of said first camera and said second camera, and

wherein a coordinate in the reference direction of a center of the search region in the second input image is deviated from a coordinate in the reference direction of a center of the reference pattern in the first input image in the same direction as a deviation corresponding to the parallax of the second input image with respect to the first input image.

18. An image recognition apparatus comprising:

a first camera operable to shoot in a first visual field and to provide a first input image;

a second camera operable to shoot in a second visual field and to provide a second input image;

an edge detecting section operable to detect edges in the first input image that is supplied from said first camera;

a matching section operable to extract, as a reference pattern, a portion in the first input image that includes at least part of a detected edge and to conduct matching between the reference pattern and the second input image that is supplied from said second camera using a pattern matching method; and

an object recognizing section operable to recognize an object in the first and second visual fields, based on a matching result,

wherein said first and second cameras are provided so as to produce a parallax,

wherein in performing the pattern matching, said matching section;

selects, one by one, a plurality of detected edges in the first input image in order from an edge having highest edge intensity;

every time an edge is selected, sets, in the first input image, a reference pattern including at least part of the selected edge;

sets, in the second input image, a search region as a processing subject of the pattern matching, based on the position of the set reference pattern in the first input image and a matching result of a reference pattern that has already been processed and the second input image; and

compares a density distribution of the set reference pattern and a density distribution of a portion in the second input image that includes at least one pixel of the search region and is congruous with the set reference pattern.

19. An image processing apparatus comprising:

a road surface image storing section operable to store an image of a road surface; and

a road surface indication recognizing section operable to recognize a portion of a road surface indication in the road surface image,

wherein the road surface indication recognizing section is further operable to;

divide the road surface image into a plurality of band-like regions extending in a reference direction corresponding to a direction parallel with the road in the image;

on the band-like region basis, search for a density variation pattern in the reference direction to determine as a candidate portion for a road surface indication, a portion having one end whose density increases according to a predetermined reference increase pattern and another end whose density decreases according to a predetermined reference decrease pattern, and

compare a width of the searched candidate portion for a road surface indication with a predetermined reference width, and when the width of the searched candidate portion for a road surface indication is smaller than or equal to the reference width, recognize that the searched candidate portion is part of a road surface indication, and

wherein the reference width is predetermined based on the width of a road surface indication that is closest to a camera that shot the road surface in an image of the road surface.

20. The image processing apparatus of claim 19,

wherein said road surface image storing section is further operable to store a displayed image of the road surface,

wherein said road surface indication recognizing section is further operable to recognize a displayed portion of the road surface indication in the road surface image, and

wherein the reference width is predetermined based on the width of a road surface indication that is closest to a camera that shot the road surface in a displayed image of the road surface.

21. An image recognition apparatus comprising:

a first camera operable to shoot in a first visual field and to provide a first input image;

a second camera operable to shoot in a second visual field and to provide a second input image;

edge detecting means for detecting edges in the first input image that is supplied from said first camera;

matching means for extracting, as a reference pattern, a portion in the first input image that includes at least part of a detected edge and for conducting matching between the reference pattern and the second input image that is supplied from said second camera using a pattern matching method;

object recognizing means for recognizing an object in the first and second visual fields, based on a matching result; and

a road surface indication recognizing means for recognizing road surface indications contained in at least one input image of the first input image and the second input image,

wherein said first and second cameras are provided so as to produce a parallax, and

wherein said edge detecting means detects remaining edges other than edges of the road surface indications among all the edges in the first input image, from the first input image, based on a road surface indication recognition result.

22. An image processing apparatus comprising:

road surface image storing means for storing a displayed image of a road surface; and

road surface indication recognizing means for recognizing a displayed portion of a road surface indication in the road surface image,

wherein the road surface indication recognizing means;

divides the road surface image into a plurality of band-like regions extending in a reference direction corresponding to a direction parallel with the road in the image;

on the band-like region basis, searches for a density variation pattern in the reference direction to determine as a candidate portion for a road surface indication, a portion having one end whose density increases according to a predetermined reference increase pattern and another end whose density decreases according to a predetermined reference decrease pattern, and

compares a width of the searched candidate portion for a road surface indication with a predetermined reference width, and when the width of the searched candidate portion for a road surface indication is smaller than or equal to the reference width, recognizes that the searched candidate portion is part of a road surface indication, and

wherein the reference width is predetermined based on the width of a road surface indication that is closest to a camera that shot the road surface in a displayed image of the road surface.