Method for detecting and recognizing an object in an image, and an apparatus and a computer program therefor

• US 9,129,156 B2
• Filed: 10/28/2011
• Issued: 09/08/2015
• Est. Priority Date: 10/28/2010
• Status: Active Grant

• Current Assignee: CycloMedia Technology B.V.
• Original Assignee: CycloMedia Technology B.V.
• Inventors: Herbschleb, Ernst David, de With, Peter Hendrik Nelis
• Primary Examiner(s): Alavi, Amir
• Application Number:
  US13/881,856
• Publication Number:
  US 20130259367A1
• Time in Patent Office: 1,411 Days
• Field of Search: 382/103, 382/104, 382/106, 382/107, 382/164, 382/165, 382/181, 382/190, 382/201, 382/203, 382/209, 382218-220, 382/284, 348/46, 348/148, 340/436, 340/901, 340/937, 700/48, 700/51, 701/1, 701/96, 706/20
• US Class Current: 1/1
• CPC Class Codes: G06F 2218/08 Feature extraction
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 20/582 of traffic signs

Abstract

The present invention relates to an apparatus and method for detecting and recognizing an object in an image, the method comprising a plurality of stages, wherein at least one of the stages comprises an integrated approach of feature detection and object recognition of at least a part of the object. In a further embodiment at least one of the stages comprises identifying an image part that contains a feature point, and matching the image part to a set of hierarchies of templates, wherein a hierarchy comprises templates for an object to be recognized, a template describes at least a part of an object to be recognized, and a child template describes a sub-part of the part of the object described by its parent template.

Claims

1. A method, comprising:

locating at least one candidate traffic sign in the prepared input image;

rejecting one or more false detections of the located at least one candidate traffic sign; and

attributing a value that indicates an accuracy of one or more correct detections of the at least one located candidate traffic sign;

wherein locating the at least one candidate traffic sign in the prepared input image comprises;

identifying a plurality of feature points in the prepared input image;

constructing, using one or more processors, a descriptor for each of the plurality of feature points, wherein the descriptor includes a description of a vicinity associated with each of the plurality of feature points;

providing a library comprising a hierarchy of a plurality of templates corresponding to a plurality of traffic signs, each one of the plurality of templates corresponding to a specific traffic sign and comprising a plurality of sub-templates that represent a part of the specific traffic sign, wherein each one of the plurality of templates comprises an image of the specific traffic sign to which it corresponds and the plurality of sub-templates each comprise a different part of the image of the specific traffic sign;

matching the descriptor constructed for each of the plurality of feature points to the plurality of sub-templates in the library or to at least one of a plurality of corresponding descriptors derived from the plurality of sub-templates;

combining the plurality of sub-templates matched with the descriptor constructed for each of the plurality of feature points to form the at least one candidate traffic sign;

storing a location and a scale of each of the plurality of feature points for which a corresponding descriptor has been matched to a sub-template;

computing a line that corresponds to a center of the at least one candidate traffic sign for the descriptor constructed for each of the plurality of feature points and matched to the plurality of sub-templates in the library that correspond to the at least one candidate traffic sign;

if it is determined that two or more lines computed for the descriptor constructed for each of the plurality of feature points and matched to the at least one of the plurality of sub-templates in the library that correspond to the at least one candidate traffic sign cross, storing information related to the crossing of the lines as a candidate center of the at least one candidate traffic sign if the scale stored for the matches in said combination, are within a predetermined range;

clustering the stored information related to the crossing of lines as a candidate center to determine possible centers; and

combining the sub-templates corresponding to the cluster of possible centers to form the at least one candidate traffic sign.

2. The method of claim 1, wherein a template in the library comprises an indication of the confidence of the detection/recognition.

3. The method of claim 2, wherein the confidence indication of the detection/recognition is split into an indication for the at least one candidate traffic sign without a traffic sign border and a confidence indication of the at least one candidate traffic sign in which the traffic sign border is considered.

4. The method of claim 1, wherein matching comprises comparing colors the prepared input image to color description in a template.

5. The method of claim 1, wherein each of the plurality of templates defines a background mask, and wherein the background mask determines one or more areas that are ignored during matching, wherein the one or more areas that are ignored during matching comprises a group of pixels and/or a group of sub-pixels.

6. The method of claim 1, wherein the stored information related to the crossing of lines as a candidate center are clustered to determine the possible centers and the sub-templates corresponding to the cluster of possible centers are combined to form the at least one candidate traffic sign after all combinations of matches corresponding to the at least one candidate traffic sign are determined.

7. A non-transitory computer-readable medium having embodied thereon computer executable instructions which, when executed by a computer, causes the computer to prepare an input image, to locate at least one candidate traffic sign in the prepared input image, to reject one or more false detections of the located at least one candidate traffic sign, to attribute a value that indicates an accuracy of one or more correct detections of the at least one located candidate traffic sign, to identify a plurality of feature points in the prepared input image, to construct a descriptor for each of the plurality of feature points, wherein the descriptor includes a description of a vicinity associated with each of the plurality of feature points, to provide a library comprising a hierarchy of a plurality of templates corresponding to a plurality of traffic signs, each one of the plurality of templates corresponding to a specific traffic sign and comprising a plurality of sub-templates that represent a part of the specific traffic sign, wherein each one of the plurality of templates comprises an image of the specific traffic sign to which it corresponds and the plurality of sub-templates each comprise a different part of the image of the specific traffic sign, to match the descriptor constructed for each of the plurality of feature points to the plurality of sub-templates in the library or to at least one of a plurality of corresponding descriptors derived from the plurality of sub-templates, to combine the plurality of sub-templates matched with the descriptor constructed for each of the plurality of feature points to form the at least one candidate traffic sign, to store a location and a scale of each of the plurality of feature points for which a corresponding descriptor has been matched to a sub-template, to compute a line that corresponds to a center of the at least one candidate traffic sign for the descriptor constructed for each of the plurality of feature points and matched to the plurality of sub-templates in the library that correspond to the at least one candidate traffic sign, if it is determined that two or more lines computed for the descriptor constructed for each of the plurality of feature points and matched to the at least one of the plurality of sub-templates in the library that correspond to the at least one candidate traffic sign cross, to store information related to the crossing of the lines as a candidate center of the at least one candidate traffic sign if the scale stored for the matches in said combination, are within a predetermined range, to cluster the stored information related to the crossing of lines as a candidate center to determine possible centers, and to combine the sub-templates corresponding to the cluster of possible centers to form the at least one candidate traffic sign.

8. The computer-readable medium of claim 7, wherein a template in the library comprises an indication of the confidence of the detection/recognition.

9. The computer-readable medium of claim 8, wherein the computer executable instructions which, when executed by a computer, further causes the computer to split the confidence indication of the detection/recognition into an indication for the at least one candidate traffic sign without a traffic sign border and a confidence indication of the at least one candidate traffic sign in which the traffic sign border is considered.

10. The computer-readable medium of method of claim 7, wherein the computer executable instructions which, when executed by a computer, further causes the computer to compare colors in the prepared input image to a color description in a template.

11. The computer-readable medium of claim 7, wherein each of the plurality of templates defines a background mask, and wherein the background mask determines one or more areas that are ignored during matching, wherein the one or more areas that are ignored during matching comprises a group of pixels and/or a group of sub-pixels.

12. The computer-readable medium of claim 7, wherein the computer executable instructions which, when executed by a computer, further causes the computer to cluster the stored information related to the crossing of lines as a candidate center are to determine the possible centers, and to combine the sub-templates corresponding to the cluster of possible centers to form the at least one candidate traffic sign after all combinations of matches corresponding to the at least one candidate traffic sign are determined.

13. An apparatus, comprising:

one or more processors; and

a memory storing instructions which, when executed by the one or more processors, causes the one or more processors to prepare an input image, to locate at least one candidate traffic sign in the prepared input image, to reject one or more false detections of the located at least one candidate traffic sign, to a value that indicates an accuracy of one or more correct detections of the at least one located candidate traffic sign, to identify a plurality of feature points in the prepared input image, to construct a descriptor for each of the plurality of feature points, wherein the descriptor includes a description of a vicinity associated with each of the plurality of feature points, to provide a library comprising a hierarchy of a plurality of templates corresponding to a plurality of traffic signs, each one of the plurality of templates corresponding to a specific traffic sign and comprising a plurality of sub-templates that represent a part of the specific traffic sign, wherein each one of the plurality of templates comprises an image of the specific traffic sign to which it corresponds and the plurality of sub-templates each comprise a different part of the image of the specific traffic sign, to match the descriptor constructed for each of the plurality of feature points to the plurality of sub-templates in the library or to at least one of a plurality of corresponding descriptors derived from the plurality of sub-templates, to combine the plurality of sub-templates matched with the descriptor constructed for each of the plurality of feature points to form the at least one candidate traffic sign, to store a location and a scale of each of the plurality of feature points for which a corresponding descriptor has been matched to a sub-template, to compute a line that corresponds to a center of the at least one candidate traffic sign for the descriptor constructed for each of the plurality of feature points and matched to the plurality of sub-templates in the library that correspond to the at least one candidate traffic sign, if it is determined that two or more lines computed for the descriptor constructed for each of the plurality of feature points and matched to the at least one of the plurality of sub-templates in the library that correspond to the at least one candidate traffic sign cross, to store information related to the crossing of the lines as a candidate center of the at least one candidate traffic sign if the scale stored for the matches in said combination, are within a predetermined range, to cluster the stored information related to the crossing of lines as a candidate center to determine possible centers, and to combine the sub-templates corresponding to the cluster of possible centers to form the at least one candidate traffic sign.

14. The apparatus of claim 13, wherein a template in the library comprises an indication of the confidence of the detection/recognition.

15. The apparatus of claim 14, wherein the memory is further configured to store instructions which, when executed by a computer, causes the one or more processors to split the confidence indication of the detection/recognition into an indication for the at least one candidate traffic sign without a traffic sign border and a confidence indication of the at least one candidate traffic sign in which the traffic sign border is considered.

16. The apparatus of method of claim 13, the memory is further configured to store instructions which, when executed by a computer, causes the one or more processors to compare colors in the prepared input image to a color description in a template.

17. The apparatus of claim 13, wherein each of the plurality of templates defines a background mask, and wherein the background mask determines one or more areas that are ignored during matching, wherein the one or more areas that are ignored during matching comprises a group of pixels and/or a group of sub-pixels.

18. The apparatus of claim 13, the memory is further configured to store instructions which, when executed by a computer, causes the one or more processors to cluster the stored information related to the crossing of lines as a candidate center are to determine the possible centers, and to combine the sub-templates corresponding to the cluster of possible centers to form the at least one candidate traffic sign after all combinations of matches corresponding to the at least one candidate traffic sign are determined.