Method and apparatus for generating a database of road sign images and positions

• US 20010043718A1
• Filed: 03/20/2001
• Published: 11/22/2001
• Est. Priority Date: 10/23/1998
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Facet Technology Corp.
• Inventors: Retterath, James Eugene, Laumeyer, Robert Anthony
• Granted Patent: US 6,453,056 B2
• Time in Patent Office: Days
• US Class Current: 382/104
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/443 by matching or filtering
• CPC Class Codes: G06V 20/582 of traffic signs

Abstract

The present invention relates to an apparatus for rapidly analyzing frame(s) of digitized video data which may include objects of interest randomly distributed throughout the video data and wherein said objects are susceptible to detection, classification, and ultimately identification by filtering said video data for certain differentiable characteristics of said objects. The present invention may be practiced on pre-existing sequences of image data or may be integrated into an imaging device for real time, dynamic, object identification, classification, logging/counting, cataloging, retention (with links to stored bitmaps of said object), retrieval, and the like. The present invention readily lends itself to the problem of automatic and semi-automatic cataloging of vast numbers of objects such as traffic control signs and utility poles disposed in myriad settings. When used in conjunction with navigational or positional inputs, such as GPS, an output from the inventive system indicates the identity of each object, calculates object location, classifies each object by type, extracts legible text appearing on a surface of the object (if any), and stores a visual representation of the object in a form dictated by the end user/operator of the system. The output lends itself to examination and extraction of scene detail which cannot practically be successfully accomplished with just human viewers operating video equipment, although human intervention can still be used to help judge and confirm a variety of classifications of certain instances and for types of identified objects.

Claims

1. A method of recognizing and determining the location of at least one of a variety of road signs from at least two image frames depicting at least one road sign wherein available known values regarding the location, orientation, and focal length of an image capture device which originally recorded the at least two image frames, comprising the steps of:

receiving at least two image frames that each depict at least a single common road sign and which correspond to an identifier tag including at least a one of the following items;

camera number, frame number, camera location coordinates, or camera orientation;

applying a fuzzy logic color filter to said at least two image frames;

filtering out and saving image frame portions containing each region that contain at least one preselected color-set from a set of at least one approved road sign color; and

saving to a memory location said image frame portions of the at least a single common road sign depicted in one of said at least two image frames which is linked to at least a one of the following items;

a camera number, an image frame number, a set of camera location coordinates, or a camera orientation direction used for recording.

2. The method of

claim 1

, and prior to completing the step of applying the fuzzy logic color filter, practicing the step of converting said at least two image frames from a native color space to a single color space portion of a L*u*v* color space and wherein the fuzzy logic color filter provides maximum value output signals for only a set of preselected color.

3. The method of

claim 2

, wherein the value output signals are determined by location in said L*u*v* color space and wherein the value output signals are assigned to a minimal set of mathematically described colors representing all the legal color names and combinations.

9. The method of

claim 1

, wherein the at least two filters is selected from a set of the following filters:

an edge filter, a color-pair filter, a color filter operating in the L*u*v* color space, an edge filter combined with a line extender, or a color filter operating in the LCH color space.

14. The method of

claim 1

, wherein in lieu of the fuzzy logic color set filter a neural network is applied to the at least two image frames.

15. The method of

claim 1

, wherein the color-set is a single color.

4. A method of rapidly recognizing road signs depicted in at least one frame of a digital videosignal, comprising the steps of:

applying at least two equally weighted filters to at least one frame of a digital depiction of a road side scene so that for each of the at least two equally weighted filters a discrete output value is obtained;

comparing the discrete output value for each respective said at least two equally weighted filters and if a discrete output of at least one of said at least two equally weighted filters does not exceed a reference value then discarding the at least one frame of digital videosignal, but if one said discrete output exceeds a reference value; and

then

setting a road sign “

image present”

flag for said at least one frame of a digital videosignal.

5. The method of

claim 4

, further comprising the step of saving a bitmap image of a portion of said at least one frame of digital videosignal.

6. The method of

claim 4

, further comprising the step of recording a location data metric corresponding to the location of the camera which originally recorded the at least one frame of digital videosignal.

7. The method of

claim 6

, wherein the location data metric further comprises the direction the camera was facing while recording, the focal length of the camera, and the location of the camera as recorded by at least one global positioning device.

8. The method of

claim 4

, further comprising the steps of:

applying another filter which differentiates between various types of road signs;

classifying as many of the images as possible by road sign type; and

creating a record in a database corresponding to the type of road sign (if known), an approximate location of the road sign, the direction the road sign faces, and at least a portion of a bitmap of the at last one frame of digital videosignal containing said road sign.

13. The method of

claim 4

, further comprising at least one more frame of digital videostream and wherein the at least one more frame of digital videostream was recorded with at least one additional camera and wherein the at least two equally weighted filters are each customized for each said at least one additional camera so that each said filter accounts for different focal lengths, illumination effects, or recording direction of each of said at one additional camera.

16. The method of

claim 4

, wherein the at least two equally weighted filters are selected from a set of the following filters:

an edge filter, a color-pair filter, a color filter operating in the L*u*v* color space, an edge filter combined with a line extender, or a color filter operating in the LCH color space.

17. The method of

claim 16

, further comprising the steps of:

growing at least two edges that were filtered until they intersect;

calculating an angle of convergence for said at least two edges;

comparing the angle of convergence to a range of acceptable angles of convergence for a corner surface of at least one each of a class of objects of interest; and

saving a record for each image from which the at least two edges were derived only if the calculated angle is within the range of acceptable angles.

18. The method of

claim 4

, further comprising the step of setting an flag to “

multiple signs present”

if more than one sign is detected in a given image frame.

19. The method of

claim 4

, further comprising the step of activating a unique visible symbol, an audible signal or tone, or a vibratory signal each time an image is detected.

20. The method of

claim 19

, wherein the symbol, signal or tone, or vibratory signal corresponds to a unique type of image.

10. A method of recognizing a single road sign depicted in a least two different frames of a digital videosignal, comprising the steps of:

searching by pairs, each location metric for each of a plurality of previously identified images of road signs having a common type, so that when at least two images appear to depict a single road sign, all of said at least two images are removed from the search space, until no more pairs are available for continued searching, and then

proceeding to the first step for a next type of road sign until no further types of road signs are available for searching

11. The method of

claim 10

, further comprising the step of recording all said pairs into an auxiliary data structure.

12. The method of

claim 11

, wherein if no other images of a road sign corresponds to any other images of a road sign of same type, then forwarding the location metric to either a human operator or a storage medium for review.