License plate recognition with an intelligent camera

• US 6,553,131 B1
• Filed: 09/15/1999
• Issued: 04/22/2003
• Est. Priority Date: 09/15/1999
• Status: Expired due to Fees

• Current Assignee: Siemens Corp. (Siemens AG)
• Original Assignee: Siemens Corporate Research Incorporated (Siemens AG)
• Inventors: Neubauer, Claus, Tyan, Jenn-Kwei
• Primary Examiner(s): Mehta, Bhavesh M.
• Assistant Examiner(s): AZARIAN, SEYED H
• Application Number:
  US09/396,950
• Time in Patent Office: 1,315 Days
• Field of Search: 382/105, 382/205, 382/177, 382/278, 382/159, 382/173, 382/176, 382/291, 382/204, 382/218, 382/104, 382/170, 382/169, 382/171, 382/172, 340/907, 340/933, 340/937, 701/117
• US Class Current: 382/105
• CPC Class Codes: G06V 10/10 Image acquisition document image scanning and transmission H04N1/00; control of digital cameras H04N23/60
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: G06V 20/625 License plates

Abstract

An intelligent camera system and method for recognizing license plates, in accordance with the invention, includes a camera adapted to independently capture a license plate image and recognize the license plate image. The camera includes a processor for managing image data and executing a license plate recognition program device. The license plate recognition program device includes a program for detecting orientation, position, illumination conditions and blurring of the image and accounting for the orientations, position, illumination conditions and blurring of the image to obtain a baseline image of the license plate. A segmenting program for segmenting characters depicted in the baseline image by employing a projection along a horizontal axis of the baseline image to identify positions of the characters. A statistical classifier is adapted for classifying the characters. The classifier recognizes the characters and returns a confidence score based on the probability of properly identifying each character. A memory is included for storing the license plate recognition program and the license plate images taken by an image capture device of the camera.

Claims

1. A method for recognizing license plates employing and intelligent camera with a processor and a memory, the method steps comprising:

capturing an image including a license plate by the intelligent camera;

detecting a region in which the license plate is located by performing a coarse localization of the image;

detecting orientation, position, and illumination conditions of the image and accounting for the orientations, position, and illumination conditions of the image to obtain a baseline image of the license plate;

performing a find localization of the baseline image to obtain a more accurate depiction of vertical resolution of the baseline image of the license plate;

segmenting characters depicted in the baseline image by employing a projection along a horizontal axis of the baseline image to identify positions of the characters;

classifying the characters based on statistical classifier to obtain a confidence score for the probability of properly identifying each character; and

recursively performing the method steps until each confidence score exceeds a threshold value to recognize the characters,

wherein the step of segmenting characters depicted in the baseline image by employing a projection along a horizontal axis of the baseline image to identify positions of the characters includes the steps of;

providing a projection profile of pixel intensities across vertical lines of pixels in the baseline image;

filtering the projection profile; and

identifying locations of characters in the baseline image depicted by area below a threshold value in the filtered projection profile.

4. The method as recited in claim 1, wherein the statistical classifier employs a convolutional network and the step of classifying the characters based on a statistical classifier to obtain a confidence score for the probability of properly identifying each character includes the step of training the classifier by employing virtual samples of characters.

5. The method as recited in claim 1, further comprising the step of comparing character blocks and characters to predetermined license plate codes and conventions to check accuracy of recognition.

6. The method as recited in claim 1, wherein the step of recursively performing the method steps until each confidence score exceeds a threshold value includes the step of considering adjacent characters together to attempt to improve the confidence score.

2. A method for recognizing license plates;

employing and intelligent camera with a processor and a memory, the method steps comprising;

capturing an image including a license plate by the intelligent camera;

detecting a region in which the license plate is located by performing a coarse localization of the image;

detecting orientation, position, and illumination conditions of the image and accounting for the orientations, position, and illumination conditions of the image to obtain a baseline image of the license plate;

performing a find localization of the baseline image to obtain a more accurate depiction of vertical resolution of the baseline image of the license plate;

segmenting characters depicted in the baseline image by employing a projection along a horizontal axis of the baseline image to identify positions of the characters;

classifying the characters based on statistical classifier to obtain a confidence score for the probability of properly identifying each character; and

recursively performing the method steps until each confidence score exceeds a threshold value to recognize the characters,

wherein the step of detecting orientation, position and illumination conditions of the image and accounting for the orientations, positions, and illumination conditions of the image to obtain a baseline image of the license plate includes the step of comparing each character in the image of the license plate with examples of images with different illuminations to account for illumination effects on the image.

3. A method for recognizing license plates employing and intelligent camera with a processor and a memory, the method steps comprising:

capturing an image including a license plate by the intelligent camera;

detecting a region in which the license plate is located by performing a coarse localization of the image;

detecting orientation, position, and illumination conditions of the image and accounting for the orientations, position, and illumination conditions of the image to obtain a baseline image of the license plate;

performing a find localization of the baseline image to obtain a more accurate depiction of vertical resolution of the baseline image of the license plate;

segmenting characters depicted in the baseline image by employing a projection along a horizontal axis of the baseline image to identify positions of the characters;

classifying the characters based on statistical classifier to obtain a confidence score for the probability of properly identifying each character; and

recursively performing the method steps until each confidence score exceeds a threshold value to recognize the characters,

wherein the step of detecting a region in which the license plate is located by performing a coarse localization of the image includes the steps of;

sub-sampling the image to reduce a number of pixels;

extracting vertical edges in the image;

generating a saliency map based upon the vertical edges to identify regions in the image with a probability of including the license plate; and

extracting a localization result which includes the image of the license plate.

7. An intelligent camera system for recognizing license plates comprising:

a camera adapted to independently capture a license plate image and recognize the license plate image including;

a processor for managing image data and executing a license plate recognition program device;

the license plate recognition program device including;

means for detecting orientation, position, illumination conditions and blurring of the image and accounting for the orientations, position, illumination conditions and blurring of the image to obtain a baseline image of the license plate;

means for segmenting characters depicted in the baseline image by employing a projection along a horizontal axis of the baseline image to identify positions of the characters;

a statistical classifier adapted for recognizing a classifying and characters based on a confidence score, the confidence score being based on a probability of properly identifying each character; and

a memory of storing the license plate recognition program and the license plate image taken by an image capture device of the camera;

wherein the means for segmenting includes;

means for providing a projection profile of pixel intensities across vertical lines of pixels in the baseline image;

a filter profile for filtering the projection profile; and

means for identifying locations of characters in the baseline image depicted by area below a threshold value in the filtered projection profile.

8. The intelligent camera system as recited in claim 7, further comprising a trigger device adapted to cause an image to be captured responsive to an event.

9. The intelligent camera system as recited in claim 8, wherein the event includes an approach of a vehicle.

11. The intelligent camera system as recited in claim 7, wherein the statistical classifier includes one of a convolutional network and a fully connected multilayer perceptron.

12. The intelligent camera system as recited in claim 7, wherein the memory includes a database for predetermined license plate codes and conventions for checking accuracy of recognition.

13. The intelligent camera system as recited in claim 7, further comprising a parking lot control system coupled to the intelligent camera system for determining parking fees based on character recognition of license plates.

10. An intelligent camera system for recognizing license plates comprising:

a camera adapted to independently capture a license plate image and recognize the license plate image including;

a processor for managing image data and executing a license plate recognition program device;

the license plate recognition program device including;

means for detecting orientation, position, illumination conditions and blurring of the image and accounting for the orientations, position, illumination conditions and blurring of the image to obtain a baseline image of the license plate;

means for segmenting characters depicted in the baseline image by employing a projection along a horizontal axis of the baseline image to identify positions of the characters;

a statistical classifier adapted for recognizing a classifying and characters based on a confidence score, the confidence score being based on a probability of properly identifying each character; and

a memory of storing the license plate recognition program and the license plate image taken by an image capture device of the camera,

wherein the means for detecting includes examples of images with different illuminations to account for illumination effects on the image for each character within the image.

14. A program storage device readable by machine, tangible embodying a program of instructions executable by the machine to perform method steps for recognizing license plates employing an intelligent camera with a processor and a memory, the method steps comprising:

capturing an image including a license plate by the intelligent camera;

detecting a region in which the license plate is located by performing a coarse localization of the image;

detecting orientation, position, and illumination conditions of the image and accounting for the orientations, position, and illumination conditions of the image to obtain a baseline image of the license plate;

performing a fine localization of the baseline image to obtain a more accurate depiction of vertical resolution of the baseline image of the license plate;

segmenting characters depicted in the baseline image by employing a projection along a horizontal axis of the baseline image to identify positions of the characters;

classifying the characters based on statistical classifier to obtain a confidence score for the probability of properly identifying each character; and

recursively performing the method steps until each confidence score exceeds a threshold value to recognize the characters,

wherein the steps of segmenting characters depicted in the baseline image by employing a projection along a horizontal axis of the baseline image to identify positions of the characters includes the steps of;

providing a projection profile of pixel intensities across vertical lines of pixels in the baseline image;

filtering the projection profile; and

identifying locations of characters in the baseline image depicted by area below a threshold value in the filtered projection profile.

15. The program storage device as recited in claim 14, wherein the step of detecting orientation, position, and illumination conditions of the image and accounting for the orientations, position, and illumination conditions of the image to obtain a baseline image of the license plate includes the step of comparing each character in the image of the license plate with examples of images with different illuminations to account for illumination effects on the image.

17. The program storage device as recited in claim 14, wherein the statistical classifier employs a convolutional network and the step of.classifying the characters based on a statistical classifier to obtain a confidence score for the probability of properly identifying each character includes the step of training the classifier by employing virtual samples of characters.

18. The program storage device as recited in claim 14, further comprising the step of comparing character blocks and characters to predetermined license plate codes and conventions to check accuracy of recognition.

19. The program storage device as recited in claim 14, wherein the step of recursively performing the method steps until each confidence score exceeds a threshold value includes the step of considering adjacent characters together to attempt to improve the confidence score.

16. A program storage device readable by machine, tangible embodying a program of instructions executable by the machine to perform method steps for recognizing license plates employing an intelligent camera with a processor and a memory, the method steps comprising:

capturing an image including a license plate by the intelligent camera;

detecting a region in which the license plate is located by performing a coarse localization of the image;

detecting orientation, position, and illumination conditions of the image and accounting for the orientations, position, and illumination conditions of the image to obtain a baseline image of the license plate;

performing a fine localization of the baseline image to obtain a more accurate depiction of vertical resolution of the baseline image of the license plate;

segmenting characters depicted in the baseline image by employing a projection along a horizontal axis of the baseline image to identify positions of the characters;

classifying the characters based on statistical classifier to obtain a confidence score for the probability of properly identifying each character; and

recursively performing the method steps until each confidence score exceeds a threshold value to recognize the characters,

wherein the step of detecting a region in which the license plate is located by performing a coarse localization of the image includes the steps of;

sub-sampling the image to reduce a number of pixels;

extracting vertical edges in the image;

generating a saliency map based upon the vertical edges to identify regions in the image with a probability of including the license plate; and

extracting a localization result which includes the images of the license plate.