Method, system, electronic device, and medium for classifying license plates based on deep learning

• US 10,528,841 B2
• Filed: 06/23/2017
• Issued: 01/07/2020
• Est. Priority Date: 06/24/2016
• Status: Active Grant

• Current Assignee: Ping An Technology Company Limited (Ping An Insurance (Group) Company of China, Ltd.)
• Original Assignee: Ping An Technology Company Limited (Ping An Insurance (Group) Company of China, Ltd.)
• Inventors: Wang, Jianzong, Liu, Ming, Xiao, Jing
• Primary Examiner(s): Moyer, Andrew M
• Application Number:
  US15/737,318
• Publication Number:
  US 20180253618A1
• Time in Patent Office: 928 Days
• Field of Search: 382156
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/2414 Smoothing the distance, e.g. radial basis function networks [RBFN]
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06N 5/046 Forward inferencing; Production systems
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/625 License plates
• CPC Class Codes: G06V 20/63 Scene text, e.g. street names
• CPC Class Codes: G06V 2201/01 Solutions for problems related to non-uniform document background
• CPC Class Codes: G06V 2201/08 Detecting or categorising vehicles
• CPC Class Codes: G08G 1/0175 by photographing vehicles, e.g. when violating traffic rules

Abstract

The present invention discloses a method, a system, an electronic device, and a medium for classifying license plates based on deep learning that are applied to an electronic device. The method includes: acquiring at least one photograph sent by a terminal device; preprocessing the acquired photograph such that the preprocessed photograph matches a plurality of input parameters of a pre-trained recognition model; and inputting the preprocessed photograph to the pre-trained recognition model to recognize corresponding vehicle use information of the photograph, and sending the corresponding vehicle use information of the photograph to the terminal device. Thus, with this disclosure, the use of a vehicle in a photograph can be automatically and accurately recognized and further the photographs can be accurately classified, thereby improving the accuracy as well as the efficiency.

Claims

1. A method of classifying license plates based on deep learning, wherein the method is applied to an electronic device and comprises the steps of:

acquiring at least one photograph sent by a terminal device, the acquired photograph having license plate information of at least one vehicle;

preprocessing the acquired photograph to obtain at least one photograph matches a plurality of input parameters of a pre-trained recognition model; and

inputting the preprocessed photograph to the pre-trained recognition model to recognize vehicle use information corresponding to the preprocessed photograph based on the corresponding license plate information, and sending the vehicle use information to the terminal device, wherein the terminal device comprises a vehicle-use-recognition operation interface including a single-photograph recognition control and a multi-photograph recognition control;

if the acquired photograph is a single photograph of a vehicle, receiving a first instruction for recognizing vehicle use information of the single photograph having license plate information of the vehicle sent by the terminal device, and sending the vehicle use information of the single photograph to the terminal device, wherein the first instruction is issued through the single-photograph recognition control to the electronic device;

or

if the acquired photograph comprises multiple photographs of a plurality of vehicles, receiving a second instruction for recognizing vehicle use information the multiple photographs having license plate information of the plurality of vehicles sent by the terminal device, and sending a title of each of the multiple photographs as well as the corresponding vehicle use information of each of the multiple photographs to the terminal device, wherein the second instruction is issued through the multi-photograph recognition control to the electronic device.

2. The method of claim 1, wherein the plurality of input parameters of the pre-trained recognition model comprises:

whitening the photograph to filter out a plurality of noise pixels in the acquired photograph, wherein the whitened photograph has a pixel mean of 0 and a pixel variance of 1; and

adjusting a size of the whitened photograph according to an input dimension of the pre-trained recognition model.

3. The method of claim 1, wherein the pre-trained recognition model is a trained deep convolutional neural network (CNN) model, and the deep CNN model is trained by the following processes:

acquiring a plurality of preset photograph samples having license plate information of a plurality of preset vehicles;

classifying the acquired photograph samples according to vehicle use information thereof to obtain at least one photograph category;

whitening each photograph sample in each photograph category, adjusting a size of the whitened photograph sample, and rotating the adjusted photograph sample to obtain a plurality of photograph sample sets under the photograph category;

acquiring a plurality of photograph sample sets to be trained under a photograph category and a plurality of photograph sample sets to be tested under a photograph category from the photograph sample sets under the photograph category; and

inputting the plurality of photograph samples of the photograph sample sets to be trained sequentially into the deep CNN model, using a preset training algorithm to train the deep CNN model used for recognizing vehicle use information based on license plate information, and testing the trained deep CNN model using the plurality of photograph samples of the plurality of photograph sample sets to be tested under each photograph category.

4. The method of claim 1, further comprising:

when the acquired photograph includes a plurality of photographs, classifying the acquired photograph according to vehicle use information thereof, so that the acquired photographs with the same vehicle use information are classified into a same photograph category, and the acquired photographs with different vehicle use information are classified into different photograph categories.

5. The method of claim 1, wherein the electronic device is a computer, a single web server, a server group consisting of a plurality of web servers, or a cloud formed by a plurality of hosts or a plurality of web servers based on cloud computing.

6. The method of claim 3, further comprising:

completing the training of the deep CNN model and using the trained CNN model for subsequent recognition of the user of a vehicle in a photograph when a recognition accuracy of the trained deep CNN model is higher than or equal to a preset threshold; and

issuing a reminder message to remind a user to increase the number of the plurality of photograph samples containing license plate information and retrain the deep CNN model according to the operations recited in claim 3 when the recognition accuracy of the trained deep CNN model is lower than the preset threshold.

7. An electronic device comprising a processing device, a storage device, a system for classifying a plurality of license plates based on deep learning stored in the storage device and at least one computer-readable instruction, wherein the computer-readable instruction is executable by the processing device to perform the following operations:

acquiring at least one photograph sent by a terminal device, the acquired photograph having license plate information of at least one vehicle;

preprocessing the acquired photograph to obtain at least one photograph matches a plurality of input parameters of a pre-trained recognition model; and

inputting the preprocessed photograph to the pre-trained recognition model to recognize vehicle use information corresponding to the preprocessed photograph based on the corresponding license plate information, and sending the vehicle use information to the terminal device, wherein the terminal device comprises a vehicle-use-recognition operation interface including a single-photograph recognition control and a multi-photograph recognition control;

if the acquired photograph is a single photograph of a vehicle, receiving a first instruction for recognizing vehicle use information of the single photograph having license plate information of the vehicle sent by the terminal device, and sending the vehicle use information of the single photograph to the terminal device, wherein the first instruction is issued through the single-photograph recognition control to the electronic device;

or

if the acquired photograph comprises multiple photographs of a plurality of vehicles, receiving a second instruction for recognizing vehicle use information the multiple photographs having license plate information of the plurality of vehicles sent by the terminal device, and sending a title of each of the multiple photographs as well as the corresponding vehicle use information of each of the multiple photographs to the terminal device, wherein the second instruction is issued through the multi-photograph recognition control to the electronic device.

8. The electronic device of claim 7, wherein the plurality of input parameters of the pre-trained recognition model comprises:

whitening the acquired photograph to filter out a plurality of noise pixels in the acquired photograph, wherein the whitened photograph has a pixel mean of 0 and a pixel variance of 1; and

adjusting a size of the whitened photograph according to an input dimension of the pre-trained recognition model.

9. The electronic device of claim 7, wherein the pre-trained recognition model is a trained deep convolutional neural network (CNN) model, and the deep CNN model is trained by the following processes:

acquiring a plurality of preset photograph samples having license plate information of a plurality of preset vehicles;

classifying the acquired photograph according to vehicle use information thereof to obtain at least one photograph category;

whitening each photograph sample in each photograph category, adjusting a size of the whitened photograph sample, and rotating the adjusted photograph sample to obtain a plurality of photograph sample sets under the photograph category;

acquiring a plurality of photograph sample sets to be trained under a photograph category and a plurality of photograph sample sets to be tested under a photograph category; and

inputting the plurality of photograph samples of the photograph sample sets to be trained sequentially into the deep CNN model, using a preset training algorithm to train the deep CNN model used for recognizing vehicle use information based on license plate information, and testing the trained deep CNN model using the plurality of photograph samples of the plurality of photograph sample sets to be tested under each photograph category.

10. The electronic device of claim 7, wherein the computer-readable instruction is executable by the processing device to further perform the following operations:

when the acquired photograph includes a plurality of photographs, classifying the photograph by vehicle use information thereof, so that the acquired photographs with the same vehicle use are classified into a same photograph category, and the acquired photographs with different vehicle use information are classified into different photograph categories.

11. The electronic device of claim 7, wherein the electronic device is a computer, a single web server, a server group consisting of a plurality of web servers, or a cloud formed by a plurality of hosts or a plurality of web servers based on cloud computing.

12. The electronic device of claim 9, wherein the computer-readable instruction is executable by the processing device to further perform the following operations:

completing the training of the deep CNN model and using the trained CNN model for subsequent recognition of the user of a vehicle in a photograph when a recognition accuracy of the trained deep CNN model is higher than or equal to a preset threshold; and

issuing a reminder message to remind a user to increase the number of the plurality of photograph samples containing license plate information and retrain the deep CNN model according to the operations recited in claim 9 when the recognition accuracy of the trained deep CNN model is lower than the preset threshold.

13. A non-transitory computer-readable storage medium, wherein the computer-readable storage medium comprises at least one computer-readable instruction executable by a processing device to perform the following operations:

acquiring at least one photograph sent by a terminal device, the acquired photograph having license plate information of at least one vehicle;

preprocessing the acquired photograph to obtain at least one photograph matches a plurality of input parameters of a pre-trained recognition model; and

inputting the preprocessed photograph to the pre-trained recognition model to recognize vehicle use information corresponding to the photograph based on the corresponding license plate information, and sending the vehicle use information to the terminal device, wherein the terminal device comprises a vehicle-use-recognition operation interface including a single-photograph recognition control and a multi-photograph recognition control;

if the acquired photograph is a single photograph of a vehicle, receiving a first instruction for recognizing vehicle use information of the single photograph having license plate information of the vehicle sent by the terminal device, and sending the vehicle use information of the single photograph to the terminal device, wherein the first instruction is issued through the single-photograph recognition control to the electronic device;

or

if the acquired photograph comprises multiple photographs of a plurality of vehicles, receiving a second instruction for recognizing vehicle use information the multiple photographs having license plate information of the plurality of vehicles sent by the terminal device, and sending a title of each of the multiple photographs as well as the corresponding vehicle use information of each of the multiple photographs to the terminal device, wherein the second instruction is issued through the multi-photograph recognition control to the electronic device.

14. The non-transitory computer-readable storage medium of claim 13, wherein the plurality of input parameters of the pre-trained recognition model comprises:

whitening the acquired photograph to filter out a plurality of noise pixels in the acquired photograph, wherein the whitened photograph has a pixel mean of 0 and a pixel variance of 1; and

adjusting a size of the whitened photograph according to an input dimension of the pretrained recognition model.

15. The non-transitory computer-readable storage medium of claim 13, wherein the pretrained recognition model is a trained deep convolutional neural network (CNN) model, and the deep CNN model is trained by the following processes:

acquiring a plurality of preset photograph samples having license plate information of a plurality of preset vehicles;

classifying the acquired photograph according to vehicle use information thereof to obtain at least one photograph category;

whitening each photograph sample in each photograph category, adjusting a size of the whitened photograph sample, and rotating the adjusted photograph sample to obtain a plurality of photograph sample sets under the photograph category;

acquiring a plurality of photograph sample sets to be trained under a photograph category and a plurality of photograph sample sets to be tested under a photograph category; and

inputting the plurality of photograph samples of the photograph sample sets to be trained sequentially into the deep CNN model, using a preset training algorithm to train the deep CNN model used for recognizing vehicle use information based on license plate information, and testing the trained deep CNN model using the plurality of photograph samples of the plurality of photograph sample sets to be tested under each photograph category.

16. The non-transitory computer-readable storage medium of claim 13, wherein the computer-readable instruction is executable by the processing device to further perform the following operations:

when the acquired photograph includes a plurality of photographs, classifying the photograph according to vehicle use information, so that the acquired photographs with the same vehicle use information are classified into a same photograph category, and the acquired photographs with different vehicle use information are classified into different photograph categories.

17. The non-transitory computer-readable storage medium of claim 15, wherein the computer-readable instruction is executable by the processing device to further perform the following operations:

completing the training of the deep CNN model and using the trained CNN model for subsequent recognition of the user of a vehicle in a photograph when a recognition accuracy of the trained deep CNN model is higher than or equal to a preset threshold; and

issuing a reminder message to remind a user to increase the number of the plurality of photograph samples containing license plate information and retrain the deep CNN model according to the operations recited in claim 15 when the recognition accuracy of the trained deep CNN model is lower than the preset threshold.