Progressive vehicle searching method and device

• US 10,152,644 B2
• Filed: 11/14/2016
• Issued: 12/11/2018
• Est. Priority Date: 08/31/2016
• Status: Active Grant

• Current Assignee: Beijing University of Posts and Telecommunications
• Original Assignee: Beijing University of Posts and Telecommunications
• Inventors: Ma, Huadong, Liu, Wu, Liu, Xinchen, Zhang, Haitao, Fu, Huiyuan
• Primary Examiner(s): Entezari, Michelle M
• Application Number:
  US15/350,813
• Publication Number:
  US 20180060684A1
• Time in Patent Office: 757 Days
• Field of Search: None
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5846 using extracted text
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/24143 Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/464 using a plurality of salient features, e.g. bag-of-words [BoW] representations
• CPC Class Codes: G06V 10/54 relating to texture
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/62 Text, e.g. of license plates, overlay texts or captions on TV images
• CPC Class Codes: G06V 20/625 License plates
• CPC Class Codes: G06V 20/63 Scene text, e.g. street names

Abstract

The present application discloses a vehicle searching method and device, which can perform the steps of: calculating an appearance similarity distance between a first image of a target vehicle and several second images containing the searched vehicle; selecting several images from the several second images as several third images; obtaining corresponding license plate features of license plate areas in the first image and each of the third images with a preset Siamese neural network model; calculating a license plate feature similarity distance between the first image and each of the third images according to license plate feature; calculating a visual similarity distance between the first image and each of the third images according to the appearance similarity distance and the license plate feature similarity distance; obtaining a the first search result of the target vehicle by arranging the several third images in an ascending order of the visual similarity distances. The solution provided by the present application is not limited by application scenes, and it also improves vehicle searching speed and accuracy while reducing requirements of hardware such as cameras that collect images of a vehicle and auxiliary devices.

Claims

1. A vehicle searching method, characterized in that it comprises:

obtaining a first image of a target vehicle;

extracting a first appearance visual feature of the target vehicle from the first image;

extracting a second appearance visual feature of the searched vehicle respectively from several second images;

wherein, the second images are the images stored in a vehicle monitoring image database;

calculating an appearance similarity distance between the first image and each of the second images according to the first appearance visual feature and each of the second appearance visual features;

selecting several images from the several second images as several third images;

determining a first license plate area in the first image and a second license plate area in each of the third images;

obtaining a first license plate feature corresponding to the first license plate area and a second license plate feature corresponding to each of the second license plate areas by inputting the first license plate area and each of the second license plate areas respectively into a preset Siamese neural network model;

calculating a license plate feature similarity distance between the first image and each of the third images according to the first license plate feature and each of the second license plate features;

calculating a visual similarity distance between the first image and each of the third images according to the appearance similarity distance and the license plate feature similarity distance;

obtaining a first search result of the target vehicle by arranging the several third images in an ascending order of the visual similarity distances;

wherein, the first appearance visual feature comprises a first texture feature, a first color feature and a first semantic attribute feature;

the second appearance visual feature comprises a second texture feature, a second color feature and a second semantic attribute feature;

the step of calculating an appearance similarity distance between the first image and each of the second images according to the first appearance visual feature and each of the second appearance visual features, comprises;

performing the following steps for the first image and each of the second images respectively;

calculating a texture similarity distance according to the first texture feature and the second texture feature;

calculating a color similarity distance according to the first color feature and the second color feature;

calculating a semantic attribute similarity distance according to the first semantic attribute feature and the second semantic attribute feature;

calculating the appearance similarity distance between the first image and the second image according to the texture similarity distance, the color similarity distance, the semantic attribute similarity distance and a third preset model;

wherein, the third preset model is;

D_appearance =α

×

d_texture +β

×

d_color +(1−

α

−

β

)×

d_attribute, wherein, D_appearance is the appearance similarity distance, d _texture, is the texture similarity distance, d_color is the color similarity distance, d_attribute is the semantic attribute similarity distance, and are empirical weights.

2. The method according to claim 1, characterized in that, the step of selecting several images from the several second images as several third images, comprises:

obtaining a second search result of the target vehicle by arranging the several second images in an ascending order of the appearance similarity distances;

determining several images that rank before a first threshold value in the second search result as the several third images.

3. The method according to claim 1, characterized in that, after the step of calculating a visual similarity distance between the first image and each of the third images, the method further comprises:

calculating time-space similarity between the first image and each of the third images according to time-space meta data contained in the first image and each of the third images;

calculating a final similarity distance between the first image and each of the third images according to the visual similarity distance and the time-space similarity;

obtaining a third search result of the target vehicle by arranging the several third images in an ascending order of the final similarity distances.

4. The method according to claim 1, characterized in that it further comprises:

determining several images that rank before a second threshold value in the first search result as several fourth images;

calculating time-space similarity between the first image and each of the fourth images according to time-space meta data contained in the first image and each of the fourth images;

calculating a final similarity distance between the first image and each of the fourth images according to the visual similarity distance and the time-space similarity;

obtaining a fourth search result of the target vehicle by arranging the several fourth images in an ascending order of the final similarity distances.

5. The method according to claim 2, characterized in that it further comprises:

determining several images that rank before a second threshold value in the first search result as several fourth images;

calculating time-space similarity between the first image and each of the fourth images according to time-space meta data contained in the first image and each of the fourth images;

calculating a final similarity distance between the first image and each of the fourth images according to the visual similarity distance and the time-space similarity;

obtaining a fourth search result of the target vehicle by arranging the several fourth images in an ascending order of the final similarity distances.

6. The method according to claim 3, characterized in that, the step of calculating time-space similarity between the first image and each of the third images comprises:

calculating the time-space similarity between the first image and each of the third images with a first preset model, wherein, the first preset model is;

7. The method according to claim 1, characterized in that, the step of calculating a final similarity distance between the first image and each of the third images comprises:

calculating the final similarity distance between the first image and each of the third images with a second preset model, wherein, the second preset model is;

D=D_visual×

ST (i, j), wherein, D is the final similarity distance, D_visual is the visual similarity distance.

8. The method according to claim 1, characterized in that, the step of calculating a visual similarity distance between the first image and each of the third images comprises:

calculating the visual similarity distance between the first image and each of the third images with a fifth preset model, wherein, the fifth preset model is;

D_visual =γ

×

D_appearance +(1−

γ

)×

D_plate , wherein, D_visual , is the visual similarity distance, D_appearance is the appearance similarity distance, D_plate is the license plate feature similarity distance, γ

is an empirical weight.

9. A vehicle searching device, characterized in that it comprises a processor, coupled to a memory, that executes or facilitates execution of executable modules, the executable modules comprising:

a first image obtaining module for obtaining a first image of a target vehicle;

a first appearance feature extracting module for extracting a first appearance visual feature of the target vehicle from the first image;

a second appearance feature extracting module for extracting a second appearance visual feature of the searched vehicle respectively from several second images;

wherein, the second images are the image stored in a vehicle monitoring image database;

a first calculating module, for calculating an appearance similarity distance between the first image and each of the second images according to the first appearance visual feature and each of the second appearance visual features;

a first selecting module for selecting several images from the several second images as several third images;

a license plate area determining module, for determining a first license plate area in the first image and a second license plate area in each of the third images;

a license plate feature obtaining module, for obtaining a first license plate feature corresponding to the first license plate area and a second license plate feature corresponding to each of the second license plate areas by inputting the first license plate area and each of the second license plate areas respectively into a preset Siamese neural network model;

a second calculating module, for calculating a license plate feature similarity distance between the first image and each of the third images according to the first license plate feature and each of the second license plate features;

a third calculating module, for calculating a visual similarity distance between the first image and each of the third images according to the appearance similarity distance and the license plate feature similarity distance;

a first search result obtaining module, for obtaining a first search result of the target vehicle by arranging the several third images in an ascending order of the visual similarity distances;

wherein, the first appearance visual feature comprises a first texture feature, a first color feature and a first semantic attribute feature;

the second appearance visual feature comprises a second texture feature, a second color feature and a second semantic attribute feature;

the step of calculating an appearance similarity distance between the first image and each of the second images according to the first appearance visual feature and each of the second appearance visual features, comprises;

performing the following steps for the first image and each of the second images respectively;

calculating a texture similarity distance according to the first texture feature and the second texture feature;

calculating a color similarity distance according to the first color feature and the second color feature;

calculating a semantic attribute similarity distance according to the first semantic attribute feature and the second semantic attribute feature;

calculating the appearance similarity distance between the first image and the second image according to the texture similarity distance, the color similarity distance, the semantic attribute similarity distance and a third preset model;

wherein, the third preset model is;

D_appearance =α

×

d_texture +β

×

d_color +(1−

α

−

β

)×

d_attribute , wherein, D_appearance is the appearance similarity distance, d_texture is the texture similarity distance, d_color is the color similarity distance, d_attribute is the semantic attribute similarity distance, and are empirical weights.

10. The device according to claim 9, characterized in that the executable modules further comprises:

a fourth calculating module, for calculating, after the third calculating module is triggered, time-space similarity between the first image and each of the third images according to time-space meta data contained in the first image and each of the third images;

a fifth calculating module, for calculating a final similarity distance between the first image and each of the third images according to the visual similarity distance and the time-space similarity;

a third search result obtaining module, for obtaining a third search result of the target vehicle by arranging the several third images in an ascending order of the final similarity distances.