Vehicle tracking and license plate recognition based on group of pictures (GOP) structure

• US 10,497,258 B1
• Filed: 09/10/2018
• Issued: 12/03/2019
• Est. Priority Date: 09/10/2018
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Georgis, Nikolaos
• Primary Examiner(s): Girma, Fekadeselassie
• Application Number:
  US16/126,669
• Time in Patent Office: 449 Days
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/62 Text, e.g. of license plates, overlay texts or captions on TV images
• CPC Class Codes: G06V 20/625 License plates
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/147 Determination of region of interest
• CPC Class Codes: G06V 30/18057 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G08G 1/0116 from roadside infrastructure, e.g. beacons
• CPC Class Codes: G08G 1/0129 for creating historical data or processing based on historical data
• CPC Class Codes: G08G 1/0175 by photographing vehicles, e.g. when violating traffic rules
• CPC Class Codes: G08G 1/04 using optical or ultrasonic detectors

Abstract

An electronic device includes an image sensor that captures a sequence of image frames that includes a plurality of different vehicles. The electronic device further includes control circuitry configured to identify a first group of frames from the captured sequence of image frames that belong to a first vehicle. A first bounding box is generated on a first vehicle region within a first frame of the identified first group of frames. A first license plate region of the first vehicle is detected within the first bounding box in the first frame and within a specific area in subsequent frames of the identified first group of frames, based on a reference of the spatial position of the first bounding box of the first vehicle in the first frame. At least the first vehicle is tracked based on the detected first license plate region of the first vehicle.

Claims

1. An electronic device, comprising:

an image sensor configured to capture a sequence of image frames that includes a plurality of vehicles; and

control circuitry configured to;

identify a group of frames from the captured sequence of image frames, wherein

the identified group of frames is associated with a vehicle of the plurality of vehicles;

generate a bounding box on a vehicle region within a first frame of the identified group of frames, wherein

the bounding box is indicative of a spatial position of the vehicle in the first frame;

detect a first license plate region of the vehicle within the first bounding box in the first frame;

detect the first license plate region of the vehicle within a first specific area in subsequent frames of the identified group of frames, based on

a reference of the spatial position of the bounding box of the vehicle in the first frame;

a resemblance between the first license plate region of the first frame and a respective second license plate region within the first specific area in the subsequent frames of the identified group of frames, and

a trained first neural network model, wherein

the first license plate region is detected in a batch; and

track at least the vehicle in the identified group of frames based on

the detected first license plate region of the vehicle within the bounding box in the first frame, and

the first specific area in the subsequent frames of the identified group of frames.

2. The electronic device of claim 1, wherein

the control circuitry is further configured to detect a plurality of vehicle regions of the plurality of vehicles within the captured sequence of image frames.

3. The electronic device of claim 1, wherein

the control circuitry is further configured to generate a first set of points demarked at four corners of

the first license plate region of the vehicle within the bounding box in the first frame, and

the first specific area in subsequent frames of the identified group of frames.

4. The electronic device of claim 3, wherein

the control circuitry is further configured to generate a second set of points within the first set of points, and

the second set of points is demarked within the first set of points to encapsulate alphanumeric characters of the first license plate region of the vehicle.

5. The electronic device of claim 3, wherein

the control circuitry is further configured to compare a position of each point of the first set of points in the first frame with a corresponding position of each point of corresponding first set of points in a second frame of the identified group of frames, based on a specified planar point matching technique.

6. The electronic device of claim 3, wherein

the control circuitry is further configured to determine an affine transformation function based on a difference between a position of each point of the first set of points in the first frame and a corresponding position of each point of a corresponding first set of points in a second frame of the identified group of frames.

7. The electronic device of claim 6, wherein

the control circuitry is further configured to update the spatial position of the bounding box of the vehicle for the second frame of the identified group of frames, based on

the detected first license plate region of the vehicle in the second frame of the identified group of frames, and

the affine transformation function.

8. The electronic device of claim 1, wherein the control circuitry is further configured to:

derive an ellipse of uncertainty within the bounding box based on at least a location of the first license plate region in the first frame; and

track the first license plate region of the vehicle within a second specific area in subsequent frames of the identified group of frames, based on the derived ellipse of uncertainty, wherein

the tracked first license plate region within the second specific area in the subsequent frames is independent of occlusion and license plate edge artifacts.

9. The electronic device of claim 1, wherein the control circuitry is further configured to:

extract, by optical character recognition (OCR), a first license plate number from the detected first license plate region of the vehicle from the first frame;

extract, by the OCR, a second license plate number from the detected first license plate region of the vehicle from a second frame of the identified group of frames; and

verify that each of the bounding box and the detected first license plate region corresponds to the vehicle, wherein

the verification is based on a comparison of the extracted first license plate number and the extracted second license plate number.

10. The electronic device of claim 9, wherein the control circuitry is further configured to receive a hotlist of vehicles comprising at least a set of license plate identifiers of a set of vehicles.

11. The electronic device of claim 10, wherein the control circuitry is further configured to validate that the extracted first license plate number from the detected first license plate region of the vehicle is one of the set of license plate identifiers in the received hotlist of vehicles.

12. The electronic device of claim 11, wherein

the control circuitry is further configured to output an alert to a law enforcement user in one of a real time or a near-real time, and

the alert is output based on the validation.

13. A method, comprising:

in an electronic device that includes control circuitry and an image sensor;

capturing, by the image sensor, a sequence of image frames that includes a plurality of vehicles;

identifying, by the control circuitry, a group of frames from the captured sequence of image frames, wherein

the identified group of frames is associated with a vehicle of the plurality of vehicles;

generating, by the control circuitry, a bounding box on a vehicle region within a first frame of the identified group of frames, wherein

the bounding box is indicative of a spatial position of the vehicle in the first frame;

detecting, by the control circuitry, a first license plate region of the vehicle within the bounding box in the first frame;

detecting, by the control circuitry, the first license plate region of the vehicle within a specific area in subsequent frames of the identified group of frames, based on

a reference of the spatial position of the bounding box of the vehicle in the first frame;

a resemblance between the first license plate region of the first frame and a respective second license plate region within the specific area in the subsequent frames of the identified group of frames, and

a trained first neural network model, wherein

the first license plate region is detected in a batch; and

tracking, by the control circuitry, at least the vehicle in the identified group of frames based on

the detected first license plate region of the vehicle within the bounding box in the first frame, and

the specific area in the subsequent frames of the identified group of frames.

14. The method of claim 13, further comprising detecting, by the control circuitry, a plurality of vehicle regions of the plurality of vehicles within the captured sequence of image frames.

15. The method of claim 13, further comprising generating, by the control circuitry, a first set of points demarked at four corners of

the first license plate region of the vehicle within the bounding box in the first frame, and

the specific area in the subsequent frames of the identified group of frames.

16. The method of claim 15, further comprising generating, by the control circuitry, a second set of points within the first set of points, wherein

the second set of points are demarked within the first set of points to encapsulate alphanumeric characters of the first license plate region of the vehicle.

17. The method of claim 13, further comprising:

in the control circuitry;

extracting, by optical character recognition (OCR), a first license plate number from the detected first license plate region of the vehicle from the first frame;

extracting, by the OCR, a second license plate number from the detected first license plate region of the vehicle from a second frame of the identified group of frames; and

verifying that each of the bounding box and the detected first license plate region correspond to the vehicle, wherein

the verification is based on a comparison of the extracted first license plate number and the extracted second license plate number.

18. The method of claim 17, further comprising validating, by the control circuitry, that the extracted first license plate number from the detected first license plate region of the vehicle is one of a set of license plate identifiers in a hotlist of vehicles.

19. An electronic device, comprising:

an image sensor configured to capture a sequence of image frames that includes a plurality of vehicles; and

control circuitry configured to;

identify a group of frames from the captured sequence of image frames, wherein

the identified group of frames is associated with a vehicle of the plurality of vehicles;

generate a bounding box on a vehicle region within a first frame of the identified group of frames, wherein

the bounding box is indicative of a spatial position of the vehicle in the first frame;

detect a first license plate region of the vehicle within the bounding box in the first frame;

derive an ellipse of uncertainty within the bounding box based on at least a location of the first license plate region in the first frame;

detect the first license plate region of the vehicle within a first specific area in subsequent frames of the identified group of frames, based on a reference of the spatial position of the bounding box of the vehicle in the first frame;

track at least the vehicle in the identified group of frames based on

the detected first license plate region of the vehicle within the bounding box in the first frame, and

the first specific area in the subsequent frames of the identified group of frames; and

track the first license plate region of the vehicle within a second specific area in the subsequent frames of the identified group of frames, based on the derived ellipse of uncertainty, wherein

the tracked first license plate region within the second specific area in the subsequent frames is independent of occlusion and license plate edge artifacts.