System and method for enhancing images and video frames

• US 10,909,845 B2
• Filed: 07/01/2013
• Issued: 02/02/2021
• Est. Priority Date: 07/01/2013
• Status: Active Grant

• Current Assignee: Conduent Business Services, LLC (Conduent, Inc.)
• Original Assignee: Conduent Business Services, LLC (Conduent, Inc.)
• Inventors: Bernal, Edgar A., Wu, Wencheng
• Primary Examiner(s): Hasan, Mainul
• Application Number:
  US13/932,453
• Publication Number:
  US 20150002745A1
• Time in Patent Office: 2,773 Days
• Field of Search: 348148
• CPC Class Codes: G06T 2207/30236 Traffic on road, railway or crossing
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06V 20/54 of traffic, e.g. cars on the road, trains or boats
• CPC Class Codes: G08G 1/0175 by photographing vehicles, e.g. when violating traffic rules
• CPC Class Codes: G08G 1/04 using optical or ultrasonic detectors
• CPC Class Codes: G08G 1/052 with provision for determining speed or overspeed speed measuring in general G01P

Abstract

A system and method for performing vehicle-velocity aware image enhancement. Embodiments generally include a video capture module configured to receive image data of the scene being monitored, an image extraction module configured to extract still images from incoming video data, a vehicle detection module that detects the approximate location of a target vehicle in the scene being monitored, a velocity determination module configured to determine the amplitude and direction of a vector that describes the velocity of the target vehicle in image pixel coordinates, and a velocity-aware enhancing module configured to enhance the image(s) of the target vehicle extracted from the video feed based on the vehicle'"'"'s velocity. Embodiments may also include an infraction detection module configured to detect the occurrence of a violation of traffic law(s) by a target vehicle.

Claims

1. A computer-implemented method comprising:

obtaining video data wherein a video camera imaging a road produced the video data;

extracting from the video data an image of a vehicle wherein the vehicle is moving;

determining a location wherein the location is the vehicle'"'"'s location within the image of the vehicle;

obtaining a velocity estimate of the vehicle'"'"'s speed and direction on the road, wherein the velocity estimate is obtained from a microwave detector or the velocity estimate is assumed according to historical speed averages; and

producing an enhanced image based at least in part on the image and the velocity estimate wherein the enhanced image shows at least a portion of the vehicle.

2. The computer-implemented method of claim 1 further comprising:

extracting from the video data a second image of the vehicle; and

determining a second location wherein the second location is the vehicle'"'"'s location within the second image of the vehicle;

wherein the velocity estimate is determined at least in part on the location and the second location.

3. The computer-implemented method of claim 1 further comprising:

extracting from the video data a second image of the vehicle; and

registering the image and the second image on a lattice, wherein the lattice is higher resolution than the image and the second image;

wherein the enhanced image is a super-resolution image constructed on the lattice from the video data.

4. The computer-implemented method of claim 1 further comprising:

extracting from the video data a plurality of additional images of the vehicle; and

performing image registrations of the image of the vehicle and the additional images of the vehicle wherein the image registrations are based at least in part on the velocity estimate;

wherein the enhanced image is produced by averaging the image of the vehicle and the additional images of the vehicle.

5. The computer-implemented method of claim 1 further comprising:

estimating a plurality of point spread functions for a plurality of pixels wherein the image comprises the pixels and wherein each of the point spread functions is based at least in part on the velocity estimate, a camera configuration, a capture parameter, and an assumption of linear motion;

wherein the enhanced image is produced by deconvolving the image with the point spread functions.

6. The computer-implemented method of claim 1 further comprising:

estimating a point spread function wherein the point spread function is based at least in part on the velocity estimate, a camera configuration, a capture parameter, and an assumption of linear motion;

obtaining a frequency domain representation of the image; and

determining a frequency domain representation the point spread function;

wherein the enhanced image is produced by frequency domain deconvolution expressed as a fraction having a numerator and a denominator, wherein the numerator comprises the frequency domain representation of the image, and wherein the denominator comprises the frequency domain representation the point spread function.

7. The computer-implemented method of claim 1 further comprising:

estimating a point spread function in the frequency domain wherein the point spread function is based at least in part on the velocity estimate, a camera configuration, a capture parameter, and an assumption of linear motion;

determining a complex conjugate of a frequency domain representation the point spread function;

determining a frequency domain representation of the image;

determining an image power spectral density that is the power spectral density of the image; and

determining a noise power spectral density that is the power spectral density of additive noise;

wherein the enhanced image is produced by frequency domain deconvolution expressed as a fraction having a numerator and a denominator, wherein the numerator comprises the product of the frequency domain representation of the image, image power spectral density, and the complex conjugate, and wherein the denominator comprises the sum of the noise power spectral density with the product of the squared magnitude of the complex conjugate and the image power spectral density.

8. The computer-implemented method of claim 1 further comprising:

extracting from the video data at least one additional image of the vehicle; and

explicitly calculating an image domain vector field based on the image and the at least one additional image wherein the image domain vector field comprises a plurality of image domain displacement vectors wherein the displacement vectors describe the displacement of the vehicle across the sequence of frames.

9. The computer-implemented method of claim 1 further comprising:

obtaining a plurality of vehicle aware velocity parameters estimated in pixel units and based at last in part on a camera configuration and an exposure time;

wherein the enhanced image is produced by velocity aware de-blurring comprising sharpening the image based on the pixel-wise vehicle velocity parameters.

10. A non-transitory computer-usable medium for producing an enhanced image, the computer-usable medium embodying computer program code, the computer program code comprising computer executable instructions configured for:

obtaining image data comprising a first image showing a vehicle moving on a road;

extracting the first image from the image data;

determining a first location that is the vehicle'"'"'s location within the first image;

obtaining a velocity estimate of the vehicle'"'"'s velocity;

producing an enhanced image based at least in part on the first image and the velocity estimate wherein the enhanced image shows at least a portion of the vehicle; and

receiving vehicle location information about the vehicle from a non-video-based triggering device, wherein the vehicle location information is in real world coordinates.

11. The non-transitory computer-usable medium of claim 10 further comprising computer executable instructions configured for mapping the vehicle location information into pixel coordinates.

12. The non-transitory computer-usable medium of claim 10 further comprising computer executable instructions configured for:

estimating a frequency domain representation of the point spread function wherein the point spread function is based at least in part on the velocity estimate, a camera configuration, a capture parameter, and an assumption of linear motion;

determining a conjugate wherein the conjugate is a complex conjugate of the frequency domain representation the point spread function;

determining a frequency domain representation of the image;

determining an image power spectral density that is the power spectral density of the image;

determining a noise power spectral density that is the power spectral density of additive noise;

calculating a frequency domain representation of the enhanced image by evaluating a fraction comprising a dividend and a divisor, wherein the dividend comprises the product of the frequency domain representation of the image, image power spectral density, and the conjugate, and wherein the divisor comprises the sum of the noise power spectral density and the product of the squared magnitude of the conjugate and the image power spectral density; and

producing the enhanced image based at least in part on the frequency domain representation of the enhanced image.