Video image alignment for video stabilization

• US 9,948,859 B2
• Filed: 08/15/2016
• Issued: 04/17/2018
• Est. Priority Date: 08/19/2015
• Status: Active Grant

• Current Assignee: Optimum Semiconductor Technologies, Inc.
• Original Assignee: Optimum Semiconductor Technologies, Inc.
• Inventors: Iancu, Daniel Sabin
• Primary Examiner(s): Lam, Hung H
• Application Number:
  US15/236,904
• Publication Number:
  US 20170054909A1
• Time in Patent Office: 610 Days
• CPC Class Codes: G06T 2207/20056 Discrete and fast Fourier transform, [DFT, FFT]
• CPC Class Codes: G06T 7/262 using transform domain methods, e.g. Fourier domain methods
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory

Abstract

A system and method relate to calculating a first edge map associated with a reference video frame, generating a second edge map associated with an incoming video frame, generating an offset between the reference video frame and the video frame based on a first frequency domain representation of the first edge map and a second frequency domain representation of the second edge map, translating locations of a plurality of pixels of the incoming video frame according to the calculated offset to align the incoming video frame with respect to the reference video frame, and transmitting the aligned video frame to a downstream device.

Claims

1. A method comprising:

generating, by a processing device executing an edge detector, a first edge map associated with a reference video frame;

generating, by the processing device executing the edge detector, a second edge map associated with an incoming video frame;

calculating, by the processing device, an offset between the reference video frame and the video frame based on a first frequency domain representation of the first edge map and a second frequency domain representation of the second edge map;

translating, by the processing device, locations of a plurality of pixels of the incoming video frame according to the calculated offset to align the incoming video frame with respect to the reference video frame; and

transmitting, by the processing device, the aligned video frame to a downstream device.

2. The method of claim 1, wherein translating the locations of the plurality of pixels further comprises:

adjusting a location of each one of the plurality of pixels of the video frame based on the offset.

3. The method of claim 1, wherein the reference video frame is updated at a rate that depends on at least one of video frame rate or video quality.

4. The method of claim 1, further comprising:

decimating the incoming video frame to generate a decimated incoming video frame; and

decimating the reference video frame to generate a decimated reference video frame.

5. The method of claim 4, further comprising:

generating the first edge map by applying the edge detector on the decimated reference video frame; and

generating the second edge map by applying the edge detector on the decimated incoming video frame.

6. The method of claim 5, further comprising:

applying a frequency transformation to the first edge map to generate the first frequency domain representation; and

applying the frequency transformation to the second edge map to generate the second frequency domain representation.

7. The method of claim 6, wherein the frequency transformation is a Fourier transformation.

8. The method of claim 1, wherein the edge detector is a Canny edge detector.

9. The method of claim 1, wherein calculating the offset between the reference video frame and the video frame further comprises:

calculating a cross-correlation between the first frequency domain representation and the second frequency domain representation;

determining a location of a peak in the cross-correlation; and

determining the offset based on the location of the peak in the cross-correlation.

10. A system, comprising:

a memory;

a processing device, operatively coupled to the memory, to;

generate, by executing an edge detector, a first edge map associated with a reference video frame;

generate, by executing the edge detector, a second edge map associated with an incoming video frame;

calculate an offset between the reference video frame and the video frame based on a first frequency domain representation of the first edge map and a second frequency domain representation of the second edge map;

translate locations of a plurality of pixels of the incoming video frame according to the calculated offset to align the incoming video frame with respect to the reference video frame; and

transmit the aligned video frame to a downstream device.

11. The system of claim 10, wherein to translate the locations of the plurality of pixels, the processing device is further to adjust a location of each one of the plurality of pixels of the video frame based on the offset.

12. The system of claim 10, wherein the processing device is further to:

decimate the incoming video frame to generate a decimated incoming video frame; and

decimate the reference video frame to generate a decimated reference video frame.

13. The system of claim 12, wherein the processing device is further to:

generate the first edge map by applying the edge detector on the decimated reference video frame; and

generate the second edge map by applying the edge detector on the decimated incoming video frame.

14. The system of claim 13, wherein the processing device is further to:

apply a frequency transformation to the first edge map to generate the first frequency domain representation; and

apply the frequency transformation to the second edge map to generate the second frequency domain representation.

15. The system of claim 14, wherein the frequency transformation is a Fourier transformation, and wherein the edge detector is a Canny edge detector.

16. A non-transitory computer-readable medium storing instructions that, when executed by a processing device, cause the processing device to:

generate, by the processing device executing an edge detector, a first edge map associated with a reference video frame;

generate, by the processing device executing the edge detector, a second edge map associated with an incoming video frame;

calculate, by the processing device, an offset between the reference video frame and the video frame based on a first frequency domain representation of the first edge map and a second frequency domain representation of the second edge map;

translate, by the processing device, locations of a plurality of pixels of the incoming video frame according to the calculated offset to align the incoming video frame with respect to the reference video frame; and

transmit, by the processing device, the aligned video frame to a downstream device.

17. The non-transitory computer-readable medium of claim 16, to translate the locations of the plurality of pixels, the processing device is further to adjust a location of each one of the plurality of pixels of the video frame based on the offset.

18. The non-transitory computer-readable medium of claim 16, wherein the processing device is further to:

decimate the incoming video frame to generate a decimated incoming video frame; and

decimate the reference video frame to generate a decimated reference video frame.

19. The non-transitory computer-readable medium of claim 18, wherein the processing device is further to:

generate the first edge map by applying the edge detector on the decimated reference video frame; and

generate the second edge map by applying the edge detector on the decimated incoming video frame.

20. The non-transitory computer-readable medium of claim 19, wherein the processing device is further to:

apply a frequency transformation to the first edge map to generate the first frequency domain representation; and

apply the frequency transformation to the second edge map to generate the second frequency domain representation.