Reduced latency video stabilization

• US 9,824,426 B2
• Filed: 08/01/2011
• Issued: 11/21/2017
• Est. Priority Date: 08/01/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Wu, Yongjun, Wozniak, Matthew, Sadhwani, Shyam, Thumpudi, Naveen
• Primary Examiner(s): Aghevli, Reza
• Application Number:
  US13/195,776
• Publication Number:
  US 20130033612A1
• Time in Patent Office: 2,304 Days
• Field of Search: 3482086, 3482221, 348448, 348607, 375219, 37524001, 37524029, 725105
• CPC Class Codes: G06T 2207/20182 Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory

Abstract

Reduced latency video stabilization methods and tools generate truncated filters for use in the temporal smoothing of global motion transforms representing jittery motion in captured video. The truncated filters comprise future and past tap counts that can be different from each other and are typically less than those of a baseline filter providing a baseline of video stabilization quality. The truncated filter future tap count can be determined experimentally by comparing a smoothed global motion transform set generated by applying a baseline filter to a video segment to those generated by multiple test filter with varying future tap counts, then settings the truncated filter future tap count based on an inflection point on an error-future tap count curve. A similar approach can be used to determine the truncated filter past tap count.

Claims

1. A computer-implemented method of digital video stabilization, the method comprising:

for at least one frame of a plurality of frames of a video segment, determining at least one global motion transform representing jittery video motion;

applying temporal smoothing to the at least one global motion transform to produce at least one smoothed global motion transform, the applying temporal smoothing utilizing a filter comprising a number of future taps and a number of past taps, the number of future taps and the number of past taps each being greater than zero, and the number of future taps being less than the number of past taps; and

processing the at least one of the frames of the video segment based on the at least one smoothed global motion transform to compensate for jittery video motion;

after the processing based on the at least one smoothed global motion transform, outputting the processed at least one of the frames of the video segment for display or subsequent processing.

2. The method of claim 1, further comprising:

as part of the determining the at least one global motion transform, processing local motion vectors of each frame of the at least one frame of the plurality of frames of the video segment to produce the at least one global motion transform.

3. The method of claim 1, wherein the at least one global motion transform is based on a similarity model of video motion.

4. The method of claim 1, wherein the filter is a truncated smooth filter.

5. The method of claim 1, wherein the number of future taps is greater than or equal to 2 and less than or equal to 4.

6. The method of claim 1, wherein the number of future taps is determined from test result data indicating a tradeoff between stabilization quality and the number of future taps or past taps in the filter.

18. The method of claim 1, wherein the video segment is a first video segment having a first frame rate, the method further comprising:

determining that a second video segment has a second frame rate that is different from the first frame rate; and

scaling the number of future taps based on the second frame rate to produce a scaled number of future taps.

19. The method of claim 1, further comprising:

increasing the number of future taps in response to determining that a video processing pipeline can tolerate an increase in latency associated with the increase in the number of future taps.

7. A computer-implemented method of determining a truncated filter future tap count for use in temporally smoothing global motion transforms representing jittery motion in a video segment, the method comprising:

determining global motion transforms for a plurality of frames in the video segment;

filtering the global motion transforms with a baseline filter comprising n future taps to generate a set of baseline smoothed global motion transforms;

filtering the global motion transforms with a plurality of test filters to generate a plurality of test smoothed global motion transform sets, each of the plurality of test filters having any of 0 through m future taps, wherein 0<

m<

n; and

determining a truncated filter future tap count based at least in part on the set of baseline smoothed global motion transforms and the plurality of test smoothed global motion transform sets, wherein the truncated filter future tap count is greater than zero, wherein the determining the truncated filter future tap count includes;

determining errors between the set of baseline smoothed global motion transforms and each of the plurality of test smoothed global motions transforms sets;

determining an inflection point along an error-future tap count curve based on the errors and the number of future taps in the plurality of test filters; and

selecting as the truncated filter future tap count, a number of future taps near the future tap count coordinate of the inflection point.

8. The method of claim 7, wherein the determining global motion transforms, the filtering the global motion transforms with a baseline filter, and the filtering the global motion transforms with a plurality of test filters are performed for a plurality of additional video segments to generate additional errors, the error-future tap count curve based on an average of the errors and the additional errors.

9. The method of claim 7, further comprising increasing the truncated filter future tap count in response to determining that a video processing pipeline can tolerate an increase in latency associated with the increase in the truncated filter future tap count.

10. Nonvolatile computer-readable storage media having encoded thereon computer-executable instructions for causing a computer system programmed thereby to perform digital video stabilization, the digital video stabilization comprising:

for at least one frame of a plurality of frames of a video segment, determining at least one global motion transform representing jittery video motion;

applying temporal smoothing to the at least one global motion transform to produce at least one smoothed global motion transform, the applying temporal smoothing utilizing a filter comprising a number of future taps and a number of past taps, the number of future taps and the number of past taps each being greater than zero, and the number of future taps being less than the number of past taps;

processing the at least one of the frames of the video segment based on the at least one smoothed global motion transform to compensate for jittery video motion; and

after the processing based on the at least one smoothed global motion transform, outputting the processed at least one of the frames of the video segment for display or subsequent processing.

11. The computer-readable storage media of claim 10, wherein the digital video stabilization further comprises:

as part of the determining the at least one global motion transform, processing local motion vectors of each frame of the at least one frame of the plurality of frames of the video segment to produce the at least one global motion transform.

12. The computer-readable storage media of claim 10, wherein the at least one global motion transform is based on a similarity model of video motion.

13. The computer-readable storage media of claim 10, wherein the filter is a truncated smooth filter.

14. The computer-readable storage media of claim 10, wherein the number of future taps is greater than or equal to 2 and less than or equal to 4.

15. The computer-readable storage media of claim 10, wherein the number of future taps is determined from test result data indicating a tradeoff between stabilization quality and the number of future taps or past taps in the filter.

16. The computer-readable storage media of claim 10, wherein the video segment is a first video segment having a first frame rate, and wherein the digital video stabilization further comprises:

determining that a second video segment has a second frame rate that is different from the first frame rate; and

scaling the number of future taps based on the second frame rate to produce a scaled number of future taps.

17. The computer-readable storage media of claim 10, wherein the digital video stabilization further comprises:

increasing the number of future taps in response to determining that a video processing pipeline can tolerate an increase in latency associated with the increase in the number of future taps.

20. A system for stabilizing digital video comprising:

a filter with a number of future taps and a number of past taps, the number of future taps and the number of past taps each being greater than zero, and the number of future taps being less than the number of past taps;

a buffer storing at least one frame of a plurality of frames of a video segment;

a device comprising;

a processor;

a memory;

wherein the memory contains code that, when executed causes the processor to;

determine at least one global motion transform representing jittery video motion of the least one frame of a plurality of frames of a video segment stored in the buffer;

utilize the filter to apply temporal smoothing to the at least one global motion transform to produce at least one smoothed global motion transform;

process the at least one of the frame of a plurality of frames of a video segment based on the at least one smoothed global motion transform to compensate for the jittery video motion; and

output the processed at least one of the frames of the video segment for display or subsequent processing.

21. The system of claim 20, wherein the code, when executed, further causes the processor to:

process local motion vectors of each frame of the at least one frame of the plurality of frames of the video segment to produce the at least one global motion transform.

22. The system of claim 20, wherein the at least one global motion transform is based on a similarity model of video motion.

23. The system of claim 20, wherein the filter is a truncated smooth filter.

24. The system of claim 20, wherein the number of future taps is greater than or equal to 2 and less than or equal to 4.

25. The system of claim 20, wherein the number of future taps is determined from test result data indicating a tradeoff between stabilization quality and the number of future taps or past taps in the filter.

26. The system of claim 20, wherein the video segment is a first video segment having a first frame rate, and wherein the code, when executed, further causes the processor to:

determine that a second video segment has a second frame rate that is different from the first frame rate; and

scale the number of future taps based on the second frame rate to produce a scaled number of future taps.

27. The system of claim 20, wherein the code, when executed, further causes the processor to:

increase the number of future taps in response to determining that a video processing pipeline can tolerate an increase in latency associated with the increase in the number of future taps.