Temporally smooth video enhancement

• US 9,582,860 B2
• Filed: 07/24/2015
• Issued: 02/28/2017
• Est. Priority Date: 07/24/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: St. Clair, Timothy Sepkoski, Liu, Tinglin, Babacan, Sevket Derin
• Primary Examiner(s): COUSO, JOSE L
• Application Number:
  US14/808,208
• Publication Number:
  US 20170024858A1
• Time in Patent Office: 585 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 2218/12 Classification; Matching
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44

Abstract

Implementations generally relate to enhancing a video. In some implementations, a method includes classifying one or more objects in one or more frames of the video. The method further includes determining one or more filter parameters of one or more filters based on the classifying of the one or more objects. The method further includes smoothing one or more of the determined filter parameters based on the classifying of the one or more objects. The method further includes applying one or more of the filters with corresponding smoothed filter parameters to one or more frames of the video.

Claims

1. A computer-implemented method to process video, the method comprising:

classifying one or more objects in a plurality of frames of the video;

determining one or more filter parameters of one or more filters based on the classifying of the one or more objects;

smoothing the one or more determined filter parameters based on the classifying of the one or more objects, wherein smoothing the one or more determined filter parameters includes adjusting one or more magnitudes of the one or more determined filter parameters across adjacent frames of the plurality of frames; and

applying the one or more filters to the plurality of frames of the video to modify one or more pixels of the plurality of frames of the video, wherein applying the one or more filters includes inputting the one or more smoothed filter parameters to the one or more filters and determining pixel values from the one or more filters.

2. The method of claim 1, further comprising tracking one or more of the objects across the plurality of frames of the video, wherein the smoothing of the one or more determined filter parameters is based on the tracking of the one or more tracked objects.

3. The method of claim 1, wherein adjusting one or more magnitudes of the one or more determined filter parameters across adjacent frames of the plurality of frames includes adjusting one or more thresholds used to determine the one or more magnitudes.

4. The method of claim 1, wherein one or more of the filters comprise one or more of a brightness filter, a contrast filter, a saturation filter, a tonal structure filter, a color filter, and a vignette filter.

5. The method of claim 1, further comprising smoothing the classifying by adjusting at least one confidence threshold.

6. The method of claim 1, wherein the classifying is based on one or more classification operations, wherein the smoothing the one or more determined filter parameters comprises smoothing the one or more classification operations, wherein the smoothing the one or more classification operations comprises smoothing a classification confidence interval across multiple frames, and wherein the multiple frames includes the plurality of frames.

7. The method of claim 1, further comprising ranking the one or more objects, wherein applying the one or more filters is based on the ranking.

8. The method of claim 1, further comprising smoothing one or more outputs of one or more of the filters based on an average of the one or more outputs across multiple frames of the video.

9. A non-transitory computer-readable storage medium carrying program instructions thereon to process video, the instructions when executed by one or more processors cause the one or more processors to perform operations comprising:

determining one or more confidence thresholds used in object detection for a video;

classifying one or more objects in a plurality of frames of the video into one or more classifications based on detecting the one or more objects using confidence values relative to the one or more confidence thresholds;

smoothing the one or more classifications of the one or more objects by adjusting the confidence values across multiple frames of the video;

determining one or more filter parameters of one or more filters based on the one or more classifications of the one or more objects; and

applying the one or more filters to one or more frames of the plurality frames of the video to modify one or more pixels of the one or more frames of the plurality of frames of the video, wherein applying the one or more filters includes inputting the one or more filter parameters to the one or more filters and determining pixel values from the one or more filters.

10. The non-transitory computer-readable storage medium of claim 9, wherein smoothing the one or more classifications includes adjusting at least one of the one or more confidence thresholds.

11. The non-transitory computer-readable storage medium of claim 9, wherein the one or more filters comprise one or more of a brightness filter, a contrast filter, a saturation filter, a tonal structure filter, a color filter, and a vignette filter.

12. The non-transitory computer-readable storage medium of claim 9, wherein the instructions further cause the one or more processors to perform operations comprising smoothing one or more of the determined filter parameters based on the classifying of the one or more objects, wherein smoothing the one or more determined filter parameters includes adjusting one or more magnitudes of the one or more determined filter parameters across adjacent frames of the plurality of frames.

13. The non-transitory computer-readable storage medium of claim 9, wherein the instructions further cause the one or more processors to perform operations comprising smoothing one or more of the determined filter parameters based on the classifying of the one or more objects.

14. The non-transitory computer-readable storage medium of claim 9, wherein the instructions further cause the one or more processors to perform operations comprising smoothing one or more outputs of at least one of the one or more filters based on average outputs from the one or more filters across two or more frames of the video.

15. A system to process video, the system comprising:

one or more processors; and

logic encoded in one or more non-transitory tangible media for execution by the one or more processors and when executed operable to perform operations comprising;

determining one or more filter parameters of one or more filters based on one or more frames of a video;

smoothing the one or more determined filter parameters based on the one or more frames of the video;

inputting the one or more smoothed filter parameters to the one or more filters and obtaining filter output values from the one or more filters across a plurality of frames of the video;

smoothing the one or more outputs of the one or more filters based on an average of the filter output values across the plurality of frames of the video, wherein the plurality of frames includes the one or more frames; and

applying the smoothed one or more outputs to modify one or more pixels of the plurality of frames of the video.

16. The system of claim 15, wherein the logic when executed is further operable to perform operations comprising tracking one or more objects across a plurality of frames of the video, wherein the plurality of frames includes the one or more frames, and wherein the smoothing of the one or more determined filter parameters is based on the tracking of the one or more tracked objects.

17. The system of claim 15, wherein, to determine the one or more filter parameters, the logic when executed is further operable to perform operations comprising:

classifying one or more objects in the one or more frames of the video into one or more classifications, wherein the classifying is based on detecting the one or more objects using one or more confidence thresholds; and

smoothing the one or more classifications used for object detection by adjusting confidence values determined for the one or more objects across multiple frames of the video.

18. The system of claim 15, wherein the one or more filters comprise one or more of a brightness filter, a contrast filter, a saturation filter, a tonal structure filter, a color filter, and a vignette filter.

19. The system of claim 15, wherein the logic when executed is further operable to perform operations comprising classifying the one or more objects in the one or more frames of the video, wherein the logic when executed is further operable to perform operations comprising smoothing one or more confidence thresholds used for object detection by adjusting the one or more confidence thresholds, wherein the classifying is based on the one or more smoothed confidence thresholds.

20. The system of claim 15, wherein the average is a weighted average of filter output values across the plurality of frames.