Generating synthetic video frames using optical flow

• US 10,127,644 B2
• Filed: 04/10/2015
• Issued: 11/13/2018
• Est. Priority Date: 04/10/2015
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Taggart, Angus M., Beale, Eric S.
• Primary Examiner(s): Wang, Xi
• Application Number:
  US14/683,638
• Publication Number:
  US 20160301848A1
• Time in Patent Office: 1,313 Days
• Field of Search: 3483841, 382232-253, 3752402, 3752401
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 5/40 using histogram techniques
• CPC Class Codes: H04N 7/0127 by changing the field or frame frequency of the incoming video signal, e.g. frame rate converter
• CPC Class Codes: H04N 7/0135 involving interpolation processes interpolation-based image scaling G06T3/4007; interpolation for video coding H04N19/587, H04N19/59

Abstract

A novel video system that detects events in a video sequence that causes such distortions and switch off optical flow based frame interpolation is provided. The system detects sudden changes in light intensity and switch to non-optical flow based frame interpolation when such sudden change is detected. When there is no such drastic change in light intensity, the system reverts back to using optical flow frame interpolation. Specifically, the system detects a flash event by computing a sum of differences in histograms of intensity levels between the pair of consecutive actual video frames. When the sum of differences is above a certain threshold, the video system switch to non-optical flow based frame interpolation. When the sum of differences is below the threshold, the video system reverts back to optical flow frame interpolation.

Claims

1. A method comprising:

receiving a sequence of actual video frames belonging to a media content; and

for each pair of consecutive actual video frames in the sequence;

computing a sum of differences in histograms of luma intensity levels between the pair of consecutive actual video frames,

wherein;

when the sum of differences is below a predetermined threshold value, generating a set of synthetic frames, corresponding to time points between the pair of consecutive actual video frames, by computing an optical flow field between the pair of consecutive actual video frames; and

when the sum of differences is above the predetermined threshold value, generating a set of synthetic frames, corresponding to time points between the pair of consecutive actual video frames, by blending between the pair of consecutive actual video frames.

2. The method of claim 1, wherein generating the set of synthetic frames between the pair of consecutive actual video frames when the sum of differences is above the predetermined threshold value comprises performing frame-blending between the pair of consecutive actual video frames.

3. The method of claim 1, further comprising displaying the generated set of synthetic frames.

4. The method of claim 1, wherein each synthetic frame in the generated set of synthetic frames is generated at an instant in time at which a display device is scheduled to display an actual video frame but the media content has no corresponding actual video frame.

5. The method of claim 4, wherein playback times of other actual video frames in the media content are shifted by a retiming operation related to the instant in time.

6. The method of claim 1, wherein computing the sum of differences is for determining whether there is a flash event occurring during the pair of consecutive actual video frames.

7. The method of claim 1, wherein computing the sum of differences is for determining whether there is an event that distorts an optical flow field between the pair of consecutive actual video frames.

8. An apparatus comprising:

a set of processing units;

a machine readable medium storing a program by execution by at least one of the processing units, the program comprising sets of instructions for;

receiving a sequence of actual video frames belonging to a media content; and

for each pair of consecutive actual video frames in the sequence;

computing a sum of differences in histograms of luma intensity levels between the pair of consecutive actual video frames,

wherein;

when the sum of differences is below a predetermined threshold value, generating a set of synthetic frames, corresponding to time points between the pair of consecutive actual video frames, by computing an optical flow field between the pair of consecutive actual video frames; and

when the sum of differences is above the predetermined threshold value, generating a set of synthetic frames, corresponding to time points between the pair of consecutive actual video frames, with blending between the pair of consecutive actual video frames.

9. The apparatus of claim 8, wherein the set of instructions for generating the set of synthetic frames between the pair of consecutive actual video frames when the sum of differences is above the predetermined threshold value comprises a set of instructions for performing frame-blending between the pair of consecutive actual video frames.

10. The apparatus of claim 8, further comprising displaying the generated set of synthetic frames.

11. The apparatus of claim 8, wherein each synthetic frame in the generated set of synthetic frames is generated at a time for which an actual video frame is scheduled for display but the media content has no corresponding actual video frame.

12. The apparatus of claim 11, wherein playback times of other actual video frames in the media content are shifted by a retiming operation related to the instant in time.

13. The apparatus of claim 8, wherein computing the sum of differences is for determining whether there is a flash event occurring during the pair of consecutive actual video frames.

14. The apparatus of claim 8, wherein computing the sum of differences is for determining whether there is an event that distorts an optical flow field between the pair of consecutive actual video frames.

15. An apparatus comprising:

a set of processing units;

a machine readable medium storing a program by execution by at least one of the processing units, the program comprising sets of instructions for;

a retiming engine specifying a set of timing instants at which synthetic frames are required when a-playing back a media content;

a frame interpolation engine for providing required synthetic frames by using either a first frame interpolation method or a second frame interpolation method, wherein the first frame interpolation method is optical flow and the second frame interpolation method is blending; and

a detection engine for detecting a particular video event in the media content, wherein the frame interpolation engine generates at least one synthetic frame corresponding to a time point missing from the media content by using the first frame interpolation method when the event is not detected and using the second frame interpolation method when the event is detected.

16. The apparatus of claim 15, wherein the detected particular video event is a surge of luma intensity and the detection engine detects the surge of luma intensity by computing a sum of differences in histograms of luma intensity levels between a pair of consecutive actual video frames in the media content.

17. The apparatus of claim 15, wherein the particular video event causes distortion in a synthetic frame of the at least one synthetic frame generated by the first frame interpolation method.

18. The apparatus of claim 15, wherein the particular video event is a change in luma intensity between a pair of consecutive actual video frames in the media content.