GENERATING SYNTHETIC VIDEO FRAMES USING OPTICAL FLOW

• US 20160301848A1
• Filed: 04/10/2015
• Published: 10/13/2016
• Est. Priority Date: 04/10/2015
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Taggart, Angus M., Beale, Eric S.
• Granted Patent: US 10,127,644 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 5/40 using histogram techniques
• CPC Class Codes: H04N 7/0127 by changing the field or frame frequency of the incoming video signal, e.g. frame rate converter
• CPC Class Codes: H04N 7/0135 involving interpolation processes interpolation-based image scaling G06T3/4007; interpolation for video coding H04N19/587, H04N19/59

Abstract

A novel video system that detects events in a video sequence that causes such distortions and switch off optical flow based frame interpolation is provided. The system detects sudden changes in light intensity and switch to non-optical flow based frame interpolation when such sudden change is detected. When there is no such drastic change in light intensity, the system reverts back to using optical flow frame interpolation. Specifically, the system detects a flash event by computing a sum of differences in histograms of intensity levels between the pair of consecutive actual video frames. When the sum of differences is above a certain threshold, the video system switch to non-optical flow based frame interpolation. When the sum of differences is below the threshold, the video system reverts back to optical flow frame interpolation.

Claims

1. A method comprising:

receiving a sequence of actual video frames belonging to a media content; and

for each pair of consecutive actual video frames in the sequence;

computing a sum of differences in histograms of intensity levels between the pair consecutive actual video frames;

wherein;

when the sum of differences is below a predetermined threshold value, generating a set of synthetic frames by computing an optical flow field between the pair of frames; and

when the sum of differences is above the predetermined threshold value generating a set of synthetic frames without computing any optical flow field between the pair of frames.

2. The method of claim 1, wherein generating the set of synthetic frames between the pair of frames when the sum of differences is above the predetermined threshold value comprises performing frame-blending between the pair of frames.

3. The method of claim 1 further comprising displaying the generated set of synthetic frames.

4. The method of claim 1, wherein each synthetic frame in the generated set of synthetic frames is generated at an instant in time at which a display device is scheduled to display a frame but the media content has no corresponding actual frame.

5. The method of claim 4, wherein playback times of actual frames in the media content are shifted by a retiming operation.

6. The method of claim 1, wherein computing the sum of differences is for determining whether there is a flash event occurring during the pair of frames.

7. The method of claim 1, wherein computing a sum of differences is for determining whether there is an event that distorts an optical flow field between the pair of frames.

8. An apparatus comprising:

a set of processing units;

a machine readable medium storing a program by execution by at least one of the processing units, the program comprising sets of instructions for;

receiving a sequence of actual video frames belonging to a media content; and

for each pair of consecutive actual video frames in the sequence;

computing a sum of differences in histograms of intensity levels between the pair consecutive actual video frames;

wherein;

when the sum of differences is below a predetermined threshold value, generating a set of synthetic frames by computing an optical flow field between the pair of frames; and

when the sum of differences is above the predetermined threshold value, generating a set of synthetic frames without computing any optical flow field between the pair of frames.

9. The apparatus of claim 8, wherein the set of instructions for generating the set of synthetic frames between the pair of frames when the sum of differences is above the predetermined threshold value comprises a set of instructions for performing frame-blending between the pair of frames.

10. The apparatus of claim 8 further comprising displaying the generated set of synthetic frames.

11. The apparatus of claim 8, wherein each synthetic frame in the generated set of synthetic frames is generated at an instant in time for which a frame is scheduled for display but the media content has no corresponding actual frame.

12. The apparatus of claim 11, wherein playback times of actual frames in the media content are shifted by a retiming operation.

13. The apparatus of claim 8, wherein computing the sum of differences is for determining whether there is a flash event occurring during the pair of frames.

14. The apparatus of claim 8, wherein computing a sum of differences is for determining whether there is an event that distorts an optical flow field between the pair of frames.

15. An apparatus comprising:

a set of processing units;

a machine readable medium storing a program by execution by at least one of the processing units, the program comprising sets of instructions for;

a retiming engine specifying a set of timing instants at which synthetic frames are required when a playing back a media content;

a frame interpolation engine for providing the required synthetic frames by using either a first frame interpolation method and a second frame interpolation method;

a detection engine for detecting a particular video event in the media content, wherein the frame interpolation engine uses the second frame interpolation method when the event is not detected and uses the second frame interpolation method when the event is detected.

16. The apparatus of claim 15, wherein the detection engine detects the surge of luma intensity by computing a sum of differences in histograms of intensity levels between a pair of consecutive actual video frames in the media content.

17. The apparatus of claim 15, wherein the first frame interpolation method is based on optical flow and the second frame interpolation method is not based on optical flow.

18. The apparatus of claim 17, wherein the second frame interpolation method is based on frame blending.

19. The apparatus of claim 17, wherein the particular video event causes distortion in a synthetic frame generated by the first frame interpolation method.

20. The apparatus of claim 15, wherein the particular video event is a sudden change in luma intensity.