Systems and methods to control judder visibility

• US 10,368,031 B2
• Filed: 02/23/2015
• Issued: 07/30/2019
• Est. Priority Date: 02/27/2014
• Status: Active Grant

• Current Assignee: Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Original Assignee: Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Inventors: Xu, Ning, Thurston, III, Kimball Darr, Daly, Scott, Crenshaw, James E., She, Alfred
• Primary Examiner(s): Chowdhury, Nigar
• Application Number:
  US15/119,117
• Publication Number:
  US 20170054938A1
• Time in Patent Office: 1,618 Days
• Field of Search: 348701, 348702
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: H04N 21/440281 by altering the temporal resolution, e.g. by frame skipping television signal recording using magnetic recording on tape for reproducing at a rate different from the recording rate H04N5/783
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51
• CPC Class Codes: H04N 5/21 Circuitry for suppressing or minimising disturbance, e.g. moiré or halo
• CPC Class Codes: H04N 5/262 Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
• CPC Class Codes: H04N 7/0112 one of the standards corresponding to a cinematograph film standard
• CPC Class Codes: H04N 7/0115 with details on the detection of a particular field or frame pattern in the incoming video signal, e.g. 3:2 pull-down pattern
• CPC Class Codes: H04N 7/0127 by changing the field or frame frequency of the incoming video signal, e.g. frame rate converter
• CPC Class Codes: H04N 7/0132 the field or frame frequency of the incoming video signal being multiplied by a positive integer, e.g. for flicker reduction
• CPC Class Codes: H04N 7/0135 involving interpolation processes interpolation-based image scaling G06T3/4007; interpolation for video coding H04N19/587, H04N19/59
• CPC Class Codes: H04N 7/014 involving the use of motion vectors motion estimation and compensation in video coding H04N19/51
• CPC Class Codes: H04N 7/0147 the interpolation using an indication of film mode or an indication of a specific pattern, e.g. 3:2 pull-down pattern

Abstract

Methods and systems for controlling judder are disclosed. Judder can be introduced locally within a picture, to restore a judder feeling which is normally expected in films. Capture time and display time of frames can be manipulated to obtain a desired amount of judder. Frames can be interpolated to obtain a film with a higher frame rate and localized control of judder.

Claims

1. A method to control judder visibility, the method comprising:

providing, by a computer, at least two input frames comprising a first frame and a second frame;

estimating, by a computer, an interpolation map based on the at least two input frames, thereby obtaining an estimated interpolation map, wherein the estimated interpolation map specifies a temporal interpolation position for at least one pixel of the at least two input frames;

interpolating, by a computer, at least one additional frame, based on the at least two input frames and the estimated interpolation map, thereby obtaining at least one interpolated frame, wherein at least one pixel of the at least one interpolated frame corresponds to at least one pixel at the time specified by the interpolation map of that at least one pixel; and

identifying, by a computer, for the at least one pixel of the at least two input frames, an amount of judder, wherein the estimating, by a computer, an interpolation map is based also on the amount of judder, and wherein identifying an amount of judder comprises;

generating a luminance change map and a contrast change map using the first frame and the second frame by calculating temporal differences between the first frame and the second frame for luminance and contrast; and

generating a judder map using the luminance change map and the contrast change map, wherein the judder map comprises judder information for the first frame.

2. The method of claim 1, further comprising providing, by a computer, user input on a desired amount of judder for at least one pixel of the at least two input frames, and wherein the estimating, by a computer, an interpolation map is based also on the user input.

3. The method of claim 1, further comprising applying, by a computer, motion blur to the at least one interpolated frame.

4. The method of claim 1, wherein at least one input frame is inserted in an output stream without modification.

5. The method of claim 1, further comprising:

identifying, by a computer, at least one region of the at least two input frames, the at least one region having an amount of judder higher than a desired amount of judder, wherein the interpolating is applied only to the at least one region.

6. The method of claim 5, wherein the at least one region is updated, by a computer, to a higher frame rate than a frame rate of the at least two input frame.

7. The method of claim 1, wherein the estimating, by a computer, an interpolation map comprises estimating a forward motion vector, a backward motion vector, a temporal motion consistency, a spatial motion consistency, a forward motion consistency and a backward motion consistency.

8. The method of claim 1, further comprising:

determining a motion speed of at least one pixel, wherein the relationship between the motion speed of a pixel and its interpolation position is a soft switching function between two speed thresholds, and wherein the estimating, by a computer, an interpolation map is based on the determined motion speed of the at least one pixel.

9. The method of claim 1, wherein the interpolating is based on frame rate conversion or based on motion compensation.

10. The method of claim 1, where a normalized measure of judder is pre-computed and delivered with an original content, such that a target device can decide how to adjust judder at a later time.

11. The method of claim 1, wherein:

an output frame rate is an integer number of an input frame rate, and

the at least one additional frame is interpolated between the at least two input frames.

12. The method of claim 11, further comprising identifying, by a computer, for the at least one pixel of the at least two input frames, an amount of judder, and wherein the estimating, by a computer, an interpolation map is based also on the amount of judder.

13. The method of claim 11, further comprising providing, by a computer, user input on a desired amount of judder for at least one pixel of the at least two input frames, and wherein the estimating, by a computer, an interpolation map is based also on the user input.

14. The method of claim 1, further comprising:

calculating a base frame rate, the base frame rate being an even number of an output frame rate and substantially close in value to an input frame rate, thereby displaying judder similarly to the input frame rate, wherein an output frame rate is a non integer number of the input frame rate; and

interpolating, by a computer, at least two additional frames, based on the at least two input frames, the base frame rate, the input frame rate, and output frame rare and the estimated interpolation map, thereby obtaining at least two interpolated frames, wherein at least one pixel of at least one of the at least two interpolated frames corresponds to at least one pixel at the time specified by the interpolation map of that at least one pixel, wherein the at least two additional frames are interpolated between the at least two input frames.

15. The method of claim 14, further comprising identifying, by a computer, for the at least one pixel of the at least two input frames, an amount of judder, and wherein the estimating, by a computer, an interpolation map is based also on the amount of judder.