Video signal interpolation using motion estimation

• US 4,383,272 A
• Filed: 04/13/1981
• Issued: 05/10/1983
• Est. Priority Date: 04/13/1981
• Status: Expired due to Term

• Current Assignee: Multimedia Patent Trust
• Original Assignee: Bell Telephone Laboratories, Inc. (Nokia Corporation)
• Inventors: Netravali, Arun N., Robbins, John D.
• Primary Examiner(s): Safourek, Benedict V.
• Assistant Examiner(s): Coles, Edward L.
• Application Number:
  US06/253,698
• Time in Patent Office: 757 Days
• Field of Search: 358/136, 358/133, 358/105, 358/138, 375/28, 364/518, 364/521
• US Class Current: 348/416.1
• CPC Class Codes: H04N 19/577 Motion compensation with bidirectional frame interpolation, i.e. using B-pictures

Abstract

Information defining elements of a picture is estimated by interpolation using information from related locations in preceding and succeeding versions of the picture. The related locations are determined by forming an estimate of the displacement of objects in the picture. Displacement estimates are advantageously formed recursively, with updates being formed only in moving areas of the picture. If desired, an adaptive technique can be used to permit motion compensated interpolation or fixed position interpolation, depending upon which produces better results.

Claims

1. Apparatus for estimating the intensities of elements (pels) in a picture in accordance with information defining intensities of pels in preceding and succeeding versions of the picture including means for determining by interpolation intensities of pels in said picture in accordance with intensities of pels in related locations in said preceding and succeeding versions,

characterized in that

said determining means includes means for selecting said related locations as a function of the displacement of objects in said picture.

2. The invention defined in claim 1 wherein said apparatus includes:

means for storing a present estimate Dⁱ of said displacement, and

means for recursively updating said estimate for each element in said picture.

3. The invention defined in claim 2 wherein said apparatus includes means for operating said updating means only in moving areas in said picture.

4. The invention defined in claim 3 wherein said apparatus further includes:

means for computing a frame difference FD(x) indicating the intensity difference at spatially corresponding locations in preceding and succeeding versions, and

means for computing a displaced frame difference [DFD(x,D)] indicating the intensity difference at the related locations determined by said displacement estimate,

wherein said selecting means is arranged to select said displaced locations if said displaced frame difference is smaller than said frame difference and to select said corresponding locations otherwise.

5. The invention defined in claim 1 wherein said apparatus further includes:

means for storing the intensity values for pels in said preceding and succeeding versions, and

means responsive to said present displacement estimate for addressing selected ones of said stored values.

6. Apparatus for estimating the intensity values of each element (pel) of a picture being processed by interpolating between the intensity values of related pels in first and second other versions of said picture, including:

means for estimating the displacement of objects in said picture occurring between said other versions, and

means for selecting said related pels in accordance with said displacement estimate.

7. The invention defined in claim 6 wherein said first and second other versions occur at intervals K₁ τ

before and K₂ τ

after said picture being processed, where K₁ and K₂ are positive integers and τ

is a predetermined constant, and wherein said related pels are at displaced locations x-K₁ D and x+K₂ D in said first and second versions, respectively, where x is the vector location of the pel in said presently processed picture and D is the vector representing said displacement estimate per time τ

.

8. The invention defined in claim 7 wherein said displacement estimate is recursively updated such that an update term is added to each estimate to form the next estimate, where said update term is a function of the intensity difference at said displaced locations.

9. The invention defined in claim 8 wherein said apparatus further includes means for comparing said intensity difference at said displaced location with the intensity difference at the same location x in said other versions, and

means for operating said selecting means only if said displaced location intensity difference is smaller than said same location intensity difference.

10. Apparatus for reducing the bandwidth needed to transmit a video signal representing a sequence of pictures by encoding the intensity values of pels in ones of said pictures in said sequence and reconstructing missing pictures using information from encoded pictures, including:

means for computing the intensity of pels in a missing picture by interpolating the intensity of pels in corresponding locations in the encoded ones of said pictures which precede and follow said missing picture, and

means for selecting said corresponding locations as a function of the displacement of objects in said picture between said preceding and following pictures.

11. The invention defined in claim 10 further including:

means for storing an estimate Dⁱ of said displacement, and

means for recursively updating said estimate to form a new estimate Dⁱ⁺¹ by adding a correction term which is a joint function of (a) the intensity difference at said corresponding location, and (b) the spatial gradient of said intensity difference.

12. The invention defined in claim 11 wherein said apparatus further includes:

means for storing the intensity values of pels in said preceding and following pictures, and

means for addressing said stored values in accordance with Dⁱ to obtain the intensities at said corresponding locations.

13. A method of estimating the intensities of elements (pels) in a picture in accordance with information defining intensities of pels in preceding and succeeding versions of the picture including the step of determining by interpolation intensities of pels in said picture in accordance with intensities of pels in related locations in said preceding and succeeding versions,

characterized in that

said determining step includes selecting said related locations as a function of the displacement of objects in said picture.

14. The method defined in claim 13 further including the steps of:

storing a present estimate Dⁱ of said displacement, and

recursively updating said estimate for each element in said picture.

15. The method defined in claim 14 further including the step of operating said updating means only in moving areas in said picture.

16. The method defined in claim 15 further including the steps of:

computing a frame difference FD(x) indicating the intensity difference at spatially corresponding locations in preceding and succeeding versions, and

computing a displaced frame difference [DFD(x,D)] indicating the intensity difference at the related locations determined by said displacement estimate,

wherein said selecting step includes selecting said displaced locations if said displaced frame difference is smaller than said frame difference and selecting said corresponding locations otherwise.

17. The method defined in claim 13 wherein said determining step further includes:

storing the intensity values for pels in said preceding and succeeding versions, and addressing selected ones of said stored values in response to said present displacement estimate.

18. A method of estimating the intensity values of each element (pel) of a picture being processed by interpolating between the intensity values of related pels in first and second other versions of said picture, including the steps of

estimating the displacement of objects in said picture occurring between said other versions, and

selecting said related pels in accordance with said displacement estimate.

19. The method defined in claim 18 wherein said first and second other versions occur at intervals K₁ τ

before and K₂ τ

after said picture being processed, where K₁ and K₂ are positive integers and τ

is a predetermined constant, and wherein said related pels are at displaced locations x-K₁ D and x+K₂ D in said first and second versions, respectively, where x is the vector location of the pel in said presently processed picture and D is the vector representing said displacement estimate per time τ

.

20. The method defined in claim 19 wherein said displacement estimating step includes recursive updating such that an update term is added to each estimate to form the next estimate, where said update term is a function of the intensity difference at said displaced locations.

21. The method defined in claim 20 further including the steps of comparing said intensity difference at said displaced location with the intensity difference at the same location x in said other versions, and

precluding said selecting step if said displaced location intensity difference is larger than said same location intensity difference.

22. A method of reducing the bandwidth needed to transmit a video signal representing a sequence of pictures by encoding the intensity values of pels in ones of said pictures in said sequence and reconstructing missing pictures using information from encoded pictures, including:

computing the intensity of pels in a missing picture by interpolating the intensity of pels in corresponding locations in the encoded ones of said pictures which precede and follow said missing picture, and

selecting said corresponding locations as a function of the displacement of objects in said picture between said preceding and following pictures.

23. The method defined in claim 22 further including the steps of:

storing an estimate Dⁱ of said displacement, and

recursively updating said estimate to form a new estimate Dⁱ⁺¹ by adding a correction term which is a joint function of (a) the intensity difference at said corresponding location, and (b) the spatial gradient of said intensity difference.

24. The method defined in claim 23 further including the steps of:

storing the intensity values of pels in said preceding and following pictures, and

addressing said stored values in accordance with Dⁱ to obtain the intensities at said corresponding locations.