Motion estimation for screen remoting scenarios

• US 10,390,039 B2
• Filed: 08/31/2016
• Issued: 08/20/2019
• Est. Priority Date: 08/31/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Zhu, Lihua, Kumar, B. Anil, Mases, Olof L. E.
• Primary Examiner(s): Wong, Allen C
• Application Number:
  US15/253,568
• Publication Number:
  US 20180063540A1
• Time in Patent Office: 1,084 Days
• Field of Search: 37524016
• CPC Class Codes: H04N 19/54 using feature points or meshes

Abstract

Innovations in motion estimation adapted for screen remoting scenarios are described herein. For example, as part of motion estimation for a current picture, a video encoder finds a pivot point in the current picture, calculates a hash value for the pivot point, and searches for a matching area in a previous picture. In doing so, the video encoder can calculate a hash index from the hash value and look up the hash index in a data structure to find candidate pivot points in the previous picture. The video encoder can compare the hash value for the pivot point in the current picture to a hash value for a candidate pivot point in the previous picture and, when the hash values match, compare sample values around the respective pivot points. In this way, the video encoder can quickly detect large areas of exact-match blocks having uniform motion.

Claims

1. A computer system comprising:

an input buffer configured to receive multiple pictures in a video sequence;

a video encoder configured to perform encoding of the multiple pictures to produce encoded data, wherein the encoding includes performing motion estimation for a current picture of the multiple pictures, the motion estimation for the current picture including;

finding a pivot point in the current picture;

calculating a hash value for the pivot point in the current picture; and

searching for a matching area in a previous picture based at least in part on the hash value for the pivot point in the current picture; and

an output buffer configured to store the encoded data for output as part of a bitstream.

2. The computer system of claim 1, wherein the motion estimation for the current picture further includes calculating multiple derivative sample values for the current picture based on base sample values for the current picture, the derivative sample values being used to find the pivot point in the current picture and to calculate the hash value for the pivot point in the current picture.

3. The computer system of claim 2, wherein a given derivative sample value, among the multiple derivative sample values, is calculated by combining multiple bits of a base luma sample value with at least one bit of a first base chroma sample value and at least one bit of a second base chroma sample value.

4. The computer system of claim 1, wherein the finding the pivot point in the current picture includes comparing sample values for the current picture to one or more patterns, each of the one or more patterns being indicative of an edge or character.

5. The computer system of claim 1, wherein the calculating the hash value uses a hashing function, and wherein the hashing function is a Cantor pairing function.

6. The computer system of claim 1, wherein the searching for the matching area includes:

calculating a hash index from the hash value for the pivot point in the current picture;

looking up the hash index in a data structure to find a list of one or more candidate pivot points in the previous picture; and

for each of at least one of the one or more candidate pivot points, comparing the hash value for the pivot point in the current picture to a hash value for the candidate pivot point.

7. The computer system of claim 6, wherein the list includes, for each of the one or more candidate pivot points, a location in the previous picture and the hash value for the candidate pivot point.

8. The computer system of claim 6, wherein the searching for the matching area further includes:

when the hash value for the pivot point in the current picture matches the hash value for a given candidate pivot point among the one or more candidate pivot points, comparing multiple sample values in an area around the pivot point in the current picture with corresponding sample values around the given candidate pivot point in the previous picture.

9. The computer system of claim 8, wherein the searching for the matching area further includes enlarging the area around the pivot point in the current picture until a stop condition occurs.

10. The computer system of claim 9, wherein the stop condition is failure to match between the sample values in the area around the pivot point in the current picture and the corresponding sample values around the given candidate pivot point in the previous picture.

11. The computer system of claim 8, wherein the pivot point in the current picture is a first pivot point, and wherein the searching for the matching area further includes:

checking whether the area around the first pivot point in the current picture overlaps a second pivot point in the current picture; and

if so, discarding the first pivot point or the second pivot point.

12. The computer system of claim 1, wherein a data structure used in the motion estimation for the current picture includes one or more lists each having one or more candidate pivot points in the previous picture, and wherein the encoding further includes updating the data structure by performing one or more of:

retaining at least one of the one or more candidate pivot points in the previous picture;

removing at least one of the one or more candidate pivot points in the previous picture; and

adding at least one pivot point in the current picture.

13. The computer system of claim 1, wherein the motion estimation for the current picture further includes identifying one or more changed regions in the current picture relative to the previous picture, and wherein the finding the pivot point in the current picture evaluates only sample values for the one or more changed regions in the current picture.

14. The computer system of claim 13, wherein a data structure used in the motion estimation for the current picture includes one or more lists each having one or more candidate pivot points in the previous picture, and wherein the encoding further includes updating the data structure by performing one or more of:

retaining any of the candidate pivot points in the previous picture that is outside the one or more changed regions;

removing any of the candidate pivot points in the previous picture that is inside the one or more changed regions; and

adding at least one pivot point in the current picture, the at least one pivot point in the current picture being inside the one or more changed regions.

15. The computer system of claim 1, wherein the motion estimation for the current picture further includes aggregating local motion information for multiple smaller areas into global motion metadata for a larger area that includes the multiple smaller areas.

16. The computer system of claim 15, wherein the motion estimation for the current picture further includes using the global motion metadata to skip block-based motion estimation operations for multiple partitions of the current picture, and wherein the using the global motion metadata includes assigning motion vectors for the multiple partitions based on the global motion metadata.

17. The computer system of claim 15, wherein the encoding further includes:

setting syntax elements based on the global motion metadata; and

signaling the syntax elements as part of the bitstream.

18. The computer system of claim 17, further comprising:

a video decoder configured to perform decoding of the multiple pictures, wherein the decoding includes;

parsing the syntax elements from the bitstream;

determining the global motion metadata from the syntax elements;

assigning motion vectors for multiple partitions of the current picture based on the global motion metadata; and

performing motion compensation for the multiple partitions of the current picture.

19. A computer-implemented method comprising:

receiving multiple pictures in a video sequence;

encoding the multiple pictures to produce encoded data, wherein the encoding includes performing motion estimation for a current picture of the multiple pictures, the motion estimation for the current picture including;

finding a pivot point in the current picture;

calculating a hash value for the pivot point in the current picture; and

searching for a matching area in a previous picture based at least in part on the hash value for the pivot point in the current picture; and

outputting the encoded data as part of a bitstream.

20. The method of claim 19, wherein the motion estimation for the current picture further includes calculating multiple derivative sample values for the current picture based on base sample values for the current picture, the derivative sample values being used to find the pivot point in the current picture and to calculate the hash value for the pivot point in the current picture.

21. The method of claim 19, wherein the finding the pivot point in the current picture includes comparing sample values for the current picture to one or more patterns, each of the one or more patterns being indicative of an edge or character.

22. The method of claim 19, wherein the searching for the matching area includes:

calculating a hash index from the hash value for the pivot point in the current picture;

looking up the hash index in a data structure to find a list of one or more candidate pivot points in the previous picture; and

for each of at least one of the one or more candidate pivot points, comparing the hash value for the pivot point in the current picture to a hash value for the candidate pivot point.

23. The method of claim 22, wherein the list includes, for each of the one or more candidate pivot points, a location in the previous picture and the hash value for the candidate pivot point.

24. The method of claim 22, wherein the searching for the matching area further includes:

when the hash value for the pivot point in the current picture matches the hash value for a given candidate pivot point among the one or more candidate pivot points, comparing multiple sample values in an area around the pivot point in the current picture with corresponding sample values around the given candidate pivot point in the previous picture.

25. The method of claim 24, wherein the searching for the matching area further includes enlarging the area around the pivot point in the current picture until a stop condition occurs.

26. The method of claim 24, wherein the pivot point in the current picture is a first pivot point, and wherein the searching for the matching area further includes:

checking whether the area around the first pivot point in the current picture overlaps a second pivot point in the current picture; and

if so, discarding the first pivot point or the second pivot point.

27. The method of claim 19, wherein a data structure used in the motion estimation for the current picture includes one or more lists each having one or more candidate pivot points in the previous picture, and wherein the encoding further includes updating the data structure by performing one or more of:

retaining at least one of the one or more candidate pivot points in the previous picture;

removing at least one of the one or more candidate pivot points in the previous picture; and

adding at least one pivot point in the current picture.

28. The method of claim 19, wherein the motion estimation for the current picture further includes identifying one or more changed regions in the current picture relative to the previous picture, and wherein the finding the pivot point in the current picture evaluates only sample values for the one or more changed regions in the current picture.

29. The method of claim 19, wherein the motion estimation for the current picture further includes aggregating local motion information for multiple smaller areas into global motion metadata for a larger area that includes the multiple smaller areas.

30. One or more computer-readable media storing computer-executable instructions for causing a computer system, when programmed thereby, to perform operations comprising:

receiving multiple pictures in a sequence;

encoding the multiple pictures to produce encoded data, wherein the encoding includes performing motion estimation for a current picture of the multiple pictures, the motion estimation for the current picture including;

finding a pivot point in the current picture;

calculating a hash value for the pivot point in the current picture; and

searching for a matching area in a previous picture based at least in part on the hash value for the pivot point in the current picture; and

outputting the encoded data as part of a bitstream.

31. The one or more computer-readable media of claim 30, wherein the motion estimation for the current picture further includes calculating multiple derivative sample values for the current picture based on base sample values for the current picture, the derivative sample values being used to find the pivot point in the current picture and to calculate the hash value for the pivot point in the current picture.

32. The one or more computer-readable media of claim 30, wherein the finding the pivot point in the current picture includes comparing sample values for the current picture to one or more patterns, each of the one or more patterns being indicative of an edge or character.

33. The one or more computer-readable media of claim 30, wherein the searching for the matching area includes:

calculating a hash index from the hash value for the pivot point in the current picture;

looking up the hash index in a data structure to find a list of one or more candidate pivot points in the previous picture; and

for each of at least one of the one or more candidate pivot points, comparing the hash value for the pivot point in the current picture to a hash value for the candidate pivot point.

34. The one or more computer-readable media of claim 33, wherein the list includes, for each of the one or more candidate pivot points, a location in the previous picture and the hash value for the candidate pivot point.

35. The one or more computer-readable media of claim 33, wherein the searching for the matching area further includes:

when the hash value for the pivot point in the current picture matches the hash value for a given candidate pivot point among the one or more candidate pivot points, comparing multiple sample values in an area around the pivot point in the current picture with corresponding sample values around the given candidate pivot point in the previous picture.

36. The one or more computer-readable media of claim 35, wherein the searching for the matching area further includes enlarging the area around the pivot point in the current picture until a stop condition occurs.

37. The one or more computer-readable media of claim 35, wherein the pivot point in the current picture is a first pivot point, and wherein the searching for the matching area further includes:

checking whether the area around the first pivot point in the current picture overlaps a second pivot point in the current picture; and

if so, discarding the first pivot point or the second pivot point.

38. The one or more computer-readable media of claim 30, wherein a data structure used in the motion estimation for the current picture includes one or more lists each having one or more candidate pivot points in the previous picture, and wherein the encoding further includes updating the data structure by performing one or more of:

retaining at least one of the one or more candidate pivot points in the previous picture;

removing at least one of the one or more candidate pivot points in the previous picture; and

adding at least one pivot point in the current picture.

39. The one or more computer-readable media of claim 30, wherein the motion estimation for the current picture further includes identifying one or more changed regions in the current picture relative to the previous picture, and wherein the finding the pivot point in the current picture evaluates only sample values for the one or more changed regions in the current picture.

40. The one or more computer-readable media of claim 30, wherein the motion estimation for the current picture further includes aggregating local motion information for multiple smaller areas into global motion metadata for a larger area that includes the multiple smaller areas.