Graphical application integration with MPEG objects

• US 9,123,084 B2
• Filed: 04/12/2012
• Issued: 09/01/2015
• Est. Priority Date: 04/12/2012
• Status: Active Grant

• Current Assignee: ActiveVideo Networks, Inc. (CommScope Holding Company, Inc.)
• Original Assignee: ActiveVideo Networks, Inc. (CommScope Holding Company, Inc.)
• Inventors: Brockmann, Ronald A., Dev, Anuj, Gorter, Onne, Hiddink, Gerrit, Hoeben, Maarten
• Primary Examiner(s): Czekaj, Dave
• Assistant Examiner(s): Beck, Leron
• Application Number:
  US13/445,104
• Publication Number:
  US 20130272394A1
• Time in Patent Office: 1,237 Days
• Field of Search: 37524001-24029
• US Class Current: 1/1
• CPC Class Codes: G06T 9/00 Image coding bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00
• CPC Class Codes: H04N 19/107 between spatial and temporal predictive coding, e.g. picture refresh
• CPC Class Codes: H04N 19/154 Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion use of rate-distortion criteria H04N19/147
• CPC Class Codes: H04N 19/174 the region being a slice, e.g. a line of blocks or a group of blocks

Abstract

System and methods are provided to cache encoded graphical objects that may be subsequently combined with other encoded video data to form a data stream decodable by a client device according to a format specification. Paint instructions relating to a graphical object are sent from a layout engine to a rendering library. A shim intercepts these instructions and determines whether the graphical object already has been rendered and encoded. If so, a cached copy of the object is transmitted to the client device. If not, the shim transparently passes the instructions to the rendering library, and the object is rendered, encoded, and cached. Hash values are used for efficiency. Methods are disclosed to detect and cache animations, and to cut and splice cached objects into encoded video data.

Claims

1. A method of providing an image to a client device from an application execution environment having a layout engine that assembles graphical components into a graphical user interface screen for a graphical application, and a rendering library that renders graphical components into pixels, the method comprising:

receiving, from the layout engine, one or more paint instructions having parameters that pertain to a given graphical object;

computing a hash value based on the received one or more paint instructions;

when the hash value is contained within a cache memory, retrieving, from the cache memory, encoded audiovisual data that are uniquely associated with the hash value, and transmitting the retrieved audiovisual data to the client device; and

when the hash value is not contained within the cache memory,

forwarding the received one or more paint instructions to the rendering library for rendering the graphical object into pixels according to the one or more paint instructions,

encoding the rendered pixels into encoded audiovisual data,

storing the hash value and the encoded audiovisual data in the cache memory, wherein the hash value and the encoded audiovisual data are uniquely associated, and

transmitting the encoded audiovisual data to the client device.

2. The method of claim 1, wherein the client device is one of the group consisting of:

a television, a television set-top box, a tablet computer, a laptop computer, a desktop computer, and a smartphone.

3. The method according to claim 1, wherein the graphical application is one of the group consisting of:

a web browser and a menu interface.

4. The method according to claim 1, wherein the encoding comprises dividing the screen into blocks of pixels, the method further comprising:

after receiving the one or more paint instructions and before computing the hash value, determining the smallest rectangle consisting of whole blocks of pixels that surrounds the graphical object;

requesting that the layout engine repaint the smallest surrounding rectangle; and

receiving, from the layout engine, painting data that include at least one paint instruction having parameters that reflect the smallest surrounding rectangle, wherein computing the hash value is based on the painting data.

5. The method according to claim 1, further comprising:

determining that the hash value is contained within the cache memory by comparing the hash value to a stored hash value of a cached image that forms part of an animation.

6. A tangible device on which is stored non-transitory computer program code for providing an image to a client device from an application execution environment having a layout engine that assembles graphical components into a graphical user interface screen for a graphical application, and a rendering library that renders graphical components into pixels, the computer program code comprising:

program code for receiving, from the layout engine, one or more paint instructions having parameters that pertain to a given graphical object;

program code for computing a hash value based on the received one or more paint instructions;

program code for retrieving, from a cache memory, encoded audiovisual data that are uniquely associated with the hash value, and transmitting the retrieved audiovisual data to the client device when the hash value is contained within the cache memory; and

program code for;

forwarding the received one or more paint instructions to the rendering library for rendering the graphical object into pixels according to the one or more paint instructions,

encoding the rendered pixels into encoded audiovisual data,

storing the hash value and the encoded audiovisual data in the cache memory, wherein the hash value and the encoded audiovisual data are uniquely associated, and

transmitting the encoded audiovisual data to the client device, when the hash value is not contained within the cache memory.

7. The device according to claim 6, wherein the client device is one of the group consisting of:

a television, a television set-top box, a tablet computer, a laptop computer, a desktop computer, and a smartphone.

8. The device according to claim 6, wherein the graphical application is one of the group consisting of:

a web browser and a menu interface.

9. The device according to claim 6, wherein the program code for encoding comprises program code for dividing the screen into blocks of pixels, the computer program code further comprising:

program code for determining the smallest rectangle consisting of whole blocks of pixels that surrounds the graphical object after receiving the painting data and before computing the hash value;

program code for requesting that the layout engine repaint the smallest surrounding rectangle; and

program code for receiving, from the layout engine, painting data that include at least one paint instruction having parameters that reflect the smallest surrounding rectangle, wherein computing the hash value is based on the painting data.

10. The device according to claim 6, further comprising:

program code for determining that the hash value is contained within the cache memory by comparing the hash value to a stored hash value of a cached image that forms part of an animation.

11. The device according to claim 6, further comprising:

program code for receiving a current image into a computing processor;

program code for receiving a previously rendered image into the computer processor, the previously rendered image being uniquely associated with an encoded image in the cache memory;

program code for transmitting to the client device the cached, encoded image without encoding the current image when the current image and the previously rendered image are identical; and

program code for;

encoding the current image according to a predictive encoding scheme,

storing the encoded current image in the cache memory, and

transmitting to the client device the encoded current image when the current image and the previously rendered image are not identical but share at least a given minimum percentage of their pixels.

12. The device according to claim 11, wherein the predictive encoding scheme is an MPEG encoding scheme.

13. The device according to claim 11, wherein the previously rendered image was not rendered immediately previously to the current image.

14. The device according to claim 11, wherein the previously rendered image is uniquely associated with a predictively encoded image in the cache memory.

15. The device according to claim 11, further comprising program code for computing a hash value for each unique chain of images that forms an animation, the hash value being a function of all images in the chain of images and a screen displacement.

16. The device according to claim 6, further comprising program code for forming two encoded MPEG slices from data comprising a given encoded MPEG slice, each encoded MPEG slice comprising a sequence of encoded macro blocks, the program code comprising:

program code for locating, in the given MPEG slice, a location of a macro block that is encoded according to a variable length code;

program code for decoding the encoded macroblock according to the variable length code;

program code for altering a DC luma value in the decoded macroblock; and

program code for encoding the altered macroblock according to the variable length code,

wherein the first formed MPEG slice consists of the data of the given MPEG slice up to but not including the encoded macro block, and the second formed MPEG slice consists of the encoded macroblock and any subsequent encoded macroblocks in the given MPEG slice.

17. The device according to claim 6, further comprising program code for combining a first encoded MPEG slice and a second encoded MPEG slice to form a third encoded MPEG slice, each encoded MPEG slice comprising a sequence of encoded macro blocks, the program code comprising:

program code for decoding the first macro block of the second slice according to a variable length code;

program code for altering a DC luma value in the decoded macroblock;

program code for encoding the altered macroblock according to the variable length code; and

program code for concatenating the data of the first slice with the encoded macro block and the undecoded data of the second slice to form the third slice.

18. A system for providing an image to a client device from an application execution environment having a layout engine that assembles graphical components into a graphical user interface screen for a graphical application, and a rendering library that renders graphical components into pixels, the system comprising:

a memory;

a shim comprising hardware or a combination of hardware and software that is configured to;

receive, from the layout engine, one or more paint instructions having parameters that pertain to a given graphical object,

compute a hash value based on the received one or more paint instructions, and

when the hash value is not contained within the memory, forward the received one or more paint instructions to the rendering library for rendering the graphical object into pixels according to the one or more paint instructions; and

a controller comprising hardware or a combination of hardware and software that is configured to;

retrieve, from the memory, encoded audiovisual data that are uniquely associated with the hash value, and

transmit the retrieved audiovisual data to the client device when the hash value is contained within the memory; and

transmit, to the client device, encoded audiovisual data comprising a rendering of the graphical object into pixels according to the received one or more paint instructions when the hash value is not contained within the memory.

19. The system according to claim 18, wherein the client device is one of the group consisting of:

a television, a television set-top box, a tablet computer, a laptop computer, a desktop computer, and a smartphone.

20. The system according to claim 18, wherein the graphical application is one of the group consisting of:

a web browser and a menu interface.

21. The system according to claim 18, wherein the memory stores a sequence of images that collectively form an animation, and wherein the controller is further configured to determine that the hash value is contained within the memory by comparing the hash value to a stored hash value of a cached image that forms part of the animation.

22. The system according to claim 18, wherein the audiovisual data are encoded according to an MPEG encoding scheme.

23. The system according to claim 18, further comprising a block-based encoder that is configured to form two encoded MPEG slices from data comprising a given encoded MPEG slice, each encoded MPEG slice comprising a sequence of encoded macro blocks, by:

locating, in the given MPEG slice, a location of a macro block that is encoded according to a variable length code;

decoding the encoded macroblock according to the variable length code;

altering a DC luma value in the decoded macroblock; and

encoding the altered macroblock according to the variable length code,

wherein the first formed MPEG slice consists of the data of the given MPEG slice up to but not including the encoded macro block, and the second formed MPEG slice consists of the encoded macroblock and any subsequent encoded macroblocks in the given MPEG slice.

24. The system according to claim 18, further comprising a block-based encoder that is configured to combine a first encoded MPEG slice and a second encoded MPEG slice to form a third encoded MPEG slice, each encoded MPEG slice comprising a sequence of encoded macroblocks, by:

decoding the first macro block of the second slice according to a variable length code;

altering a DC luma value in the decoded macroblock;

P

encoding the altered macroblock according to the variable length code; and

concatenating the data of the first slice with the encoded macro block and the undecoded data of the second slice to form the third slice.