Method and system for remotely communicating a computer rendered image sequence

• US 8,520,734 B1
• Filed: 07/31/2009
• Issued: 08/27/2013
• Est. Priority Date: 07/31/2009
• Status: Active Grant

• Current Assignee: Teradici Corporation (HP Inc.)
• Original Assignee: Teradici Corporation (HP Inc.)
• Inventors: Xu, Zhan
• Primary Examiner(s): Kelley, Christopher S
• Assistant Examiner(s): VAZQUEZ COLON, MARIA E
• Application Number:
  US12/462,235
• Time in Patent Office: 1,488 Days
• Field of Search: 375/240.16, 375/240.01, 375/240.02
• US Class Current: 375/240.16
• CPC Class Codes: H04N 19/105 Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
• CPC Class Codes: H04N 19/139 Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/34 Scalability techniques involving progressive bit-plane based encoding of the enhancement layer, e.g. fine granular scalability [FGS]
• CPC Class Codes: H04N 19/517 by encoding

Abstract

A method and system for communicating a computer rendered image sequence from a host computer to a remote computer. The method comprises determining, at the host computer, while performing a progressive encoding of an image portion of the computer rendered image sequence, motion of the image portion, wherein the progressive encoding comprises generating a lossy encoding of a frequency transform of the image portion and a first refinement encoding of the frequency transform; generating, at the host computer, a motion vector representing the motion; and communicating, from the host computer to the remote computer, the lossy encoding, the first refinement encoding, and the motion vector.

Claims

1. A method of communicating an image sequence rendered by a host computer, the method comprising:

determining, by the host computer and after performing a progressive encoding of a constant image at a constant spatial location in the image sequence, a lossless motion of the constant image in the image sequence, wherein the progressive encoding comprises generating, in absence of a lossless match of the constant image to a previous image, a lossy encoded state of the constant image comprising a lossy encoding of a frequency transform of the constant image and a first refinement encoding of the frequency transform, wherein determining the lossless motion comprises detecting the constant image at a new spatial location in the image sequence, wherein the constant image comprises exact pixel value matches between frames of a period of the constant image;

generating, by the host computer, a lossless motion vector as an encoding of the lossless motion;

communicating, from the host computer to a remote computer, the lossy encoding, the first refinement encoding, and the lossless motion vector;

determining, subsequent to communicating the lossless motion vector, persistence of the constant image at the new spatial location; and

communicating, to the remote computer, a lossless re-encoding of the constant image at the new spatial location in response to the persistence and the lossy encoded state.

2. The method of claim 1, wherein the constant image at the constant spatial location and the constant image at the new spatial location comprise identical visual image data.

3. The method of claim 2, wherein determining the lossless motion of the constant image comprises identifying, subsequent to generating the lossy encoding, the new spatial location based on comparing sequential image frames of the image sequence.

4. The method of claim 1, wherein the lossless motion vector identifies to a decoder a perfect pixel value match between the constant image at the new spatial location and the constant image at the constant spatial location.

5. The method of claim 1, wherein communicating the lossless re-encoding comprises prioritizing an encoding of a subset of pixels within the constant image based on an accumulated image quality of a decoding of communicated data of the constant image and a duration of the persistence.

6. The method of claim 1, further comprising communicating, from the host computer to the remote computer, subsequent to determining the lossless motion, at least one additional refinement encoding of the frequency transform, the at least one additional refinement encoding to be decoded at the remote computer prior to decoding the lossless motion vector at the remote computer.

7. The method of claim 1 further comprising maintaining first and second quality state information for a first and a second spatial section, respectively, of a frame buffer associated with the image sequence, wherein (i) the first spatial section comprises a first pixel of the constant image prior to the determining the lossless motion, (ii) the second spatial section comprises a second pixel of the constant image at the new spatial location associated with the first pixel subsequent to the lossless motion, and (iii) the second quality state information, subsequent to the lossless motion, comprises a portion of the first quality state information prior to the lossless motion.

8. The method of claim 7 wherein, subsequent to the lossless motion, the second quality state information comprises transmitted encoded quality levels and an image source spatial location for the first pixel prior to the lossless motion.

9. The method of claim 1 further comprising:

determining, by the host computer, a change to a pixel value of a pixel associated with the new spatial location subsequent to the lossless motion, the change exclusive of changes resulting from the lossless motion; and

communicating, from the host computer to the remote computer, an encoding of a second image portion of the image sequence, wherein the second image portion comprises the pixel.

10. The method of claim 1 further comprising reviewing, by the host computer, a first, a second, and a third period of rendering of the image sequence for determining (i) changed image content, resulting in the constant image, for generating the lossy encoding, (ii) persistence of the constant image and the constant spatial location of the constant image within the image sequence for generating the first refinement encoding, and (iii) persistence of the constant image and the new spatial location of the constant image within the image sequence for determining the lossless motion, respectively.

11. The method of claim 10, wherein reviewing the first, the second, and the third period comprises at least one of analyzing a respective sequence of rendering operations used to render the image sequence or analyzing a respective image frame sequence of the image sequence.

12. The method claim of 10, wherein each of the first, the second, and the third periods comprises at least one of a defined period of time or at least a period of time necessary to communicate a respective previous encoding of the image sequence.

13. The method of claim 1, wherein the image sequence comprises a sequence of images in a format for display by a computer display.

14. The method of claim 1, further comprising:

evaluating, by the host computer, subsequent to determining the lossless motion and prior to communicating the lossless motion vector, a criteria for performing a refinement encoding of at least a subset of the constant image; and

performing, based on the evaluating, one of (i) communicating, from the host computer to the remote computer, a refinement encoding of a frequency transform of the at least a subset of the constant image prior to communicating the lossless motion vector, (ii) communicating, from the host computer to the remote computer, the refinement encoding in conjunction with communicating the lossless motion vector, or (iii) communicating the lossless motion vector.

19. The method of claim 1, further comprising starting a new progressive encoding of a second portion of the image sequence based on an absence of a previously encoded lossless copy of the second portion, wherein the lossless re-encoding is a lossless encoding of the constant image at the new spatial location.

15. A method of communicating a computer rendered image sequence from a host computer to a remote computer, comprising:

(a) determining, by the host computer, a portion, of constant shape and lossless persistent image content, of a period of the computer rendered image sequence, wherein at least one lossless movement of the portion occurs between image frames of the period, wherein determining the at least one lossless movement comprises detecting a lossless copy of the portion at a new spatial location of a subsequent frame in the computer rendered image sequence, wherein the portion comprises exact pixel value matches between frames of the computer rendered image sequence during the period;

(b) communicating, from the host computer to the remote computer, during the period, a lossy encoding of the portion and at least one progressive refinement of the portion, prior to the at least one lossless movement, wherein the lossy encoding and the at least one progressive refinement comprise respective initial and subsequent layers of a frequency transformation of the portion;

(c) determining, by the host computer, during the period, one of (i) a stationary location of the portion for a plurality of frames or (ii) a lossless movement of the portion;

(d) generating, by the host computer, compressed data comprising one of (i) a refinement of the portion in response to determining the stationary location and to determining an incomplete refinement of an encoding of the portion, the incomplete refinement determined from an encoded state record of the portion, or (ii) at least one lossless motion vector indicating the lossless movement in response to determining the lossless movement;

(e) communicating, from the host computer to the remote computer, one of (i) first compressed data comprising refinements of the portion in response to determining the stationary location, or (ii) second compressed data comprising at least one lossless motion vector representing the lossless movement in response to determining the lossless movement; and

(f) repeating steps (c)-(e) while the period persists and until communicating the first compressed data after determining the lossless movement.

16. An encoder system at a host computer for encoding a computer rendered image sequence, the encoder system comprising:

a progressive image encoder enabled to (i) generate encoded refinements of a frequency transform of a section of the computer rendered image sequence while detecting the section remains pixel value constant and spatially static, and (ii) communicate the encoded refinements to a remote computer, wherein the remote computer is enabled for coupling to the host computer by a packet switched network;

a motion compensation encoder enabled to (i) detect lossless motion, and generate lossless motion vectors, for the section during a lossless motion period wherein the section remains pixel value constant and spatially moving, wherein the motion compensation encoder determines that the section remains pixel value constant and spatially moving by detecting a lossless copy of the section at a new spatial location of a new frame in the computer rendered image, and (ii) communicate the lossless motion vectors to the remote computer; and

a storage for storing encoding information for the section, wherein the progressive image encoder is further enabled to generate an additional encoding of the section, decodable to generate a lossless decoding of the section and subsequent to the lossless motion period followed by a period of frames of the computer rendered image sequence where the section remains pixel value constant.

17. The encoder system of claim 16, wherein the additional encoding comprises at least one of a residual encoding, an initial encoding, or an encoding of additional frequency transform refinements.

18. The encoder system of claim 16, wherein the progressive image encoder is further enabled to (i) generate decoded pixel values by decoding a combination of the encoded refinements and the lossless motion vectors, (ii) generate an encoded difference between decoded pixel values and source pixel values of the section, and (iii) communicate the encoded difference to the remote computer.