Updating an image

• US 8,754,827 B2
• Filed: 09/21/2010
• Issued: 06/17/2014
• Est. Priority Date: 06/30/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Skype S.Ã R.L. (Microsoft Corporation)
• Inventors: Braghis, Florin, Lindblom, Jonas, Ledin, Renars
• Primary Examiner(s): Abdulselam, Abbas
• Assistant Examiner(s): STEPP JONES, SHAWNA T
• Application Number:
  US12/924,133
• Publication Number:
  US 20120001832A1
• Time in Patent Office: 1,365 Days
• Field of Search: 345/2.2, 345/1.3
• US Class Current: 345/2.1
• CPC Class Codes: G06F 3/1454 involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
• CPC Class Codes: G06F 3/1462 with means for detecting differences between the image stored in the host and the images displayed on the remote displays
• CPC Class Codes: G06T 9/00 Image coding bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00
• CPC Class Codes: G09G 2310/04 Partial updating of the display screen
• CPC Class Codes: G09G 2340/02 Handling of images in compressed format, e.g. JPEG, MPEG
• CPC Class Codes: G09G 2360/122 Tiling
• CPC Class Codes: H04L 12/1822 Conducting the conference, e.g. admission, detection, selection or grouping of participants, correlating users to one or more conference sessions, prioritising transmission
• CPC Class Codes: H04N 19/119 Adaptive subdivision aspects, e.g. subdivision of a picture into rectangular or non-rectangular coding blocks
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/146 Data rate or code amount at the encoder output
• CPC Class Codes: H04N 19/147 according to rate distortion criteria rate-distortion as a criterion for motion estimation H04N19/567
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/507 using conditional replenishment
• CPC Class Codes: H04N 19/60 using transform coding

Abstract

Method, terminal, network and computer program product for processing updates of an image for transmission from a first terminal to a second terminal for use in screen sharing between the first terminal and the second terminal wherein updates to the image displayed on a first display at the first terminal are also displayed on a second display at the second terminal. The image is divided into an array of contiguous sub-tiles, each sub-tile comprising more than one picture element of the image. It is determined that a plurality of sub-tiles of the image have changed at the first terminal and at least one of said changed sub-tiles and at least one contiguous sub-tile are selected to form a tile. The tile is encoded for transmission to the second terminal, said encoding determining sub-tile data identifying which sub-tiles are included in the tile such that the image can be updated at the second terminal in accordance with the changes to said changed sub-tiles, wherein the configuration of the tile is flexible such that the number of contiguous sub-tiles encoded in the tile can be varied.

Claims

1. A method of processing updates of an image for transmission from a first terminal to a second terminal for use in screen sharing between the first terminal and the second terminal wherein updates to the image displayed on a first display at the first terminal are also displayed on a second display at the second terminal, the image being divided into an array of contiguous sub-tiles, each sub-tile comprising more than one picture element of the image, the method comprising:

determining that a plurality of sub-tiles of the image have changed at the first terminal;

selecting at least one of said changed sub-tiles and at least one contiguous sub-tile and forming a tile by;

assigning respective weights to the changed sub-tiles; and

using the weights in one of a clustering technique and a dimension reduction technique to determine the configuration of the tile; and

encoding the tile for transmission to the second terminal, said encoding determining sub-tile data identifying which sub-tiles are included in the tile such that the image can be updated at the second terminal in accordance with the changes to said changed sub-tiles,

wherein the configuration of the tile is flexible such that the number of contiguous sub-tiles encoded in the tile can be varied.

2. The method of claim 1 wherein said forming comprises forming another tile by determining the configuration of the another tile in dependence upon the arrangement of changed sub-tiles in another array.

3. The method of claim 2 wherein the configuration of the another tile is determined using an algorithm to optimize the encoding of the changed sub-tiles in the another tile.

4. The method of claim 2 wherein the configuration of the another tile is determined in dependence upon at least one of:

the complexity of encoding the another tile;

the total amount of data in the encoded another tile;

the tile header overhead of the encoded another tile; and

the compression efficiency of compressing the another tile.

5. The method of claim 1 wherein said forming a tile further comprises for another array of contiguous sub-tiles:

creating a preliminary tile for each contiguous area of said changed sub-tiles; and

grouping at least two of the preliminary tiles together to form said tile.

6. The method of claim 1 further comprising assigning weights to unchanged sub-tiles according to their proximity to said changed sub-tiles.

7. The method of claim 1 wherein the at least one contiguous sub-tile is encoded in the tile whether or not it has changed.

8. The method of claim 1 wherein said encoding the tile comprises compressing the tile.

9. The method of claim 8 wherein the tile is compressed using one of JPEG compression and PNG compression.

10. The method of claim 1 wherein said step of determining that a plurality of sub-tiles of the image have changed comprises one of:

determining that the sub-tiles have changed since the last time a tile was formed for the image; and

determining that the sub-tiles have changed since the last time a tile was transmitted to the second terminal.

11. The method of claim 1 further comprising transmitting the encoded tile from the first terminal directly to the second terminal.

12. The method of claim 1 further comprising transmitting the encoded tile from the first terminal to a server.

13. The method of claim 12 wherein the server either (i) transmits the encoded tile to the second terminal, or (ii) determines whether to transmit the encoded tile to the second terminal.

14. The method of claim 1 further comprising storing the encoded tile at the first terminal.

15. The method of claim 1 wherein there are a plurality of second terminals.

16. The method of claim 1 wherein the tile comprises a rectangular section of the image.

17. The method of claim 1 wherein each sub-tile comprises a block of 32 by 32 picture elements of the image.

18. The method of claim 1 wherein in at least some instances, where a plurality of contiguous sub-tiles have changed, they are encoded in a common tile.

19. The method of claim 1 wherein in at least some instances, where a plurality of sub-tiles in close proximity in the array have changed, they are encoded in a common tile.

20. The method of claim 1 wherein a plurality of tiles are formed and encoded for transmission to the second terminal.

21. A terminal configured to transmit updates of an image to a further terminal for use in screen sharing between the terminal and the further terminal wherein updates to the image displayed on a first display at the terminal are also displayed on a second display at the further terminal, the image being divided into an array of contiguous sub-tiles, each sub-tile comprising more than one picture element of the image, the terminal comprising:

a determiner configured to determine that a plurality of sub-tiles of the image have changed at the terminal;

a selecter configured to select at least one of said changed sub-tiles and at least one contiguous sub-tile to form a tile by;

assigning respective weights to changed sub-tiles; and

using the weights in one of a clustering technique and a dimension reduction technique to determine the configuration of the tile; and

an encoder configured to encode the tile for transmission to the second terminal, said encoder being configured to determine sub-tile data identifying which sub-tiles are included in the tile such that the image can be updated at the further terminal in accordance with the changes to said changed sub-tiles,

wherein the configuration of the tile is flexible such that the number of contiguous sub-tiles encoded in the tile by the encoder can be varied.

22. The terminal of claim 21 further comprising the first display for use in screen sharing between the terminal and the further terminal.

23. The terminal of claim 21 further comprising a transmitter configured to transmit the encoded tile from the terminal.

24. The terminal of claim 21 further comprising a store configured to store the encoded tile.

25. A network comprising:

a first terminal; and

a second terminal,

wherein the first terminal is configured to transmit updates of an image to the second terminal for use in screen sharing between the first terminal and the second terminal wherein updates to the image displayed on a first display at the first terminal are also displayed on a second display at the second terminal, the image being divided into an array of contiguous sub-tiles, each sub-tile comprising more than one picture element of the image, the first terminal comprising;

a determiner configured to determine that a plurality of sub-tiles of the image have changed at the first terminal;

a selecter configured to select at least one of said changed sub-tiles and at least one contiguous sub-tile to form a tile by;

assigning respective weights to the changed sub-tiles; and

using the weights in one of a clustering technique and a dimension reduction technique to determine the configuration of the tile; and

an encoder configured to encode the tile for transmission to the second terminal, said encoder being configured to determine sub-tile data identifying which sub-tiles are included in the tile such that the image can be updated at the second terminal in accordance with the changes to said changed sub-tiles,

wherein the configuration of the tile is flexible such that the number of contiguous sub-tiles encoded in the tile by the encoder can be varied, and wherein the second terminal comprises;

a receiver configured to receive the encoded tile; and

an updater configured to update the image at the second terminal in accordance with the changes to said changed sub-tiles.

26. The network of claim 25 wherein the second terminal further comprises a decoder configured to decode the encoded tile to thereby determine which sub-tiles are included in the tile using the sub-tile data.

27. The network of claim 25 wherein the second terminal is configured to execute a view client in a web browser to display the image at the second terminal.

28. The network of claim 25 further comprising a server configured to receive the encoded tile from the first terminal.

29. A computer system comprising:

a processor; and

a computer program product comprising stored computer readable instructions for execution by the processor at a first terminal for transmitting updates of an image from the first terminal to a second terminal for use in screen sharing between the first terminal and the second terminal wherein updates to the image displayed on a first display at the first terminal are also displayed on a second display at the second terminal, the image being divided into an array of contiguous sub-tiles, each sub-tile comprising more than one picture element of the image, the instructions comprising instructions to perform operations comprising;

determining that a plurality of sub-tiles of the image have changed at the first terminal;

selecting at least one of said changed sub-tiles and at least one contiguous sub-tile and forming a tile by;

assigning respective weights to changed sub-tiles; and

using the weights in one of a clustering technique and a dimension reduction technique to determine the configuration of the tile; and

encoding the tile for transmission to the second terminal, said encoding determining sub-tile data identifying which sub-tiles are included in the tile such that the image can be updated at the second terminal in accordance with the changes to said changed sub-tiles,

wherein the configuration of the tile is flexible such that the number of contiguous sub-tiles encoded in the tile can be varied.