System and method for transmitting video and user interface elements

• US 8,724,696 B2
• Filed: 09/23/2010
• Issued: 05/13/2014
• Est. Priority Date: 09/23/2010
• Status: Active Grant

• Current Assignee: Vmware LLC (Broadcom, Inc.)
• Original Assignee: VMware, Inc. (Broadcom, Inc.)
• Inventors: Cannon, Anthony, Byford, Dustin Michael, Langdale, Philip
• Primary Examiner(s): Ustaris, Joseph
• Assistant Examiner(s): NAWAZ, TALHA M
• Application Number:
  US12/889,090
• Publication Number:
  US 20120076197A1
• Time in Patent Office: 1,328 Days
• Field of Search: 375/240
• US Class Current: 375/240.01
• CPC Class Codes: H04N 19/00 Methods or arrangements for coding, decoding, compressing or decompressing digital video signals

Abstract

A method for offloading decoding of encoded video data from a computer system executing a video player application playing the encoded video data to a remote display terminal still enables the video player application to overlay transparent or semi-transparent graphical elements, such elements, such as user interface controls, onto the video, despite not having access to decoded video data. A video decoding function call from the video player application is intercepted and replacement video data is provided to the video player application rather than a decoded version of the encoded video data. The video player application is thereby able to overlay graphical elements onto the replacement video data to create composite video from which the graphical elements are then able to be subsequently extracted and transmitted with the encoded video data to remote display terminal, which decodes the video data and overlays the graphical elements for display.

Claims

1. A method for offloading decoding of encoded video data from a computer system executing a video player application to a remote display terminal, the method comprising:

intercepting a video decoding function call from the video player application requesting graphics hardware assistance from the computer system to decode the encoded video data, wherein the video player application is configured to decode encoded video data and composite graphical overlays onto decoded video data;

providing replacement video data to the video player application rather than a decoded version of the encoded video data in response to the video decoding function call;

receiving, by the computer system, composite video data composited by the video player application, wherein the composite video data comprises the provided replacement video data and a graphical overlay;

extracting, by the computer system, the graphical overlay from the received composite video data by subtracting the replacement video data from the composite video data; and

transmitting the encoded video data and the extracted graphical overlay from the computer system to the remote display terminal, wherein the remote display terminal decodes the encoded video data and composites the graphical overlay onto the decoded video data to generate a final composite video data for display.

2. The method of claim 1, wherein the video player application executes within a virtual machine and the remote display terminal receives graphical user interface displays from the virtual machine.

3. The method of claim 2, wherein the replacement video data is generated with the assistance of a virtual GPU.

4. The method of claim 2, wherein the receiving, extracting and transmitting steps are performed by a display protocol application running in the virtual machine and wherein the display protocol application is configured to transmit displays of the virtual machine to the remote display terminal in accordance with a display protocol.

5. The method of claim 4, wherein the composite video data composited by the video player application is transmitted by the video player application to a video adapter driver that is configured to transmit the composite video data to the display protocol application.

6. The method of claim 1, wherein the intercepting step is performed by a proxy library configured to intercept function calls transmitted to a pre-existing video rendering library.

7. The method of claim 1, wherein each video frame in the replacement video data comprises a graphical image, wherein each adjacent pixel of the graphical image alternates between one of two colors and each corresponding pixel in a prior graphical image of an immediately prior video frame in the replacement video data is the other of the two colors.

8. The method of claim 1, wherein pixels of the graphical overlay comprise alpha transparency values.

9. A computer system configured to offload decoding of encoded video data from a video player application on the computer system to a remote display terminal, the computer system comprising a processor configured to perform the steps of:

intercepting a video decoding function call from the video player application requesting graphics hardware assistance from the computer system to decode the encoded video data, wherein the video player application is configured to decode encoded video data and composite graphical overlays onto decoded video data;

providing replacement video data to the video player application rather than a decoded version of the encoded video data in response to the video decoding function call;

receiving, by the computer system, composite video data composited by the video player application, wherein the composite video data comprises the provided replacement video data and a graphical overlay;

extracting, by the computer system, the graphical overlay from the received composite video data by subtracting the replacement video data from the composite video data; and

transmitting the encoded video data and the extracted graphical overlay from the computer system to the remote display terminal, wherein the remote display terminal decodes the encoded video data and composites the graphical overlay onto the decoded video data to generate a final composite video data for display.

10. The computer system of claim 9, wherein the video player application executes within a virtual machine on the computer system and the remote display terminal receives graphical user interface displays from the virtual machine.

11. The computer system of claim 10, wherein the replacement video data is generated with the assistance of a virtual GPU.

12. The computer system of claim 10, wherein the receiving, extracting and transmitting steps are performed by a display protocol application running in the virtual machine and wherein the display protocol application is configured to transmit displays of the virtual machine to the remote display terminal in accordance with a display protocol.

13. The computer system of claim 12, wherein the composite video data composited by the video player application is transmitted by the video player application to a video adapter driver that is configured to transmit the composite video data to the display protocol application.

14. The computer system of claim 9, wherein the intercepting step is performed by a proxy library configured to intercept function calls transmitted to a pre-existing video rendering library.

15. The computer system of claim 9, wherein each video frame in the replacement video data comprises a graphical image, wherein each adjacent pixel of the graphical image alternates between one of two colors and each corresponding pixel in a prior graphical image of an immediately prior video frame in the replacement video data is the other of the two colors.

16. A non-transitory computer readable storage medium including instruction that, when executed by a processor of a computer system, offloads decoding of encoded video data from a computer system to a remote display terminal, by performing the steps of:

intercepting a video decoding function call from the video player application requesting graphics hardware assistance from the computer system to decode the encoded video data, wherein the video player application is configured to decode encoded video data and composite graphical overlays onto decoded video data;

providing replacement video data to the video player application rather than a decoded version of the encoded video data in response to the video decoding function call;

receiving, by the computer system, composite video data composited by the video player application, wherein the composite video data comprises the provided replacement video data and a graphical overlay;

extracting, by the computer system, the graphical overlay from the received composite video data by subtracting the replacement video data from the composite video data; and

transmitting the encoded video data and the extracted graphical overlay from the computer system to the remote display terminal, wherein the remote display terminal decodes the encoded video data and composites the graphical overlay onto the decoded video data to generate a final composite video data for display.

17. The non-transitory computer-readable storage medium of claim 16, wherein the video player application executes within a virtual machine and the remote display terminal receives graphical user interface displays from the virtual machine.

18. The non-transitory computer-readable storage medium of claim 17, wherein the receiving, extracting and transmitting steps are performed by a display protocol application running in the virtual machine and wherein the display protocol application is configured to transmit displays of the virtual machine to the remote display terminal in accordance with a display protocol.

19. The non-transitory computer-readable storage medium of claim 18, wherein the composite video data composited by the video player application is transmitted by the video player application to a video adapter driver that is configured to transmit the composite video data to the display protocol application.

20. The non-transitory computer-readable storage medium of claim 16, wherein the intercepting step is performed by a proxy library configured to intercept function calls transmitted to a pre-existing video rendering library.

21. The non-transitory computer-readable storage medium of claim 16, wherein each video frame in the replacement video data comprises a graphical image, wherein each adjacent pixel of the graphical image alternates between one of two colors and each corresponding pixel in a prior graphical image of an immediately prior video frame in the replacement video data is the other of the two colors.