STREAMING PER-PIXEL TRANSPARENCY INFORMATION USING TRANSPARENCY-AGNOSTIC VIDEO CODECS

• US 20200133696A1
• Filed: 11/27/2019
• Published: 04/30/2020
• Est. Priority Date: 10/26/2018
• Status: Active Grant

• Current Assignee: NVIDIA Corporation
• Original Assignee: NVIDIA Corporation
• Inventors: Zimmermann, Johannes, Bosnjakovic, Andrija, Reid, Ashley
• Granted Patent: US 11,403,121 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 2203/04804 Transparency, e.g. transparent or translucent windows
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 9/452 Remote windowing, e.g. X-Window System, desktop virtualisation protocols for virtual reality H04L67/131
• CPC Class Codes: H04L 65/612 for unicast
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04L 65/764 at the destination reformatting of additional data in video clients H04N21/4355

Abstract

The disclosure relates to the transfer of per-pixel transparency information using video codecs that do not provide an alpha channel (alternatively referred to as “transparency-agnostic video codecs”). For example, alpha information of visual elements may be transcoded into the supported channels of a video stream to generate additional samples of a supported color space, which are representative of the alpha information. After being encoded by a “transparency-agnostic video codec” and transmitted, the received alpha information may then be extracted from the supported channels of the video stream to render the received visuals with corresponding per-pixel transparency.

Claims

1. A method comprising:

determining application visuals corresponding to one or more applications to include in a video stream;

transcoding alpha information of the application visuals to a color space that is supported by the video stream to generate alpha samples that represent the alpha information using the color space;

storing the alpha samples in one or more color channels of the color space in image data representative of at least one video frame of the video stream; and

generating at least a portion of the video stream comprising the image data.

2. The method of claim 1, wherein the alpha samples include per-pixel transparency information of at least some of the application visuals.

3. The method of claim 1, wherein the storing comprises attaching groups of the alpha samples to the video stream as a single color sample of the color space.

4. The method of claim 1, wherein the color space is a Luma-Chrominance (YUV) color space or a Red Green Blue (RGB) color space.

5. The method of claim 1, wherein the storing of the alpha samples includes a video codec of the video stream processing the alpha samples as color samples of the color space.

6. The method of claim 1, wherein non-alpha color samples of the application visuals are stored in a first designated region of the at least one video frame and the alpha samples are stored in a second designated region of the at least one video frame.

7. The method of claim 1, wherein the transcoding of the alpha information includes swizzling 2×

3 blocks of the alpha information to 2×

2 blocks of the color space to account for subsequent Luma-Chrominance (YUV) chroma sub-sampling.

8. A method comprising:

receiving, in a video stream, image data representative of at least one video frame of the video stream;

extracting, from one or more color channels of a color space that is supported by the video stream, alpha samples of application visuals from the image data; and

rendering the application visuals to exhibit transparency properties based on correlating the alpha samples of the application visuals with non-alpha color samples of the application visuals.

9. The method of claim 8, wherein the alpha samples include per-pixel transparency information of at least some of the application visuals.

10. The method of claim 8, wherein the receiving of the video stream is from a server and the rendering of the application visuals is on a client.

11. The method of claim 8, wherein the alpha samples are extracted as groups of the alpha samples from individual color samples of the color space that is supported by the video stream.

12. The method of claim 8, wherein the color space is a Luma-Chrominance (YUV) color space or a Red Green Blue (RGB) color space.

13. The method of claim 8, wherein at least some of the alpha samples correspond to a visual element and the rendering of the application visuals includes identifying the visual element of the application visuals from metadata of the visual element that identifies the visual element within the video stream.

14. The method of claim 8, wherein the extracting of the alpha samples includes a video codec of the video stream processing the alpha samples as color samples of the color space that is supported by the video stream.

15. A system comprising:

a computing device including one or more processing devices and one or more memory devices communicatively coupled to the one or more processing devices storing programmed instructions thereon, which when executed by the one or more processing devices causes the instantiation of;

a communications manager to receive, in a video stream, image data representative of at least one video frame of the video stream;

a stream decoder to extract alpha samples of a visual element from one or more color channels of a color space of the image data that is supported by the video stream; and

a renderer to render the visual element, over at least one image generated by the computing device, using transparency that corresponds to the alpha samples based on correlating the alpha samples of the visual element with non-alpha color samples of the visual element.

16. The system of claim 15, wherein the renderer renders the visual element based at least in part on properties of the visual element that the communications manager received in association with the image data.

17. The system of claim 15, wherein the alpha samples include per-pixel transparency information of the visual element.

18. The system of claim 15, wherein the stream decoder extracts groups of the alpha samples from individual color samples of the color space.

19. The system of claim 15, wherein the color space is a Luma-Chrominance (YUV) color space or a Red Green Blue (RGB) color space.

20. The system of claim 15, wherein the stream decoder extracts the non-alpha color samples of the visual element from a first defined region of the at least one video frame and the alpha samples from a second defined region of the at least one video frame.