Subpicture overlay using fragment shader

• US 7,542,042 B1
• Filed: 11/10/2004
• Issued: 06/02/2009
• Est. Priority Date: 11/10/2004
• Status: Active Grant

• Current Assignee: NVIDIA Corporation
• Original Assignee: NVIDIA Corporation
• Inventors: Allen, Roger L., Bastos, Rui M., Abdalla, Karim M., Legakis, Justin S.
• Primary Examiner(s): Chauhan; Ulka
• Assistant Examiner(s): PRENDERGAST, ROBERTA D
• Application Number:
  US10/985,575
• Time in Patent Office: 1,665 Days
• Field of Search: 345/629, 345/501, 345/506, 345/522, 345/505
• US Class Current: 345/506
• CPC Class Codes: G06F 9/5044 considering hardware capabilities
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 15/005 General purpose rendering architectures
• CPC Class Codes: G09G 2360/121 using a cache memory
• CPC Class Codes: G09G 2360/18 Use of a frame buffer in a display terminal, inclusive of the display panel
• CPC Class Codes: G09G 5/14 Display of multiple viewports

Abstract

A new method of operating a fragment shader to produce complex video content comprised of a video image or images, such as from a DVD player, that overlays a fragment shader-processed background. Pixels are fragment shader-processed during one loop or set of loops through a texture processing stations to produce a fragment shader-processed background. Then, at least some of those pixels are merged with the video or images to produce complex video content. The resulting complex image is then made available for further processing.

Claims

1. A method of operating a fragment shader that includes a plurality of shader pipelines and a fragment shader distributor for selectively receiving rasterized pixel data and video image data, for selectively processing said rasterized pixel data, and for distributing said processed rasterized pixel data and said video image data to said plurality of shader pipelines, wherein each of the shader pipelines processes selected pixels of a video image to be composited, the method comprising the steps of:

receiving very long instruction words (VLIWs), each VLIW including a plurality of instructions for programming separate processing stations within a shader pipeline and a program flag bit, wherein different shader pipelines within the plurality of shader pipelines are programmed with different VLIWs;

separately programming a first plurality of processing stations within a first of the plurality of shader pipelines based on the plurality of instructions in a first VLIW;

receiving first video pixel data by recalling rasterized video image data from a frame buffer memory;

processing said first video pixel data and said rasterized video image data to produce a first pixel data set that includes said rasterized video image data;

storing the first pixel data set;

programming a second plurality of processing stations within a second of the plurality of shader pipelines based on a second VLIW and the availability of the second shader pipeline;

receiving second video image data;

processing said second video image data to produce a second pixel data set; and

merging the first and second pixel data sets to produce a composite video image data set, wherein corresponding pixels of the first and second pixel data sets are processed in a same shader pipeline within the plurality of shader pipelines.

2. The method of claim 1, wherein the step of processing said first video pixel data is performed by passing said first video pixel data through a programmable fragment shader having multiple texture processing stations.

3. The method of claim 2, wherein the step of passing said first video pixel data through the programmable fragment shader includes looping through multiple texture processing stations.

4. The method of claim 3, wherein the step of passing said first video pixel data through the programmable fragment shader includes segmenting rasterized pixel data.

5. The method of claim 1, wherein the step of processing said first video pixel data comprises adding texture information.

6. The method of claim 1, wherein the step of receiving second video image data includes obtaining video image data from a DVD player.

7. The method of claim 1, wherein the step of merging the stored first pixel data set and the second pixel data set together is performed by overlaying the second pixel data set on the stored first pixel data set.

8. The method of claim 1, wherein the step of merging the first and second pixel data sets is performed by blending the first and second pixel data sets together.

9. The method of claim 1, wherein each of the first video pixel data and the second video image data are processed segment by segment, each segment comprising a plurality of pixels.

10. A computer system, comprising:

a central processing unit for running an application program that causes the central processing unit to produce raw graphics data;

a video source for producing a video image;

a frame buffer memory for storing the raw graphics data and the video image; and

a graphics processing unit for processing said raw graphics data into pixel data and for merging said pixel data with said video image, said graphics processing unit including;

a host interface for receiving said raw graphics data;

a geometry engine for producing two-dimensional frame-buffer co-ordinates of graphics primitives within said raw graphics data;

a rasterizer that rasterizes said raw graphics data using said two-dimensional frame buffer co-ordinates to produce rasterized pixel data;

a fragment shader having a plurality of fragment shader pipelines that each contain programming processing stations that are programmed with incoming very long instruction words (VLIWs) that each include a program flag bit for processing said rasterized pixel data, different fragment shader pipelines being programmed with different VLIWs that each include a plurality of different instructions for programming the processing stations; and

a scheduler for separately and differently programming each of the fragment shader pipelines of said fragment shader, wherein said application program causes said fragment shader to process said rasterized pixel data and to merge said pixel data with said video image, said fragment shader being configured to;

receive a first VLIW that includes a plurality of instructions for programming separate processing stations within a first fragment shader pipeline and including a program flag bit;

program a plurality of processing stations within the first fragment shader pipeline based on the plurality of instructions in the first VLIW;

receive first video pixel data by recalling rasterized video image data from a frame buffer memory;

process said first video pixel data and said rasterized video image data to produce a first pixel data set that includes said rasterized video image data;

store the first pixel data set;

receive a second VLIW that includes a plurality of instructions for programming separate processing stations within a second fragment shader pipeline and including a program flag bit;

program a second plurality of processing stations within the second shader pipeline based on the second VLIW and the availability of the second shader pipeline;

receive second video image data;

process said second video image data to produce a second pixel data set; and

merge the first and second pixel data sets to produce a composite video image data set, wherein corresponding pixels of the first and second pixel data sets are processed in a same fragment shader Pipeline within the plurality of fragment shader pipelines.

11. The computer system of claim 10, wherein each of the video pixel data and video image data are processed segment by segment, each segment comprising a plurality of pixels.

12. A graphics processing integrated circuit, comprising:

a front end for receiving raw graphics data and video image data;

a geometry engine for organizing said raw graphics data into geometric primitives;

a rasterizer for converting said geometric primitives into rasterized pixel data; and

a fragment shader for processing said rasterized pixel data to produce pixel data and for merging said pixel data and said video image data together, said fragment shader including;

a plurality of fragment shader pipelines that each contain programmable processing stations that are programmed with incoming very long instruction words (VLIW)s that each include a program flag bit, each of the fragment shader pipelines for processing said rasterized pixel data to form said background pixel data and for merging said pixel data and said video image data together; and

a fragment shader distributor for selectively receiving rasterized pixel data and video image data, for selectively processing said rasterized pixel data, and for distributing said processed rasterized pixel data and said video image data to said plurality of fragment shader pipelines, wherein said fragment shader includes a scheduler for applying the very long instruction words (VLIWs) that each comprise a plurality of instructions to said plurality of fragment shader pipelines, wherein different ones of VLIWs are applied to different ones of the fragment shader pipelines and said program commands control the processing of said rasterized pixel data and the merging of said pixel data and said video image data together, said fragment shader being configured to;

receive a first VLIW that includes a plurality of instructions for programming separate processing stations within a first one of the fragment shader pipelines and including a program flag bit;

program a plurality of processing stations within the first fragment shader pipeline based on the plurality of instructions in the first VLIW;

receive first video pixel data by recalling rasterized video image data from a frame buffer memory;

process said first video pixel data and said rasterized video image data to produce a first pixel data set that includes said rasterized video image data;

store the first pixel data set;

receive a second different VLIW that includes a plurality of instructions for programming separate processing stations within a second one of the fragment shader pipelines and including a program flag bit;

program a second plurality of processing stations within the second fragment shader pipeline based on the second VLIW and the availability of the second shader pipeline;

receive second video image data;

process said second video image data to produce a second pixel data set; and

merge the first and second pixel data sets to produce a composite video image data set, corresponding pixels of the first and second pixel data sets are processed in a same fragment shader pipeline within the plurality of fragment shader pipelines.

13. The graphics processing integrated circuit of claim 12, wherein said rasterized pixel data is divided into segments, each of the segments being able to fit into one or more of the fragment shader pipelines without loss of said rasterized pixel data.

14. The graphics processing integrated circuit of claim 12, wherein at least one of the fragment shader pipelines shades processed rasterized pixel data to add texture information.