Detection of video windows and graphics windows

• US 6,873,341 B1
• Filed: 11/04/2002
• Issued: 03/29/2005
• Est. Priority Date: 11/04/2002
• Status: Expired due to Term

• Current Assignee: Lattice Semiconductor Corporation
• Original Assignee: Silicon Image, Inc. (Lattice Semiconductor Corporation)
• Inventors: Mruzik, Michael R., Adams, Dale R., Keating, Stephen J.
• Primary Examiner(s): Luu, Matthew
• Application Number:
  US10/287,976
• Time in Patent Office: 876 Days
• Field of Search: 345/629, 345/630, 345/632, 345/633, 345/634, 345/790
• US Class Current: 345/629
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06V 30/413 Classification of content, e.g. text, photographs or tables
• CPC Class Codes: G09G 2320/0686 with two or more screen areas displaying information with different brightness or colours
• CPC Class Codes: G09G 5/14 Display of multiple viewports

Abstract

The invention relates to computer graphics and computer imaging on a video display, and includes the dynamic detection of video windows and graphical images overlapping one another. A display processor identifies differences between typical video and graphics data sources to detect the edges of video windows. By detecting the edges of active video windows within a graphics image, a display processor may uniquely adjust image characteristics of an exposed video window. These characteristics include, for example, hue, brightness, intensity and contrast.

Claims

1. A method of automatically detecting video pixels within a stream of pixel data, the method comprising:

selecting a first partial set of pixel positions from and fewer than a full set of pixel positions of a frame template;

sampling pixel data for each pixel position of the first partial set of pixel positions from a first frame in the stream of pixel data, thereby defining a first set of pixel data;

saving the first set of pixel data;

sampling corresponding pixel data for each pixel position of the first partial set of pixel positions from a second frame within the stream of pixel data, thereby defining a second set of pixel data; and

calculating a temporal difference value between each pixel data of the first set of pixel data and the corresponding pixel data of the second set of pixel data, thereby defining a first set of difference values between the first frame and the second frame.

2. The method of claim 1, wherein the method repeats over a series of frames greater than two frames and wherein each occurrence of the step of selecting the first partial set of pixel positions further comprises selecting a non-intersecting set of pixel positions, the method repeating after each pixel position of the frame template is selected an equal number of times.

3. The method of claim 2, wherein subsequent occurrences of the step of selecting the first partial set of pixel positions further comprises:

selecting a reference pixel location; and

selecting additional pixel locations based on a relative position to the reference pixel location;

wherein the reference pixel location of a current step of selecting the first partial set of pixel positions is based on a previous reference pixel location selected during an immediately preceding step of selecting the first partial set of pixel positions.

4. The method of claim 3, wherein each pixel position of the frame template is indexed by a row value and a column value, and wherein the reference pixel location of the current step is set to a pixel location set to a fixed row distance and a fixed column distance from the previous reference pixel location.

5. The method of claim 1, wherein the method further comprises comparing each temporal difference value from the first set of difference values to a first threshold and a second threshold defining a first subrange of differences, and if the temporal difference value is within the first subrange of differences, setting a motion indicator to indicate an in-motion pixel value, otherwise setting the motion indicator to indicate a not-in-motion pixel value, thereby setting a plurality of motion indicators.

6. The method of claim 5, wherein the step of selecting the first partial set of pixel positions includes:

dividing the frame template into a plurality of stripes across the frame template, wherein each of the plurality of stripes contains an integer number of rows greater than two;

selecting a representative row from each of the plurality of stripes; and

selecting from the representative row a plurality of representative pixel positions, wherein the plurality of representative pixel positions includes only a partial set of pixel positions contained in the representative row, and

wherein a compilation of the plurality of representative pixel positions of the representative row from each of the plurality of stripes defines the first partial set of pixel positions.

7. The method of claim 6, wherein the method further comprises:

forming one or more moving line segments such that each of the moving line segments include a group of motion indicators contained along the representative row such that at least a majority of the group of motion indicators indicate the in-motion pixel value.

8. The method of claim 7, wherein the method further comprises determining borders of an exposed video window encompassing the video pixels defined by the one or more moving line segments.

9. The method of claim 8, wherein the borders are polygonal.

10. The method of claim 9, wherein the borders have all right angle coners.

11. The method of claim 8, wherein the step of determining the borders of the exposed video window comprises:

comparing one or more right ends of the one or more moving line segments to determine one or more right edges of the exposed video window; and

comparing one or more left ends of the one or more moving line segments to determine one or more left edges of the exposed video window;

examining one or more top moving line segments of the one or more moving line segments to identifying one or more top edges of the exposed video window;

examining one or more bottom moving line segments of the one or more moving line segments to identifying one or more bottom edges of the exposed video window; and

connecting the one or more right, left, top and bottom edges define either a non-rectangular exposed video window or multiple exposed video windows.

12. The method of claim 1, wherein the method further comprises:

determining a second partial set of pixel positions from and fewer than the full set of pixel positions of the frame template, wherein the second partial set of pixel positions includes a plurality of center pixel positions corresponding to the first partial set of pixel positions and a corresponding one or more neighboring pixel positions each located adjacent to one of the plurality of center pixel positions;

sampling center pixel data for each pixel position of the plurality of center pixel positions from a third frame in the stream of pixel data, thereby defining a third set of pixel data;

sampling a corresponding one or more neighboring pixel data for each pixel position of the one or more neighboring pixel positions from the third frame within the stream of pixel data, thereby defining a fourth set of pixel data; and

calculating for each center pixel data, a set of spatial difference values, wherein the set of spatial difference values includes one difference value between the center pixel data and each of the corresponding one or more neighboring pixel data, thereby defining a second set of difference values between the center pixel data and the one or more neighboring pixel data.

13. The method of claim 12, wherein the third frame is the second frame.

14. The method of claim 12, wherein the method further comprises comparing each of the set of spatial difference values from the second set of differences to a third threshold and a fourth threshold defining a second subrange of differences, and if any of the set of spatial difference values for the center pixel data is outside the second subrange of difference, setting a graphics indicator to indicate a graphics pixel value for the center pixel position, otherwise setting the graphics indicator to indicate a not-graphics pixel value for the center pixel position, thereby setting a plurality of graphics indicators.

15. The method of claim 14, wherein the method further comprises forming one or more graphics line segments such that each of the graphics line segments include a group of graphics indicators contained along the representative row such that at least a majority of the group of graphics indicators indicate the graphics pixel value.

16. The method of claim 7, wherein the method further comprises:

determining a second partial set of pixel positions from and fewer than the full set of pixel positions of the frame template, wherein the second partial set of pixel positions includes a plurality of center pixel positions corresponding to the first partial set of pixel positions and a corresponding one or more neighboring pixel positions each located adjacent to one of the plurality of center pixel positions;

sampling center pixel data for each pixel position of the plurality of center pixel positions from a said frame in the stream of pixel data, thereby defining a third set of pixel data;

sampling a corresponding one or more neighboring pixel data for each pixel position of the one or more neighboring pixel positions from the same said frame within the stream of pixel data, thereby defining a fourth set of pixel data; and

calculating for each center pixel data, a set of spatial difference values, wherein the set of spatial difference values includes one difference value between the center pixel data and each of the corresponding one or more neighboring pixel data, thereby defining a second set of difference values between the center pixel data and the one or more neighboring pixel data.

17. The method of claim 16, wherein the method further comprises:

comparing each of the set of spatial difference values from the second set of differences to a third threshold and a fourth threshold defining a second subrange of differences, and if any of the set of spatial difference values for the center pixel data is outside the second subrange of differences, setting a graphics indicator to indicate a graphics pixel value for the center pixel position, otherwise setting the graphics indicator to indicate a not-graphics pixel value, thereby setting a plurality of graphics indicators; and

forming one or more graphics line segments such that each of the graphics line segments includes a group of graphics indicators contained along the representative row such that at least a majority of the group of graphics indicators indicate the graphics pixel value.

18. The method of claim 17, wherein the method further comprises defining as a video line segment each segment of each of the one or more moving line segments that do not overlap with one of the one or more graphics line segments.

19. The method of claim 18, wherein the method further comprises:

performing an auto correlation function to detect where in each of the one or more graphics line segments a blocking artifact exists wherein the blocking artifact exists due to a compressing of a video image; and

defining an additional video line segment for each segment exhibiting the blocking artifact.

20. The method of claim 18, wherein the method further comprises:

determining the borders of an exposed video window using the video line segments such that the borders enclose the video line segments.

21. The method of claim 20, wherein the step of determining the borders the exposed video window comprises:

comparing one or more right ends of the one or more video line segments to determine one or more right edges of the exposed video window; and

comparing one or more left ends of the one or more video line segments to determine one or more left edges of the exposed video window;

examining one or more top moving line segments of the one or more video line segments to identifying one or more top edges of the exposed video window; and

examining one or more bottom moving line segments of the one or more video line segments to identifying one or more bottom edges of the exposed video window;

whereby the one or more right, left, top and bottom edges define the exposed video window.

22. An apparatus for automatically detecting video pixels within a stream of pixel data, the apparatus comprising:

a means for selecting a first partial set of pixel positions from and fewer than a full set of pixel positions of a frame template;

a means for sampling pixel data for each pixel position of the first partial set of pixel positions from a first frame in the stream of pixel data, thereby defining a first set of pixel data, and for sampling corresponding pixel data for each pixel position of the first partial set of pixel positions from a second frame within the stream of pixel data, thereby defining a second set of pixel data;

a memory for saving the first set of pixel data; and

a processor programmed to calculate a temporal difference value between each of the pixel data of the first set of pixel data and the corresponding pixel data of the second set of pixel data, thereby defining a first set of differences.

23. The apparatus of claim 22, wherein the apparatus further comprises:

a means for comparing each of the temporal difference values;

a means for forming one or more moving line segments; and

a means for determining borders of the exposed video window using the one or more moving line segments.

24. An apparatus for automatically detecting video pixels within a stream of pixel data, the apparatus comprising:

a means for sampling a first set of pixel data for each pixel position of the partial set of pixel positions from a first frame in the stream of pixel data, wherein the partial set of pixel positions are selected from and fewer than a full set of pixel positions of a frame template;

a means for sampling a second set of pixel data for each pixel position of the partial set of pixel positions from a second frame in the stream of pixel data;

a means for sampling a third set of pixel data for a select set of pixel positions adjacent to each pixel position of the partial set of pixel positions from the first or second frames in the stream of pixel data;

a temporal computation means for performing calculations between the first set of pixel data from the first frame and the second set of pixel data from the second frame;

a spatial computational means for performing calculations among the first or second sets of pixel data and the third set of pixel data;

a correlation means to generate one or more video line segments by correlating results from the temporal computation means and the spatial computational means;

a means for determine boundaries of an exposed video window encompassing the video pixels by using the results from the correlation means; and

a means for modifying pixel data within the stream of pixel data for pixel data located within the boundaries of the exposed video window.

25. An semiconductor device for automatically detecting and modifying video pixels within a stream of pixel data, the semiconductor device comprising:

a memory;

a processor coupled to the memory;

an input coupled to the stream of pixel data and coupled to the processor;

a sampler circuit;

a program for execution on the processor, wherein the program controls the sampler circuit such that the sampler circuit samples only a partial set pixel data from frame to frame of fewer than all pixel data in the stream of pixel data, and wherein the program determines boundaries to encompass the video pixels, thereby defining an exposed video window based on the partial set of pixel data;

a pixel modifying unit to modify pixel data identified as falling within the boundaries of the exposed video window; and

an output coupled to an output stream of pixel data and of the pixel modifying unit.

26. The semiconductor device of claim 25, wherein the output provides an output stream of pixel data formatted in a digital video interface standard.

27. The semiconductor device of claim 25, wherein the input accepts the stream of pixel data when formatted in a digital video interface standard.

28. A method for detecting video pixel data in a stream of pixel data arranged as sequential frames, with each said frame including a set of pixel positions to which the pixel data corresponds, the method comprising:

sampling the pixel data for only a subset of the pixel positions of a first said frame;

sampling the pixel data for a same subset of the pixel positions for a second said frame;

determining an amount of change in a value of the pixel data for each pixel position of the subset of pixel positions between the first and second sets of pixel data;

determining from the amount of the change in the value whether the pixel data for each pixel position of the subset is the video pixel data.

29. The method of claim 28, further comprising:

sampling the pixel data for only a subset of the pixel positions of one of said frames, thereby defining a third set of pixel data, said third set including pixel data corresponding to plural groups of neighboring pixel positions of the frame;

determining an amount of change in a value of the pixel data for at least some of the neighboring pixel positions of each group; and

determining from the amount of change in the value whether the pixel data corresponding to at least one of the pixel positions of each said group is video pixel data or not video pixel data.

30. A method for detecting whether a pixel data in a stream of pixel data is a video pixel data or not a video pixel data, wherein each pixel data of the stream corresponds to one of plural pixel position of a fame, the method comprising:

sampling the pixel data corresponding to a first subset of the pixel positions of the frame, said first subset including plural pixel positions but fewer than all of said pixel positions of the frame;

sampling the pixel data corresponding to at least one pixel position neighboring each of the pixel positions of the first subset of the pixel positions;

determining a difference in a value between the sampled pixel data corresponding to each pixel position of the first subset and the respective neighboring pixels; and

determining from said difference in the value whether the pixel data for each pixel position of the first subset is a said video pixel data or not a said video pixel data.