Apparatus and method for mixing graphics with video images

• US 9,148,639 B2
• Filed: 01/09/2014
• Issued: 09/29/2015
• Est. Priority Date: 12/20/2005
• Status: Active Grant

• Current Assignee: Savant Systems, LLC
• Original Assignee: Savant Systems, LLC
• Inventors: Madonna, Robert P., Noonan, Michael E., Eschholz, Siegmar K., Allen, James F., Corsini, Peter H.
• Primary Examiner(s): Lee, Michael
• Application Number:
  US14/151,246
• Publication Number:
  US 20140125868A1
• Time in Patent Office: 628 Days
• Field of Search: 348563-569, 348580-581, 348/843, 348/298, 348/629, 348/634, 348584-601, 345629-641
• US Class Current: 1/1
• CPC Class Codes: G09G 2340/10 Mixing of images, i.e. displayed pixel being the result of an operation, e.g. adding, on the corresponding input pixels
• CPC Class Codes: G09G 2340/125 wherein one of the images is motion video
• CPC Class Codes: G09G 2370/20 Details of the management of multiple sources of image data
• CPC Class Codes: G09G 5/026 Control of mixing and/or overlay of colours in general G09G5/022 and G09G5/024 take precedence
• CPC Class Codes: G09G 5/397 Arrangements specially adapted for transferring the contents of two or more bit-mapped memories to the screen simultaneously, e.g. for mixing or overlay G09G5/02 takes precedence
• CPC Class Codes: H04L 12/2803 Home automation networks
• CPC Class Codes: H04L 12/282 based on user interaction within the home receiver circuitry for displaying additional information being controlled by a remote control apparatus H04N21/42204
• CPC Class Codes: H04L 12/2838 Distribution of signals within a home automation network, e.g. involving splitting/multiplexing signals to/from different paths adaptations of television systems for transmission by electric cable for domestic distribution H04N7/106; hybrid transport H04L12/6418; home network arrangements specially adapted for distribution of digital video signals H04N7/24
• CPC Class Codes: H04L 2012/2849 Audio/video appliances
• CPC Class Codes: H04L 2012/285 Generic home appliances, e.g. refrigerators
• CPC Class Codes: H04N 21/4112 having fewer capabilities than the client, e.g. thin client having less processing power or no tuning capabilities
• CPC Class Codes: H04N 21/42646 for reading from or writing on a non-volatile solid state storage medium, e.g. DVD, CD-ROM
• CPC Class Codes: H04N 21/43615 Interfacing a Home Network, e.g. for connecting the client to a plurality of peripherals home Audio Video Interoperability [HAVI] data switching networks H04L12/2805
• CPC Class Codes: H04N 21/4438 Window management, e.g. event handling following interaction with the user interface
• CPC Class Codes: H04N 5/265 Mixing
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay
• CPC Class Codes: H04N 5/765 Interface circuits between an apparatus for recording and another apparatus
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 5/775 between a recording apparatus and a television receiver
• CPC Class Codes: H04N 5/85 on discs or drums
• CPC Class Codes: H04N 9/74 : for obtaining special effects H04N9/65 - H04N9/73 take precedence
• CPC Class Codes: H04N 9/76 : for mixing of colour signals H04N9/75 takes precedence

Abstract

In one embodiment, graphics frames are received, where each graphics frame includes one or more regions where pixels depict graphics that represent an on screen display (OSD) used to interact with a programmable multimedia controller, and a background region where pixels are set to the one or more predetermined colors. Further, video images are received, where at least some of the video images correspond to the plurality of graphics frames. Mixed images are created by mixing the graphics frames and the corresponding video images, the mixing to, where pixels of the graphics frame are not set to the one more predetermined colors, blend a color of at least some of the pixels of the graphics frame with a color of pixels of a corresponding video image, and, where pixels of the graphics frame are set to the one more predetermined colors, pass pixels of the corresponding video image.

Claims

1. A method comprising:

receiving graphics frames generated by a processing subsystem of a programmable multimedia controller at a module of the programmable multimedia controller, each graphics frame including one or more regions where pixels depict graphics that represent an on screen display (OSD) that includes a menu for interacting with the programmable multimedia controller, and a background region where pixels are set to the one or more predetermined colors;

receiving video images from a video source external to the programmable multimedia controller at the module of the programmable multimedia controller, at least some of the video images to correspond to the plurality of graphics frames;

synchronizing the graphics frames to coincide with the corresponding video images;

creating mixed images by mixing the graphics frames and the corresponding video images, the mixing to examine colors of each of the pixels of each graphics frame, the colors represented by a color value within a color space of the graphics frame, and,

where pixels of the graphics frame are not set to the one more predetermined colors, blend a color of at least some of the pixels of the graphics frame with a color of pixels of a corresponding video image, and,

where pixels of the graphics frame are set to the one more predetermined colors, pass pixels of the corresponding video image; and

outputting the mixed images to a display device separate from the programmable multimedia controller that is coupled to the module of the programmable multimedia controller.

2. The method of claim 1, wherein the mixing to blend the color of the at least some of the pixels of the graphics frame with the color of the pixels of the corresponding video image is based on alpha blend values that are calculated according to a selected blend mode of a plurality of supported blend modes, wherein the selected blend mode calculates the alpha blend values based on at least one of intensity of pixels of the graphic frame or intensity of pixels of the corresponding video image, and the mixing yields an at least partially semi-transparent OSD.

3. The method of claim 2, wherein the mixing to blend the color of the at least some of the pixels of the graphics frame with the color of the pixels of the corresponding video image further comprises:

multiplying values that represent the color of the at least some of the pixels of the graphics frame and values that represent the color of the pixels of the corresponding video image by the alpha blend values;

combining the values that represent the color of the at least some of the pixels of the graphics frame and the values that represent the color of the pixels of the corresponding video image; and

scaling the combined values.

4. The method of claim 1, further comprising:

eliminating any natural occurrences of the one or more predetermined colors from the graphics frames.

5. The method of claim 4, wherein the eliminating further comprises:

scanning the graphics frames for pixels that are naturally set to the one or more predetermined colors; and

replacing the pixels that are naturally set to the one or more predetermined colors within the graphics frame with pixels of another color.

6. The method of claim 1, wherein a location of the one or more regions where pixels depict the graphics differs on a frame-by-frame basis among ones of the graphics frames.

7. The method of claim 1, wherein the one or more predetermined colors are a repeating pattern of predetermine colors defined by two or more adjacent pixels having color values within the color space of the graphics frame that have a predetermined less than or greater than relation to one or more thresholds, and the creating mixed images operates to,

where adjacent pixels of the graphics frame have colors represented by color values that do not have the predetermined less than or greater than relation to the one or more thresholds, pass pixels of the graphics frame, and,

where adjacent pixels of the graphics frame have colors represented by color values that have the predetermined less than or greater than relation to one or more thresholds, pass pixels of a corresponding video image.

14. The method of claim 1, wherein the programmable multimedia controller is configured to control at least one of lighting devices;

heating, ventilation, and air conditioning (HVAC) devices;

automatic window shade or blinds;

or electronic door locks.

8. A method comprising:

receiving graphics frames, each graphics frame including one or more regions where pixels depict graphics that represent an on screen display (OSD) generated by a programmable multimedia controller, and a background region where pixels are set to a repeating pattern of predetermine colors, wherein the repeating pattern of predetermined colors is a 2-pixel pattern defined by two or more adjacent pixels that includes at least a first pixel having a first color represented by a first color value that has a first particular less than or greater than relation to a first threshold, and a second pixel having a second color represented by a second color value that has a second particular less than or greater than relation to a second threshold;

receiving video images, at least some of the video images to correspond to the plurality of graphics frames;

creating mixed images by mixing the graphics frames and the corresponding video images, the mixing to examine colors of each of the pixels of each graphics frame, the colors represented by a color value within a color space of the graphics frame, and,

where adjacent pixels of the graphics frame have colors represented by color values that do not have the first particular less than or greater than relation to the first threshold and the second particular less than or greater than relation to the second threshold, pass pixels of the graphics frame, and

where adjacent pixels of the graphics frame have colors represented by color values that have the first particular less than or greater than relation to the first threshold and the second particular less than or greater than relation to the second threshold, pass pixels of a corresponding video image; and

outputting the mixed images to a display device.

15. The method of claim 8, wherein the mixing to blend the color of the at least some of the pixels of the graphics frame with the color of the pixels of the corresponding video image is based on alpha blend values that are calculated according to a selected blend mode of a plurality of supported blend modes, wherein the selected blend mode calculates the alpha blend values based on at least one of intensity of pixels of the graphic frame or intensity of pixels of the corresponding video image, and the mixing yields an at least partially semi-transparent OSD.

16. The method of claim 15, wherein the mixing to blend the color of the at least some of the pixels of the graphics frame with the color of the pixels of the corresponding video image further comprises:

multiplying values that represent the color of the at least some of the pixels of the graphics frame and values that represent the color of the pixels of the corresponding video image by the alpha blend values;

combining the values that represent the color of the at least some of the pixels of the graphics frame and the values that represent the color of the pixels of the corresponding video image; and

scaling the combined values.

17. The method of claim 8, further comprising:

eliminating any natural occurrences of the first color or the second color from the graphics frames.

18. The method of claim 17, wherein the eliminating further comprises:

scanning the graphics frames for pixels that are naturally set to the first color or the second color; and

replacing the pixels that are naturally set to the first color or the second color within the graphics frame with pixels of another color.

19. The method of claim 8, wherein a location of the one or more regions where pixels depict the graphics differs on a frame-by-frame basis among ones of the graphics frames.

20. The method of claim 8, wherein the graphics frames originate from a processing subsystem of the programmable multimedia controller and the video images originate from a video source external to the programmable multimedia controller.

21. The method of claim 8, wherein at least the receiving graphics frames, receiving video images, creating, and outputting are performed by a module of the programmable multimedia controller.

22. The method of claim 8, wherein the programmable multimedia controller is configured to control at least one of lighting devices;

heating, ventilation, and air conditioning (HVAC) devices;

automatic window shade or blinds;

or electronic door locks.

9. An apparatus comprising:

a first interface of a module of a programmable multimedia controller configured to receive graphics frames generated by a processing subsystem of the programmable multimedia controller, each graphics frame including one or more regions where pixels depict graphics that represent an on screen display (OSD) that includes a menu for interacting with the programmable multimedia controller, and a background region where pixels are set to one or more predetermined colors;

a second interface of the module configured to receive video images from a video source external to the programmable multimedia controller, at least some of the video images to correspond to the plurality of graphics frames;

logic configured to synchronize the graphics frames to coincide with the corresponding video images;

a video mixer of the module configured to create mixed images by mixing the graphics frames and the corresponding video images, the mixing to examine colors of each of the pixels of each graphics frame, the colors represented by a color value within a color space of the graphics frame, and to blend a color of at least some of the pixels of a graphics frame with a color of pixels of a corresponding video image where pixels of the graphics frame are not set to the one or more predetermined colors to create an intermediate color, and to pass pixels of the corresponding video image where pixels of the graphics frame are set to the one or more predetermined colors; and

one or more output interfaces of the module configured to output the mixed images to a display device.

10. The apparatus of claim 9, further comprising:

an alpha blend value generator configured to calculate alpha blend values according to a selected blend mode of a plurality of supported blend modes, wherein the selected blend mode causes the alpha blend value generator to calculate the alpha blend values based on at least one of intensity of pixels of the graphic frame or intensity of pixels of the corresponding video image, and

wherein the video mixer is further configured to blend the color of the at least some of the pixels of the graphics frame with the color of the pixels of the corresponding video image by multiplication of the values that represent the color of the at least some of the pixels of the graphics frame and the values that represent the color of the pixels of the corresponding video image by the alpha blend values, combination of the values that represent the color of the at least some of the pixels of the graphics frame and the values that represent the color of the pixels of the corresponding video image, and use of a scaled version of the combined values.

11. The apparatus of claim 9, wherein a location of the one or more regions where pixels depict the graphics differs on a frame-by-frame basis among ones of the graphics frames.

12. The apparatus of claim 9, wherein the one or more predetermined colors are a repeating pattern of predetermine colors defined by two or more pixels having color values within the color space of the graphics frame that have a predetermined less than or greater than relation to one or more thresholds, and video mixer configured is configured to create mixed images by, where adjacent pixels of the graphics frame have colors represented by color values that do not have the predetermined less than or greater than relation to the one or more thresholds, pass pixels of the graphics frame, and, where adjacent pixels of the graphics frame have colors represented by color values that have the predetermined less than or greater than relation to one or more thresholds, pass pixels of a corresponding video image.

13. An apparatus comprising:

a first interface configured to receive graphics frames, each graphics frame including one or more regions where pixels depict graphics that represent an on screen display (OSD) generated by a programmable multimedia controller, and a background region where pixels are set to a repeating pattern of predetermined colors defined by two or more adjacent pixels, wherein the repeating pattern of predetermined colors is a 2-pixel pattern defined by at least a first pixel having a first color represented by a first color value that has a first particular less than or greater than relation to a first threshold, and a second pixel having a second color represented by a second color value that has a second particular less than or greater than relation to a second threshold;

a second interface configured to receive video images, at least some of the video images to correspond to the plurality of graphics frames;

a video mixer configured to create mixed images by mixing the graphics frames and the corresponding video images, the mixing to examine colors of each of the pixels of each graphics frame, the colors represented by a color value within a color space of the graphics frame, and where adjacent pixels of the graphics frame have colors represented by color values that do not have the first particular less than or greater than relation to the first threshold and the second particular less than or greater than relation to the second threshold, pass pixels of the graphics frame, and, where adjacent pixels of the graphics frame have colors represented by color values that have the first particular less than or greater than relation to the first threshold and the second particular less than or greater than relation to the second threshold, pass pixels of a corresponding video image; and

one or more output interfaces configured to output the mixed images to a display device.

23. The apparatus of claim 13, wherein a location of the one or more regions where pixels depict the graphics differs on a frame-by-frame basis among ones of the graphics frames.

24. The apparatus of claim 13, further comprising:

a processing subsystem of the programmable multimedia controller configured to originate the graphics frames.