Post production visual enhancement rendering

• US 7,444,656 B2
• Filed: 03/27/2003
• Issued: 10/28/2008
• Est. Priority Date: 08/02/2001
• Status: Expired due to Term

• Current Assignee: OpenTV, Inc. (Kudelski SA)
• Original Assignee: Intellocity USA, Inc. (Kudelski SA)
• Inventors: Lemmons, Thomas
• Primary Examiner(s): Lonsberry, Hunter B.
• Application Number:
  US10/403,317
• Publication Number:
  US 20040031062A1
• Time in Patent Office: 2,042 Days
• Field of Search: 725/32, 725 34- 35, 725/40
• US Class Current: 725/32
• CPC Class Codes: H04N 21/23412 for generating or manipulating the scene composition of objects, e.g. MPEG-4 objects
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/234318 by decomposing into objects, e.g. MPEG-4 objects
• CPC Class Codes: H04N 21/235 Processing of additional data, e.g. scrambling of additional data or processing content descriptors
• CPC Class Codes: H04N 21/25808 Management of client data terminal profiles in network data switching protocols H04L67/303
• CPC Class Codes: H04N 21/25883 being end-user demographical data, e.g. age, family status or address arrangements for identifying locations of users in broadcast systems H04H60/52
• CPC Class Codes: H04N 21/4312 involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations
• CPC Class Codes: H04N 21/4314 for fitting data in a restricted space on the screen, e.g. EPG data in a rectangular grid
• CPC Class Codes: H04N 21/435 Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream
• CPC Class Codes: H04N 21/44012 involving rendering scenes according to scene graphs, e.g. MPEG-4 scene graphs
• CPC Class Codes: H04N 21/4622 Retrieving content or additional data from different sources, e.g. from a broadcast channel and the Internet web site content organization and management for information retrieval from the Internet G06F16/958; transmission by internet of broadcast information H04H60/82; stock exchange data over packet-switching network H04L12/1804; push services including data channel over packet-switching network H04L12/1859
• CPC Class Codes: H04N 21/47205 for manipulating displayed content, e.g. interacting with MPEG-4 objects, editing locally
• CPC Class Codes: H04N 21/47815 Electronic shopping payment schemes, payment architectures or payment protocols for electronic shopping systems G06Q20/12
• CPC Class Codes: H04N 21/4782 Web browsing , e.g. WebTV
• CPC Class Codes: H04N 21/812 involving advertisement data advertising per se G06Q30/02
• CPC Class Codes: H04N 21/854 Content authoring
• CPC Class Codes: H04N 21/8543 using a description language, e.g. Multimedia and Hypermedia information coding Expert Group [MHEG], eXtensible Markup Language [XML] information retrieval of semistructured data, the underlying structure being taken into account, e.g. mark-up language structure data G06F16/80
• CPC Class Codes: H04N 21/858 Linking data to content, e.g. by linking an URL to a video object, by creating a hotspot
• CPC Class Codes: H04N 21/8586 by using a URL processing chained hypermedia data for information retrieval G06F16/94; information retrieval from the Internet by using URLs G06F16/955; URL in broadcast information H04H20/93; Web-based protocols H04L67/02
• CPC Class Codes: H04N 7/17318 Direct or substantially direct transmission and handling of requests

Abstract

Disclosed is a method and system that renders enhancement images onto receptor sites of a scene of a pre-existing program. A receptor site defines an area of a scene onto which an enhancement image may be rendered and provides information that may include size, scale, rotation, perspective, contour and coloration. Rendering processes may be direct, planar, conformal, blended, or migrated and may produce photo-realistic images that appear to be a natural part of a scene. Selection of enhancements may be targeted or opportunistic and may employ secondary selection criteria including demographics, viewer preferences, pricing and other information. Receptor site identification may employ human or software resources, or a combination thereof. Receptor site information may be determined through human or software resources or a combination thereof and may include the acquisition of characteristics of a displayed object image that then may be applied to an enhancement image.

Claims

1. A method for rendering an enhancement image onto a scene of pre-existing program content, comprising:

detecting a receptor site for said enhancement;

determining a size of said receptor site;

selecting said enhancement image from a plurality of enhancement images based on said size of said receptor site;

accessing enhancement image information;

processing said enhancement image based on said enhancement image information; and

rendering said processed enhancement image at said receptor site based on said selected enhancement image information.

2. The method of claim 1 wherein said step of detecting a receptor site further comprises:

receiving an input from an operator viewing said scene.

3. The method of claim 1 wherein said step of detecting a receptor site further comprises:

applying an image recognition software program to said scene.

4. The method of claim 1 wherein said step of processing said enhancement image further comprises:

scaling said enhancement image.

5. The method of claim 1 wherein said step of processing said enhancement image further comprises:

rotating said enhancement image.

6. The method of claim 1 wherein said step of processing said enhancement image further comprises:

warping said enhancement image.

7. The method of claim 1 wherein said step of processing said enhancement image further comprises:

warping said enhancement image to conform to the contour of a recognized object.

8. The method of claim 1 wherein said step of determining a receptor site characteristic comprises:

determining a coloration characteristic of said receptor site.

9. The method of claim 8 wherein said step of processing said enhancement image further comprises:

applying said coloration characteristics characteristic to said enhancement image.

42. The method of claim 1, wherein said enhancement image information further comprises an instruction for rendering said enhancement image.

43. The method of claim 1, wherein said enhancement image information further comprises an instruction for combining said enhancement image with program content.

44. The method of claim 1, wherein the enhancement image information further comprises an instruction for applying a property of the receptor site to the enhancement image.

45. The method of claim 1, wherein the enhancement image information further comprises an instruction for applying a property of the enhancement image to the receptor site.

46. The method of claim 1, wherein the enhancement image information further comprises an instruction for scaling or clipping.

47. The method of claim 1, wherein the enhancement image information further comprises control color information or reflectivity information.

10. A method for rendering an enhancement image onto pre-existing content comprising:

accessing a first database containing a plurality of enhancement images and information associated with said enhancement images including types of said enhancement images;

selecting said enhancement image;

accessing, after the step selecting said enhancement image, a second database containing a plurality of receptor sites and information associated with said receptor sites;

selecting a receptor site from said plurality of receptor sites based on said type of said enhancement image;

accessing enhancement image information for selecting either a receptor site property or an enhancement image property;

processing said enhancement image based on said receptor site information; and

rendering said processed enhancement image at said receptor site based on said enhancement image information.

11. The method of claim 10 wherein said step of processing said enhancement image further comprises:

scaling said enhancement image.

12. The method of claim 10 wherein said step of processing said enhancement image further comprises:

rotating said enhancement image.

13. The method of claim 10 wherein said step of processing said enhancement image further comprises:

warping said enhancement image.

14. The method of claim 10 wherein said step of processing said enhancement image further comprises:

warping said enhancement image to conform to the contour of a recognized object.

15. The method of claim 10 wherein said step of processing said enhancement image further comprises:

applying a coloration characteristic of said receptor site to said enhancement image.

16. A method for rendering an enhancement image onto pre-exist1ng content comprising:

accessing a first database containing a plurality of receptor sites and information associated with said receptor sites including sizes of said receptor sites;

selecting a receptor site from said plurality of receptor sites;

accessing a second database containing a plurality of enhancement images and information associated with said enhancement images including types of said enhancement images;

selecting an enhancement image from said plurality of enhancement images based on said size of said receptor site;

accessing enhancement image information;

processing said enhancement image based on said enhancement image information; and

rendering said processed enhancement image at said receptor site based on said enhancement image information.

17. The method of claim 16 wherein said step of processing said enhancement image further comprises:

scaling said enhancement image.

18. The method of claim 16 wherein said step of processing said enhancement image further comprises:

rotating said enhancement image.

19. The method of claim 16 wherein said step of processing said enhancement image further comprises:

warping said enhancement image.

20. The method of claim 16 wherein said step of processing said enhancement image further comprises:

warping said enhancement image to conform to the contour of a recognized object.

21. The method of claim 16 wherein said step of processing said enhancement image further comprises:

applying a coloration characteristic of said one receptor site to said enhancement image.

22. A method for identifying a receptor site in a scene of a program at which an enhancement image may be rendered comprising:

accessing said scene of said program;

recognizing an object in said scene, said object having an object characteristic;

defining a receptor site in said scene based upon recognition of said object; and

assigning a type to said receptor site;

determining a receptor site characteristic of said receptor site based on said object characteristic;

selecting an enhancement image from a plurality of enhancement images using said receptor site characteristic;

accessing enhancement image information;

accessing enhancement image information h decticg either =, receptu

processing said enhancement image based on said enhancement image information; and

replacing said object in said scene with said processed enhancement image by rendering said processed enhancement image at said defined receptor site based on said enhancement image information.

23. The method of claim 22 wherein said step of recognizing an object further comprises:

receiving an operator input.

24. The method of claim 22 wherein said step of recognizing an object further comprises:

applying a pattern recognition process to said scene.

25. The method of claim 22 further comprising:

determining a receptor site dimension, perspective, contour or illumination.

26. The method of claim 22 further comprising:

comparing the image of said recognized object with a model of said object and determining a scale or illumination characteristic of said object.

27. A machine-readable medium having program code stored thereon which, when executed by a machine, causes the machine to perform a method for rendering an enhancement image onto a scene, the method comprising:

detecting a receptor site for said enhancement;

determining a size of said receptor site;

selecting an enhancement image from a plurality of enhancement images based on said size of said receptor site;

accessing enhancement image information;

processing said enhancement image based on said enhancement image information; and

rendering said processed enhancement image at said position of said receptor site based on said enhancement image information.

28. The machine-readable medium of claim 27, wherein the method further comprises:

receiving an input from an operator viewing said scene.

29. The machine-readable medium of claim 27, wherein the method further comprises:

applying a pattern recognition process to said scene.

30. The machine-readable medium of claim 27, wherein said the method further comprises:

scaling said enhancement image.

31. The machine-readable medium of claim 27, wherein the method further comprises:

rotating said enhancement image.

32. The machine-readable medium of claim 27, wherein the method further comprises:

warping said enhancement image.

33. The machine-readable medium of claim 27, wherein the method further comprises:

warping said enhancement image to conform to the contour of a recognized object.

34. The machine-readable medium of claim 27, wherein the method further comprises:

determines determining a coloration characteristic of said receptor site.

35. The machine-readable medium of claim 34, wherein the method further comprises:

applying said coloration characteristic to said enhancement image.

36. The machine-readable medium of claim 27, wherein the method for selecting said enhancement image further comprises:

a secondary selection process.

37. The machine-readable medium of claim 36, wherein said secondary selection process further comprises:

employing viewer demographic information in selecting said enhancement image.

38. The machine-readable medium of claim 36, wherein said secondary selection process further comprises:

employing viewer preference information in selecting said enhancement image.

39. The machine-readable medium of claim 36, wherein said secondary selection process further comprises:

employing price information in selecting said enhancement image.

40. A system for rendering an enhancement image onto a scene of pre-existing program content comprising:

first access means for accessing a program scene;

second access means for accessing receptor site information for said program scene, said receptor site information including a size of said receptor site;

selection means for selecting an enhancement image based on said size of said receptor site;

accessing means for accessing enhancement image information;

processing means for processing said enhancement image based on said enhancement image information; and

rendering means for rendering said processed enhancement image at said receptor site based on said selected enhancement image information.

41. An interactive television system programmed to perform the following steps:

receive a program scene;

access receptor site information for said program scene, said receptor site information including a size of said receptor site;

select an enhancement image based upon said size of said receptor site;

access enhancement image information;

process said enhancement image based on said receptor site information; and

render said processed enhancement image based on said enhancement image information.