Method and apparatus for insertion of additional content into video

• US 20060026628A1
• Filed: 07/29/2005
• Published: 02/02/2006
• Est. Priority Date: 07/30/2004
• Status: Abandoned Application

• Current Assignee: Chang Sheng Xu, Joo Hwee Lim, Kong Wah Wan, Xin Guo Yu
• Original Assignee: Chang Sheng Xu, Joo Hwee Lim, Kong Wah Wan, Xin Guo Yu
• Inventors: Lim, Joo Hwee, Yu, Xin Guo, Wan, Kong Wah, Xu, Chang Sheng
• Application Number:
  US11/192,590
• Publication Number:
  US 20060026628A1
• Time in Patent Office: Days
• US Class Current: 725/32
• CPC Class Codes: H04N 19/27 involving both synthetic and natural picture components, e.g. synthetic natural hybrid coding [SNHC]
• CPC Class Codes: H04N 21/235 Processing of additional data, e.g. scrambling of additional data or processing content descriptors
• CPC Class Codes: H04N 21/435 Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream
• CPC Class Codes: H04N 21/478 Supplemental services, e.g. displaying phone caller identification, shopping application
• CPC Class Codes: H04N 21/812 involving advertisement data advertising per se G06Q30/02
• CPC Class Codes: H04N 21/8456 by decomposing the content in the time domain, e.g. in time segments
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay
• CPC Class Codes: H04N 5/2723 Insertion of virtual advertisement; Replacing advertisements physical present in the scene by virtual advertisement data processing systems or methods specially adapted for marketing G06Q30/02
• CPC Class Codes: H04N 5/44504 Circuit details of the additional information generator, e.g. details of the character or graphics signal generator, overlay mixing circuits

Abstract

A method and apparatus inserts virtual advertisements or other virtual contents into a sequence of frames of a video presentation by performing real-time content-based video frame processing to identify suitable locations in the video for implantation. Such locations correspond to both the temporal segments within the video presentation and the regions within an image frame that are commonly considered to be of lesser relevance to the viewers of the video presentation. This invention presents a method and apparatus that allows a non-intrusive means to incorporate additional virtual content into a video presentation, facilitating an additional channel of communications to enhance greater video interactivity.

Claims

1. A method of inserting additional content into a video segment of a video stream, the video segment comprising a series of video frames, the method comprising:

receiving the video segment;

determining the frame content of at least one frame of the video segment;

determining the suitability of the frame for insertion of additional content, based on the determined frame content; and

inserting the additional content into the frames of the video segment depending on the determined suitability.

2. A method according to claim 1, wherein

determining the suitability of the frame for insertion comprises determining at least one first measure for the at least one frame indicative of the suitability of the frame for insertion of the additional content;

and

inserting the additional content depends on the determined at least one first measure.

3. A method according to claim 2, wherein the at least one first measure relative to the determined frame content is operator definable.

4. A method according to claim 2 or 3, wherein the at least one first measure indicative of the suitability of insertion of the additional content comprises a measure of the suitability of the frame for insertion of the additional content therein.

5. A method according to any one of claims 2 to 4, wherein the frame is determined to be suitable for insertion therein of additional content if the first measure is on a first side of a first threshold.

6. A method according to claim 5, wherein the frame is determined not to be suitable for insertion therein of additional content if the first measure is on a second side of the first threshold.

7. A method according to claim 1, further comprising:

determining the presence of at least one predetermined type of spatial region within the frames of the video segment; and

inserting the additional content into the video frames at a position depending on the predetermined type of spatial region determined to be present.

8. A method according to claim 7, wherein the presence of a predetermined type of spatial region is determined based on the determined frame content of at least one frame of the video segment.

9. A method according to claim 1, wherein the suitability of the frame for insertion is determined based on decision of the relevance of the frame to viewers.

10. A method according to claim 2, wherein the suitability of the frame for insertion is determined based on decision of the relevance of the frame to viewers;

and

wherein the at least one first measure comprises a first relevance-to-viewer measure, of the at least one frame.

11. A method according to claim 10, wherein the first relevance-to-viewer measure is an output derived from a table, with the frame content as an input to the table.

12. A method according to claim 1, further comprising determining how exciting the video segment is, and determining the suitability of the frame for insertion of additional content, is further based on how exciting the frame is determined to be.

13. A method according to claim 12 further comprising determining the suitability of the frame for insertion comprises determining at least one first measure for the at least one frame indicative of the suitability of the frame for insertion of the additional content;

and

inserting the additional content depends on the determined at least one first measure;

wherein the first relevance-to-viewer measure is derived from the frame content and from the determination as to how exciting the video segment is.

14. A method according to claim 13 wherein the first relevance-to-viewer measure is an output derived from a table, with the frame content as an input to the table;

and wherein the determination as to how exciting the video segment is comprises a further input to the table.

15. A method according to claim 12, wherein determining how exciting the video segment is comprises tracking the content of preceding video segments within the video stream.

16. A method according to claim 12, wherein determining how exciting the video segment is comprises analysing audio associated with the video segment.

17. A method according to claim 12, wherein determining how exciting the video segment is comprises analysing audio associated with preceding video segments within the video stream.

18. A method according to claim 1 further comprising pre-learning a plurality of parameters by analysing video segments of the same subject matter as the current video segment and using the pre-learned parameters to determine the suitability of the frame for insertion of additional content.

19. A method according to claim 18, wherein the pre-learned parameters are used to determine the at least one first measure.

20. A method according to claim 7, further comprising pre-learning a plurality of parameters by analysing video segments of the same subject matter as the current video segment and using the pre-learned parameters to determine the presence of the at least one predetermined type of spatial region.

21. A method according to any one of claims 18 to 20, further comprising modifying the use of the parameters based on an earlier portion of the video stream, preceding the current video segment.

22. A method according to claim 21, wherein determining the frame content of at least one frame of the video segment and determining the frame insertion suitability comprises performing content-based analysis of the video and the modified parameters to identify suitable frames and regions in the video segment for additional content insertion.

23. A method according to claim 1 further comprising selecting the additional content to be inserted prior to inserting the additional content.

24. A method according to claim 23, wherein selecting the additional content to be inserted is based on the size and/or the aspect ratio of the spatial region into which the additional content is to be inserted.

25. A method according to claim 1 further comprising detecting static spatial regions within the video stream and inserting further content into the detected static spatial regions.

26. A method according to claim 25, wherein if the further content inserted into the detected static spatial regions and the additional content overlap, the further content occludes the overlapping portion of the additional content.

28. A method according to any one of claims 25 to 27, wherein detecting static spatial regions comprises sampling and averaging pixel properties of a sequence of frames in the video stream to determine if pixels are stationary in the sequence of frames.

29. A method according to claim 28, wherein averaging comprises generating a time-lagged moving average.

30. A method according to any one of claims 25 to 27, wherein detecting static spatial regions comprises:

sampling pixel properties at image co-ordinates of a sequence of frames of the video stream in a time-lag window, the pixel properties comprising directional edge strength and pixel RGB intensity;

moving-average filtering the pixel properties at the same co-ordinates between frames to provide a change deviation over the time-lag window;

comparing the change deviation for different co-ordinates against a pre-defined threshold to determine if the pixels at the co-ordinates are stationary; and

determining regions of pixels so determined to be stationary.

40. Apparatus according to claim 38 or 39, operable according to the method of claim 1.

41. A computer program product for inserting additional content into a video segment of a video stream, the video segment comprising a series of video frames, the computer program product comprising:

a computer usable medium; and

a computer readable program code means embodied in the computer usable medium and for operating according to the method of claim 1.

27. A method of inserting further content into a video segment of a video stream, the video segment comprising a series of video frames, the method comprising:

receiving the video stream;

detecting static spatial regions within the video stream; and

inserting the further content into the detected static spatial regions.

31. A method according to claim 27, wherein determining the frame content comprises:

determining one or more dominant colours in the frame;

determining the size of interconnected regions of the same colour for one or more of the dominant colours in the frame; and

comparing the determined size against a relevant predetermined threshold

32. A method according to claim 31, wherein determining one or more dominant colours in the frame comprises classifying areas as green or non-green and comparing the determined size of the largest interconnected green area against a relevant predetermined threshold determines if the frame is of a field view.

33. A method according to claim 27, wherein the video stream is a live broadcast.

34. A method according to claim 27, wherein the video stream is a broadcast of a game.

35. A method according to claim 34, wherein the game is a game of association football.

36. A method according to claim 27, further comprising transmitting the video stream with the additional content therein to viewers.

37. Video integration apparatus operable according to the method of claim 27.

42. A computer program product for inserting additional content into a video segment of a video stream, the video segment comprising a series of video frames, the computer program product comprising:

a computer usable medium; and

a computer readable program code means embodied in the computer usable medium and which, when downloaded onto a computer, renders the computer into apparatus as according to any one of claims 37 to 40.

38. Video integration apparatus for inserting additional content into a video segment of a video stream, the video segment comprising a series of video frames, the apparatus comprising:

means for receiving the video segment;

means for determining the frame content of at least one frame of the video segment;

means for determining the suitability of the at least one frame for insertion of additional content, based on the determined frame content; and

means for inserting the additional content into the frames of the video segment depending on the determined suitability.

39. Video integration apparatus for inserting further content into a video segment of a video stream, the video segment comprising a series of video frames, the apparatus comprising:

means for receiving the video stream;

means for detecting static spatial regions within the video stream; and

means for inserting the further content into the detected static spatial regions.