Summarization of video content

• US 7,203,620 B2
• Filed: 05/23/2002
• Issued: 04/10/2007
• Est. Priority Date: 07/03/2001
• Status: Expired due to Fees

• Current Assignee: Sharp Kabushiki Kaisha (Hon Hai Precision Industry Co., Ltd.)
• Original Assignee: Sharp Laboratories of America Incorporated (Hon Hai Precision Industry Co., Ltd.)
• Inventors: Li, Baoxin
• Primary Examiner(s): Raymond; Edward
• Application Number:
  US10/155,358
• Publication Number:
  US 20030081937A1
• Time in Patent Office: 1,783 Days
• Field of Search: 702/181, 702182-185, 345/461, 345/473
• US Class Current: 702/181
• CPC Class Codes: A63B 2220/806 Video cameras
• CPC Class Codes: A63B 2220/807 Photo cameras
• CPC Class Codes: A63B 24/0003 Analysing the course of a movement or motion sequences during an exercise or trainings sequence, e.g. swing for golf or tennis
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06F 16/7834 using audio features
• CPC Class Codes: G06F 16/785 using colour or luminescence
• CPC Class Codes: G06F 16/786 using motion, e.g. object motion or camera motion
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/102 Programmed access in sequence to addressed parts of tracks of operating record carriers access by moving the head G11B3/08, G11B5/54, G11B7/085, G11B21/022; by moving the record carrier G11B15/005, G11B17/005, by driving of both record carrier and head G11B15/1816
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence

Abstract

Summarization of video content including sports.

Claims

1. A method of processing a video including sports comprising:

(a) identifying a plurality of segments of said video based upon an event, wherein said event is determined automatically by a computer based upon a probabilistic inference technique, where each of said segments includes a plurality of frames of said video; and

(b) creating a summarization of said video by including said plurality of segments, where said summarization includes fewer frames than said video.

2. The method of claim 1 wherein said probabilistic inference technique includes a Bayesian Network.

3. The method of claim 1 wherein said probabilistic inference technique includes a Hidden Markov Model.

4. The method of claim 1 wherein said probabilistic inference technique is applied to pre-identified segments.

5. The method of claim 1 wherein said probabilistic inference technique is free from being applied to pre-identified segments.

6. The method of claim 1 wherein said probabilistic inference technique uses training sequences of segments.

7. A method of processing a video including sports comprising:

(a) identifying a plurality of segments of said video, where each of said segments is automatically identified by a computer includes a plurality of frames of said video, based upon a series of activities defined by the rules of said sports that could potentially result in at least one of;

(i) a score;

(ii) preventing a score; and

(b) creating a summarization of said video by including said plurality of segments, where said summarization includes fewer frames than said video.

8. The method of claim 7 wherein said summarization of said plurality of segments is in the same temporal order as said plurality of segments within said video.

9. The method of claim 7 wherein said activities are determined based upon the color characteristics of said video.

10. The method of claim 7 wherein said activities are determined based upon scene changes.

11. A method of processing a video including sports comprising:

(a) identifying a plurality of segments of said video based upon detecting a play of said sports, wherein said identifying includes a computer automatically detecting the start and end of said play, where each of said segments includes a plurality of frames of said video; and

(b) creating a summarization of said video by including said plurality of segments, where said summarization includes fewer frames than said video.

12. The method of claim 11 wherein said detecting the end of said play is based upon detecting said start of said play.

13. The method of claim 11 wherein said summarization identifies said plurality of segments of said video.

14. The method of claim 11 wherein said summarization is a summarized video comprising said plurality of segments excluding at least a portion of said video other than said plurality of segments.

15. A method of processing a video including sports comprising:

(a) identifying a plurality of segments of said video, wherein the start of said plurality of segments is identified automatically by a computer based upon a frame of said video having color characteristics, where each of said segments includes a plurality of frames of said video; and

(b) creating a summarization of said video by including said plurality of segments, where said summarization includes fewer frames than said video.

16. The method of claim 15 wherein said color characteristics are based on spatial regions having sufficient color homogeneity.

17. The method of claim 15 further comprising said color characteristics are based upon different spatially displaced regions having different colors.

18. A method of processing a video including sports comprising:

(a) identifying a plurality of segments of said video, wherein the start of said plurality of segments is identified automatically by a computer based upon anticipated characteristics of said plurality of segments from an anticipated camera angle used to obtain said start of said plurality of segments, where each of said segments includes a plurality of frames of said video; and

(b) creating a summarization of said video by including said plurality of segments, where said summarization includes fewer frames than said video.

19. The method of claim 18 wherein said characteristics includes spatially displaced regions of predetermined characteristics.

20. The method of claim 18 wherein said characteristics includes predefined motion characteristics.

21. The method of claim 18 wherein the end of said plurality of segments is based upon a scene change.

22. A method of processing a video including sports comprising:

(a) identifying a plurality of segments of said video automatically by a computer wherein the start of said plurality of segments is based upon a play of said sports, where each of said segments includes a plurality of frames of said video;

(b) creating a first summarization of said video by including said plurality of segments, where said summarization includes fewer frames than said video; and

(c) creating another summarization of said video from said plurality of segments, wherein said another summarization has a duration shorter than said first summarization.

23. The method of claim 22 wherein said another summarization is selected based upon audio information.

24. The method of claim 22 wherein said another summarization is selected based upon textual information.

25. The method of claim 22 further comprising creating a further summarization of said video from said plurality of segments, wherein said further summarization has a duration shorter than said another summarization.

26. A method of processing a video including sports comprising:

(a) identifying a plurality of segments of said video wherein the start of said plurality of segments is based upon a play of said sports, where each of said segments includes a plurality of frames of said video;

(b) creating a summary of said video from said plurality of segments, wherein said summary has a duration based upon the bandwidth requirements of the transmitting said summary to a user.

27. The method of claim 26 including transmitting to said user through a wireless connection.

28. The method of claim 26 including transmitting to said user through the Internet.

29. The method of claim 26 wherein said summary is based upon a user profile of said user.

30. A method of processing a video including sports comprising:

(a) identifying a plurality of segments of said video wherein the start of said plurality of segments is based upon a play of said sports, where each of said segments includes a plurality of frames of said video;

(b) creating a summary of said video from said plurality of segments, wherein said summary includes fewer frames than said sports video, wherein said plurality of segments were selected based upon a caption module of said sports video.

31. The method of claim 30 wherein said caption module detects captions within said video.

32. The method of claim 31 wherein said caption module recognizes information with said caption.

33. A method of processing a video including sports comprising:

(a) identifying a plurality of segments of said video automatically by a computer wherein said identifying for at least one of said segments includes detecting the start of said segment based upon processing of a first single frame of said video, where each of said segments includes a plurality of frames of said video;

(b) verifying that said first single frame is an appropriate start of said segment based upon processing of another single frame temporally relevant to said first single frame; and

(c) creating a summarization of said video by including said plurality of segments, where said summarization includes fewer frames than said video.

34. A method of processing a video including sports comprising:

(a) identifying a plurality of segments of said sports video automatically by a computer, wherein said identifying for the end of at least one of said segments is based upon detecting a scene change, where each of said segments includes a plurality of frames of said sports video; and

(b) creating a summarization of said sports video by including said plurality of segments, where said summarization includes fewer frames than said sports video.

35. The method of claim 34 wherein said scene change is based upon a threshold between at least two frames.

36. The method of claim 34 wherein said scene change is based upon a gradual transition below a threshold level.

37. A method of processing a video including sports comprising:

(a) identifying a plurality of segments of said video automatically by a computer, where each of said segments includes a plurality of frames of said video;

(b) identifying a plurality of segments, automatically by a computer, that are temporally separated by a sufficiently short duration;

(c) based upon said identifying as a result of (b) connecting said identified plurality of segments; and

(d) creating a summarization of said video by including said plurality of segments, where said summarization includes fewer frames than said video.

38. The method of claim 37 wherein said connecting includes discarding the frames of said video between said identified plurality of segments.

39. The method of claim 37 wherein said connecting results in a single segment that includes said identified plurality of segments together with the frames of said video between said identified plurality of segments.

40. method of processing a video including sports comprising:

(a) identifying a plurality of segments of said video automatically by a computer, where each of said segments includes a plurality of frames of said video;

(b) identifying at least one of said segments, automatically by a computer, that has a temporally sufficiently short duration;

(c) based upon said identifying as a result of (b) removing said identified segment from said summarization; and

(d) creating a summarization of said video by including said plurality of segments, where said summarization includes fewer frames than said video.

41. The method of claim 40 wherein said connecting includes discarding the frames of said video between said identified plurality of segments.

42. The method of claim 40 wherein said connecting results in a single segment that includes said identified plurality of segments together with the frames of said video between said identified plurality of segments.

43. A method of processing a video including sports comprising:

(a) identifying a plurality of segments of said video automatically by a computer wherein each of said segments includes a play of said sports, wherein said segments include full-speed plays and slow motion plays of said full-speed plays; and

(b) creating a summarization of said video by including said plurality of segments, where said summarization includes fewer frames than said video, where a user may select from;

(i)said summarization including only full-speed plays;

(ii)said summarization including only slow motion plays;

(iii) said summarization including both full-speed plays and slow motion plays.

44. A method of processing a video including sports comprising:

(a) identifying a plurality of segments of said video automatically by a computer wherein each of said segments includes a play of said sports;

(b) creating a summarization of said video by including said plurality of segments, where said summarization includes fewer frames than said video; and

(c) removing at least one of said segments from said summary based, at least in part, upon audio information related to said at least one of said segments.

45. The method of claim 44 wherein said audio information is obtained exclusively from a temporal analysis.

46. A method of processing a video including sports comprising:

(a) identifying a plurality of segments of said video automatically by a computer wherein each of said segments includes a play of said sports;

(b) creating a summarization of said video by including said plurality of segments, where said summarization includes fewer frames than said video; and

(c) modifying the duration of at least one of said segments from said summary based, at least in part, upon audio information related to said at least one of said segments.

47. The method of claim 46 wherein said audio information is obtained exclusively from a temporal analysis.

48. A method of processing a video comprising:

(a) providing a description of said video;

(b) identifying a plurality of segments of said video automatically by a computer based upon said description;

(c) generating another video based upon said identified segments, wherein said another video has less frames than said video.

49. The method of claim 48 wherein said video is transcoded in accordance with the channel capacity of a transmission medium.

50. The method of claim 48 wherein said video is transcoded in accordance with the storage capacity of a storage device.

51. The method of claim 48 wherein said description is compliant with MPEG-7 standard.

52. The method of claim 48 wherein said description is compliant with TV-Anytime standard.

53. A method of processing a video including sports comprising:

(a) identifying a plurality of segments of said video wherein each of said segments includes a play of said sports;

(b) detecting a commercial within said video; and

(c) creating a summarization of said video by including said plurality of segments, where said summarization includes fewer frames than said video, where said plurality of segments of said video sufficiently temporally close to said commercial are free from being included within said summarization.