Summarization of soccer video content

• US 7,657,836 B2
• Filed: 09/27/2002
• Issued: 02/02/2010
• Est. Priority Date: 07/25/2002
• Status: Expired due to Fees

• Current Assignee: Sharp Kabushiki Kaisha (Hon Hai Precision Industry Co., Ltd.)
• Original Assignee: Sharp Laboratories of America Incorporated (Hon Hai Precision Industry Co., Ltd.)
• Inventors: Pan, Hao, Li, Baoxin
• Primary Examiner(s): Pitaro; Ryan F
• Application Number:
  US10/260,110
• Publication Number:
  US 20040017389A1
• Time in Patent Office: 2,685 Days
• Field of Search: 715/722, 715/724, 715/716
• US Class Current: 715/723
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06F 16/7834 using audio features
• CPC Class Codes: G06F 16/785 using colour or luminescence
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: H04N 21/233 Processing of audio elementary streams monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/235 Processing of additional data, e.g. scrambling of additional data or processing content descriptors
• CPC Class Codes: H04N 21/2662 Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
• CPC Class Codes: H04N 21/435 Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream
• CPC Class Codes: H04N 21/47202 for requesting content on demand, e.g. video on demand
• CPC Class Codes: H04N 21/84 Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73
• CPC Class Codes: H04N 21/8456 by decomposing the content in the time domain, e.g. in time segments
• CPC Class Codes: H04N 21/8543 using a description language, e.g. Multimedia and Hypermedia information coding Expert Group [MHEG], eXtensible Markup Language [XML] information retrieval of semistructured data, the underlying structure being taken into account, e.g. mark-up language structure data G06F16/80
• CPC Class Codes: H04N 21/8549 Creating video summaries, e.g. movie trailer retrieval in video databases by using presentations in form of a video summary G06F16/739

Abstract

Summarization of video content including soccer.

Claims

1. A method of automatically processing a video comprising a series of sequentially ordered frames and including soccer and that includes at least one replay segment corresponding to a live action segment, said replay segment replaying at least a portion of said live action segment from a different camera perspective than does said live action segment, said method comprising:

(a) automatically identifying a first plurality of first frames in said series of sequentially ordered frames that presents said replay segment;

(b) using the previously identified said replay segment, automatically identifying a second plurality of frames in said series of sequentially ordered frames that presents at least a portion of said live action segment; and

(c) creating a summarization of said video that includes said first and second plurality of frames, respectively, where said summarization includes fewer frames than does said video.

2. The method of claim 1 wherein the last frame of said live action segment is identified by detecting an amount of field color in a frame being less than a threshold, measured in a normalized color space.

3. The method of claim 2 wherein said last frame of said live action segment is identified based on detection of a scene change.

4. The method of claim 1 wherein the first frame of said live action segment is identified using a predetermined interval measured from an identified last frame of said live action segment.

5. A method of processing a video comprising a series of sequentially ordered frames and including soccer and that includes at least one replay segment replaying a live action segment, said method comprising:

(a) identifying a first plurality of frames in said series of sequentially ordered frames that presents said replay segment;

(b) using the previously identified said replay segment, automatically identifying a second plurality of frames in said series of sequentially ordered frames that presents at least a portion of said live action segment, wherein the first frame of said live action segment is identified using an automatically selected one of a plurality of predetermined intervals, each respectively measured from an identified last frame of said live action segment; and

(c) creating a summarization of said video by including said first and second plurality of frames, respectively, where said summarization includes fewer frames than said video.

6. The method of claim 5 wherein said video includes an audio signal and wherein selection of one of said plurality of said predetermined intervals is based on a comparison of the average energy of said audio signal respectively measured over each one of said plurality of intervals.

7. The method of claim 6 wherein said average energy is calculated using a Hamming window.

8. The method of claim 5 wherein said signal includes an audio signal including voice commentary and wherein selection of one of said plurality of predetermined intervals is based on an audio pitch contour of said voice commentary.

9. The method of claim 5 wherein selection of one of said plurality of predetermined intervals is based on the length of said replay segment.

10. The method of claim 5 wherein the smallest of said plurality of predetermined intervals is selected unless a criterion is met.

11. A method of processing a video comprising a series of sequentially ordered frames and including soccer and that includes at least one replay segment replaying a live action segment, said method comprising:

(a) automatically identifying a first plurality of frames in said series of sequentially ordered frames that presents said replay segment;

(b) using the previously identified said replay segment, automatically identifying a second plurality of frames in said series of sequentially ordered frames that presents at least a portion of said live action segment;

(c) automatically identifying a third plurality of frames in said series of sequentially ordered frames that presents an intervening segment of said video between said first plurality of frames and said second plurality of frames, where said third plurality of frames is also used to automatically identify said second plurality of frames;

(d) creating a summarization of said video by including said first, second, and third plurality of frames, where said summarization includes fewer frames than said video.

12. The method of claim 11 wherein said third segment is a close up of a participant in said soccer.

13. The method of claim 11 wherein said third segment is at least 25% of the combined duration of said first, second, and third segments.

14. The method of claim 11 wherein said third segment is at least 50% of the combined duration of said first, second, and third segments.

15. A method of processing a video comprising a series of sequentially ordered frames and including soccer and that includes at least one replay segment replaying a live action segment, said method comprising:

(a) automatically identifying a first plurality of frames in said series of sequentially ordered frames that presents said replay segment;

(b) using the previously identified said replay segment, automatically identifying a second plurality of frames in said series of sequentially ordered frames that presents at least a portion of said live action segment;

(c) creating a first summary of said video comprising said first plurality of frames and excluding said second plurality of frames;

(d) creating a second summary of said video comprising said first plurality of frames and said second plurality of frames; and

(e) presenting, at the selection of a user, a selected one of said first summary and said second summary.

16. The method of claim 15 wherein the first frame of said live action segment is identified using an automatically selected one of a plurality of predetermined intervals, each respectively measured from an identified last frame of said live action segment.

17. The method of claim 16 wherein said video includes an audio signal and wherein selection of one of said plurality of said predetermined intervals is based on a comparison of the average energy of said audio signal respectively measured over each one of said plurality of intervals.

18. The method of claim 16 wherein said signal includes an audio signal including voice commentary and wherein selection of one of said plurality of predetermined intervals is based on an audio pitch contour of said voice commentary.

19. The method of claim 16 wherein selection of one of said plurality of predetermined intervals is based on the length of said replay segment.

20. A method of processing a video comprising a series of sequentially ordered frames and including soccer and that includes at least one replay segment replaying a live action segment, said method comprising:

(a) automatically identifying a plurality of frames in said series of sequentially ordered frames that presents said replay segment;

(b) using the previously identified said replay segment, automatically identifying a second plurality of frames in said series of sequentially ordered frames that presents at least a portion of said live action segment;

(c) automatically identifying a third plurality of frames in said series of sequentially ordered frames that presents an intervening segment of said video between said first plurality of frames and said second plurality of frames;

(d) creating a first summary of said video comprising said first plurality of frames and excluding said second and third plurality of frames;

(e) creating a second summary of said video comprising said first and second plurality of frames and excluding said third plurality of frames;

(f) creating a third summary of said video comprising said first, second, and third plurality of frames; and

(g) presenting, at the selection of a user, a selected one of said first summary and said second summary and said third summary.

21. The method of claim 20 wherein the first frame of said live action segment is identified using an automatically selected one of a plurality of predetermined intervals, each respectively measured from an identified last frame of said live action segment.

22. The method of claim 21 wherein said video includes an audio signal and wherein selection of one of said plurality of said predetermined intervals is based on a comparison of the average energy of said audio signal respectively measured over each one of said plurality of intervals.

23. The method of claim 21 wherein said signal includes an audio signal including voice commentary and wherein selection of one of said plurality of predetermined intervals is based on an audio pitch contour of said voice commentary.

24. The method of claim 21 wherein selection of one of said plurality of predetermined intervals is based on the length of said replay segment.

25. The method of claim 20 wherein said intervening segment includes a close-up image of a participant in said soccer.

26. The method of claim 20 wherein said third segment is at least 25% of the combined duration of said first, second, and third segments.

27. The method of claim 20 wherein said third segment is at least 50% of the combined duration of said first, second, and third segments.

28. The method of claim 20 wherein said third segment is at least 75% of the combined duration of said first, second, and third segments.

29. The method of claim 20 wherein the last frame of said live action segment is identified by detecting an amount of field color in a frame being less than a threshold, measured in a normalized color space.

30. The method of claim 29 wherein said last frame of said live action segment is identified based on detection of a scene change.