Systems and methods for smart media content thumbnail extraction

• US 20060026524A1
• Filed: 08/02/2004
• Published: 02/02/2006
• Est. Priority Date: 08/02/2004
• Status: Active Grant

• Current Assignee: Zhigu Holdings Limited
• Original Assignee: Microsoft Corporation
• Inventors: Ma, Wei-Ying, Lin, Bin, Zhang, Hong-Jiang, Zou, Xinli, Kong, Zhike, Ma, Yu-Fei
• Granted Patent: US 7,986,372 B2
• Time in Patent Office: Days
• US Class Current: 715/713
• CPC Class Codes: G06F 16/743 a collection of video files or sequences
• CPC Class Codes: G06F 16/784 the detected or recognised objects being people
• CPC Class Codes: G06F 16/785 using colour or luminescence
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence

Abstract

Systems and methods for smart media content thumbnail extraction are described. In one aspect program metadata is generated from recorded video content. The program metadata includes one or more key-frames from one or more corresponding shots. An objectively representative key-frame is identified from among the key-frames as a function of shot duration and frequency of appearance of key-frame content across multiple shots. The objectively representative key-frame is an image frame representative of the recorded video content. A thumbnail is created from the objectively representative key-frame.

Claims

1. A method comprising:

generating program metadata from recorded video content, the program metadata comprising one or more key-frames from one or more corresponding shots;

identifying an objectively representative key-frame from the key-frames as a function of shot duration and frequency of appearance of key-frame content across multiple shots, the objectively representative key-frame being an image frame representative of the recorded video content; and

creating a thumbnail from the objectively representative key-frame.

2. A method as recited in claim 1, wherein the recorded video content is a recorded television program.

3. A method as recited in claim 1, wherein generating further comprises:

decomposing the recorded video content into multiple shots;

deriving a set of sub-shots from each of the multiple shots; and

for each set of sub-shots, identifying a respective key-frame of the key-frames.

4. A method as recited in claim 1, wherein identifying further comprises selecting the objectively representative key-frame as a further function of low motion intensity.

5. A method as recited in claim 1, wherein identifying further comprises:

for each shot of the shots;

(a) determining whether the shot is of short or long duration relative to other ones of the shots;

(b) evaluating whether a key-frame of the shot is of objective high image quality;

(c) detecting whether the shot represents commercial content;

in view of the determining, evaluating, and detecting, removing shot(s) of short duration, objectively low image quality, or that include commercial content from the program metadata.

6. A method as recited in claim 1, wherein identifying further comprises:

clustering the shots based on appearance representation to generate one or more clusters; and

if a key-frame associated with a largest cluster of clusters includes a human face, selecting a key-frame with a largest facial area as the objectively representative key-frame.

7. A method as recited in claim 1, wherein identifying further comprises:

clustering the shots based on appearance representation to generate one or more clusters; and

if a key-frame associated with a largest cluster of clusters does not include a human face, selecting a key-frame with an attended area of objectively determined high attention ranking as the objectively representative key-frame.

8. A method as recited in claim 1, the method further comprising presenting the thumbnail as a representative identifier of the recorded video content.

9. A computer-readable medium comprising computer-program instructions executable by a processor for:

generating program metadata from recorded video content, the program metadata comprising one or more key-frames from one or more corresponding shots;

identifying an objectively representative key-frame from the key-frames as a function of shot duration and frequency of appearance of key-frame content across multiple shots, the objectively representative key-frame being an image frame representative of the recorded video content; and

creating a thumbnail from the objectively representative key-frame.

10. A computer-readable medium as recited in claim 9, wherein the recorded video content is a recorded television program.

11. A computer-readable medium as recited in claim 9, wherein the computer-program instructions for generating further comprise instructions for:

decomposing the recorded video content into multiple shots;

deriving a set of sub-shots from each of the multiple shots; and

for each set of sub-shots, identifying a respective key-frame of the key-frames.

12. A computer-readable medium as recited in claim 9, wherein the computer-program instructions for identifying further comprise instruction for selecting the objectively representative key-frame as a further function of low motion intensity.

13. A computer-readable medium as recited in claim 9, wherein the computer-program instructions for identifying further comprise instructions for:

for each shot of the shots;

(a) determining whether the shot is of short or long duration relative to other ones of the shots;

(b) evaluating whether a key-frame of the shot is of objective high image quality;

(c) detecting whether the shot represents commercial content;

in view of the determining, evaluating, and detecting, removing shot(s) of short duration, objectively low image quality, or that include commercial content from the program metadata.

14. A computer-readable medium as recited in claim 9, wherein the computer-program instructions for identifying further comprise instructions for:

clustering the shots based on appearance representation to generate one or more clusters; and

if a key-frame associated with a largest cluster of clusters includes a human face, selecting a key-frame with a largest facial area as the objectively representative key-frame.

15. A computer-readable medium as recited in claim 9, wherein the computer-program instructions for identifying further comprise instructions for:

clustering the shots based on appearance representation to generate one or more clusters; and

if a key-frame associated with a largest cluster of clusters does not include a human face, selecting a key-frame with an attended area of objectively determined high attention ranking as the objectively representative key-frame.

16. A computer-readable medium as recited in claim 9, wherein the computer-program instructions further comprise instructions for presenting the thumbnail as a representative identifier of the recorded video content.

20. A computer-readable medium as recited in claim 9, wherein the computer-program instructions for identifying further comprise instruction for selecting the objectively representative key-frame as a further function of low motion intensity.

17. A computing device comprising:

a processor; and

a memory coupled to the processor, the memory comprising computer-program instructions executable by the processor for;

generating program metadata from recorded video content, the program metadata comprising one or more key-frames from one or more corresponding shots;

identifying an objectively representative key-frame from the key-frames as a function of shot duration and frequency of appearance of key-frame content across multiple shots, the objectively representative key-frame being an image frame representative of the recorded video content; and

creating a thumbnail from the objectively representative key-frame.

18. A computing device as recited in claim 17, wherein the recorded video content is a recorded television program.

19. A computing device as recited in claim 17, wherein the computer-program instructions for generating further comprise instructions for:

decomposing the recorded video content into multiple shots;

deriving a set of sub-shots from each of the multiple shots; and

for each set of sub-shots, identifying a respective key-frame of the key-frames.

21. A computing device as recited in claim 17, wherein the computer-program instructions for identifying further comprise instructions for:

for each shot of the shots;

(a) determining whether the shot is of short or long duration relative to other ones of the shots;

(b) evaluating whether a key-frame of the shot is of objective high image quality;

(c) detecting whether the shot represents commercial content;

in view of the determining, evaluating, and detecting, removing shot(s) of short duration, objectively low image quality, or that include commercial content from the program metadata.

22. A computing device as recited in claim 17, wherein the computer-program instructions for identifying further comprise instructions for:

clustering the shots based on appearance representation to generate one or more clusters; and

if a key-frame associated with a largest cluster of clusters includes a human face, selecting a key-frame with a largest facial area as the objectively representative key-frame.

23. A computing device as recited in claim 17, wherein the computer-program instructions for identifying further comprise instructions for:

clustering the shots based on appearance representation to generate one or more clusters; and

if a key-frame associated with a largest cluster of clusters does not include a human face, selecting a key-frame with an attended area of objectively determined high attention ranking as the objectively representative key-frame.

24. A computing device as recited in claim 17, wherein the computer-program instructions further comprise instructions for presenting the thumbnail as a representative identifier of the recorded video content.

25. A computing device comprising:

generating means to generate program metadata from recorded video content, the program metadata comprising one or more key-frames from one or more corresponding shots; and

identifying means to identify an objectively representative key-frame from the key-frames as a function of shot duration and frequency of appearance of key-frame content across multiple shots, the objectively representative key-frame being an image frame representative of the recorded video content.

26. A computing device as recited in claim 25, wherein the generating means further comprise:

decomposing means to decompose the recorded video content into multiple shots;

deriving means to determine a set of sub-shots from each of the multiple shots; and

for each set of sub-shots, identifying means to identify a respective key-frame of the key-frames.

27. A computing device as recited in claim 25, wherein the identifying means further comprise:

for each shot of the shots;

(a) determining means to determine whether the shot is of short or long duration relative to other ones of the shots;

(b) evaluating means to evaluate whether a key-frame of the shot is of objective high image quality;

(c) detecting means to detect whether the shot represents commercial content;

in view of the determining, evaluating, and detecting means, removing means to remove shot(s) of short duration, objectively low image quality, or that include commercial content from the program metadata.

28. A computing device as recited in claim 25, wherein the identifying means further comprises:

clustering means to cluster the shots based on appearance representation to generate one or more clusters; and

if a key-frame associated with a largest cluster of clusters includes a human face, selecting means to select a key-frame with a largest facial area as the objectively representative key-frame.

29. A computing device as recited in claim 25, wherein the identifying means further comprises:

clustering means to cluster the shots based on appearance representation to generate one or more clusters; and

if a key-frame associated with a largest cluster of clusters does not include a human face, selecting means to select a key-frame with an attended area of objectively determined high attention ranking as the objectively representative key-frame.

30. A computing device as recited in claim 25, further comprising presenting means to present the thumbnail as a representative identifier of the recorded video content.