Indexing of media content on a network

• US 6,282,549 B1
• Filed: 03/29/1999
• Issued: 08/28/2001
• Est. Priority Date: 05/24/1996
• Status: Expired due to Term

• Current Assignee: The Procter & Gamble Company
• Original Assignee: Magnifi, Inc. (Emmperative Marketing, Inc.)
• Inventors: Cremin, Karl, Hoffert, Eric M., Smoot, Stephen R., Ali, Adnan
• Primary Examiner(s): Alam, Hosain T.
• Assistant Examiner(s): Colbert, Ella
• Application Number:
  US09/280,173
• Time in Patent Office: 883 Days
• Field of Search: 707/101, 707/102, 707/103, 707/104, 707/500, 707/501, 707/513, 707/1-3, 345/339, 348/17, 382/296, 725/107, 725/110, 725/111, 725/112, 725/113, 725/37
• US Class Current: 1/1
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/683 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/685 using automatically derived transcript of audio data, e.g. lyrics speech recognition G10L15/00
• CPC Class Codes: G06F 16/78 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/7834 using audio features
• CPC Class Codes: G06F 16/7844 using original textual content or text extracted from visual content or transcript of audio data
• CPC Class Codes: G06F 16/785 using colour or luminescence
• CPC Class Codes: G06F 16/786 using motion, e.g. object motion or camera motion
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G10H 1/0058 Transmission between separate instruments or between individual components of a musical system G10H1/0083 takes precedence
• CPC Class Codes: G10H 2210/046 for differentiation between music and non-music signals, based on the identification of musical parameters, e.g. based on tempo detection
• CPC Class Codes: G10H 2240/056 MIDI or other note-oriented file format
• CPC Class Codes: G10H 2240/066 MPEG audio-visual compression file formats, e.g. MPEG-4 for coding of audio-visual objects MP3 G10H2240/061
• CPC Class Codes: G10H 2240/135 Library retrieval index, i.e. using an indexing scheme to efficiently retrieve a music piece
• CPC Class Codes: G10H 2240/305 Internet or TCP/IP protocol use for any electrophonic musical instrument data or musical parameter transmission purposes
• CPC Class Codes: Y10S 707/99931 Database or file accessing
• CPC Class Codes: Y10S 707/99932 Access augmentation or optimizing
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99942 : Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99943 : Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99944 : Object-oriented database structure
• CPC Class Codes: Y10S 707/99945 : Object-oriented database structure processing
• CPC Class Codes: Y10S 707/99948 : Application of database or data structure, e.g. distributed, multimedia, or image

Abstract

A method and apparatus for searching for multimedia files in a distributed database and for displaying results of the search based on the context and content of the multimedia files.

Claims

1. A method comprising:

a computer examining Hyper Text Markup Language (HTML) to locate media, by

locating an HTML page and scanning the page for predetermined HTML tags;

for each HTML object, determining if there is a media Uniform Resource Locator (URL);

the computer locating lexical information near the media Uniform Resource Locator (URL) in the HTML, by identifying relevant text from the HTML;

examining the media object for lexical information;

the computer generating a media index for later use in retrieval of the media, by categorizing the content of the media based on both video motion activity the media index based upon the lexical information near the media and the content of the media; and

the computer weighting the media index based on a set of predetermined weighting values.

2. The method of claim 1, wherein the identifying relevant text from the HTML comprises:

using text from the associated table element, if the media URL is in a table;

using text from the associated list element, if the media URL is in a list; and

using text from headings associated with the media.

3. The method of claim 1, wherein said examining the media object comprises examining the media file header for textual information and examining the media file content for media information.

4. The method of claim 3, wherein said examining the media file content for media information comprises examining for closed captioning text.

5. The method of claim 1, wherein said examining the media object for text further comprises streaming the media file and examining the streaming media for text.

6. The method of claim 1, wherein predetermined values are set for:

text in the URL of the media file;

text embedded in the media file;

text annotations in the media file; and

script dialogue, lyrics, and closed captioning in the media file.

7. The method of claim 1 further comprising:

decompressing frames of the media object at N points wherein N is determined based on the size of the preview and the size of images to be displayed in the preview;

analyzing each of the frames to determine if the frames meet a predetermined value and if the frames do meet the predetermined value, selecting the frame for display; and

selecting a substitute frame, if one of the frames do not meet the predetermined value.

8. The method of claim 7 further comprising:

scaling each of the selected frames;

determining a predetermined value for the frames.

9. A method comprising:

searching a network for media content;

by examining HyperText Markup Language (HTML lexical information surrounding media objects;

examining the media objects for lexical information and non-lexical information;

indexing the media content based upon the context and content of media objects;

using lexical and non-lexical information associated with the media objects and

building previews of the media content for retrieval by indexing media content based upon categorization of the media content according to video motion activity and audio amplitude activity, the previews including information allowing display feedback of the information at predetermined locations.

10. The method of claim 9 further comprising:

computing an aspect ratio A;

determining a target height IH of the preview image as the preview height FH less the height of any top and bottom border;

determining a target IW of the preview image as a function of the target height IH and the aspect ratio A.

11. The method of claim 10 further comprising computing the number of images for display as the preview width divided by the sum of the target width IW and any spacing between images.

12. The method of claim 9, comprising

using text from an associated table element, if the media URL is in a table;

using text from an associated list element;

if the media URL is in a list; and

using text from headings associated with the media objects.

13. The method of claim 9 further comprising examining a media file header for textual information and examining the media file content for media information.

14. The method of claim 13, wherein the examining the media file content further comprises examining for closed captioning text.

15. The method of claim 9, comprising streaming a media file and examining the streaming media for text.

16. The method of claim 9, wherein the non-lexical information comprises brightness, contrast, and color.

17. The method of claim 9, wherein the non-lexical information comprises motion and chrominance.

18. The method of claim 9, wherein the preview comprises snippets of audio information.

19. A method comprising:

searching a network for media files, by examining HyperText Markup Language (HTML) lexical information surrounding references to media files;

analyzing the content of the media files for lexical information and non-lexical information; and

indexing the media files based on the content of the media files and the context surrounding references to the media files by categorizing the content of the media files based on both video motion activity and audio amplitude activity.

20. The method of claim 19, wherein the context indexing is based on weighted values associated with the location of the lexical information relative to the reference to the media file.

21. The method of claim 19, wherein the media file comprises viewable media, and the media file indexing is based on brightness, contrast, and color.

22. The method of claim 19, wherein the media file comprises viewable media, and the media indexing is based on chrominance.

23. The method of claim 19, wherein the relative motion is determined by examining differences between frames in the video.

24. The method of claim 19, wherein the video motion activity is determined by a motion metric and a brightness metric.

25. The method of claim 24, wherein the motion metric is an estimate of the type of content found in a video file.

26. The method of claim 25, wherein the video file may contain a low, medium, or high degree of motion.

27. The method of claim 26, wherein the low degree of motion is indicative of a talk show.

28. The method of claim 26, wherein the medium degree of motion is indicative of newscasts or commercials.

29. The method of claim 26, wherein the high degree of motion is indicative of action films and sports.

30. The method of claim 24, wherein the brightness metric is an estimate of the brightness content found in the video file.

31. The method of claim 30, wherein the video file may contain a dark, moderate or bright intensity of brightness.

32. The method of claim 24, wherein the motion metric and brightness metric, combined, may enhance a user ability to search a video.

33. The method of claim 19, wherein the audio amplitude activity is determined by a music-speech metric.

34. The method of claim 33, wherein the music-speech metric is an estimate of the type of content found in an audio file.

35. The method of claim 34, wherein the audio file may contain a low, moderate, or high rate of change in amplitude activity.

36. The method of claim 35, wherein the low rate of change in amplitude activity is indicative of music signals.

37. The method of claim 35, wherein the moderate rate of change in amplitude activity is indicative of music and speech signals.

38. The method of claim 35, wherein the high rate of change in amplitude activity is indicative of speech signals.

39. The method of claim 34, wherein the audio file may contain a continuous or discontinuous signal.

40. The method of claim 39, wherein the continuous signal is indicative of music signals.

41. The method of claim 40, wherein the discontinuous signal is indicative of speech signals.