System and method for indexing, searching, identifying, and editing portions of electronic multimedia files

• US 20020069218A1
• Filed: 07/23/2001
• Published: 06/06/2002
• Est. Priority Date: 07/24/2000
• Status: Active Grant

• Current Assignee: Scenera, Inc.
• Original Assignee: VMark, Inc. (VALID Soluciones Tecnológicas SA)
• Inventors: Lee, Keansub, Oh, Jeongtaek, Song, S. Moon-Ho, Kim, Yunam, Lee, Sangwook, Oh, Sangwook, Chung, Min Gyo, Kim, Hyeokman, Sull, Sanghoon, Chun, Seong Soo, Choi, Hyungseok, Kim, Jung Rim, Yoon, Ja-Cheon
• Granted Patent: US 7,624,337 B2
• Time in Patent Office: Days
• US Class Current: 715/501.1
• CPC Class Codes: G06F 16/71 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/7844 using original textual content or text extracted from visual content or transcript of audio data
• CPC Class Codes: G06F 16/7847 using low-level visual features of the video content
• CPC Class Codes: G06F 16/7867 using information manually generated, e.g. tags, keywords, comments, title and artist information, manually generated time, location and usage information, user ratings
• CPC Class Codes: G06T 3/4092 Image resolution transcoding, e.g. by using client-server architectures
• CPC Class Codes: G11B 2220/20 Disc-shaped record carriers
• CPC Class Codes: G11B 2220/41 Flat as opposed to hierarchical combination, e.g. library of tapes or discs, CD changer, or groups of record carriers that together store one title
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: G11B 27/105 of operating discs
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D

Abstract

A method and system are provided for tagging, indexing, searching, retrieving, manipulating, and editing video images on a wide area network such as the Internet. A first set of methods is provided for enabling users to add bookmarks to multimedia files, such as movies, and audio files, such as music. The multimedia bookmark facilitates the searching of portions or segments of multimedia files, particularly when used in conjunction with a search engine. Additional methods are provided that reformat a video image for use on a variety of devices that have a wide range of resolutions by selecting some material (in the case of smaller resolutions) or more material (in the case of larger resolutions) from the same multimedia file. Still more methods are provided for interrogating images that contain textual information (in graphical form) so that the text may be copied to a tag or bookmark that can itself be indexed and searched to facilitate later retrieval via a search engine.

Claims

1. A system for accessing multimedia content stored in a multimedia file having a beginning and an intermediate point, the content having at least one segment at the intermediate point, the system comprising:

a multimedia bookmark, the multimedia bookmark having content information about the segment at the intermediate point;

wherein a user can utilize the multimedia bookmark to access the segment without accessing the beginning of the multimedia file.

2. The system of claim 1 further comprising a search mechanism that locates the segment in the multimedia file.

3. The system of claim 2 further comprising an access mechanism that reads the multimedia content at the segment designated by the multimedia bookmark.

4. The system of claim 1, wherein the multimedia content is partial data related to a particular at least one segment.

5. The system of claim 1, wherein the multimedia content is visual data comprising one or more frames of video.

6. The system of claim 1, wherein the multimedia content is audio data.

7. The system of claim 1, wherein the multimedia content is a string of characters.

8. The system of claim 1, wherein the multimedia bookmark further comprises positional information about the segment.

9. The system of claim 8, wherein the positional information is a URI.

10. The system of claim 8, wherein the positional information includes an elapsed time.

11. The system of claim 8, wherein the positional information includes a time code.

12. The system of claim 1, wherein the multimedia file is contained on local storage.

13. The system of claim 12, wherein the local storage includes a database.

14. The system of claim 1, wherein the multimedia file is stored on a device accessible via a network.

15. The system of claim 14, wherein the network is the Internet.

17. The system of claim 16 further comprising a search mechanism that locates the segment in the multimedia file.

18. The system of claim 17 further comprising an access mechanism that reads the multimedia content at the segment designated by the multimedia bookmark.

19. The system of claim 16, wherein the at least two variations are accessible from a network.

20. The system of claim 19, wherein the network is the Internet.

21. The system of claim 16, wherein the multimedia bookmark is accessible from a network.

22. The system of claim 21, wherein the multimedia bookmark is stored in a database.

23. The system of claim 21, wherein the multimedia bookmark is indexed in a search engine.

24. The system of claim 16 further comprising metadata constructed and arranged to store a media profile for each variation file, the media profile containing offset information representing a start time and an end time of the segment that is correlated with the master file.

25. The system of claim 24, wherein the offset information of a variation file is calculated by aligning a referential segment between two different time points from the master file and the variation file.

26. The system of claim 25, wherein the master file is a video.

27. The system of claim 26, wherein the referential segment is between two successive shot boundaries.

28. The system of claim 16 wherein the multimedia bookmark can be copied.

29. The system of claim 28, wherein the multimedia bookmark can be e-mailed.

16. A system for accessing multimedia content encoded in a master file having a beginning point and an end point and at least one variation file derived from the master file, the system comprising:

a segment of the file having a beginning point after the beginning point of the master file and an end point before the end point of the master file that are designated by a user;

a multimedia bookmark, the multimedia bookmark having content information about the segment;

wherein the user can access the same segment on the master file and the variation file via the multimedia bookmark.

30. A method of enabling access to multimedia content having a beginning point and an intermediate point, the intermediate point starting a segment of the multimedia content that is designated by a user, the method comprising:

saving content information describing the segment in a multimedia bookmark.

31. The method of claim 30 further comprising:

searching for a segment that matches content information criteria.

32. The method of claim 30 further comprising:

accessing the segment multimedia content matching the content information criteria.

33. A method of enabling access to multimedia content having a beginning point and an intermediate point, the intermediate point starting a segment of the multimedia content that is designated by a user, the method comprising:

selecting a multimedia content from a server;

playing the multimedia content downloaded from the server by a user;

receiving at the server an add-bookmark command from the user;

saving content information pertaining to a segment of the multimedia content designated by the user;

displaying a bookmarked position of the multimedia content;

searching for a multimedia file satisfying search criteria of content information;

accessing multimedia content starting from the segment having content information matching the search criteria.

34. The system of claim 33, wherein the multimedia content is partial data related to a particular at least one segment.

35. The system of claim 33, wherein the multimedia content is visual data comprising one or more frames of video.

36. The system of claim 33, wherein the multimedia content is audio data.

37. The system of claim 33, wherein the multimedia content is a string of characters.

38. The system of claim 33, wherein the multimedia bookmark further comprises positional information about the segment.

39. The system of claim 38, wherein the positional information is a URI.

40. The system of claim 38, wherein the positional information includes an elapsed time.

41. The system of claim 38, wherein the positional information includes a time code.

42. The system of claim 41, wherein the multimedia file is contained on local storage.

43. The system of claim 42, wherein the local storage includes a database.

44. The system of claim 33, wherein the multimedia file is stored on a device accessible via a network.

45. The system of claim 44, wherein the network is the Internet.

49. The method of claim 48, the method further comprising:

determining if another segment is to be selected; and

if another segment is to be selected, then performing the step of selecting a segment in a metafile.

50. The method of claim 49, the method further comprising:

determining if another metafile is to be browsed; and

if another metafile is to be browsed, then performing the step of providing a metafile.

51. The method of claim 46 wherein the metafile is an XML file.

52. The method of claim 47 wherein the metafile is an XML file.

53. The method of claim 48 wherein the metafile is an XML file.

46. A method for virtual editing multimedia files, the method comprising:

providing one or more video files;

creating a metadata file for each of the video files, each of the metadata files having at least one segment to be edited; and

creating a single edited metafile containing the segments to be edited from each of the metadata files;

wherein when the edited metadata file is accessed, the user is able to play the segments to be edited in the edited order.

47. A method for virtual editing multimedia files, the method comprising:

providing one or more video files;

creating a metadata file for each of the video files, each of the metadata files having at least one segment to be edited; and

creating a single edited metafile containing links to the segments to be edited from each of the metadata files in an edited order;

wherein when the edited metadata file is accessed, the user is able to play the segments to be edited in the edited order.

48. A method for editing a multimedia file comprising:

providing a metafile, the metafile having at least one segment that is selectable;

selecting a segment in the metafile;

determining if a composing segment should be created, and if the composing segment should be created, then creating a composing segment in a hierarchical structure;

specifying the composing segment as a child of a parent composing segment;

determining if metadata is to be copied or if a URI is to be used;

if the metadata is to be copied, then copying metadata of the selected segment to the component segment;

if the URI is to be used, then writing a URI of the selected segment to the component segment;

writing a URL of an input video file to the component segment;

determining if all URLs of any sibling files are the same; and

if the URL is the same as any of the sibling'"'"'s URLs, then writing the URL to the parent composing segment and deleting the URLs of all sibling segments.

54. A virtual video editor comprising:

a network controller, the network controller constructed and arranged to access remote metafiles and remote video files;

a file controller, the file controller in operative connection to the network controller, the file controller constructed and arranged to access local metafiles and local video files, and to access the remote metafiles and the remote video files via the network controller;

a parser, the parser constructed and arranged to receive information about the files from the file controller;

an input buffer, the input buffer constructed and arranged to receive parser information from the parser;

a structure manager, the structure manager constructed and arranged to provide structure data to the input buffer;

a composing buffer, the composing buffer constructed and arranged to receive input information from the input buffer and structure information from the structure manager to generate composing information; and

a generator, the generator constructed and arranged to receive the composing information from the composing buffer;

wherein the generator generates output information in a pre-selected format.

55. The virtual video editor of claim 54, the editor further comprising:

a playlist generator, the playlist generator constructed and arranged to receive structure information from the structure manager in order to generate playlist information; and

a video player, the video player constructed and arranged to receive the playlist information from the playlist generator and file information from the file controller in order to generate display information.

56. The virtual video editor of claim 55, the editor further having a display device constructed and arranged to receive the display information from the video player and to display the display information to a user.

59. The method of claim 58 further comprising:

transmitting image data saved in the multimedia bookmark; and

using the image data as a query frame for a frame based search of the storage devices.

60. The method of claim 58 farther comprising:

transmitting positional information saved in the multimedia bookmark; and

using the positional information as a query frame for a frame based search of the storage devices.

61. The method of claim 58 further comprising:

transmitting annotated text saved in the multimedia bookmark; and

using the annotated text as keywords for a text based search of the storage devices.

62. The method of claim 58 further comprising:

determining whether one or more of the search results contains annotated text;

using the annotated text as keywords in a text based search for relevant multimedia content; and

presenting, to the user, search results from the text based search.

63. The method of claim 58 further comprising the at least one feature saved in a multimedia bookmark including image data and annotated text.

64. The method of claim 58 further comprising the at least one feature saved in a multimedia bookmark including image data and positional information.

65. The method of claim 58 further comprising the at least one feature saved in a multimedia bookmark including positional information and annotated text.

66. The method of claim 58 further comprising the at least one feature saved in a multimedia bookmark including image data, annotated text and positional data.

68. The method of claim 67 further comprising the sending and recipient devices including wireless devices.

70. The method of claim 69 further comprising streaming video content to a personal digital assistant.

57. A method for transcoding an image for display at multiple resolutions, the method comprising:

providing a multimedia file;

designating one or more regions of the multimedia file as focus zones;

providing a vector to each of the focus zones;

reading the multimedia file with a client device, the client device having a maximum display resolution;

determining if the resolution of the multimedia file exceeds the maximum display resolution of the client device;

if the multimedia file resolution exceeds the maximum display resolution of the display device, then determining the maximum number focus zones can be displayed on the client device; and

displaying the maximum number of focus zones on the client device.

58. A method for searching for relevant multimedia content based on at least one feature saved in a multimedia bookmark, the method comprising:

transmitting at least one feature saved in a multimedia bookmark from a client system to a server system in response to user selection of the multimedia bookmark;

generating a query for each feature saved in the multimedia bookmark and received by the server system;

searching one or more storage devices using each query generated; and

presenting, to the user, search results produced from at least one storage device search.

67. A method for sending a multimedia bookmark between devices over a wireless network, the method comprising:

submitting a multimedia bookmark to a video bookmark message service center by a sending device;

acknowledging receipt of the multimedia bookmark by the video bookmark message service center to the sending device;

requesting routing information for a recipient device from a home location register by the video bookmark message service center;

receiving the routing information from the home location register by the video bookmark message service center;

invoking a send multimedia bookmark at a mobile switching center;

sending the multimedia bookmark to a recipient device by the mobile switching center;

acknowledging receipt of the multimedia bookmark by the recipient device; and

notifying the video bookmark message service center when the multimedia bookmark has been received by the recipient device.

69. A method for sending multimedia content to a mobile device for playback over a wireless network, the method comprising:

submitting a multimedia bookmark and a request for multimedia content playback from the mobile device to a mobile switching center;

sending the multimedia bookmark and the request for playback to a video bookmark message service center by the mobile switching center;

determining a bit rate suitable for transmission of the multimedia content to the mobile device by the video bookmark message service center;

calculating a new multimedia bookmark based on the transmission bit rate and characteristics of the mobile device;

sending the new multimedia bookmark to a multimedia server; and

streaming the multimedia content from the multimedia server to the video bookmark message service center before delivering the multimedia content to the mobile device via the mobile switching center.

71. A method for verifying inclusion of attachments to electronic mail messages, the method comprising:

scanning the electronic mail message for at least one indicator of an attachment to be included;

determining whether at least one attachment to the electronic mail message is present upon detection of the at least one indicator of an attachment to be included; and

displaying a reminder to a user in the event at least one indicator of an attachment to be included is found but no attachment is determined to be present.

72. The method of claim 71 further comprising comparing contents of the electronic mail message with language settings designated by the user to determine whether at least one indicator of an attachment to be included is present.

73. A content transcoder for modifying and forwarding multimedia content maintained in one or more multimedia content databases to a wide area network for display on a requesting client device, the content transcoder comprising:

a policy engine operably coupled to the multimedia content database;

a content analyzer operably coupled to the policy engine and the multimedia content database;

a content selection module operably coupled to the policy engine and the content analyzer;

a content manipulation module operably coupled to the content selection module;

a content analysis and manipulation library operably coupled to the content analyzer, the content selection module and the content manipulation module; and

wherein

the policy engine is operable to receive a request for multimedia content from the requesting client device via the wide area network and to receive policy information from the multimedia content database;

the content analyzer is operable to retrieve multimedia content from the multimedia content database and to forward the multimedia content to the content selection module;

the content selection module is operable to select portions of the multimedia content based on the policy information and information from the content analysis and manipulation library and to forward the selected portions of multimedia content to the content manipulation module;

the content manipulation module is operable to modify the multimedia content for display on a requesting client device prior to transmitting the modified multimedia content over the wide area network to the requesting client device.

74. The content transcoder of claim 73 further comprising the requesting client device including a personal digital assistant (PDA).

75. The content transcoder of claim 73 further comprising the requesting client device including a laptop computer.

76. The content transcoder of claim 73 further comprising the requesting client device including a television.

77. The content transcoder of claim 73 further comprising the requesting client device including a personal computer.

78. The content transcoder of claim 73 further comprising the requesting client device including a personal data appliance.

79. The content transcoder of claim 73 further comprising the requesting client device including a mobile telephone.

80. A method of searching for multimedia content in a peer to peer environment, the method comprising:

broadcasting a message from a user system to announce entrance to the peer to peer environment;

acknowledging receipt of the broadcast message by one or more active nodes in the peer to peer environment;

tracking the active nodes by the user system;

broadcasting a query message including multimedia features to the peer to peer environment upon initiation of a search request by the user system;

executing a multimedia search engine on a multimedia database included in a storage device on one or more active nodes upon receipt of the query message; and

responding to the query message with a search results message including a listing of found filenames and network address locations.