Method and system for operating a data processor to index primary data in real time with iconic table of contents

• US 5,550,965 A
• Filed: 12/27/1993
• Issued: 08/27/1996
• Est. Priority Date: 12/27/1993
• Status: Expired due to Term

• Current Assignee: Lucent Technologies, Inc. (Nokia Corporation)
• Original Assignee: Lucent Technologies, Inc. (Nokia Corporation)
• Inventors: Gabbe, John D., Ginsberg, Allen, Robinson, Bethany S.
• Primary Examiner(s): Zimmerman, Mark K.
• Assistant Examiner(s): Burwell, Joseph R.
• Application Number:
  US08/173,843
• Time in Patent Office: 974 Days
• Field of Search: 395/153, 395/154, 395/160, 395/161, 395/159, 395/155, 395/600, 348/15, 348/19, 348/904, 358/335, 360/69
• US Class Current: 715/209
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/41 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing
• CPC Class Codes: Y10S 707/99948 Application of database or data structure, e.g. distributed, multimedia, or image

Abstract

A method and processing system for the indexing of data for one or more users involving the steps of receiving primary data, delineating episode data from the primary data, generating media representations to correspond with the episode data, generating a table of contents from the episode data and media representations in real time, and storing and/or presenting the table of contents on a graphical user interface. Primary data may constitute audio data, video data, text data, combinations of these as well as any medium-generated data which may be in or converted to digital form. Episode data is generated on the basis of significant video images determined by scene changes and other information. In addition, episode data may be based on a significant audio change, delineated from the primary data, e.g. based on speaker changes and other information. Media representations are iconic images, symbolic representations or graphical images which refer to episode data to ultimately aid the user in making decisions from reviewing the real time table of contents.

Claims

1. A method of operating a data processor to index data for at least one user comprising the computer-implemented steps of:

receiving primary data from a data source, the primary data consisting of audio-data and audiovisual-data that has been converted to a sequence of binary words;

delineating a plurality of episode data from the primary data, each one of said plurality of episode data being at least one word of said sequence of binary words, said each one of said plurality of episode data being stored in memory, wherein said delineating includes comparing and filtering audio spectral frequency components in said audio-data and audiovisual-data in the primary data;

generating a plurality of icons, to correspond with said plurality of episode data;

generating a table of contents as said primary data is being received by configuring said plurality of icons in an organized fashion;

presenting the table of contents to said at least one user on at least one display device, the table of contents permitting said at least one user to select one of said plurality of icons to replay the corresponding one of said plurality of episode data.

2. The method of operating a data processor to index data according to claim 1 wherein said sequence of binary words has at least one identifiable word, said at least one identifiable word being within the primary data received during a discrete, identifiable time interval;

and all steps following the step of receiving the primary data from said data source are executed immediately after said at least one identifiable word is received such that said one of said plurality of icons derived from said one of said plurality of episode data is presented to said at least one user immediately after said at least one identifiable word is received.

3. The method of operating a data processor to index data according to claim 1 wherein the step of generating said plurality of icons comprises the step of executing procedures and relationships that relate respective ones of said plurality of episode data with ones of said plurality of icons via a plurality of first knowledge representations.

4. The method of operating a data processor to index data according to claim 1 further comprising step of storing said plurality of icons in a database.

5. The method of operating a data processor to index data according to claim 1 further comprising the steps of:

presenting the primary data on at least one display device selected from the group consisting of audio interfaces, graphical user interfaces, visual display devices, and audiovisual display devices; and

storing the primary data in a database.

6. The method of operating a data processor to index data according to claim 1 further comprising the steps of:

retrieving selective ones of said plurality of episode data by selecting corresponding ones of said plurality of icons;

inputting at least one ancillary data by said at least one user to annotate the selected one of said plurality of episode data that the user retrieved;

presenting the at least one ancillary data to a user;

storing the at least one ancillary data in a database such that the at least one ancillary data is appended to the selected one of said plurality of episode data that was retrieved;

indexing the at least one ancillary data such that each of the at least one ancillary data has a first ancillary address stored in the database; and

retrieving the at least one ancillary data such that the at least one

ancillary data is presented to said user.

7. The method of operating a data processor to index data according to claim 6 further comprising the steps of:

sorting the at least one ancillary data by using input values selected by a user and by using intrinsic values of the at least one ancillary data as sorting criteria;

generating a second ancillary address corresponding to the sorted at least one ancillary data, the second ancillary address being stored in the database; and

displaying the sorted at least one ancillary data on a display device by using the second ancillary address.

8. The method of operating a data processor to index data according to claim 7 further comprising the steps of:

sorting the plurality of episode data by sorting respective ones of at least one ancillary data that were appended to the corresponding ones of said plurality of episode data in claim 6.

9. The method of operating a data processor to index data according to claim 1 further comprising the steps of:

allowing a user to input ancillary data to annotate one of the plurality of episode data that occur red with said user'"'"'s input of ancillary data;

presenting the at least one ancillary data to said user; and

storing the at least one ancillary data such that the at least one ancillary data is appended to said one of said plurality of episode data that was retrieved.

10. The method of operating a data processor to index data according to claim 1 further comprising the steps of:

permitting a user to select one of said plurality of episode data by inputting corresponding ones of a plurality of icon values, each one of said plurality of icons having a designated one of said plurality of icon values; and

displaying said one of said plurality of episode data that the user selected on a display device.

11. The method of operating a data processor to index data according to claim 1 wherein the step of delineating said plurality of episode data is further performed periodically at predetermined time intervals by comparing and filtering luminance values of bitmaps in said audiovisual-data in the primary data to previously filtered prior bitmaps in the plurality of episode data to determine whether the subsequent bitmaps differ more than 20% to constitute episode data.

12. The method of operating a data processor to index data according to index data according to claim 11 wherein the step of delineating said plurality of episode data is accomplished in response to correlating changes in said audio data in the primary data, and changes in luminance values of bitmaps in said audiovisual data in the primary data.

13. The method of operating a data processor to index data according to claim 1 wherein the step of delineating said plurality of episode data is further accomplished by comparing a plurality of color values of bitmaps in said audiovisual data in the primary data and wherein delineating said plurality of episode data is performed periodically at a variable rate based upon feedback of the processor concerning available memory and memory utilization.

14. The method of operating a data processor to index data according to claim 1 wherein the step of generating is performed by generating an icon such that said icon is a miniature graphical image of at least one episode data of the plurality of episode data.

15. A data processor system for indexing data comprising:

a receiver, the receiver capable of demodulating a primary data signal consisting of audio-data and audiovisual-data to obtain primary data, the receiver having a memory buffer selected from the group of first input memory buffers and second input memory buffers to store the primary data in digital form, the primary data consisting of at least one sequence of binary words;

a plurality of episode data, each one of said plurality of episode data being at least one binary word of the at least one sequence of binary words;

a plurality of first knowledge representations relating the plurality of episode data to the primary data;

a plurality of second knowledge representations relating respective ones of a plurality of icons with corresponding ones of said plurality of episode data;

a processor selected from the group of first processor, second processor, and first and second processor considered collectively to generate each of said plurality of icons from any given one of said plurality of episode data, from the plurality of episode data, based on said second knowledge representations, and to generate each one of said plurality of episode data from any given primary data based on said first knowledge representations, wherein said generating includes comparing and filtering audio spectral frequency components in said studio-data and audiovisual-data in the primary data; and

a means for real time generating of a table of contents with respect to receiving and storing said primary data from the plurality of episode data and from the plurality of icons.

16. The system according to claim 15 further comprising a means for selecting respective ones of said plurality of icons such that a user may display corresponding ones of said plurality of episode data.

17. The system according to claim 15 further comprising means for presenting primary data to a user before the primary data is stored.

18. The system according to claim 15 further comprising means for detecting and delineating a plurality of episode data from the primary data.

19. The system according to claim 15 further comprising means for generating said plurality of icons and means for associating respective ones of said plurality of episode data with corresponding ones of said plurality of icons.

20. The system according to claim 15 further comprising means for storing a table of contents of the plurality of episode data and said plurality of icons.

21. The system according to claim 15 further comprising means for presenting said plurality of icons to the user as said plurality of icons are being formed.

22. The system according to claim 15 further comprising means for retrieving any one of said plurality of icons from memory and displaying any one of said plurality of icons.

23. The system according to claim 15 further comprising means for annotating any one of said plurality of episode data with at least one ancillary data.

24. The system according to claim 15 further comprising means for indexing and sorting said episode data by using the at least one ancillary data as sorting criteria.

25. The system according to claim 15 further comprising means for identifying, selecting, and presenting any one of said plurality of episode data immediately after the primary data relating to said one of said plurality of episode data has been received.

26. A multimedia conferencing system comprising:

a plurality of participating clients, the plurality of participating clients being work stations, the work stations supporting a graphical user interface;

external sources, the external sources having external data, external data including primary data, metadata and events consisting of audio-data and audiovisual data in digital form from other networks;

internal sources, the internal sources encompassing local sources of metadata and event data,

an episode server, the episode server having a means for storing data, said means for storing data responsive to input of external data, event data, client requests or client notes, the episode server having a means for retrieving data, said means for retrieving data being responsive to input of a client query, clients selection or client request, said means for storing data including delineating episode data by executing procedures in conjunctions with a plurality of first knowledge representations and the episode server having a means for generating a plurality of icons in real time and a table of contents in real time with respect to storing and retrieving data, wherein said procedures include comparing and filtering audio spectral frequency components in said audio-data and audiovisual data in the external data;

a network manager, the network manager being operably, electrically connected to said episode server, said external sources and said participating clients, the network manager having a means for providing communication, switching and control function between ones of said plurality of participating clients and the episode server, between ones of the participating clients, between the external sources and the episode server, and between the internal sources and the episode server; and

an archive, the archive operably attached to the episode server, the archive being a high-capacity storage device for storing the episode data, and for storing other common files for the plurality of participating clients.

27. The multimedia conferencing system of claim 26 wherein the plurality of first knowledge representations contain knowledge concerning luminance values of bitmaps, correlation of simultaneous changes in audio and video and color of bitmaps.

28. The multimedia conferencing system according to claim 26 wherein the means for storing data include storing data such that annotating of episode data occurs in response to said client notes and said event data, and such that indexing occurs in response to said client request.

29. The multimedia conferencing system according to claim 26 wherein the means for retrieving data include retrieving data such that primary data, said plurality of icons, table of contents, ancillary data, and episode data are retrieved from the archive in response to said client query, said client selection or said client request.