Indirect manipulation of data using temporally related data, with particular application to manipulation of audio or audiovisual data

• US 6,172,675 B1
• Filed: 12/05/1996
• Issued: 01/09/2001
• Est. Priority Date: 12/05/1996
• Status: Expired due to Term

• Current Assignee: Vulcan Patents LLC
• Original Assignee: Interval Research Corporation
• Inventors: Ahmad, Subutai, Farber, Emanuel E.
• Primary Examiner(s): Zimmerman, Mark
• Assistant Examiner(s): Nguyen, Kimbinh T.
• Application Number:
  US08/761,025
• Time in Patent Office: 1,496 Days
• Field of Search: 707/100, 707/101, 707/530, 707/104, 707/531, 364/481, 395/615, 704/255, 345/302, 345/382, 345/328
• US Class Current: 715/251
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing

Abstract

The invention enables a body of information to be easily manipulated (e.g., summarized, rearranged, edited, selectively accessed or searched) to facilitate use of the information by a user. The body of information is represented by at least two sets of temporally related data. Manipulation of a second one of the sets of data in a predetermined manner is accomplished indirectly through direct manipulation of a first one of the sets of data in the predetermined manner. Where the body of information is represented by more than two sets of temporally related data, the additional sets of data can also be manipulated in a similar fashion. In a particular embodiment of the invention, digital text data representing a body of information (which is also represented by video and/or audio data) is manipulated, the digital text data is aligned with either the digital video data or digital audio data of the body of information, and the digital video data or digital audio data is manipulated according to the manipulation of the digital text data and the alignment of the digital text data with the digital video data or digital audio data. The other of the video or audio data (if present) can be manipulated by aligning that data with either the text data or the previously manipulated video or audio data, then manipulating the other of the video or audio data according to the previous manipulation of the data with which the other of the video or audio data has been aligned.

Claims

1. A method for indirectly modifying a second set of data that is temporally related to a first set of data, comprising the steps of:

modifying the first set of data in a predetermined manner to produce a modified first set of data;

aligning the first set of data with the second set of data to establish a temporal correspondence between the first and second sets of data, wherein a temporal correspondence does not exist between the first and second sets of data prior to the step of aligning; and

automatically modifying data from the second set of data that is aligned with data from the modified first set of data, in accordance with the modification of the first set of data, to produce the modified second set of data.

2. A method as in claim 1, wherein the first and second sets of data are sets of digital data.

3. A method as in claim 1, wherein the first and second sets of data are sets of analog data.

4. A method as in claim 3, further comprising the steps of:

digitizing the first set of analog data to produce a first set of digital data; and

digitizing the second set of analog data to produce a second set of digital data.

5. A method as in claim 1, wherein:

the first set of data is an audio data set; and

the second set of data is a video data set.

6. A method as in claim 1, wherein:

the first set of data is a text data set; and

the second set of data is an audio data set.

7. A method as in claim 6, further comprising the step of producing the text data set from the audio data set.

8. A method as in claim 6, wherein the step of aligning the audio data set with the text data set further comprises the steps of:

generating a linguistic transcription network that models the text data set in terms of audio features;

performing a feature analysis on the audio data set to produce a set of audio feature data; and

determining the path through the linguistic transcription network which represents the best fit with the set of audio feature data.

9. A method as in claim 1, wherein a third set of data is temporally related to both the first and second sets of data, the method further comprising the steps of:

aligning the third set of data with either the first or second sets of data to establish a temporal correspondence between the third set of data and the first or second sets of data; and

automatically modifying data from the third set of data that is aligned with data from the modified first or second set of data to produce a modified third set of data.

10. A method as in claim 9, wherein:

the first set of data is a text data set;

the second set of data is an audio data set; and

the third set of data is a video data set.

11. A method as in claim 10, wherein:

the step of aligning the first set of data with the second set of data further comprises the step of aligning the text data set with the audio data set; and

the step of aligning the third set of data with either the first or second sets of data further comprises the step of aligning the video data set with the audio data set.

12. A method as in claim 11, wherein:

the step of modifying the first set of data further comprises the step of summarizing the text data set in the predetermined manner, thereby producing a text summary;

the step of automatically modifying data from the second set of data further comprises the step of selecting audio data that is aligned with the text summary, thereby producing an audio summary; and

the step of automatically modifying data from the third set of data further comprises the step of selecting video data that is aligned with the audio summary, thereby producing a video summary.

13. A method as in claim 1, wherein the step of modifying the first set of data further comprises the step of summarizing the first set of data in the predetermined manner to produce a summarized first set of data, the step of automatically modifying the second set of data thereby producing a summarized second set of data.

14. A method as in claim 13, wherein:

the first set of data is a text data set; and

the second set of data is an audio data set.

15. A method as in claim 1, wherein the step of modifying the first set of data further comprises the step of editing the first set of data in the predetermined manner to produce an edited first set of data, the step of automatically modifying the second set of data thereby producing an edited second set of data.

18. A method as in claim 1, wherein:

each of the first and second sets of data include a plurality of segments, the segments of the first set of data being temporally related to the segments of the second set of data;

the step of modifying the first set of data in a predetermined manner to produce a modified first set of data further comprises the step of modifying the segments of the first set of data in a predetermined manner; and

the step of automatically modifying data from the second set of data further comprises the step of modifying segments of the second set of data in accordance with the modification of the corresponding segments of the first set of data.

19. A method as in claim 18, wherein:

the step of modifying the first set of data further comprises the step of summarizing one or more of the segments of the first set of data; and

the step of automatically modifying the second set of data further comprises the step of summarizing segments of the second set of data that correspond to summarized segments of the first set of data.

20. A method as in claim 19, wherein the first and second sets of data each represent a series of news stories, each segment representing a particular news story.

21. A method as in claim 18, wherein

one of the first and second sets of data is a video data set including a plurality of frames of video data;

and

the method further comprises the step of adding the first and/or last video frame of one or more of the segments of the modified video data set one or more times to the beginning and/or end, respectively, of the segment.

22. A method as in claim 18, wherein:

the first set of data is a text data set;

the second set of data is an audio data set; and

the method further comprises the step of defining one or more segments of the audio data set so that the beginning of the audio segment is temporally prior to, or temporally coincident with, the beginning of the corresponding segment of the text data set, and so that the end of the audio segment is temporally subsequent to, or temporally coincident with, the end of the corresponding segment of the text data set.

23. A method as in claim 1, wherein:

the first set of data is a text data set;

the second set of data is an audio data set; and

the method further comprises the step of defining the audio data set so that the beginning of the audio data set is temporally prior to, or temporally coincident with, the beginning of the text data set, and so that the end of the audio data set is temporally subsequent to, or temporally coincident with, the end of the text data set.

29. A method as in claim 1, wherein:

the first set of data is a text data set; and

the second set of data is a video data set.

30. A method as in claim 1, further comprising the step of displaying the modified second set of data.

31. A method as in claim 8, wherein the step of performing a feature analysis on the audio data set to produce a set of audio feature data further comprises the step of grouping consecutive audio samples of the audio data into audio frames spanning a duration of time of about 45 milliseconds or less.

16. A system for indirectly modifying a second set of data that is temporally related to a first set of data, comprising:

means for modifying the first set of data in a predetermined manner to produce a modified first set of data;

means for aligning the first set of data with the second set of data to establish a temporal correspondence between the first and second sets of data, wherein a temporal correspondence does not exist between the first and second sets of data prior to alignment of the first and second sets of data; and

means for automatically modifying data from the second set of data that is aligned with data from the modified first set of data, in accordance with the modification of the first set of data, to produce the modified second set of data.

32. A system as in claim 16, wherein:

the first set of data is an audio data set; and

the second set of data is a video data set.

33. A system as in claim 16, wherein:

the first set of data is a text data set; and

the second set of data is an audio data set.

34. A system as in claim 33, further comprising means for producing the text data set from the audio data set.

35. A system as in claim 16, wherein a third set of data is temporally related to both the first and second sets of data, the system further comprising:

means for aligning the third set of data with either the first or second sets of data to establish a temporal correspondence between the third set of data and the first or second sets of data; and

means for automatically modifying data from the third set of data that is aligned with data from the modified first or second set of data to produce a modified third set of data.

36. A system as in claim 35, wherein:

the first set of data is a text data set;

the second set of data is an audio data set; and

the third set of data is a video data set.

37. A system as in claim 36, wherein:

the means for aligning the first set of data with the second set of data further comprises means for aligning the text data set with the audio data set; and

the means for aligning the third set of data with either the first or second sets of data further comprises means for aligning the video data set with the audio data set.

38. A system as in claim 37, wherein:

the means for modifying the first set of data further comprises means for summarizing the text data set in the predetermined manner, thereby producing a text summary;

the means for automatically modifying data from the second set of data further comprises means for selecting audio data that is aligned with the text summary, thereby producing an audio summary; and

the means for automatically modifying data from the third set of data further comprises means for selecting video data that is aligned with the audio summary, thereby producing a video summary.

39. A system as in claim 16, wherein the means for modifying the first set of data further comprises means for summarizing the first set of data in the predetermined manner to produce a summarized first set of data, the means for automatically modifying the second set of data thereby producing a summarized second set of data.

40. A system as in claim 39, wherein:

the first set of data is a text data set; and

the second set of data is an audio data set.

41. A system as in claim 16, wherein the means for modifying the first set of data further comprises means for editing the first set of data in the predetermined manner to produce an edited first set of data, the means for automatically modifying the second set of data thereby producing an edited second set of data.

42. A system as in claim 16, wherein:

each of the first and second sets of data include a plurality of segments, the segments of the first set of data being temporally related to the segments of the second set of data, wherein each segment represents a particular news story;

the means for modifying the first set of data in a predetermined manner to produce a modified first set of data further comprises means for modifying the news stories represented by the first set of data in a predetermined manner, wherein the means for modifying the news stories represented by the first set of data further comprises means for summarizing one or more of the news stories represented by the first set of data; and

the means for automatically modifying data from the second set of data further comprises means for modifying the news stories represented by the second set of data in accordance with the modification of the news story as represented by the first set of data, wherein the means for modifying news stories represented by the second set of data further comprises means for summarizing news stories represented by the second set of data in accordance with the summarization of the news story as represented by the first set of data.

43. A system as in claim 16, wherein:

one of the first and second sets of data is a video data set including a plurality of frames of video data;

each of the first and second sets of data include a plurality of segments, the segments of the first set of data being temporally related to the segments of the second set of data;

the means for modifying the first set of data in a predetermined manner to produce a modified first set of data further comprises means for modifying the segments of the first set of data in a predetermined manner;

the means for automatically modifying data from the second set of data further comprises means for modifying segments of the second set of data in accordance with the modification of the corresponding segments of the first set of data; and

the system further comprises means for adding the first and/or last video frame of one or more of the segments of the modified video data set one or more times to the beginning and/or end, respectively, of the segment.

44. A system as in claim 16, wherein:

the first set of data is a text data set;

the second set of data is an audio data set;

each of the first and second sets of data include a plurality of segments, the segments of the first set of data being temporally related to the segments of the second set of data;

the means for modifying the first set of data in a predetermined manner to produce a modified first set of data further comprises means for modifying the segments of the first set of data in a predetermined manner;

the means for automatically modifying data from the second set of data further comprises means for modifying segments of the second set of data in accordance with the modification of the corresponding segments of the first set of data; and

the system further comprises means for defining one or more segments of the audio data set so that the beginning of the audio segment is temporally prior to, or temporally coincident with, the beginning of the corresponding segment of the text data set, and so that the end of the audio segment is temporally subsequent to, or temporally coincident with, the end of the corresponding segment of the text data set.

45. A system as in claim 16, wherein:

the first set of data is a text data set;

the second set of data is an audio data set; and

the system further comprises means for defining the audio data set so that the beginning of the audio data set is temporally prior to, or temporally coincident with, the beginning of the text data set, and so that the end of the audio data set is temporally subsequent to, or temporally coincident with, the end of the text data set.

46. A system as in claim 16, wherein the means for aligning further comprises:

means for dividing the first set of data into a plurality of parts;

means for dividing the second set of data into a plurality of parts such that each part of the second set of data approximately temporally corresponds to a part of the first set of data; and

means for aligning corresponding parts of the first and second sets of data.

47. A system as in claim 16, wherein the means for modifying the first set of data operates at least in part at the same time as the means for aligning the first set of data with the second set of data.

48. A system as in claim 16, further comprising means for identifying, after operation of the means for modifying the first set of data and prior to operation of the means for aligning, parts of the second set of data that correspond approximately to parts of the first set of data that have been modified, wherein the means for aligning operates only on the modified parts of the first set of data and the corresponding identified parts of the second set of data.

49. A system as in claim 16, wherein:

the first set of data is a text data set; and

the means for automatically modifying data from the second set of data further comprises means for including in the modified second set of data all data from the second set of data except data which is aligned with data in the text data set that is not included in the modified text data set.

50. A system as in claim 16, wherein:

the first set of data is a text data set; and

the means for automatically modifying data from the second set of data further comprises means for including in the modified second set of data all data from the second set of data that is between data corresponding to adjacent text data in the modified text data set.

51. A system as in claim 16, wherein:

the first set of data is a text data set; and

the second set of data is a video data set.

52. A system as in claim 16, further comprising means for displaying the modified second set of data.

17. A computer readable storage medium on which are stored instructions for indirectly modifying a second set of data that is temporally related to a first set of data, comprising:

instructions for modifying the first set of data in a predetermined manner to produce a modified first set of data;

instructions for aligning the first set of data with the second set of data to establish a temporal correspondence between the first and second sets of data, wherein a temporal correspondence does not exist between the first and second sets of data prior to the step of aligning; and

instructions for automatically modifying data from the second set of data that is aligned with data from the modified first set of data, in accordance with the modification of the first set of data, to produce the modified second set of data.

53. A computer readable storage medium as in claim 17, wherein:

the first set of data is an audio data set; and

the second set of data is a video data set.

54. A computer readable storage medium as in claim 17, wherein:

the first set of data is a text data set; and

the second set of data is an audio data set.

55. A computer readable storage medium as in claim 54, further comprising instructions for producing the text data set from the audio data set.

56. A computer readable storage medium as in claim 17, wherein a third set of data is temporally related to both the first and second sets of data, the computer readable storage medium further comprising:

instructions for aligning the third set of data with either the first or second sets of data to establish a temporal correspondence between the third set of data and the first or second sets of data; and

instructions for automatically modifying data from the third set of data that is aligned with data from the modified first or second set of data to produce a modified third set of data.

57. A computer readable storage medium as in claim 56, wherein:

the first set of data is a text data set;

the second set of data is an audio data set; and

the third set of data is a video data set.

58. A computer readable storage medium as in claim 57, wherein:

the instructions for aligning the first set of data with the second set of data further comprise instructions for aligning the text data set with the audio data set; and

the instructions for aligning the third set of data with either the first or second sets of data further comprise instructions for aligning the video data set with the audio data set.

59. A computer readable storage medium as in claim 58, wherein:

the instructions for modifying the first set of data further comprise instructions for summarizing the text data set in the predetermined manner, thereby producing a text summary;

the instructions for automatically modifying data from the second set of data further comprise instructions for selecting audio data that is aligned with the text summary, thereby producing an audio summary; and

the instructions for automatically modifying data from the third set of data further comprise instructions for selecting video data that is aligned with the audio summary, thereby producing a video summary.

60. A computer readable storage medium as in claim 17, wherein the instructions for modifying the first set of data further comprise instructions for summarizing the first set of data in the predetermined manner to produce a summarized first set of data, the instructions for automatically modifying the second set of data thereby producing a summarized second set of data.

61. A computer readable storage medium as in claim 60, wherein:

the first set of data is a text data set; and

the second set of data is an audio data set.

62. A computer readable storage medium as in claim 17, wherein the instructions for modifying the first set of data further comprise instructions for editing the first set of data in the predetermined manner to produce an edited first set of data, the instructions for automatically modifying the second set of data thereby producing an edited second set of data.

63. A computer readable storage medium as in claim 17, wherein:

each of the first and second sets of data include a plurality of segments, the segments of the first set of data being temporally related to the segments of the second set of data, each segment representing a particular news story;

the instructions for modifying the first set of data in a predetermined manner to produce a modified first set of data further comprise instructions for modifying the news stories represented by the first set of data in a predetermined manner, wherein the instructions for modifying the news stories represented by the first set of data further comprise instructions for summarizing one or more of the news stories represented by the first set of data; and

the instructions for automatically modifying data from the second set of data further comprise instructions for modifying the news stories represented by the second set of data in accordance with the modification of the news stories as represented by the first set of data, wherein the instructions for modifying news stories represented by the second set of data further comprise instructions for summarizing news stories represented by the second set of data in accordance with the summarization of the news stories as represented by the first set of data.

64. A computer readable storage medium as in claim 17, wherein:

one of the first and second sets of data is a video data set including a plurality of frames of video data;

each of the first and second sets of data include a plurality of segments, the segments of the first set of data being temporally related to the segments of the second set of data;

the instructions for modifying the first set of data in a predetermined manner to produce a modified first set of data further comprise instructions for modifying the segments of the first set of data in a predetermined manner;

the instructions for automatically modifying data from the second set of data further comprise instructions for modifying segments of the second set of data in accordance with the modification of the corresponding segments of the first set of data; and

the computer readable storage medium further comprises instructions for adding the first and/or last video frame of one or more of the segments of the modified video data set one or more times to the beginning and/or end, respectively, of the segment.

65. A computer readable storage medium as in claim 17, wherein:

the first set of data is a text data set;

the second set of data is an audio data set;

each of the first and second sets of data include a plurality of segments, the segments of the first set of data being temporally related to the segments of the second set of data;

the instructions for modifying the first set of data in a predetermined manner to produce a modified first set of data further comprise instructions for modifying the segments of the first set of data in a predetermined manner;

the instructions for automatically modifying data from the second set of data further comprise instructions for modifying segments of the second set of data in accordance with the modification of the corresponding segments of the first set of data; and

the computer readable storage medium further comprises instructions for defining one or more segments of the audio data set so that the beginning of the audio segment is temporally prior to, or temporally coincident with, the beginning of the corresponding segment of the text data set, and so that the end of the audio segment is temporally subsequent to, or temporally coincident with, the end of the corresponding segment of the text data set.

66. A computer readable storage medium as in claim 17, wherein:

the first set of data is a text data set;

the second set of data is an audio data set; and

the computer readable storage medium further comprises instructions for defining the audio data set so that the beginning of the audio data set is temporally prior to, or temporally coincident with, the beginning of the text data set, and so that the end of the audio data set is temporally subsequent to, or temporally coincident with, the end of the text data set.

67. A computer readable storage medium as in claim 17, wherein the instructions for aligning further comprise:

instructions for dividing the first set of data into a plurality of parts;

instructions for dividing the second set of data into a plurality of parts such that each part of the second set of data approximately temporally corresponds to a part of the first set of data; and

instructions for aligning corresponding parts of the first and second sets of data.

68. A computer readable storage medium as in claim 17, wherein the instructions for modifying the first set of data execute at least in part at the same time as the instructions for aligning the first set of data with the second set of data.

69. A computer readable storage medium as in claim 17, further comprising instructions for identifying, after execution of the instructions for modifying the first set of data and prior to execution of the instructions for aligning, parts of the second set of data that correspond approximately to parts of the first set of data that have been modified, wherein the instructions for aligning only align the modified parts of the first set of data and the corresponding identified parts of the second set of data.

70. A computer readable storage medium as in claim 17, wherein:

the first set of data is a text data set; and

the instructions for automatically modifying data from the second set of data further comprise instructions for including in the modified second set of data all data from the second set of data except data which is aligned with data in the text data set that is not included in the modified text data set.

71. A computer readable storage medium as in claim 17, wherein:

the first set of data is a text data set; and

the instructions for automatically modifying data from the second set of data further comprise instructions for including in the modified second set of data all data from the second set of data that is between data corresponding to adjacent text data in the modified text data set.

72. A computer readable storage medium as in claim 17, wherein:

the first set of data is a text data set; and

the second set of data is a video data set.

73. A computer readable storage medium as in claim 17, further comprising instructions for displaying the modified second set of data.

24. A method for indirectly modifying a second set of data that is temporally related to a first set of data, comprising the steps of:

modifying the first set of data in a predetermined manner to produce a modified first set of data;

aligning the first set of data with the second set of data to establish a temporal correspondence between the first and second sets of data, wherein the step of aligning further comprises the steps of;

dividing the first set of data into a plurality of parts;

dividing the second set of data into a plurality of parts such that each part of the second set of data approximately temporally corresponds to a part of the first set of data;

aligning corresponding parts of the first and second sets of data; and

automatically modifying data from the second set of data that is aligned with data from the modified first set of data, in accordance with the modification of the first set of data, to produce the modified second set of data.

25. A method for indirectly modifying a second set of data that is temporally related to a first set of data, comprising the steps of:

modifying the first set of data in a predetermined manner to produce a modified first set of data;

aligning the first set of data with the second set of data to establish a temporal correspondence between the first and second sets of data, wherein at least part of the step of modifying the first set of data is performed at the same time as at least part of the step of aligning the first set of data with the second set of data; and

automatically modifying data from the second set of data that is aligned with data from the modified first set of data, in accordance with the modification of the first set of data, to produce the modified second set of data.

26. A method for indirectly modifying a second set of data that is temporally related to a first set of data, comprising the steps of:

modifying the first set of data in a predetermined manner to produce a modified first set of data;

identifying parts of the second set of data that correspond approximately to parts of the first set of data that have been modified;

aligning the modified parts of the first set of data and the corresponding identified parts of the second set of data to establish a temporal correspondence between the first and second sets of data; and

automatically modifying data from the second set of data that is aligned with data from the modified first set of data, in accordance with the modification of the first set of data, to produce the modified second set of data.

27. A method for indirectly modifying a second set of data that is temporally related to a first set of data, wherein the first set of data is a text data set, comprising the steps of:

modifying the first set of data in a predetermined manner to produce a modified first set of data;

aligning the first set of data with the second set of data to establish a temporal correspondence between the first and second sets of data; and

automatically modifying data from the second set of data that is aligned with data from the modified first set of data, in accordance with the modification of the first set of data, to produce the modified second set of data, wherein the step of automatically modifying data from the second set of data further comprises including in the modified second set of data all data from the second set of data except data which is aligned with data in the first set of data that is not included in the modified first set of data.

28. A method for indirectly modifying a second set of data that is temporally related to a first set of data, wherein the first set of data is a text data set, comprising the steps of:

modifying the first set of data in a predetermined manner to produce a modified first set of data;

aligning the first set of data with the second set of data to establish a temporal correspondence between the first and second sets of data; and

automatically modifying data from the second set of data that is aligned with data from the modified first set of data, in accordance with the modification of the first set of data, to produce the modified second set of data, wherein the step of automatically modifying data from the second set of data further comprises including in the modified second set of data all data from the second set of data that is between data corresponding to adjacent data in the modified first set of data.