Method and Apparatus for Video Digest Generation

• US 20090022472A1
• Filed: 07/16/2008
• Published: 01/22/2009
• Est. Priority Date: 07/16/2007
• Status: Active Grant

• Current Assignee: Vdoqwest, Inc.
• Original Assignee: Novafora, Inc.
• Inventors: BRONSTEIN, MICHAEL, Bronstein, Alexander, Rakib, Shlomo Selim, Matatyaou, Asaf
• Granted Patent: US 8,442,384 B2
• Time in Patent Office: Days
• US Class Current: 386/278
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: G11B 27/329 on a disc [VTOC]
• CPC Class Codes: H04N 5/783 Adaptations for reproducing at a rate different from the recording rate

Abstract

Automated video digest system and method. The system decomposes video and audio portions of video media into low-level descriptors, producing a large number of such descriptors for each video image and scene. These descriptors are optimized to be both compatible with rapid automated descriptor creation, and also to have some positive or negative statistical correlation with the viewing preferences of typical human viewers. By comparing the number of descriptors correlated with positive human viewing preferences, versus the number of descriptors correlated with negative human viewing preferences, the automated system may analyze an unknown video program and make educated guesses as to which portions of the program are likely to be most interesting to the viewer on playback. Less interesting portions may be skipped or fast-forwarded though. The method may also be optimized to individual viewers. Such methods are useful for controlling the playback of digital video recorders and other systems.

Claims

1. A digital video recorder system for storage and/or playback of media content, comprising:

at least one video processor capable of analyzing input media content and automatically generating descriptors that are associated with the image and/or audio contents of various time portions of the input media content;

at least one sensor to detect and record user viewing preferences while the input media content is being played;

persistent memory capable of storing a persistent database that correlates the descriptors with the recorded user viewing preferences;

at least one control processor, and associated control software;

in which the control processor, under the control of the control software, controls the storage and/or playback of various time portions of a previously unanalyzed input media content (unknown media content) by;

a) sending the unknown media content to the video processor, and directing the video processor to generate descriptors of the video and or audio portion of various time portions of the unknown media content (unknown video descriptors);

b) using the unknown video descriptors as an index to the persistent database, and retrieving the user viewing preferences associated with the unknown video descriptors;

c) constructing an estimated user preference of the various time portions of the unknown media content by using the retrieved user viewing preferences and a suitable weighing function;

and using this estimated user preference to control the storage and/or playback of various time portions of the unknown media content.

2. The system of claim 1, in which the video processor and the control processor are the same processor and the video processor is capable of compressing and/or decompressing a lossy or lossless video codec.

3. The system of claim 1, in which the video processor and the control processor are different processors, and in which the video processor is optimized for generating the descriptors.

4. The system of claim 1, in which the descriptors are selected from a population of descriptors known to have a positive or negative statistical correlation with user preferences.

5. The system of claim 4, in which the descriptors that are associated with the image contents of various time portions of the input media content are the result of applying mathematical processing to a subset of media content selected from the group consisting of three dimensional blocks of pixels, three dimensional edges, spatio-temporal regions with consistent motion, a priori objects, and collections of simple features.

6. The system of claim 4, in which the descriptors that are associated with the image contents of various time portions of the input media content are further optimized to detect action, conversation, cartoons, outdoor scenes, and text over video.

7. The system of claim 1, in which the descriptors that are associated with the audio contents of various portions of the input media content are the result of applying mathematical processing to a subset of media content selected from the group consisting of functions that analyze sound frequency range, sound energy versus frequency range, voice frequency range, music frequency range, word recognition, and language recognition.

8. The system of claim 1, in which the database contains both descriptors associated with positive viewer preference, and descriptors associated with negative viewer preference, and the suitable weighting function contains a subtraction operation that subtracts the negative viewer preference from the positive viewer preference.

9. The system of claim 1, in which the system memory and/or database may be supplemented with a supplemental database of other video descriptors and user preferences obtained from a plurality of other systems, other media content, and other users.

10. The system of claim 1, in which the one or more sensors to detect and record user viewing preferences detect information selected from the group consisting of:

video that the user plays at normal speed, video that the user skips past, video that the user plays at high speed, video that the user designates as preferred, video that the user designates as not preferred, video that image sensors on the system indicate is playing to an audience, video that image sensors indicate is not playing to an audience, video that image and sound sensors indicate is playing to an interested audience, and video that image and sound sensors indicate is playing to a non-interested audience.

11. The system of claim 1, in which the system compresses video to fit within a designated time duration that is shorter than the length of the video by choosing the input video time portions associated with the highest user preference, and not choosing the input video time portions associated with lower user preference, so as to cause the input video to be played back within the designated time duration.

12. The system of claim 1, in which the descriptors that are associated with the image contents of various time portions of the input media content are produced from image feature recognition algorithms selected from the group consisting of the Viola-Jones algorithm, the Rowley-Baluja-Kanade algorithm, the Schneiderman-Kanade algorithm, the Roth-Yang-Ahuja algorithm, and the Chum algorithm.

13. The system of claim 1, in which the database is a relational database.

14. The system of claim 1, in which the system is capable of distinguishing between different users, and the associated control software controls a media content based upon an individual user'"'"'s profile of viewing preferences or history of viewing preferences.

15. The system of claim 1, in which the system further computes one or more video signatures that identify the overall appearance of the various time portions of the unknown media content (overall video identifiers);

correlates these overall video identifiers with overall user viewing preferences, and also stores this data in the persistent database;

in which the control processor and/or the video processor, under control of the control software, further controls the storage and/or playback of various time portions of a previously unanalyzed input media content (unknown media content) by;

d) generating video signatures of the various time portions of the unknown media content (overall video identifiers);

e) using the overall video identifiers as an index to the persistent database, and retrieving the overall user viewing preferences associated with the overall video identifiers;

f) using the overall user viewing preferences and the estimated user preference as input into a master user preference function;

g) and using this master user preference function to control the storage and playback of various time portions of the unknown media content.

16. The system of claim 15, in which the video signature is computed by a hash function or video DNA creation algorithm.

17. The system of claim 15, in which the master user preference function allows the overall user viewing preference to override the estimated user preference.

18. A digital video recorder system for storage and/or playback of media content, comprising:

at least one video processor capable of analyzing input media content and automatically generating one or more video signatures that identify the overall appearance of the various time portions of the unknown media content (overall video identifiers);

at least one sensor to detect and record overall user viewing preferences while the input media content is being played;

persistent memory capable of storing a persistent database that correlates the overall video identifiers with the recorded overall user viewing preferences;

at least one control processor, and associated control software;

in which the control processor and/or the video processor, under control of the control software, controls the storage and/or playback of various time portions of a previously unanalyzed input media content (unknown media content) by;

a) generating video signatures of the various time portions of the unknown media content (overall video identifiers);

b) using the overall video identifiers as an index to the persistent database, and retrieving the overall user viewing preferences associated with the overall video identifiers;

c) using the overall user viewing preferences and the estimated user preference as input into an overall user preference function;

d) and using this overall user preference function to control the storage and/or playback of various time portions of the unknown media content.

19. The system of claim 18, in which the video processor and the control processor are the same processor, and the video processor is cap able of compressing and/or decompressing a lossy or lossless video codec.

20. The system of claim 18, in which the video signature is computed by a hash function or video DNA creation algorithm.

21. A digital video recorder device for storage and playback of media content, comprising:

at least one video processor capable of analyzing input media content and automatically generating descriptors that are associated with the image and audio contents of various time portions of the input media content;

at least one sensor to detect and record user viewing preferences while the input media content is being played;

persistent memory capable of storing a persistent database that correlates the descriptors with the recorded user viewing preferences;

at least one control processor and associated operational software;

in which the control processor, under the control of the control software, controls the storage and playback of various time portions of a previously unanalyzed input media content (unknown media content), the processor including a descriptor generator configured to receive the unknown media content and to generate descriptors of the video and or audio portion of various time portions of the unknown media content (unknown video descriptors);

an index unit configured to utilize the unknown video descriptors as an index to the persistent database to retrieve user viewing preferences associated with the unknown video descriptors;

and a preference constructor configured to construct an estimated user preference of the various time portions of the unknown media content by using the retrieved user viewing preferences and a suitable weighing function;

wherein the processor is further configured to utilize this estimated user preference to control the storage and/or playback of various time portions of the unknown media content.

22. The device of claim 21, in which the video processor implements one or more lossy or lossless codecs selected from the group consisting of MPEG-1, MPEG-2, MPEG-4, MPEG-10, MPEG-4 (Advanced Simple Profile), H.264, Blackbird, Cinepak, Indeo 3/4/5, MJPEG, Sorenson 3, Microsoft Windows Media Video (WMV), RealVideo, CorePNG, Huffyuv, Lagarith, LCL, SheerVideo, Lossy codecs, or lossless codecs.

23. The device of claim 22, in which the video processor and the control processor are the same processor.

24. The device of claim 21, in which one or more of the descriptor generator, the index unit, or the preference constructor are implemented by software running on the video or control processor, software embedded into the video or control processor, dedicated hardware processor circuitry embedded into the video or control processor, or dedicated hardware circuitry located outside the video or control processor.

25. The device of claim 21, in which the system further computes one or more video signatures that identify the overall appearance of the various time portions of the unknown media content (overall video identifiers);

correlates these overall video identifiers with overall user viewing preferences, and also stores this data in the persistent database;

in which the control processor and/or the video processor, under control of the control software, further controls the storage and/or playback of various time portions of a previously unanalyzed input media content (unknown media content) by a signature generator configured to generate video signatures of the various time portions of the unknown media content (overall video identifiers);

a signature indexer configured to use the overall video identifiers as an index to the persistent database, and retrieve the overall user viewing preferences associated with the overall video identifiers;

and a signature preference constructor configured to use the overall user viewing preferences and the estimated user preference as input into a master user preference control unit;

and using this master user preference control unit controls the storage and/or playback of various time portions of the unknown media content.

26. The device of claim 25, in which the video signature is computed by a hash function or video DNA creation algorithm.

27. The device of claim 25, in which the persistent database that correlates the descriptors with the recorded user viewing preferences contains example data selected from a user-viewing-preference-neutral database of descriptors.

28. A method of controlling the storage and/or playback of media content, comprising:

inputting a plurality of different media contents, and automatically generating a plurality of descriptors that are associated with the image and/or audio contents of various time portions of the media content; and

detecting and recording the user viewing preferences associated with these various time portions of the different media contents;

correlating the descriptors with the user video viewing preferences in a persistent database so that the descriptors may be used as an index to the persistent database to retrieve user video viewing preferences;

and using this database to control the storage and/or playback of a previously unrated (unknown) media content by;

automatically generating a plurality of descriptors that are associated with the unknown media content (unknown content descriptors);

using the unknown content descriptors as an index into the database;

retrieving the user viewing preferences;

constructing an estimated user preference of the various time portions of the unknown media content by using the retrieved user viewing preferences and a suitable weighing function;

and using this estimated user preference to control the storage and/or playback of the unknown media content.

29. The method of claim 28, in which the descriptors are selected from population of descriptors known to have a positive or negative statistical correlation with user preferences.

30. The method of claim 28, in which the descriptors that are associated with the image contents of various time portions of the input media content are the outputs of mathematical functions selected from the group consisting of three dimensional blocks of pixels, three dimensional edges, spatio-temporal regions with consistent motion, a priori objects, and collections of simple features.

31. The method of claim 30, in which the descriptors that are associated with the image contents of various time portions of the input media content are further optimized to detect action, conversation, cartoons, outdoor scenes, and text over video.

32. The method of claim 28, in which the descriptors that are associated with the audio contents of various portions of the input media content are the outputs of mathematical functions selected from the group consisting of functions that analyze sound frequency range, sound energy versus frequency range, voice frequency range, music frequency range, word recognition, and language recognition.

33. The method of claim 28, in which the database contains both descriptors associated with positive viewer preference, and descriptors associated with negative viewer preference, and the suitable weighting function contains a subtraction operation that subtracts the negative viewer preference from the positive viewer preference.

34. The method of claim 28, in which the database may be supplemented with a supplemental database of other video descriptors and user preferences obtained from a plurality of other media content, and other users.

35. The method of claim 28, in which the user viewing preferences are selected from the group consisting of:

video that the user plays at normal speed, video that the user skips past, video that the user plays at high speed, video that the user designates as preferred, video that the user designates as not preferred, video that is playing to an audience, video that is not playing to an audience, video that is playing to an interested audience, and video that playing to a non-interested audience.

36. The method of claim 28, further compressing video to fit within a designated time duration that is shorter than the length of the video by choosing the input video time portions associated with the higher estimated user preference, and not choosing the input video time portions associated with lower estimated user preference, so as to cause the input video to be played back within the designated time duration.

37. The method of claim 28, in which the database is a relational database.

38. The method of claim 28, in which the persistent database that correlates the descriptors with the recorded user viewing preferences is supplemented with additional descriptors selected from a user viewing preference neutral database of descriptors.

39. The method of claim 38, in which the additional descriptors are selected based on a matching function between user selected metadata and additional descriptor metadata.

40. The method of claim 39, in which the user selected metadata is generated by computing the selected video signature of video images selected by the viewer, using the selected video signature as an index to a preference neutral database, retrieving database linked metadata, and using the database linked metadata to retrieve the additional descriptors.

41. The method of claim 38, further comprising:

computing one or more video signatures that identify the overall appearance of the various time portions of the unknown media content (overall video identifiers);

correlating these overall video identifiers with overall user viewing preferences, and also storing this data in the persistent database;

and controlling the storage and/or playback of various time portions of a previously unanalyzed input media content (unknown media content) by;

d) generating video signatures of the various time portions of the unknown media content (overall video identifiers);

e) using the overall video identifiers as an index to the persistent database, and retrieving the overall user viewing preferences associated with the overall video identifiers;

f) using the overall user viewing preferences and the estimated user preference as input into a master user preference function;

g) and using this master user preference function to control the storage and/or playback of various time portions of the unknown media content.

42. The method of claim 41, in which the video signature is computed by a hash function, video DNA creation algorithm, or other mathematical algorithm.

43. The method of claim 41, in which the master user preference function allows the overall user viewing preference to override the estimated user preference.

44. A method of controlling the storage and/or playback of media content, said method comprising:

inputting a plurality of different media contents, and automatically generating a plurality of descriptors that are associated with the image and/or audio contents of various time portions of the media content;

said descriptors being selected from population of descriptors known to have a positive or negative statistical correlation with user preferences;

said descriptors being the outputs of mathematical functions selected from the group consisting of three dimensional blocks of pixels, three dimensional edges, spatio-temporal regions with consistent motion, a priori objects, and collections of simple features;

or

said descriptors being the outputs of mathematical functions selected from the group consisting of functions that analyze sound frequency range, sound energy versus frequency range, voice frequency range, music frequency range, word recognition, and language recognition; and

detecting and recording the user viewing preferences associated with these various time portions of the different media contents;

correlating the descriptors with the user video viewing preferences in a persistent database so that the descriptors may be used as an index to the persistent database to retrieve user video viewing preferences;

and using this database to control the storage and/or playback of a previously unrated (unknown) media content by;

automatically generating a plurality of descriptors that are associated with the unknown media content (unknown content descriptors);

using the unknown content descriptors as an index into the database;

retrieving the user viewing preferences;

constructing an estimated user preference of the various time portions of the unknown media content by using the retrieved user viewing preferences and a suitable weighing function;

and using this estimated user preference to control the storage and/or playback of the unknown media content.

45. The method of claim 44, in which the descriptors that are associated with the image contents of various time portions of the input media content are further optimized to detect action, conversation, cartoons, outdoor scenes, and text over video.

46. The method of claim 44, in which the database contains both descriptors associated with positive viewer preference, and descriptors associated with negative viewer preference, and the suitable weighting function contains a subtraction operation that subtracts the negative viewer preference from the positive viewer preference.

47. The method of claim 44, in which the database may be supplemented with a supplemental database of other video descriptors and user preferences obtained from a plurality of other media content, and other users.

48. The method of claim 44, in which the user viewing preferences are selected from the group consisting of:

video that the user plays at normal speed, video that the user skips past, video that the user plays at high speed, video that the user designates as preferred, video that the user designates as not preferred, video that is playing to an audience, video that is not playing to an audience, video that is playing to an interested audience, and video that playing to a non-interested audience.

49. The method of claim 44, further compressing video to fit within a designated time duration that is shorter than the length of the video by choosing the input video time portions associated with the higher estimated user preference, and not choosing the input video time portions associated with lower estimated user preference, so as to cause the input video to be played back within the designated time duration.

50. The method of claim 44, in which the database is a relational database.

51. The method of claim 44, in which the persistent database that correlates the descriptors with the recorded user viewing preferences is supplemented with additional descriptors selected from a user viewing preference neutral database of descriptors.

52. The method of claim 44, in which the additional descriptors are selected based on a matching function between user selected metadata and additional descriptor metadata.

53. The method of claim 44, in which the user selected metadata is generated by computing the selected video signature of video images selected by the viewer, using the selected video signature as an index to a preference neutral database, retrieving database linked metadata, and using the database linked metadata to retrieve the additional descriptors.

54. The method of claim 44, further comprising:

computing one or more video signatures that identify the overall appearance of the various time portions of the unknown media content (overall video identifiers);

correlating these overall video identifiers with overall user viewing preferences, and also storing this data in the persistent database;

and controlling the storage and/or playback of various time portions of a previously unanalyzed input media content (unknown media content) by;

a) generating video signatures of the various time portions of the unknown media content (overall video identifiers);

b) using the overall video identifiers as an index to the persistent database, and retrieving the overall user viewing preferences associated with the overall video identifiers;

c) using the overall user viewing preferences and the estimated user preference as input into a master user preference function;

d) and using this master user preference function to control the storage and playback of various time portions of the unknown media content.

55. The method of claim 54, in which the video signature is computed by a hash function, video DNA creation algorithm, or other mathematical algorithm.

56. The method of claim 54, in which the master user preference function allows the overall user viewing preference to override the estimated user preference.

57. A method for controlling the storage and/or playback of media content, comprising:

inputting a plurality of different media contents, and automatically generating a plurality of video signatures that identify the overall appearance of various time portions of the unknown media content (overall video identifiers);

detecting and recording the overall user viewing preferences associated with these various time portions of the different media contents;

correlating the overall video identifiers with the overall user viewing preferences in a persistent database so that the overall video identifiers may be used as an index to the persistent database to retrieve the overall user viewing preferences;

and controlling the storage and/or playback of various time portions of a previously unanalyzed input media content (unknown media content) by;

a) generating video signatures of the various time portions of the unknown media content (overall video identifiers);

b) using the overall video identifiers as an index to the persistent database, and retrieving the overall user viewing preferences associated with the overall video identifiers;

c) using the overall user viewing preferences as input into an overall user preference function;

d) and using this overall user preference function to control the storage and/or playback of various time portions of the unknown media content.

58. The method of claim 57, in which the video signature is computed by a hash function or video DNA creation algorithm.