Method and apparatus for video digest generation

• US 8,442,384 B2
• Filed: 07/16/2008
• Issued: 05/14/2013
• Est. Priority Date: 07/16/2007
• Status: Active Grant

• Current Assignee: Vdoqwest, Inc.
• Original Assignee: Novafora, Inc.
• Inventors: Bronstein, Michael, Bronstein, Alexander, Rakib, Shlomo Selim, Matatyaou, Asaf
• Primary Examiner(s): Louie, Oscar
• Assistant Examiner(s): Shayanfar, Ali
• Application Number:
  US12/174,558
• Publication Number:
  US 20090022472A1
• Time in Patent Office: 1,763 Days
• Field of Search: 386/278, 725/46
• US Class Current: 386/278
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: G11B 27/329 on a disc [VTOC]
• CPC Class Codes: H04N 5/783 Adaptations for reproducing at a rate different from the recording rate

Abstract

Automated video digest system and method. The system decomposes video and audio portions of video media into low-level descriptors, producing a large number of such descriptors for each video image and scene. These descriptors are optimized to be both compatible with rapid automated descriptor creation, and also to have some positive or negative statistical correlation with the viewing preferences of typical human viewers. By comparing the number of descriptors correlated with positive human viewing preferences, versus the number of descriptors correlated with negative human viewing preferences, the automated system may analyze an unknown video program and make educated guesses as to which portions of the program are likely to be most interesting to the viewer on playback. Less interesting portions may be skipped or fast-forwarded though. The method may also be optimized to individual viewers. Such methods are useful for controlling the playback of digital video recorders and other systems.

Claims

1. A digital video recorder system for storage and/or playback of video media content, said video media content comprising audio portions and video image portions, said recorder comprising:

at least one video processor capable of analyzing at least the video image portions of said input media content and automatically generating descriptors that are associated with the video image and/or audio contents of various time portions of the input media content;

at least one sensor to detect and record user viewing preferences while the input media content is being played;

persistent memory capable of storing a persistent database that correlates the descriptors with the recorded user viewing preferences;

at least one control processor, and associated control software;

in which the control processor, under the control of the control software, controls the storage and/or playback of unknown media content comprising various time portions of a previously unanalyzed input media content by;

a) sending the unknown media content to the video processor, and directing the video processor to generate unknown video descriptors comprising descriptors of the video and or audio portion of various time portions of the unknown media content;

b) retrieving the user viewing preferences associated with the unknown video descriptors by using the unknown video descriptors as an index to the persistent database;

c) constructing an estimated user preference of the various time portions of the unknown media content by using the retrieved user viewing preferences and a suitable weighing function;

d) generating overall video identifiers comprising video signatures of the various time portions of the unknown media content;

e) retrieving the overall user viewing preferences associated with the overall video identifiers by using the overall video identifiers as an index to the persistent database, and;

f) inputting the overall user viewing preferences and the estimated user preference into a master user preference function; and

g) controlling the storage and playback of various time portions of the unknown media content with this master user preference function.

2. The system of claim 1, in which the video processor and the control processor are the same processor and the video processor is capable of compressing and/or decompressing a lossy or lossless video codec.

3. The system of claim 1, in which the video processor and the control processor are different processors, and in which the video processor is optimized for generating the descriptors.

4. The system of claim 1, in which the descriptors are selected from a population of descriptors known to have a positive or negative statistical correlation with user preferences.

5. The system of claim 4, in which the descriptors that are associated with the image contents of various time portions of the input media content are the result of applying mathematical processing to a subset of media content selected from the group consisting of three dimensional blocks of pixels, three dimensional edges, spatio-temporal regions with consistent motion, a priori objects, and collections of simple features.

6. The system of claim 4, in which the descriptors that are associated with the image contents of various time portions of the input media content are further optimized to detect action, conversation, cartoons, outdoor scenes, and text over video.

7. The system of claim 1, in which the descriptors that are associated with the audio contents of various portions of the input media content are the result of applying mathematical processing to a subset of media content selected from the group consisting of functions that analyze sound frequency range, sound energy versus frequency range, voice frequency range, music frequency range, word recognition, and language recognition.

8. The system of claim 1, in which the database contains both descriptors associated with positive viewer preference, and descriptors associated with negative viewer preference, and the suitable weighting function contains a subtraction operation that subtracts the negative viewer preference from the positive viewer preference.

9. The system of claim 1, in which the system memory and/or database may be supplemented with a supplemental database of other video descriptors and user preferences obtained from a plurality of other systems, other media content, and other users.

10. The system of claim 1, in which the one or more sensors to detect and record user viewing preferences detect information selected from the group consisting of:

video that the user plays at normal speed, video that the user skips past, video that the user plays at high speed, video that the user designates as preferred, video that the user designates as not preferred, video that image sensors on the system indicate is playing to an audience, video that image sensors indicate is not playing to an audience, video that image and sound sensors indicate is playing to an interested audience, and video that image and sound sensors indicate is playing to a non-interested audience.

11. The system of claim 1, in which the system compresses video to fit within a designated time duration that is shorter than the length of the video by choosing the input video time portions associated with the highest user preference, and not choosing the input video time portions associated with lower user preference, so as to cause the input video to be played back within the designated time duration.

12. The system of claim 1, in which the descriptors that are associated with the image contents of various time portions of the input media content are produced from image feature recognition algorithms selected from the group consisting of the Viola-Jones algorithm, the Rowley-Baluja-Kanade algorithm, the Schneiderman-Kanade algorithm, the Roth-Yang-Ahuja algorithm, and the Chum algorithm.

13. The system of claim 1, in which the database is a relational database.

14. The system of claim 1, in which the system is capable of distinguishing between different users, and the associated control software controls a media content based upon an individual user'"'"'s profile of viewing preferences or history of viewing preferences.

15. The system of claim 1, in which the video signature is computed by a hash function or video DNA creation algorithm.

16. The system of claim 1, in which the master user preference function allows the overall user viewing preference to override the estimated user preference.

17. A digital video recorder device for storage and playback of video media content, said video media content comprising audio portions and video image portions, said recorder comprising:

at least one video processor capable of analyzing at least the video image portions of said input media content and automatically generating descriptors that are associated with the video image and audio contents of various time portions of the input media content;

at least one sensor to detect and record user viewing preferences while the input media content is being played;

persistent memory capable of storing a persistent database that correlates the descriptors with the recorded user viewing preferences;

at least one control processor and associated operational software;

in which the control processor, under the control of the control software, is configured to controls the storage and playback of unknown media content comprising various time portions of a previously unanalyzed input media content, the processor including a descriptor generator configured to receive the unknown media content and to generate unknown video descriptors comprising descriptors of the video and or audio portion of various time portions of the unknown media content;

an index unit configured to utilize the unknown video descriptors as an index to the persistent database to retrieve user viewing preferences associated with the unknown video descriptors;

a preference constructor configured to construct an estimated user preference of the various time portions of the unknown media content by using the retrieved user viewing preferences and a suitable weighing function;

wherein the processor is further configured to utilize this estimated user preference to control the storage and/or playback of various time portions of the unknown media content;

in which said at least one video processor or control processor is further configured to compute one or more video signatures that identify overall video identifiers comprising the overall appearance of the various time portions of the unknown media content;

said at least one video processor or control processor is further configured to correlate these overall video identifiers with overall user viewing preferences, and also to store this data in the persistent database;

in which said at least one video processor or control processor, under control of the control software, are further configured to control the storage and/or playback of unknown media content comprising various time portions of a previously unanalyzed input media content by a signature generator configured to generate overall video identifiers comprising video signatures of the various time portions of the unknown media content;

a signature indexer configured to use the overall video identifiers as an index to the persistent database, and retrieve the overall user viewing preferences associated with the overall video identifiers;

a signature preference constructor configured to use the overall user viewing preferences and the estimated user preference as input into a master user preference control unit;

wherein said master user preference control unit is further configured to control the storage and/or playback of various time portions of the unknown media content.

18. The device of claim 17, in which the video processor implements one or more lossy or lossless codecs selected from the group consisting of MPEG-1, MPEG-2, MPEG-4, MPEG-10, MPEG-4 (Advanced Simple Profile), H.264, Blackbird, Cinepak, Indeo 3/4/5, MJPEG, Sorenson 3, Microsoft Windows Media Video (WMV), RealVideo, CorePNG, Huffyuv, Lagarith, LCL, SheerVideo, Lossy codecs, or lossless codecs.

19. The device of claim 18, in which the video processor and the control processor are the same processor.

20. The device of claim 17, in which one or more of the descriptor generator, the index unit, or the preference constructor are implemented by software running on the video or control processor, software embedded into the video or control processor, dedicated hardware processor circuitry embedded into the video or control processor, or dedicated hardware circuitry located outside the video or control processor.

21. The device of claim 17, in which the video signature is computed by a hash function or video DNA creation algorithm.

22. The device of claim 17, in which the persistent database that correlates the descriptors with the recorded user viewing preferences contains example data selected from a user-viewing-preference-neutral database of descriptors.

23. A computerized method of controlling the storage and/or playback of video media content said video media content comprising audio portions and video image portions, comprising:

inputting a plurality of different video media contents, and using at least one processor to perform the steps of;

automatically generating a plurality of descriptors that are associated with the video image portions and/or audio portions contents of various time portions of the media content; and

detecting and recording the user viewing preferences associated with these various time portions of the different media contents;

correlating the descriptors with the user video viewing preferences in a persistent database so that the descriptors may be used as an index to the persistent database to retrieve user video viewing preferences; and

using this database to control the storage and/or playback of unknown media content comprising a previously unrated media content by;

automatically generating unknown content descriptors comprising a plurality of descriptors that are associated with the unknown media content;

using the unknown content descriptors as an index into the database;

retrieving the user viewing preferences;

constructing an estimated user preference of the various time portions of the unknown media content by using the retrieved user viewing preferences and a suitable weighing function; and

controlling the storage and/or playback of the unknown media content based on said estimated user preference;

wherein the persistent database that correlates the descriptors with the recorded user viewing preferences is supplemented with additional descriptors from a user viewing preference neutral database of descriptors.

24. The method of claim 23, in which the descriptors are selected from a population of descriptors known to have a positive or negative statistical correlation with user preferences.

25. The method of claim 23, in which the descriptors that are associated with the image contents of various time portions of the input media content are the outputs of mathematical functions selected from the group consisting of three dimensional blocks of pixels, three dimensional edges, spatio-temporal regions with consistent motion, a priori objects, and collections of simple features.

26. The method of claim 25, in which the descriptors that are associated with the image contents of various time portions of the input media content are further optimized to detect action, conversation, cartoons, outdoor scenes, and text over video.

27. The method of claim 23, in which the descriptors that are associated with the audio contents of various portions of the input media content are the outputs of mathematical functions selected from the group consisting of functions that analyze sound frequency range, sound energy versus frequency range, voice frequency range, music frequency range, word recognition, and language recognition.

28. The method of claim 23, in which the database contains both descriptors associated with positive viewer preference, and descriptors associated with negative viewer preference, and the suitable weighting function contains a subtraction operation that subtracts the negative viewer preference from the positive viewer preference.

29. The method of claim 23, in which the database may be supplemented with a supplemental database of other video descriptors and user preferences obtained from a plurality of other media content, and other users.

30. The method of claim 23, in which the user viewing preferences are selected from the group consisting of:

video that the user plays at normal speed, video that the user skips past, video that the user plays at high speed, video that the user designates as preferred, video that the user designates as not preferred, video that is playing to an audience, video that is not playing to an audience, video that is playing to an interested audience, and video that playing to a non-interested audience.

31. The method of claim 23, further compressing video to fit within a designated time duration that is shorter than the length of the video by choosing the input video time portions associated with the higher estimated user preference, and not choosing the input video time portions associated with lower estimated user preference, so as to cause the input video to be played back within the designated time duration.

32. The method of claim 23, in which the database is a relational database.

33. The method of claim 23, in which the additional descriptors are selected based on a matching function between user selected metadata and additional descriptor metadata.

34. The method of claim 33, in which the user selected metadata is generated by computing the selected video signature of video images selected by the viewer, using the selected video signature as an index to a preference neutral database, retrieving database linked metadata, and using the database linked metadata to retrieve the additional descriptors.

35. The method of claim 23, further comprising:

computing one or more overall video identifiers comprising video signatures that identify the overall appearance of the various time portions of the unknown media content;

correlating these overall video identifiers with overall user viewing preferences, and also storing this data in the persistent database;

and controlling the storage and/or playback of unknown media content comprising various time portions of a previously unanalyzed input media content by;

d) generating overall video identifiers comprising video signatures of the various time portions of the unknown media content;

e) retrieving the overall user viewing preferences associated with the overall video identifiers by using the overall video identifiers as an index to the persistent database;

f) inputting the overall user viewing preferences and the estimated user preference into a master user preference function;

g) and controlling the storage and/or playback of various time portions of the unknown media content based on said master user preference function.

36. The method of claim 35, in which the video signature is computed by a hash function, video DNA creation algorithm, or other mathematical algorithm.

37. The method of claim 35, in which the master user preference function allows the overall user viewing preference to override the estimated user preference.

38. A computerized method of controlling the storage and/or playback of video media content said video media content comprising audio portions and video image portions, said method comprising:

inputting a plurality of different media contents, and using at least one processor to perform the steps of;

automatically generating a plurality of descriptors that are associated with the video image portions and/or audio portion contents of various time portions of the media content;

said descriptors being selected from population of descriptors known to have a positive or negative statistical correlation with user preferences;

said descriptors being the outputs of mathematical functions selected from the group consisting of three dimensional blocks of pixels, three dimensional edges, spatio-temporal regions with consistent motion, a priori objects, and collections of simple features;

or said descriptors being the outputs of mathematical functions selected from the group consisting of functions that analyze sound frequency range, sound energy versus frequency range, voice frequency range, music frequency range, word recognition, and language recognition;

and detecting and recording the user viewing preferences associated with these various time portions of the different media contents;

correlating the descriptors with the user video viewing preferences in a persistent database so that the descriptors may be used as an index to the persistent database to retrieve user video viewing preferences;

and controlling the storage and/or playback of an unknown media content comprising previously unrated media content with said persistent database by;

automatically generating a plurality of unknown content descriptors comprising descriptors that are associated with the unknown media content;

database indexing the database with the unknown content descriptors;

retrieving the user viewing preferences;

constructing an estimated user preference of the various time portions of the unknown media content by using the retrieved user viewing preferences and a suitable weighing function; and

controlling the storage and/or playback of the unknown media content by further;

computing one or more overall video identifiers comprising video signatures that identify the overall appearance of the various time portions of the unknown media;

correlating these overall video identifiers with overall user viewing preferences, and also storing this data in the persistent database;

controlling the storage and/or playback of unknown media content comprising various time portions of a previously unanalyzed input media content by;

a) generating overall video identifiers comprising video signatures of the various time portions of the unknown media content;

b) retrieving the overall user viewing preferences associated with the overall video identifiers by using the overall video identifiers as an index to the persistent database;

c) inputting the overall user viewing preferences and the estimated user preference into a master user preference function;

d) and controlling the storage and playback of various time portions of the unknown media content with this master user preference function.

39. The method of claim 38, in which the descriptors that are associated with the image contents of various time portions of the input media content are further optimized to detect action, conversation, cartoons, outdoor scenes, and text over video.

40. The method of claim 38, in which the database contains both descriptors associated with positive viewer preference, and descriptors associated with negative viewer preference, and the suitable weighting function contains a subtraction operation that subtracts the negative viewer preference from the positive viewer preference.

41. The method of claim 38, in which the database may be supplemented with a supplemental database of other video descriptors and user preferences obtained from a plurality of other media content, and other users.

42. The method of claim 38, in which the user viewing preferences are selected from the group consisting of:

video that the user plays at normal speed, video that the user skips past, video that the user plays at high speed, video that the user designates as preferred, video that the user designates as not preferred, video that is playing to an audience, video that is not playing to an audience, video that is playing to an interested audience, and video that playing to a non-interested audience.

43. The method of claim 38, further compressing video to fit within a designated time duration that is shorter than the length of the video by choosing the input video time portions associated with the higher estimated user preference, and not choosing the input video time portions associated with lower estimated user preference, so as to cause the input video to be played back within the designated time duration.

44. The method of claim 38, in which the database is a relational database.

45. The method of claim 38, in which the persistent database that correlates the descriptors with the recorded user viewing preferences is supplemented with additional descriptors selected from a user viewing preference neutral database of descriptors.

46. The method of claim 38, in which the additional descriptors are selected based on a matching function between user selected metadata and additional descriptor metadata.

47. The method of claim 38, in which the user selected metadata is generated by computing the selected video signature of video images selected by the viewer, using the selected video signature as an index to a preference neutral database, retrieving database linked metadata, and using the database linked metadata to retrieve the additional descriptors.

48. The method of claim 38, in which the video signature is computed by a hash function, video DNA creation algorithm, or other mathematical algorithm.

49. The method of claim 38, in which the master user preference function allows the overall user viewing preference to override the estimated user preference.