AUTOMATICALLY GENERATING NOTES AND CLASSIFYING MULTIMEDIA CONTENT SPECIFIC TO A VIDEO PRODUCTION

• US 20150244943A1
• Filed: 02/24/2014
• Published: 08/27/2015
• Est. Priority Date: 02/24/2014
• Status: Active Grant

• Current Assignee: Invent.ly LLC
• Original Assignee: Invent.ly LLC
• Inventors: Brown, Stephen J.
• Granted Patent: US 9,582,738 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06V 20/35 Categorising the entire scene, e.g. birthday party or wedding scene
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: H04N 5/262 Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

Automatically classifying multimedia content that is specific to a video production includes obtaining, from a video sensor embedded in a video capturing device that captures a video, a time series of location data, direction data, orientation data, and a position of the first user and the second user, identifying for any given duration in the video, corresponding first user data, corresponding second user data, and corresponding video sensor data, annotating the video with the corresponding first user data, the corresponding second user data, and the corresponding video sensor data to obtain an annotated multimedia content, performing a comparison of a data pattern of the annotated multimedia content with data patterns of a script stored in a database to obtain a recommended section for the annotated multimedia content, and automatically classifying the annotated multimedia content by associating the annotated multimedia content with the recommended section.

Claims

1. A method of automatically classifying multimedia content that is specific to a video production based on a user context, said method comprising:

obtaining from a video sensor embedded in a video capturing device that captures a video associated with a first user and a second user, video sensor data comprising a time series of location data, direction data, orientation data, and a position of said first user and said second user being recorded;

identifying, for any given duration associated with said video, corresponding first user data, corresponding second user data, and corresponding video sensor data;

annotating said video with said corresponding first user data, said corresponding second user data, and said corresponding video sensor data to obtain an annotated multimedia content;

performing, by said processor, a comparison of a data pattern of said annotated multimedia content with data patterns that correspond to a plurality of predefined sections of a script stored in a database to obtain a recommended section for said annotated multimedia content, wherein said plurality of predefined sections are specific to said video production, and wherein said data pattern comprises first user data, second user data, and video sensor data associated with a section of said annotated multimedia content; and

automatically classifying, by said processor, said annotated multimedia content by associating said annotated multimedia content with said recommended section.

2. The method of claim 1, further comprising generating a list of prioritized predefined sections from said plurality of predefined sections based said comparison, wherein said recommended section is a highest priority section from said list of prioritized predefined sections.

3. The method of claim 2, further comprising:

receiving at least one user-suggested section from a user;

processing said at least one user-suggested section from said user; and

associating said annotated multimedia content with said at least one user-suggested section instead of said recommended section.

4. The method of claim 3, further comprising:

establishing a quantitative weight factor associated with at least one of said video sensor data associated with said video;

modifying said quantitative weight factor associated with said at least one of said video sensor data associated with said video to obtain a modified weight factor such that said user-suggested section becomes prioritized over said recommended section upon said comparison being performed based on said modified weight factor; and

determining a modified recommended section for a subsequent video to be recorded at a subsequent time based on said modified weight factor.

5. The method of claim 4, wherein said script comprises at least one of identities, dialogues, activities, characters, locations, directions, orientations, positions, movements, scenes, and themes associated with said first user and said second user.

6. The method of claim 3, further comprising automatically generating notes for at least one of said recommended section or said user-suggested section of said annotated multimedia content from at least one of said first user data, said second user data, and said video sensor data based on said comparison and said quantitative weight factor, wherein said notes comprise at least one of identities of said first user and said second user, a line in said script at which said first user or said second user is talking, character information, scene information, and event information, and wherein said activities of said first user or said second user comprise at least one of said first user facing said second user, said first user approaching said video capturing device at a distance from said first user, said second user talking to said first user, said second user talking and facing said video capturing device, and health information of said first user or said second user.

7. The method of claim 3, further comprising obtaining data from at least one additional sensor that senses identities or health-related information comprising heart rates, perspirations, and respirations of said first user and said second user, wherein said data pattern comprises said data from said at least one additional sensor that is specific to at least one of said recommended section or said user-suggested section of said annotated multimedia content.

8. The method of claim 1, further comprising synchronizing said first user data, said second user data, and said video sensor data to identify said corresponding first user data, said corresponding second user data, and said corresponding video sensor data for said any given duration.

9. The method of claim 1, wherein said recommended section of said annotated multimedia content is obtained based on a set of rules that are stored in said database.

10. The method of claim 1, further comprising obtaining, from a central network computing device, data from prior scenes and shoots that are specific to said video production, said location data, and weather information of said shoots.

11. The method of claim 3, wherein an annotation of said video, a classification of said recommended section or said user-suggested section of said annotated multimedia content, and a generation of notes for said recommended section or said user-suggested section, is based on prior multimedia content comprising predefined data, user information, and classifications associated with videos that were previously captured by said video production.

12. A system for automatically classifying multimedia content that is specific to a video production based on a user context, said system comprising:

a video capturing device that captures a video associated with a first user and a second user;

a video sensor embedded in said video capturing device, wherein said video sensor captures a video sensor data comprising a time series of a location data, a direction data, an orientation data, and a position of said first user and said second user being recorded;

a memory unit that stores instructions;

a database operatively connected to said memory unit; and

a processor, when configured by said instructions, executes a set of modules comprising;

a sensor data obtaining module that obtains said first user data, said second user data, and said video sensor data;

an identification module that identifies for any given duration associated with said video, corresponding first user data, corresponding second user data, and corresponding video sensor data;

an annotation module that annotates said video with said corresponding first user data, said corresponding second user data, and said corresponding video sensor data to obtain an annotated multimedia content;

a comparison module that performs a comparison of a data pattern of said annotated multimedia content with data patterns that correspond to a plurality of predefined sections of a script stored in said database to obtain a recommended section, wherein said plurality of predefined sections are specific to said video production, and wherein said data pattern comprises a first user data, a second user data and a video sensor data associated with a section of said annotated multimedia content; and

a classification module that automatically classifies said annotated multimedia content by associating said annotated multimedia content with said recommended section.

13. The system of claim 12, wherein a list of prioritized predefined sections from said plurality of predefined sections is generated based said comparison, wherein the recommended section is a highest priority section from the list of prioritized predefined sections.

14. The system of claim 13, wherein said set of modules further comprises:

a weight assigning module that establishes a quantitative weight factor associated with at least one of said video sensor data associated with said video, modifies said quantitative weight factor associated with said at least one of said video sensor data associated with said video to obtain a modified weight factor such that a user-suggested section becomes prioritized over said recommended section upon said comparison being performed based on said modified weight factor, and determines a modified recommended section for a subsequent video to be recorded at a subsequent time based on said modified weight factor; and

a generation module that generates notes for at least one of said user-suggested section or said recommended section of said annotated multimedia content from at least one of said first user data, said second user data, and said video sensor data.

15. The system of claim 14, wherein said notes for said user-suggested section or said recommended section of said annotated multimedia content comprise at least one of identities of said first user and said second user, a line in said script at which said first user or said second user is talking, activities of said first user or said second user, character information, scene information, and event information.

16. The system of claim 15, further comprising at least one additional sensor that senses identities, or health-related information comprising heart rates, perspirations, respirations of said first user and said second user.

17. The system of claim 16, wherein said set of modules comprise a synchronization module that synchronizes said video, said first user data, said second user data, and said video sensor data to identify said corresponding first user data, said corresponding second user data, and said corresponding video sensor data for said any given duration.

18. The system of claim 17, further comprising a central network computing device that comprises data from prior scenes and shoots that are specific to said video production, a location data, and weather information from said shoots.

19. A system for automatically generating notes for a multimedia content that is specific to a video production based on a user context, said system comprising:

a first audio capturing device adapted to be attached to a first user, wherein said first audio capturing device captures a first audio;

a second audio capturing device adapted to be attached to a second user, wherein said second audio capturing device captures a second audio;

a first audio sensor coupled to said first audio capturing device, wherein said first audio sensor captures a first user data comprising a time series of a location data, a direction data, and an orientation data associated with said first user;

a second audio sensor coupled to said second audio capturing device, wherein said second audio sensor captures a second user data comprising a time series of a location data, a direction data, and an orientation data associated with said second user;

a video capturing device that captures a video associated with said first user and said second user;

a video sensor embedded in said video capturing device, wherein said video sensor captures a video sensor data comprising a time series of a location data, a direction data, an orientation data, and a position of said first user and said second user being recorded;

a memory unit that stores instructions;

a database operatively connected to said memory unit; and

a processor, when configured by said instructions, executes a set of module comprising;

a first module that obtains said first user data, said second user data, and said video sensor data;

a second module that identifies for any given duration associated with said first audio, said second audio, or said video, corresponding first user data, corresponding second user data, and corresponding video sensor data;

a third module that annotates at least one of said first audio, said second audio and said video with said corresponding first user data, said corresponding second user data, and said corresponding video sensor data to obtain an annotated multimedia content;

a fourth module that identifies that performs a comparison of a data pattern of said annotated multimedia content with data patterns that correspond to a plurality of predefined sections of a script stored in said database to obtain a recommended section, wherein said plurality of predefined sections are specific to said video production, and wherein said data pattern comprises a first user data, a second user data and a video sensor data associated with a section of said annotated multimedia content;

a fifth module that automatically classifies said annotated multimedia content by associating said annotated multimedia content with said recommended section; and

a sixth module that automatically generates notes for said recommended section of said annotated multimedia content from at least one of said first user data, said second user data, and said video sensor data that are associated with said section, and said predefined data based on said comparison.

20. The system of claim 19, wherein said script comprises at least one of identities, dialogues, activities, characters, locations, directions, orientations, positions, movements, scenes, and themes associated with said first user and said second user.