Automated creation of storyboards from screenplays

• US 9,106,812 B1
• Filed: 12/29/2011
• Issued: 08/11/2015
• Est. Priority Date: 12/29/2011
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Price, Roy F., Schwabland, Peter A., King, Joseph
• Primary Examiner(s): Tran, Quoc A
• Application Number:
  US13/340,029
• Time in Patent Office: 1,321 Days
• Field of Search: 715/716, 715200-204, 715719-723, 715/731, 715/756, 715/732, 715/726, 715/209, 715/273
• US Class Current: 1/1
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/43 Querying
• CPC Class Codes: G06F 16/433 using audio data
• CPC Class Codes: G06F 16/483 using metadata automatically derived from the content
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G10L 13/027 Concept to speech synthesisers; Generation of natural phrases from machine-based concepts generation of parameters for speech synthesis out of text G10L13/08
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: G11B 27/036 Insert-editing
• CPC Class Codes: H04N 1/212 Motion video recording combined with still video recording television signal recording H04N5/76
• CPC Class Codes: H04N 1/2129 Recording in, or reproducing from, a specific memory area or areas, or recording or reproducing at a specific moment
• CPC Class Codes: H04N 21/4302 Content synchronisation processes, e.g. decoder synchronisation
• CPC Class Codes: H04N 21/47205 for manipulating displayed content, e.g. interacting with MPEG-4 objects, editing locally
• CPC Class Codes: H04N 21/8549 Creating video summaries, e.g. movie trailer retrieval in video databases by using presentations in form of a video summary G06F16/739

Abstract

Computing resources may automatically generate a rough draft of storyboard by using textual analysis to extract meaning from a screenplay. The storyboard may be accompanied by spoken dialog, sound effects, and a soundtrack. Users may provide pre-production decisions regarding appearance of characters and backgrounds. The users may include a community of different individuals who contribute content and suggest modifications to the storyboard. The computing resources may also access logic to select visual elements for the storyboard from libraries of existing content or from custom content prepared by one of the users. Pagination of the screenplay and location of dialog and action on the pages of the screenplay may be used to create a timeline for the storyboard and any accompanying audio.

Claims

1. A computer-implemented method of generating a storyboard from a screenplay, the method comprising:

receiving, by one or more computing devices, the screenplay from a first user, the screenplay comprising one or more pages of text that include one or more sluglines, a character dialogue, and one or more screenplay descriptions, the one or more screenplay descriptions corresponding to a description of a character, scene or action;

creating, by at least one of the one or more computing devices, a timeline having a predetermined amount of time associated with each page of the screenplay, the timeline comprising one or more scenes, each scene beginning at a time point corresponding to a location of one of the one or more sluglines on the one or more pages of the screenplay;

selecting, by at least one of the one or more computing devices, a background image for one of the one or more scenes based at least in part on text contained in a slugline indicating a beginning of the scene;

associating, by at least one of the one or more computing devices, a facial expression from a plurality of facial expressions to at least one character based at least in part on the one or more screenplay descriptions or the character dialogue, wherein each facial expression of the plurality of facial expressions comprises a predetermined, anatomically-based facial appearance;

associating, by at least one of the one or more computing devices, the facial expression with a character image for the at least one character based at least in part on the one or more screenplay descriptions or the character dialogue;

positioning, by at least one of the one or more computing devices, the character image in front of the background image;

creating, by at least one of the one or more computing devices, one or more audio tracks comprising speech corresponding to the character dialogue;

synchronizing, by at least one of the one or more computing devices, the one or more audio tracks with the timeline based at least in part on a relative length in fractions of the pages of the character dialogue; and

compiling, by at least one of the one or more computing devices, the storyboard based at least in part on the background image, the character image, the timeline, and the one or more audio tracks.

2. The method of claim 1, wherein the one or more screenplay descriptions comprise at least one of a sentiment, a mood or an emotion of a character.

3. The method of claim 1, wherein the background image is based at least in part on one of the one or more screenplay descriptions.

4. A computer-implemented method comprising:

receiving, by one or more computing devices, a screenplay comprising text and formatting, the screenplay having at least one or more screenplay descriptions that correspond to a description of a character, scene or action;

analyzing, by at least one of the one or more computing devices, the text and the formatting of the screenplay to extract storyboard content that includes identifying at least one character, and a character dialogue;

determining, by at least one of the one or more computing devices, a facial expression of a plurality of facial expressions of the at least one character based at least in part on the screenplay description or the character dialogue, wherein the facial expression comprises a predetermined, anatomically-based facial appearance;

associating, by at least one of the one or more computing devices, the facial expression with a character image of the at least one character based at least in part on the screenplay description or the character dialogue; and

compiling, by at least one of the one or more computing devices, a storyboard based at least in part on the storyboard content.

5. The method of claim 4, wherein analyzing the text and the formatting comprises ascertaining a context based at least in part on the formatting and ascertaining a meaning of the text based at least in part on text analytics and the context.

6. The method of claim 4, wherein the analyzing comprises identifying a slugline that includes the scene description, wherein the slugline is identified at least in part by margins surrounding the slugline and capitalization of text in the slugline.

7. The method of claim 4, wherein the description of the character comprises a gender or a physical characteristic.

8. The method of claim 4, wherein the storyboard content further includes a scene description that comprises a scene location, and wherein the generating the storyboard includes placing a background image that corresponds to the scene location.

9. The method of claim 8, wherein the scene description comprises a camera technique.

10. The method of claim 4, wherein generating the storyboard further comprises selecting, for inclusion in the storyboard, audio based at least in part on the character dialogue.

11. The method of claim 10, wherein the audio comprises synthesized speech generated by text-to-speech conversion of text of the character dialogue.

12. A computer-readable storage medium having stored therein instructions, which, when executed by a processor, cause a computing device to perform acts comprising:

receiving a screenplay, the screenplay having at least one or more screenplay descriptions that correspond to a description of a character, scene or action;

applying text analytics to identify a character, character dialogue, and a scene in the screenplay;

performing a sentiment analysis of the one or more screenplay descriptions to associate the character with at least one of a sentiment, a mood, or an emotion, the sentiment analysis based at least in part on the text analytics;

determining a facial expression of the character from a plurality of facial expressions based at least in part on the sentiment analysis, wherein each facial expression of the plurality of facial expressions comprises a predetermined, anatomically-based facial appearance;

associating the facial expression with the character;

generating a timeline for the screenplay wherein a time range of the timeline is selected based at least in part on a length of the screenplay;

electronically publishing a project comprising the screenplay;

receiving custom content for the project; and

generating a storyboard based at least in part on the character, the dialogue, the scene, the timeline, and the custom content.

13. The medium of claim 12, wherein the custom content comprises a recording of lines of the character dialogue.

14. The medium of claim 13, wherein the acts further comprise:

making the project comprising the screenplay available for editing; and

receiving custom content for the project, wherein the custom content comprises another recording of the lines of the character dialogue such that the project comprises two different recordings of the lines of the character dialogue.

15. The medium of claim 14, wherein the acts further comprise:

presenting a user interface configured to receive a selection between the two different recordings; and

generating the storyboard based at least in part on the selected one of the two different recordings.

16. A computing device comprising:

one or more processors; and

a memory coupled to the one or more processors, the memory comprising;

a screenplay analysis module configured to identify, in a screenplay having one or more pages, sections of dialogue, sections of action, and relative locations of the sections of dialogue and the sections of action within the one or more pages;

a timeline generation module configured to generate a timeline for the screenplay based at least in part on a number of pages in the screenplay;

a synchronization module configured to map audio corresponding to the sections of dialogue to the timeline based at least in part on the relative locations of the sections of dialogue and the sections of action within the one or more pages;

a character image generation module configured to;

analyze audio that corresponds to a section of dialogue to identify a mood, sentiment, or emotion of at least one character;

determine a facial expression of a plurality of facial expressions of the at least one character based at least in part on the audio analysis, wherein the facial expression comprises a predetermined, anatomically-based facial appearance; and

associate the facial expression with a character image of the at least one character for inclusion in the storyboard; and

a storyboard generation module configured to generate a storyboard based at least in part on the timeline, the storyboard comprising an audio track based at least in part on the section of dialogue mapped to the timeline.

17. The computing device of claim 16, wherein the timeline generation module allocates an average of one minute of time in the timeline for one page of the screenplay.

18. The computing device of claim 16, wherein the timeline generation module divides the timeline into scenes based at least in part on one or more sluglines in the screenplay.

19. The computing device of claim 18, wherein the timeline generation module divides the scenes into camera shots based at least in part on an analysis of a slugline at a beginning of a scene and at least one of a section of dialogue or a section of action in the scene.

20. The computing device of claim 16, wherein the character image is further based at least in part on textual analysis of sections of dialogue associated with the character.

21. The computing device of claim 16, wherein the memory further comprises a background selection module configured to select a background for the storyboard based at least in part on textual analysis of a slugline.

22. The computing device of claim 16, wherein the audio analysis comprises segmenting a speech sample into voiced and unvoiced sounds, words or syllables to allow a quantitative description of homogenous parts of the audio.

23. The computer device of claim 16, wherein the memory further comprises a screenplay analysis module configured to perform a disambiguation analysis of a section of dialogue, the disambiguation analysis determining at least one interpretation of multiple possible interpretations of the section of dialogue.