Creating multimedia content for animation drawings by synchronizing animation drawings to audio and textual data

• US 10,423,716 B2
• Filed: 10/28/2013
• Issued: 09/24/2019
• Est. Priority Date: 10/30/2012
• Status: Active Grant

• Current Assignee: Sergey Anatoljevich Gevlich
• Original Assignee: Sergey Anatoljevich Gevlich
• Inventors: Gevlich, Sergey Anatoljevich
• Primary Examiner(s): Zuberi, Mohammed H
• Assistant Examiner(s): Barnes, Jr., Carl E
• Application Number:
  US14/437,424
• Publication Number:
  US 20150278181A1
• Time in Patent Office: 2,157 Days
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 40/131 Fragmentation of text files, e.g. creating reusable text-blocks; Linking to fragments, e.g. using XInclude; Namespaces
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes

Abstract

A method and system for creating multimedia prototype presentations use the linear compression or stretching of playback speed record of creating and editing graphic images, which does not lead to degradation of quality perception. When using manually created drafts, schemes and drawings as graphic images of the presentation, the playback of record of their creating and editing process can be sped up or slowed down in a broad time range without losing quality perception of the visual content. The synchronization of the audio and the video track is made by a linear compression or stretching playback speed record of creating and editing the graphic images until the playback duration of the frame visual content and the duration of its sound match.

Claims

1. A computer-implemented method for creating an animation, the method comprising:

using a processor of a touchscreen device, defining animation sequences and an order of transition between the animation sequences;

preparing a composition of a visual content and an audio track of the animation by performing

(a) on the touchscreen device, drawing dynamic elements corresponding to the animation sequences, wherein the dynamic elements are added to a static background image of each animation sequence, by recording a process of the drawing that uses a finger or a stylus on the touchscreen device;

(b) entering a text to be narrated as the audio track into the touchscreen device;

(c) specifying boundaries between fragments of the text by using a stylus or a finger to manually indicate the boundaries, wherein the fragments of the text correspond to neighboring animation sequences, and placing markers of a text markup at locations of the boundaries;

(d) recording the audio track, using a microphone of the touchscreen device, by narrating the text so that pauses in the audio track correspond to the markers and represent transitions between the neighboring animation sequences;

(e) splitting the audio track into portions associated with the animation sequences during the narration;

(f) creating a configuration file for each animation sequence by determining a playback duration of the corresponding animation sequence and by making a uniform time scaling of the visual representation of the animation, so that a duration of the visual representation is adjusted to match a duration of the audio track recorded through the microphone;

(g) setting an aspect ratio, resolution, color, transparency and FPS of the animation;

simultaneously playing back, on the touchscreen device, the audio track and a time scaled visual representation of the dynamic elements as they were being drawn, and recording a resulting mix of the audio track and the dynamic elements into a configuration file of each animation sequence;

playing back the configuration file of the animation sequence on the touchscreen device;

combining the configuration files of the animation sequences in the order of transition of the animation sequences into a configuration multimedia file of the animation; and

saving the configuration multimedia file of the animation.

2. The method of claim 1, further comprising editing the text, the audio track and the dynamic elements and re-configuring the animation film.

3. The method of claim 1, wherein the text is entered into the touchscreen device by any of:

loading prepared text from an external source;

entering the text using an on-screen keyboard; and

inserting the text verbally via a microphone with speech-to-text recognition.

4. The method of claim 1, further comprising marking up the audio track, including the steps of:

displaying the text and audio frames synchronized at a particular moment in time visually presented as segments corresponding to the animation sequences, so that a visual representation of the text, bars or edges of the animation sequences for which the audio track has already been marked up is visually different from a presentation of the text, bars or edges of the animation sequences for which the audio track has not yet been marked up; and

displaying a visual border of the text corresponding to splitting animation sequences into the animation sequences with marked up and not marked up audio content, so that a visual location of the border on a screen stays constant as a result of scrolling of the frame bars in a process of marking up;

inserting a marker of an audio track markup where synchronization of one animation sequence ends and synchronization of another animation sequence begins in case of a manual markup; and

automatically matching recognized words to texts of the animation sequences and automatically marking up the audio track in case of an automatic speech recognition.

5. The method of claim 1, further comprising creation and editing dynamic elements of a animation sequence by:

automatically detecting drawn dynamic elements and correcting their geometry, wherein a duration of drawing of the dynamic elements is equal to a duration of the drawing of the original dynamic elements, excluding an actual time for automatic correction and reviewing suggested versions;

attaching indicating lines and captions to the dynamic elements and automatically move matching lines and text along with the dynamic elements when moving them during editing;

setting a transition mode between the dynamic elements of two adjacent animation sequences;

playing back a fragment of the audio track of a current animation sequence in order to check creation of the dynamic elements of the animation sequence; and

testing an animation of an already created part of the visual imagery of the animation sequence in order to check the creation of the dynamic elements of the animation sequence.

6. The method of claim 1, further comprising creating a configuration file of the animation sequence for synchronization of animation with audio, wherein a time of visual representation of creating and editing dynamic elements of is determined:

from a start marker of the audio track to the end marker of the audio track; and

from a start of an actual sound of a first word of the text of the animation sequence to of a end of a sound of a last word of the text of the animation sequence,

wherein additional pauses are inserted before the start of the sound of the first word of the first animation sequence and after ending of the sounding of the last word of the last animation sequence.

7. The method of claim 1, further comprising correction of the animation sequences by:

manually indicating a boundary of two text fragments for two new neighboring animation sequences, and placing markers of a text markup at a location of the boundary;

determining which one of the two new created animation sequence has the dynamic elements of an original animation sequence;

creating dynamic elements for the second animation sequence;

deleting markers of text markup and audio track of the animation sequences, if there are no dynamic elements found in the animation sequences;

deleting the marker of the text markup and the audio track of the animation sequences and attaching the dynamic elements to the resulting integrated animation sequence, if the dynamic elements are present only in one of the integrated animation sequences; and

if there are no dynamic elements present in both integrated animation sequences;

deleting the markers of the text markup and the audio track of the animation sequences;

creating the resulting audio track by consecutive connection of two original audio tracks keeping their original sequence; and

creating resulting dynamic elements by consecutive connection of two original dynamic elements keeping their original sequence and a frame rate of the original animation sequences.

8. The method of claim 1, further comprising correcting markup of the audio track by:

determining two adjacent animation sequences and location of a marker of the audio track markup to be corrected;

displaying an image of the audio track with a sound end marker;

playing back continuous consistent sounds of both animation sequences and simultaneously displaying a time line with a locator of a current position of the marker of the audio track markup;

moving a locator of the marker of audio track markup on time line right or left after listening to an end of one animation sequence and beginning of the following one; and

checking result after moving to playback audio of the track from time mark N seconds to a new position of marker of audio track markup, where N is from 0 to 4 seconds.

9. The method of claim 1, further comprising executing a transition between two adjacent animation sequences when creating a new frame by:

saving an image of the previous animation sequence without changes as a start image of a new animation sequence;

erasing the image of the previous animation sequence completely;

erasing the entire image of the previous animation sequence by moving an eraser tool on a screen;

erasing the image of the previous animation sequence up to a background image;

erasing a part of a screen field by the eraser tool horizontally;

erasing chosen elements of the image of the previous animation sequence;

restoring a final image of the animation sequence before the previous one;

rotating a visual field of the previous animation sequence;

opening a clean space for creating dynamic elements of the new animation sequence and leaving in sight a small part of the image of the previous frame;

minimizing into an icon the resulting image of the previous animation sequence and placing the icon on the screen field of the new animation sequence;

minimizing the resulting image of the previous animation sequence and leaving it on the screen field of the new animation sequence; and

extending the resulting image of the previous animation sequence and leaving the part visible within the frame edges on the screen field of the new animation sequence.

10. A system for creating an animation, the system comprising:

a touchscreen device having a processor, the processor configured to define animation sequences and an order of transition between the animation sequences;

the processor configured to prepare a composition of a visual content and an audio track of the animation and combining them into an animation file, preparing a composition of a visual content and an audio track of the animation and combining them into an animation file, by performing

(a) on the touchscreen device, drawing dynamic elements corresponding to the animation sequences, wherein the dynamic elements are added to a static background image of each animation sequence, and recording a visual representation of a process of the drawing;

(b) preparing a text to be narrated as the audio track and entering the text into the touchscreen device;

(c) specifying boundaries between fragments of the text by using a stylus or a finger to manually indicate the boundaries, wherein the fragments of the text correspond to neighboring animation sequences, and placing markers of a text markup at locations of the boundaries;

(d) recording the audio track, using a microphone of the touchscreen device, by narrating the text so that pauses in the audio track correspond to the markers and represent transitions between the neighboring animation sequences;

(e) splitting the audio track into portions associated with the animation sequences during the narration;

(f) creating a configuration file for each animation sequence by determining a playback duration of the animation sequence and by making a uniform time scaling of the visual representation of the animation, so that a duration of the visual representation is adjusted to match a duration of the audio track recorded through the microphone;

(g) setting an aspect ratio, resolution, color, transparency and FPS of the animation;

simultaneously playing back, on the touchscreen device, the audio track and a time scaled visual representation of the dynamic elements as they were being drawn, and recording a resulting mix of the audio track and the dynamic elements into a configuration file of each animation sequence;

playing back the configuration file of the animation sequence on the touchscreen device;

combining the configuration files of the animation sequences in the order of transition of the animation sequences into a configuration multimedia file of the animation; and

saving the configuration multimedia file of the animation.

11. The system of claim 10, wherein the processor is configured to:

automatically detect verbal text pronunciation;

automatically divide the recorded audio track into audio fragments and connect these audio fragments with the matching text sequences; and

automatically detect drawn graphic figures and select their smoothed out and regular-shaped equivalents from an image library.

12. The system of claim 10, wherein upon detecting a touch on different areas of the touch screen device, an on-screen mode of an operation selection panel is presented in a form of a bar divided into field-buttons with icons of operation modes, wherein when the field-buttons are pressed on, they expand lengthwise and sub-buttons available in these modes of operations of the touchscreen device appear;

and

wherein;

a mode of the operations selection panel is located in an upper section of the screen across the entire width and is provided with the fields-buttons “

Presentation management”

, “

Text editing”

, “

Sound editing”

, and “

Graphic editing”

;

a work field of an active operation mode is displayed under the selection panel on the touchscreen device;

at startup of the touchscreen device, if a processed prototype is not selected, the touchscreen device switches to the “

Presentation management”

mode and the icons available for review or editing of prototypes with captions are displayed in a work field; and

if the prototype is selected at launch, the touchscreen device switches to the “

Text editing”

mode of the selected prototype, thus the text of the prototype is displayed in the work field.

13. The system of claim 10, wherein the field-button “

Presentation management”

contains sub-buttons “

Create presentation”

, “

Open presentation for editing”

, “

Launch demonstration of presentation”

, “

Delete presentation”

, “

Save presentation as” and

wherein the icons of the presentation prototypes with captions are displayed in the work field.

14. The system of claim 10, wherein the field-button “

Text editing”

contains sub-buttons “

Text import”

, “

Text verbal dictation with recognition”

, “

Text input from on-screen keyboard”

, “

Insert the marker of the frame text edge.”

15. The system of claim 10, wherein the field-button “

Sound editing”

contains sub-buttons “

Import sound from file and markup sound into words”

, “

Record sound from microphone and markup sound into words”

, wherein the text of the prototype is split into segments corresponding to fragments of the animation and is displayed in the work field, so that a text part of the animation for which an audio track is already marked up into fragments is different from a part for which markup is not done yet.

16. The system of claim 10, wherein the field-button “

Graphic editing”

contains sub-buttons “

Turn on/turn off figures automatic recognition mode”

, “

Turn on/turn off text narrator”

, “

Move forward/back through frames”

, wherein an image sheet, tools and a palette for creating and editing graphic images of a current frame is displayed in the work field.

17. The system of claim 10, wherein upon pressing on the “

Sound editing”

button, the icons of graphic images corresponding to the frames are displayed in the segments of a work field along with the text.

18. The system of claim 17, wherein when the segments of the work field are pressed on, a pronounceable or a recognizable text is attached to the frame text displayed in a current segment of the work field.

19. The system of claim 17, wherein a sub-button “

Move forward/back through frames”

for the frame starting from the second is implemented in a form of two buttons “

Scroll forward/back” and

“

Select the look of transition between frames”

, wherein when the button “

Select the look of transition between frames”

is pressed, a list of the selection of the transition mode is displayed as a set of the buttons;

“

Save frame image as starting image of the next frame”

, “

Erase frame image entirely and instantly”

, “

Erase frame image entirely with delay”

, “

Erase frame image entirely with eraser tool”

, “

Erase frame image up to background”

, “

Erase part of frame image with eraser tool”

, “

Erase selected elements of frame image”

, “

Restore final image of the frame before the previous”

, “

Move (rotate) the image opening blank space to create figures of a new frame and leaving in sight a small part of the previous frame image”

, “

Minimize frame image placing the icon on the field of a new frame”

, “

Shrink frame image and leave it on the field of a new frame”

, “

Expand frame image and leave it on the field of a new frame”

.

20. A computer program product comprising a computer-readable non-transitory medium containing computer code for creating an animation file by performing the steps of:

(a) defining animation sequences of the animation file on a touchscreen device, each animation sequence including a static background image;

(b) drawing dynamic elements corresponding to the animation sequences, wherein the dynamic elements are added to the static background image of each animation sequence, and recording a process of the drawing that uses a finger or a stylus on the touchscreen device;

(c) entering a text to be narrated into the touchscreen device;

(d) specifying boundaries between fragments of the text by using a stylus or a finger to manually indicate the boundaries, wherein the fragments of the text correspond to neighboring animation sequences, and placing markers of a text markup at locations of the boundaries;

(e) recording an audio track, using a microphone of the touchscreen device, by narrating the text so that pauses in the audio track correspond to the markers and represent transitions between the neighboring animation sequences;

(f) splitting the audio track into portions associated with the corresponding animation sequences;

(g) creating a configuration file for each animation sequence by determining a playback duration of the animation sequence and by making a uniform time scaling of the visual representation of the each animation sequence so that a duration of the visual representation is adjusted to match a duration of the audio track;

(h) setting a transparency and FPS of the animation sequences;

(i) simultaneously playing back, on the touchscreen device, the audio track and a time scaled visual representation of the dynamic elements as they were being drawn, and recording a resulting mix of the audio track and the dynamic elements into a configuration file of each animation sequence;

(j) playing back the configuration files of the animation sequences on the touchscreen device;

(k) combining the configuration files of the animation sequences into a configuration multimedia file; and

(l) saving the configuration multimedia file.