Interactive cinemagrams

• US 10,586,367 B2
• Filed: 07/14/2017
• Issued: 03/10/2020
• Est. Priority Date: 01/04/2017
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Ravindran, Sourabh, Yoo, Youngjun
• Primary Examiner(s): Hsu, Amy R
• Application Number:
  US15/650,531
• Publication Number:
  US 20190019320A1
• Time in Patent Office: 970 Days
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06T 13/20 3D [Three Dimensional] animation
• CPC Class Codes: G06T 13/80 2D [Two Dimensional] animation, e.g. using sprites
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20116 Active contour; Active surface; Snakes
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/149 involving deformable models, e.g. active contour models
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: H04N 1/387 Composing, repositioning or otherwise geometrically modifying originals

Abstract

A method, apparatus, and computer readable medium for interactive cinemagrams. The method includes displaying a still frame of a cinemagram on a display of an electronic device, the cinemagram having an animated portion. The method also includes after the displaying, identifying occurrence of a triggering event based on an input from one or more sensors of the electronic device. Additionally, the method includes initiating animation of the animated portion of the cinemagram in response to identifying the occurrence of the triggering event. The method may also include generating the image as a cinemagram by identifying a reference frame from a plurality of frames and an object in the reference frame, segmenting the object from the reference frame, tracking the object across multiple of the frames, determining whether a portion of the reference frame lacks pixel information during motion of the object, and identifying pixel information to add to the portion.

Claims

1. A method for interactive cinemagrams, the method comprising:

identifying a reference frame from a plurality of frames and an object in the reference frame;

generating a motion component of a cinemagram by tracking the object across multiple of the frames;

determining whether a portion of the reference frame lacks pixel information during motion of the object for the motion component of the cinemagram;

identifying pixel information to add to the portion of the reference frame during motion of the object;

displaying a still image frame of the cinemagram on a display of an electronic device;

identifying a triggering event for initiating animation of the motion component of the cinemagram from metadata associated with the cinemagram;

after displaying the still image frame, identifying occurrence of the triggering event based on an input from one or more sensors of the electronic device; and

initiating animation of the motion component of the cinemagram while displaying a portion of the still image frame in a remaining portion of the cinemagram in response to identifying the occurrence of the triggering event.

2. The method of claim 1, further comprising:

identifying audio for the cinemagram based on the metadata associated with the cinemagram; and

initiating play of the audio for the cinemagram in response to identifying the occurrence of the triggering event.

3. The method of claim 1, wherein:

the cinemagram further includes a static component, and

the method further comprises generating the static component by segmenting the object from the reference frame.

4. The method of claim 1, wherein identifying the pixel information to add to the portion of the reference frame during motion of the object comprises:

identifying the portion of the reference frame in multiple of the frames other than the reference frame; and

identifying the pixel information for the portion of the reference frame based on pixel information for the portion in the other frames.

5. The method of claim 1, wherein identifying the pixel information to add to the portion of the reference frame during motion of the object comprises filling the pixel information into the portion of the reference frame using nearby pixel interpolation.

6. The method of claim 1, further comprising:

before generating the cinemagram, analyzing movement of multiple objects across the plurality of frames; and

generating a recommendation of one or more objects to be animated for the motion component of the cinemagram.

7. The method of claim 1, wherein at least one of:

the one or more sensors include a microphone and the triggering event is based on sound sensed by the microphone; and

the one or more sensors include a touch sensor and the triggering event is based on touch sensed by the touch sensor.

8. The method of claim 1, wherein the one or more sensors include a motion sensor and the triggering event is based on sensed movement of the electronic device.

9. An electronic device comprising:

a display configured to display a still image frame of a cinemagram;

one or more sensors; and

a processor operably connected to the display and the one or more sensors, the processor configured to;

identify a reference frame from a plurality of frames and an object in the reference frame;

generate a motion component of the cinemagram by tracking the object across multiple of the frames;

determine whether a portion of the reference frame lacks pixel information during motion of the object for the motion component of the cinemagram;

identify pixel information to add to the portion of the reference frame during motion of the object;

identify a triggering event for initiating animation of the motion component of the cinemagram from metadata associated with the cinemagram;

identify, after display of the still image frame, occurrence of the triggering event based on an input from the one or more sensors; and

initiate, in response to identification of the occurrence of the triggering event, animation of the motion component of the cinemagram by the display while displaying a portion of the still image frame in a remaining portion of the cinemagram.

10. The electronic device of claim 9, wherein the processor is further configured to:

identify audio for the cinemagram based on the metadata associated with the cinemagram; and

initiate play of the audio for the cinemagram in response to identifying the occurrence of the triggering event.

11. The electronic device of claim 9, wherein:

the cinemagram further includes a static component, and

the processor is further configured to generate the static component by segmenting the object from the reference frame.

12. The electronic device of claim 9, wherein to identify the pixel information to add to the portion of the reference frame during motion of the object, the processor is further configured to:

identify the portion of the reference frame in multiple of the frames other than the reference frame; and

identify the pixel information for the portion of the reference frame based on pixel information for the portion in the other frames.

13. The electronic device of claim 9, wherein to identify the pixel information to add to the portion of the reference frame during motion of the object, the processor is further configured to fill the pixel information into the portion of the reference frame using nearby pixel interpolation.

14. The electronic device of claim 9, wherein the processor is further configured to:

before generation of the cinemagram, analyze movement of multiple objects across the plurality of frames; and

generate a recommendation of one or more objects to be animated for the motion component of the cinemagram.

15. The electronic device of claim 9, wherein at least one of:

the one or more sensors include a microphone and the triggering event is based on sound sensed by the microphone; and

the one or more sensors include a touch sensor and the triggering event is based on touch sensed by the touch sensor.

16. The electronic device of claim 9, wherein the one or more sensors include a motion sensor and the triggering event is based on sensed movement of the electronic device.

17. A non-transitory, computer-readable medium comprising computer code that, when executed by at least one processor of an electronic device, causes the electronic device to:

identify a reference frame from a plurality of frames and an object in the reference frame;

generate a motion component of a cinemagram by tracking the object across multiple of the frames;

determine whether a portion of the reference frame lacks pixel information during motion of the object for the motion component of the cinemagram;

identify pixel information to add to the portion of the reference frame during motion of the object;

cause a display of the electronic device to display a still image frame of the cinemagram;

identify a triggering event for initiating animation of the motion component of the cinemagram from metadata associated with the cinemagram;

identify, after display of the still image frame, occurrence of the triggering event based on an input from one or more sensors of the electronic device; and

initiate, in response to identification of the occurrence of the triggering event, animation of the motion component of the cinemagram by the display while displaying a portion of the still image frame in a remaining portion of the cinemagram.

18. The non-transitory, computer-readable medium of claim 17, further comprising computer code that, when executed by the at least one processor, causes the electronic device to:

identify audio for the cinemagram based on the metadata associated with the cinemagram; and

initiate play of the audio for the cinemagram in response to identifying the occurrence of the triggering event.

19. The non-transitory, computer-readable medium of claim 17, wherein:

the cinemagram further includes a static component, and

the non-transitory, computer-readable medium further comprises computer code that, when executed by the at least one processor, causes the electronic device to generate the static component by segmenting the object from the reference frame.

20. The non-transitory, computer-readable medium of claim 17, wherein the computer code that, when executed by the at least one processor, causes the electronic device to identify the pixel information to add to the portion of the reference frame during motion of the object, comprises computer code that, when executed by the at least one processor, causes the electronic device to:

identify the portion of the reference frame in multiple of the frames other than the reference frame; and

identify the pixel information for the portion of the reference frame based on pixel information for the portion in the other frames.

21. The non-transitory, computer-readable medium of claim 17, wherein the computer code that, when executed by the at least one processor, causes the electronic device to identify the pixel information to add to the portion of the reference frame during motion of the object, comprises computer code that, when executed by the at least one processor, causes the electronic device to fill the pixel information into the portion of the reference frame using nearby pixel interpolation.

22. The non-transitory, computer-readable medium of claim 17, further comprising computer code that, when executed by the at least one processor, causes the electronic device to:

before generation of the cinemagram, analyze movement of multiple objects across the plurality of frames; and

generate a recommendation of one or more objects to be animated for the motion component of the cinemagram.

23. The non-transitory, computer-readable medium of claim 17, wherein at least one of:

the one or more sensors include a microphone and the triggering event is based on sound sensed by the microphone; and

the one or more sensors include a touch sensor and the triggering event is based on touch sensed by the touch sensor.

24. The non-transitory, computer-readable medium of claim 17, wherein the one or more sensors include a motion sensor and the triggering event is based on sensed movement of the electronic device.