Automated effects generation for animated content

• US 10,474,877 B2
• Filed: 01/09/2017
• Issued: 11/12/2019
• Est. Priority Date: 09/22/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Huang, Peter Chi-Hao, Hachenburg, Erick, Hart, Bryan, Zulkower, Valentin
• Primary Examiner(s): Lewis, Cheryl
• Application Number:
  US15/402,189
• Publication Number:
  US 20170193280A1
• Time in Patent Office: 1,037 Days
• CPC Class Codes: G06F 16/434 using image data, e.g. images, photos, pictures taken by a user
• CPC Class Codes: G06F 16/435 Filtering based on additional data, e.g. user or group profiles
• CPC Class Codes: G06F 16/44 Browsing; Visualisation therefor
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/784 the detected or recognised objects being people
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 13/80 2D [Two Dimensional] animation, e.g. using sprites
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2210/12 Bounding box
• CPC Class Codes: G06T 7/285 using a sequence of stereo image pairs
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships

Abstract

Various embodiments relate generally to a system, a device and a method for automating effects generation for media content. A media content item is received in a media content management system. Data representing one or more static regions associated with the media content item are determined. A canvas mapping object comprising the data representing the one or more static regions of the media content item is generated based on the frames of the media content item. The media content item is stored in a database in the media content management system in association with the canvas mapping object, and the media content item and the canvas mapping object are provided in response to a search request associated with the media content item.

Claims

1. A method comprising:

receiving a media content item in an media content management system, wherein the media content item comprises a video file in a graphics interchange format (GIF);

determining one or more image detection data items associated with the media content item, wherein determining one or more image detection data items associated with the media content item further comprises;

determining data representing a detected facial feature in the video file using a facial recognition module; and

determining the data representing the detected facial feature as an image detection data item of the one or more image detection data items;

generating, by a processor, a canvas mapping object comprising at least one of the one or more image detection data items;

determining data representing a canvas region based on the at least one of the one or more image detection data items; and

storing the data representing the canvas region in the canvas mapping object in association with the media content item in a database in the media content management system.

2. The method of claim 1, wherein determining one or more image detection data items associated with the media content item further comprises:

determining data representing a detected eye tracking feature in the video file using an eye tracking module; and

determining the data representing the detected eye tracking feature as an image detection data item of the one or more image detection data items.

3. The method of claim 1, and wherein determining one or more image detection data items associated with the media content item further comprises:

determining data representing a detected static space in the video file using a static space detection module; and

determining the data representing the detected static space as an image detection data item of the one or more image detection data items.

4. The method of claim 3, wherein the data representing static space detection in the video file comprises a bounded region of the video file, the bounded region comprising the data representing the canvas region.

5. The method of claim 3, wherein the data representing static space detection in the video file comprises a plurality of bounded regions in the video file, and wherein determining data representing a canvas region further comprises:

selecting a bounded region of the plurality of bounded regions based on a predetermined pixel region threshold associated with one or more effects generation criteria.

6. The method of claim 1, wherein generating a canvas mapping object comprising at least one of the one or more image detection data items further comprises:

determining data representing a detected facial feature in the media content item;

determining one or more bounded regions adjacent to the detected facial feature based on the one or more image detection data items; and

storing the one or more bounded regions in the canvas mapping object.

7. The method of claim 1, wherein determining data representing a canvas region based on the at least one of the one or more image detection data items further comprises:

determining one or more effects generation criteria associated with automated effects generation; and

selecting an image detection data item from the one or more image detection data items based on the one or more effects generation criteria as the canvas region.

8. The method of claim 1, further comprising:

receiving a request associated with the media content item from a user device; and

providing the media content item and the canvas mapping object to the user device responsive to the request.

9. The method of claim 8, wherein the request comprises a search request from a dynamic keyboard interface on the user device.

10. The method of claim 9, wherein providing the media content item and the canvas mapping object further comprises:

providing a plurality of pixel location coordinates defining the canvas region within the media content item; and

enabling an effects generation module on the dynamic keyboard interface to modify the media content item at the plurality of pixel location coordinates.

11. A method comprising:

receiving, in an media content management system, a media content item comprising a plurality of frames;

determining data representing one or more static regions associated with the plurality of frames of the media content item, wherein the data representing the one or more static regions associated with the plurality of frames is determined based on a color difference value determined as a percentage change in color value among a plurality of pixels in the plurality of frames;

generating, by a processor, a canvas mapping object comprising the data representing the one or more static regions associated with the plurality of frames of the media content item;

storing the canvas mapping object in association with the media content item in a database in the media content management system; and

providing the media content item and the canvas mapping object responsive to a search request associated with the media content item, the search request querying the database in the media content management system.

12. The method of claim 11, wherein the canvas mapping object is stored as a metadata attribute of the media content item.

13. The method of claim 11, wherein the data representing the one or more static regions associated with the plurality of frames is determined based on a movement analysis of an object across the plurality of frames.

14. A system comprising:

a server configured to receive a media content item comprising a plurality of frames, receive a search query, retrieve at least two media content items in the database, generate at least two renderings of the at least two media content items in animation in response to the received search query, provide to display concurrently the at least two renderings of the at least two media content items in animation, and to provide associated canvas mapping objects in relation with the at least two media content items;

a preprocessing module configured to determine data representing one or more static regions associated with the plurality of frames of the media content item, generate a canvas mapping object comprising the data representing the one or more static regions associated with the plurality of frames, and preprocessing module further configured to determine one or more image detection data items based on the plurality of frames of the media content item;

a processor configured to determine data representing a canvas region from the data representing the one or more static regions based on the one or more image detection data items, to configure the canvas mapping object to identify a plurality of location coordinates associated with the data representing the canvas region, the plurality of location coordinates defined in relation to the plurality of frames of the media content item, and the processor further configured to define the plurality of location coordinates based on at least one of the one or more image detection data items; and

a database configured to store, by the processor, the media content item in association with the canvas mapping object comprising the data representing the canvas region in association with the plurality of frames.

15. The system of claim 14, wherein the one or more image detection data items associated with the media content item further comprises at least one of a facial detection data item, an eye tracking detection data item, a movement analysis data item, and a static space detection data item.

16. The system of claim 14, wherein the processor is further configured to generate the canvas mapping object using a third party application communicatively coupled to the server.

17. The system of claim 14, wherein preprocessing module is further configured to determine the data representing the one or more static regions based on a color difference value defined as a percentage change in color value among a plurality of color values associated with a plurality of pixels in each frame of the plurality of frames of the media content item.

18. A method comprising:

receiving a media content item in an media content management system, wherein the media content item comprises a video file in a graphics interchange format (GIF);

determining one or more image detection data items associated with the media content item, and wherein determining one or more image detection data items associated with the media content item further comprises;

determining data representing a detected static space in the video file using a static space detection module; and

determining the data representing the detected static space as an image detection data item of the one or more image detection data items;

generating, by a processor, a canvas mapping object comprising at least one of the one or more image detection data items;

determining data representing a canvas region based on the at least one of the one or more image detection data items; and

storing the data representing the canvas region in the canvas mapping object in association with the media content item in a database in the media content management system.

19. A system comprising:

a server configured to receive a media content item comprising a plurality of frames;

a preprocessing module configured to determine data representing one or more static regions associated with the plurality of frames of the media content item, generate a canvas mapping object comprising the data representing the one or more static regions associated with the plurality of frames, and preprocessing module further configured to determine one or more image detection data items based on the plurality of frames of the media content item;

a processor configured to determine data representing a canvas region from the data representing the one or more static regions based on the one or more image detection data items, to configure the canvas mapping object to identify a plurality of location coordinates associated with the data representing the canvas region, the plurality of location coordinates defined in relation to the plurality of frames of the media content item, and the processor further configured to define the plurality of location coordinates based on at least one of the one or more image detection data items, wherein the one or more image detection data items associated with the media content item further comprises at least one of a facial detection data item, an eye tracking detection data item, a movement analysis data item, and a static space detection data item; and

a database configured to store, by the processor, the media content item in association with the canvas mapping object comprising the data representing the canvas region in association with the plurality of frames.

20. A system comprising:

a server configured to receive a media content item comprising a plurality of frames;

a preprocessing module configured to determine data representing one or more static regions associated with the plurality of frames of the media content item, generate a canvas mapping object comprising the data representing the one or more static regions associated with the plurality of frames, and preprocessing module further configured to determine one or more image detection data items based on the plurality of frames of the media content item;

a processor configured to determine data representing a canvas region from the data representing the one or more static regions based on the one or more image detection data items, generate a canvas mapping object using a third party application communicatively coupled to the server, and configure the canvas mapping object to identify a plurality of location coordinates associated with the data representing the canvas region, the plurality of location coordinates defined in relation to the plurality of frames of the media content item, and the processor further configured to define the plurality of location coordinates based on at least one of the one or more image detection data items; and

a database configured to store, by the processor, the media content item in association with the canvas mapping object comprising the data representing the canvas region in association with the plurality of frames.