Architecture for augmenting video data obtained by a client device with one or more effects during rendering

• US 10,291,678 B2
• Filed: 10/01/2016
• Issued: 05/14/2019
• Est. Priority Date: 10/01/2016
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Pique Corchs, Hermes Germi, Pugin, Kirill A., Racasanu, Razvan Gabriel, Miller, Colin Todd, Srinivasan, Ragavan, Bar, Tomer, Redd, Bryce David
• Primary Examiner(s): Amin, Jwalant
• Application Number:
  US15/283,292
• Publication Number:
  US 20180097866A1
• Time in Patent Office: 955 Days
• Field of Search: None
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04L 65/762 at the source reformatting of additional data in video distribution servers H04N21/2355
• CPC Class Codes: H04L 65/764 at the destination reformatting of additional data in video clients H04N21/4355
• CPC Class Codes: H04L 65/765 intermediate
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/47205 for manipulating displayed content, e.g. interacting with MPEG-4 objects, editing locally
• CPC Class Codes: H04N 21/6582 Data stored in the client, e.g. viewing habits, hardware capabilities, credit card number arrangements where receivers interact with the broadcast H04H20/38
• CPC Class Codes: H04N 21/8133 specifically related to the content, e.g. biography of the actors in a movie, detailed information about an article seen in a video program

Abstract

A video effects application executes on a client device having an image capture device and receives video data captured by the image capture device. The video effects application extracts information from the captured video data and stores the extracted information as metadata associated with the captured video data. For example, the video effects application identifies objects in the captured video data or identifies optical flow of the captured video data and stores the identified objects or identified optical flow as metadata associated with the captured video data. The video effects application stores information describing modifications to the captured video data in association with the captured video data. When the captured video data is presented, the captured video data, associated metadata, and information describing the modifications is communicated to a renderer, which uses the metadata to perform the identified modifications to the captured video data when presenting the captured video data.

Claims

1. A method comprising:

obtaining video data at a client device;

extracting information from the video data by the client device applying one or more processes to the video data when the video data is obtained at the client device;

storing the extracted information as metadata associated with the video data;

receiving information identifying one or more modifications to the video data;

storing the received information identifying modifications to the video data in association with the video data; and

providing the video data, the metadata, and the identified modifications to a plurality of client devices for display, each of the plurality of client devices configured to modify the video data differently during presentation of the video data based on differences in data captured by one or more components of each of the plurality of client devices, modification of the video data by a client device of the plurality of client devices performed when the video data is rendered at the client device of the plurality of client devices for presentation via a display device of the client device of the plurality of client devices.

2. The method of claim 1, further comprising:

providing the video data and the metadata associated with the video data to an online system.

3. The method of claim 2, wherein providing the video data and the metadata associated with the video data to the online system comprises:

communicating an identifier of a user of the online system to the online system in association with the video data and the metadata associated with the video data.

4. The method of claim 2, further comprising:

providing the received information identifying modifications to the video data to the online system in association with the video data and the metadata associated with the video data.

5. The method of claim 1, the metadata associated with the video data identifies one or more portions of the video data including one or more objects.

6. The method of claim 5, wherein modifying the video data comprises:

modifying one or more portions of the video data based on an object included in the one or more portions of the video data.

7. The method of claim 1, wherein a modification to the video data identifies a portion of the video data and, modifying the video data comprises:

modifying the portion of the video data based on one or more inputs received from the user of the client device during times when the portion of the video data is rendered by the client device.

8. The method of claim 1, wherein a modification to the video data identifies a portion of the video data, and modifying the video data comprises:

modifying the portion of the video data based on information captured by one or more components of the client device during times when the portion of the video data is rendered by the client device.

9. The method of claim 8, wherein a component of the client device is selected from a group consisting of:

an accelerometer, a position sensor, and any combination thereof.

10. The method of claim 1, wherein modifying the video data by performing the modifications identified by the received information, one or more of the modifications performed based on the metadata associated with the video data comprises:

overlaying one or more images on a portion of the video data identified as including an object by the metadata associated with the video data and modifying audio data of the video data during the portion of the video data.

11. The method of claim 1, wherein modifying the video data by performing the modifications identified by the received information comprises:

performing a plurality of modifications to a portion of the video data based on one or more characteristics of the portion of the video data identified by the metadata associated with the video data.

12. The method of claim 1, wherein obtaining video data at the client device comprises:

capturing the video data via an image capture device included in the client device.

13. The method of claim 1, wherein obtaining video data at the client device comprises:

receiving the video data from another client device.

14. The method of claim 1, wherein a process applied to the video data when the video data is obtained at the client device identifies one or more objects included in the video data.

15. The method of claim 14, wherein an object included in the video data comprises a face.

16. A method comprising:

receiving, at an online system, video data from a client device associated with a user of the online system;

receiving, at the online system, metadata associated with the video data by the client device, the metadata including information extracted from the video data by the client device when the client device obtained the video data;

storing the video data in association with the metadata at the online system; and

providing the video data, the metadata, and the identified modifications to a plurality of client devices for display, each of the plurality of client devices configured to modify the video data differently during presentation of the video data based on differences in data captured by one or more components of each of the plurality of client devices, modification of the video data by a client device of the plurality of client devices performed when the video data is rendered at the client device of the plurality of client devices for presentation via a display device of the client device of the plurality of client devices.

17. The method of claim 16, wherein receiving, at the online system, metadata associated with the video data by the client device comprises:

receiving metadata associated with the video data by the client device and information describing one or more modifications to the video data received by the client device.

18. The method of claim 17, wherein providing the video data and the metadata associated with the video data to the additional client device comprises:

providing the video data, the metadata associated with the video data, and the information describing one or more modifications to the video data to the additional client device, the additional client device configured to perform the one or more modifications to the video data when rendering the video data.

19. The method of claim 17, wherein the information describing one or more modifications to the video data identifies a plurality of modifications to a portion of the video data based on one or more characteristics of the portion of the video data identified by the metadata associated with the video data.

20. The method of claim 17, wherein the information describing one or more modifications to the video data identifies a modification to a portion of the video data based on one or more inputs received by the additional client device during times corresponding to the portion of the video data.

21. A computer program product comprising a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by a processor, cause the processor to:

obtain video data and metadata associated with the video data at a client device, the metadata associated with the video data by an additional client device;

receive information identifying one or more modifications to the video data at the client device;

modify the video data by performing the modifications identified by the received information during rendering of the video data for display, one or more of the modifications performed using the metadata associated with the video data, one or more of the modifications based on data captured by one or more components of the client device; and

present the modified video data via a display device of the client device.

22. The computer program product of claim 21, wherein a modification to the video data identifies a portion of the video data, and modifying the video data comprises:

modify the portion of the video data based on information obtained by the client device during times when the portion of the video data is rendered by the client device.

23. The computer program product of claim 22, wherein the information obtained by the client device during times when the portion of the video data is rendered by the client device comprises one or more inputs received from a user of the client device during times when the portion of the video data is rendered by the client device.

24. The computer program product of claim 22, wherein the information obtained by the client device during times when the portion of the video data is rendered by the client device comprises information captured by one or more components of the client device during times when the portion of the video data is rendered by the client device.

25. The computer program product of claim 21, wherein obtaining video data and metadata associated with the video data at a client device, the metadata associated with the video data by the additional client device comprises:

receive the video data and the metadata associated with the video data from an online system.