Video stream customization using graphics

• US 10,529,109 B1
• Filed: 04/18/2019
• Issued: 01/07/2020
• Est. Priority Date: 05/16/2017
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Chen, Crystal Rhan-Tsor, Han, Lydia, Shorman, Aigerim, Christensen, Tanner Cotant, Blecher, Noah Bradley, Burdakov, Daniil, Narayan, Karthik, Singh, Harshdeep, Mathias, Pradeep George, Diwakar, Deepa, Bar Lev, Tamar, Lang, Christopher Jon, Mao, Shupin
• Primary Examiner(s): Harrison, Chante E
• Application Number:
  US16/388,549
• Time in Patent Office: 264 Days
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G09G 2340/12 Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
• CPC Class Codes: G09G 2340/125 wherein one of the images is motion video

Abstract

An online system customizes video conversations between users of the online system by providing graphics that are likely to interest the users. The online system may present composite views overlaying selected graphics on a video stream, which may be part of an augmented reality (AR) environment. The graphics include, for example, background graphics, masks, props, visual or particular effects, frames or borders, etc. The online system may use a machine learning model to predict whether a user is likely to select or interact with a particular graphic. The online system can also filter or rank graphics according to user affinities or user connections on the online system. Users may be encouraged to interact with the graphics that are customized to their specific interests, which can promote an engaging video conversation or AR experience.

Claims

1. A computer-implemented method comprising:

receiving a video stream from a client device of a user of an online system;

determining user information of the user based on actions performed by the user through the online system;

identifying a plurality of graphics provided at least in part by a plurality of users of the online system;

determining characteristics of the plurality of graphics using image processing algorithms;

associating the determined characteristics with the plurality of graphics;

determining a set of candidate graphics by filtering the plurality of graphics based at least in part on the user information of the user, the filtering including;

determining interests of the user based on the user information;

applying a machine learning model to determine, for a graphic of the set of candidate graphics, whether the user is likely to select the graphic;

including the selected graphic of the plurality of graphics within the candidate graphics responsive to the determining that the user is likely to select the graphic;

providing the set of candidate graphics for presentation to the user on the client device;

receiving a selection for a candidate graphic of the set of candidate graphics; and

providing a composite view generated by combining the selected candidate graphic with the video stream into a composite view for presentation to the user on the client device.

2. The computer-implemented method of claim 1, wherein the plurality of graphics is provided further by at least a third party system, wherein the set of candidate graphics further includes a graphic provided by the third party system, and wherein the user information indicates that the user interacted with an object of the online system associated with the third party system.

3. The computer-implemented method of claim 1, wherein the composite view includes a visual representation of the user, and wherein the selected candidate graphic is overlaid on at least a portion of the visual representation of the user.

4. The computer-implemented method of claim 1, wherein the plurality of graphics comprises one or more background graphics and one or more visual effects,

and wherein the composite view is generated further by applying a visual effect of the selected candidate graphic to at least a background portion of the video stream.

5. The computer-implemented method of claim 1, wherein determining the set of candidate graphics by filtering further comprises:

generating feature vectors based at least in part on the determined interests of the user and on the associated characteristics of the plurality of graphics;

wherein the machine learning model is trained using support vector machines and a training label based on a known outcome to the generated feature vectors.

6. A computer-implemented method comprising:

receiving an image from a client device of a user of an online system;

determining user information of the user based on actions performed by the user through the online system;

identifying a plurality of graphics provided at least in part by a plurality of users of the online system;

determining characteristics of the plurality of graphics using image processing algorithms;

associating the determined characteristics with the plurality of graphics;

determining a set of candidate graphics by filtering the plurality of graphics based at least in part on the user information of the user, the filtering including;

determining interests of the user based on the user information;

applying a machine learning model to determine, for a graphic of the set of candidate graphics, whether the user is likely to select the graphic;

including the selected graphic of the plurality of graphics within the candidate graphics responsive to the determining that the user is likely to select the graphic;

providing the set of candidate graphics for presentation to the user on the client device;

receiving a selection for a candidate graphic of the set of candidate graphics; and

providing a composite view generated by combining the selected candidate graphic with the image into a composite view for presentation to the user on the client device.

7. The computer-implemented method of claim 6, wherein the plurality of graphics is provided further by at least a third party system, wherein the set of candidate graphics further includes a graphic provided by the third party system, and wherein the user information indicates that the user interacted with an object of the online system associated with the third party system.

8. The computer-implemented method of claim 6, wherein the composite view includes a visual representation of the user, and wherein the selected candidate graphic is overlaid on at least a portion of the visual representation of the user.

9. The computer-implemented method of claim 6, wherein the plurality of graphics comprises one or more background graphics and one or more visual effects,

and wherein the composite view is generated further by applying a visual effect of the selected candidate graphic to at least a background portion of the image.

10. The computer-implemented method of claim 6, wherein determining the set of candidate graphics by filtering further comprises:

generating feature vectors based at least in part on the determined interests of the user and on the associated characteristics of the plurality of graphics;

wherein the machine learning model is trained using support vector machines and a training label based on a known outcome to the generated feature vectors.

11. A non-transitory computer-readable storage medium storing instructions that when executed by a processor cause the processor to perform steps including:

receiving a video stream from a client device of a user of an online system;

determining user information of the user based on actions performed by the user through the online system;

identifying a plurality of graphics provided at least in part by a plurality of users of the online system;

determining characteristics of the plurality of graphics using image processing algorithms;

associating the determined characteristics with the plurality of graphics;

determining a set of candidate graphics by filtering the plurality of graphics based at least in part on the user information of the user, the filtering including;

determining interests of the user based on the user information;

applying a machine learning model to determine, for a graphic of the set of candidate graphics, whether the user is likely to select the graphic;

including the selected graphic of the plurality of graphics within the candidate graphics responsive to the determining that the user is likely to select the graphic;

providing the set of candidate graphics for presentation to the user on the client device;

receiving a selection for a candidate graphic of the set of candidate graphics; and

providing a composite view generated by combining the selected candidate graphic with the video stream into a composite view for presentation to the user on the client device.

12. The non-transitory computer-readable storage medium of claim 11, wherein the plurality of graphics is provided further by at least a third party system, wherein the set of candidate graphics further includes a graphic provided by the third party system, and wherein the user information indicates that the user interacted with an object of the online system associated with the third party system.

13. The non-transitory computer-readable storage medium of claim 11, wherein the composite view includes a visual representation of the user, and wherein the selected candidate graphic is overlaid on at least a portion of the visual representation of the user.

14. The non-transitory computer-readable storage medium of claim 11, wherein the plurality of graphics comprises one or more background graphics and one or more visual effects,

and wherein the composite view is generated further by applying a visual effect of the selected candidate graphic to at least a background portion of the video stream.

15. The non-transitory computer-readable storage medium of claim 11, wherein determining the set of candidate graphics by filtering further comprises:

generating feature vectors based at least in part on the determined interests of the user and on the associated characteristics of the plurality of graphics;

wherein the machine learning model is trained using support vector machines and a training label based on a known outcome to the generated feature vectors.