Video stream customization using graphics

• US 10,311,619 B1
• Filed: 08/15/2018
• Issued: 06/04/2019
• Est. Priority Date: 05/16/2017
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Chen, Crystal Rhan-Tsor, Han, Lydia, Shorman, Aigerim, Christensen, Tanner Cotant, Blecher, Noah Bradley, Burdakov, Daniil, Narayan, Karthik, Singh, Harshdeep, Mathias, Pradeep George, Diwakar, Deepa, Bar Lev, Tamar, Lang, Christopher Jon, Mao, Shupin
• Primary Examiner(s): Harrison, Chante E
• Application Number:
  US16/103,927
• Time in Patent Office: 293 Days
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G09G 2340/12 Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
• CPC Class Codes: G09G 2340/125 wherein one of the images is motion video

Abstract

An online system customizes video conversations between users of the online system by providing graphics that are likely to interest the users. The online system may present composite views overlaying selected graphics on a video stream, which may be part of an augmented reality (AR) environment. The graphics include, for example, background graphics, masks, props, visual or particular effects, frames or borders, etc. The online system may use a machine learning model to predict whether a user is likely to select or interact with a particular graphic. The online system can also filter or rank graphics according to user affinities or user connections on the online system. Users may be encouraged to interact with the graphics that are customized to their specific interests, which can promote an engaging video conversation or AR experience.

Claims

1. A computer-implemented method comprising:

receiving, by a server of an online system, a video stream from a client device of a user of the online system;

determining user information of the user based on actions performed by the user stored in the online system;

identifying a plurality of graphics provided at least in part by a plurality of users of the online system;

determining a set of candidate graphics by filtering the plurality of graphics based at least in part on the user information of the user;

for each graphic of the set of candidate graphics, generating a score for the graphic indicating a likelihood of the user selecting the graphic;

providing, by the server to the client device, the set of candidate graphics for presentation to the user on the client device in an order based on the scores;

receiving, by the server from the client device, a selection for a candidate graphic of the set of candidate graphics; and

providing, by the server to the client device, a composite view generated by combining the selected candidate graphic with the video stream for presentation to the user on the client device.

2. The method of claim 1, wherein the scores are generated by a model trained based at least in part on feature vectors derived from actions performed by a population of users of the online system.

3. The method of claim 2, wherein the actions performed by the population of users of the online system include a plurality of selections of graphics of the plurality of graphics to be included in composite views presented to the population of users.

4. The method of claim 1, wherein each graphic of the set of candidate graphics is associated with one or more characteristics, and wherein filtering the plurality of graphics based at least in part on user information describing the user comprises:

determining interests of the user based on the user information; and

determining, for a graphic of the set of candidate graphics, that the corresponding one or more characteristics of the graphic matches at least one of the interests of the user.

5. The method of claim 1, wherein the plurality of graphics is provided further by at least a third party system, wherein the set of candidate graphics further includes a graphic provided by the third party system, and wherein the user information indicates that the user interacted with an object of the online system associated with the third party system.

6. The method of claim 1, wherein the composite view includes a visual representation of the user, and wherein the selected candidate graphic is combined with at least a portion of the visual representation of the user.

7. The method of claim 1, further comprising:

determining a background portion of the video stream, wherein the composite view is generated further by applying a visual effect of the selected candidate graphic to at least the background portion.

8. A computer-implemented method comprising:

receiving, by a server of an online system, an image from a client device of a user of the online system;

determining user information of the user based on actions performed by the user stored in the online system;

identifying a plurality of graphics provided at least in part by a plurality of users of the online system;

determining a set of candidate graphics by filtering the plurality of graphics based at least in part on the user information of the user;

for each graphic of the set of candidate graphics, generating a score for the graphic indicating a likelihood of the user selecting the graphic;

providing, by the server to the client device, the set of candidate graphics for presentation to the user on the client device in an order based on the scores;

receiving, by the server from the client device, a selection for a candidate graphic of the set of candidate graphics; and

providing, by the server to the client device, a composite view generated by combining the selected candidate graphic with the image for presentation to the user on the client device.

9. The method of claim 8, wherein the scores are generated by a model trained based at least in part on feature vectors derived from actions performed by a population of users of the online system.

10. The method of claim 9, wherein the actions performed by the population of users of the online system include a plurality of selections of graphics of the plurality of graphics to be included in composite views presented to the population of users.

11. The method of claim 8, wherein each graphic of the set of candidate graphics is associated with one or more characteristics, and wherein filtering the plurality of graphics based at least in part on user information describing the user comprises:

determining interests of the user based on the user information; and

determining, for a graphic of the set of candidate graphics, that the corresponding one or more characteristics of the graphic matches at least one of the interests of the user.

12. The method of claim 8, wherein the plurality of graphics is provided further by at least a third party system, wherein the set of candidate graphics further includes a graphic provided by the third party system, and wherein the user information indicates that the user interacted with an object of the online system associated with the third party system.

13. The method of claim 8, wherein the composite view includes a visual representation of the user, and wherein the selected candidate graphic is combined with at least a portion of the visual representation of the user.

14. A non-transitory computer-readable storage medium storing instructions that when executed by a processor cause the processor to perform steps including:

receiving, by a server of an online system, a video stream from a client device of a user of an online system;

determining user information of the user based on actions performed by the user stored in the online system;

identifying a plurality of graphics provided at least in part by a plurality of users of the online system;

determining a set of candidate graphics by filtering the plurality of graphics based at least in part on the user information of the user;

for each graphic of the set of candidate graphics, generating a score for the graphic indicating a likelihood of the user selecting the graphic;

providing, by the server to the client device, the set of candidate graphics for presentation to the user on the client device in an order based on the scores;

receiving, by the server from the client device, a selection from the user for a candidate graphic of the set of candidate graphics; and

providing, by the server to the client device, a composite view generated by combining the selected candidate graphic with the video stream for presentation to the user on the client device.

15. The non-transitory computer readable storage medium of claim 14, wherein the scores are generated by a model trained based at least in part on feature vectors derived from actions performed by a population of users of the online system.

16. The non-transitory computer readable storage medium of claim 15, wherein the actions performed by the population of users of the online system include a plurality of selections of graphics of the plurality of graphics to be included in composite views presented to the population of users.

17. The non-transitory computer readable storage medium of claim 14, wherein each graphic of the set of candidate graphics is associated with one or more characteristics, and wherein filtering the plurality of graphics based at least in part on user information describing the user comprises:

determining interests of the user based on the user information; and

determining, for a graphic of the set of candidate graphics, that the corresponding one or more characteristics of the graphic matches at least one of the interests of the user.

18. The non-transitory computer readable storage medium of claim 14, wherein the plurality of graphics is provided further by at least a third party system, wherein the set of candidate graphics further includes a graphic provided by the third party system, and wherein the user information indicates that the user interacted with an object of the online system associated with the third party system.

19. The non-transitory computer readable storage medium of claim 14, wherein the composite view includes a visual representation of the user, and wherein the selected candidate graphic is combined with at least a portion of the visual representation of the user.

20. The non-transitory computer readable storage medium of claim 14, having further instructions that when executed by the processor cause the processor to:

determining a background portion of the video stream, wherein the composite view is generated further by applying a visual effect of the selected candidate graphic to at least the background portion.