Method and computer readable medium for presentation of content items synchronized with media display

• US 10,643,264 B2
• Filed: 07/25/2016
• Issued: 05/05/2020
• Est. Priority Date: 07/25/2016
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Jones, Christopher William, D'Arcy, Mark Gerard, Lee, Ji Byol
• Primary Examiner(s): Smith, Jeffrey A.
• Assistant Examiner(s): Gibson-Wynn, Kennedy
• Application Number:
  US15/218,319
• Publication Number:
  US 20180025405A1
• Time in Patent Office: 1,380 Days
• Field of Search: 705 261- 272
• CPC Class Codes: G06Q 30/0631 Item recommendations
• CPC Class Codes: G06Q 30/0633 Lists, e.g. purchase orders, compilation or processing
• CPC Class Codes: G06Q 30/0643 Graphical representation of items or shoppers
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 2201/09 Recognition of logos
• CPC Class Codes: H04N 21/4532 involving end-user characteristics, e.g. viewer profile, preferences monitoring of user activities for profile generation for accessing a video database G06F16/739; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18

Abstract

An online system displays relevant products along with a video watched by a user in a synchronized manner, which allows the user to explore relevant content (e.g., products recommended to the user) while the user is watching the video. The online system receives a video including a sequence of video frames to be displayed to the user, and the video may be associated with a specific theme, e.g., a soccer game, and one or more video frames may include points of interest (referred as “Q points”). The online system detects or receives Q points associated with corresponding video frames, where a Q point in a video frame identifies an object of interest in the video frame. The online system also correlates user profile information of the user with the detected points to select relevant products tailored to the user.

Claims

1. A computer-implemented method, the method comprising:

receiving a video for display to a user, the video comprising a plurality of video frames;

for each video frame;

analyzing visual features of the video frame; and

detecting one or more points of interest in the video frame based on the analysis of the visual features of the video frame, a point of interest in the video frame being associated with an object in the video frame, and the object in the video frame being related to one or more different products;

for each of the identified point of interests in the plurality of video frames, selecting a product of the one or more different products related to the object associated with the point of interest, by;

analyzing user profile information associated with the user to derive a user preference; and

selecting the product from the one or more different products based on the derived user preference; and

providing for display at a display interface;

a video frame of the plurality of video frames in a first portion of the display interface shown to the user, and

a scrolling carousel of products located in a second portion of the display interface, the carousel displaying content associated with a selected product for an identified point of interest of the identified points of interest corresponding to the video frame being concurrently displayed to the user in the first portion of the display interface, and wherein the scrolling carousel further concurrently displays a portion of content associated with a next selected product associated with a next video frame of the plurality of video frames containing a point of interest.

2. The method of claim 1, wherein providing for display at the display interface comprises:

for each video frame currently being displayed to the user;

determining whether the video frame has a detected point of interest; and

selecting one or more products associated with the detected point of interest in the video frame.

3. The method of claim 1,

wherein the selected product is same or similar to a product shown in the video frame.

4. The method of claim 1, wherein providing for display the scrolling carousel of products comprises:

displaying one or more tools in the second portion of the display interface, wherein the user interacts with the displayed products through the one or more tools.

5. The method of claim 4, wherein the one or more tools displayed in the second portion of the display interface comprise one of the following:

a purchase tool for the user to purchase the displayed product; and

a bookmark tool for the user to bookmark the displayed product.

6. The method of claim 1, wherein a point of interest in a video frame is identified by one of a timestamp associated with the video frame and frame number of the video frame.

7. The method of claim 1, wherein the selected product associated with the identified point of interest shares one or more characteristics with the object in the video frame associated with the identified point of interest.

8. The method of claim 1, wherein analyzing visual features of the video frame comprises:

applying a trained image feature extraction model to the video frame to extract a plurality of visual features;

identifying an object in the video frame based on the extracted visual features; and

identifying one or more items associated with the identified object, each identified item corresponding to a product that is likely to be of interest to the user.

9. The method of claim 1, wherein the video displayed to the user is one of a pre-recorded video and a broadcast video streamed to the user in real time.

19. The method of claim 1, wherein the scrolling carousel applies a gradual replacement effect to replace the content associated with the selected product with the content associated with the next selected product between display of the video frame and the next video frame.

10. A non-transitory computer readable storage medium storing computer program instructions, the computer program instructions when executed by a computer processor causes the processor to perform the steps of:

receiving a video for display to a user, the video comprising a plurality of video frames;

for each video frame;

analyzing visual features of the video frame; and

detecting one or more points of interest in the video frame based on the analysis of the visual features of the video frame, a point of interest in the video frame being associated with an object in the video frame, and the object in the video frame being related to one or more different products;

for each of the identified point of interests in the plurality of video frames, selecting a product of the one or more different products related to the object associated with the point of interest, by;

analyzing user profile information associated with the user to derive a user preference;

and

selecting the product from the one or more different products based on the derived user preference; and

providing for display at a display interface;

a video frame of the plurality of video frames in a first portion of the display interface shown to the user, and

a scrolling carousel of products located in a second portion of the display interface, the carousel displaying content associated with a selected product for an identified point of interest of the identified points of interest corresponding to the video frame being concurrently displayed to the user in the first portion of the display interface, and wherein the scrolling carousel further concurrently displays a portion of content associated with a next selected product associated with a next video frame of the plurality of video frames containing a point of interest.

11. The non-transitory computer readable storage medium of claim 10, wherein providing for display at the display interface comprises:

for each video frame currently being displayed to the user;

determining whether the video frame has a detected point of interest; and

selecting one or more products associated with the detected point of interest in the video frame.

12. The non-transitory computer readable storage medium of claim 10,

wherein the selected product is same or similar to a product shown in the video frame.

13. The non-transitory computer readable storage medium of claim 10, wherein providing for display the scrolling carousel of products comprises:

displaying one or more tools in the second portion of the display interface, wherein the user interacts with the displayed products through one or more tools.

14. The non-transitory computer readable storage medium of claim 13, wherein the one or more tools displayed in the second portion of the display interface comprise one of the following:

a purchase tool for the user to purchase the displayed product; and

a bookmark tool for the user to bookmark the displayed product.

15. The non-transitory computer readable storage medium of claim 10, wherein a point of interest in a video frame is identified by one of a timestamp associated with the video frame and frame number of the video frame.

16. The non-transitory computer readable storage medium of claim 10, wherein the selected product associated with the identified point of interest shares one or more characteristics with the object in the video frame associated with the identified point of interest.

17. The non-transitory computer readable storage medium of claim 10, wherein analyzing visual features of the video frame comprises:

applying a trained image feature extraction model to the video frame to extract a plurality of visual features;

identifying an object in the video frame based on the extracted visual features; and

identifying one or more items associated with the identified object, each identified item corresponding to a product that is likely to be of interest to the user.

18. The non-transitory computer readable storage medium of claim 10, wherein the video displayed to the user is one of a pre-recorded video and a broadcast video streamed to the user in real time.

20. The non-transitory computer readable storage medium of claim 10, wherein the scrolling carousel applies a gradual replacement effect to replace the content associated with the selected product with the content associated with the next selected product between display of the video frame and the next video frame.