Generating a shoppable video

• US 10,354,290 B2
• Filed: 06/16/2015
• Issued: 07/16/2019
• Est. Priority Date: 06/16/2015
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Inc.
• Inventors: Yadav, Vikas, Krishnamurthy, Balaji, Sarkar, Mausoom, Mangla, Rajiv, Malik, Gitesh
• Primary Examiner(s): Kumar, Pankaj
• Assistant Examiner(s): Riaz, Sahar Aqil
• Application Number:
  US14/741,111
• Publication Number:
  US 20160371546A1
• Time in Patent Office: 1,491 Days
• Field of Search: 725 36, 725 43
• CPC Class Codes: G06Q 30/0276 Advertisement creation
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06V 10/757 Matching configurations of points or features
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04N 21/254 Management at additional data server, e.g. shopping server, rights management server arrangements for maintenance or administration in data networks H04L41/00; network services using third party service providers H04L67/53
• CPC Class Codes: H04N 21/44222 Analytics of user selections, e.g. selection of programs or purchase activity monitoring of user selections in data processing systems G06F11/34; arrangements for monitoring the user's behaviour or opinions in broadcast systems H04H60/33
• CPC Class Codes: H04N 5/7605 on discs or drums H04N5/781, H04N5/805, H04N5/83, H04N5/85 take precedence
• CPC Class Codes: H04N 5/93 Regeneration of the television signal or of selected parts thereof

Abstract

Embodiments of the present invention provide systems and methods for automatically generating a shoppable video. A video is parsed into one or more scenes. Products and their corresponding product information are automatically associated with the one or more scenes. The shoppable video is then generated using the associated products and corresponding product information such that the products are visible in the shoppable video based on a scene in which the products are found.

Claims

1. One or more non-transitory computer storage media storing computer-useable instructions that, when used by one or more computing devices, cause the one or more computing devices to generate a shoppable video, comprising:

parsing, by a scene parsing module, a video into one or more video scenes;

automatically identifying, by a product determining module, one or more products in a first video scene of the one or more video scenes, the identifying comprising computing a product score for a plurality of products having corresponding product images stored in a database by calculating a feature vector for each of a plurality of tiles corresponding to the first video scene and comparing values of the feature vector corresponding to each of the plurality of tiles to values of feature vectors of product images stored in a database, wherein the product score indicates a likelihood that a product is visible in the first video scene;

retrieving, by a video generation module, product information corresponding to the one or more products in the first video scene based on the automatically identified one or more products;

automatically associating, by the video generation module, the one or more products and the corresponding product information with the first video scene by storing an indication that the one or more products in the first video scene corresponds to the retrieved product information; and

generating the shoppable video using the automatically associated one or more products and the corresponding product information, the shoppable video comprising at least the first video scene having the product information corresponding to the one or more products.

2. The one or more computer storage media of claim 1, wherein the identifying of the one or more products in the first video scene is based on a plurality of features identified in the video frames of the first video scene compared to a plurality of features identified in a plurality of product images stored in a database.

3. The one or more computer storage media of claim 1, wherein automatically associating the one or more products and the corresponding product information with the first video scene further comprises:

parsing the one or more video scenes into a plurality of frames;

for a first frame of the plurality of frames, producing one or more copy frames having different resolutions; and

partitioning the first frame and the one or more copy frames into a plurality of tiles.

4. The one or more computer storage media of claim 3, further comprising:

for each of the plurality of tiles in the first frame and the one or more copy frames,

(1) computing a feature vector,

(2) comparing values of the feature vector corresponding to each of the plurality of tiles to values of feature vectors of product images stored in a database, and

(3) based on the comparison, computing the product score for the plurality of products having corresponding product images stored in the database.

5. The one or more computer storage media of claim 4, further comprising based at least on the computed product score, determining the one or more products to associate with the at least the portion of the one or more video scenes.

6. The one or more computer storage media of claim 4, further comprising:

for a second frame of the plurality of frames, producing one or more copy frames having different resolutions than the second frame;

partitioning the second frame and the one or more copy frames into a plurality of tiles; and

for each of the plurality of tiles in the second frame and the one or more copy frames,

(1) computing a feature vector,

(2) comparing values of the feature vector corresponding to each of the plurality of tiles to values of the feature vectors of the product images stored in the database, and

(3) based on the comparison, computing the product score for the plurality of products having corresponding product images stored in the database.

7. The one or more computer storage media of claim 4, wherein the feature vector is an n-dimensional vector of numerical features that represent an object.

8. The one or more computer storage media of claim 4, wherein the feature vector numerically represents one or more of color components, length, area, shape description, gradient magnitude, or gradient direction of the object in the plurality of tiles and the product images.

9. The one or more computer storage media of claim 4, wherein the feature vectors are computed using a deep convolutional neural network.

10. The one or more computer storage media of claim 1, further comprising:

determining the one or more products to automatically associate with the at least the portion of the one or more scenes, wherein the determining is based, at least, on a computed product scores for a plurality of products having corresponding product images stored in a database.

11. The one or more computer storage media of claim 10, wherein the computed product scores are based on, at least, monitoring votes received for the plurality of products, such that a first product having a higher vote count than a second product indicates an increased likelihood that the first product is visible in the video.

12. The one or more computer storage media of claim 3, wherein at least a portion of the plurality of tiles overlap with another tile.

13. The one or more computer storage media of claim 1, wherein the shoppable video presents the one or more products based on a scene in which each product is found.

14. A computerized method for generating a shoppable video, the computerized method comprising:

computing, by an image feature vector module, a feature vector for each of a plurality of product images stored in a database;

automatically identifying, by a product determining module, one or more products in a first video scene of the one or more video scenes, the identifying comprising;

computing a feature vector using a tile feature vector module for a first tile in a first frame of a video, the first frame included in a first video scene;

comparing values, by the product determining module, of the feature vector of the first tile to values of the feature vectors for the plurality of product images; and

based on the comparing, determining that the first tile includes an image of a first product that corresponds to a first set of images of the plurality of product images stored in the database wherein the determining comprises computing a product score for the first product based on the feature vector of the first tile and the feature vectors for the plurality of product images, wherein the product score indicates a likelihood that the first product is visible in the first video scene; and

generating the shoppable video based on the automatically identified one or more products, the shoppable video comprising the first video scene having product information corresponding to the first product.

15. The computerized method of claim 14, further comprising:

computing a feature vector for a second tile in a first frame of a video;

comparing values of the feature vector of the second tile to values of the feature vectors for the plurality of product images;

based on the comparing, determining that the second tile includes an image of a second product that corresponds to a second set of images of the plurality of product images stored in the database; and

generating the shoppable video that comprises the product information corresponding to the first product and the second product.

16. The computerized method of claim 14, further comprising grouping the plurality of frames into one or more video scenes, wherein the shoppable video includes a presentation of the first product and one or more other products found in the video based on a video scene in which each product is found.

17. The computerized method of claim 14, wherein determining that the first tile includes an image of a first product that corresponds to a first set of images of the plurality of product images stored in the database further comprises monitoring votes received for the first product, and wherein the first product having a higher vote count than a second product indicates an increased likelihood that the first product is visible in the video.

18. A computerized system comprising:

a datastore storing product images and associated feature vectors computed for the product images;

one or more processors; and

one or more computer storage media storing computer-useable instructions that, when used by the one or more processors, cause the one or more processors to;

divide, by a scene parsing module, a video into one or more video scenes, each comprising a plurality of frames,

for each of the plurality of frames, generate one or more copy frames using a copy resolution component such that the one or more copy frames have a different resolution,

partition, by a tile parsing module, each of the plurality of frames and each of the one or more copy frames into a plurality of tiles,

for each of the plurality of tiles, compute, by a tile feature vector module, a feature vector,

compare values of the feature vector via a feature vector comparing module, for each of the plurality of tiles to values of the feature vectors computed for the product images stored in the database,

for a first video scene, use the comparison of the values of the feature vectors to automatically determine which products are visible in each of the plurality of frames, wherein the comparison includes determining a product score for each of the plurality of tiles, wherein the product score indicates a likelihood that a product is visible in a frame; and

generate a shoppable video based on the automatically determined products visible in each of the plurality of frames, the shoppable video comprising at least the first video scene having product information corresponding to the one or more product images.

19. The system of claim 18, wherein shoppable video is generated using the products that are determined to be visible in each of the plurality of frames and a computed product score for the products determined to be visible.

20. The system of claim 19, further comprising grouping the plurality of frames into one or more video scenes, wherein the shoppable video includes a presentation of the products that are determined to be visible in each of the plurality of frames based on a video scene in which each product is found.