Interactive content generation

• US 9,336,459 B2
• Filed: 10/20/2015
• Issued: 05/10/2016
• Est. Priority Date: 07/03/2014
• Status: Expired due to Fees

• Current Assignee: OIM Squared Inc.
• Original Assignee: OIM Squared Inc.
• Inventors: Williams, Eric, Cordova-Diba, Bertin Rudolph, Munguia, Jose Adrian, Simon, Tyson
• Primary Examiner(s): TSAI, TSUNG YIN
• Application Number:
  US14/918,478
• Publication Number:
  US 20160042250A1
• Time in Patent Office: 203 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/73 Querying
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 20/20 in augmented reality scenes

Abstract

Generation of interactive content. In an embodiment, a representation of candidate object(s) in content of a digital media asset are received. For each of the candidate object(s), feature(s) of the candidate object are compared to corresponding feature(s) of a plurality of reference objects to identify reference object(s) that match the candidate object. For each of the matched candidate object(s), a hotspot package is generated. The hotspot package may comprise a visual overlay which comprises information associated with the reference object(s) matched to the respective candidate object.

Claims

1. A computer-implemented method that comprises using at least one hardware processor to:

receive a representation of one or more candidate objects in content of a digital media asset;

for each of the one or more candidate objects, compare a plurality of features of the candidate object to a plurality of corresponding features of each of a plurality of reference objects to identify one or more reference objects that match the candidate object, wherein the comparison comprises, for each of the plurality of reference objects,

for each of the plurality of features of the candidate object, comparing the feature of the candidate object to a corresponding feature of the reference object, and generating a feature score based on the comparison of the feature of the candidate object to the corresponding feature of the reference object,

determining a weight for each feature score,

weighting each feature score according to the weight determined for that feature score, and

generating a matching score based on each weighted feature score; and

,

for each of the one or more candidate objects that is matched to one or more reference objects, generate a hotspot package, wherein the hotspot package comprises a visual overlay, and wherein the visual overlay comprises information associated with the one or more reference objects matched to the candidate object.

2. The method of claim 1, wherein the representation of one or more candidate objects is received in a visual query that is received from a network device via at least one network.

3. The method of claim 2, further comprising using the at least one hardware processor of the network device to generate the visual query, wherein generating the visual query comprises:

displaying the content of the digital media asset;

receiving a selection of a portion of the displayed content via a user operation; and

generating the visual query based on the selected portion of the displayed content.

4. The method of claim 1, wherein the representation of one or more candidate objects comprises, for each of the one or more candidate objects, an image of the candidate object.

5. The method of claim 4, wherein the digital media asset is a digital image, and wherein each of the one or more images, in the representation of one or more candidate objects, is a portion of the digital image.

6. The method of claim 4, wherein comparing a plurality of features of the candidate object to a plurality of corresponding features of each of a plurality of reference objects further comprises deriving the plurality of features of the candidate object from visual attributes in the image of the candidate object.

7. The method of claim 6, wherein deriving the plurality of features of the candidate object from visual attributes in the image of the candidate object comprises detecting the candidate object in the image of the candidate object prior to deriving the plurality of features of the candidate object.

8. The method of claim 7, wherein detecting the candidate object in the image of the candidate object comprises:

normalizing the image of the candidate object; and

determining a boundary that surrounds the candidate object within the image of the candidate object.

9. The method of claim 8, wherein normalizing the image of the candidate object comprises:

removing image noise from the image of the candidate object;

converting the image of the candidate object to a photometric invariant color space; and

converting the image of the candidate object to one or both of a predefined size and a predefined aspect ratio.

10. The method of claim 8, wherein determining a boundary that surrounds the candidate object within the image of the candidate object comprises:

segmenting the image of the candidate object into regions;

merging similar neighboring regions until no similar neighboring regions remain; and

determining a boundary around one or more remaining regions as the boundary that surrounds the candidate object.

11. The method of claim 10, wherein merging similar neighboring regions comprises, for a pair of neighboring regions:

calculating a variation between the pair of neighboring regions;

merging the pair of neighboring regions when the variation is less than a threshold, and

not merging the pair of neighboring regions when the variation is greater than the threshold.

12. The method of claim 1, wherein the digital media asset is a video, and wherein the representation of one or more candidate objects comprises a frame of the video.

13. The method of claim 1, wherein the plurality of features comprise two or more of a color feature, a texture feature, a shape feature, and a keypoints feature.

14. The method of claim 1, wherein, for each of the one or more candidate objects that is matched to one or more reference objects, the matched one or more reference objects comprise an object that is identified as the candidate object.

15. The method of claim 14, wherein the visual overlay of each hotspot package comprises one or more of an image, title, description, and price associated with the object that is identified as the candidate object.

16. The method of claim 1, wherein, for each of the one or more candidate objects that is matched to one or more reference objects, the matched one or more reference objects comprise an object that is visually similar to the candidate object.

17. The method of claim 16, wherein the visual overlay of each hotspot package comprises a representation of each object, from the matched one or more reference objects, that is visually similar to the candidate object.

18. The method of claim 1, wherein generating a hotspot package for each of the one or more candidate objects that is matched to one or more reference objects comprises, for each of the one or more reference objects matched to the candidate object:

retrieving information associated with the reference object using an identifier of the reference object; and

incorporating the information into the hotspot package.

19. The method of claim 18, wherein each of the plurality of reference objects represents a product, and wherein the information associated with the one or more reference objects comprises one or more of an image, a title, a description, and a price.

20. The method of claim 1, further comprising using the at least one hardware processor to generate a navigation hotspot package, wherein the navigation hotspot package comprises a visual overlay, and wherein the visual overlay comprises one or more inputs for one or both of searching and selecting each of the hotspot packages for the one or more candidate objects that were matched to one or more reference objects.

21. The method of claim 1, further comprising using the at least one hardware processor to embed each hotspot package with the digital media asset, wherein embedding each hotspot package with the digital media asset comprises:

generating an asset template;

embedding a viewer for the digital media asset into the asset template; and

, for each hotspot package,

generating a hotspot package template, and

embedding the hotspot package template into the asset template.

22. The method of claim 21, wherein the asset template and each hotspot package template are generated in a markup language.

23. The method of claim 1, wherein the visual overlay of each hotspot package for the one or more candidate objects that were matched to one or more reference objects comprises one or more inputs for initiating a purchase for at least one of the matched one or more reference objects.

24. The method of claim 1, further comprising using the at least one hardware processor to, for each of the one or more candidate objects that is matched to one or more reference objects:

determine a position of the candidate object in the content of the digital media asset; and

generate a hotspot, wherein the hotspot comprises a visual indication to be overlaid at a hotspot position in the content of the digital media asset corresponding to the determined position of the candidate object, and wherein the hotpot is associated with the hotspot package generated for the candidate object.

25. The method of claim 24, wherein the visual indication of each hotspot is selectable via a user operation so as toggle the associated hotspot package between a visible and invisible state.

26. The method of claim 25, further comprising rendering the content of the digital media asset with the visual indication of each hotspot overlaid at its respective hotspot position in the content of the digital media asset.

27. The method of claim 1, wherein comparing a plurality of features of a candidate object to a plurality of corresponding features of each of a plurality of reference objects is performed according to a first feature-matching algorithm defined by a first feature-matching software module, and wherein the method further comprises using the at least one hardware processor to:

receive a second feature-matching software module, defining a second feature-matching algorithm, via an interface; and

,

subsequently, compare a plurality of features of a candidate object to a plurality of corresponding features of each of a plurality of reference objects according to the second feature-matching algorithm defined by the second feature-matching software module, instead of the first feature-matching algorithm defined by the first feature-matching software module.

28. The method of claim 1, further comprising using the at least one hardware processor to embed each hotspot package with the digital media asset.

29. A system comprising:

at least one hardware processor; and

one or more software modules that are configured to, when executed by the at least on hardware processor,

receive a representation of one or more candidate objects in content of a digital media asset,

for each of the one or more candidate objects, compare a plurality of features of the candidate object to a plurality of corresponding features of each of a plurality of reference objects to identify one or more reference objects that match the candidate object, wherein the comparison comprises, for each of the plurality of reference objects,

for each of the plurality of features of the candidate object, comparing the feature of the candidate object to a corresponding feature of the reference object, and generating a feature score based on the comparison of the feature of the candidate object to the corresponding feature of the reference object,

determining a weight for each feature score,

weighting each feature score according to the weight determined for that feature score, and

generating a matching score based on each weighted feature score, and,

for each of the one or more candidate objects that is matched to one or more reference objects, generate a hotspot package, wherein the hotspot package comprises a visual overlay, and wherein the visual overlay comprises information associated with the one or more reference objects matched to the candidate object.

30. A non-transitory computer-readable medium having instructions stored thereon, wherein the instructions, when executed by a processor, cause the processor to:

receive a representation of one or more candidate objects in content of a digital media asset;

for each of the one or more candidate objects, compare a plurality of features of the candidate object to a plurality of corresponding features of each of a plurality of reference objects to identify one or more reference objects that match the candidate object, wherein the comparison comprises, for each of the plurality of reference objects,

for each of the plurality of features of the candidate object, comparing the feature of the candidate object to a corresponding feature of the reference object, and generating a feature score based on the comparison of the feature of the candidate object to the corresponding feature of the reference object,

determining a weight for each feature score,

weighting each feature score according to the weight determined for that feature score, and

generating a matching score based on each weighted feature score; and

,

for each of the one or more candidate objects that is matched to one or more reference objects, generate a hotspot package, wherein the hotspot package comprises a visual overlay, and wherein the visual overlay comprises information associated with the one or more reference objects matched to the candidate object.