Interactive content generation

• US 9,317,778 B2
• Filed: 10/20/2015
• Issued: 04/19/2016
• Est. Priority Date: 07/03/2014
• Status: Expired due to Fees

• Current Assignee: OIM Squared Inc.
• Original Assignee: OIM Squared Inc.
• Inventors: Williams, Eric, Cordova-Diba, Bertin Rudolph, Munguia, Jose Adrian, Simon, Tyson
• Primary Examiner(s): TSAI, TSUNG YIN
• Application Number:
  US14/918,491
• Publication Number:
  US 20160042251A1
• Time in Patent Office: 182 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/73 Querying
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 20/20 in augmented reality scenes

Abstract

Generation of interactive content. In an embodiment, a representation of candidate object(s) in content of a digital media asset are received. For each of the candidate object(s), feature(s) of the candidate object are compared to corresponding feature(s) of a plurality of reference objects to identify reference object(s) that match the candidate object. For each of the matched candidate object(s), a hotspot package is generated. The hotspot package may comprise a visual overlay which comprises information associated with the reference object(s) matched to the respective candidate object.

Claims

1. A computer-implemented method that comprises using at least one hardware processor to:

receive a representation of one or more candidate objects in content of a digital media asset;

for each of the one or more candidate objects, compare one or more features of the candidate object to one or more corresponding features of a plurality of reference objects, according to a first feature-matching algorithm defined by a first feature-matching software module, to identify one or more reference objects that match the candidate object;

for each of the one or more candidate objects that is matched to one or more reference objects, generate a hotspot package, wherein the hotspot package comprises a visual overlay, and wherein the visual overlay comprises information associated with the one or more reference objects matched to the candidate object;

receive a second feature-matching software module, defining a second feature-matching algorithm, via an interface; and

,

subsequently, compare one or more features of one or more other candidate objects to one or more corresponding features of a plurality of reference objects, according to the second feature-matching algorithm defined by the second feature-matching software module, instead of the first feature-matching algorithm defined by the first feature-matching software module.

2. The method of claim 1, wherein the representation of one or more candidate objects is received in a visual query that is received from a network device via at least one network.

3. The method of claim 2, further comprising using the at least one hardware processor of the network device to generate the visual query, wherein generating the visual query comprises:

displaying the content of the digital media asset;

receiving a selection of a portion of the displayed content via a user operation; and

generating the visual query based on the selected portion of the displayed content.

4. The method of claim 1, wherein the representation of one or more candidate objects comprises, for each of the one or more candidate objects, an image of the candidate object.

5. The method of claim 4, wherein the digital media asset is a digital image, and wherein each of the one or more images, in the representation of one or more candidate objects, is a portion of the digital image.

6. The method of claim 4, wherein comparing one or more features of the candidate object to one or more corresponding features of a plurality of reference objects comprises deriving the one or more features of the candidate object from visual attributes in the image of the candidate object.

7. The method of claim 6, wherein deriving the one or more features of the candidate object from visual attributes in the image of the candidate object is performed according to a first feature-extraction algorithm defined by a first feature-extraction software module, and wherein the method further comprises using the at least one hardware processor to:

receive a second feature-extraction software module, defining a second feature-extraction algorithm, via an interface; and

,

subsequently, derive one or more features of a candidate object from visual attributes in the image of a candidate object, according to the second feature-extraction algorithm defined by the second feature-extraction software module, instead of the first feature-extraction algorithm defined by the first feature-extraction software module.

8. The method of claim 6, wherein deriving the one or more features of the candidate object from visual attributes in the image of the candidate object comprises detecting the candidate object in the image of the candidate object prior to deriving the one or more features of the candidate object.

9. The method of claim 8, wherein detecting the candidate object in the image of the candidate object comprises:

normalizing the image of the candidate object; and

determining a boundary that surrounds the candidate object within the image of the candidate object.

10. The method of claim 9, wherein normalizing the image of the candidate object comprises:

removing image noise from the image of the candidate object;

converting the image of the candidate object to a photometric invariant color space; and

converting the image of the candidate object to one or both of a predefined size and a predefined aspect ratio.

11. The method of claim 9, wherein determining a boundary that surrounds the candidate object within the image of the candidate object comprises:

segmenting the image of the candidate object into regions;

merging similar neighboring regions until no similar neighboring regions remain; and

determining a boundary around one or more remaining regions as the boundary that surrounds the candidate object.

12. The method of claim 1, wherein the digital media asset is a video, and wherein the representation of one or more candidate objects comprises a frame of the video.

13. The method of claim 11, wherein merging similar neighboring regions comprises, for a pair of neighboring regions:

calculating a variation between the pair of neighboring regions;

merging the pair of neighboring regions when the variation is less than a threshold; and

not merging the pair of neighboring regions when the variation is greater than the threshold.

14. The method of claim 9, wherein determining a boundary that surrounds the candidate object within the image of the candidate object is performed according to a first object-recognition algorithm defined by a first object-recognition software module, and wherein the method further comprises using the at least one hardware processor to:

receive a second object-recognition software module, defining a second object-recognition algorithm, via an interface; and

,

subsequently, determining a boundary that surrounds a candidate object within an image of the candidate object, according to the second object-recognition algorithm defined by the second object-recognition software module, instead of the first object-recognition algorithm defined by the first object-recognition software module.

15. The method of claim 1, wherein comparing one or more features of the candidate object to one or more corresponding features of a plurality of reference objects to identify one or more reference objects that match the candidate object comprises, for each of the plurality of reference objects:

for each of the one or more features of the candidate object, comparing the feature of the candidate object to a corresponding feature of the reference object, and generating a feature score based on the comparison of the feature of the candidate object to the corresponding feature of the reference object; and

generating a matching score based on each feature score.

16. The method of claim 15, wherein generating a matching score based on each feature score is performed according to a first data-fusion algorithm defined by a first data-fusion software module, and wherein the method further comprises using the at least one hardware processor to:

receive a second data-fusion software module, defining a second data-fusion algorithm, via an interface; and

,

subsequently, generate a matching score based on each feature score, according to the second data-fusion algorithm defined by the second data-fusion software module, instead of the first data-fusion algorithm defined by the first data-fusion software module.

17. The method of claim 1, wherein the one or more features comprise two or more of a color feature, a texture feature, a shape feature, and a keypoints feature.

18. The method of claim 1, wherein, for each of the one or more candidate objects that is matched to one or more reference objects, the matched one or more reference objects comprise an object that is identified as the candidate object.

19. The method of claim 18, wherein the visual overlay of each hotspot package comprises one or more of an image, title, description, and price associated with the object that is identified as the candidate object.

20. The method of claim 1, wherein, for each of the one or more candidate objects that is matched to one or more reference objects, the matched one or more reference objects comprise an object that is visually similar to the candidate object.

21. The method of claim 20, wherein the visual overlay of each hotspot package comprises a representation of each object, from the matched one or more reference objects, that is visually similar to the candidate object.

22. The method of claim 1, wherein generating a hotspot package for each of the one or more candidate objects that is matched to one or more reference objects comprises, for each of the one or more reference objects matched to the candidate object:

retrieving information associated with the reference object using an identifier of the reference object; and

incorporating the information into the hotspot package.

23. The method of claim 1, further comprising using the at least one hardware processor to generate a navigation hotspot package, wherein the navigation hotspot package comprises a visual overlay, and wherein the visual overlay comprises one or more inputs for one or both of searching and selecting each of the hotspot packages for the one or more candidate objects that were matched to one or more reference objects.

24. The method of claim 1, further comprising using the at least one hardware processor to embed each hotspot package with the digital media asset, wherein embedding each hotspot package with the digital media asset comprises:

generating an asset template;

embedding a viewer for the digital media asset into the asset template; and

,

for each hotspot package,

generating a hotspot package template, and

embedding the hotspot package template into the asset template.

25. The method of claim 24, wherein the asset template and each hotspot package template are generated in a markup language.

26. The method of claim 1, wherein the visual overlay of each hotspot package for the one or more candidate objects that were matched to one or more reference objects comprises one or more inputs for initiating a purchase for at least one of the matched one or more reference objects.

27. The method of claim 1, further comprising using the at least one hardware processor to, for each of the one or more candidate objects that is matched to one or more reference objects:

determine a position of the candidate object in the content of the digital media asset; and

generate a hotspot, wherein the hotspot comprises a visual indication to be overlaid at a hotspot position in the content of the digital media asset corresponding to the determined position of the candidate object, and wherein the hotpot is associated with the hotspot package generated for the candidate object.

28. The method of claim 27, wherein the visual indication of each hotspot is selectable via a user operation so as toggle the associated hotspot package between a visible and invisible state.

29. A system comprising:

at least one hardware processor; and

one or more software modules that are configured to, when executed by the at least on hardware processor,

receive a representation of one or more candidate objects in content of a digital media asset,

for each of the one or more candidate objects, compare one or more features of the candidate object to one or more corresponding features of a plurality of reference objects, according to a first feature-matching algorithm defined by a first feature-matching software module, to identify one or more reference objects that match the candidate object,

for each of the one or more candidate objects that is matched to one or more reference objects, generate a hotspot package, wherein the hotspot package comprises a visual overlay, and wherein the visual overlay comprises information associated with the one or more reference objects matched to the candidate object,

receive a second feature-matching software module, defining a second feature-matching algorithm, via an interface, and,

subsequently, compare one or more features of one or more other candidate objects to one or more corresponding features of a plurality of reference objects, according to the second feature-matching algorithm defined by the second feature-matching software module, instead of the first feature-matching algorithm defined by the first feature-matching software module.

30. A non-transitory computer-readable medium having instructions stored thereon, wherein the instructions, when executed by a processor, cause the processor to:

receive a representation of one or more candidate objects in content of a digital media asset;

for each of the one or more candidate objects, compare one or more features of the candidate object to one or more corresponding features of a plurality of reference objects, according to a first feature-matching algorithm defined by a first feature-matching software module, to identify one or more reference objects that match the candidate object;

for each of the one or more candidate objects that is matched to one or more reference objects, generate a hotspot package, wherein the hotspot package comprises a visual overlay, and wherein the visual overlay comprises information associated with the one or more reference objects matched to the candidate object;

receive a second feature-matching software module, defining a second feature-matching algorithm, via an interface; and

,

subsequently, compare one or more features of one or more other candidate objects to one or more corresponding features of a plurality of reference objects, according to the second feature-matching algorithm defined by the second feature-matching software module, instead of the first feature-matching algorithm defined by the first feature-matching software module.