METHODS AND APPARATUS FOR MEDIA SEARCH, CHARACTERIZATION, AND AUGMENTED REALITY PROVISION

• US 20190238952A1
• Filed: 01/25/2019
• Published: 08/01/2019
• Est. Priority Date: 01/31/2018
• Status: Active Grant

• Current Assignee: WowYow, Inc.
• Original Assignee: WowYow, Inc.
• Inventors: BOSKOVICH, ADAM
• Granted Patent: US 10,939,182 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/433 using audio data
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/635 Overlay text, e.g. embedded captions in a TV program
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: H04N 19/114 Adapting the group of pictures [GOP] structure, e.g. number of B-frames between two anchor frames H04N19/107 takes precedence
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/2668 Creating a channel for a dedicated end-user group, e.g. insertion of targeted commercials based on end-user profiles information retrieval from the Internet by querying with filtering and personalisation G06F16/9535; arrangements for replacing or switching information during the broadcast H04H20/10; push services over packet-switching network H04L12/1859; adaptation of message content in packet-switching networks H04L51/063
• CPC Class Codes: H04N 21/4728 for selecting a Region Of Interest [ROI], e.g. for requesting a higher resolution version of a selected region
• CPC Class Codes: H04N 21/8133 specifically related to the content, e.g. biography of the actors in a movie, detailed information about an article seen in a video program
• CPC Class Codes: H04N 21/858 Linking data to content, e.g. by linking an URL to a video object, by creating a hotspot

Abstract

Methods and apparatus for providing attribute-based search of media assets such as video and audio assets, and associated augmented reality functions including dynamic provision of relevant secondary content relating to the identified attribute(s). In one embodiment, media or content assets are ingested into a processing system and processed according to one or more attribute detection, identification, and characterization algorithms. Attributes detected may include for example the presence of tangible items such as clothing or chattels, particular persons such as celebrities, and/or certain contexts such as sporting activities and musical performances, as rendered within the media asset. In one implementation, the characterized assets are provided a unique ID, and stored so as to permit cross-correlation based on, e.g., the identified and characterized attributes. Secondary content (e.g., advertising or promotional content) is correlated to each identified asset, and dynamically served upon a user providing some threshold level of interaction with the attribute.

Claims

1. A computerized method for automatically providing secondary digital content relating to a user-cognizable digital element present in a digital content asset, the method comprising:

identifying a plurality of digital content assets;

processing, using a plurality of computer programs, the plurality of digital content assets, the processing comprising;

identifying at least one user-cognizable digital element within each of the plurality of identified digital content assets;

characterizing the identified at least one user-cognizable digital element in each of the plurality of identified digital content assets to generate characterizing data;

assigning data indicative of a unique identifier to each of the plurality of identified digital content assets;

storing the generated characterizing data along with at least a portion of the data indicative of the unique identifier in a database; and

forming an association between (i) at least a portion of the stored generated characterizing data of the identified at least one of the user-cognizable digital element in each of the plurality of identified digital content assets, and (ii) secondary digital content;

receiving data indicative of interest by a user in the at least one identified user-cognizable digital element within at least one of the plurality of identified digital content assets; and

based at least on the received data indicative of interest, causing provision of the secondary digital content to a computerized device associated with the user.

2. The computerized method of claim 1, wherein:

the plurality of identified digital content assets comprise digitally encoded video data having a plurality of I-Frames, P-Frames, and B-Frames; and

the identifying the at least one user-cognizable digital element within each of the plurality of identified digital content assets comprises utilizing at least one shape recognition algorithm to analyze at least one I-Frame associated with the digitally encoded video data of each of the plurality of identified digital content assets.

3. The computerized method of claim 2, wherein the utilizing the at least one shape recognition algorithm to analyze at least one I-Frame associated with the digitally encoded video data of each of the plurality of identified digital content assets comprises:

decoding the digital video data;

identifying the at least one I-frame within the decoded digital video data; and

utilizing the at least one polygonal shape recognition algorithm on data of the identified at least one I-frame to identify at least one polygon therein; and

utilizing the identified at least one polygon to identify an attribute.

4. The computerized method of claim 3, wherein the utilizing the identified at least one polygon to identify an attribute comprises using one or more geometric relationships associated with the polygon to search a database of known geometric relationships associated with one or more tangible items.

5. The computerized method of claim 1, wherein the processing the plurality of digital content assets using the plurality of computer programs at least in part comprises pre-processing the plurality of digital content assets during a first prescribed period of time and prior to a second, later prescribed period of time during which the pre-processed plurality of digital content assets are made available to one or more computerized user devices, the computerized device associated with the user comprising one of the computerized user devices.

6. The computerized method of claim 1, wherein the receiving data indicative of interest by the user in the at least one identified user-cognizable digital element within at least one of the plurality of identified digital content assets comprises receiving data generated by a JavaScript element operatively associated with an Internet website, the data generated by the JavaScript element received as part of an HTTPS session established between the computerized device associated with the user and at least one server apparatus.

7. The computerized method of claim 1, wherein the receiving data indicative of interest by the user in the at least one identified user-cognizable element within at least one of the plurality of identified digital content assets comprises receiving data generated by a JavaScript element operatively associated with an Internet website, the data generated based at least in part on a user performing at least one of the following via an input device of the computerized device associated with the user:

(i) causing, via the input device, a cursor to be located within a boundary of the at least one identified user-cognizable digital element that is rendered on a display device of the computerized device associated with the user; and

/or (ii) using the input device to pause rendering of the at least one identified digital content asset by the computerized device associated with the user.

8. The computerized method of claim 1, wherein the forming the association between (i) the at least portion of the stored generated characterizing data of the at least one of the identified user-cognizable digital elements in each of the identified digital content assets, and (ii) secondary digital content comprises:

receiving data indicative of an identity of a tangible item;

searching the generated characterizing data of the database based at least in part on the data indicative of the identity of the tangible item to identify one or more instances of the tangible item within the generated characterizing data; and

associating the secondary digital content to the identified one or more instances, the secondary digital content comprising digitally rendered content relating to the tangible item.

9. The computerized method of claim 1, wherein:

the identifying at least one user-cognizable digital element within each of the plurality of identified digital content assets comprises;

decoding an audio stream of each of the plurality of identified digital content assets; and

processing the decoded audio stream of each of the plurality of identified digital content assets to identify at least one user-cognizable audio element within each; and

the characterizing of the at least one identified user-cognizable digital element in each of the plurality of identified digital content assets to generate characterizing data comprises characterizing the at least one identified user-cognizable audio element in each of the plurality of identified digital content assets based on a plurality of tones or sounds therein, the plurality of tones or sounds corresponding to at least one of a (i) trademark or service mark, and/or (ii) a trade dress element, associated with a particular good or service.

10. A computerized system for media processing, analysis, and selective provision of secondary content, the computerized system comprising:

a computerized server apparatus;

a database in data communication with the computerized server apparatus; and

a customer portal environment operative to execute on one or more computerized customer devices associated with an operator of the system.

11. The computerized system of claim 10, wherein the computerized server apparatus comprises an application programming interface (API) configured to enable each of the one or more computerized customer devices to, via a lightweight script operative to execute on a network site accessible to each of the one or more computerized customer devices, transparently access augmented reality (AR) information having secondary content associated with one or more identifiable elements within data frames or containers of media assets accessed on the network site and renderable by the one or more computerized customer devices.

12. The computerized system of claim 11, wherein the server apparatus comprises a plurality of servers configured to operate in logical and data communication with one another so as to enable (i) substantially real time (near-real time) processing of one or more user-selected media assets to identify and characterize the identifiable elements, the one or more user-selected media assets selected by at least one of the one or more computerized customer devices, and (ii) dynamic return of related secondary content to the at least one of the one or more computerized customer devices while the rendering of the one or more user-selected media asset occurs.

13. A computerized apparatus configured for automatic provision of secondary digital content relating to a user-cognizable digital element present in a digital content asset, the computerized apparatus comprising:

data interface apparatus configured for data communication with a plurality of computerized user devices and a database;

data processor apparatus in data communication with the data interface apparatus; and

data storage apparatus in data communication with the data processor apparatus, the data storage apparatus having at least one computer program stored thereon, the at least one computer program comprising a plurality of instructions, the plurality of instructions configured to, when executed on the data processor apparatus, cause the computerized apparatus to;

identify a plurality of digital content assets; and

perform processing of the plurality of digital content assets, the processing comprising;

identification a plurality of user-cognizable digital elements, one or more of the plurality of user-cognizable digital elements being within each of the plurality of identified digital content assets;

generation of characterizing data related to individual ones of the plurality of user-cognizable digital elements;

generation of data indicative of a plurality of unique identifiers, each of the plurality of unique identifiers corresponding ton one of the plurality of identified digital content assets;

storage, in the database, of the generated characterizing data related to individual ones of the plurality of user-cognizable digital elements associated with corresponding ones of the plurality of unique identifiers; and

generation of data indicative of an association between (i) at least a portion of the stored characterizing data, and (ii) secondary digital content.

14. The computerized apparatus of claim 13, wherein the plurality of instructions are further configured to, when executed on the data processor apparatus, cause the computerized apparatus to receive data indicative of interest by a user in at least one of the plurality of user-cognizable digital elements within at least one of the plurality of identified digital content assets, the data indicative of interest by the user comprising data indicative of a user action performed via an input device of at least one of the plurality of computerized user devices which is associated with the user, the data indicative of the user action comprising data indicative of one or more of (i) the user causing, via the input device, a cursor to be located within a boundary of the at least one identified user-cognizable digital element, or (ii) the user using the input device to pause rendering of the at least one identified digital content asset by the computerized device associated with the user.

15. The computerized apparatus of claim 14, wherein the plurality of instructions are further configured to, when executed on the data processor apparatus, cause the computerized apparatus to, based at least on the received data indicative of interest by the user, cause provision of at least one element of the secondary digital content to the at least one of the plurality of computerized user devices which is associated with the user.