Watermarking and signal recognition for managing and sharing captured content, metadata discovery and related arrangements

• US 9,979,691 B2
• Filed: 09/26/2016
• Issued: 05/22/2018
• Est. Priority Date: 05/03/2013
• Status: Active Grant

• Current Assignee: Digimarc Corporation
• Original Assignee: Digimarc Corporation
• Inventors: Lord, John D.
• Primary Examiner(s): NAKHJAVAN, SHERVIN K
• Application Number:
  US15/276,505
• Publication Number:
  US 20170085518A1
• Time in Patent Office: 603 Days
• Field of Search: None
• CPC Class Codes: G06T 1/0021 Image watermarking
• CPC Class Codes: G06T 2201/005 Image watermarking
• CPC Class Codes: G10L 19/018 Audio watermarking, i.e. embedding inaudible data in the audio signal
• CPC Class Codes: H04L 51/10 Multimedia information
• CPC Class Codes: H04L 51/52 for supporting social networking services
• CPC Class Codes: H04N 21/21805 enabling multiple viewpoints, e.g. using a plurality of cameras
• CPC Class Codes: H04N 21/2187 Live feed
• CPC Class Codes: H04N 21/233 Processing of audio elementary streams monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/2665 Gathering content from different sources, e.g. Internet and satellite
• CPC Class Codes: H04N 21/2668 Creating a channel for a dedicated end-user group, e.g. insertion of targeted commercials based on end-user profiles information retrieval from the Internet by querying with filtering and personalisation G06F16/9535; arrangements for replacing or switching information during the broadcast H04H20/10; push services over packet-switching network H04L12/1859; adaptation of message content in packet-switching networks H04L51/063
• CPC Class Codes: H04N 21/2743 Video hosting of uploaded data from client
• CPC Class Codes: H04N 21/42203 sound input device, e.g. microphone
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/4622 Retrieving content or additional data from different sources, e.g. from a broadcast channel and the Internet web site content organization and management for information retrieval from the Internet G06F16/958; transmission by internet of broadcast information H04H60/82; stock exchange data over packet-switching network H04L12/1804; push services including data channel over packet-switching network H04L12/1859
• CPC Class Codes: H04N 21/4788 communicating with other users, e.g. chatting arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104
• CPC Class Codes: H04N 21/6581 Reference data, e.g. a movie identifier for ordering a movie or a product identifier in a home shopping application
• CPC Class Codes: H04N 21/8358 involving watermark protecting executable software by watermarking G06F21/16; image watermarking in general G06T1/0021; watermarks inserted in still images for transmission purposes H04N1/32144; inserting watermarks during video coding H04N19/467

Abstract

Content is identified using watermarking and/or other content recognition combined with contextual metadata, which facilitates identification and correlation with other content and metadata when it is posted to a network.

Claims

1. A method comprising:

in a network system, receiving items of content captured by electronic devices associated with users, the received items being uploaded to the network system, and the network system is configured to stream or download the received items to other users of the network system;

with one or more processors, executing instructions to perform the acts of;

extracting digital watermarks in audio signals from the received items of captured content;

with watermark IDs decoded from the digital watermarks, querying a database to obtain a first instance of metadata associated with a first watermark ID extracted from a corresponding first instance of received item and querying the database to obtain other instances of metadata associated with other watermark IDs extracted from corresponding other instances of the received items;

correlating the first instance of metadata associated with the first instance of received item with the other instances of metadata corresponding to the other instances of the received items, the correlating identifying related received items that have correlated metadata; and

aggregating the related received items for presentation to a user of the network system.

2. The method of claim 1, wherein the received items of captured content comprise audio content captured by microphones of the users at a live event, wherein each of the electronic devices captures instances of ambient audio at the live event, the instances each being embedded with the digital watermark, which is associated with the live event.

3. The method of claim 1, wherein the received items of captured content comprise imagery.

4. The method of claim 1 wherein the watermark IDs are embedded in captured live audio, which is captured from a live performance in electronic audio processing equipment;

the captured live audio being processed by the electronic audio processing equipment to embed a watermark ID and then output audio with a watermark ID embedded in the output to the users at the live performance with low latency.

5. The method of claim 1, further comprising:

deriving content fingerprints from the received items;

in the network system, identifying the received items by matching the content fingerprints with corresponding fingerprints associated with identifying metadata;

obtaining second metadata of the received items associated with the identifying metadata; and

correlating the second metadata of the received items.

6. The method of claim 1, further comprising correlating the received items with each other based on the watermark IDs.

7. The method of claim 1 wherein the first instance and other instances of metadata are comprised of metadata obtained by plural stages of aggregating metadata, including a first stage of obtaining first metadata associated with a watermark ID extracted from one of the received items and a second stage of obtaining metadata associated with the first metadata, and wherein the correlating determines the correlated metadata by correlating metadata of plural different received items that has been aggregated for the received items in the plural stages.

8. The method of claim 7 wherein the second stage of obtaining metadata comprises obtaining metadata indexed in plural databases by the first metadata.

9. The method of claim 7 comprising;

receiving upload metadata provided by users to the network system along with the received items when the received items are posted to the network system; and

aggregating the upload metadata with other metadata associated with a first received item by associating the upload metadata and the other metadata with a watermark ID extracted from the first received item.

10. The method of claim 1 further comprising:

updating metadata associated with the received items in the network system by receiving metadata update information at the network system and in response, querying an ID database to obtain updated metadata for the received items that is associated with the received items based on the watermark IDs extracted from the received items.

11. The method of claim 10 wherein the updating is performed after the received items are posted to the network system to update metadata of the received items with new metadata created after the received items are posted to the network system.

12. The method of claim 1 wherein the correlating is updated by searching metadata of a second set of received items posted to the network system after a first set of received items has been posted to the network system;

and updating the first set of received items by linking received items from the second set with received items that have correlated metadata.

13. A system comprising:

an uploading system, the uploading system configured to receive items of content captured by electronic devices associated with users and transform the received items to a format for downloading or streaming to other users of the system;

a watermark recovery system communicatively coupled to the uploading system, the watermark recovery system configured to extract digital watermarks in audio signals from the received items of captured content; and

a server communicatively coupled with the watermark recovery system to receive watermark IDs decoded from the digital watermarks, and configured to query a database to obtain a first instance of metadata associated with a first watermark ID extracted from a corresponding first instance of received item and to query the database to obtain other instances of metadata associated with other watermark IDs extracted from corresponding other instances of the received items and provide the first and other instances of metadata to the uploading system;

wherein the uploading system is configured to correlate the first instance of metadata associated with the first instance of received item with the other instances of metadata corresponding to the other instances of the received items, the uploading system executing the correlating to identify related received items that have correlated metadata; and

the uploading system configured to aggregate the related received items for presentation to a user.

14. The system of claim 13 wherein the first instance and other instances of metadata are comprised of metadata obtained by plural stages of aggregating metadata;

wherein the uploading system is configured to obtain, in a first stage, first metadata associated with a watermark ID extracted from one of the received items, and is configured to obtain, in a second stage, metadata associated with the first metadata; and

wherein the uploading system is configured to obtain the correlated metadata by correlating metadata of plural different received items that has been aggregated for the received items in the first and second stages.

15. The system of claim 14 wherein the uploading system is configured to obtain, in the second stage, metadata indexed in plural databases by the first metadata.

16. The system of claim 13 wherein the uploading system is configured to receive upload metadata provided by users along with the received items when the received items are posted to the uploading system, and is configured to aggregate the upload metadata with other metadata associated with a first received item by associating the upload metadata and the other metadata with a watermark ID extracted from the first received item.

17. The system of claim 13 wherein the uploading system is configured to update metadata associated with the received items by receiving metadata update information, and in response, querying an ID database to obtain updated metadata for the received items that is associated with the received items based on the watermark IDs extracted from the received items.

18. The system of claim 17 wherein the uploading system is configured to update metadata of the received items after the received items are posted to the uploading system to update metadata of the received items with new metadata created after the received items are posted to the uploading system.

19. The system of claim 13 wherein the uploading system is configured to search metadata of a second set of received items posted to the network system after a first set of received items has been posted to the network system, and is configured to update the first set of received items by linking received items from the second set with received items that have correlated metadata.

20. The system of claim 13 wherein the watermark IDs are embedded in captured live audio, which is captured from a live performance in electronic audio processing equipment;

the captured live audio being processed by the electronic audio processing equipment to embed a watermark ID and then output audio with a watermark ID embedded in the output at the live performance with low latency.