Annotation of metadata through capture infrastructure

• US 10,074,108 B2
• Filed: 09/08/2016
• Issued: 09/11/2018
• Est. Priority Date: 11/26/2008
• Status: Active Grant

• Current Assignee: Samba TV Incorporated
• Original Assignee: Free Stream Media Corporation
• Inventors: Navin, Ashwin, Harrison, David, Smallman, Josh
• Primary Examiner(s): Ibrahim, Mohamed
• Application Number:
  US15/259,146
• Publication Number:
  US 20160381435A1
• Time in Patent Office: 733 Days
• Field of Search: None
• CPC Class Codes: G06F 16/783 using metadata automatically derived from the content
• CPC Class Codes: G06F 21/10 Protecting distributed programs or content, e.g. vending or licensing of copyrighted material protection in video systems or pay television H04N7/16 ; Digital rights management [DRM]
• CPC Class Codes: G06F 21/53 by executing in a restricted environment, e.g. sandbox or secure virtual machine
• CPC Class Codes: G06F 2221/033 Test or assess software
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06Q 30/0255 based on user history
• CPC Class Codes: G06Q 30/0269 based on user profile or attribute
• CPC Class Codes: G08C 17/02 using a radio link
• CPC Class Codes: H04L 12/18 for broadcast or conference , e.g. multicast
• CPC Class Codes: H04L 2101/668 Internet protocol [IP] address subnets
• CPC Class Codes: H04L 61/2514 between local and global IP addresses
• CPC Class Codes: H04L 61/2517 using port numbers
• CPC Class Codes: H04L 61/256 NAT traversal
• CPC Class Codes: H04L 61/2575 using address mapping retrieval, e.g. simple traversal of user datagram protocol through session traversal utilities for NAT [STUN]
• CPC Class Codes: H04L 61/4511 using domain name system [DNS]
• CPC Class Codes: H04L 61/4541 Directories for service discovery
• CPC Class Codes: H04L 61/5007 Internet protocol [IP] addresses
• CPC Class Codes: H04L 63/10 for controlling access to devices or network resources
• CPC Class Codes: H04L 63/102 Entity profiles
• CPC Class Codes: H04L 65/1069 Session establishment or de-establishment
• CPC Class Codes: H04L 65/61 : for supporting one-way streaming services, e.g. Internet radio
• CPC Class Codes: H04L 67/02 : based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/10 : in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/141 : Setup of application sessions admission control or resource allocation in data switching networks H04L47/70
• CPC Class Codes: H04L 67/51 : Discovery or management thereof, e.g. service location protocol [SLP] or web services
• CPC Class Codes: H04N 21/23424 : involving splicing one content stream with another content stream, e.g. for inserting or substituting an advertisement
• CPC Class Codes: H04N 21/2541 : Rights Management protecting software against unauthorised usage in a vending or licensing environment G06F21/10; security in data switching network management H04L41/28; security management or policies for network security H04L63/20; access security in wireless networks H04W12/08
• CPC Class Codes: H04N 21/278 : Content descriptor database or directory service for end-user access details of content or meta data based information retrieval of video data in video databases G06F16/739
• CPC Class Codes: H04N 21/4126 : The peripheral being portable, e.g. PDAs or mobile phones
• CPC Class Codes: H04N 21/41407 : embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/4147 : PVR [Personal Video Recorder] H04N5/76 takes precedence ; arrangements for broadcast with accumulation-type receivers H04H20/40
• CPC Class Codes: H04N 21/435 : Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream
• CPC Class Codes: H04N 21/4415 : using biometric characteristics of the user, e.g. by voice recognition or fingerprint scanning cryptography using biological data H04L9/3231; authentication in networks using biometric H04L63/0861
• CPC Class Codes: H04N 21/466 : Learning process for intelligent management, e.g. learning user preferences for recommending movies services using the results of monitoring in broadcast systems H04H60/61
• CPC Class Codes: H04N 21/6175 : involving transmission via Internet broadcast-related systems characterised by the transmission system being the Internet H04H60/82
• CPC Class Codes: H04N 21/64322 : IP
• CPC Class Codes: H04N 21/812 : involving advertisement data advertising per se G06Q30/02
• CPC Class Codes: H04N 21/835 : Generation of protective data, e.g. certificates protecting software against unauthorised usage in a vending or licensing environment G06F21/10
• CPC Class Codes: H04N 21/8352 : involving content or source identification data, e.g. Unique Material Identifier [UMID]
• CPC Class Codes: H04N 21/8358 : involving watermark protecting executable software by watermarking G06F21/16; image watermarking in general G06T1/0021; watermarks inserted in still images for transmission purposes H04N1/32144; inserting watermarks during video coding H04N19/467
• CPC Class Codes: H04N 21/84 : Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73

Abstract

A method, apparatus and system annotation of meta-data through a capture infrastructure are disclosed. In one embodiment, a method of a client device includes applying an automatic content recognition algorithm to determine a content identifier of an audio-visual data. The client device then associates the content identifier with an advertisement data based on a semantic correlation between a meta-data of the advertisement provided by a content provider and/or the content identifier. A capture infrastructure annotates the audio-visual data with a brand name and/or a product name by comparing entries in the master database with a closed captioning data of the audio-visual data and/or through an application of an optical character recognition algorithm in the audio-visual data. The content identifier may involve a music identification, an object identification, a facial identification, and/or a voice identification. A minimal functionality including accessing a tuner and/or a stream decoder that identifies a channel and/or a content may be found in the networked media device. The networked media device may produce an audio fingerprint and/or a video fingerprint that is communicated with the capture infrastructure.

Claims

1. A method of a client device comprising:

constraining an executable environment in a security sandbox;

executing a sandboxed application in the constrained executable environment;

through the sandboxed application, associating with a networked media device by way of a sandbox reachable service executing on the networked media device using a communication provided by the networked media device;

applying an automatic content recognition algorithm to determine a content identifier of an audio-visual data rendered through the sandboxed application;

associating the content identifier with an advertisement data based on a semantic correlation between a metadata of the advertisement data provided by a content provider and the content identifier; and

targeting the advertisement data at the networked media device based on the association of the content identifier with the advertisement data and the association between the client device and the networked media device in accordance with;

enabling, through a capture infrastructure, annotation of the audio-visual data.

2. The method of claim 1, wherein at least one of:

the method comprises enabling the annotation of the audio-visual data with at least one of a brand name and a product name by comparing entries in a master database with at least one of a closed captioning data of the audio-visual data and through an application of an optical character recognition algorithm in the audio-visual data,

the content identifier is at least one of a music identification, an object identification, a facial identification, and a voice identification,

a minimal functionality comprising accessing at least one of a tuner and a stream decoder that identifies at least one of a channel and a content is found in the networked media device,

the networked media device produces at least one of an audio fingerprint and a video fingerprint that is communicated with the capture infrastructure,

the capture infrastructure compares the at least one of the audio fingerprint and the video fingerprint with the master database,

the capture infrastructure further annotates the audio-visual data with a logo name by comparing the entries in the master database with a logo data of the audio-visual data identified using a logo detection algorithm,

the capture infrastructure automatically divides the audio-visual data into a series of scenes based on a semantic grouping of actions in the audio-visual data,

the audio-visual data is analyzed in advance of a broadcast to determine content identifiers associated with each commercial in the audio-visual data such that advertisements are pre-inserted into the audio-visual data prior to the broadcast,

the capture infrastructure applies a time-order algorithm to automatically match the advertisements to the audio-visual data when a correlation pattern is identified by the capture infrastructure with other audio-visual content previously analyzed,

the capture infrastructure includes a buffer that is saved to a persistent storage and for which a label is generated to facilitate identification of reoccurring sequences,

a post processing operation is at least one of automated through a post-processing algorithm and a crowd-sourced operation using a plurality of users in which a turing test is applied to determine a veracity of an input,

a device pairing algorithm is used in which a cookie data associated with a web page visited by a user stored on a browser on the client device is paired with the networked media device when the client device is communicatively coupled with the networked media device,

a transitive public IP matching algorithm is utilized in which at least one of the client device and the networked media device communicates each public IP address with any paired entity to the capture infrastructure,

a tag that is unconstrained from a same-origin policy is used to automatically load an advertisement in the browser, and

the tag is at least one of an image tag, a frame, an iframe, and a script tag.

3. The method of claim 2, further comprising:

referencing an additional metadata comprising at least one of the content identifier and the advertisement based on a video processing algorithm,

wherein the additional metadata is at least one of a title, a description, a thumbnail, a name of an individual, and a historical data, and

wherein the additional metadata is determined from a browser history captured from the client device based on a capture policy, and correlating a relevance of the browser history with the at least one of the content identifier and the advertisement;

automatically instantiating a connection between the sandboxed application and an unannounced device associated with the networked media device based on determining that an IP address of a port from the unannounced device is associated with the networked media device;

processing an identification data associated with the sandbox reachable service sharing a public address with the client device;

determining a private address pair of the sandbox reachable service based on the identification data;

establishing a communication session between the sandboxed application and the sandbox reachable service using a cross-site scripting technique of the security sandbox; and

appending a header of a hypertext transfer protocol to permit the networked media device to communicate with the sandboxed application as a permitted origin domain through a Cross-origin resource sharing (CORS) algorithm, the header being either one of an origin header when the CORS algorithm is applied and a referrer header in an alternate algorithm.

4. The method of claim 3, further comprising:

accessing a pairing server when processing the identification data associated with the sandbox reachable service sharing the public address with the client device,

wherein the pairing server performs a discovery lookup of any device that has announced sharing of the public address associated with the client device, and

wherein the sandbox reachable service announces itself to the pairing server prior to the establishment of the communication session between the sandboxed application and the sandbox reachable service.

5. The method of claim 4, wherein at least one of:

the sandbox reachable service announces an availability of the sandbox reachable service across a range of public addresses such that the sandboxed application communicates with the sandbox reachable service in any one of the range of the public addresses,

the range of public addresses is known by the pairing server so that the announcement of the availability of the sandbox reachable service across the range of public addresses is unnecessary,

the sandbox reachable service communicates at least one of a global unique identifier and an alphanumeric name to the pairing server along with the private address pair of the sandbox reachable service, and

the private address pair includes a private IP address and a port number associated with the sandbox reachable service.

6. The method of claim 3, further comprising:

extending the security sandbox with a discovery algorithm and a relay algorithm through a discovery module and a relay module added to the security sandbox; and

bypassing the pairing server having the discovery algorithm and the relay algorithm when establishing the communication session between the sandboxed application and the sandbox reachable service when the security sandbox is extended with the discovery algorithm and the relay algorithm through the discovery module and the relay module added to the security sandbox.

7. The method of claim 6, further comprising:

applying the discovery algorithm of the security sandbox to determine that the networked media device having the sandbox reachable service communicates in a shared network common to the client device and the networked media device; and

applying the relay algorithm of the security sandbox to establish the communication session between the sandboxed application and the sandbox reachable service of the networked media device.

8. The method of claim 6:

wherein the discovery algorithm utilizes a protocol comprising at least one of a Bonjour®

protocol, a SSDP protocol, a LSD uTorrent®

protocol, a multicast protocol, an anycast protocol, and another Local Area Network based protocol that discovers services in a Local Area Network based on a broadcast from any one of an operating system service, the security sandbox, the client device, the sandbox reachable service, and the networked media device.

9. The method of claim 3, wherein the sandboxed application is at least one of a web page, a script, a binary executable, an intermediate bytecode, an abstract syntax tree, and an executable application in the security sandbox, wherein the sandboxed application comprises at least one of a HyperText Markup Language 5 (HTML5) application, a Javascript®

Title;

ANNOTATION OF METADATA THROUGH CAPTURE INFRASTRUCTURE application, an Adobe®

Flash®

application, a Microsoft®

Silverlight®

application, a JQuery®

application, an Asynchronous Javascript® and

an XML (AJAX) application, and wherein an access control algorithm governs a policy through which a secondary authentication is required when establishing the communication session between the sandboxed application and the networked media device.

10. The method of claim 9, further comprising:

utilizing an exception to a same origin policy through a use of at least one of a hyperlink, a form, the script, a frame, a header, and an image when establishing the communication session between the sandboxed application and the sandbox reachable service.

11. The method of claim 3, further comprising:

eliminating a communication through a centralized infrastructure when the sandboxed application and the sandbox reachable service communicate in a shared network common to the client device and the networked media device when the connection is instantiated, the shared network being at least one of a local area network, a multicast network, an anycast network, and a multilan network;

minimizing a latency in the communication session when the sandboxed application and the sandbox reachable service communicate in the shared network common to the client device and the networked media device when the connection is instantiated; and

improving privacy in the communication session when the sandboxed application and the sandbox reachable service communicate in the shared network common to the client device and the networked media device when the connection is instantiated.

12. The method of claim 1,

wherein the client device and the networked media device reside on networks that are incommunicable with each other comprising at least one of a firewall separation, a different network separation, a physical separation, and an unreachable connection separation, and

wherein the sandboxed application of the security sandbox of the client device and the sandbox reachable service of the networked media device communicate with each other through a relay service employed by a pairing server having a discovery module and a relay module to facilitate a trusted communication between the sandboxed application and the sandbox reachable service.

13. The method of claim 12:

wherein the trusted communication is facilitated such that the sandboxed application never learns at least one of a private IP address and a hardware address of the networked media device when;

a first Network Address Translator (NAT) device coupled with a network on which the client device operates receives communications from a public IP address of a different network on which the sandbox reachable service operates, and

a second NAT device coupled with the different network on which the networked media device operates translates the private IP address of the networked media device to the public IP address visible to the sandboxed application.

14. The method of claim 12:

wherein the networked media device comprises a plurality of sandbox reachable applications including the sandbox reachable application,

wherein a service agent module of the networked media device coordinates communications with a discovery module of at least one of the security sandbox and the pairing server,

wherein the security sandbox is at least one of an operating system on which the sandboxed application is hosted and a browser application of the operating system, and

wherein the networked media device is at least one of a television, a projection screen, a multimedia display, a touchscreen display, an audio device, and a multidimensional visual presentation device.

15. The method of claim 12, a further comprising:

utilizing at least one of a WebSocket and a long polling service message query interface to reduce a latency of message delivery during the trusted communication between the sandboxed application and the sandbox reachable service; and

optimizing a polling period between polling to be less than a timeout period of a session through the relay service.

16. The method of claim 12, a further comprising:

initiating the relay service through at least one of a series of web pages where information is communicated using hyperlinks that point at the pairing server, and a form having a confirmation dialog that is submitted back to the pairing server,

wherein a global unique identifier is masked through the pairing server when the confirmation dialog is served from the pairing server.

17. The method of claim 1,

wherein a cookie associated with the security sandbox is used to store a remote access token on a storage of the client device,

wherein the remote access token identifies at least one of a set of communicable private IP addresses and hardware addresses associated with sandbox reachable services that previously operated on a common shared network with the client device, and

wherein the client device can communicate with the sandbox reachable services that previously operated on the common shared network through the remote access token.

18. A method of a networked device comprising:

associating with a client device executing a sandboxed application by way of a sandbox reachable service executing on the networked device using a communication provided by the networked device, the client device constraining an executable environment in a security sandbox and executing the sandboxed application in the constrained executable environment;

applying an automatic content recognition algorithm to determine a content identifier of an audio-visual data;

associating the content identifier with an advertisement data based on a semantic correlation between a metadata of the advertisement data provided by a content provider and the content identifier; and

targeting the advertisement data at the client device based on the association of the content identifier with the advertisement data and the association between the client device and the networked device in accordance with;

enabling, through a capture infrastructure, annotation of the audio-visual data.

19. The method of claim 18, further comprising:

accessing a pairing server when processing an identification data associated with the sandbox reachable service of the networked device that shares a public address with the client device,

wherein the pairing server performs a discovery lookup of any device that has announced that it shares the public address associated with the client device, and

wherein the sandbox reachable service announces itself to the pairing server prior to establishment of a communication session between the sandboxed application and the sandbox reachable service; and

appending a header of a hypertext transfer protocol to permit the networked device to communicate with the sandboxed application as a permitted origin domain through a CORS algorithm, the header being either one of an origin header when the CORS algorithm is applied and a referrer header in an alternate algorithm in accordance with the client device;

processing the identification data associated with the sandbox reachable service sharing the public address with the client device;

determining a private address pair of the sandbox reachable service based on the identification data; and

establishing the communication session between the sandboxed application and the sandbox reachable service using a cross-site scripting technique of the security sandbox.

20. The method of claim 19, wherein:

the pairing server performs a discovery lookup of any device that has announced sharing of the public address associated with the client device, and

the method further comprises the sandbox reachable service announcing itself to the pairing server prior to the establishment of the communication session between the sandboxed application and the sandbox reachable service.

21. The method of claim 20, further comprising at least one of:

announcing an availability of the sandbox reachable service across a range of public addresses such that the sandboxed application communicates with the sandbox reachable service in any one of the range of the public addresses; and

communicating at least one of a global unique identifier, a hardware address, and an alphanumeric name to the pairing server along with the private address pair of the sandbox reachable service,

wherein the private address pair includes a private IP address and a port number associated with the sandbox reachable service.

22. The method of claim 19, wherein the communication session is established between the sandboxed application and the sandbox reachable service in accordance with the client device:

extending the security sandbox with a discovery algorithm and a relay algorithm through a discovery module and a relay module added to the security sandbox; and

bypassing the pairing server having the discovery algorithm and the relay algorithm when establishing the communication session between the sandboxed application and the sandbox reachable service when the security sandbox is extended with the discovery algorithm and the relay algorithm through the discovery module and the relay module added to the security sandbox.

23. The method of claim 22:

wherein the client device and the networked device reside on networks that are incommunicable with each other comprising at least one of a firewall separation, a different network separation, a physical separation, and an unreachable connection separation, and

wherein the sandboxed application of the security sandbox of the client device and the sandbox reachable service of the networked device communicate with each other through a relay service employed by the pairing server having the discovery module and the relay module to facilitate a trusted communication between the sandboxed application and the sandbox reachable service.

24. The method of claim 23:

wherein the trusted communication is facilitated such that the sandboxed application never learns at least one of a private IP address and a hardware address of the networked device when;

a first NAT device coupled with a network on which the client device operates receives communications from a public IP address of a different network on which the sandbox reachable service operates, and

a second NAT device coupled with the different network on which the networked device operates translates the private IP address of the networked device to the public IP address visible to the sandboxed application.

25. The method of claim 22, wherein the communication session is established between the sandboxed application and the sandbox reachable service in accordance with the client device:

applying the discovery algorithm of the security sandbox to determine that the networked device having the sandbox reachable service communicates in a shared network common to the client device and the networked device; and

applying the relay algorithm of the security sandbox to establish the communication session between the sandboxed application and the sandbox reachable service of the networked device.

26. The method of claim 22:

wherein the discovery algorithm utilizes a protocol comprising at least one of a Bonjour®

protocol, a SSDP protocol, a LSD uTorrent®

protocol, a multicast protocol, an anycast protocol, and another Local Area Network based protocol that discovers services in a Local Area Network based on a broadcast from any one of an operating system service, the security sandbox, the client device, the sandbox reachable service, and the networked device.

27. The method of claim 22:

wherein the networked device comprises a plurality of sandbox reachable applications including the sandbox reachable application,

wherein a service agent module of the networked device coordinates communications with the discovery module of at least one of the security sandbox and the pairing server,

wherein the security sandbox is at least one of an operating system on which the sandboxed application is hosted and a browser application of the operating system, and

wherein the networked device is at least one of a television, a projection screen, a multimedia display, a touchscreen display, an audio device, a weather measurement device, a traffic monitoring device, a status update device, a global positioning device, a geospatial estimation device, a tracking device, a bidirectional communication device, a unicast device, a broadcast device, and a multidimensional visual presentation device.

28. The method of claim 23, wherein the communication session is established between the sandboxed application and the sandbox reachable service in accordance with the client device:

utilizing at least one of a WebSocket and a long polling service message query interface to reduce a latency of message delivery during the trusted communication between the sandboxed application and the sandbox reachable service; and

optimizing a polling period between polling to be less than a timeout period of a session through the relay service.

29. The method of claim 23, wherein the communication session is established between the sandboxed application and the sandbox reachable service in accordance with the client device:

initiating the relay service through at least one of a series of web pages where information is communicated using hyperlinks that point at the pairing server, and a form having a confirmation dialog that is submitted back to the pairing server,

wherein a global unique identifier is masked through the pairing server when the confirmation dialog is served from the pairing server.

30. The method of claim 19:

wherein the sandboxed application is at least one of a web page, a script, a binary executable, an intermediate bytecode, an abstract syntax tree, and an executable application in the security sandbox,

wherein the sandboxed application comprises at least one of a HyperText Markup Language 5 (HTML5) application, a Javascript®

application, an Adobe®

Flash®

application, a Microsoft®

Silverlight®

application, a JQuery®

application, an Asynchronous Javascript® and

an XML (AJAX) application, and

wherein an access control algorithm governs a policy through which a secondary authentication is required when establishing the communication session between the sandboxed application and the networked device.

31. The method of claim 30, wherein the communication session is established between the sandboxed application and the sandbox reachable service in accordance with the client device:

utilizing an exception to a same origin policy through a use of at least one of a hyperlink, a form, the script, a frame, a header, and an image.

32. The method of claim 19, further comprising:

eliminating a communication through a centralized infrastructure when the sandboxed application and the sandbox reachable service communicate in a shared network common to the client device and the networked device when the communication session is established, the shared network being at least one of a local area network, a multicast network, an anycast network, and a multilan network;

minimizing a latency in the communication session when the sandboxed application and the sandbox reachable service communicate in the shared network common to the client device and the networked device when the communication session is established; and

improving privacy in the communication session when the sandboxed application and the sandbox reachable service communicate in the shared network common to the client device and the networked device when the communication session is established.

33. The method of claim 18, wherein at least one of:

the method comprises enabling the annotation of the audio-visual data with at least one of a brand name and a product name by comparing entries in a master database with at least one of a closed captioning data of the audio-visual data and through an application of an optical character recognition algorithm in the audio-visual data,

the content identifier is at least one of a music identification, an object identification, a facial identification, and a voice identification,

a minimal functionality comprising accessing at least one of a tuner and a stream decoder that identifies at least one of a channel and a content is found in the networked device,

the method further comprises producing, through the networked device, at least one of an audio fingerprint and a video fingerprint that is communicated with the capture infrastructure,

the capture infrastructure compares the at least one of the audio fingerprint and the video fingerprint with the master database,

the capture infrastructure annotates the audio-visual data with a logo name by comparing the entries in the master database with a logo data of the audio-visual data identified using a logo detection algorithm,

the capture infrastructure automatically divides the audio-visual data into a series of scenes based on a semantic grouping of actions in the audio-visual data,

the audio-visual data is analyzed in advance of a broadcast to determine content identifiers associated with each commercial in the audio-visual data such that advertisements are pre-inserted into the audio-visual data prior to the broadcast,

the capture infrastructure applies a time-order algorithm to automatically match the advertisements to the audio-visual data when a correlation pattern is identified by the capture infrastructure with other audio-visual content previously analyzed,

the capture infrastructure includes a buffer that is saved to a persistent storage and for which a label is generated to facilitate identification of reoccurring sequences,

a post processing operation is at least one of automated through a post-processing algorithm and a crowd-sourced operation using a plurality of users in which a turing test is applied to determine a veracity of an input,

a device pairing algorithm is used in which a cookie data associated with a web page visited by a user stored on a browser on a client device is paired with the networked device when the client device is communicatively coupled with the networked device,

a transitive public IP matching algorithm is utilized in which at least one of the client device and the networked device communicates each public IP address with any paired entity to the capture infrastructure,

a tag that is unconstrained from a same-origin policy is used to automatically load an advertisement in the browser, and

the tag is at least one of an image tag, a frame, an iframe, and a script tag.

34. The method of claim 18:

wherein a cookie associated with the security sandbox is used to store a remote access token on a storage of the client device,

wherein the remote access token identifies at least one of a set of communicable private IP addresses and hardware addresses associated with sandbox reachable services that previously operated on a common shared network with the client device, and

wherein the client device can communicate with the sandbox reachable services that previously operated on the common shared network through the remote access token.

35. A system comprising:

a client device to;

constrain an executable environment in a security sandbox, and

execute a sandboxed application in the constrained executable environment; and

a networked device to execute a sandbox reachable service thereon,

wherein the client device and the networked device are associated with each other through the sandboxed application by way of the sandbox reachable service using a communication provided by the networked device,

wherein at least one of the;

networked device and the client device is configured to apply an automatic content recognition algorithm to determine a content identifier of an audio-visual data and to associate the content identifier with an advertisement data based on a semantic correlation between a metadata of the advertisement data provided by a content provider and the content identifier, and

wherein the at least one of the networked device and the client device is further configured to target the advertisement data at a corresponding at least one of;

the client device and the networked device based on the association of the content identifier with the advertisement data and the association between the client device and the networked device in accordance with enabling, through a capture infrastructure, annotation of the audio-visual data.

36. The system of claim 35:

wherein a communication session is established between the sandboxed application and the sandbox reachable service by appending a header of a hypertext transfer protocol to permit the networked device to communicate with the sandboxed application as a permitted origin domain through a CORS algorithm, the header being either one of an origin header when the CORS algorithm is applied and a referrer header in an alternate algorithm.

37. The system of claim 36, wherein:

the client device accesses a pairing server when processing identification data associated with the sandbox reachable service sharing a public address with the client device,

the pairing server performs a discovery lookup of any device that has announced sharing of the public address associated with the client device, and

the sandbox reachable service announces itself to the pairing server prior to the establishment of the communication session between the sandboxed application and the sandbox reachable service.

38. The system of claim 37, wherein the client device is configured to:

extend the security sandbox with a discovery algorithm and a relay algorithm through a discovery module and a relay module added to the security sandbox, and

bypass the pairing server having the discovery algorithm and the relay algorithm when establishing the communication session between the sandboxed application and the sandbox reachable service when the security sandbox is extended with the discovery algorithm and the relay algorithm through the discovery module and the relay module added to the security sandbox.

39. The system of claim 38:

wherein the client device and the networked device reside on networks that are incommunicable with each other comprising at least one of a firewall separation, a different network separation, a physical separation, and an unreachable connection separation, and

wherein the sandboxed application of the security sandbox of the client device and the sandbox reachable service of the networked device communicate with each other through a relay service employed by the pairing server having the discovery module and the relay module to facilitate a trusted communication between the sandboxed application and the sandbox reachable service.

40. The system of claim 39:

wherein the trusted communication is facilitated such that the sandboxed application never learns at least one of a private IP address and a hardware address of the networked device when;

a first NAT device coupled with a network on which the client device operates receives communications from a public IP address of a different network on which the sandbox reachable service operates, and

a second NAT device coupled with the different network on which the networked device operates translates the private IP address of the networked device to the public IP address visible to the sandboxed application.

41. The system of claim 38, wherein the client device is configured to:

apply the discovery algorithm of the security sandbox to determine that the networked device having the sandbox reachable service communicates in a shared network common to the client device and the networked device, and

apply the relay algorithm of the security sandbox to establish the communication session between the sandboxed application and the sandbox reachable service of the networked device.

42. The system of claim 38:

wherein the discovery algorithm utilizes a protocol comprising at least one of a Bonjour®

protocol, a SSDP protocol, a LSD uTorrent®

protocol, a multicast protocol, an anycast protocol, and another Local Area Network based protocol that discovers services in a Local Area Network based on a broadcast from any one of an operating system service, the security sandbox, the client device, the sandbox reachable service, and the networked device.

43. The system of claim 38:

wherein the networked device comprises a plurality of sandbox reachable applications including the sandbox reachable application,

wherein a service agent module of the networked device coordinates communications with the discovery module of at least one of the security sandbox and the pairing server,

wherein the security sandbox is at least one of an operating system on which the sandboxed application is hosted and a browser application of the operating system, and

wherein the networked device is at least one of a television, a projection screen, a multimedia display, a touchscreen display, an audio device, a weather measurement device, a traffic monitoring device, a status update device, a global positioning device, a geospatial estimation device, a tracking device, a bidirectional communication device, a unicast device, a broadcast device, and a multidimensional visual presentation device.

44. The system of claim 39, wherein the client device is configured to:

utilize at least one of a WebSocket and a long polling service message query interface to reduce a latency of message delivery during the trusted communication between the sandboxed application and the sandbox reachable service, and

optimize a polling period between polling to be less than a timeout period of a session through the relay service.

45. The system of claim 39, wherein the client device is configured to:

initiate the relay service through at least one of a series of web pages where information is communicated using hyperlinks that point at the pairing server, and a form having a confirmation dialog that is submitted back to the pairing server,

wherein a global unique identifier is masked through the pairing server when the confirmation dialog is served from the pairing server.

46. The system of claim 37, wherein the networked device is configured to at least one of:

announce the sandbox reachable service to a discovery module using a processor and memory,

announce an availability of the sandbox reachable service across a range of public addresses such that the sandboxed application communicates with the sandbox reachable service in any one of the range of the public addresses, and

communicate at least one of a global unique identifier and an alphanumeric name to the pairing server along with at least one of a hardware address associated with the networked device, a public address pair associated with the sandbox reachable service of the networked device, and a private address pair associated with the sandbox reachable service of the networked device,

wherein the private address pair includes a private IP address and a port number associated with the sandbox reachable service.

47. The system of claim 36:

wherein the sandboxed application is at least one of a web page, a script, a binary executable, an intermediate bytecode, an abstract syntax tree, and an executable application in the security sandbox,

wherein the sandboxed application comprises at least one of a HyperText Markup Language 5 (HTML5) application, a Javascript®

application, an Adobe®

Flash®

application, a Microsoft®

Silverlight®

application, a JQuery®

application, an Asynchronous Javascript® and

an XML (AJAX) application, and

wherein an access control algorithm governs a policy through which a secondary authentication is required when establishing the communication session between the sandboxed application and the networked device.

48. The system of claim 47, wherein the client device is configured to:

utilize an exception to a same origin policy through a use of at least one of a hyperlink, a form, the script, a frame, a header, and an image when establishing the communication session between the sandboxed application and the sandbox reachable service.

49. The system of claim 36, wherein the client device is further configured to:

enable elimination of a communication through a centralized infrastructure when the sandboxed application and the sandbox reachable service communicate in a shared network common to the client device and the networked device when the communication session is established, the shared network being at least one of a local area network, a multicast network, an anycast network, and a multilan network,

minimize a latency in the communication session when the sandboxed application and the sandbox reachable service communicate in the shared network common to the client device and the networked device when the communication session is established, and

improve privacy in the communication session when the sandboxed application and the sandbox reachable service communicate in the shared network common to the client device and the networked device when the communication session is established.

50. The system of claim 35, wherein at least one of:

the at least one of;

the networked device and the client device enables, through the capture infrastructure, the annotation of the audio-visual data with at least one of a brand name and a product name by comparing entries in a master database with at least one of a closed captioning data of the audio-visual data and through an application of an optical character recognition algorithm in the audio-visual data,

the content identifier is at least one of a music identification, an object identification, a facial identification, and a voice identification,

a minimal functionality comprising accessing at least one of a tuner and a stream decoder that identifies at least one of a channel and a content is found in the networked device,

the networked device produces at least one of an audio fingerprint and a video fingerprint that is communicated with the capture infrastructure,

the capture infrastructure compares the at least one of the audio fingerprint and the video fingerprint with the master database,

the capture infrastructure annotates the audio-visual data with a logo name by comparing the entries in the master database with a logo data of the audio-visual data identified using a logo detection algorithm,

the capture infrastructure automatically divides the audio-visual data into a series of scenes based on a semantic grouping of actions in the audio-visual data,

the audio-visual data is analyzed in advance of a broadcast to determine content identifiers associated with each commercial in the audio-visual data such that advertisements are pre-inserted into the audio-visual data prior to the broadcast,

the capture infrastructure applies a time-order algorithm to automatically match the advertisements to the audio-visual data when a correlation pattern is identified by the capture infrastructure with other audio-visual content previously analyzed,

the capture infrastructure includes a buffer that is saved to a persistent storage and for which a label is generated to facilitate identification of reoccurring sequences,

a post processing operation is at least one of automated through a post-processing algorithm and a crowd-sourced operation using a plurality of users in which a turing test is applied to determine a veracity of an input,

a device pairing algorithm is used in which a cookie data associated with a web page visited by a user stored on a browser on the client device is paired with the networked device when the client device is communicatively coupled with the networked device,

a transitive public IP matching algorithm is utilized in which the at least one of the client device and the networked device communicates each public IP address with any paired entity to the capture infrastructure,

a tag that is unconstrained from a same-origin policy is used to automatically load an advertisement in the browser, and

the tag is at least one of an image tag, a frame, an iframe, and a script tag.

51. The system of claim 35:

wherein a cookie associated with the security sandbox is used to store a remote access token on a storage of the client device,

wherein the remote access token identifies at least one of a set of communicable private IP addresses and hardware addresses associated with sandbox reachable services that previously operated on a common shared network with the client device, and

wherein the client device can communicate with the sandbox reachable services that previously operated on the common shared network through the remote access token.