Social and Interactive Applications for Mass Media

• US 20070130580A1
• Filed: 11/27/2006
• Published: 06/07/2007
• Est. Priority Date: 11/29/2005
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Baluja, Shumeet, Fink, Michael, Covell, Michele
• Granted Patent: US 8,479,225 B2
• Time in Patent Office: Days
• US Class Current: 725/18
• CPC Class Codes: G06F 16/635 Filtering based on additional data, e.g. user or group profiles
• CPC Class Codes: G06F 16/683 using metadata automatically derived from the content
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: H04H 20/14 for monitoring programmes
• CPC Class Codes: H04H 2201/90 characterised by the use of signatures
• CPC Class Codes: H04H 60/06 Arrangements for scheduling broadcast services or broadcast-related services
• CPC Class Codes: H04H 60/37 for identifying segments of broadcast information, e.g. scenes or extracting programme ID
• CPC Class Codes: H04H 60/372 Programme
• CPC Class Codes: H04H 60/375 Commercial
• CPC Class Codes: H04H 60/45 for identifying users
• CPC Class Codes: H04H 60/58 of audio determination or detection of speech characteristics in general G10L25/00; speech recognition in general G10L15/00
• CPC Class Codes: H04H 60/59 of video recognising characters or patterns in general G06F18/00, G06V20/00
• CPC Class Codes: H04H 60/64 for providing detail information
• CPC Class Codes: H04H 60/65 for using the result on users' side
• CPC Class Codes: H04H 60/66 for using the result on distributors' side
• CPC Class Codes: H04H 60/80 characterised by transmission among terminal devices
• CPC Class Codes: H04H 60/87 accessed over computer networks
• CPC Class Codes: H04N 21/252 Processing of multiple end-users' preferences to derive collaborative data
• CPC Class Codes: H04N 21/26603 : for automatically generating descriptors from content, e.g. when it is not made available by its provider, using content analysis techniques
• CPC Class Codes: H04N 21/41407 : embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/4143 : embedded in a Personal Computer [PC]
• CPC Class Codes: H04N 21/42203 : sound input device, e.g. microphone
• CPC Class Codes: H04N 21/4394 : involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/44016 : involving splicing one content stream with another content stream, e.g. for substituting a video clip
• CPC Class Codes: H04N 21/440236 : by media transcoding, e.g. video is transformed into a slideshow of still pictures, audio is converted into text
• CPC Class Codes: H04N 21/44213 : Monitoring of end-user related data arrangements for monitoring the users' behaviour or opinions in broadcast systems H04H60/33
• CPC Class Codes: H04N 21/4542 : Blocking scenes or portions of the received content, e.g. censoring scenes
• CPC Class Codes: H04N 21/4622 : Retrieving content or additional data from different sources, e.g. from a broadcast channel and the Internet web site content organization and management for information retrieval from the Internet G06F16/958; transmission by internet of broadcast information H04H60/82; stock exchange data over packet-switching network H04L12/1804; push services including data channel over packet-switching network H04L12/1859
• CPC Class Codes: H04N 21/4722 : for requesting additional data associated with the content
• CPC Class Codes: H04N 21/4756 : for rating content, e.g. scoring a recommended movie
• CPC Class Codes: H04N 21/4782 : Web browsing , e.g. WebTV
• CPC Class Codes: H04N 21/4788 : communicating with other users, e.g. chatting arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104
• CPC Class Codes: H04N 21/4828 : for searching program descriptors retrieval of video data G06F16/739
• CPC Class Codes: H04N 21/8106 : involving special audio data, e.g. different tracks for different languages
• CPC Class Codes: H04N 21/812 : involving advertisement data advertising per se G06Q30/02
• CPC Class Codes: H04N 21/8133 : specifically related to the content, e.g. biography of the actors in a movie, detailed information about an article seen in a video program
• CPC Class Codes: H04N 21/858 : Linking data to content, e.g. by linking an URL to a video object, by creating a hotspot
• CPC Class Codes: H04N 5/445 : for displaying additional information H04N5/50 takes precedence
• CPC Class Codes: H04N 7/17318 : Direct or substantially direct transmission and handling of requests
• CPC Class Codes: H04N 7/17336 : Handling of requests in head-ends

Abstract

Systems, methods, apparatuses, user interfaces and computer program products provide social and interactive applications for mass media based on real-time ambient-audio and/or video identification.

Claims

1. A method comprising:

receiving a descriptor identifying ambient audio associated with a media broadcast;

comparing the descriptor to reference descriptors associated with the media broadcast; and

aggregating personalized information related to the media broadcast based on the result of the comparison.

2. The method of claim 1, where comparing the descriptor to reference descriptors further comprises:

querying a database of reference descriptors using the received descriptor; and

determining one or more reference descriptors that match the received descriptor based on matching criteria.

3. The method of claim 2, where determining one or more reference descriptors further comprises:

determining a set of candidate reference descriptors based on the matching criteria; and

validating the set of candidate references descriptors using a validation procedure.

4. The method of claim 3, where determining a set of candidate reference descriptors further comprises:

scoring the reference descriptors based on temporal consistency with the received descriptor; and

determining the set of candidate reference descriptors from the scores.

5. The method of claim 1, where

receiving the reference descriptors from a database of reference descriptors.

6. The method of claim 1, where aggregating personalized information further comprises:

providing a communication link with a commenting medium.

7. A method comprising:

receiving a first descriptor identifying ambient audio associated with a first media broadcast;

receiving a second descriptor identifying ambient audio associated with a second media broadcast;

comparing the first and second descriptors to determine if the first and second media broadcasts are the same; and

aggregating personalized information based on the result of the comparison.

8. A method comprising:

detecting ambient audio associated with a media broadcast;

generating descriptors identifying the media broadcast;

transmitting the descriptors to a network resource; and

receiving aggregated personalized information from the network resource application server based on the descriptors.

9. The method of claim 8, where generating descriptors further comprises:

recording snippets of ambient audio;

decomposing the ambient audio snippets into overlapping frames; and

converting the frames into descriptors identifying statistical summaries of the ambient audio snippets.

10. The method of claim 8, further comprising:

training the descriptors to overcome noise.

11. A system comprising:

a database of reference descriptors; and

a server operatively coupled to the database and to a client system, the database server configurable to receive a descriptor from the client system for identifying ambient audio associated with a media broadcast, comparing the received descriptor with one or more reference descriptors, and aggregating personalized information related to the media broadcast based on the result of the comparison.

12. The system of claim 11, where the received descriptor is generated from ambient audio samples.

13. The system of claim 11, where the received descriptor is a compressed file containing one or more audio signature components.

14. The system of claim 11, where the database server receives a client system identifier from the client system for identifying the client system.

15. The system of claim 11, where the personalized information includes information associated with a commenting medium.

16. The system of claim 11, where the personalized information includes information associated with advertising.

17. The system of claim 11, where the reference descriptors are periodically updated from recent media broadcasts.

18. The system of claim 11, where the personalized information is served to the client system in a web page.

19. The system of claim 11, where the client system includes a display device for enabling a user to interact with a mass personalization application.

20. The system of claim 11, where the client system monitors and records ambient audio at selectable periods of time.

21. The system of claim 11, where the received descriptor is encoded so that the ambient audio signal cannot be recovered.

22. The system of claim 11, where the received descriptor is encrypted.

23. The system of claim 11, where the received descriptor is sent to the database server as a query submission in response to a trigger event at the client system.

24. The system of claim 11, where the received descriptor is sent to the audio database server as part of a streaming process.

25. The system of claim 11, where the reference descriptors are generated by one or more client systems.

26. The system of claim 11, where the personalized information includes information for establishing a communication link between users in the same geographic location.

27. The system of claim 11, where the match is determined using locality sensitive hashing.

28. The system of claim 11, where the received descriptor represents an identifying statistical summary of an audio sample.

29. The system of claim 11, where the match is determined using reference descriptors that are temporally consistent with the descriptor generated by the client system.

30. A system comprising:

an audio detector configurable for sampling ambient audio; and

a client interface operatively coupled to the audio detector and configurable to generate descriptors identifying a media broadcast, the client interface configurable for transmitting the descriptors to a network resource, and for receiving aggregated personalized information from the network resource based on the descriptors.

31. The system of claim 30, further comprising:

a display for enabling a user to interact with a mass personalization application.

32. The system of claim 30, where the ambient audio is sampled at selectable periods of time.

33. The system of claim 30, where ambient audio sampling begins when a service is invoked.

34. The system of claim 30, where at least some of the descriptors are encoded so that the ambient audio signal cannot be recovered.

35. The system of claim 30, where at least some of the descriptors are encrypted.

36. The system of claim 30, where at least some of the descriptors are sent to an audio database server as a query submission in response to a trigger event.

37. The system of claim 36, where the descriptors are sent to the audio database server as part of a streaming process.

38. A system, comprising:

means for receiving a descriptor identifying ambient audio associated with a media broadcast;

means for comparing the descriptor to reference descriptors associated with the media broadcast; and

means for aggregating personalized information related to the media broadcast based on the result of the comparison.

39. A computer-readable medium having instructions stored thereon, which, when executed by a processor, causes the processor to perform the operations of:

receiving a descriptor identifying ambient audio associated with a media broadcast;

comparing the descriptor to reference descriptors associated with the media broadcast; and

aggregating personalized information related to the media broadcast based on the result of the comparison.

40. A computer-readable medium having instructions stored thereon, which, when executed by a processor, causes the processor to perform the operations of:

detecting ambient audio associated with a media broadcast;

generating descriptors identifying the media broadcast;

transmitting the descriptors to a network resource; and

receiving aggregated personalized information from the network resource based on the descriptors.

41. A method comprising:

receiving descriptors from multiple client systems, the descriptors identifying ambient audio associated with a real time media broadcast;

comparing the descriptors with reference descriptors to determine positive matches, where positive matches are determined based at least in part on temporal consistency between the received descriptors and the reference descriptors;

creating a social community based on the positive matches; and

transmitting information relating to the social community to the client systems.

42. The method of claim 41, where the transmitted information includes a communication link for providing a communication channel between the client systems and the social community.

43. The method of claim 41, where the descriptors are generated using wavelets.