SECOND SCREEN INTERACTIVE PLATFORM

• US 20130111514A1
• Filed: 09/16/2012
• Published: 05/02/2013
• Est. Priority Date: 09/16/2011
• Status: Abandoned Application

• Current Assignee: Umami Company
• Original Assignee: Umami Company
• Inventors: Rosenberg, Scott, Glennon, Aron, SLAVIN, Bryan
• Application Number:
  US13/621,277
• Publication Number:
  US 20130111514A1
• Time in Patent Office: Days
• US Class Current: 725/18
• CPC Class Codes: G06F 16/433 using audio data
• CPC Class Codes: G06F 16/60 of audio data
• CPC Class Codes: H04H 2201/40 characterised in that additional data relating to the broadcast data are available via a different channel than the broadcast channel
• CPC Class Codes: H04H 60/58 of audio determination or detection of speech characteristics in general G10L25/00; speech recognition in general G10L15/00
• CPC Class Codes: H04H 60/64 for providing detail information
• CPC Class Codes: H04N 21/4126 The peripheral being portable, e.g. PDAs or mobile phones
• CPC Class Codes: H04N 21/42203 sound input device, e.g. microphone
• CPC Class Codes: H04N 21/43079 of additional data with content streams on multiple devices
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/442 Monitoring of processes or resources, e.g. detecting the failure of a recording device, monitoring the downstream bandwidth, the number of times a movie has been viewed, the storage space available from the internal hard disk arrangements for monitoring broadcast services or broadcast-related services H04H60/29; arrangements for identifying or recognising characteristics with a direct linkage to broadcast information H04H60/35; monitoring of user activities for profile generation for accessing a video database G06F16/739; monitoring in wireless networks H04W24/00
• CPC Class Codes: H04N 21/4722 for requesting additional data associated with the content
• CPC Class Codes: H04N 21/8126 involving additional data, e.g. news, sports, stocks, weather forecasts

Abstract

Interactive digital media platform, methods and apparatus for detecting and dynamically synchronizing to media content (e.g., television (TV) programs or movies) that a viewer is watching while providing related content on a second screen for enhancing the viewer experience. In one embodiment, the primary content is determined by detecting an audio signal of the primary content via the second screen device; the audio signal may then be processed to generate a fingerprint for comparison with a data store of primary content. The primary content can be classified by various categories (e.g., unique program, advertising, repeat airing, theme song . . . ) and the classification used to aid in the identification and/or in selection of the content to be presented on the interactive second screen device. The system allows a substantially real time comparison and recognition of what primary content a viewer is watching on a first screen device and presentation to the user of content that is substantially synchronous to the viewer'"'"'s location in the primary content. The viewer can actively engaged with the content presented and can share the content with others via social networking and the like.

Claims

1. A second screen interactive content delivery system comprising:

a portable interactive second screen device for use while watching a primary content comprising television programming on a first screen device, the second screen device having an audio analyzer for audibly detecting an audio portion of the currently viewed primary content on the first screen device and the second screen device having an interactive display screen for presenting interactive content contextually related to the detected primary content,

the second screen device including a processor executing a first stored program for communicating with an identification process that determines a match between a detected primary content and a known primary content, wherein the first stored program operates to;

process the detected audio portion and communicate the processed audio portion to the identification process to determine a match that identifies the detected primary content; and

based on that identification, process and present on the display screen an interactive content that is contextually related to the detected primary content.

2. The system of claim 1, wherein the first stored program utilizes a fingerprinting algorithm for processing the detected audio portion.

3. The system of claim 2, wherein the known primary content comprises fingerprints and for each fingerprint an associated television program and a time offset within the program.

4. The system of claim 1, wherein the interactive content is presented as a series of web-based pages synchronized in time with respect to the detected primary content.

5. The system of claim 4, wherein the series of pages are synchronized to time codes in the television programming.

6. The system of claim 4, wherein the series of pages can be scrolled via the interactive display screen.

7. The system of claim 4, wherein individual pages can be selected via the interactive display screen.

8. The system of claim 5, wherein the series of pages comprises a flipbook, organized horizontally, vertically or stacked, and in order of the time codes.

9. The system of claim 4, further including the presentation of one or more asynchronous pages at the beginning and/or end of the series of synchronized pages.

10. The system of claim 5, wherein the first stored program operates to automatically after a designed time period, or in response to a communication from a user selectable option on the display screen, return to a page having a time code closest to but not exceeding a current time.

11. The system of claim 5, wherein each page is displayed only after its associated time code has passed in the primary content being detected.

12. The system of claim 5, wherein the first stored program operates to conceal a page until after its associated time code has passed in the primary content begin detected, and presents a user selectable option on the display screen to reveal the page.

13. The system of claim 1, wherein the first stored program operates to automatically advance through the time synchronized pages presented on the interactive display screen.

14. The system of claim 13, wherein the first stored program halts the automatic advancement in response to an input signal from the display screen indicating a user interaction with the display screen.

15. The system of claim 1, wherein the first stored program communicates as a client with a fingerprinting identification server external to the second screen device for determining a match of a detected primary contact and a stored primary contact.

16. The system of claim 15, wherein the client and server accumulate and share matching information over several request-response transactions to determine a match.

17. The system of claim 16, wherein the server executes a second stored program that sends a cookie to the client with partial match information.

18. The system of claim 17, wherein the first stored program receives the cookie and sends the cookie back to the server along with a subsequently detected audio portion.

19. The system of claim 1, wherein the first stored program communicates with a fingerprinting identification service to search for a match across a data store of known primary content, and once a match is identified, subsequent searches for matches with subsequent detected audio portions are performed within a neighborhood of the identified match.

20. The system of claim 19, wherein the neighborhood is a range of time prior to and after to the matched primary content.

21. The system of claim 1, wherein the second screen device is Internet-enabled for communicating with the identification process and source(s) of the interactive content.

22. The system of claim 1, wherein the second screen device includes a browser process communicating with an external web server for aggregating the interactive content.

23. The system of claim 1, wherein the second screen device includes a web browser, a data store of detected primary content, and an inter-process interface communicating with the browser and data store for processing the interactive content.

24. The system of claim 1, wherein the first stored program on the second screen device includes a fingerprinting generation process communicating with an external web service that stores detected primary content.

25. The system of claim 1, wherein the second screen device includes a browser process having a fingerprinting generation process embedded in the browser process.

26. The system of claim 1, wherein the second screen device includes a fingerprinting process and a browser process embedded in a primary application stored on the second screen device.

27. The system of claim 1, wherein the identification process utilizes metadata of the known primary content and the detected audio portion to determine a match.

28. The system of claim 27, wherein the metadata comprises a characteristic of the known primary content including one or more of:

Unique Program;

Advertising;

Repeat Airing of a Program;

Theme Song;

Silence;

Noise;

Speaking.

29. The system of claim 27, wherein the first stored program utilizes the metadata to determine one or more of:

a program boundary;

an advertisement boundary.

30. The system of claim 1, wherein the first screen device comprises a television, a personal computer, a Smartphone, a portable media player, a cable or satellite set-top box, an Internet-enabled streaming device, a gaming device, or a DVD/blue ray device.

31. The system of claim 1, wherein the second screen device comprises a tablet computer, a Smartphone, a laptop computer, or a portable media player.

32. The system of claim 19, wherein the second screen device continually tracks the detected audio portion and the first stored program presents in substantially real time interactive content which changes as the detected audio portion changes.

33. The system of claim 32, wherein once a match is determined the identification service sends a portion of the data store defined by the neighborhood to the second screen device which portion is then stored on the second screen device for use locally on the second screen device in subsequent identification searches.

34. The system of claim 1, wherein the second screen device includes a user selectable input and the first stored program responds to a communication from the user selectable input to advance through the time synchronized pages.

35. The system of claim 34, wherein the first stored program operates to process communications from the user selectable input including one or more of:

requesting more information;

conducting a search;

viewing advertisements;

scheduling a future event;

contributing to the interactive content;

interacting with other viewers and/or non-viewers having an interest in the primary or interactive content;

social networking associated with the primary or interactive content;

purchasing services or goods.

36. The system of claim 1, wherein the primary content comprises a live broadcast, streaming content, or stored video content.

37. The system of claim 1, wherein the interactive content comprises one or more of a direct connection to a web page, and a link to a web page.

38. A method for substantially real time comparison and recognition of what primary content a viewer is watching on a first screen device comprising:

a. detecting on a portable interactive second screen device an audio signal from a primary video content that a viewer is watching on a first screen device;

b. identifying the primary video content utilizing the detected audio signal or a representation thereof for comparison with a primary content detection signal or representation thereof;

c. based on the identification, presenting content on the second screen device substantially synchronous to the viewer'"'"'s location in the primary content.

39. The method of claim 38, including utilizing metadata of the primary content detection signal or representation thereof in the step of identifying the detected primary content or in a step of selecting the content presented on the second screen device.

40. The method of claim 39, including the step of extracting information from one or more content streams to generate the metadata.

41. The method of claim 40 wherein the streams comprise one or more of video, audio and closed captioning of the primary content.

42. A method of substantially real time sharing of video content a viewer is watching on a first screen device comprising:

a. detecting on a portable interactive second screen device an audio signal from a primary video content that a viewer is watching on a first screen device;

b. identifying the primary video content utilizing the detected audio signal or a representation thereof for comparison with a primary content detection signal or representation thereof;

c. based on the identification, presenting content on the second screen device substantially synchronous to the viewer'"'"'s location in the primary content;

d. the content presented on the second screen device including one or more images or videos from the primary content that are substantially synchronous in time to the viewer'"'"'s location in the primary content and a user selectable input for sharing the content via a social network, email or other communications protocol.

43. The method of claim 42 including

e. storing audio fingerprints of the primary video content in a data store with an associated content identifier and time code that identifies a location in the primary content;

f. storing video and/or images from the primary content in a data store with an associated content identifier and time code that identifies a location in the primary content; and

g. utilizing the audio fingerprints in the identifying step and utilizing the video and/or images that correspond to the time code of the identified audio fingerprint or within a designated time range before and/or after that time code to select the content presented on the second screen.