Event-based speech interactive media player

• US 9,978,366 B2
• Filed: 10/09/2015
• Issued: 05/22/2018
• Est. Priority Date: 10/09/2015
• Status: Active Grant

• Current Assignee: XappMedia, Inc.
• Original Assignee: XappMedia, Inc.
• Inventors: Higbie, Patrick B., Kelvie, John P., Myers, Michael M., Raines, Franklin D.
• Primary Examiner(s): GUERRA-ERAZO, EDGAR X
• Application Number:
  US14/880,084
• Publication Number:
  US 20170103754A1
• Time in Patent Office: 956 Days
• Field of Search: 704235, 704246, 704270, 7042701, 704275, 704277, 704278
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/228 of application context
• CPC Class Codes: G10L 25/78 Detection of presence or absence of voice signals switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10
• CPC Class Codes: H04N 21/23424 involving splicing one content stream with another content stream, e.g. for inserting or substituting an advertisement
• CPC Class Codes: H04N 21/42203 sound input device, e.g. microphone
• CPC Class Codes: H04N 21/44016 involving splicing one content stream with another content stream, e.g. for substituting a video clip
• CPC Class Codes: H04N 21/812 involving advertisement data advertising per se G06Q30/02

Abstract

Interactive content containing audio or video may be provided in conjunction with non-interactive content containing audio or video to enhance user engagement and interest with the contents and to increase the effectiveness of the distributed information. Interactive content may be directly inserted into the existing, non-interactive content. Additionally or alternatively, interactive content may be streamed in parallel to the existing content, with only minimal modification to the existing content. For example, the server may monitor content from a content provider; detect an event (e.g., detecting a marker embedded in the content stream, or in a data source external to the content stream); upon detection of the event, play interactive content at one or more designated times while silencing the content stream of the content provider (e.g., by muting, pausing, playing silence.) The marker may be a sub-audible tone or metadata associated with the content stream. The user may respond to the interactive content by voice, touch or gesture.

Claims

1. A device comprising:

one or more processors;

a memory storing one or more computer programs that include computer instructions, which when executed by the one or more processors, cause the device to;

monitor media content streamed from a content provider to a user, the media content comprising both non-interactive content and interactive content in at least an audio or video format;

detect a first event in the media content streamed to the user, the first event comprising information associated with activating a speech recognizer to detect voice commands spoken by the user;

in response to detecting the first event in the media content streamed to the user, activate the speech recognizer;

while the speech recognizer is activated, receive a voice command spoken by the user and detected by the speech recognizer;

in accordance with a determination that the voice command spoken by the user is an action phrase, cause an action associated with the action phrase to be performed; and

in accordance with a determination that the voice command spoken by the user is not the action phrase, cause the media content to continue to be played.

2. The device of claim 1, wherein the first event in the media content is in the form of metadata.

3. The device of claim 2, wherein the first event in the media content is in the form of at least ID3 metadata or HLS metadata.

4. The device of claim 1, wherein the first event in the media content is a sub-audible tone embedded in the media content, wherein the sub-audible tone is not detectable by an unaided human ear.

5. The device of claim 4, wherein the sub-audible tone has a frequency of 20 Hertz or less.

6. The device of claim 1, wherein the media content is associated with metadata comprising instructions for performing the action according to the action phrase, and the metadata is either supplied from the content provider along with the media content or supplied separately from an external source.

7. The device of claim 1, wherein the media content is silenced during a period for which the speech recognizer is activated, and the media content is silenced by at least one of:

muting a playback mechanism used to play the media content,

pausing the playback mechanism used to play the media content, or

playing silence over the playback mechanism used to play the media content.

8. The device of claim 1, wherein the content provider is at least one of a remote media server, a local file system, and a local memory.

9. The device of claim 1, wherein the computer instructions, which when executed by the one or more processors, further cause the device to:

prior to activating the speech recognizer, cause the interactive content to be played followed by a prompt, the prompt comprising a message instructing the user to say the action phrase, wherein the first event is associated with an end of the prompt such that the speech recognizer is activated after causing the prompt to be played.

10. A method for providing speech interactive content, the method comprising:

monitoring media content streamed from a content provider to a user, the media content comprising both non-interactive content and interactive content in at least an audio or video format;

detecting a first event in the media content streamed to the user, the first event comprising information associated with activating a speech recognizer to detect voice commands spoken by the user;

in response to detecting the first event in the media content streamed to the user, activating the speech recognizer;

while the speech recognizer is activated, receive a voice command spoken by the user and detected by the speech recognizer;

in accordance with a determination that the voice command spoken by the user is an action phrase, cause an action associated with the action phrase to be performed; and

in accordance with a determination that the voice command spoken by the user is not the action phrase, causing the media content to continue to be played.

11. The method of claim 10, wherein the first event in the media content is in the form of metadata.

12. The method of claim 11, wherein the first event in the media content is in the form of at least ID3 metadata or HLS metadata.

13. The method of claim 10, wherein the first event in the media content is a sub-audible tone embedded in the media content, wherein the sub-audible tone is not detectable by an unaided human ear.

14. The method of claim 13, wherein the sub-audible tone has a frequency of 20 Hertz or less.

15. The method of claim 10, wherein the media content is associated with metadata comprising instructions for performing the action according to the action phrase, and the metadata is either supplied from the content provider along with the media content or supplied separately from an external source.

16. The method of claim 10, wherein the media content is silenced during a period for which the speech recognizer is activated, and the media content is silenced by at least one of:

muting a playback mechanism used to play the media content,

pausing the playback mechanism used to play the media content, or

playing silence over the playback mechanism used to play the media content.

17. The method of claim 10, wherein the content provider is at least one of a remote media server, a local file system, and a local memory.

18. The method of claim 10, further comprising:

prior to activating the speech recognizer, cause the interactive content to be played followed by a prompt, the prompt comprising a message instructing the user to say the action phrase, wherein the first event is associated with an end of the prompt such that the speech recognizer is activated after causing the prompt to be played.

19. A non-transitory computer readable medium comprising one or more computer programs, which when executed by a device with one or more processors, cause the device to:

monitor media content streamed from a content provider to a user, the media content comprising both non-interactive content and interactive content in at least an audio or video format;

detect a first event in the media content streamed to the user, the first event comprising information associated with activating a speech recognizer to detect voice commands spoken by the user;

in response to detecting the first event in the media content streamed to the user, activate the speech recognizer;

while the speech recognizer is activated, receive a voice command spoken by the user and detected by the speech recognizer;

in accordance with a determination that the voice command spoken by the user is an action phrase, cause an action associated with the action phrase to be performed; and

in accordance with a determination that the voice command spoken by the user is not the action phrase, cause the media content to continue to be played.

20. The medium of claim 19, wherein the media content is silenced during a period for which the speech recognizer is activated, and the media content is silenced by at least one of:

muting a playback mechanism used to play the media content,

pausing the playback mechanism used to play the media content, or

playing silence over the playback mechanism used to play the media content.

21. A device comprising:

one or more processors;

a memory storing one or more computer programs that include computer instructions, which when executed by the one or more processors, cause the device to;

monitor non-interactive media content in at least an audio or video format that is streamed from a first source to a user;

detect a first event in the non-interactive media content streamed to the user, the first event indicating a start time of a playback of interactive media content and comprising information associated with activating a speech recognizer to detect voice commands spoken by the user;

in response to detecting the first event in the non-interactive media content streamed to the user;

cause the interactive media content in at least an audio or video format to be played by a second source to the user at the indicated start time, and

activate the speech recognizer;

while the speech recognizer is activated, receive a voice command spoken by the user and detected by the speech recognizer;

in accordance with a determination that the voice command spoken by the user is an action phrase associated with the interactive media content, cause an action associated with the action phrase to be performed; and

in accordance with a determination that the voice command spoken by the user is not the action phrase associated with the interactive media content, cause the non-interactive media content to continue to be played.

22. The device of claim 21, wherein the first event in the non-interactive media content is in the form of metadata.

23. The device of claim 22, wherein the first event in the non-interactive media content is in the form of at least ID3 metadata or HLS metadata.

24. The device of claim 21, wherein the first event in the non-interactive media content is a sub-audible tone embedded in the media content, wherein the sub-audible tone is not detectable by an unaided human ear.

25. The device of claim 24, wherein the sub-audible tone has a frequency of 20 Hertz or less.

26. The device of claim 21, wherein the first event is associated with metadata comprising instructions for selecting the interactive media content and the associated action phrase, and the metadata is either supplied from the first source along with the non-interactive media content or supplied separately from an external source.

27. The device of claim 21, wherein the non-interactive media content is silenced during at least a period for which the speech recognizer is activated, and the non-interactive media content is silenced by at least one of:

muting a first playback mechanism of the first source used to play the non-interactive media content,

pausing the first playback mechanism of the first source used to play the non-interactive media content, or

playing silence over the first playback mechanism of the first source used to play the non-interactive media content.

28. The device of claim 21, wherein the interactive media content is silenced during at least a period for which the speech recognizer is activated, and the interactive media content is silenced by at least one of:

muting a second playback mechanism of the second source used to play the interactive media content,

pausing the second playback mechanism of the second source used to play the interactive media content, or

playing silence over the second playback mechanism of the second source used to play the interactive media content.

29. The device of claim 21, wherein the first source and the second source are each at least one of a remote media server, a local file system, and a local memory.

30. The device of claim 21, wherein the computer instructions, which when executed by the one or more processors, further cause the device to:

prior to activating the speech recognizer, cause the interactive media content to be played followed by a prompt, the prompt comprising a message instructing the user to say the action phrase, wherein the second event is associated with an end of the prompt such that the speech recognizer is activated after causing the prompt to be played.

31. A method for providing speech interactive content, the method comprising:

monitoring non-interactive media content in at least an audio or video format that is streamed from a first source to a user;

detecting a first event in the non-interactive media content streamed to the user, the first event indicating a start time of a playback of interactive media content and comprising information associated with activating a speech recognizer to detect voice commands spoken by the user;

in response to detecting the first event in the non-interactive media content streamed to the user;

causing the interactive media content in at least an audio or video format to be played by a second source to the user at the indicated start time, and

activating the speech recognizer;

while the speech recognizer is activated, receiving a voice command spoken by the user and detected by the speech recognizer;

in accordance with a determination that the voice command spoken by the user is an action phrase associated with the interactive media content, cause an action associated with the action phrase to be performed; and

in accordance with a determination that the voice command spoken by the user is not the action phrase associated with the interactive media content, cause the non-interactive media content to continue to be played.

32. The method of claim 31, wherein the first event in the non-interactive media content is in the form of metadata.

33. The method of claim 32, wherein the first event in the non-interactive media content is in the form of at least ID3 metadata or HLS metadata.

34. The method of claim 31, wherein the first event in the non-interactive media content is a sub-audible tone embedded in the media content, wherein the sub-audible tone is not detectable by an unaided human ear.

35. The method of claim 34, wherein the sub-audible tone has a frequency of 20 Hertz or less.

36. The method of claim 31, wherein the first event is associated with metadata comprising instructions for selecting the interactive media content and the associated action phrase, and the metadata is either supplied from the first source along with the non-interactive media content or supplied separately from an external source.

37. The method of claim 31, wherein the non-interactive media content is silenced during at least a period for which the speech recognizer is activated, and the non-interactive media content is silenced by at least one of:

muting a first playback mechanism of the first source used to play the non-interactive media content,

pausing the first playback mechanism of the first source used to play the non-interactive media content, or

playing silence over the first playback mechanism of the first source used to play the non-interactive media content.

38. The method of claim 31, wherein the interactive media content is silenced during at least a period for which the speech recognizer is activated, and the interactive media content is silenced by at least one of:

muting a second playback mechanism of the second source used to play the interactive media content,

pausing the second playback mechanism of the second source used to play the interactive media content, or

playing silence over the second playback mechanism of the second source used to play the interactive media content.

39. The method of claim 31, wherein the first source and the second source are each at least one of a remote media server, a local file system, and a local memory.

40. The method of claim 31, further comprising:

prior to activating the speech recognizer, cause the interactive media content to be played followed by a prompt, the prompt comprising a message instructing the user to say the action phrase, wherein the second event is associated with an end of the prompt such that the speech recognizer is activated after causing the prompt to be played.

41. A non-transitory computer readable medium comprising one or more computer programs, which when executed by a device with one or more processors, cause the device to:

monitor non-interactive media content in at least an audio or video format that is streamed by a first source to a user;

detect a first event in the non-interactive media content streamed to the user, the first event indicating a start time of a playback of interactive media content and comprising information associated with activating a speech recognizer to detect voice commands spoken by the user;

in response to detecting the first event in the non-interactive media content streamed to the user;

cause the interactive media content in at least an audio or video format to be played by a second source to the user at the indicated start time, and

activate the speech recognizer;

while the speech recognizer is activated, receive a voice command spoken by the user and detected by the speech recognizer;

in accordance with a determination that the voice command spoken by the user is an action phrase associated with the interactive media content, cause an action associated with the action phrase to be performed; and

in accordance with a determination that the voice command spoken by the user is not the action phrase associated with the interactive media content, cause the non-interactive media content to continue to be played.