Ambient noise based augmentation of media playback

• US 10,073,670 B2
• Filed: 03/05/2015
• Issued: 09/11/2018
• Est. Priority Date: 12/08/2008
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Kraut, Joel
• Primary Examiner(s): Ell, Matthew
• Application Number:
  US14/639,849
• Publication Number:
  US 20150178040A1
• Time in Patent Office: 1,286 Days
• Field of Search: 715768
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/162 Interface to dedicated audio devices, e.g. audio drivers, interface to CODECs
• CPC Class Codes: H03G 3/02 Manually-operated control H03G3/001 and H03G3/002 take precedence
• CPC Class Codes: H04M 1/72442 for playing music files
• CPC Class Codes: H04M 1/72478 for hearing-impaired users
• CPC Class Codes: H04N 21/4126 The peripheral being portable, e.g. PDAs or mobile phones
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/42202 environmental sensors, e.g. for detecting temperature, luminosity, pressure, earthquakes
• CPC Class Codes: H04N 21/42203 sound input device, e.g. microphone
• CPC Class Codes: H04N 21/42224 Touch pad or touch panel provided on the remote control
• CPC Class Codes: H04N 21/439 Processing of audio elementary streams
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/4396 by muting the audio signal
• CPC Class Codes: H04N 21/4884 for displaying subtitles
• CPC Class Codes: H04N 5/60 for the sound signals
• CPC Class Codes: H04W 4/025 using location based information parameters

Abstract

Ambient noise sampled by a mobile device from a local environment is used to automatically trigger actions associated with content currently playing on the mobile device. In some implementations, subtitles or closed captions associated with the currently playing content are automatically invoked and displayed on a user interface based on a level of ambient noise. In some implementations, audio associated with the currently playing content is adjusted or muted. Actions can be automatically triggered based on a comparison of the sampled ambient noise, or an audio fingerprint of the sampled ambient noise, with reference data, such as a reference volume level or a reference audio fingerprint. In some implementations, a reference volume level can be learned on the mobile device based on ambient noise samples.

Claims

1. A method performed by a mobile device comprising:

playing audio content by the mobile device;

sampling ambient noise by the mobile device from a local environment of the mobile device to obtain a sampled ambient noise signal, the ambient noise interfering with the played audio content;

generating, based on the sampled ambient noise signal, an audio fingerprint indicating characteristic(s) of the sampled ambient noise signal;

comparing the audio fingerprint to another previously-registered noise sample;

when a match is identified between the audio fingerprint and the previously-registered noise sample, altering a graphical user interface of the mobile device; and

performing a function of the mobile device in response to input to the graphical user interface.

2. The method of claim 1, wherein the altering the graphical user interface of the mobile device comprises:

overlaying a partially transparent sheet on the graphical user interface, the sheet including one or more options associated with the currently playing audio content.

3. The method of claim 2, further comprising:

overlaying one or more controls on the graphical user interface which are operable through touch input to control the audio content.

4. The method of claim 1, wherein the performing a function of the mobile device comprises:

displaying subtitles associated with the currently playing audio content on the graphical user interface.

5. The method of claim 1, wherein the performing a function of the mobile device comprises:

displaying closed captions associated with the currently playing audio content.

6. The method of claim 1, wherein the performing a function of the mobile device comprises:

pausing the currently playing audio content.

7. The method of claim 1, wherein the performing a function of the mobile device comprises:

adjusting a volume level of the audio content.

8. The method of claim 1, wherein the performing a function of the mobile device comprises:

muting the audio content.

9. The method of claim 1, wherein the function of the mobile device is related to the currently playing audio content.

10. A non-transitory computer-readable medium having instructions stored thereon, which, when executed by a processor, causes the processor to perform operations comprising:

playing audio content by a mobile device;

sampling ambient noise by the mobile device from a local environment of the mobile device to obtain a sampled ambient noise signal, the ambient noise interfering with the audio content;

generating, based on the sampled ambient noise signal, an audio fingerprint indicating characteristic(s) of the sampled ambient noise signal;

comparing the audio fingerprint to another previously-registered ambient noise audio fingerprint; and

when a match is identified between the audio fingerprint and the previously-registered ambient noise audio fingerprint, altering the playing of the audio content to play the audio content in an augmented format.

11. The computer-readable medium of claim 10, wherein the augmented format comprises:

overlaying a partially transparent sheet on a graphical user interface, the sheet including one or more options associated with the currently playing audio content.

12. The computer-readable medium method of claim 11, further comprising:

overlaying one or more controls on the graphical user interface which are operable through touch input to control the content.

13. The computer-readable medium of claim 10, wherein the augmented format comprises:

displaying subtitles associated with the currently playing audio content on a graphical user interface.

14. The computer-readable medium of claim 10, wherein the augmented format comprises:

displaying closed captions associated with the currently playing audio content.

15. The computer-readable medium of claim 10, wherein the augmented format comprises:

pausing the currently playing audio content.

16. The computer-readable medium of claim 10, wherein the augmented format comprises:

adjusting a volume level of the audio content.

17. The computer-readable medium method of claim 10, wherein the augmented format comprises:

muting the audio content.

18. The computer-readable medium of claim 10, where the operations further comprise:

obtaining input from one or more controls or sensors of the mobile device; and

determining the altering to be performed on the playing of the audio content based on the sampled ambient noise signal and the input.

19. A method performed by a mobile device comprising:

playing audio content by the mobile device;

estimating location of the mobile device during the playing via a global positioning system function of the mobile device;

comparing the estimated location to other locations previously stored at which noise events were detected that interfered with other playing events, wherein the other locations were determined by a global positioning function of at least one of the mobile device or another mobile device;

when the estimated location matches a previously stored location, altering the playing of the audio content to play the audio content in an augmented format.

20. The method of claim 19, wherein the augmented format comprises:

overlaying a partially transparent sheet on a user interface of the mobile device, the sheet including one or more options associated with the currently playing audio content.

21. The method of claim 20, wherein the augmented format comprises:

overlaying one or more controls on a user interface of the mobile device which are operable through touch input to control the audio content.

22. The method of claim 19, wherein the augmented format comprises:

displaying subtitles associated with the currently playing audio content on a user interface of the mobile device.

23. The method of claim 19, wherein the augmented format comprises:

displaying closed captions associated with the currently playing audio content.

24. The method of claim 19, wherein the augmented format comprises:

pausing the currently playing audio content.

25. The method of claim 19, wherein the augmented format comprises:

adjusting the volume level of the audio content.

26. The method of claim 19, wherein the augmented format comprises:

muting the audio content.

27. A method performed by a mobile device comprising:

playing media content by the mobile device;

sampling ambient noise by the mobile device from a local environment of the mobile device to obtain a sampled ambient noise signal, the ambient noise interfering with the played media content;

generating, based on the sampled ambient noise signal, an audio fingerprint indicating characteristic(s) of the sampled ambient noise signal;

comparing the audio fingerprint to another previously-registered noise sample;

when a match is identified between the audio fingerprint and the previously-registered noise sample, altering a graphical user interface of the mobile device; and

performing a function of the mobile device in response to input to the graphical user interface.