System and method for predictive generation of visual sequences

• US 10,319,395 B2
• Filed: 03/13/2017
• Issued: 06/11/2019
• Est. Priority Date: 03/11/2016
• Status: Active Grant

• Current Assignee: Limbic Media Corporation
• Original Assignee: Limbic Media Corporation
• Inventors: Love, Justin Thomas, Benning, Manjinder Singh, Elliot, Adam, Leben, Jakob, Odowichuk, Gabrielle
• Primary Examiner(s): Tsang, Fan S
• Assistant Examiner(s): Siegel, David
• Application Number:
  US15/457,758
• Publication Number:
  US 20180061438A1
• Time in Patent Office: 820 Days
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 21/10 Transforming into visible information
• CPC Class Codes: G10L 25/18 the extracted parameters being spectral information of each sub-band
• CPC Class Codes: G10L 25/51 for comparison or discrimination
• CPC Class Codes: G10L 25/90 Pitch determination of speech signals
• CPC Class Codes: H05B 47/12 by detecting audible sound
• CPC Class Codes: H05B 47/155 Coordinated control of two or more light sources
• CPC Class Codes: Y02B 20/40 Control techniques providing energy savings, e.g. smart controller or presence detection

Abstract

A system and method for predictively generating visual experiences based on streaming audio is disclosed. More specifically, the present invention is directed to systems and apparatus for analyzing streaming audio and predictively mapping the information in the stream to a sequence of visual patterns generated by a lighting system in a manner that induces a perceptual association between the streaming audio and visual patterns.

Claims

1. A method for controlling one or more light sources, comprising:

receiving, in a buffer, a streaming audio signal;

receiving, from the buffer, a buffer contents signal representing at least a portion of the streaming audio signal in the buffer;

analyzing the buffer contents signal;

generating a lighting control signal based on said analysis;

outputting a real-time audio signal from the buffer; and

outputting the lighting control signal in synchronization with the real-time audio signal;

wherein;

the real-time audio signal comprises an impulsive sound feature having a first duration; and

said generating comprises generating a corresponding feature in the lighting control signal having a second duration that is longer than the first duration, thereby adapting a part of the lighting control signal corresponding to the impulsive sound feature to a human'"'"'s visual response time.

2. The method of claim 1, wherein the second duration is three orders of magnitude longer than the first duration.

3. The method of claim 1, wherein said analysis determines whether the streaming audio signal includes an instance of a human voice.

4. The method of claim 1, wherein said analysis determines whether the streaming audio signal includes music.

5. The method of claim 1, further comprising storing pre-analyzed audio signals and their corresponding lighting control signals in a database, wherein the analyzing step comprises:

identifying, in the database, a match for said portion of the streaming audio signal;

retrieving, from the database, the corresponding lighting control signal for said match; and

basing the output lighting control signal on the retrieved corresponding lighting control signal.

6. The method of claim 1, further comprising:

storing previously analyzed audio signals and their corresponding lighting control signals in a database, in a format representing a light show; and

retrieving the stored signals to perform the light show.

7. The method of claim 1, wherein said analysis comprises:

detection and classification of music genre or music pattern based on one or more initial bars of a track in the streaming audio signal; and

retrieval of a lighting control signal corresponding to the classification from a database.

8. The method of claim 1, further comprising:

analyzing an upcoming portion of a track, a whole track, or a playlist in the buffer contents signal;

planning a light show based on said upcoming portion of a track, a whole track, or a playlist; and

maintaining continuity of the light show between successive tracks, based on said upcoming portion of a track, a whole track, or a playlist.

9. The method of claim 1, further comprising:

analyzing a past portion of a track or whole track received in the buffer; and

maintaining continuity of a light show between successive tracks, based on said past portion of a track or whole track received in the buffer.

10. The method of claim 1, further comprising:

detecting a frequency range of the buffer contents signal; and

dividing the buffer contents signal into sub-octave bins spanning the frequency range;

wherein generating the lighting control signal is based on a content of each said bin.

11. The method of claim 1, wherein the lighting control signal includes a specification of color and a specification of brightness for at least one light source.

12. The method of claim 1, wherein the lighting control signal includes a specification of color and a specification of brightness for each of a plurality of light sources based on a physical location of each of the light sources.

13. The method of claim 1, wherein said lighting control signal is based on one or more of:

frequency analysis;

detection and classification of a percussive sound;

detection of a melodic sound;

strength of a melodic sound;

a further received signal;

a musical key;

rhythmic dissonance;

harmonic dissonance;

tempo detection;

beat detection;

beat strength estimation;

song transition;

inter-song transition;

a non-musical event;

artificial intelligence;

an interaction of multiple intelligent agents;

an introduction of random colors;

meta data of the streaming audio signal;

an audience response;

a sound envelope;

maximizing a response of a lighting display system in relation to the real-time audio signal;

varying a frequency range of a sub-octave bin used in said analysis;

detection and classification of music genre or music pattern;

detection and classification of speech style or pattern;

a linear mapping of a sound feature in the streamed audio signal to the lighting control signal; and

a non-linear mapping of a sound feature in the streamed audio signal to the lighting control signal.

14. A system for controlling one or more light sources, comprising:

a buffer configured to;

receive a streaming audio signal; and

output a real-time audio signal; and

a predictive mapping unit comprising a processor and a computer readable memory storing computer readable instructions which, when executed by the processor, cause the predictive mapping unit to;

receive, from the buffer, a buffer contents signal representing at least a portion of the streaming audio signal in the buffer;

analyze the buffer contents signal;

generate a lighting control signal based on said analysis; and

output the lighting control signal in synchronization with the real-time audio signal;

wherein;

the real-time audio signal comprises an impulsive sound feature having a first duration; and

said generating comprises generating a corresponding feature in the lighting control signal having a second duration that is longer than the first duration, thereby adapting a part of the lighting control signal corresponding to the impulsive sound feature to a human'"'"'s visual response time.

15. The system of claim 14, wherein the predictive mapping unit further comprises:

a frequency analyzer, defined by the computer readable instructions, that receives the buffer contents signal and outputs a frequency domain signal; and

a binning module, defined by the computer readable instructions, that receives the frequency domain signal and outputs a plurality of sub-octave bin signals.

16. The system of claim 15, wherein the predictive mapping unit further comprises:

a temporal filter that outputs an envelope analysis signal;

a feature extractor, defined by the computer readable instructions, that is configured to detect one or more of genre;

musical key;

rhythmic dissonance;

harmonic dissonance;

tempo;

beat;

beat strength;

song transition;

inter-song transition;

melody;

melody strength; and

percussive elements; and

to output a corresponding feature signal; and

a non-music detector, defined by the computer readable instructions, configured to detect one or more of speech;

a whistle;

a yell;

clapping;

singing;

wind; and

rain; and

to output a non-musical event signal.

17. The system of claim 14, wherein the predictive mapping unit further comprises a color space mapper, defined by the computer readable instructions, configured to map the lighting control signal through a subset of color space that is delimited by chromaticities and maximum spectral radiant intensities of one or more light sources to which the lighting control signal is output.