METHODS, APPARATUS AND SYSTEMS FOR INDIVIDUALIZING AUDIO, MUSIC AND SPEECH ADAPTIVELY, INTELLIGENTLY AND INTERACTIVELY

• US 20140169572A1
• Filed: 12/19/2012
• Published: 06/19/2014
• Est. Priority Date: 12/19/2012
• Status: Active Grant

• Current Assignee: Duo Zhang
• Original Assignee: Hung Tran
• Inventors: TRAN, HUNG
• Granted Patent: US 9,055,362 B2
• Time in Patent Office: Days
• US Class Current: 381/56
• CPC Class Codes: H04R 1/10 Earpieces; Attachments therefor ; Earphones; Monophonic headphones H04R1/28 takes precedence; stereophonic headphones H04R5/033
• CPC Class Codes: H04R 3/04 for correcting frequency response
• CPC Class Codes: H04R 5/033 Headphones for stereophonic communication details thereof, e.g. relating to batteries, cables or control elements H04R1/10

Abstract

The present invention discloses methods, apparatus and systems for individualizing music, audio and speech adaptively, intelligently and interactively according to a listener'"'"'s personal hearing ability, unique hearing preference, characteristic feedback, and real-time surrounding environment.

Claims

1. A sound individualizing system, comprising the steps of:

(a) adjusting a sound by an automatic fluctuation control unit;

(b) performing operations of windowing through a windowing unit and padding a plurality of zeros by a zero padding unit;

(c) transforming a first output of said zero padding unit into a plurality of time-frequency bins by a forward transform unit;

(d) passing said time-frequency bins through a cepstrum calculation unit to output a cepstrum;

(e) processing said cepstrum by at least one cepstrum-domain lifter;

(f) conveying a second output of said lifter into an adaptive classification unit;

(g) directing a third output of said forward transform unit to a weighted fusion unit, wherein said weighted fusion unit merges said adjacent time-frequency bins according to human auditory scaling;

(h) employing a fourth output of said weighted fusion unit by a long-term moment calculation unit, wherein said long-term moment calculation unit computes a plurality of long-term variance, skewness, kurtosis and higher-order moments;

(i) conveying said fourth output of said weighted fusion unit to a short-term moment calculation unit, wherein said short-term moment calculation unit computes a plurality of short-term variance, skewness, kurtosis and higher-order moments;

(j) directing said long-term and short-term variances, skewnesses, kurtosises and higher-order moments to said adaptive classification unit;

(k) passing said fourth output of said weighted fusion unit to a multi-block weighted averaging unit, wherein said multi-block weighted averaging unit suppresses a plurality of undesired components;

(l) calculating a fifth output and a sixth output, wherein said fifth output is a long-term mean value and said sixth output is a short-term mean value;

(m) sending said long-term and short-term mean values to said adaptive classification unit, wherein said adaptive classification unit utilizes said cepstrum vector, said long-term and short-term mean values, variances, skewnesses, kurtosises and higher-order moments to classify said sound into a beat category and a non-beat category;

(n) converting said beat category and said non-beat category to a beat signal;

(o) updating said automatic fluctuation control unit, said windowing unit, and said weighting coefficients, wherein said updated weighting coefficients control said multi-block weighted averaging unit to compute said long-term and short-term mean values; and

(p) employing said beat signal to enhance auditory perception of an individual listener by an individualized auditory enhancer in accordance to a human input unit.

2. A sound individualizing system according to claim 1, wherein said beat signal drives an individualized multimodal enhancer, and said individualized multimodal enhancer activates at least one tactile actuator.

3. A sound individualizing system, comprising the steps of:

(a) generating a first sound in a stimulus generating unit, wherein a sound delivering unit delivers said first sound;

(b) extracting a second sound through a sound sensing unit, wherein a sound analyzing unit processes said second sound to output a plurality of time-frequency characteristics;

(c) controlling a stimulus searching unit according to said time-frequency characteristics, wherein said stimulus searching unit determines a plurality of stimulus properties;

(d) processing said stimulus parameters in said stimulus generating unit to update said first sound and adapt to said sound analyzing unit;

(e) switching to a first channel in a channel selecting unit, wherein said sound analyzing unit controls switching, and said first channel is employed by said sound delivering unit in generating said first sound;

(f) sending a first control signal from a mode selecting unit to said sound analyzing unit, wherein said mode selecting unit determines operation of said sound analyzing unit, and said sound sensing unit detects a first output of an incoming sound and modifies a second output of said mode selecting unit; and

(g) extracting a user input through a human interface unit, wherein said human interface unit determines to latch a third output to a choice storing unit from a plurality of choice units, and said human interface unit controls said mode selecting unit through a user adjustment unit.

4. A sound individualizing system according to claim 3, wherein said first channel are substituted by at least two channels.

5. A sound individualizing system, comprising the steps of:

(a) sending a sound input to an input analyzing unit for adapting to quality and amplitude of said sound input;

(b) processing a first output of said input analyzing unit through a direct current removing unit to remove direct current components;

(c) delivering a second output of said direct current removing unit to a multiplexing unit to pre-process multi-dimensional properties of said sound input for a first forward transform unit;

(d) applying a windowing unit to conduct a window function to a third output of said multiplexing unit;

(e) padding zeros to a fourth output of said windowing unit through a first zero padding unit;

(f) performing a forward transform on a fifth output of said first zero padding unit by said first forward transform unit, whereas said first forward transform unit generates a first transformed stream;

(g) delivering said first transformed stream to a beat sensing unit, wherein said beat sensing extracts a beat signal from said first transformed stream;

(h) sending said beat signal to a visual animation unit, wherein said visual animation unit stimulates individual visual perception;

(i) employing an individual motion sensing unit to detect an individual motion, wherein said individual motion unit stimulates an individual motion conversion unit;

(j) conveying a converted motion waveform from said individual motion conversion unit to said visual animation unit, a spatial data loading unit, an equalization curve searching unit, and a filter shaping unit, wherein said spatial data loading unit loads a transformed frequency response of a spatial impulse response to a channel arranging unit, said equalization curve searching unit searches for an equalization curve for an individual, and said filter shaping unit adjusts a response contour of a function combining unit;

(k) sending a sixth output of a test result converter unit to said function combining unit, wherein said test result converter unit extracts a seventh output of a hearing test unit;

(l) providing a combined stream from said test result converter, said equalization curve searching unit, and said filter shaping unit to a first reverse transform unit, wherein said first reverse transform unit conducts a reverse transform;

(m) delivering an eighth output of said first reverse transform unit to a second zero padding unit, wherein said second zero padding unit pads zeros to said eighth output of said first reverse transform unit;

(n) conveying a second stream combined from said spatial data loading unit, said beat sensing unit, and a second forward transform unit, wherein said second forward transform unit conducts a forward transform on a ninth output of said second zero padding unit;

(o) delivering said second stream to a magnitude and phase manipulating unit, where said magnitude and phase manipulating unit adjusts magnitude and phase of said first stream; and

(p) sending a tenth output of said magnitude and phase manipulating unit to a second reverse transform unit for enhancing auditory perception.

6. A sound individualizing system, comprising the steps of:

(a) sending a sound to a sensory analysis unit for extracting a first stream and classifying said sound;

(b) processing said first stream to a sound combining unit, wherein said sound combining unit maps a plurality of dimensions of said first stream to a plurality of dimensions of a second stream;

(c) providing said second stream to a sound externalization unit, wherein said sound externalization unit filters said second stream to enhance externalization auditory effect;

(d) performing a forward transform to a first output of said sound externalization unit through a forward transform unit;

(e) conveying a spatialization effect to a second output of said forward transform unit through a sound spatialization unit, wherein said sound spatialization unit adjusts spatialization based on said classification of said sensory analysis unit;

(f) obtaining a first control signal from a listener through a human input unit, wherein said human input unit converts said first control signal to a second control signal to said sound externalization unit through a personalization structuring unit;

(g) providing a third control signal to a magnitude and phase manipulating unit to adjust magnitude responses and phase responses of said second output of said forward transform unit through said personalization structuring unit;

(h) delivering a fourth control signal from said personalization structuring unit to a dynamic database unit to extract an individual interaural spatialization response, wherein said individual interaural spatialization response is processed to improve a spatial resolution by a multiple-dimensional interpolation unit; and

(i) conducting a reverse transform to a third output of said sound spatialization unit through a reverse transform unit.

7. A sound individualizing system, comprising the steps of:

(a) sending a sound to an input adapting unit for extracting a first stream;

(b) processing said first stream in a forward transform unit, wherein said forward transform unit performs a forward transform to generate a transformed signal;

(c) delivering said transformed signal to a magnitude and phase manipulating unit, wherein said magnitude and phase manipulating unit adjusts magnitude and phase of said transformed signal;

(d) extracting an individual input in an individual interface unit, wherein said individual interface unit stimulates a time-frequency analysis unit to generate a time-varying and frequency-selective signal for a low frequency effect unit;

(e) sending an adjusting signal from said low frequency effect unit to said magnitude and phase manipulating unit; and

(f) analyzing said first stream in a music analyzing unit to classify said first stream, wherein said low frequency effect unit is controlled according to said classification.

8. A sound individualizing system, comprising the steps of:

(a) extracting a sound through a sound acquiring unit, wherein said sound acquiring unit delivers a first stream;

(b) processing said first stream in an environment analyzing unit and an automatic range adjustment unit, wherein said automatic range adjustment unit is controlled by said environment analyzing unit;

(c) conveying a first output of said automatic range adjustment unit to a re-centering unit, where said re-centering unit is controlled by a fine structure unit to deliver a second stream to a re-scaling unit;

(d) scaling time-domain resolution of said second stream through said re-scaling unit, wherein said re-scaling unit is adjusted by said fine structure unit;

(e) processing a second output of said re-scaling unit in a time-frequency analysis unit, wherein said time-frequency analysis unit analyzes time-variation and frequency-selectivity of said second output and delivers a third output to an individual output unit;

(f) delivering said third output to a sound classifying unit, wherein said sound classifying unit controls said individual output unit;

(g) extracting a human input from a human interface unit, where said human interface unit stimulates said environment analyzing unit, said fine structure unit, a time-frequency distribution unit, a weighting unit;

(h) processing said human input in said time-frequency distribution unit, wherein said time-frequency distribution unit determines transform kernel functions of said time-frequency analysis unit;

(i) employing said weighting unit to control said individual output unit; and

(j) storing an instantaneous status of said individual output unit to a status storing unit, wherein said human interface unit retrieves said instantaneous status from said status storing unit.

9. A sound individualizing system, comprising the steps of:

(a) extracting a sound input from an environment monitoring unit, where said environment monitoring unit stimulates an environment analyzing unit to generate a first stream, a second stream, a third stream, a fourth stream, a fifth stream, a sixth stream and a seventh stream;

(b) arranging sequential order of a plurality of stimulation sounds stored in a sound sequencing unit, wherein said first stream controls said sequential order;

(c) generating a first sound in a sound generating unit, wherein said second stream determines a plurality of characteristics of said first sound;

(d) adjusting bandwidth of said stimulation sounds in a bandwidth adjusting unit, wherein a group delay unit sound receives a first output of said bandwidth adjusting unit, applies phase spectrum that matches a group delay to generate a first signal, and sends said first signal to a sound mixing unit;

(e) mixing said first signal with said first sound to generate a mixed signal according to said third stream;

(f) providing a binaural signal for a binaural strategy unit based on said mixed signal, wherein said fourth stream determines a plurality of characteristics of said binaural signal for a sound manipulating unit;

(g) driving an ear interface unit according to a second output of a human interface unit, wherein said sound manipulating unit delivers a second sound to said ear interface unit;

(h) controlling said sound manipulating unit by said human interface unit, wherein said human interface unit interfaces with an individual listener;

(i) processing said fifth stream in a user-data analyzing unit, wherein said user-data analyzing unit combines a third output of said human interface unit with said fifth stream to generate a confidence level;

(j) sending said confidence level to said confidence level unit for storage;

(k) delivering said sixth stream to a result output unit, wherein said result output unit converts said sixth stream for visual stimulation;

(l) providing an indication to an individual listener through said seventh stream on a plurality of characteristics of time-frequency analysis;

(m) identifying a plurality of functions of a platform through a platform identifying unit, wherein said platform identifying unit transmits said functions to a sound calibrating unit; and

(n) adjusting said sound mixing unit according to a calibration mode unit, wherein said calibration mode unit is changed by said human interface unit.

10. A sound individualizing system, comprising the steps of:

(a) extracting a sound input through an automatic gain control unit, wherein said automatic gain control unit transmits a gain-adjusted signal to a windowing unit;

(b) conducting a forward transform in a forward transform unit, wherein said forward transform unit receives a windowed signal from said windowing unit;

(c) calculating and performing magnitude adjustment in a magnitude manipulating unit, wherein said magnitude manipulating unit sends a magnitude-adjusted signal to a group delay manipulating unit;

(d) remapping a plurality of frequency components from a first output of said group delay manipulating unit, wherein a frequency remapping unit transmits a remapped signal to a reverse transform unit;

(e) extracting a second input of an individual listener through a human interface unit, wherein said human interface unit activates a hearing test unit to collect individual information of said listener'"'"'s hearing, and said magnitude manipulating unit is controlled by said hearing test unit for generating said magnitude-adjusted signal;

(f) analyzing said individual information in a test rating unit, wherein said test rating unit determines a third output of a response optimizing unit to adjust said magnitude manipulating unit and said group delay manipulating unit;

(g) sending said sound input to an environment analyzing unit, wherein said environment analyzing unit calculates to control said automatic gain control unit, and adaptively generates a first control signal to said magnitude manipulating unit, and a second control signal to said group delay manipulating unit;

(h) providing phase compensation in said group delay manipulating unit, wherein said hearing test unit conducts said phase compensation for generating a group delay according to said response optimizing unit;

(i) detecting a type of a hearing device, whereas said type is used as an index to look up a device compensation library;

(j) conducting a magnitude and phase compensation in a response compensation unit, wherein said response compensation unit extracts a fourth output of said device compensation library; and

(k) obtaining a user input through a peripheral selecting unit and a peripheral compensation library, wherein a fifth output of said peripheral compensation library controls said response compensating unit jointly with said device compensation unit.

11. A sound individualizing system, comprising the steps of:

(a) conducting a forward transform of a sound in a forward transform unit, wherein said forward transform unit sends a plurality of channels to a channel selection unit;

(b) providing a first output from said channel selection unit to a magnitude manipulating unit, wherein magnitude of said first output is changed according to a first control signal of an equalization library unit;

(c) sending a second output from said magnitude manipulating unit to a phase manipulating unit, wherein phase of said second output is changed to output a third output according to a second control signal of said equalization library unit;

(d) conducting a reverse transform of said third output in a reverse transform unit;

(e) extracting a user input through a human interface unit, wherein said user input controls a search criterion unit;

(f) sending a fourth output from said search criterion unit to a selection result unit, wherein said selection result unit determines said first and second control signals of said equalization library unit; and

(g) controlling a plurality of user choice units for said human interface unit to select a fifth output, wherein said fifth output is latched into said selection result unit.

12. A sound individualizing system, comprising the steps of:

(a) conducting a forward transform of a sound in a forward transform unit, wherein said forward transform unit sends a first output to a range selection unit;

(b) selecting a plurality of frequency ranges through said range selection unit, wherein a range table unit provides a look-up table for said range selection unit;

(c) conducting analysis of correlation between a plurality of channels through a cross-channel analysis unit, wherein said cross-channel analysis unit delivers a second output to a metric accumulating unit;

(d) computing a metric to quantize said correlation and generate a third output, wherein said third output is stored into a metric optimization unit, and said metric optimization unit is controlled by an iteration control unit;

(e) adjusting said frequency ranges according to a fourth output of said metric optimization unit, wherein said iteration control unit determines storage depth of said third output; and

(f) sending signal components that reside outside said frequency ranges to a reverse transform unit, wherein a fifth output of said reverse transform unit is mixed with a sixth output of a target removing unit, and said target removing unit extracts said sixth output by removing a target component from said sound.