Estimating and Tracking Multiple Attributes of Multiple Objects from Multi-Sensor Data

• US 20160061934A1
• Filed: 03/24/2015
• Published: 03/03/2016
• Est. Priority Date: 03/28/2014
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Audience Corporation
• Inventors: Avendano, Carlos, Goodwin, Michael M., Woodruff, John
• Granted Patent: US 9,500,739 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G01S 3/802 Systems for determining direction or deviation from predetermined direction
• CPC Class Codes: G01S 5/20 Position of source determined by a plurality of spaced direction-finders
• CPC Class Codes: G10L 25/18 the extracted parameters being spectral information of each sub-band
• CPC Class Codes: G10L 25/48 specially adapted for particular use

Abstract

Systems and methods for estimating and tracking multiple attributes of multiple objects from multi-sensor data are provided. An exemplary method includes identifying features associated with sensor data. The sensor data represents data captured by at least one of a plurality of acoustic and non-acoustic sensors. Identification of the features associated with the sensor data may be based variously on detected sounds, motions, images, and the like. The exemplary method further includes determining, in parallel, multiple probable objects based at least in part on the identified features. Various embodiments of the method also include forming hypotheses based at least in part on associating identified features with the multiple probable objects and attributing the formed hypotheses to channels. Sequence of the formed hypotheses are constructed. The exemplary system includes a tracking module configured to provide the channels and constructed sequences for use in various signal processing, such as signal enhancement.

Claims

1. A system for estimating and tracking multiple attributes of multiple objects from multi-sensor data, the system comprising:

a feature module operable to identify features associated with sensor data, the sensor data representing data captured by at least one of a plurality of sensors;

an object detector operable to determine multiple probable objects based at least in part on the identified features, the determination being made in parallel for the identified features;

a hypothesis generator operable to form hypotheses based at least in part on associating identified features with the multiple probable objects;

a grouping module operable to attribute the formed hypotheses to channels; and

a tracking module operable to construct sequences of the formed hypotheses;

the tracking module being configured to provide the channels and constructed sequences.

2. The system of claim 1, wherein the determination is performed simultaneously for the multiple probable objects.

3. The system of claim 1, wherein the sensor data comprises at least one audio signal representing at least one captured sound, at least some of the multiple probable objects are sounds sources, and the plurality of the sensors includes at least two microphones configured to capture the at least one sound.

4. The system of claim 3, further comprising a front end module operable to transform data representing the at least one captured sound to obtain a sequence of frames of the sensor data.

5. The system of claim 4, wherein the transformation includes:

performing an analog-to-digital conversion of the at least one audio signal to generate a digital acoustic signal;

transforming the digital acoustic signal from a time domain to a frequency domain; and

separating the transformed acoustic signal into sub-band signals.

6. The system of claim 5, wherein the identifying the features includes:

detecting variance in the sub-band signals that is not relevant to the multiple objects; and

removing the detected variance from the sub-band signals, so as to provide the sensor data, associated with the at least one audio signal, with the detected variance removed.

7. The system of claim 5, wherein the features includes one or more of the following:

energy level differences between the sub-band signals, energies of the sub-band signals, pitches, saliences of the pitches, and spatial locations.

8. The system of claim 7, wherein the spatial locations include relative angles associated with probable sound sources.

9. The system of claim 1, wherein the attributing the formed hypotheses to channels includes assigning the formed hypotheses to one or more sub-bands, the one or more sub-bands being associated with filter bank channels.

10. The system of claim 1, wherein:

the plurality of the sensors includes a motion sensor operable to detect motions associated with the probable objects and an image sensor operable to detect images associated with the multiple probable objects; and

identification of the features is at least partially based on the detected motions and detected images.

11. The system of claim 1, wherein the determining the probable objects includes selecting the probable objects from a list of pre-determined object types.

12. The system of claim 11, wherein the pre-determined object types are generated via a neural network previously trained using a plurality of known objects and features associated with each of the known objects.

13. The system of claim 1, wherein the forming hypotheses includes:

enumerating possible combinations of the features associated with the probable objects; and

determining probabilities of the possible combinations of the features.

14. The system of claim 13, wherein the probabilities are generated using a machine learning technique previously trained using a plurality of known objects and combinations of features associated with each of the known objects.

15. The system of claim 13, wherein number of the formed hypotheses is determined based at least on a minimum probability threshold.

16. The system of claim 1, wherein a number of the formed hypotheses is fixed to a pre-determined number.

17. The system of claim 4, wherein constructing the sequences of the formed hypotheses includes linking the formed hypotheses associated with multiple frames from the sequence of the frames of the sensor data.

18. The system of claim 1, wherein the sequences of the formed hypotheses includes a number of the formed hypotheses, attributes of objects associated with each of the formed hypotheses, and features associated with each of the formed hypotheses.

19. A method for estimating and tracking multiple attributes of multiple objects from multi-sensor data, the method comprising:

identifying features associated with sensor data, the sensor data representing data captured by at least one of a plurality of sensors;

determining multiple probable objects based at least in part on the identified features, the determining being performed in parallel for the identified features;

forming hypotheses based at least in part on associating identified features with the multiple probable objects;

attributing the formed hypotheses to channels; and

constructing sequences of the formed hypotheses;

the channels and constructed sequences being configured for use in signal processing.

20. A non-transitory computer-readable storage medium having embodied thereon instructions, which when executed by one or more processors, perform steps of a method, the method comprising:

identifying features associated with sensor data, the sensor data representing data captured by at least one of a plurality of sensors;

determining multiple probable objects based at least in part on the identified features, the determination being made in parallel for the identified features;

forming hypotheses based at least in part on associating identified features with the multiple probable objects;

attributing the formed hypotheses to channels; and

constructing sequences of the formed hypotheses.