Estimating and tracking multiple attributes of multiple objects from multi-sensor data

• US 9,500,739 B2
• Filed: 03/24/2015
• Issued: 11/22/2016
• Est. Priority Date: 03/28/2014
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Knowles Electronics Llc (Knowles Corporation)
• Inventors: Woodruff, John, Avendano, Carlos, Goodwin, Michael M.
• Primary Examiner(s): Baker, Charlotte M
• Application Number:
  US14/666,312
• Publication Number:
  US 20160061934A1
• Time in Patent Office: 609 Days
• Field of Search: 704/233, 725/105, 340/945, 348/143, 348/144, 348/148, 348/152, 348/153, 348/154, 348/155, 348/156, 348/159, 348/E7.086, 701/14
• US Class Current: 1/1
• CPC Class Codes: G01S 3/802 Systems for determining direction or deviation from predetermined direction
• CPC Class Codes: G01S 5/20 Position of source determined by a plurality of spaced direction-finders
• CPC Class Codes: G10L 25/18 the extracted parameters being spectral information of each sub-band
• CPC Class Codes: G10L 25/48 specially adapted for particular use

Abstract

Systems and methods for estimating and tracking multiple attributes of multiple objects from multi-sensor data are provided. An exemplary method includes identifying features associated with sensor data. The sensor data represents data captured by at least one of a plurality of acoustic and non-acoustic sensors. Identification of the features associated with the sensor data may be based variously on detected sounds, motions, images, and the like. The exemplary method further includes determining, in parallel, multiple probable objects based at least in part on the identified features. Various embodiments of the method also include forming hypotheses based at least in part on associating identified features with the multiple probable objects and attributing the formed hypotheses to channels. Sequence of the formed hypotheses are constructed. The exemplary system includes a tracking module configured to provide the channels and constructed sequences for use in various signal processing, such as signal enhancement.

Claims

1. A system for estimating and tracking multiple attributes of multiple objects from multi-sensor data, the system comprising:

an object detector stored in memory and executed by at least one processor to determine multiple probable objects based at least in part on features associated with sensor data, the sensor data representing data captured by at least one of a plurality of sensors, the determination being made in parallel for the features;

a hypothesis generator stored in memory and executed by the at least one processor to form hypotheses based at least in part on associating the features with the multiple probable objects;

a grouping module stored in memory and executed by the at least one processor to attribute the formed hypotheses to channels; and

a tracking module stored in memory and executed by the at least one processor to construct sequences of the formed hypotheses;

the tracking module being configured to provide the channels and constructed sequences for use in signal processing.

2. The system of claim 1, wherein the determination is performed simultaneously for the multiple probable objects.

3. The system of claim 1, wherein the sensor data comprises at least one audio signal representing at least one captured sound, at least some of the multiple probable objects are sound sources, and the plurality of the sensors includes at least two microphones configured to capture the at least one sound.

4. The system of claim 3, further comprising a front end module operable to transform data representing the at least one captured sound to obtain a sequence of frames of the sensor data.

5. The system of claim 4, wherein the transformation includes:

performing an analog-to-digital conversion of the at least one audio signal to generate a digital acoustic signal;

transforming the digital acoustic signal from a time domain to a frequency domain; and

separating the transformed digital acoustic signal into sub-band signals.

6. The system of claim 5, wherein identification of the features includes:

detecting variance in the sub-band signals that is not relevant to the multiple probable objects; and

removing the detected variance from the sub-band signals, so as to provide the sensor data, associated with the at least one audio signal, with the detected variance removed.

7. The system of claim 5, wherein the features include one or more of the following:

energy level differences between the sub-band signals, energies of the sub-band signals, pitches, saliences of the pitches, and spatial locations.

8. The system of claim 7, wherein the spatial locations include relative angles associated with probable sound sources.

9. The system of claim 1, wherein the attributing the formed hypotheses to the channels includes assigning the formed hypotheses to one or more sub-bands, the one or more sub-bands being associated with filter bank channels.

10. The system of claim 1, wherein:

the plurality of the sensors includes a motion sensor operable to detect motions associated with the multiple probable objects and an image sensor operable to detect images associated with the multiple probable objects; and

identification of the features is at least partially based on the detected motions and detected images.

11. The system of claim 1, wherein the determining the multiple probable objects includes selecting the multiple probable objects from a list of pre-determined object types.

12. The system of claim 11, wherein the pre-determined object types are generated via a neural network previously trained using a plurality of known objects and features associated with each of the known objects.

13. The system of claim 1, wherein the forming hypotheses includes:

enumerating possible combinations of the features associated with the multiple probable objects; and

determining probabilities of the possible combinations of the features.

14. The system of claim 13, wherein the probabilities are generated using a machine learning technique previously trained using a plurality of known objects and combinations of the features associated with each of the known objects.

15. The system of claim 13, wherein a number of the formed hypotheses is determined based at least on a minimum probability threshold.

16. The system of claim 1, wherein a number of the formed hypotheses is fixed to a pre-determined number.

17. The system of claim 4, wherein constructing the sequences of the formed hypotheses includes linking the formed hypotheses associated with multiple frames from the sequence of the frames of the sensor data.

18. The system of claim 1, wherein the sequences of the formed hypotheses includes a number of the formed hypotheses, attributes of objects associated with each of the formed hypotheses, and features associated with each of the formed hypotheses.

19. A method for estimating and tracking multiple attributes of multiple objects from multi-sensor data, the method comprising:

determining multiple probable objects based at least in part on features associated with sensor data, the sensor data representing data captured by at least one of a plurality of sensors, the determining being performed in parallel for the features;

forming hypotheses based at least in part on associating the features with the multiple probable objects;

attributing the formed hypotheses to channels; and

constructing sequences of the formed hypotheses;

the channels and constructed sequences being configured for use in signal processing.

20. A non-transitory computer-readable storage medium having embodied thereon instructions, which when executed by one or more processors, perform steps of a method, the method comprising:

determining multiple probable objects based at least in part on features associated with sensor data, the sensor data representing data captured by at least one of a plurality of sensors, the determination being made in parallel for the features;

forming hypotheses based at least in part on associating the features with the multiple probable objects;

attributing the formed hypotheses to channels; and

constructing sequences of the formed hypotheses, the channels and constructed sequences being configured for use in signal processing.