Monitoring

• US 10,402,984 B2
• Filed: 08/10/2016
• Issued: 09/03/2019
• Est. Priority Date: 08/14/2015
• Status: Active Grant

• Current Assignee: Nokia Technologies Oy (Nokia Corporation)
• Original Assignee: Nokia Technologies Oy (Nokia Corporation)
• Inventors: Lehtiniemi, Arto, Leppanen, Jussi, Eronen, Antti
• Primary Examiner(s): Seth, Manav
• Application Number:
  US15/750,598
• Publication Number:
  US 20190019295A1
• Time in Patent Office: 1,119 Days
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G10L 2021/02166 Microphone arrays; Beamforming
• CPC Class Codes: H04R 1/406 microphones
• CPC Class Codes: H04R 2430/23 Direction finding using a sum-delay beam-former
• CPC Class Codes: H04R 2430/25 Array processing for suppression of unwanted side-lobes in directivity characteristics, e.g. a blocking matrix
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407
• CPC Class Codes: H04R 5/027 Spatial or constructional arrangements of microphones, e.g. in dummy heads
• CPC Class Codes: H04R 5/04 Circuit arrangements, e.g. for selective connection of amplifier inputs/outputs to loudspeakers, for loudspeaker detection, or for adaptation of settings to personal preferences or hearing impairments combinations of amplifiers H03F3/68; stereophonic systems H04S
• CPC Class Codes: H04S 2400/15 Aspects of sound capture and related signal processing for recording or reproduction
• CPC Class Codes: H04S 7/303 Tracking of listener position or orientation

Abstract

A method comprising: processing received image data and/or received depth data to track an object moving in a monitored scene space; determining a bearing of the tracked moving object in the monitored scene space; and adaptively filtering outputs from a plurality of spatially diverse microphones in dependence upon the bearing of the tracked moving object in the monitored scene space to produce a combined audio signal.

Claims

1. A method comprising:

processing received image data and/or received depth data to track an object moving in a monitored scene space;

determining a bearing of the moving object in the monitored scene space;

filtering outputs from a plurality of spatially diverse microphones in dependence upon the bearing of the moving object in the monitored scene space to produce a combined audio output signal, wherein the combined audio output signal has directional gain in a target direction;

defining at least a first portion of the monitored scene space and a second portion of the monitored scene space separated by at least one virtual boundary;

detecting movement of the moving object from the first portion of the monitored scene space to the second portion of the monitored scene space across the at least one virtual boundary; and

in dependence upon detecting movement of the moving object from the first portion of the monitored scene space to the second portion of the monitored scene space across the at least one virtual boundary, changing the target direction from a first direction being one of one or more target directions associated with the first portion of the monitored scene space to a second direction being one of the one or more target directions associated with the second portion of the monitored scene space, wherein the second direction is different from the first direction.

2. A method as claimed in claim 1, wherein the combined audio output signal has directional gain enhancing audio produced by the moving object.

3. A method as claimed in claim 1, wherein the target direction is determined by the bearing of the moving object and/or wherein the target direction is a bearing of the moving object.

4. A method as claimed in claim 1 comprising:

defining a plurality of portions of the monitored scene space demarcated by virtual boundaries;

detecting movement of the moving object from an originating portion of the monitored scene space to a destination portion of the monitored scene space across one or more virtual boundaries; and

in dependence upon detecting movement of the moving object from an originating portion of the monitored scene space to a destination portion of the monitored scene space across one or more virtual boundaries, changing the target direction to point towards a predetermined bearing within the originating portion of the monitored scene space to a predetermined bearing within the destination portion of the monitored scene space.

5. A method as claimed in claim 1 further comprising providing for user predefinition of one or more virtual boundaries and/or user predefinition of one or more target directions.

6. A method as claimed in claim 1, wherein filtering outputs from a plurality of spatially diverse microphones in dependence upon the bearing of the moving object in the monitored scene space to produce a combined audio output signal comprises using a filter-sum beamformer or delay-sum beamformer configured in dependence upon the bearing of the moving object in the monitored scene space.

7. A method as claimed in claim 1 comprising determining whether the filtering of outputs from the plurality of spatially diverse microphones has directional gain configured to enhance audio produced by the moving object or directional gain configured to enhance audio produced other than by the moving object

and/or

determining whether the filtering of outputs from the plurality of spatially diverse microphones has directional gain configured to enhance audio produced by the moving object or directional gain configured to enhance audio produced by a different object.

8. A method as claimed in claim 1, wherein the processing of the received image data and/or the received depth data to track an object moving in a monitored scene space;

the determining a bearing of the moving object in the monitored scene space; and

the filtering of outputs from the plurality of spatially diverse microphones in dependence upon the bearing of the moving object in the monitored scene space, occur in real-time to produce in real-time the combined audio output signal.

9. A method as claimed in claim 1, wherein the plurality of spatially diverse microphones form a uniformly spaced array and/or form a fixed array and/or are internal microphones of a device.

10. A method as claimed in claim 1, comprising:

processing received data to generate a response event when there is, relative to at least one virtual boundary, a change in a portion of the monitored scene space; and

in response to the response event, changing the filtering of outputs from the plurality of spatially diverse microphones to produce a combined audio output signal.

11. An apparatus comprising at least one processor and at least one memory, the memory comprising machine-readable instructions, that when executed cause the apparatus to:

process received image data and/or received depth data to track an object moving in a monitored scene space;

determine a bearing of the moving object in the monitored scene space;

filter outputs from a plurality of spatially diverse microphones in dependence upon the bearing of the moving object in the monitored scene space to produce a combined audio output signal, wherein the combined audio output signal has directional gain in a target direction;

define at least a first portion of the monitored scene space and a second portion of the monitored scene space separated by at least one virtual boundary;

detect movement of the moving object from the first portion of the monitored scene space to the second portion of the monitored scene space across the at least one virtual boundary; and

in dependence upon detecting movement of the moving object from the first portion of the monitored scene space to the second portion of the monitored scene space across the at least one virtual boundary, change the target direction from a first direction being one of one or more target directions associated with the first portion of the monitored scene space to a second direction being one of the one or more target directions associated with the second portion of the monitored scene space, wherein the second direction is different from the first direction.

12. The apparatus of claim 11, wherein the combined audio output signal has directional gain enhancing audio produced by the moving object.

13. The apparatus of claim 11, wherein the target direction is determined by the bearing of the moving object and/or wherein the target direction is a bearing of the moving object.

14. The apparatus of claim 11, further caused to:

define a plurality of portions of the monitored scene space demarcated by virtual boundaries;

detect movement of the moving object from an originating portion of the monitored scene space to a destination portion of the monitored scene space across one or more virtual boundaries; and

in dependence upon detecting movement of the moving object from an originating portion of the monitored scene space to a destination portion of the monitored scene space across one or more virtual boundaries, change the target direction to point towards a predetermined bearing within the originating portion of the monitored scene space to a predetermined bearing within the destination portion of the monitored scene space.

15. The apparatus of claim 11, further caused to provide for user predefinition of one or more virtual boundaries and/or user predefinition of one or more target directions.

16. The apparatus of claim 11, wherein filtering outputs from a plurality of spatially diverse microphones in dependence upon the bearing of the moving object in the monitored scene space to produce a combined audio output signal comprises using a filter-sum beamformer or delay-sum beamformer configured in dependence upon the bearing of the moving object in the monitored scene space.

17. The apparatus of claim 11, further caused to:

determine whether the filtering of outputs from the plurality of spatially diverse microphones has directional gain configured to enhance audio produced by the moving object or directional gain configured to enhance audio produced other than by the moving object

and/or

determine whether the filtering of outputs from the plurality of spatially diverse microphones has directional gain configured to enhance audio produced by the moving object or directional gain configured to enhance audio produced by a different object.

18. The apparatus of claim 11, wherein the processing of the received image data and/or the received depth data to track an object moving in a monitored scene space;

the determining a bearing of the moving object in the monitored scene space; and

the filtering of outputs from the plurality of spatially diverse microphones in dependence upon the bearing of the moving object in the monitored scene space, occur in real-time to produce in real-time the combined audio output signal.

19. The apparatus of claim 11, further caused to:

process received data to generate a response event when there is, relative to at least one virtual boundary, a change in a portion of the monitored scene space; and

in response to the response event, change the filtering of outputs from the plurality of spatially diverse microphones to produce a combined audio output signal.

20. At least one non-transitory computer readable medium comprising instructions that, when executed, perform:

process received image data and/or received depth data to track an object moving in a monitored scene space;

determine a bearing of the moving object in the monitored scene space;

filter outputs from a plurality of spatially diverse microphones in dependence upon the bearing of the moving object in the monitored scene space to produce a combined audio output signal, wherein the combined audio output signal has directional gain in a target direction;

define at least a first portion of the monitored scene space and a second portion of the monitored scene space separated by at least one virtual boundary;

detect movement of the moving object from the first portion of the monitored scene space to the second portion of the monitored scene space across the at least one virtual boundary; and

in dependence upon detecting movement of the moving object from the first portion of the monitored scene space to the second portion of the monitored scene space across the at least one virtual boundary, change the target direction from a first direction being one of one or more target directions associated with the first portion of the monitored scene space to a second direction being one of the one or more target directions associated with the second portion of the monitored scene space, wherein the second direction is different from the first direction.