Augmented auditory perception for the visually impaired

• US 8,797,386 B2
• Filed: 04/22/2011
• Issued: 08/05/2014
• Est. Priority Date: 04/22/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Chou, Philip A., Zhang, Zhengyou, Florencio, Dinei
• Primary Examiner(s): Rao, Andy
• Assistant Examiner(s): WALKER, JARED T
• Application Number:
  US13/092,276
• Publication Number:
  US 20120268563A1
• Time in Patent Office: 1,201 Days
• Field of Search: 348/46, 348/62, 348/148, 348/163, 704/260, 704/271
• US Class Current: 348/46
• CPC Class Codes: A61H 2201/0157 portable
• CPC Class Codes: A61H 2201/165 Wearable interfaces
• CPC Class Codes: A61H 2201/501 connected to external computer devices or networks
• CPC Class Codes: A61H 2201/5048 Audio interfaces, e.g. voice or music controlled
• CPC Class Codes: A61H 2201/5058 Sensors or detectors
• CPC Class Codes: A61H 2201/5092 Optical sensor
• CPC Class Codes: A61H 3/061 with electronic detecting or guiding means
• CPC Class Codes: G01S 15/86 Combinations of sonar systems with lidar systems; Combinations of sonar systems with systems not using wave reflection
• CPC Class Codes: G01S 15/89 for mapping or imaging
• CPC Class Codes: G01S 15/93 for anti-collision purposes
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04R 2420/07 Applications of wireless loudspeakers or wireless microphones
• CPC Class Codes: H04R 5/033 Headphones for stereophonic communication details thereof, e.g. relating to batteries, cables or control elements H04R1/10

Abstract

A person is provided with the ability to auditorily determine the spatial geometry of his current physical environment. A spatial map of the current physical environment of the person is generated. The spatial map is then used to generate a spatialized audio representation of the environment. The spatialized audio representation is then output to a stereo listening device which is being worn by the person.

Claims

1. A computer-implemented process for providing a person with the ability to auditorily determine the spatial geometry of his current physical environment, comprising:

using a computer to perform the following process actions;

generating a depth map of the current physical environment of the person;

analyzing the depth map to identify objects and passageways therein;

analyzing the depth map to determine the current spatial coordinates, from the perspective of an environment sensing device being held by the person, of each of the identified objects and passageways;

using a first spatial orientation sensor within the sensing device to determine the current spatial orientation of the sensing device;

using a second spatial orientation sensor within a stereo listening device being worn on the head of the person to determine the current spatial orientation of the head of the person, wherein the second spatial orientation sensor is located at a distance from the first spatial orientation sensor;

comparing the current spatial orientation of the sensing device to the current spatial orientation of the head of the person to determine a relative current pose between the sensing device and head;

determining the current distance between the sensing device and head;

using said current pose and current distance to transform the current spatial coordinates of each of the identified objects and passageways to be from the perspective of the head of the person;

using the depth map to generate a spatialized audio representation of the current physical environment; and

outputting said spatialized audio representation to the stereo listening device.

2. The process of claim 1, wherein the process action of generating a depth map of the current physical environment of the person comprises the actions of:

using an infrared projector to project a known infrared pattern into said environment;

using an infrared camera that is matched to the infrared projector to capture an ongoing series of images of a reflected infrared pattern being reflected off of objects in said environment; and

generating the depth map of said environment by comparing the known infrared pattern to the reflected infrared pattern in each captured image.

3. The process of claim 1, wherein the process action of generating a depth map of the current physical environment of the person comprises the actions of:

using an ultrasound transmitter to transmit an original ultrasound signal into said environment;

using an ultrasound receiver that is matched to the ultrasound transmitter to capture a return ultrasound signal being reflected off of objects in said environment; and

generating the depth map of said environment by comparing the original ultrasound signal to the return ultrasound signal.

4. The process of claim 1, wherein the process action of generating a depth map of the current physical environment of the person comprises the actions of:

using a pair of visible light video cameras to capture a stereo video stream of said environment; and

generating the depth map of said environment by processing the stereo video stream.

5. The process of claim 1, wherein the process action of generating a depth map of the current physical environment of the person comprises the actions of:

using a single visible light video camera to capture a video stream of said environment; and

generating the depth map of said environment by processing the video stream using a structure from motion method.

6. The process of claim 1, wherein the process action of analyzing the depth map to identify objects and passageways therein comprises the actions of:

using a visible light video camera to capture a video stream of objects and passageways in the current physical environment of the person; and

analyzing the video stream to identify one or more attributes of each of said objects and passageways.

7. The process of claim 1, wherein the process action of analyzing the depth map to identify objects and passageways therein comprises the actions of:

using an ultrasound transmitter to transmit an original ultrasound signal into the current physical environment of the person;

using an ultrasound receiver that is matched to the ultrasound transmitter to capture a return ultrasound signal being reflected off of objects in said environment; and

identifying one or more attributes of each of the objects and passageways in said environment by comparing the original ultrasound signal to the return ultrasound signal.

8. The process of claim 7, wherein the attributes of each of the objects comprise whether the object is hard or soft.

9. The process of claim 1, wherein, the process action of using the depth map to generate a spatialized audio representation of the current physical environment comprises an action of using a head-related transfer function (HRTF) for the person to spatialize a different virtual sound source at the current spatial coordinates, from the perspective of the head of the person, of each of the identified objects and passageways, each sound source generating an identifiable sound.

10. The process of claim 9, wherein the process action of using the depth map to generate a spatialized audio representation of the current physical environment further comprises using the HRTF for the person to spatialize one or more additional virtual sound sources at prescribed spatial coordinates from the perspective of the head of the person, wherein each additional virtual sound source communicates a specific type of direction information to the person, said information comprising one of a prescribed cardinal direction, or a prescribed ordinal direction, or a waypoint.

11. The process of claim 9, wherein the process action of using the depth map to generate a spatialized audio representation of the current physical environment further comprises providing the person with audible navigation information which is specifically related to said environment, said information comprising verbal directions to a particular landmark in said environment.

12. The process of claim 9, wherein whenever the current physical environment of the person comprises one or more fast moving objects, the process action of using the depth map to generate a spatialized audio representation of the current physical environment further comprises using the HRTF for the person to spatialize a virtual sound source at spatial coordinates, from the perspective of the head of the person, that correspond to the current locus of the fast moving objects, wherein said sound source simulates the sound of the fast moving objects.

15. The process of claim 9, wherein the process action of using the depth map to generate a spatialized audio representation of the current physical environment further comprises an action of mixing a direct path of the sound that emanates from each different virtual sound source with either virtual reflections off of any floor or any walls identified in the depth map, or reverberation and noise, or a combination of said virtual reflections, reverberation and noise.

16. The process of claim 9, wherein the process action of using the depth map to generate a spatialized audio representation of the current physical environment further comprises an action of adjusting the direct path to reflected path energy ratio for each different virtual sound source.

17. The process of claim 9, wherein the process action of using the depth map to generate a spatialized audio representation of the current physical environment further comprises an action of adjusting the direct path to reverberation energy ratio for each different virtual sound source.

13. A computer-implemented process for providing a person with the ability to auditorily determine the spatial geometry of his current physical environment, comprising:

using a computer to perform the following process actions;

using an infrared projector to project a known infrared pattern into the current physical environment of the person;

using an infrared camera that is matched to the infrared projector to capture an ongoing series of images of a reflected infrared pattern being reflected off of objects in said environment;

generating a depth map of said environment by comparing the known infrared pattern to the reflected infrared pattern in each captured image;

analyzing the depth map to identify objects and passageways therein;

analyzing the depth map to determine the current spatial coordinates, from the perspective of an environment sensing device being held by the person, of each of the identified objects and passageways;

using a first spatial orientation sensor within the sensing device to determine the current spatial orientation of the sensing device;

using a second spatial orientation sensor within a stereo listening device being worn by the person to determine the current spatial orientation of the head of the person, wherein the second spatial orientation sensor is located at a distance from the first spatial orientation sensor;

comparing the current spatial orientation of the sensing device to the current spatial orientation of the head of the person to determine a relative current pose between the sensing device and head;

determining the current distance between the sensing device and head;

using said current pose and current distance to transform the current spatial coordinates of each of the identified objects and passageways to be from the perspective of the head of the person;

using a head-related transfer function for the person to generate a spatialized audio representation of said environment by spatializing a different virtual sound source at the transformed current spatial coordinates of each of the identified objects and passageways, wherein each sound source generates an identifiable sound; and

outputting said spatialized audio representation to the listening device.

14. The process of claim 13, further comprising the actions of:

using a visible light video camera to capture a video stream of said environment; and

analyzing the video stream to identify one or more attributes of each of the identified objects and passageways,

wherein the identifiable sound generated by each sound source comprises characteristics which are tailored to the identified attributes of the particular object or passageway the sound source is attached to.