AUGMENTED AUDITORY PERCEPTION FOR THE VISUALLY IMPAIRED

• US 20120268563A1
• Filed: 04/22/2011
• Published: 10/25/2012
• Est. Priority Date: 04/22/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Chou, Philip A., Zhang, Zhengyou, Florencio, Dinei
• Granted Patent: US 8,797,386 B2
• Time in Patent Office: Days
• US Class Current: 348/46
• CPC Class Codes: A61H 2201/0157 portable
• CPC Class Codes: A61H 2201/165 Wearable interfaces
• CPC Class Codes: A61H 2201/501 connected to external computer devices or networks
• CPC Class Codes: A61H 2201/5048 Audio interfaces, e.g. voice or music controlled
• CPC Class Codes: A61H 2201/5058 Sensors or detectors
• CPC Class Codes: A61H 2201/5092 Optical sensor
• CPC Class Codes: A61H 3/061 with electronic detecting or guiding means
• CPC Class Codes: G01S 15/86 Combinations of sonar systems with lidar systems; Combinations of sonar systems with systems not using wave reflection
• CPC Class Codes: G01S 15/89 for mapping or imaging
• CPC Class Codes: G01S 15/93 for anti-collision purposes
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04R 2420/07 Applications of wireless loudspeakers or wireless microphones
• CPC Class Codes: H04R 5/033 Headphones for stereophonic communication details thereof, e.g. relating to batteries, cables or control elements H04R1/10

Abstract

A person is provided with the ability to auditorily determine the spatial geometry of his current physical environment. A spatial map of the current physical environment of the person is generated. The spatial map is then used to generate a spatialized audio representation of the environment. The spatialized audio representation is then output to a stereo listening device which is being worn by the person.

Claims

1. A computer-implemented process for providing a person with the ability to auditorily determine the spatial geometry of his current physical environment, comprising:

using a computer to perform the following process actions;

generating a spatial map of the current physical environment of the person;

using the spatial map to generate a spatialized audio representation of the current physical environment; and

outputting said spatialized audio representation to a stereo listening device being worn by the person.

2. The process of claim 1, wherein the process action of generating a spatial map of the current physical environment of the person comprises the actions of:

using an infrared projector to project a known infrared pattern into said environment;

using an infrared camera that is matched to the infrared projector to capture an ongoing series of images of a reflected infrared pattern being reflected off of objects in said environment; and

generating a depth map of said environment by comparing the known infrared pattern to the reflected infrared pattern in each captured image.

3. The process of claim 1, wherein the process action of generating a spatial map of the current physical environment of the person comprises the actions of:

using an ultrasound transmitter to transmit an original ultrasound signal into said environment;

using an ultrasound receiver that is matched to the ultrasound transmitter to capture a return ultrasound signal being reflected off of objects in said environment; and

generating a depth map of said environment by comparing the original ultrasound signal to the return ultrasound signal.

4. The process of claim 1, wherein the process action of generating a spatial map of the current physical environment of the person comprises the actions of:

using a pair of visible light video cameras to capture a stereo video stream of said environment; and

generating a depth map of said environment by processing the stereo video stream.

5. The process of claim 1, wherein the process action of generating a spatial map of the current physical environment of the person comprises the actions of:

using a single visible light video camera to capture a video stream of said environment; and

generating a depth map of said environment by processing the video stream using a structure from motion method.

6. The process of claim 1, wherein the process action of generating a spatial map of the current physical environment of the person comprises an action of analyzing the spatial map to identify objects and passageways therein.

7. The process of claim 6, wherein the process action of analyzing the spatial map to identify objects and passageways therein comprises the actions of:

using a visible light video camera to capture a video stream of objects and passageways in the current physical environment of the person; and

analyzing the video stream to identify one or more attributes of each of said objects and passageways.

8. The process of claim 6, wherein the process action of analyzing the spatial map to identify objects and passageways therein comprises the actions of:

using an ultrasound transmitter to transmit an original ultrasound signal into the current physical environment of the person;

using an ultrasound receiver that is matched to the ultrasound transmitter to capture a return ultrasound signal being reflected off of objects in said environment; and

identifying one or more attributes of each of the objects and passageways in said environment by comparing the original ultrasound signal to the return ultrasound signal.

9. The process of claim 8, wherein the attributes of each of the objects comprise whether the object is hard or soft.

10. The process of claim 1, wherein,

the spatial map comprises a depth map,

the process action of generating a spatial map of the current physical environment of the person comprises an action of analyzing the depth map to identify objects and passageways therein, and

the process action of using the spatial map to generate a spatialized audio representation of the current physical environment comprises the actions of:

determining the current spatial coordinates, from the perspective of the head of the person, of each of the identified objects and passageways; and

using a head-related transfer function (HRTF) for the person to spatialize a different virtual sound source at the current spatial coordinates, from the perspective of the head of the person, of each of the identified objects and passageways, each sound source generating an identifiable sound.

11. The process of claim 10, wherein the process action of using the spatial map to generate a spatialized audio representation of the current physical environment further comprises at least one of the actions of:

mixing a direct path of the sound that emanates from each different virtual sound source with either virtual reflections off of any floor or any walls identified in the depth map, or reverberation and noise, or a combination of said virtual reflections, reverberation and noise;

or

adjusting the relative loudness of each different virtual sound source;

or

adjusting the direct path to reflected path energy ratio for each different virtual sound source;

or

adjusting the direct path to reverberation energy ratio for each different virtual sound source.

12. The process of claim 10, wherein the process action of using the spatial map to generate a spatialized audio representation of the current physical environment further comprises using the HRTF for the person to spatial ize one or more additional virtual sound sources at prescribed spatial coordinates from the perspective of the head of the person, wherein each additional virtual sound source communicates a specific type of direction information to the person, said information comprising one of a prescribed cardinal direction, or a prescribed ordinal direction, or a waypoint.

13. The process of claim 10, wherein the process action of using the spatial map to generate a spatialized audio representation of the current physical environment further comprises providing the person with audible navigation information which is specifically related to said environment, said information comprising at least one of verbal directions to a particular landmark in said environment, or a verbal explanation of one or more aspects of said environment.

14. The process of claim 10, wherein whenever the current physical environment of the person comprises one or more fast moving objects, the process action of using the spatial map to generate a spatialized audio representation of the current physical environment further comprises using the HRTF for the person to spatialize a virtual sound source at spatial coordinates, from the perspective of the head of the person, that correspond to the current locus of the fast moving objects, wherein said sound source simulates the sound of the fast moving objects.

15. The process of claim 10, wherein the process action of determining the current spatial coordinates, from the perspective of the head of the person, of each of the identified objects and passageways comprises the actions of:

analyzing the depth map to determine the current spatial coordinates, from the perspective of an environment sensing device being held by the person, of each of the identified objects and passageways;

using a first spatial orientation sensor within the sensing device to determine the current spatial orientation of the sensing device;

using a second spatial orientation sensor within the stereo listening device to determine the current spatial orientation of the head of the person;

comparing the current spatial orientation of the sensing device to the current spatial orientation of the head of the person to determine a relative current pose between the sensing device and head;

determining the current distance between the sensing device and head; and

using said current pose and current distance to transform the current spatial coordinates of each of the identified objects and passageways to be from the perspective of the head of the person.

16. A sensing device for providing a person with the ability to auditorily determine the spatial geometry of his current physical environment, comprising:

an object sensor;

a spatial orientation sensor;

an audio reproduction module comprising a left side sound reproduction element and a right side sound reproduction element; and

a computing device coupled to the object sensor, spatial orientation sensor and audio reproduction module, wherein,

the object sensor and computing device operate together to generate a spatial map of the current physical environment of the person,

said map identifies the current location of objects and passageways in said environment,

the spatial orientation sensor and computing device operate together to determine the current spatial orientation of the sensing device, and

the computing device uses said map to generate a spatialized audio representation of said objects and passageways, and outputs said representation to the audio reproduction module.

17. The sensing device of claim 16, wherein the audio reproduction module is integral to the sensing device and the sensing device is worn by the person either over his ears, or in his ears, or in close proximity to his ears.

18. The sensing device of claim 16, wherein the sensing device is held by the person, and the left side and right side sound reproduction elements are implemented in a separate listening device which is worn by the person either over his ears, or in his ears, or in close proximity to his ears.

19. A computer-implemented process for providing a person with the ability to auditorily determine the spatial geometry of his current physical environment, comprising:

using a computer to perform the following process actions;

using an infrared projector to project a known infrared pattern into the current physical environment of the person;

using an infrared camera that is matched to the infrared projector to capture an ongoing series of images of a reflected infrared pattern being reflected off of objects in said environment;

generating a depth map of said environment by comparing the known infrared pattern to the reflected infrared pattern in each captured image;

analyzing the depth map to identify objects and passageways therein;

analyzing the depth map to determine the current spatial coordinates, from the perspective of an environment sensing device being held by the person, of each of the identified objects and passageways;

using a first spatial orientation sensor within the sensing device to determine the current spatial orientation of the sensing device;

using a second spatial orientation sensor within a stereo listening device being worn by the person to determine the current spatial orientation of the head of the person;

comparing the current spatial orientation of the sensing device to the current spatial orientation of the head of the person to determine a relative current pose between the sensing device and head;

determining the current distance between the sensing device and head;

using said current pose and current distance to transform the current spatial coordinates of each of the identified objects and passageways to be from the perspective of the head of the person;

using a head-related transfer function for the person to generate a spatialized audio representation of said environment by spatializing a different virtual sound source at the transformed current spatial coordinates of each of the identified objects and passageways, wherein each sound source generates an identifiable sound; and

outputting said spatialized audio representation to the listening device.

20. The process of claim 19, further comprising the actions of:

using a visible light video camera to capture a video stream of said environment; and

analyzing the video stream to identify one or more attributes of each of the identified objects and passageways,

wherein the identifiable sound generated by each sound source comprises characteristics which are tailored to the identified attributes of the particular object or passageway the sound source is attached to.