Audio navigation assistance

• US 9,140,554 B2
• Filed: 01/24/2014
• Issued: 09/22/2015
• Est. Priority Date: 01/24/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Jerauld, Robert
• Primary Examiner(s): CHEUNG, CALVIN K
• Application Number:
  US14/163,943
• Publication Number:
  US 20150211858A1
• Time in Patent Office: 606 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: A61F 9/08 Devices or methods enabling eye-patients to replace direct visual perception by another kind of perception
• CPC Class Codes: A61H 2201/1604 Head
• CPC Class Codes: A61H 2201/165 Wearable interfaces
• CPC Class Codes: A61H 2201/5007 computer controlled
• CPC Class Codes: A61H 2201/5048 Audio interfaces, e.g. voice or music controlled
• CPC Class Codes: A61H 2201/5058 Sensors or detectors
• CPC Class Codes: A61H 2201/5064 Position sensors
• CPC Class Codes: A61H 2201/5092 Optical sensor
• CPC Class Codes: A61H 3/061 with electronic detecting or guiding means
• CPC Class Codes: G01C 21/20 Instruments for performing navigational calculations G01C21/24, G01C21/26 take precedence
• CPC Class Codes: G01C 21/206 specially adapted for indoor navigation
• CPC Class Codes: G01C 21/3837 Data obtained from a single source
• CPC Class Codes: G02B 2027/0127 comprising devices increasing the depth of field
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G09B 21/003 using tactile presentation of the information, e.g. Braille displays
• CPC Class Codes: G09B 21/006 using audible presentation of the information
• CPC Class Codes: G09B 21/008 using visual presentation of the information for the partially sighted
• CPC Class Codes: H04S 7/303 : Tracking of listener position or orientation

Abstract

Embodiments that relate to a head-mounted computing device for providing navigation assistance via audio output are disclosed. For example, in one disclosed embodiment depth image and visible image data are used to generate a three-dimensional mesh of at least a portion of an environment. Using the mesh, at least one feature in the environment is detected. When operating in a familiar navigation mode and based on detecting the feature, a first audio navigation cue is outputted to the user. When operating in an unfamiliar navigation mode and based on detecting the feature, a second audio navigation cue is outputted to the user, where the second audio navigation cue differs from the first audio navigation cue.

Claims

1. A head-mounted computing device for providing navigation assistance in an environment via audio output, the head-mounted computing device comprising:

one or more depth sensors for generating depth image data from the environment;

one or more visible light sensors for generating visible image data from the environment;

one or more transducers; and

a navigation module executed by a processor of the head-mounted computing device, the navigation module comprising a familiar navigation mode and an unfamiliar navigation mode, the navigation module configured to;

using the depth image data and visible image data, generate a three-dimensional mesh of at least a portion of the environment;

using the three-dimensional mesh, detect at least one feature in the environment;

when operating in the familiar navigation mode and based on detecting the feature, output a first audio navigation cue of the environment to the user via the one or more transducers; and

when operating in the unfamiliar navigation mode and based on detecting the feature, output a second audio navigation cue of the environment to the user via the one or more transducers, wherein the second audio navigation cue differs from the first audio navigation cue.

2. The head-mounted computing device of claim 1, wherein the navigation module is further configured to output one or more of the first audio navigation cue and the second audio navigation cue to the user in a manner that causes the user to perceive the cue as originating at a three-dimensional location of the feature.

3. The head-mounted computing device of claim 1, wherein the feature comprises a slope, and the first audio navigation cue and the second audio navigation cue comprise a plurality of ascending pitched sounds that indicate an inclined slope or a plurality of descending pitched sounds that indicate a declined slope.

4. The head-mounted computing device of claim 3, wherein the first audio navigation cue and the second audio navigation cue further comprise a plurality of continuous pitched sounds that indicate a continuous slope or a plurality of discrete pitched sounds that indicate a plurality of steps.

5. The head-mounted computing device of claim 1, wherein the navigation module is further configured to output the first audio navigation cue and the second audio navigation cue to the user at a volume that increases as a proximity of the feature to the user decreases.

6. The head-mounted computing device of claim 1, wherein the navigation module is further configured to:

receive user input of a room identifier that identifies a room in the environment; and

using the room identifier, generate a map of the environment that includes the room tagged with the room identifier.

7. The head-mounted computing device of claim 6, wherein the navigation module is further configured to:

subsequently detect the room tagged with the room identifier; and

based on detecting the room tagged with the room identifier, engage the familiar navigation mode and use the map of the environment to provide the first audio navigation cue to the user.

8. The head-mounted computing device of claim 1, wherein the navigation module is further configured to:

using one or more of the depth image data and the visible image data, recognize a face in the environment;

associate the face with an identity of a person; and

inform the user of the identity of the person.

9. The head-mounted computing device of claim 1, wherein the navigation module is further configured to output a haptic navigation cue to the user via the one or more transducers.

10. The head-mounted computing device of claim 1, further comprising a display system for displaying a visual navigation cue to the user, wherein the visual navigation cue is based on at least a three-dimensional location of the feature in the environment.

11. A method for providing navigation assistance to a user in an environment via audio output in a head-mounted computing device, the method comprising:

receiving depth image data from the environment;

receiving visible image data from the environment;

using the depth image data and visible image data, generating a three-dimensional mesh of at least a portion of the environment;

using the three-dimensional mesh, detecting at least one feature in the environment;

when the head-mounted computing device is operating in a familiar navigation mode and based on detecting the feature, outputting a first audio navigation cue of the environment to the user via the one or more transducers; and

when the head-mounted computing device is operating in the unfamiliar navigation mode and based on detecting the feature, outputting a second audio navigation cue of the environment to the user via the one or more transducers, wherein the second audio navigation cue differs from the first audio navigation cue.

12. The method of claim 11, further comprising outputting one or more of the first audio navigation cue and the second audio navigation cue to the user in a manner that causes the user to perceive the cue as originating at a three-dimensional location of the feature.

13. The method of claim 11, wherein the feature comprises a slope, and the first audio navigation cue and the second audio navigation cue comprise a plurality of ascending pitched sounds that indicate an inclined slope or a plurality of descending pitched sounds that indicate a declined slope.

14. The method of claim 13, wherein the first audio navigation cue and the second audio navigation cue further comprise a plurality of continuous pitched sounds that indicate a continuous slope or a plurality of discrete pitched sounds that indicate a plurality of steps.

15. The method of claim 11, further comprising outputting the first audio navigation cue and the second audio navigation cue to the user at a volume that increases as a proximity of the detected object to the user decreases.

16. The method of claim 11, further comprising:

recognizing a face in the environment using one or more of the depth image data and the visible image data;

associating the face with an identity of a person; and

informing the user of the identity of the person.

17. The method of claim 16, further comprising:

assigning an identity earcon to the identity of the person; and

outputting the identity earcon to the user via the one or more transducers.

18. The method of claim 11, further comprising outputting a haptic navigation cue to the user via the one or more transducers.

19. The method of claim 11, further comprising, based on detecting the feature, displaying a visual navigation cue to the user via a display system of the head-mounted computing device, wherein the visual navigation cue is based on at least a three-dimensional location of feature in the environment.

20. A method for providing navigation assistance to a user in an environment via audio output in a head-mounted computing device, the method comprising:

receiving depth image data from the environment;

receiving visible image data from the environment;

using the depth image data and visible image data, generating a three-dimensional mesh of at least a portion of the environment;

using the three-dimensional mesh, detecting at least one feature in the environment;

when the head-mounted computing device is operating in a familiar navigation mode and based on detecting the feature, outputting a first audio navigation cue of the environment to the user via the one or more transducers;

when the head-mounted computing device is operating in the unfamiliar navigation mode and based on detecting the feature, outputting a second audio navigation cue of the environment to the user via the one or more transducers, wherein the second audio navigation cue differs from the first audio navigation cue; and

outputting one or more of the first audio navigation cue and the second audio navigation cue to the user in a manner that causes the user to perceive the cue as originating at a three-dimensional location of the feature.