Wearable display system with indicators of speakers

• US 20020103649A1
• Filed: 01/31/2001
• Published: 08/01/2002
• Est. Priority Date: 01/31/2001
• Status: Active Grant

• Current Assignee: Uniloc 2017 LLC (SoftBank Group Corp.)
• Original Assignee: International Business Machines Corporation
• Inventors: Basson, Sara H., Kanevsky, Dimitri
• Granted Patent: US 6,975,991 B2
• Time in Patent Office: Days
• US Class Current: 704/270
• CPC Class Codes: A61F 11/04 Methods or devices for enabling ear patients to achieve auditory perception through physiological senses other than hearing sense, e.g. through the touch sense electronic stimulation of auditory nerves A61N1/36036
• CPC Class Codes: G01S 5/18 using ultrasonic, sonic, or infrasonic waves
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G09B 21/00 Teaching, or communicating with, the blind, deaf or mute audible presentation of material to be studied G09B5/04; devices or methods for replacing direct visual or auditory perception by another kind of perception A61F9/08, A61F11/04; audible indication of meter readings or of colour G01D7/12; watches for blind persons G04B25/02; speech analysis, speech recognition G10L; sound-recording or reproducing, per se G11B
• CPC Class Codes: G09B 21/04 Devices for conversing with the deaf-blind
• CPC Class Codes: G10L 15/25 using position of the lips, movement of the lips or face analysis
• CPC Class Codes: G10L 25/78 Detection of presence or absence of voice signals switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10
• CPC Class Codes: H04R 25/502 using analog signal processing

Abstract

Methods and systems for creating a comfortable, user-friendly environment that allows a hearing impaired user to identify who is speaking and preferably what is being said during interactions with other individuals, e.g., at any form of meeting. In accordance with the invention, it is determined whether or not someone is speaking. If yes, then the speaker'"'"'s position is identified. It is also determined whether the speaker is in the range of view for the user'"'"'s display. If yes, an illuminated dot is projected above the speaker on a wearable display to show the user where the speaker is located. If no, a directional arrow is projected on the display to indicate to the user which way he should look to see the current speaker.

Claims

1. A method of providing a user with one or more visual indications, in accordance with a display system associated with the user, of who is currently speaking during an event in which the user is engaged, the event including one or more other individuals, the method comprising the steps of:

identifying the location of the individual who is currently speaking during the event;

determining whether the individual identified as the current speaker is within a field of view of the user;

displaying a first visual indicator to the user, in accordance with the display system, in association with the individual identified as the current speaker when the individual is within the field of view of the user; and

displaying a second visual indicator to the user, in accordance with the display system, when the individual identified as the current speaker is not within the field of view of the user.

2. The method of claim 1, wherein the display system is worn by the user.

3. The method of claim 1, wherein the step of identifying the location of the individual who is currently speaking during the event further comprises:

capturing one or more video images of the one or more individuals participating in the event;

analyzing the one or more captured video images to determine which individual has one or more facial features indicative of speech;

designating the individual with the one or more facial features indicative of speech as the current speaker; and

determining the location of the individual designated as the current speaker.

4. The method of claim 3, wherein the step of determining whether the individual identified as the current speaker is within the field of view of the user further comprises capturing one or more video images of the field of view of the user.

5. The method of claim 4, wherein the step of displaying the first visual indicator further comprises correlating at least a portion of the one or more video images captured of the individuals participating in the event with at least a portion of the one or more video images captured of the field of view of the user.

6. The method of claim 1, wherein the step of identifying the location of the individual who is currently speaking during the event further comprises:

capturing audio data of the one or more individuals participating in the event;

analyzing the audio data to determine which individual is uttering sound indicative of speech;

designating the individual uttering sound that is indicative of speech as the current speaker; and

determining the location of the individual designated as the current speaker.

7. The method of claim 6, wherein the step of determining whether the individual identified as the current speaker is within the field of view of the user further comprises capturing directional data associated with the display system and positional data associated with the user.

8. The method of claim 7, wherein the step of displaying the first visual indicator further comprises correlating the location of the current speaker with the directional data associated with the display system and the positional data associated with the user.

9. The method of claim 1, wherein the first visual indicator comprises a marker displayed in proximity to a representation of the individual identified as the current speaker on the display system.

10. The method of claim 1, wherein the first visual indicator comprises a change in at least one attribute associated with a representation of the individual identified as the current speaker on the display system.

11. The method of claim 10, wherein the attribute is one of color and brightness.

12. The method of claim 1, wherein the second visual indicator comprises a directional symbol displayed on the display system indicating to the user the direction to turn such that the current speaker is in the user'"'"'s field of view.

13. The method of claim 1, further comprising the steps of:

obtaining a textual transcription of audio content associated with the event as provided by the one or more individuals; and

displaying the textual transcription of the audio content to the user in accordance with the display system.

14. The method of claim 13, wherein the step of obtaining the textual transcription comprises at least one of human stenography and automatic speech recognition.

16. The apparatus of claim 15, wherein the at least one processing device and the display are worn by the user.

17. The apparatus of claim 16, wherein the display is a head mounted display.

18. The apparatus of claim 15, wherein the first visual indicator comprises a marker displayed in proximity to a representation of the individual identified as the current speaker on the display.

19. The apparatus of claim 15, wherein the first visual indicator comprises a change in at least one attribute associated with a representation of the individual identified as the current speaker on the display.

20. The apparatus of claim 19, wherein the attribute is one of color and brightness.

21. The apparatus of claim 15, wherein the second visual indicator comprises a directional symbol displayed on the display indicating to the user the direction to turn such that the current speaker is in the user'"'"'s field of view.

22. The apparatus of claim 15, wherein the at least one processing device is further operative to obtain a textual transcription of audio content associated with the event as provided by the one or more individuals, and the display is further operative to display the textual transcription of the audio content to the user in accordance with the display system.

24. The system of claim 23, wherein the first visual indicator comprises a marker displayed in proximity to a representation of the individual identified as the current speaker on the wearable display.

25. The system of claim 23, wherein the first visual indicator comprises a change in at least one attribute associated with a representation of the individual identified as the current speaker on the wearable display.

26. The system of claim 25, wherein the attribute is one of color and brightness.

27. The system of claim 23, wherein the second visual indicator comprises a directional symbol displayed on the wearable display indicating to the user the direction to turn such that the current speaker is in the user'"'"'s field of view.

28. The system of claim 23, further comprising a transcription service coupled to the wearable personal computer and operative to obtain a textual transcription of audio content associated with the event as provided by the one or more individuals in accordance with one or more microphones, such that the wearable personal computer causes the display of the textual transcription of the audio content to the user in accordance with the wearable display.

15. Apparatus for providing a user with one or more visual indications of who is currently speaking during an event in which the user is engaged, the event including one or more other individuals, the apparatus comprising:

at least one processing device operative to, in accordance with data captured in association with the event;

(i) identify the location of the individual who is currently speaking during the event; and

(ii) determine whether the individual identified as the current speaker is within a field of view of the user; and

a display, coupled to the at least one processing device, and operative to;

(i) display a first visual indicator to the user in association with the individual identified as the current speaker when the individual is within the field of view of the user; and

(ii) display a second visual indicator to the user when the individual identified as the current speaker is not within the field of view of the user.

23. A system for providing a user with one or more visual indications of who is currently speaking during an event in which the user is engaged, the event including one or more other individuals, the system comprising:

one or more video cameras for capturing video images of the one or more individuals participating in the event;

a video server coupled to the one or more video cameras and operative to;

(i) analyze the captured video images to determine which individual has one or more facial features indicative of speech; and

(ii) identify the location of the individual who is currently speaking during the event;

a second video camera for capturing video images of a field of view of the user;

a wearable personal computer coupled to the video server and the second video camera and operative to;

(i) obtain the location of the individual who is currently speaking during the event from the video server and at least a portion of the video images of the field of view of the user from the second video camera;

(ii) determine whether the individual identified as the current speaker is within a field of view of the user;

(iii) generate a first visual indicator in association with the individual identified as the current speaker when the individual is within the field of view of the user; and

(iv) generate a second visual indicator when the individual identified as the current speaker is not within the field of view of the user; and

a wearable display, coupled to the wearable personal computer, and operative to display the first and second visual indicators to the user.