Method and system for improving audiovisual communication

• US 7,710,654 B2
• Filed: 05/11/2004
• Issued: 05/04/2010
• Est. Priority Date: 05/12/2003
• Status: Active Grant

• Current Assignee: Elbit Systems Limited
• Original Assignee: Elbit Systems Limited
• Inventors: Ashkenazi, Asaf, Shmuely, Yoram
• Primary Examiner(s): Harrington; Alicia M
• Application Number:
  US10/556,322
• Publication Number:
  US 20060238877A1
• Time in Patent Office: 2,184 Days
• Field of Search: 359630-634, 345 7- 9
• US Class Current: 359/630
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 5/30 Polarising elements light-modulating devices with active elements G02F1/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G10L 21/02 Speech enhancement, e.g. noise reduction or echo cancellation reducing echo effects in line transmission systems H04B3/20; echo suppression in hands-free telephones H04M9/08

Abstract

System for improving audiovisual communication, the system including a line of sight determining system for determining a user line of sight of the eyes of a user, and an image display system coupled with the line of sight determining system, the image display system displaying a supplementary image for the eyes, and controlling a property of the supplementary image and of a scene image, according to the determined user line of sight.

Claims

1. System for improving audiovisual communication, the system comprising:

a line-of-sight determining system for determining a user line-of-sight of at least one eye of a user; and

an image display system coupled with said line-of-sight determining system, said image display system configured to display a first scene image including at least one object within said line of sight of the user, and at least one supplementary image for said at least one eye, according to said user line of sight , said supplementary image including information relating to said at least one object being directly viewed by the user and included in the first scene image, wherein said image display system is further configured to control at least one property of said at least one supplementary image and of said scene image, according to said determined user line-of-sight.

2. The system according to claim 1, further comprising a multi-dimensional audio system coupled with said line-of-sight determining system, said multi-dimensional audio system producing multi-dimensional sound to be heard by said user, according to the location of at least one input signal relative to the position and orientation of the head of said user.

3. The system according to claim 1, further comprising a directional voice system for processing the voice of said user according to said determined user line-of-sight.

4. The system according to claim 1, wherein a microphone receives an interrogation respective of said scene image from said user, respective of said object at which said user is looking directly at.

5. The system according to claim 4, wherein said microphone is directional and is located inside a conical volume which defines a dissipation geometry of sound waves generated by the speech of said user.

6. The system according to claim 4, wherein a loudspeaker produces a verbal sound for said user, according to said interrogation, said verbal sound being respective of said object and associated with said determined user line-of-sight.

7. The system according to claim 4, wherein said image display system displays said at least one supplementary image, according to said interrogation, said at least one supplementary image being respective of said scene image and associated with said determined user line-of-sight, and

wherein said image display system displays said at least one supplementary image on a location on a visor of a head-mounted device, which is on a current line-of-sight of said user and said scene image.

8. The system according to claim 4, wherein said image display system displays said at least one supplementary image, according to said interrogation, said at least one supplementary image being respective of said scene image and associated with the position and orientation of the head of said user, said position and orientation being determined by a head tracking system, and

wherein said image display system displays said at least one supplementary image on a location on a visor of a head-mounted device, which is substantially aligned with said position and orientation and said scene image.

9. The system according to claim 4, wherein said image display system displays said at least one supplementary image, according to said interrogation, said at least one supplementary image being respective of said scene image and associated with said determined user line-of-sight, and

wherein said image display system displays said at least one supplementary image on a location on a windshield of a vehicle, which is on a current line-of-sight of said user and said scene image.

10. The system according to claim 4, wherein said image display system displays said at least one supplementary image, according to said interrogation, said at least one supplementary image being respective of said scene image and associated with the position and orientation of the head of said user, said position and orientation being determined by said line-of-sight determining system, and

wherein said image display system displays said at least one supplementary image on a location on a windshield of a vehicle, which is substantially aligned with said position and orientation and said scene image.

11. The system according to claim 1, wherein a memory records verbal information provided by said user via a microphone, while said user is gazing at said scene image, said verbal information being respective of said scene image and associated with said determined user line-of-sight.

12. The system according to claim 1, further comprising:

a directional camera coupled with said line-of-sight determining system; and

at least one moving mechanism coupled with said directional camera,

wherein said at least one moving mechanism moves said directional camera according to said determined user line-of-sight.

13. The system according to claim 1, wherein said line-of-sight determining system comprises and eye tracker, said eye tracker is coupled with a head-mounted device of said user, said eye tracker determining said user line-of-sight in a first coordinate system.

14. The system according to claim 13, wherein said line-of-sight determining system further includes a head tracking system, said head tracking system being coupled with said eye tracker and with said head-mounted device,

wherein said head tracking system determines the position and orientation of said user in a second coordinate system, and

wherein said eye tracker determines said user line-of-sight in said second coordinate system, according to said position and orientation.

15. The system according to claim 14, further comprising:

a back-up positioning system; and

a processor coupled with said back-up positioning system, said line-of-sight determining system and with said image display system,

wherein said back-up positioning system determines said position and orientation in said second coordinate system, and

wherein said processor controls the operation of said line-of-sight determining system and of said image display system.

16. The system according to claim 14, further comprising:

a back-up positioning system; and

a processor coupled with said back-up positioning system, said line-of-sight determining system and with said image display system.

17. The system according to claim 1, further comprising an audio communication noise reduction system, wherein said audio communication noise reduction system produces a substantially noise-free signal of an acoustic sound, said audio communication noise reduction system produces at least one sound, said at least one sound including a desired sound and an anti-phase noise sound, said anti-phase noise sound being in anti-phase relative to a noise, said audio communication noise reduction system comprising:

an acoustoelectric transducer for producing a noise bearing sound signal by detecting said acoustic sound and said noise;

a reference-acoustoelectric transducer for producing a reference noise signal by detecting said noise in a noisy environment; and

an audio controller coupled with said reference-acoustoelectric transducer and said acoustoelectric transducer,

wherein said audio controller produces said substantially noise-free signal, according to said reference noise signal and said noise bearing sound signal.

18. The system according to claim 1, wherein said audio communication noise reduction system, said line-of-sight determining system and said image display system are coupled together via a processor and further coupled with a head-mounted device.

19. The system according to claim 1, further comprising:

a communication interface coupled with said processor;

a network coupled with said communication interface; and

at least one stationary computer coupled with said network,

wherein said processor controls the operation of said line-of-sight determining system and of said image display system, according to signals received from said communication interface.

20. The system according to claim 19, wherein said at least one stationary computer is selected from the list consisting of:

local on-board computer;

remote on-board computer;

ground based computer; and

satellite based computer.

21. The system according to claim 1, wherein said at least one property is selected from the list consisting of:

location of said at least one supplementary image relative to said determined user line-of-sight;

light intensity of said at least one supplementary image; and

light intensity of said scene image.

22. The system according to claim 1, wherein said supplementary image is configured to include text relating to said at least one object being directly viewed by the user and included in the first scene image.

23. Method for improving audiovisual communication, the method comprising the procedures of:

determining the user line-of-sight of at least one eye of a user

displaying a first scene image including at least one object within said line of sight of the user, and at least one supplementary image for said at least one eye, according to said user line of sight, said supplementary image including information relating to said at least one object directly viewed by the user and included in the first scene image, and

controlling at least one property of said at least one supplementary image and of said scene image, according to said determined user line-of-sight.

24. The system according to claim 23, wherein said supplementary image is configured to include text relating to said at least one object being directly viewed by the user and included in the first scene image.