Systems and methods for providing audio to a user based on gaze input

• US 10,635,386 B2
• Filed: 08/21/2017
• Issued: 04/28/2020
• Est. Priority Date: 08/23/2013
• Status: Active Grant

• Current Assignee: Tobii Technology AB
• Original Assignee: Tobii Technology AB
• Inventors: Vennstrom, Anders, Lindh, Fredrik
• Primary Examiner(s): Zhai, Kyle
• Application Number:
  US15/682,149
• Publication Number:
  US 20180129469A1
• Time in Patent Office: 981 Days
• Field of Search: None
• CPC Class Codes: G02B 27/00 Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: H04S 2400/01 Multi-channel, i.e. more than two input channels, sound reproduction with two speakers wherein the multi-channel information is substantially preserved
• CPC Class Codes: H04S 2400/13 Aspects of volume control, not necessarily automatic, in stereophonic sound systems
• CPC Class Codes: H04S 2420/07 Synergistic effects of band splitting and sub-band processing
• CPC Class Codes: H04S 7/303 Tracking of listener position or orientation
• CPC Class Codes: H04S 7/307 Frequency adjustment, e.g. tone control H04S7/301 takes precedence
• CPC Class Codes: H04S 7/308 Electronic adaptation dependent on speaker or headphone connection

Abstract

According to the invention, a method for providing audio to a user is disclosed. The method may include determining, with an eye tracking device, a gaze point of a user on a display. The method may also include causing, with a computer system, an audio device to produce audio to the user, where content of the audio may be based at least in part on the gaze point of the user on the display.

Claims

1. A method for providing visual depth of field adjustment to a scene on a display device comprising:

displaying a scene on a display device, the scene comprising at least a first portion and a second portion;

determining a gaze direction of a user with an eye tracking device;

determining a gaze target where the gaze direction intersects the scene, wherein the gaze target comprises an object in the first portion of the scene;

determining a particular sound channel that is associated with the first portion of the scene;

altering a portion of a displayed scene instead of altering an entirety of the scene, modifying a depth of the second portion to be different than a depth of the first portion,

wherein modifying the depth of the second portion comprises;

simulating a ray being projected from the object in the first portion of the scene to an object in the second portion of the scene,

determining a distance between the object in the first portion of the scene and the object in the second portion of the scene,

altering a depth of the object in the second portion of the scene differently than the object in the first portion of the scene, and

causing audio to be produced to the user, wherein the audio is produced by the particular sound channel associated with the object in the first portion of the scene, and content of the audio is based at least in part on (i) the gaze target, (ii) head information of the user, and (iii) at least one or more virtual sounds produced by the object in the first portion of the scene.

2. The method of claim 1, wherein determining the distance and the direction between the gaze target and the first portion of the scene further comprises:

dividing the scene into horizontal sections; and

determining a distance between at least two of the horizontal sections.

3. The method of claim 2, wherein altering display of the first portion of the scene comprises:

altering display of each of at least one of the horizontal sections to simulate a depth of field.

4. The method of claim 1 wherein the altering of the display comprises altering display of the first portion of the scene based at least in part on the distance and the direction.

5. The method of claim 1 wherein the distance is measured from a sound producing object in the second portion of the display to the gaze target in the first portion of the display and the direction is measured from the gaze target from the sound producing object.

6. A system for providing visual depth of field adjustment to a scene comprising:

a display device for displaying a scene comprising a first portion and a second portion;

an eye tracking device for determining a gaze direction of a user; and

at least one processor for;

determining a gaze target where the gaze direction intersects the first portion of the scene and determining a particular sound channel is associated with the first portion of the scene;

and

altering display of the first portion of the scene instead of altering an entirety of the scene, wherein the altering comprises modifying a depth of the second portion to be different than a depth of the first portion,

wherein modifying the depth of the second portion comprises;

simulating a ray being projected from the object in the first portion of the scene to an object in the second portion of the scene,

determining a distance between the object in the first portion of the scene and the object in the second portion of the scene,

altering a depth of the object in the second portion of the scene differently than the object in the first portion of the scene; and

causing audio to be produced to the user, wherein the audio is produced by the particular sound channel associated with the object in the first portion of the scene, and content of the audio is based at least in part on (i) the gaze target, (ii) head information of the user, and (iii) at least one or more virtual sounds produced by the object in the first portion of the scene.

7. The system of claim 6, wherein altering display of the first portion of the scene comprises:

enlarging the gaze target relative to the first portion of the scene.

8. The system of claim 6, wherein determining the distance and the direction between the gaze target and the first portion of the scene comprises:

dividing the scene into a plurality of horizontal sections;

determining a distance between at least two of the horizontal sections;

determining which of the horizontal sections contains the gaze target;

determining which of the horizontal sections contains the first portion of the scene; and

determining the distance and the direction between the gaze target and the first portion of the scene based on at least the distance between the horizontal section that contains the gaze target and the horizontal section that contains the first portion of the scene.

9. A non-transitory computer-readable medium having instructions stored thereon executable by a computing device to cause the computing device to perform operations comprising:

displaying a scene on a display device, the scene comprising a first portion and a second portion;

determining a gaze direction of a user with an eye tracking device;

determining a gaze target where the gaze direction intersects the first portion of the scene and determining a particular sound channel that is associated with the first portion of the scene;

and

altering display of the first portion of the scene instead of altering an entirety of the scene, wherein the altering comprises or modifying a depth of the second portion to be different than a depth of the first portion, wherein modifying the depth of the second portion comprises;

simulating a ray being projected from the object in the first portion of the scene to an object in the second portion of the scene,

determining a distance between the object in the first portion of the scene and the object in the second portion of the scene,

altering a depth of the object in the second portion of the scene differently than the object in the first portion of the scene; and

causing audio to be produced to the user, wherein the audio is produced by the particular sound channel associated with the object in the first portion of the scene, and content of the audio is based at least in part on (i) the gaze target, (ii) head information of the user, and (iii) at least one or more virtual sounds produced by the object in the first portion of the scene.

10. The non-transitory computer-readable medium of claim 9, wherein determining the distance and the direction between the gaze target and the portion of the scene comprises:

simulating a first ray extending from the gaze target to the first portion of the scene; and

determining the distance and the direction based on a distance and a direction traversed by the first ray.

11. The non-transitory computer-readable medium of claim 9, wherein altering display of the portion of the scene comprises:

altering a rendering characteristic of the portion of the scene.