Visual cognition aware display and visual data transmission architecture

• US 7,850,306 B2
• Filed: 08/28/2008
• Issued: 12/14/2010
• Est. Priority Date: 08/28/2008
• Status: Active Grant

• Current Assignee: Nokia Technologies Oy (Nokia Corporation)
• Original Assignee: Nokia Corporation
• Inventors: Oksanen, Markku A., Aaltonen, Viljakaisa, Jarvenpaa, Toni J., Uusitalo, Mikko A.
• Primary Examiner(s): Hasan; Mohammed
• Application Number:
  US12/231,227
• Publication Number:
  US 20100056274A1
• Time in Patent Office: 838 Days
• Field of Search: 351200-246
• US Class Current: 351/209
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: G02B 2027/0112 comprising device for genereting colour display
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0147 comprising a device modifying the resolution of the displayed image
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence

Abstract

Methods, apparatus and computer program products reduce bandwidth requirements in mobile video and gaming applications by tracking a gaze fixation point of a user on a video image at a remote location; generating information identifying the gaze fixation point at the remote location; transmitting the gaze fixation point information to a source of video information over a network; determining a portion of a video image to transmit at a high resolution and a remaining portion to transmit at a low resolution using the transmitted gaze fixation point information; and transmitting the portion of the video image selected for transmission at a high resolution at the high resolution and the portion of the video image selected for transmission at a low resolution at the low resolution. Further methods, apparatus and computer program product also reduce bandwidth requirements by receiving gaze fixation point information from a remote location, wherein the gaze fixation point information indicates where in a video image a user is currently looking; selecting a portion of video images to transmit at a high resolution and a portion of video images to transmit at a low resolution using the transmitted gaze fixation point information; and transmitting the portions of the video images at the determined resolutions.

Claims

1. A method comprising:

tracking a gaze fixation point of a user on dynamic imagery;

generating gaze fixation point information identifying a current gaze fixation point of the user; and

transmitting the gaze fixation point information to a source of the dynamic imagery

where tracking the gaze fixation point of the user on the dynamic imagery occurs at a remote location and wherein transmitting the gaze fixation point information to the source of the dynamic imagery further comprises transmitting the gaze fixation point information to a remote source of the dynamic imagery over a network.

2. The method of claim 1 wherein generating the gaze fixation point information identifying the current gaze fixation point of the user further comprises generating new gaze fixation point information at a rate greater than a frame rate of the dynamic imagery.

3. The method of claim 1 wherein tracking the gaze fixation point of the user on the dynamic imagery further comprises detecting eye movement greater than a pre-determined threshold, and suspending generation of new gaze fixation point information while the eye continues to move at a rate greater than the pre-determined threshold.

4. The method of claim 1 wherein the dynamic imagery further comprises video imagery.

5. The method of claim 1 wherein the dynamic imagery further comprises dynamic imagery generated by a game.

6. The method of claim 1, wherein transmitting the gaze fixation point information to the remote source of the dynamic imagery over the network further comprises transmitting the gaze fixation point information over a wireless network.

7. The method of claim 1, wherein transmitting the gaze fixation point information to the remote source of the dynamic imagery over the network further comprises transmitting the gaze fixation point information over a wired network.

8. The method of claim 1, the gaze fixation point information is used by the remote source of the dynamic imagery to selectively modify images comprising the dynamic imagery, wherein the method further comprises:

receiving the dynamic imagery as modified using the gaze fixation point information; and

displaying the modified dynamic imagery.

9. The method of claim 8 wherein the remote source of the dynamic imagery uses the gaze fixation point information to selectively modify resolutions of portions of each image comprising the dynamic imagery.

10. The method of claim 8 wherein the remote source of the dynamic imagery uses the gaze fixation point information to selectively modify color content of portions of each image comprising the dynamic imagery.

11. The method of claim 1, wherein the gaze fixation point information is used by the remote source of dynamic imagery to select a portion of each image comprising the dynamic imagery to be transmitted at a high resolution and a portion of each image comprising the dynamic imagery to be transmitted at a low resolution, wherein the method further comprises:

receiving the portions of each image transmitted respectively at high and low resolutions over a wireless network; and

displaying the respective image portions at the selected resolutions.

12. A method comprising:

receiving from a remote location gaze fixation point information identifying where in dynamic imagery a user is currently looking;

using the gaze fixation point information to selectively modify dynamic imagery next to be transmitted to the user; and

transmitting the dynamic imagery to the user as modified using the gaze fixation point information, wherein the gaze fixation point information is also received from a plurality of users, wherein the gaze fixation point information received from each user is used to determine how to selectively modify the dynamic imagery next to be transmitted to that particular user, wherein the dynamic imagery is transmitted to that particular user as modified using the gaze fixation point information received from that particular user.

13. The method of claim 12 wherein using the gaze fixation point information to selectively modify dynamic imagery next to be transmitted to the user further comprises selectively modifying the resolution of portions of each image comprising the dynamic imagery using the gaze fixation point information.

14. The method of claim 13 where selectively modifying the resolution of portions of each image comprising the dynamic image using the gaze fixation point information further comprises selectively varying the resolution of the portions in dependence on the respective distances of the portions from the gaze fixation point.

15. The method of claim 12 wherein using the gaze fixation point information to selectively modify dynamic imagery next to be transmitted to the user further comprises selectively modifying each image comprising the dynamic imagery to reduce color content in portions of each image that are determined to be in the peripheral vision of a viewer.

16. The method of claim 12 wherein the dynamic imagery further comprises video imagery.

17. The method of claim 12 wherein the dynamic imagery further comprises dynamic imagery generated by a game.

18. The method of claim 12 wherein transmitting the dynamic imagery to the user as modified using the gaze fixation point information further comprises transmitting the modified dynamic imagery over a wireless network.

19. The method of claim 12 wherein transmitting the dynamic imagery to the user as modified using the gaze fixation point information further comprises transmitting the modified dynamic imagery over a wired network.

20. An apparatus comprising:

an eye tracker configured to track a gaze fixation point of an eye on dynamic imagery, the eye tracker further configured to generate gaze fixation point information identifying a current gaze fixation point of a user; and

communications apparatus configured to transmit the gaze fixation point information to a remote source of dynamic imagery, where the communications apparatus is further configured to transmit the dynamic imagery wherein at least a portion of each image comprising the dynamic imagery is transmitted at a high resolution and at least a portion of each image comprising the dynamic imagery is transmitted at a low resolution, and wherein the gaze fixation point information is used to determine what portions of individual images to transmit at high and low resolutions.

21. The apparatus of claim 20 wherein the eye tracker is further configured to generate new gaze fixation point information identifying a current gaze fixation point of a user at a rate greater than a frame rate of the dynamic imagery.

22. The apparatus of claim 20 wherein the eye tracker is further configured to detect eye movement greater than a pre-determined threshold, and to suspend generation of new gaze fixation point information while the eye continues to move at a rate greater than the pre-determined threshold.

23. The apparatus of claim 20 wherein the dynamic imagery further comprises video imagery.

24. The apparatus of claim 20 wherein the dynamic imagery further comprises dynamic imagery generated by a game.

25. The apparatus of claim 20 wherein the communications apparatus further comprises a wireless transceiver configured to transmit the gaze fixation point information to the remote source of dynamic imagery over a wireless network.

26. An apparatus comprising:

communications apparatus configured to receive gaze fixation point information from a remote location, wherein the gaze fixation point information indicates where in dynamic imagery a user is currently looking, and configured to receive the gaze fixation point information from a plurality of users;

image modification apparatus configured to receive the gaze fixation point information from the communication apparatus, to use the gaze fixation point information to selectively modify dynamic imagery next to be transmitted to the user, and to receive gaze fixation point information from each user of the plurality of users to determine how to selectively modify the dynamic imagery next to be transmitted to that particular user; and

wherein the communications apparatus is further configured to transmit the dynamic imagery as modified using the gaze fixation point information and is configured to transmit to that particular user the dynamic imagery as modified using the gaze fixation point information received from that particular user.

27. The apparatus of claim 26 wherein the dynamic imagery further comprises video imagery.

28. The apparatus of claim 26 wherein the dynamic imagery further comprises dynamic imagery generated by a game.

29. The apparatus of claim 26 wherein the communications apparatus further comprises wireless communications apparatus configured to transmit and to receive information over a wireless network.

30. The apparatus of claim 26 wherein the communications apparatus is further configured to transmit and to receive information over a wired network.

31. A non-transitory computer readable storage medium tangibly embodying a computer program, which, when executed by a digital processing apparatus, performs actions as follows:

tracking a gaze fixation point of a user on dynamic imagery;

generating gaze fixation point information identifying a current gaze fixation point of the user; and

controlling an apparatus to transmit the gaze fixation point information to a source of dynamic imagery,

where tracking the gaze fixation point of the user on the dynamic imagery occurs at a remote location and wherein transmitting the gaze fixation point information to the source of the dynamic imagery further comprises transmitting the gaze fixation point information to a remote source of the dynamic imagery over a network.

32. The non-transitory computer readable storage medium of claim 31, further comprising operating the apparatus to generate gaze fixation point information identifying a current gaze fixation point of the user at a rate greater than a frame rate of the dynamic imagery.

33. The non-transitory computer readable storage medium of claim 31, further comprising operating the apparatus to detect eye movement greater than a pre-determined threshold, and to suspend generation of new gaze fixation point information while the eye continues to move at a rate greater than the pre-determined threshold.

34. The non-transitory computer readable storage medium of claim 31, wherein the dynamic imagery further comprises video imagery.

35. The non-transitory computer readable storage medium of claim 31, wherein the dynamic imagery further comprises dynamic imagery generated by a game.

36. The non-transitory computer readable storage medium of claim 31, wherein transmitting the gaze fixation point information to the source of the dynamic imagery over a network further comprises transmitting the gaze fixation point information over a wireless network.

37. The non-transitory computer readable storage medium of claim 31, wherein transmitting the gaze fixation point information to the source of the dynamic imagery over a network further comprises transmitting the gaze fixation point information over a wired network.

38. The non-transitory computer readable storage medium of claim 31, wherein the gaze fixation point information is used by the source of dynamic imagery to select a portion of each image comprising the dynamic imagery to be transmitted at a high resolution and a portion of each image comprising the dynamic imagery to be transmitted at a low resolution, further comprising receiving the portions of each image transmitted at high and low resolutions over a wireless network;

and displaying the respective image portions at the selected resolutions.

39. The non-transitory computer readable storage medium of claim 31, wherein the gaze fixation point information is used by the source of the dynamic imagery to select a portion of each image comprising the dynamic imagery to be transmitted at a high resolution and a portion of each image comprising the dynamic imagery to be transmitted at a low resolution, further comprising receiving the portions of each image transmitted respectively at high and low resolutions over a wired network;

and displaying the respective image portions at the selected resolutions.

40. A non-transitory computer readable storage medium tangibly embodying a computer program, which, when executed by a digital processing apparatus, performs actions as follows:

receiving from a remote location gaze fixation point information from a remote location identifying where in dynamic imagery a user is currently looking;

selectively modifying dynamic imagery next to be transmitted to the remote location to reduce image content using the gaze fixation point information; and

transmitting the dynamic imagery to the user as modified using the gaze fixation point information,

wherein the gaze fixation point information is also received from a plurality of users, wherein the gaze fixation point information received from each user is used to determine how to selectively modify the dynamic imagery next to be transmitted to that particular user, wherein the dynamic imagery is transmitted to that particular user as modified using the gaze fixation point information received from that particular user.

41. The non-transitory computer readable storage medium of claim 40, wherein the dynamic imagery further comprises video imagery.

42. The non-transitory computer readable storage medium of claim 40, wherein the dynamic imagery further comprises dynamic imagery generated by a game.

43. The non-transitory computer readable storage medium of claim 40, wherein selectively modifying dynamic imagery further comprises reducing resolution in portions of images comprising the dynamic imagery that are determined to be in the peripheral vision of the user with the gaze fixation point information.

44. The non-transitory computer readable storage medium of claim 40, wherein selectively modifying dynamic imagery further comprises reducing color content in portions of images comprising the dynamic imagery that are determined to be in the peripheral vision of the user with the gaze fixation point information.