Visual cognition aware display and visual data transmission architecture

• US 20100056274A1
• Filed: 08/28/2008
• Published: 03/04/2010
• Est. Priority Date: 08/28/2008
• Status: Active Grant

• Current Assignee: Nokia Technologies Oy (Nokia Corporation)
• Original Assignee: Nokia Corporation
• Inventors: Uusitalo, Mikko A., Oksanen, Markku A., Aaltonen, Viljakaisa, Jarvenpaa, Toni J.
• Granted Patent: US 7,850,306 B2
• Time in Patent Office: Days
• US Class Current: 463/31
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: G02B 2027/0112 comprising device for genereting colour display
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0147 comprising a device modifying the resolution of the displayed image
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence

Abstract

Methods, apparatus and computer program products reduce bandwidth requirements in mobile video and gaming applications by tracking a gaze fixation point of a user on a video image at a remote location; generating information identifying the gaze fixation point at the remote location; transmitting the gaze fixation point information to a source of video information over a network; determining a portion of a video image to transmit at a high resolution and a remaining portion to transmit at a low resolution using the transmitted gaze fixation point information; and transmitting the portion of the video image selected for transmission at a high resolution at the high resolution and the portion of the video image selected for transmission at a low resolution at the low resolution. Further methods, apparatus and computer program product also reduce bandwidth requirements by receiving gaze fixation point information from a remote location, wherein the gaze fixation point information indicates where in a video image a user is currently looking; selecting a portion of video images to transmit at a high resolution and a portion of video images to transmit at a low resolution using the transmitted gaze fixation point information; and transmitting the portions of the video images at the determined resolutions.

Claims

1. A method comprising:

tracking a gaze fixation point of a user on dynamic imagery;

generating gaze fixation point information identifying a current gaze fixation point of the user; and

transmitting the gaze fixation point information to a source of dynamic imagery.

2. The method of claim 1 where tracking a gaze fixation point of a user on dynamic imagery occurs at a remote location and wherein transmitting the gaze fixation point information to a source of dynamic imagery further comprises transmitting the gaze fixation point information to a remote source of source dynamic imagery over a network.

3. The method of claim 1 where tracking a gaze fixation point of a user on dynamic imagery occurs at a location that is close to a local source of dynamic imagery and wherein transmitting the gaze fixation point information to a source of dynamic imagery further comprises transmitting the gaze fixation point information to the local source of source dynamic imagery over a local network.

4. The method of claim 1 wherein generating gaze fixation point information identifying a current gaze fixation point of a user further comprises generating new gaze fixation point information at a rate greater than a frame rate of the dynamic imagery.

5. The method of claim 1 wherein tracking a gaze fixation point of a user on a dynamic imagery further comprises detecting eye movement greater than a pre-determined threshold, and suspending generation of new gaze fixation point information while the eye continues to move at a rate greater than the pre-determined threshold.

6. The method of claim 1 wherein the dynamic imagery further comprises video imagery.

7. The method of claim 1 wherein the dynamic imagery further comprises dynamic imagery generated by a game.

8. The method of claim 2 wherein transmitting the gaze fixation point information to a remote source of dynamic imagery over a network further comprises transmitting the gaze fixation point information over a wireless network.

9. The method of claim 2 wherein transmitting the gaze fixation point information to a remote source of dynamic imagery over a network further comprises transmitting the gaze fixation point information over a wired network.

10. The method of claim 2 wherein the gaze fixation point information is used by the remote source of dynamic imagery to selectively modify images comprising the dynamic imagery, wherein the method further comprises:

receiving the dynamic imagery as modified using the gaze fixation point information; and

displaying the modified dynamic imagery.

11. The method of claim 10 wherein the remote source of dynamic imagery uses the gaze fixation point information to selectively modify resolutions of portions of each image comprising the dynamic imagery.

12. The method of claim 10 wherein the remote source of dynamic imagery uses the gaze fixation point information to selectively modify color content of portions of each image comprising the dynamic imagery.

13. The method of claim 2 wherein the gaze fixation point information is used by the remote source of dynamic imagery to select a portion of each image comprising the dynamic imagery to be transmitted at a high resolution and a portion of each image comprising the dynamic imagery to be transmitted at a low resolution, wherein the method further comprises:

receiving the portions of each image transmitted respectively at high and low resolutions over a wireless network; and

displaying the respective image portions at the selected resolutions.

14. A method comprising:

receiving from a remote location gaze fixation point information identifying where in dynamic imagery a user is currently looking;

using the gaze fixation point information to selectively modify dynamic imagery next to be transmitted to the user; and

transmitting the dynamic imagery to the user as modified using the gaze fixation point information.

15. The method of claim 14 wherein using the gaze fixation point information to selectively modify dynamic imagery next to be transmitted to the user further comprises selectively modifying the resolution of portions of each image comprising the dynamic imagery using the gaze fixation point information.

16. The method of claim 15 where selectively modifying the resolution of portions of each image comprising the dynamic image using the gaze fixation point information further comprises selectively varying the resolution of the portions in dependence on the respective distances of the portions from the gaze fixation point.

17. The method of claim 14 wherein using the gaze fixation point information to selectively modify dynamic imagery next to be transmitted to the user further comprises selectively modifying each image comprising the dynamic imagery to reduce color content in portions of each image that are determined to be in the peripheral vision of a viewer.

18. The method of claim 14 wherein receiving from a remote location gaze fixation point information further comprises receiving gaze fixation point information from a plurality of users;

wherein using the gaze fixation point information to selectively modify dynamic imagery next to be transmitted to the user further comprises using the gaze fixation point information received from each user to determine how to selectively modify the dynamic imagery next to be transmitted to that particular user; and

wherein transmitting the dynamic imagery further comprises transmitting to each user the dynamic imagery as modified using the gaze fixation point information received from that particular user.

19. The method of claim 14 wherein the dynamic imagery further comprises video imagery.

20. The method of claim 14 wherein the dynamic imagery further comprises dynamic imagery generated by a game.

21. The method of claim 14 wherein transmitting the dynamic imagery to the user as modified using the gaze fixation point information further comprises transmitting the modified dynamic imagery over a wireless network.

22. The method of claim 14 wherein transmitting the dynamic imagery to the user as modified using the gaze fixation point information further comprises transmitting the modified dynamic imagery over a wired network.

23. An apparatus comprising:

an eye tracker configured to track a gaze fixation point of an eye on dynamic imagery, the eye tracker further configured to generate gaze fixation point information identifying a current gaze fixation point of a user; and

communications apparatus configured to transmit the gaze fixation point information to a remote source of dynamic imagery.

24. The apparatus of claim 23 wherein the eye tracker is further configured to generate new gaze fixation point information identifying a current gaze fixation point of a user at a rate greater than a frame rate of the dynamic imagery.

25. The apparatus of claim 23 wherein the eye tracker is further configured to detect eye movement greater than a pre-determined threshold, and to suspend generation of new gaze fixation point information while the eye continues to move at a rate greater than the pre-determined threshold.

26. The apparatus of claim 23 wherein the dynamic imagery further comprises video imagery.

27. The apparatus of claim 23 wherein the dynamic imagery further comprises dynamic imagery generated by a game.

28. The apparatus of claim 23 wherein the communications apparatus further comprises a wireless transceiver configured to transmit the gaze fixation point information to the remote source of dynamic imagery over a wireless network.

29. The apparatus of claim 23 where the communications apparatus is further configured to receive dynamic imagery wherein at least a portion of each image comprising the dynamic imagery is transmitted at a high resolution and at least a portion of each image comprising the dynamic imagery is transmitted at a low resolution, and wherein the gaze fixation point information is used to determine what portions of individual images to transmit at high and low resolutions.

30. An apparatus comprising:

communications apparatus configured to receive gaze fixation point information from a remote location, wherein the gaze fixation point information indicates where in dynamic imagery a user is currently looking;

image modification apparatus configured to receive the gaze fixation point information from the communication apparatus and to use the gaze fixation point information to selectively modify dynamic imagery next to be transmitted to the user; and

wherein the communications apparatus is further configured to transmit the dynamic imagery as modified using the gaze fixation point information.

31. The apparatus of claim 30 wherein the dynamic imagery further comprises video imagery.

32. The apparatus of claim 30 wherein the dynamic imagery further comprises dynamic imagery generated by a game.

33. The apparatus of claim 30 wherein the communications apparatus further comprises wireless communications apparatus configured to transmit and to receive information over a wireless network.

34. The apparatus of claim 30 wherein the communications apparatus is further configured to transmit and to receive information over a wired network.

35. A computer program product comprising a computer readable memory medium tangibly embodying a computer program, wherein when executed by digital processing apparatus the computer program is configured to control an apparatus to track a gaze fixation point of a user on dynamic imagery;

to generate gaze fixation point information identifying a current gaze fixation point of a user; and

to control communications apparatus to transmit the gaze fixation point information to a source of dynamic imagery over a network.

36. The computer program product of claim 35 wherein when executed the computer program is further configured to operate the apparatus to generate gaze fixation point information identifying a current gaze fixation point of a user at a rate greater than a frame rate of the dynamic imagery.

37. The computer program product of claim 35 wherein when executed the computer program is further configured to operate the apparatus to detect eye movement greater than a pre-determined threshold, and to suspend generation of new gaze fixation point information while the eye continues to move at a rate greater than the pre-determined threshold.

38. The computer program product of claim 35 wherein the dynamic imagery further comprises video imagery.

39. The computer program product of claim 35 wherein the dynamic imagery further comprises dynamic imagery generated by a game.

40. The computer program product of claim 35 wherein to control communications apparatus to transmit the gaze fixation point information to a source of dynamic imagery over a network further comprises to control the communications apparatus to transmit the gaze fixation point information over a wireless network.

41. The computer program product of claim 35 wherein to control communications apparatus to transmit the gaze fixation point information to a source of dynamic imagery over a network further comprises to transmit the gaze fixation point information over a wired network.

42. The computer program product of claim 35 wherein the gaze fixation point information is used by the source of dynamic imagery to select a portion of images comprising the dynamic imagery to be transmitted at a high resolution and a portion of images comprising the dynamic imagery to be transmitted at a low resolution, wherein when executed by digital processing apparatus the computer program is further configured to control the apparatus to receive the respective image portions transmitted at high and low resolutions over a wireless network;

and to display the respective image portions at the selected resolutions.

43. The computer program product of claim 35 wherein the gaze fixation point information is used by the source of dynamic imagery to select a portion of images comprising the dynamic imagery to be transmitted at a high resolution and a portion of images comprising the dynamic imagery to be transmitted at a low resolution, wherein when executed by digital processing apparatus the computer program is further configured to control the apparatus to receive the respective image portions transmitted at high and low resolutions over a wired network;

and to display the respective image portions at the selected resolutions.

44. A computer program product comprising a computer readable memory medium tangibly embodying a computer program, wherein when executed by digital processing apparatus the computer program is configured to control an apparatus to receive gaze fixation point information from a remote location, wherein the gaze fixation point information indicates where in dynamic imagery a user is currently looking;

to selectively modify dynamic imagery next to be transmitted to the remote location to reduce image content using the gaze fixation point information; and

to control communications apparatus to transmit the dynamic imagery as modified using the gaze fixation point information.

45. The computer program product of claim 44 wherein the dynamic imagery further comprises video imagery.

46. The computer program product of claim 44 wherein the dynamic imagery further comprises dynamic imagery generated by a game.

47. The computer program product of claim 44 wherein to selectively modify dynamic imagery further comprises to reduce resolution in portions of images comprising the dynamic imagery that are determined to be in the peripheral vision of the user with the gaze fixation point information.

48. The computer program product of claim 44 wherein to selectively modify dynamic imagery further comprises to reduce color content in portions of images comprising the dynamic imagery that are determined to be in the peripheral vision of the user with the gaze fixation point information.