Image changes based on viewer's gaze

• US 9,990,035 B2
• Filed: 03/14/2016
• Issued: 06/05/2018
• Est. Priority Date: 03/14/2016
• Status: Active Grant

• Current Assignee: Jeffrey T. Haley
• Original Assignee: Robert L. Richmond
• Inventors: Richmond, Robert L, Haley, Jeffrey, Schworer, Jr., Thomas Leonard, Richmond, Maxwell Wolfe, Richmond, Eli Newton, Richmond, Owen Sagan
• Primary Examiner(s): Xiao, Ke
• Assistant Examiner(s): LIU, GORDON G
• Application Number:
  US15/069,829
• Publication Number:
  US 20170262052A1
• Time in Patent Office: 813 Days
• Field of Search: 345619
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0487 using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text

Abstract

An embodiment combines the concepts of image enhancement and gaze tracking to enhance the experience of viewing images by tracking where the viewer is looking. The result is to make the viewing experience more like viewing the original scene, or to enhance the viewing experience in new ways beyond the original experience, either automatically, or by interacting with a photographer'"'"'s previously specified intentions for what should happen when the viewer looks at a particular portion of an image or images taken by that photographer.

Claims

1. A method, comprising:

displaying a first image of a scene with a display screen;

capturing an image of a face of a viewer with an image-capture device as the viewer gazes at a location within the first image, a portion of the first image at the location having a sharpness;

determining, with computing circuitry, the location in response to the captured image of the face of the viewer;

searching, with computing circuitry, electronically stored images of one or more portions of the scene for an image portion corresponding to the portion of the first image and having a sharpness that is different than the sharpness of the portion of the first image;

modifying, with computing circuitry, the sharpness at the location by replacing the portion of the first image with the corresponding image portion;

capturing another image of the face of the viewer with an image-capture device;

determining, with computing circuitry in response to the captured other image of the face of the viewer, whether the viewer is gazing at the location within the first image; and

modifying again, with computing circuitry, the sharpness at the location by replacing the corresponding image portion with the portion of the first image in response to the computing circuitry determining that the viewer is no longer gazing at the location within the first image.

2. The method of claim 1 where the modifying includes replacing the first image with a second image that includes the corresponding image portion, where the contents of the second image are the same as the contents of the first image but with some or all pixels modified.

3. The method of claim 1 where the modifying includes changing the sharpness of closer objects within the location that appear to be in front of farther objects within the location.

4. The method of claim 1 where the modifying includes changing the sharpness of farther objects within the location that appear to be behind closer objects within the location.

5. The method of claim 1 where the modifying the sharpness includes modifying the apparent depth of field at the location.

6. The method of claim 1 where:

the determining includes determining, with computing circuitry and in response to the captured image, whether eyes of the viewer are diverging as if the viewer is looking at a distant object; and

in response to determining that the eyes of the viewer are diverging as if the viewer is looking at a distant object, modifying the sharpness at the location by decreasing the sharpness of closer objects at the location and increasing the sharpness of farther objects at the location.

7. The method of claim 1 wherein the modifying includes increasing the sharpness at the location to generate a region of sharper view, and moving the region of sharper view across the display screen to draw a gaze of the viewer across the display screen.

8. The method of claim 1 where the first image is a still image.

9. The method of claim 1 where the first image is part of a video stream of images.

10. The method of claim 1 where the first image is a still image with moving sprites.

11. The method of claim 1 where the first image is two dimensional.

12. The method of claim 1 where the first image is three dimensional.

13. The method of claim 1 where the modifying is according to an algorithm that is custom determined by an artist who selected the first image.

14. The method of claim 1 where the modifying is according to an algorithm that is pre-determined by a company that supplied software for selection and use by an artist or content creator who selected the first image.

15. The method of claim 1 where the modifying is according to an algorithm that is pre-determined by a company that supplied software and the algorithm works with any image or set of images.

16. The method of claim 1 where the modifying is according to an algorithm that allows an artist or content creator to specify a speed at which a gradual change modification will be performed.

17. The method of claim 1 wherein, as the sharpness at the location is modified, the sharpness at at least one other location within the first image is not modified in a same way with respect to a way in which the sharpness at the location is modified.

18. A method, comprising:

displaying a first image of a scene with a display screen;

capturing an image of a face of a viewer with an image-capture device as the viewer gazes at a location within the first image, a portion of the first image at the location having a brightness;

determining, with computing circuitry, the location in response to the captured image of the face of the viewer;

searching, with computing circuitry, a database of images of one or more portions of the scene for an image portion corresponding to the portion of the first image and having a brightness that is different than the brightness of the portion of the first image;

modifying, with computing circuitry, the brightness at the location by substituting the portion of the first image with the corresponding image portion from the database;

capturing another image of the face of the viewer with an image-capture device;

determining, with computing circuitry in response to the captured other image of the face of the viewer, whether the viewer is gazing at the location within the first image; and

modifying again, with computing circuitry, the brightness at the location by replacing the corresponding image portion with the portion of the first image in response to the computing circuitry determining that the viewer is no longer gazing at the location within the first image.

19. The method of claim 18 where the modifying includes replacing the first image with a second image that includes the corresponding image portion, where the contents of the second image are the same as the contents of the first image but with some or all pixels modified.

20. The method of claim 19 where the first and second images are a pair of photographic images taken with a camera where the first and second images are substantially the same except for a difference in respective exposures of the pair of photographic images.

21. The method of claim 18 where the first image is one of a series of video images.

22. The method of claim 18 where the modifying is according to an algorithm that is custom determined by an artist who selected the first image.

23. The method of claim 18 where the modifying is according to an algorithm that is pre-determined by a company that supplied software for selection and use by an artist or content creator who selected the first image.

24. The method of claim 18 where the modifying is according to an algorithm that is pre-determined by a company that supplied software and the algorithm works with any image or set of images.

25. The method of claim 18 where the modifying is according to an algorithm that allows an artist or content creator to specify a speed at which a gradual-change modification will be performed.

26. The method of claim 18 wherein, as the brightness at the location is modified, the brightness at at least one other location within the first image is not modified in a same way with respect to a way in which the brightness at the location is modified.

27. A method, comprising:

displaying an original image of a scene with a display screen;

capturing an image of a face of a viewer with an image-capture device as the viewer gazes at a region of the displayed original image, the region having a color balance, color saturation, color depth, or contrast;

determining, with computing circuitry, the region of the displayed original image in response to the captured image of the face of the viewer;

searching, with computing circuitry, a database of images of one or more regions of the scene for an image region corresponding to the region of the displayed original image and having at least one of a color balance, color saturation, color depth, and contrast that is different than a respective at least one of the color balance, color saturation, color depth, and contrast of the region of the displayed original image;

modifying, with computing circuitry, at least one of the color balance, the color saturation, the color depth, and the contrast of the region of the displayed original image by replacing the region with the corresponding image region from the database;

capturing another image of the face of the viewer with an image-capture device;

determining, with computing circuitry in response to the captured other image of the face of the viewer, whether the viewer is gazing at the region of the displayed original image; and

returning each of the modified at least one of the color balance, the color saturation, the color depth, and the contrast of the region of the displayed original image to a respective pre-modified value if the computing circuitry determines that the viewer is no longer gazing at the region of the displayed original image.

28. The method of claim 27 where the modifying includes replacing the original image with a substitute image that includes the corresponding image region, where the contents of the substitute image are the same as the contents of the original image but with some or all pixels within the corresponding image region of the substitute image modified relative to the region of the original image.

29. The method of claim 27 where the original image is a still image.

30. The method of claim 27 where the original image is one of a series of video images.

31. The method of claim 27 where the modifying is according to an algorithm that is custom determined by an artist who selected the original image.

32. The method of claim 27 where the modifying is according to an algorithm that is pre-determined by a company that supplied software for selection and use by an artist or content creator who selected the original image.

33. The method of claim 27 where the modifying is according to an algorithm that is pre-determined by a company that supplied software and the algorithm works with any image or set of images.

34. The method of claim 27 where the modifying is according to an algorithm that allows an artist or content creator to specify a speed at which a gradual-change modification will be performed.

35. A method, comprising:

displaying an original image with a display screen;

capturing an image of a face of a viewer with an image-capture device as the viewer gazes at a location within the displayed original image;

determining, with computing circuitry, the location within the displayed original image in response to the captured image of the face of the viewer;

searching, with computing circuitry, electronically stored sprites for a moving sprite corresponding to a still portion of the displayed original image at the location; and

modifying, with computing circuitry, the displayed original image by replacing the still portion of the displayed original image at the location with the corresponding moving sprite such that the moving sprite moves only at the location within the displayed original image.

36. A method, comprising:

displaying an original image with a display screen;

capturing an image of a face of a viewer with an image-capture device as the viewer gazes at a location within the displayed original image;

determining, with computing circuitry, the location within the displayed original image in response to the captured image of the face of the viewer;

searching, with computing circuitry, a database for a multi-branch video corresponding to a portion of the displayed original image at the location;

modifying, with computing circuitry, the displayed original image by selecting a branch of the corresponding multi-branch video, replacing the portion of the displayed original image at the location with the selected branch of the corresponding multi-branch video, and rendering, in a viewer-perceivable manner, the selected branch of the multi-branch video within the location;

capturing another image of the face of the viewer with an image-capture device;

determining, with computing circuitry in response to the captured other image of the face of the viewer, whether the viewer is gazing at the location within the displayed original image; and

returning, with computing circuitry, the portion of the displayed original image to the location within the displayed original image in response to the computing circuitry determining that the viewer is no longer gazing at the location within the displayed original image.

37. The method of claim 36 wherein replacing the portion of the original image at the location includes replacing a still portion of the original image at the location.

38. A method, comprising:

displaying an original image with a display screen;

capturing an image of a face of a viewer with an image-capture device as the viewer gazes at a location within the original image;

determining, with computing circuitry, the location within the original image in response to the captured image of the face of the viewer;

searching, with computing circuitry, electronically stored images for a still-image portion corresponding to the location within the original image; and

modifying, with computing circuitry, the original image by replacing a moving sprite at the location with the corresponding still-image portion.