Image changes based on viewer's gaze

• US 10,416,765 B2
• Filed: 06/04/2018
• Issued: 09/17/2019
• Est. Priority Date: 03/14/2016
• Status: Active Grant

• Current Assignee: Jeffrey T. Haley
• Original Assignee: Jeffrey T. Haley
• Inventors: Richmond, Robert L, Haley, Jeffrey T., Schworer, Jr., Thomas Leonard, Richmond, Maxwell Wolfe, Richmond, Eli Newton, Richmond, Owen Sagan
• Primary Examiner(s): Liu, Gordon G
• Application Number:
  US15/997,468
• Publication Number:
  US 20180284888A1
• Time in Patent Office: 470 Days
• Field of Search: 345619
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0487 using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text

Abstract

An embodiment combines the concepts of image enhancement and pointing to enhance the experience of viewing images by tracking where the viewer is pointing. The result is to make the viewing experience more like viewing the original scene, or to enhance the viewing experience in new ways beyond the original experience, either automatically, or by interacting with a photographer'"'"'s previously specified intentions for what should happen when the viewer points at a particular portion of an image or images taken by that photographer.

Claims

1. A method, comprising:

displaying a first image of a scene with a display screen;

determining, with computing circuitry, a location being pointed to within the first image, a portion of the first image at the location having an image characteristic;

searching, with computing circuitry, electronically stored images of one or more portions of the scene for an image portion corresponding to the portion of the first image and having an image characteristic that is different than the image characteristic of the portion of the first image;

modifying, with computing circuitry, the image characteristic at the location by replacing the portion of the first image with the corresponding image portion;

determining, with computing circuitry, whether the location within the first image is still being pointed to; and

modifying again, with computing circuitry, the image characteristic at the location by replacing the corresponding image portion with the portion of the first image in response to the computing circuitry determining that the location is no longer being pointed to.

2. The method of claim 1 wherein determining a location includes determining, with computing circuitry, a location being pointed to within the first image with a pointing device.

3. The method of claim 1 wherein determining a location includes determining, with computing circuitry, a location being pointed to within the first image with a mouse.

4. The method of claim 1 wherein determining a location includes determining, with computing circuitry, a location being pointed to within the first image with a body part.

5. The method of claim 1 wherein determining a location includes determining, with computing circuitry, a location being pointed to within the first image with a finger.

6. The method of claim 1 wherein the image characteristic includes sharpness.

7. The method of claim 1 wherein the image characteristic includes focus.

8. The method of claim 1 wherein the image characteristic includes brightness.

9. The method of claim 1 wherein the image characteristic includes at least one of color balance, color saturation, color depth, and contrast.

10. The method of claim 1 wherein the modifying includes replacing the first image with a second image that includes the corresponding image portion, where the contents of the second image are the same as the contents of the first image but with some or all pixels modified.

11. The method of claim 10 wherein the first and second images are a pair of photographic images taken with a camera and substantially the same except for a difference in respective exposures of the pair of photographic images.

12. The method of claim 1 wherein the modifying includes changing the sharpness of closer objects within the location that appear to be in front of farther objects within the location.

13. The method of claim 1 wherein the modifying includes changing the sharpness of farther objects within the location that appear to be behind closer objects within the location.

14. The method of claim 1 wherein the modifying the image characteristic includes modifying the apparent depth of field at the location.

15. The method of claim 1 wherein the modifying includes:

increasing the sharpness at the location to generate a region of sharper view; and

moving the region of sharper view across the display screen to draw a gaze of the viewer across the display screen.

16. The method of claim 1 wherein the modifying includes replacing the first image with a substitute image that includes the corresponding image portion, where the contents of the substitute image are the same as the contents of the first image but with some or all pixels within the corresponding image portion of the substitute image modified relative to the portion of the first image.

17. The method of claim 1 wherein the first image is a still image.

18. The method of claim 1 wherein the first image is part of a video stream of images.

19. The method of claim 1 wherein the first image is a still image with moving sprites.

20. The method of claim 1 wherein the first image is two dimensional.

21. The method of claim 1 wherein the first image is three dimensional.

22. The method of claim 1 wherein the modifying is according to an algorithm that is custom determined by an artist who selected the first image.

23. The method of claim 1 wherein the modifying is according to an algorithm that is pre-determined by a company that supplied software for selection and use by an artist or content creator who selected the first image.

24. The method of claim 1 wherein the modifying is according to an algorithm that is pre-determined by a company that supplied software and the algorithm works with any image or set of images.

25. The method of claim 1 wherein the modifying is according to an algorithm that allows an artist or content creator to specify a speed at which a gradual-change modification will be performed.

26. The method of claim 1 wherein the first image is one of a series of video images.

27. The method of claim 1 wherein, as the image characteristic at the location is modified, the image characteristic at at least one other location within the first image is not modified in a same way with respect to a way in which the image characteristic at the location is modified.

28. The method of claim 1 wherein replacing the portion of the first image at the location includes replacing a still portion of the first image at the location.

29. A method, comprising:

displaying an image with a display screen;

determining, with computing circuitry, a location being pointed to within the image;

searching, with computing circuitry, electronically stored sprites for a moving sprite corresponding to a still portion of the displayed image at the location;

modifying, with computing circuitry, the displayed image by replacing the still portion of the displayed image at the location with the corresponding moving sprite such that the moving sprite moves only at the location within the displayed image.

30. A method, comprising:

displaying an image with a display screen;

determining, with computing circuitry, a location being pointed to within the image;

searching, with computing circuitry, a database for a multi-branch video corresponding to a portion of the displayed image at the location;

modifying, with computing circuitry, the displayed image by selecting a branch of the corresponding multi-branch video, replacing the portion of the displayed image at the location with the selected branch of the corresponding multi-branch video, and rendering, in a viewer-perceivable manner, the selected branch of the multi-branch video within the location; and

returning, with computing circuitry, the portion of the displayed image to the location within the displayed image in response to the computing circuitry determining that the location within the displayed image is no longer being pointed to.

31. A method, comprising:

displaying an image with a display screen;

determining, with computing circuitry, a location being pointed to within the image;

searching, with computing circuitry, electronically stored images for a still-image portion corresponding to the location within the image; and

modifying, with computing circuitry, the image by replacing a moving sprite at the location with the corresponding still-image portion.