COMPOSING AND REALIZING A VIEWER'S INTERACTION WITH DIGITAL MEDIA

• US 20170262052A1
• Filed: 03/14/2016
• Published: 09/14/2017
• Est. Priority Date: 03/14/2016
• Status: Active Grant

• Current Assignee: Jeffrey T. Haley
• Original Assignee: Jeffrey T. Haley
• Inventors: Richmond, Robert L, Haley, Jeffrey, Schworer, JR., Thomas Leonard, Richmond, Maxwell Wolfe, Richmond, Eli Newton, Richmond, Owen Sagan
• Granted Patent: US 9,990,035 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0487 using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text

Abstract

An embodiment combines the concepts of image enhancement and gaze tracking to enhance the experience of viewing images by tracking where the viewer is looking. The result is to make the viewing experience more like viewing the original scene, or to enhance the viewing experience in new ways beyond the original experience, either automatically, or by interacting with a photographer'"'"'s previously specified intentions for what should happen when the viewer looks at a particular portion of an image or images taken by that photographer.

Claims

1. A method, comprising:

displaying a first image of a scene with a display screen;

capturing an image of a face of a viewer with an image-capture device as the viewer gazes at a location within the first image, a portion of the first image at the location having a sharpness;

determining, with computing circuitry, the location in response to the captured image of the face of the viewer; and

modifying, with computing circuitry, the sharpness at the location by replacing the portion of the first image with a corresponding portion of a second image of the scene, the corresponding portion having a sharpness that is different than the sharpness of the portion.

7. The method of claim 1 where the modifying includes replacing the first image with the second image where the contents of the second image are the same as the contents of the first image but with some or all pixels modified.

8. The method of claim 1 where the modifying includes changing the sharpness of closer objects within the location that appear to be in front of farther objects within the location.

9. The method of claim 1 where the modifying includes changing the sharpness of farther objects within the location that appear to be behind closer objects within the location.

10. The method of claim 1 where the modifying the sharpness includes modifying the apparent depth of field at the location.

11. The method of claim 1 where:

the determining includes determining, with computing circuitry and in response to the captured image, whether eyes of the viewer are diverging as if the viewer is looking at a distant object; and

in response to determining that the eyes of the viewer are diverging as if the viewer is looking at a distant object, modifying the sharpness at the location by decreasing the sharpness of closer objects at the location and increasing the sharpness of farther objects at the location.

12. The method of claim 1 wherein the modifying includes increasing the sharpness at the location to generate a region of sharper view, and moving the region of sharper view across the display screen to draw a gaze of the viewer across the display screen.

22. The method of claim 1 where the first image is a still image.

23. The method of claim 1 where the first image is part of a video stream of images.

24. The method of claim 1 where the first image is a still image with moving sprites.

25. The method of claim 1 where the first image is two dimensional.

26. The method of claim 1 where the first image is three dimensional.

27. The method of claim 1 where the modifying is according to an algorithm that is custom determined by an artist who selected the first image.

28. The method of claim 1 where the modifying is according to an algorithm that is pre-determined by a company that supplied software for selection and use by an artist or content creator who selected the first image.

29. The method of claim 1 where the modifying is according to an algorithm that is pre-determined by a company that supplied software and the algorithm works with any image or set of images.

30. The method of claim 1 where the modifying is according to an algorithm that allows an artist or content creator to specify a speed at which a gradual change modification will be performed.

60. The method of claim 1 wherein, as the sharpness at the location is modified, the sharpness at at least one other location within the first image is not modified in a same way with respect to a way in which the sharpness at the location is modified.

2-3. -3. (canceled)

4. A method, comprising:

displaying a first image of a scene with a display screen;

capturing an image of a face of a viewer with an image-capture device as the viewer gazes at a location within the first image, a portion of the first image at the location having a brightness;

determining, with computing circuitry, the location in response to the captured image of the face of the viewer; and

modifying, with computing circuitry, the brightness at the location by substituting the portion of the first image with a corresponding portion of a second image of the scene, the corresponding portion having a brightness that is different from the brightness of the portion.

5. The method of claim 4 where the modifying includes replacing the first image with the second image where the contents of the second image are the same as the contents of the first image but with some or all pixels modified.

6. The method of claim 5 where the first and second images are a pair of photographic images taken with a camera where the first and second images are substantially the same except for a difference in respective exposures of the pair of photographic images.

49. The method of claim 4 where the first image is one of a series of video images.

50. The method of claim 4 where the modifying is according to an algorithm that is custom determined by an artist who selected the first image.

51. The method of claim 4 where the modifying is according to an algorithm that is pre-determined by a company that supplied software for selection and use by an artist or content creator who selected the first image.

52. The method of claim 4 where the modifying is according to an algorithm that is pre-determined by a company that supplied software and the algorithm works with any image or set of images.

53. The method of claim 4 where the modifying is according to an algorithm that allows an artist or content creator to specify a speed at which a gradual-change modification will be performed.

61. The method of claim 4 wherein, as the brightness at the location is modified, the brightness at at least one other location within the first image is not modified in a same way with respect to a way in which the brightness at the location is modified.

13-16. -16. (canceled)

17. A method, comprising:

displaying an original image of a scene with a display screen;

capturing an image of a face of a viewer with an image-capture device as the viewer gazes at a region of the displayed original image, the region having a color balance, color saturation, color depth, or contrast;

determining, with computing circuitry, the region of the displayed original image in response to the captured image of the face of the viewer; and

modifying, with computing circuitry, at least one of the color balance, the color saturation, the color depth, and the contrast of the region of the original image by replacing the region with a corresponding region of a substitute image of the scene.

18. The method of claim 17 where the modifying includes replacing the original image with the substitute image where the contents of the substitute image are the same as the contents of the original image but with some or all pixels within the corresponding region of the substitute image modified relative to the region of the original image.

54. The method of claim 17 where the original image is a still image.

55. The method of claim 17 where the original image is one of a series of video images.

56. The method of claim 17 where the modifying is according to an algorithm that is custom determined by an artist who selected the original image.

57. The method of claim 17 where the modifying is according to an algorithm that is pre-determined by a company that supplied software for selection and use by an artist or content creator who selected the original image.

58. The method of claim 17 where the modifying is according to an algorithm that is pre-determined by a company that supplied software and the algorithm works with any image or set of images.

59. The method of claim 17 where the modifying is according to an algorithm that allows an artist or content creator to specify a speed at which a gradual-change modification will be performed.

19. (canceled)

20. A method, comprising:

displaying an original image with a display screen;

capturing an image of a face of a viewer with an image-capture device as the viewer gazes at a location within the displayed original image;

determining, with computing circuitry, the location within the displayed original image in response to the captured image of the face of the viewer; and

modifying, with computing circuitry, the original image by replacing a still portion of the original image at the location with a moving sprite, or by replacing a moving sprite of the original image at the location with a still portion of a substitute image.

21. A method, comprising:

displaying an original image with a display screen;

capturing an image of a face of a viewer with an image-capture device as the viewer gazes at a location within the displayed original image;

determining, with computing circuitry, the location within the displayed original image in response to the captured image of the face of the viewer; and

modifying, with computing circuitry, the original image by selecting a branch of a multi-branch video, replacing a portion of the original image at the location with the branch of the multi-branch video, and rendering, in a viewer-perceivable manner, the branch of the multi-branch video within the location.

62. The method of claim 21 wherein replacing the portion of the original image at the location includes replacing a still portion of the original image at the location.

31-48. -48. (canceled)