Interactive images

• US 20060133667A1
• Filed: 11/09/2005
• Published: 06/22/2006
• Est. Priority Date: 11/30/2001
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Uyttendaele, Matthew, Toyama, Kentaro, Schoelkopf, Bernhard
• Granted Patent: US 7,444,016 B2
• Time in Patent Office: Days
• US Class Current: 382/162
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77

Abstract

A system and process for creating an interactive digital image, which allows a viewer to interact with a displayed image so as to change it with regard to a desired effect, such as exposure, focus or color, among others. An interactive image includes representative images which depict a scene with some image parameter varying between them. The interactive image also includes an index image, whose pixels each identify the representative image that exhibits the desired effect related to the varied image parameter at a corresponding pixel location. For example, a pixel of the index image might identify the representative image having a correspondingly-located pixel that depicts a portion of the scene at the sharpest focus. One primary form of interaction involves selecting a pixel of a displayed image whereupon the representative image identified in the index image at a corresponding pixel location is displayed in lieu of the currently displayed image.

Claims

1. A computer-implemented process for allowing a viewer to interact with a displayed image so as to cause the displayed image to change with regard to a desired effect, comprising:

an accessing step for accessing an interactive image, said interactive image comprising a set of representative images which depict a scene with one or more image parameters varying between images;

an inputting step for inputting an interaction mode selection made by the viewer;

an implementing step for implementing the mode of interaction by displaying one of said representative images, or a modified version thereof, on the image display device in accordance with requirements of the selected mode of interaction.

2. The process of claim 1, wherein the viewer selects an ordinal mode of interaction, and wherein the implementing step comprises:

a second inputting step for inputting a viewer selection of a particular representative image; and

a displaying step for displaying the viewer-selected representative image.

3. The process of claim 1, wherein the interactive image further comprises an index image, each pixel of which identifies the representative image that exhibits an optimum condition related to the one or more varying image parameters at a corresponding pixel location.

4. The process of claim 3, wherein the viewer selects a pixel-index mode of interaction, and wherein the implementing step comprises:

a second inputting step for inputting a viewer selection of a pixel location in a currently displayed image;

an identifying step for identifying the representative image assigned to the pixel of the index image corresponding to the viewer selected pixel location in the currently displayed image; and

a displaying step for displaying the identified representative image in lieu of the currently displayed image.

5. The process of claim 3, wherein the viewer selects a cumulative mode of interaction, and wherein the implementing step comprises:

(a) a second inputting step for inputting a first viewer selection of a pixel location in a currently displayed image;

(b) a first identifying step for identifying the representative image assigned to the pixel of the index image corresponding to the first viewer selected pixel location in the currently displayed image;

(c) a first displaying step for displaying the identified representative image on the display device in lieu of the currently displayed image;

(d) a third inputting step for inputting a subsequent viewer selection of a pixel location in the new currently displayed image;

(e) a second identifying step for identifying the representative image assigned to the pixel of the index image corresponding to the subsequently-selected pixel location in the new currently displayed image;

(f) a determining step for determining whether the currently identified representative image has been identified previously;

(g) whenever it is determined that the currently identified representative image has not been identified previously, performing a combining step for combining the currently identified representative image with the currently displayed image, and a second displaying step for displaying the combined image in lieu of the currently displayed image; and

(h) a repeating step for repeating steps (d) through (g) each time the viewer subsequently selects another pixel location in the currently displayed image.

6. The process of claim 3, wherein the viewer selects a comprehensive mode of interaction, and wherein the implementing step comprises:

a generating step for generating an image from the representative images which exhibits at each pixel the optimal condition related to the one or more parameter varied among the representative images;

a displaying step for displaying the generated image.

7. The process of claim 6, wherein the parameter varied among the representative images is the exposure level, and wherein the generating step comprises:

a maximum contrast image generating step for generating an image which exhibits a maximum contrast in each of a set of prescribed sub-regions; and

a blending step for blending the resulting contrast maximized image to smooth the transition in contrast between sub-regions.

8. The process of claim 6, wherein the parameter varied among the representative images is the depth of focus, and wherein the generating step comprises a sharpest focus image generating step for generating an image which exhibits at each pixel the sharpest focus for that pixel location.

9. The process of claim 6, wherein the parameter varied among the representative images is the color space range depicted, and wherein the generating step comprises a fully saturated color image generating step for generating an image which exhibits at each pixel a fully saturated color associated with that pixel location.

10. A computer-implemented process for allowing a viewer to interact with the video so as to cause frames thereof to exhibit a desired effect as they are being displayed, comprising:

an inputting step for inputting an interaction mode selection made by the viewer; and

for each frame of the video that is scheduled for play after the viewer interaction mode selection has been input, performing

an accessing step for accessing an interactive image associated with the frame, wherein said interactive image comprises a set of representative images which depict a scene with one or more image parameters varying between images, and

an implementing step for implementing the selected mode of interaction by displaying one of said representative images, or a modified version thereof, on the video display device in accordance with requirements of the selected mode of interaction.

11. The process of claim 10, wherein the viewer selects an ordinal mode of interaction, and wherein the implementing step comprises:

(a) a first displaying step for displaying an initial one of the representative images of the interactive image associated with the frame scheduled for play and pausing the video such that the initial representative image remains displayed pending further viewer input;

(b) a second inputting step for inputting a viewer selection of a particular representative image among the representative images making up the interactive image associated with the frame under consideration;

(c) a second displaying step for displaying the viewer-selected representative image;

(d) a repeating step for repeating steps (b) and (c) each time a subsequent viewer selection of a representative image is input;

(e) a resuming step for resuming video play upon input of a viewer request to do so.

12. The process of claim 10, wherein the interactive image further comprises an index image, each pixel of which identifies the representative image that exhibits an optimum condition related to the one or more varying image parameters at a corresponding pixel location.

13. The process of claim 12, wherein the viewer selects a pixel-index mode of interaction, and wherein the implementing step comprises:

(a) a second inputting step for inputting a viewer selection of a pixel location in a currently displayed frame of the video;

(b) an identifying step for identifying the representative image assigned to the pixel of the index image corresponding to the viewer selected pixel location of the interactive image associated with the next frame scheduled to be played;

(c) a displaying step for displaying the identified representative image at the time scheduled for playing the frame under consideration;

(d) a repeating step for repeating steps (b) and (c) for each subsequent frame scheduled of play, until another viewer-selected pixel location is input.

14. The process of claim 12, wherein the viewer selects a cumulative mode of interaction, and wherein the implementing step comprises:

(a) a second inputting step for inputting a first viewer selection of a pixel location in a currently displayed frame of the video;

(b) a first identifying step for identifying the representative image assigned to the pixel of the index image corresponding to the first viewer selected pixel location of the interactive image associated with the next frame scheduled to be played;

(c) a first displaying step for displaying the identified representative image at the time scheduled for playing said next frame;

(d) a first repeating step for repeating step (b) for each subsequent frame scheduled to play, and repeating step (c) for each subsequent frame scheduled to play until another viewer-selected pixel location is input;

(e) a third inputting step for inputting a viewer selection of a pixel location in a subsequently displayed image;

(f) second identifying step for identifying the representative image assigned to the pixel of the index image corresponding to the currently selected pixel location of the interactive image associated with the next frame scheduled to be played;

(g) a determining step for determining whether the identified representative image has been identified previously;

(h) whenever it is determined that the identified representative image has not been identified previously, performing a combining step for combining the identified representative image with all other identified representative images associated with the frame scheduled to be played next, and a second displaying step for displaying the combined image at the time scheduled for the next frame;

(i) a second repeating step for repeating steps (f) and (g) for each subsequent frame scheduled to play, and repeating step (h) for each subsequent frame scheduled to play until another viewer-selected pixel location is input;

(j) a third repeating step for repeating steps (e) through (i) each time a viewer subsequently selects a pixel location in a currently displayed image.

15. The process of claim 12, wherein the viewer selects a comprehensive mode of interaction, and wherein the implementing step comprises:

a generating step for generating, for each frame of the video prior to its being played, an image from the representative images of the interactive image associated with the frame which exhibits at each pixel the optimal condition related to the one or more parameter varied among the representative images; and

a displaying step for displaying the generated image associated with a frame at the time scheduled for the frame to be played.

16. A computer-implemented process for allowing a viewer to interact with virtual images so as to cause the virtual images to exhibit a desired effect when displayed, comprising:

a first inputting step for inputting a viewer-selected viewpoint and viewing direction;

a second inputting step for inputting a viewer-selected interaction mode;

an accessing step for accessing an interactive image that is capable of being used to generate a virtual image showing a scene from the viewpoint and viewing direction selected by the viewer, wherein said interactive image comprises a set of representative images which depict the scene with one or more image parameters varying between images; and

an implementing step for implementing the selected mode of interaction, said implementing step comprising,

a generating step for generating from one of said representative images, or a modified version thereof, the virtual image in accordance with requirements of the selected mode of interaction which shows the scene from the selected viewpoint and viewing direction, and

a displaying step for displaying the generated virtual image on an image display device.

17. The process of claim 16, wherein the viewer selects an ordinal mode of interaction, and wherein the generating step comprises:

a third inputting step for inputting a viewer selection of a particular representative image which shows the scene from the selected viewpoint and viewing direction; and

a designating step for designating the selected representative image as the virtual image.

18. The process of claim 16, wherein the interactive image further comprises an index image, each pixel of which identifies the representative image that exhibits an optimum condition related to the one or more varying image parameters at a corresponding pixel location, and wherein the viewer selects a pixel-index mode of interaction, and wherein the generating step comprises:

a third inputting step for inputting a viewer selection of a pixel location in a currently displayed image which shows the scene from the selected viewpoint and viewing direction;

an identifying step for identifying the representative image assigned to the pixel of the index image corresponding to the viewer selected pixel location in the currently displayed image; and

a designating step for designating the identified representative image as the virtual image.

19. The process of claim 16, wherein the interactive image further comprises an index image, each pixel of which identifies the representative image that exhibits an optimum condition related to the one or more varying image parameters at a corresponding pixel location, and wherein the viewer selects a cumulative mode of interaction, and wherein the implementing step comprises:

(a) a third inputting step for inputting a first viewer selection of a pixel location in a currently displayed image which shows the scene from the selected viewpoint and viewing direction;

(b) a first identifying step for identifying the representative image assigned to the pixel of the index image corresponding to the first viewer selected pixel location in the currently displayed image;

(c) a first designating step for designating the identified representative image as the current virtual image;

(d) a second displaying step for displaying the current virtual image on the display device in lieu of the currently displayed image;

(e) a fourth inputting step for inputting a subsequent viewer selection of a pixel location in the new currently displayed image;

(f) a second identifying step for identifying the representative image assigned to the pixel of the index image corresponding to the selected pixel location in the new currently displayed image;

(g) a determining step for determining whether the currently identified representative image has been identified previously;

(h) whenever it is determined that the currently identified representative image has not been identified previously, performing a combining step for combining the currently identified representative image with the currently displayed image, and a second designating step for designating the combined image as the new current virtual image;

(i) a third displaying step for displaying the new current virtual image in lieu of the currently displayed image; and

(j) a repeating step for repeating steps (e) through (i) each time a viewer subsequently selects another pixel location in the currently displayed image.

20. The process of claim 16, wherein the interactive image further comprises an index image, each pixel of which identifies the representative image that exhibits an optimum condition related to the one or more varying image parameters at a corresponding pixel location, and wherein the viewer selects a comprehensive mode of interaction, and wherein the generating step comprises:

an optimal condition image generating step for generating an image from the representative images which exhibits at each pixel the optimal condition related to the one or more parameter varied among the representative images; and

a designating step for designating the generated image as the virtual image.