Interactive images

• US 20030103670A1
• Filed: 11/30/2001
• Published: 06/05/2003
• Est. Priority Date: 11/30/2001
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Uyttendaele, Matthew T., Toyama, Kentaro, Schoelkopf, Bernhard
• Granted Patent: US 7,120,293 B2
• Time in Patent Office: Days
• US Class Current: 382/162
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77

Abstract

A system and process for creating an interactive digital image, which allows a viewer to interact with a displayed image so as to change it with regard to a desired effect, such as exposure, focus or color, among others. An interactive image includes representative images which depict a scene with some image parameter varying between them. The interactive image also includes an index image, whose pixels each identify the representative image that exhibits the desired effect related to the varied image parameter at a corresponding pixel location. For example, a pixel of the index image might identify the representative image having a correspondingly-located pixel that depicts a portion of the scene at the sharpest focus. One primary form of interaction involves selecting a pixel of a displayed image whereupon the representative image identified in the index image at a corresponding pixel location is displayed in lieu of the currently displayed image.

Claims

1. A computer-implemented process for creating an interactive image, comprising using a computer to perform the following process actions:

inputting at least one digital image of a scene;

deriving from the at least one inputted image a set of representative images which depict the scene with one or more image parameters varying between images;

computing an index image, each pixel of which identifies the representative image that exhibits a desired effect related to the varied parameter at the corresponding pixel location.

2. The process of claim 1, wherein the process action of inputting at least one digital image of the scene comprises inputting a plurality of digital images each one of which varies in said one or more parameters from the others.

3. The process action of claim 2, wherein the one or more parameters are parameters which can be varied by changing the settings on a digital camera.

4. The process of claim 3, wherein the one or more parameters are chosen from a set of variable camera parameters comprising exposure, focus, aperture size, brightness, contrast, and white balance.

5. The process of claim 1, wherein the process action of inputting at least one digital image of the scene comprises inputting at least one digital image captured using a digital camera.

6. The process of claim 1, wherein the process action of inputting at least one digital image of the scene comprises inputting at least one synthetic image.

7. The process of claim 2, wherein the process action of deriving from the plurality of inputted digital images a set of representative images which depict the scene with one or more image parameters varying between images, comprises designating the inputted images as the representative images.

8. The process of claim 2, wherein the process action of deriving from the plurality of inputted digital images a set of representative images which depict the scene with one or more image parameters varying between images, comprises the actions of:

generating additional images of the scene depicted in the inputted images by at least on of (i) interpolating or (ii) extrapolating the one or more varied parameters between the input images; and

designating the resulting images as representative images.

9. The process of claim 8, wherein the process action of deriving from the plurality of inputted digital images a set of representative images which depict the scene with one or more image parameters varying between images, further comprises an action of also designating the inputted images as representative images.

10. The process of claim 2, wherein the exposure level varies between the input images, and wherein the process action of deriving from the plurality of inputted digital images a set of representative images which depict the scene with one or more image parameters varying between images, comprises the actions of:

combining the plurality of input images into a high-dynamic-range image;

establishing a transfer function which when applied to a pixel value of the high-dynamic range image, produces pixel values falling within an exposure level range determined by a selected one of a set of exposure level center values and a prescribed factor that controls the extent of contraction or expansion of pixel values an either side of the selected center value; and

for each representative image being derived,

designating a previously unselected value from the set of exposure level center values as the selected center value of the transfer function,

applying the transfer function to each pixel of the high-dynamic-range image, and

respectively assigning each pixel value computed using the transfer function to a pixel location of the representative image under consideration that corresponds to the location of the high-dynamic-range image pixel value from which it was computed.

11. The process of claim 10, wherein the transfer function is a sigmoid transfer function.

12. The process of claim 1, wherein the process action of inputting at least one digital image of the scene comprises inputting a single digital image.

13. The process of claim 12, wherein the process action of deriving from the single inputted image a set of representative images which depict the scene with one or more image parameters varying between images, comprises the actions of:

extracting from the single input image a different portion of the image data contained therein for each representative image to be created; and

generating a representative image of the scene comprising the portion of the data extracted for that image.

14. The process of claim 13, wherein the single input image is a full color image, and wherein the process action of extracting a different portion of the image data contained in the single input image comprises the actions of:

identifying all the pixels of the input image that fall within each one of a prescribed number of color space ranges;

associating each color space range to a separate representative image; and

for each representative image,

assigning the pixel value of each pixel identified in the input image as falling into the color space range associated with that representative image to a correspondingly-located pixel location in the representative image, and

assigning a pixel value representing a color unsaturated version of each pixel not identified in the input image as falling into the color space range associated with that representative image to a correspondingly-located pixel location in the representative image.

15. The process of claim 13, wherein the single input image is a high resolution image, and wherein the process action of extracting a different portion of the image data contained in the single input image comprises the action of generating a plurality of representative images from the high resolution image wherein the generated representative images comprise a set of hierarchically ordered images ranging in resolution from high to low with the high end resolution image being lower in resolution than the high resolution input image.

16. The process of claim 1, wherein the representative images vary in respect to exposure level, and wherein the process action of computing an index image, comprises the action of computing the index image wherein each pixel identifies the representative image that exhibits the best exposure level for the corresponding pixel location.

17. The process of claim 16, wherein the process action of identifying the representative image that exhibits the best exposure level at a pixel location corresponding to the index image pixel location under consideration, comprises an action of identifying the representative image that maximizes the local contrast about that pixel location among all the representative images.

18. The process of claim 1, wherein the representative images vary in respect to what color space range is depicted by showing as color saturated only those pixels having color values falling within the color space range associated with the representative image, and wherein the process action of computing an index image, comprises the actions of, for each pixel of the index image:

computing the color difference between, the pixel of each representative image corresponding in location to the pixel of the index image under consideration, and each of a set of representative RGB color coordinates defined as the center RGB value of each of said color space ranges;

identifying which of the computed color differences is the overall minimum difference; and

assigning an indicator identifying the representative image associated with the overall minimum color difference to the pixel of the index image under consideration.

19. The process of claim 1, wherein the representative images vary in respect to depth of focus, and wherein the process action of computing an index image, comprises the action of computing the index image wherein each pixel identifies the representative image that exhibits the sharpest focus at the corresponding pixel location.

20. The process of claim 19, wherein the process action of identifying the representative image that exhibits the sharpest focus at a pixel location corresponding the index image pixel location under consideration, comprises an action of identifying the representative image that maximizes the local contrast about that pixel location among all the representative images.

21. The process of claim 20, further comprising the process actions of:

prior to computing the index image, reducing camera noise in the representative images by convolving each representative image using a Gaussian filter; and

after computing the index image, smoothing the index image by convolving it using a Gaussian filter.

22. The process action of claim 20, further comprising a process action of performing an anisotropic diffusion on the index image to better identify the representative image providing the sharpest focus for each pixel location residing in substantially untextured regions of the scene.

23. The process of claim 1, further comprising the process action of assigning a unique index number to each representative image, and wherein the process action of computing the index image, comprises an action of identifying the representative image that exhibits the desired effect related to the varied parameter at the corresponding pixel location by assigning the index number of the associated representative image to the index image pixel location under consideration.

24. The process of claim 1, wherein the interactive image is a frame of video.

25. The process of claim 24, wherein at least some of the frames of said video are interactive images.

26. The process of claim 1, wherein the interactive image is a panoramic image.

27. The process of claim 26, wherein the interactive panoramic image is part of a panoramic video-based virtual reality system.

28. The process of claim 27, wherein at least some of the panoramic images making up the panoramic video-based virtual reality system are interactive panoramic images.

29. The process of claim 1, wherein the interactive image is a slot image associated with a concentric mosaic.

30. The process of claim 29, wherein the interactive slot image is part of a concentric mosaic-based virtual reality system.

31. The process of claim 30, wherein at least some of the slot images making up the concentric mosaic-based virtual reality system are interactive slot images.

32. The process of claim 3, wherein the process action of computing the index image, further comprises the actions of:

prior to computing the index image, assigning a unique index number to each representative image;

for each pixel location of the index image,

identifying the representative image that exhibits the desired effect related to the varied parameter at a pixel location corresponding to the index image pixel location under consideration,

assigning the index number of the identified representative image to the index image pixel location under consideration; and

smoothing the index image.

33. The process of claim 1, wherein the scene is just a portion of a larger, overall image such that the interactive image acts as an interactive region in the overall image.

34. An interactive image system for allowing a viewer to interact with a displayed image so as to cause the displayed image to change with regard to a desired effect, comprising:

a general purpose computing device comprising an image display device;

a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to,

access an interactive image, said interactive image comprising a set of representative images which depict a scene with one or more image parameters varying between images,

input an interaction mode selection made by the viewer,

implement the mode of interaction by displaying one of said representative images, or a modified version thereof, on the image display device in accordance with requirements of the selected mode of interaction.

35. The system of claim 34, wherein the program module for accessing the interactive image, comprises a sub-module for retrieving a previously-created interactive image from a local storage device.

36. The system of claim 34, wherein the program module for accessing the interactive image, comprises a sub-module for requesting and receiving the interactive image from a remote computing device over a computer network.

37. The system of claim 34, wherein the program module for accessing the interactive image, comprises a sub-module for creating the interactive image from at least one digital image of a scene upon a request by the viewer to interact with the interactive image.

38. The system of claim 34, wherein the program module for accessing the interactive image, comprises a sub-module for displaying an initial image to the viewer which is representative of the scene depicted by the interactive image.

39. The system of claim 38, wherein the sub-module for displaying the initial image to the viewer, comprises a sub-module for displaying one of the representative images at random.

40. The system of claim 38, wherein the sub-module for displaying the initial image to the viewer, comprises a sub-module for displaying a prescribed one of the representative images.

41. The system of claim 38, wherein the sub-module for displaying the initial image to the viewer, comprises a sub-module for displaying a representative image selected by the viewer from the set of representative images.

42. The system of claim 38, wherein the sub-module for displaying the initial image to the viewer, comprises a sub-module for displaying an image created by combining two or more of the representative images.

43. The system of claim 38, wherein the one or more image parameters varying between images pertains to what color space range is depicted in that each representative image depicts a different color space range by showing as color-saturated only those pixels having color values falling within the color space range associated with the representative image, and wherein the sub-module for displaying the initial image to the viewer, comprises a sub-module for displaying an image of the scene whose pixels are unsaturated in color.

44. The system of claim 34, wherein the viewer selects an ordinal mode of interaction, and wherein the program module for implementing the ordinal mode comprises sub-modules for:

inputting a viewer selection of a particular representative image; and

displaying the viewer-selected representative image.

45. The system of claim 44, wherein the representative images are indexed and wherein the sub-module for inputting the viewer selection of a particular representative image comprises a sub-module for inputting a viewer-specified index of said representative image.

46. The system of claim 45, wherein the program module for implementing the ordinal mode further comprises sub-modules for:

displaying to the viewer on said display device a graphic user interface slider which the user can move back and forth between slider endpoints via an input device, said slider representing the range of indices assigned to the representative images such that a first endpoint of the slider is associated with a first of the indexed representative images and the second endpoint of the slider is associated with a last of the indexed representative images and positions on the slider between the endpoints are associated with representative images in sequential index order from the first endpoint to the second endpoint; and

wherein the sub-module for inputting a viewer-specified index of said representative image comprises inputting the index associated with the position of the slider.

47. The system of claim 45, wherein the one or more image parameters varying between images pertains to what color space range is depicted in that each representative image depicts a different color space range by showing as color-saturated only those pixels having color values falling within the color space range associated with the representative image, and wherein the color space range is a RGB color space range in that each range specifies a particular range of red color component values, a particular range of green color component values and a particular range of blue color component values, and wherein the program module for inputting a viewer-specified index representing the selected representative image comprises sub-modules for:

allowing the viewer to respectively select a red, green and blue color component value;

identifying the particular RGB color space range the selected color component values fall into;

identifying the representative image associated with the identified RGB color space range; and

inputting the index associated with the identified representative image.

48. The system of claim 34, wherein the interactive image further comprises an index image, each pixel of which identifies the representative image that exhibits an optimum condition related to the one or more varying image parameters at a corresponding pixel location.

49. The system of claim 48, wherein the viewer selects a pixel-index mode of interaction, and wherein the program module for implementing the pixel-index mode comprises sub-modules for:

inputting a viewer selection of a pixel location in a currently displayed image on the display device;

identifying the representative image assigned to the pixel of the index image corresponding to the viewer selected pixel location in the currently displayed image; and

displaying the identified representative image on the display device in lieu of the currently displayed image.

50. The system of claim 49, further comprising a sub-module for capturing the displayed representative image as a still image.

51. The system of claim 48, wherein the viewer selects a cumulative mode of interaction, and wherein the program module for implementing the cumulative mode comprises sub-modules for:

(a) inputting a first viewer selection of a pixel location in a currently displayed image on the display device;

(b) identifying the representative image assigned to the pixel of the index image corresponding to the first viewer selected pixel location in the currently displayed image;

(c) displaying the identified representative image on the display device in lieu of the currently displayed image;

(d) inputting a subsequent viewer selection of a pixel location in the new currently displayed image;

(e) identifying the representative image assigned to the pixel of the index image corresponding to the subsequently-selected pixel location in the new currently displayed image;

(f) determining whether the currently identified representative image has been identified previously;

(g) whenever it is determined that the currently identified representative image has not been identified previously, combining the currently identified representative image with the currently displayed image, and displaying the combined image on the display device in lieu of the currently displayed image; and

(h) repeating sub-modules (d) through (g) each time a viewer subsequently selects another pixel location in the currently displayed image.

52. The process of claim 51, wherein the parameter varied among the representative images is the exposure level, and wherein the sub-module for combining the currently identified representative image with the currently displayed image, comprises sub-modules for:

ascertaining the number of representative image that have been combined to form the currently displayed image and adding one to the ascertained number to account for the currently identified representative image which is to be combined with the currently displayed image, wherein the resulting sum m represents the number of images that will make up the combined image; and

for each pixel of the combined image being generated,

computing a weighted sum of the values of the correspondingly-located pixels of the currently identified representative image and the currently displayed image, wherein the pixel value of the currently identified representative image is given a weight of 1/m and the pixel value of the currently displayed image is given a weight of (m−

1)/m, and

assigning the computed weighted sum as the pixel value of the correspondingly-located pixel of the combined image.

53. The system of claim 51, further comprising a sub-module for capturing the displayed combined image as a still image.

54. The system of claim 48, wherein the viewer selects a comprehensive mode of interaction, and wherein the program module for implementing the comprehensive mode comprises sub-modules for:

generating an image from the representative images which exhibits at each pixel the optimal condition related to the one or more parameter varied among the representative images;

displaying the generated image on the display device.

55. The system of claim 54, wherein the parameter varied among the representative images is the exposure level, and wherein the sub-module for generating the comprehensive image from the representative images comprises sub-modules for:

generating an image which exhibits a maximum contrast in each of a set of prescribed sub-regions;

blending the resulting contrast maximized image to smooth the transition in contrast between sub-regions.

56. The system of claim 54, wherein the parameter varied among the representative images is the depth of focus, and wherein the sub-module for generating the comprehensive image from the representative images comprises generating an image which exhibits at each pixel the sharpest focus for that pixel location.

57. The system of claim 54, wherein the parameter varied among the representative images is the color space range depicted, and wherein the sub-module for generating the comprehensive image from the representative images comprises generating an image which exhibits at each pixel a fully saturated color associated with that pixel location.

58. The system of claim 54, further comprising a sub-module for capturing the displayed comprehensive representative image as a still image.

59. An interactive video system for allowing a viewer to interact with the video so as to cause frames thereof to exhibit a desired effect as they are being displayed, comprising:

a general purpose computing device comprising a video display device;

a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to,

input an interaction mode selection made by the viewer, and

for each frame of the video that is scheduled for play after the viewer interaction mode selection has been input,

access an interactive image associated with the frame, wherein said interactive image comprises a set of representative images which depict a scene with one or more image parameters varying between images, and

implement the selected mode of interaction by displaying one of said representative images, or a modified version thereof, on the video display device in accordance with requirements of the selected mode of interaction.

60. The system of claim 59, wherein the viewer selects an ordinal mode of interaction, and wherein the program module for implementing the ordinal mode comprises sub-modules for:

(a) displaying an initial one of the representative images of the interactive image associated with the frame scheduled for play and pausing the video such that the initial representative image remains displayed on the video display screen pending further viewer input;

(b) inputting a viewer selection of a particular representative image among the representative images making up the interactive image associated with the frame under consideration;

(c) displaying the viewer-selected representative image;

(d) repeating sub-modules (b) and (c) each time a subsequent viewer selection of a representative image is input;

(e) resuming video play upon input of a viewer request to do so.

61. The system of claim 59, wherein the interactive image further comprises an index image, each pixel of which identifies the representative image that exhibits an optimum condition related to the one or more varying image parameters at a corresponding pixel location.

62. The system of claim 61, wherein the viewer selects a pixel-index mode of interaction, and wherein the program module for implementing the pixel-index mode comprises sub-modules for:

(a) inputting a viewer selection of a pixel location in a currently displayed frame of the video;

(b) identifying the representative image assigned to the pixel of the index image corresponding to the viewer selected pixel location of the interactive image associated with the next frame scheduled to be played;

(c) displaying the identified representative image on the video display device at the time scheduled for playing the frame under consideration;

(d) repeating sub-modules (b) and (c) for each subsequent frame scheduled of play, until another viewer-selected pixel location is input.

63. The system of claim 61, wherein the viewer selects a cumulative mode of interaction, and wherein the program module for implementing the cumulative mode comprises sub-modules for:

(a) inputting a first viewer selection of a pixel location in a currently displayed frame of the video;

(b) identifying the representative image assigned to the pixel of the index image corresponding to the first viewer selected pixel location of the interactive image associated with the next frame scheduled to be played;

(c) displaying the identified representative image on the video display device at the time scheduled for playing said next frame;

(d) repeating sub-module (b) for each subsequent frame scheduled to play, and repeating sub-module (c) for each subsequent frame scheduled to play until another viewer-selected pixel location is input;

(e) inputting a viewer selection of a pixel location in a subsequently displayed image;

(f) identifying the representative image assigned to the pixel of the index image corresponding to the currently selected pixel location of the interactive image associated with the next frame scheduled to be played;

(g) determining whether the identified representative image has been identified previously;

(h) whenever it is determined that the identified representative image has not been identified previously, combining the identified representative image with all other identified representative images associated with the frame scheduled to be played next, and displaying the combined image on the video display device at the time scheduled for the next frame;

(i) repeating sub-modules (f) and (g) for each subsequent frame scheduled to play, and repeating sub-module (h) for each subsequent frame scheduled to play until another viewer-selected pixel location is input;

(j) repeating sub-modules (e) through (i) each time a viewer subsequently selects a pixel location in a currently displayed image.

64. The system of claim 61, wherein the viewer selects a comprehensive mode of interaction, and wherein the program module for implementing the comprehensive mode comprises sub-modules for:

generating, for each frame of the video prior to its being played, an image from the representative images of the interactive image associated with the frame which exhibits at each pixel the optimal condition related to the one or more parameter varied among the representative images;

displaying the generated image associated with a frame at the time scheduled for the frame to be played.

65. An interactive virtual reality system for allowing a viewer to interact with virtual images generated by the system so as to cause the virtual images to exhibit a desired effect when displayed, comprising:

a general purpose computing device comprising an image display device;

a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to,

input a viewer-selected viewpoint and viewing direction,

input a viewer-selected interaction mode,

access an interactive image that is capable of being used to generate a virtual image showing a scene from the viewpoint and viewing direction selected by the viewer, wherein said interactive image comprises a set of representative images which depict the scene with one or more image parameters varying between images, and

implement the selected mode of interaction by,

generating from one of said representative images, or a modified version thereof, the virtual image in accordance with requirements of the selected mode of interaction which shows the scene from the selected viewpoint and viewing direction, and

displaying the generated virtual image on the image display device.

66. The system of claim 65, wherein the viewer selects an ordinal mode of interaction, and wherein the program module for generating the virtual image comprises sub-modules for:

inputting a viewer selection of a particular representative image which shows the scene from the selected viewpoint and viewing direction; and

designating the selected representative image as the virtual image.

67. The system of claim 65, wherein the interactive image further comprises an index image, each pixel of which identifies the representative image that exhibits an optimum condition related to the one or more varying image parameters at a corresponding pixel location.

68. The system of claim 67, wherein the viewer selects a pixel-index mode of interaction, and wherein the program module for generating the virtual image comprises sub-modules for:

inputting a viewer selection of a pixel location in a currently displayed image on the display device which shows the scene from the selected viewpoint and viewing direction;

identifying the representative image assigned to the pixel of the index image corresponding to the viewer selected pixel location in the currently displayed image; and

designating the identified representative image as the virtual image.

69. The system of claim 67, wherein the viewer selects a cumulative mode of interaction, and wherein the program module for implementing the cumulative mode further comprises sub-modules for:

(a) inputting a first viewer selection of a pixel location in a currently displayed image on the display device which shows the scene from the selected viewpoint and viewing direction;

(b) identifying the representative image assigned to the pixel of the index image corresponding to the first viewer selected pixel location in the currently displayed image;

(c) designating the identified representative image as the current virtual image;

(d) displaying the current virtual image on the display device in lieu of the currently displayed image;

(e) inputting a subsequent viewer selection of a pixel location in the new currently displayed image;

(f) identifying the representative image assigned to the pixel of the index image corresponding to the selected pixel location in the new currently displayed image;

(g) determining whether the currently identified representative image has been identified previously;

(h) whenever it is determined that the currently identified representative image has not been identified previously, combining the currently identified representative image with the currently displayed image, and designating the combined image as the new current virtual image;

(i) displaying the new current virtual image on the display device in lieu of the currently displayed image; and

(j) repeating sub-modules (e) through (i) each time a viewer subsequently selects another pixel location in the currently displayed image.

70. The system of claim 67, wherein the viewer selects a comprehensive mode of interaction, and wherein the program module for generating the virtual image comprises sub-modules for:

generating an image from the representative images which exhibits at each pixel the optimal condition related to the one or more parameter varied among the representative images;

designating the generated image as the virtual image.

71. A computer-readable medium having computer-executable instructions for creating an interactive image, said computer-executable instructions comprising:

inputting at least one digital image of a scene;

deriving from the at least one inputted image a set of representative images which depict the scene with one or more image parameters varying between images;

computing an index image, each pixel of which identifies the representative image that exhibits a desired effect related to the varied parameter at the corresponding pixel location; and

compressing the representative images to facilitate storage of the representative images or transfer of the representative images over a computer network to a remote computing device.

72. The computer-readable medium of claim 71, wherein the instruction for compressing the representative images comprises sub-modules for:

combining the representative images into a single comprehensive image; and

creating an auxiliary image that provides information for reconstructing the representative images.

73. A computer-implemented process for creating an interactive image, comprising using a computer to perform the following process actions:

inputting at least one digital image of a scene;

deriving from the at least one inputted image a set of representative images which depict the scene with one or more image parameters varying between images;

computing a n-dimensional index hypervolume that maps n-dimensional coordinates to the representative image that exhibits a desired effect related to the varied parameter at a pixel location corresponding to the n-dimensional coordinates.

74. An interactive virtual reality system for allowing a viewer to interact with virtual images generated by the system so as to cause the virtual images to exhibit a desired effect when displayed, comprising:

a general purpose computing device comprising an image display device;

a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to,

input a viewer-selected viewpoint and viewing direction,

access an interactive image that is capable of being used to generate a virtual image showing a scene from the viewpoint and viewing direction selected by the viewer, wherein said interactive image comprises a set of representative images which depict the scene with one or more image parameters varying between images, and

implement a prescribed mode of interaction dictated by the by the selected viewpoint and viewing direction,

generating from one of said representative images, or a modified version thereof, the virtual image in accordance with requirements of the prescribed mode of interaction which shows the scene from the selected viewpoint and viewing direction, and

displaying the generated virtual image on the image display device.