Alternate viewpoint image enhancement

• US 9,256,961 B2
• Filed: 06/28/2012
• Issued: 02/09/2016
• Est. Priority Date: 06/28/2012
• Status: Active Grant

• Current Assignee: HERE Global B.V. (Nokia Corporation)
• Original Assignee: HERE Global B.V. (Nokia Corporation)
• Inventors: Lynch, James D.
• Primary Examiner(s): He, Yingchun
• Application Number:
  US13/536,536
• Publication Number:
  US 20140002439A1
• Time in Patent Office: 1,321 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G01C 21/367 Details, e.g. road map scale, orientation, zooming, illumination, level of detail, scrolling of road map or positioning of current position marker
• CPC Class Codes: G06F 16/29 Geographical information databases
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 2200/32 involving image mosaicing
• CPC Class Codes: G06T 2210/61 Scene description
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G09B 29/007 using computer methods

Abstract

In one embodiment, panoramic images, images bubbles, or any two-dimensional views of three-dimensional subject matter are enhanced with one or more alternate viewpoints. A controller receives data indicative of a point on the two-dimensional perspective and accesses a three-dimensional location based on the point. The controller selects an image bubble based on the three-dimensional location. The three-dimensional location may be determined according to a depth map corresponding to the point. A portion of the image bubble is extracted and incorporated into the two-dimensional perspective. The resulting image may be a seamless enhanced resolution image or include a picture-in-picture enhanced resolution window including subject matter surrounding the selected point.

Claims

1. A method comprising:

receiving data indicative of a point on a two-dimensional perspective image, wherein the point is associated with a desired object;

accessing, using a processor, a three-dimensional location computed from a depth map, wherein the depth map includes depth values for pixels of the two-dimensional perspective image;

selecting, using the processor, an image bubble from a plurality of image bubbles based on the three-dimensional location computed from the depth map, wherein the plurality of image bubbles are panoramic images; and

incorporating, using the processor, a window inset including at least a portion of the image bubble with the two-dimensional perspective image based on the depth map, wherein the at least the portion of the image bubble represents the desired object,

wherein the at least the portion of the image bubble represents a different view angle than the two-dimensional perspective image.

2. The method of claim 1, further comprising:

receiving a command to manipulate the at least the portion of the image bubble by view angle or zoom.

3. The method of claim 1, further comprising:

receiving a command to switch to a subsequent image bubble; and

incorporating at least a portion of the subsequent image bubble with the two-dimensional perspective image according to the command.

4. The method of claim 1, wherein incorporating the at least the portion of the image bubble with the two-dimensional perspective image comprises:

generating a final image including the two-dimensional perspective image and data derived from the at least the portion of the image bubble.

5. The method of claim 1, further comprising:

rotating the at least the portion of the image bubble to approximate the two-dimensional perspective image.

6. The method of claim 1, wherein incorporating the at least the portion of the image bubble with the two-dimensional perspective image comprises:

projecting the at least the portion of the image bubble on a three-dimensional model; and

transforming data from the three-dimensional model to the two-dimensional perspective image.

7. The method of claim 1, wherein detail from multiple image bubbles is incorporated with the two-dimensional perspective image.

8. The method of claim 1, wherein a first spatial resolution is associated with the desired object in the two-dimensional perspective image and a second spatial resolution is associated with the desired object in the image bubble, and wherein the first spatial resolution is less than the second spatial resolution.

9. The method of claim 1, wherein the at least the portion of the image bubble includes scenery obstructed in the two-dimensional perspective image.

10. An apparatus comprising:

a memory storing a plurality of image bubbles; and

a processor configured to access a three-dimensional location from a depth map associated with a two-dimensional image, select an image bubble from the plurality of image bubbles and select a portion of the selected image bubble, based on the three-dimensional location, to be incorporated, based on the depth map, with the two-dimensional image as a window inset, wherein the plurality of image bubbles are panoramic images, wherein a first resolution associated with the portion of the selected image bubble is greater than a second resolution associated with the two-dimensional image at the point selected from the two-dimensional image,

wherein the portion of the selected image bubble at the first resolution and the two-dimensional image at the second resolution are displayed together at the same time,

wherein the at least the portion of the image bubble represents a different view angle than the two-dimensional perspective image.

11. The apparatus of claim 10, wherein the point is selected from a user interface and the two-dimensional image is a photograph.

12. The apparatus of claim 10, wherein the processor is configured to rotate and size the portion of the image bubble to align with the two-dimensional image using the depth map.

13. The apparatus of claim 10, wherein detail from multiple image bubbles is added to the two-dimensional image.

14. The apparatus of claim 10, wherein the portion of the image bubble includes objects obstructed in the two-dimensional image.

15. The apparatus of claim 10, wherein the processor is configured to change the portion of the image bubble by rotating the image bubble about a center of the image bubble.

16. The apparatus of claim 10, wherein the processor is configured to access a subsequent image bubble according to input data and incorporate at least a portion of the subsequent image bubble into the two-dimensional image.

17. The apparatus of claim 16, wherein the portion of the subsequent image bubble includes an object corresponding to the point selected from the two-dimensional image.

18. A non-transitory computer readable medium including instructions that when executed are operable to:

determine a first image bubble associated with a first position and a first view angle, wherein the first image bubble includes data indicative of an object at a first spatial resolution;

identify a second image bubble associated with a second position and a second view angle in a region represented by the first image bubble, wherein the second image bubble includes data indicative of the object at a second spatial resolution;

generate an image with the first image bubble at the first view angle; and

incorporate at least a portion of the second bubble into the image as a window inset and according to a depth map associated with the first image, wherein the first image bubble and the second image bubble are panoramic images.

19. The non-transitory computer readable medium of claim 18, the instruction operable to:

query the depth map using a point selected from a panoramic image of the first image bubble to receive a depth value;

determine the second image bubble as being for the point based on the depth value, wherein the at least the portion of the second image bubble is incorporated with the panoramic image.

20. The non-transitory computer readable medium of claim 19, wherein the second spatial resolution is greater than the first spatial resolution.