Alternate Viewpoint Image Enhancement

• US 20140002439A1
• Filed: 06/28/2012
• Published: 01/02/2014
• Est. Priority Date: 06/28/2012
• Status: Active Grant

• Current Assignee: HERE Global B.V. (HERE Holding Corporation)
• Original Assignee: HERE Global B.V. (HERE Holding Corporation)
• Inventors: Lynch, James D.
• Granted Patent: US 9,256,961 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G01C 21/367 Details, e.g. road map scale, orientation, zooming, illumination, level of detail, scrolling of road map or positioning of current position marker
• CPC Class Codes: G06F 16/29 Geographical information databases
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 2200/32 involving image mosaicing
• CPC Class Codes: G06T 2210/61 Scene description
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G09B 29/007 using computer methods

Abstract

In one embodiment, panoramic images, images bubbles, or any two-dimensional views of three-dimensional subject matter are enhanced with one or more alternate viewpoints. A controller receives data indicative of a point on the two-dimensional perspective and accesses a three-dimensional location based on the point. The controller selects an image bubble based on the three-dimensional location. The three-dimensional location may be determined according to a depth map corresponding to the point. A portion of the image bubble is extracted and incorporated into the two-dimensional perspective. The resulting image may be a seamless enhanced resolution image or include a picture-in-picture enhanced resolution window including subject matter surrounding the selected point.

Claims

1. A method comprising:

receiving data indicative of a point on a two-dimensional perspective, wherein the point is associated with a desired object;

accessing, using a processor, a three-dimensional location computed from a depth map;

selecting, using the processor, an image bubble based on the three-dimensional location computed from the depth map; and

incorporating, using the processor, at least a portion of the image bubble with the two-dimensional perspective, wherein the at least the portion of the image bubble represents the desired object.

2. The method of claim 1, wherein the at least the portion of the image bubble represents a different view angle than the two-dimensional perspective.

3. The method of claim 1, further comprising:

receiving a command to manipulate the at least the portion of the image bubble by view angle or zoom.

4. The method of claim 1, further comprising:

receiving a command to switch to a subsequent image bubble; and

incorporating at least a portion of the subsequent image bubble with the two-dimensional perspective according to the command.

5. The method of claim 1, wherein incorporating the at least the portion of the image bubble with the two-dimensional image comprises:

generating an image including the two-dimensional perspective and data derived from the at least the portion of the image bubble.

6. The method of claim 1, further comprising:

rotating the at least the portion of the image bubble to approximate the two-dimensional perspective.

7. The method of claim 1, wherein incorporating the at least the portion of the image bubble with the two-dimensional image comprises:

projecting the at least the portion of the image bubble on a three-dimensional model; and

transforming data from the three-dimensional model to the two-dimensional perspective.

8. The method of claim 1, wherein detail from multiple image bubbles is incorporated with the two-dimensional perspective.

9. The method of claim 1, wherein incorporating the at least the portion of the image bubble with the two-dimensional image comprises:

augmenting the two-dimensional perspective with a window inset including the at least the portion of the image bubble.

10. The method of claim 1, wherein a first spatial resolution is associated with the desired object in the two-dimensional perspective and a second spatial resolution is associated with the desired object in the image bubble, and wherein the first spatial resolution is less than the second spatial resolution.

11. The method of claim 1, wherein the at least the portion of the image bubble includes scenery obstructed in the two-dimensional perspective.

12. An apparatus comprising:

a memory storing a plurality of image bubbles; and

a processor configured to access a three-dimensional location associated with a two-dimensional image and select a portion of an image bubble, based on the three-dimensional location, to be incorporated with the two-dimensional image, wherein a first resolution associated with the portion of the image bubble is greater than a second resolution associated with the two-dimensional image at the point selected from the two-dimensional image.

13. The apparatus of claim 12, wherein the point is selected from a user interface and the two-dimensional image is a photograph.

14. The apparatus of claim 12, wherein the processor is configured to rotate and size the portion of the image bubble to align with the two-dimensional image using a depth map.

15. The apparatus of claim 12, wherein detail from multiple image bubbles is added to the two-dimensional image.

16. The apparatus of claim 12, wherein the processor is configured to incorporate the portion of the image bubble into the two-dimensional image as a window inset positioned near the selected point.

17. The apparatus of claim 12, wherein the portion of the image bubble includes objects obstructed in the two-dimensional image.

18. The apparatus of claim 12, wherein the processor is configured to change the portion of the image bubble by rotating the image bubble about a center of the image bubble.

19. The apparatus of claim 12, wherein the processor is configured to access a subsequent image bubble according to input data and incorporate at least a portion of the subsequent image bubble into the two-dimensional image.

20. The apparatus of claim 19, wherein the portion of the subsequent image bubble includes an object corresponding to the point selected from the two-dimensional image.

21. A non-transitory computer readable medium including instructions that when executed are operable to:

determine a first image bubble associated with a position, wherein the first image bubble includes data indicative of an object at a first spatial resolution;

identify a second image bubble associated with a region represented by the first image bubble, wherein the second image bubble includes data indicative of the object at a second spatial resolution;

generate an image with the first image bubble; and

incorporate at least a portion of the second bubble into the image.

22. The non-transitory computer readable medium of claim 21, the instruction operable to:

query a depth map using a point selected from a panoramic image of the first image bubble to receive a depth value;

determine the second image bubble as being for the point based on the depth value; and

incorporate the at least the portion of the second image bubble with the panoramic image.

23. The non-transitory computer readable medium of claim 22, wherein the second spatial resolution is greater than the first spatial resolution.