Alternate viewpoint image enhancement

• US 10,030,990 B2
• Filed: 01/18/2016
• Issued: 07/24/2018
• Est. Priority Date: 06/28/2012
• Status: Active Grant

• Current Assignee: HERE Global B.V. (Nokia Corporation)
• Original Assignee: HERE Global B.V. (Nokia Corporation)
• Inventors: Lynch, James D.
• Primary Examiner(s): He, Yingchun
• Application Number:
  US14/997,819
• Publication Number:
  US 20160133044A1
• Time in Patent Office: 918 Days
• Field of Search: None
• CPC Class Codes: G01C 21/367 Details, e.g. road map scale, orientation, zooming, illumination, level of detail, scrolling of road map or positioning of current position marker
• CPC Class Codes: G06F 16/29 Geographical information databases
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 2200/32 involving image mosaicing
• CPC Class Codes: G06T 2210/61 Scene description
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G09B 29/007 using computer methods

Abstract

In one embodiment, panoramic images, images bubbles, or any two-dimensional views of three-dimensional subject matter are enhanced with one or more alternate viewpoints. A controller receives data indicative of a point on the two-dimensional perspective and accesses a three-dimensional location based on the point. The controller selects an image bubble based on the three-dimensional location. The three-dimensional location may be determined according to a depth map corresponding to the point. A portion of the image bubble is extracted and incorporated into the two-dimensional perspective. The resulting image may be a seamless enhanced resolution image or include a picture-in-picture enhanced resolution window including subject matter surrounding the selected point.

Claims

1. A method comprising:

receiving first image data for a first image bubble captured from a first perspective;

receiving second image data for a second image bubble captured from a second perspective; and

combining, using a processor, the first image data and the second image data into a composite image, wherein the composite image includes a spatial resolution greater than a resolution of the first image bubble and a resolution of the second image bubble,

wherein one or more of the first image data and second image data comprise image data for multiple viewpoints within the first image bubble or second image bubble, respectively,

wherein the composite image is from a third perspective, and

wherein the composite image comprises an inset window, the inset window including viewpoints otherwise unavailable from the third perspective.

2. The method of claim 1, further comprising:

receiving a first depth map associated with the first image bubble; and

receiving a second depth map associated with the second image bubble,

wherein the composite image is generated based on the first depth map and the second depth map.

3. The method of claim 1, further comprising:

identifying a third image bubble, wherein the first image bubble and the second image bubble are combined with the third image bubble to create the composite image.

4. The method of claim 3, further comprising:

identifying an object of interest in the first image bubble; and

identifying the object of interest in the third image bubble, wherein the object of interest in the first image bubble has a higher resolution than the object of interest in the third image bubble.

5. The method of claim 4, wherein the object of interest is identified from an image processing algorithm.

6. The method of claim 5, wherein the image processing algorithm includes edge detection, object recognition, optical character recognition, or feature extraction.

7. The method of claim 1, further comprising:

receiving a user selection of a point of the second image data; and

transitioning from the first perspective to the second perspective in response to the user selection.

8. The method of claim 1, wherein at least a portion of the first image bubble includes scenery obstructed from the third perspective.

9. The method of claim 1, further comprising:

receiving a command to manipulate the composite image by view angle or zoom.

10. The method of claim 1, wherein the composite image includes image data for multiple viewpoints within the composite image.

11. The method of claim 1, wherein the third perspective is the same as the first perspective.

12. An apparatus comprising:

a memory configured to store first image data for a first image bubble captured from a first perspective and second image data for a second image bubble captured from a second perspective; and

a processor configured to combine the first image data and the second image data into a composite image, wherein the composite image includes a spatial resolution greater than a resolution of the first image bubble and a resolution of the second image bubble,

wherein the composite image is from a third perspective, and

wherein the composite image comprises an inset window, the inset window including views not otherwise available from the third perspective.

13. The apparatus of claim 12, wherein the processor is configured to access a first depth map associated with the first image bubble and a second depth map associated with the second image bubble,

wherein the composite image is generated based on the first depth map and the second depth map.

14. The apparatus of claim 13, wherein the first depth map and the second depth map includes depth values for three-dimensional locations in the first and second image bubbles.

15. The apparatus of claim 12, wherein the processor is configured to identify a first view of an object of interest in the first image bubble and a second view of the object of interest in the third image bubble, wherein the object of interest in the first image bubble has a higher resolution than the object of interest in the third image bubble.

16. The apparatus of claim 12, wherein the processor is configured to receive a user selection of a point of the second image data and transition from the first perspective to the second perspective in response to the user selection.

17. The apparatus of claim 12, wherein the composite image includes image data for multiple viewpoints within the composite image.

18. A method comprising:

receiving first image data for a first image bubble captured from a first perspective, the first perspective comprising a field of view in a first direction;

receiving second image data for a second image bubble captured from a second perspective, the second perspective comprising a field of view in a second direction, the second direction different than the first direction;

combining the first image data and the second image data into a composite image, wherein the composite image comprises an inset window containing the second image data from the second perspective;

receiving a user selection of a point of the second image data; and

transitioning, using a processor, from the first perspective to the second perspective in response to the user selection.

19. The method of claim 18, further comprising:

combining the first image data and the second image data into a composite image, wherein the composite image includes a spatial resolution greater than a resolution of the first image bubble and a resolution of the second image bubble.

20. An apparatus comprising:

a memory configured to store first image data for a first image bubble captured from a first perspective and second image data for a second image bubble captured from a second perspective, wherein the first perspective comprises a field of view in a first direction and the second perspective comprises a field of view in a second direction, the second direction different than the first direction;

an interface configured to display the second image data from the second perspective in an inset window within a display of the first image data from the first perspective and to receive a user selection of a point of the second image data; and

a processor configured to calculate a transition from the first perspective to the second perspective in response to the user selection.