Method and apparatus for visualization of geo-located media contents in 3D rendering applications

• US 10,311,633 B2
• Filed: 09/30/2014
• Issued: 06/04/2019
• Est. Priority Date: 01/17/2014
• Status: Active Grant

• Current Assignee: Nokia Technologies Oy (Nokia Corporation)
• Original Assignee: Nokia Technologies Oy (Nokia Corporation)
• Inventors: Roimela, Kimmo
• Primary Examiner(s): Nguyen, Kimbinh T
• Application Number:
  US15/111,212
• Publication Number:
  US 20160335796A1
• Time in Patent Office: 1,708 Days
• Field of Search: 345419
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 17/05 Geographic models

Abstract

An approach is provided for accurate processing and registering of media content for rendering in 3D maps and other applications. The approach includes determining at least one first pixel of at least one image that geometrically corresponds to at least one second pixel of at least one rendered three-dimensional map. Further, the approach includes processing and/or facilitating a processing of (a) the at least one first pixel; (b) the at least one second pixel; (c) metadata associated with at least one of the at least one first pixel and the second pixel; or (d) a combination thereof to determine at least one confidence value, wherein the at least one confidence value is indicative of an estimated level of geometric distortion resulting from projecting the at least one first pixel onto the at least one second pixel. Furthermore, the approach includes determining whether to cause, at least in part, a rendering of the at least one first pixel onto the at least one rendered three-dimensional map based, at least in part, on the confidence value.

Claims

1. An apparatus comprising:

at least one processor; and

at least one memory including computer program code for one or more programs, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to perform at least the following,

determine each of a plurality of first pixels of an image that geometrically corresponds to a second pixel of a rendered three-dimensional map, wherein the three-dimensional map is a three-dimensional geographic map;

process the each first pixel and the second pixel to determine a confidence value that increases as a transparency level for the rendering of the each first pixel onto the three-dimensional map decreases, wherein the confidence value indicates an estimated level of geometric distortion of the each first pixel as projected onto the three-dimensional map resulting from a difference between an image capturing perspective of the image with respect to the each first pixel and a virtual viewing perspective of the three-dimensional map;

initiate a rendering of the first pixels onto the three-dimensional map from the virtual viewing perspective based on the confidence value;

determine at least one user interaction with the three-dimensional map; and

initiate an updating of the confidence value, the rendering of the first pixels onto the three-dimensional map, or a combination thereof in real time or substantially in real time based on the at least one user interaction,

wherein the confidence value of a respective first pixel decreases as the virtual viewing perspective moves further away from the image capturing perspective.

2. An apparatus of claim 1, wherein the at least one user interaction results in a change in (a) the virtual viewing perspective for the three-dimensional map, (b) at least one rendering characteristic for at least one of the first pixels, the image, the second pixel, and the three-dimensional map;

or (c) a combination thereof; and

wherein the at least one rendering characteristic includes a fade parameter, a toggle parameter, a size parameter, or a combination thereof.

3. An apparatus of claim 1, wherein the apparatus is further caused to:

initiate a comparison of the confidence value against at least one threshold confidence criterion to determine whether to initiate the rendering of the first pixels onto the three-dimensional map.

4. An apparatus of claim 1, wherein the apparatus is further caused to:

process metadata associated with the each first pixel and the second pixel;

determine a first camera location associated with the image and a second camera location associated with the three-dimensional map based on the metadata; and

determine an offset between the first camera location and the second camera location,

wherein the confidence value is further based on the offset.

5. An apparatus of claim 4, wherein the apparatus is further caused to:

determine at least one first ray extending from the first camera location to at least one pixel location of one of the first pixels and the second pixel;

determine at least one second ray extending from the second camera location to the at least one pixel location; and

determine at least one angle formed between the at least one first ray and the at least one second ray,

wherein the confidence value is determined based on the at least one angle.

6. An apparatus of claim 5, wherein the apparatus is further caused to:

determine the transparency level for the rendering of the each first pixel onto the three-dimensional map based on an accumulated confidence value added from a plurality of confidence values determined per pixel for a plurality of images including the image.

7. An apparatus of claim 6, wherein the transparency level decreases with an increase in the number of the images.

8. An apparatus of claim 4, wherein the apparatus is further caused to:

initiate an estimation of at least one depth offset value for the image with reference to the second camera location; and

initiate an iterative updating of a depth offset value buffer with at least one estimated depth offset value.

9. An apparatus of claim 8, wherein the rendering of the first pixels onto the three-dimensional map is based on the at least one depth offset value.

10. An apparatus of claim 8, wherein the rendering of the first pixels onto the three-dimensional map is based on at least one user interaction and the at least one depth offset value.

11. An apparatus of claim 8, wherein the apparatus is further caused to:

generate at least one new candidate depth offset value for the image with reference to the second camera location;

initiate a first rendering of at least one test image onto the three-dimensional map based on the at least one depth offset value;

initiate a second rendering of the at least one test image onto the three-dimensional map based on the at least one new candidate depth offset value; and

initiate an analysis of the first rendering and the second rendering to determine one or more rendering characteristics associated with the first rendering and the second rendering.

12. An apparatus of claim 8, wherein the apparatus is further caused to:

initiate an updating of the depth offset value buffer with the at least one depth offset value or the at least one new candidate depth offset value based on the analysis.

13. An apparatus of claim 1, wherein the apparatus is further caused to:

determine that the image includes a plurality of overlapping images; and

initiate a sorting, a blending, or a combination thereof of the plurality of overlapping images based on the confidence value.

14. An apparatus of claim 1, wherein the image includes at least one video, the apparatus is further caused to:

determine at least one three-dimensional motion track for the at least one video,

wherein the confidence value, the rendering of the first pixels onto the three-dimensional map is based on the at least one three-dimensional motion track.

15. A method comprising:

determining, by an apparatus, each of a plurality of first pixels of an image that geometrically corresponds to a second pixel of a rendered three-dimensional map, wherein the three-dimensional map is a three-dimensional geographic map;

processing, by the apparatus, the each first pixel and the second pixel to determine a confidence value that increases as a transparency level for the rendering of the each first pixel onto the three-dimensional map decreases, wherein the confidence value indicates an estimated level of geometric distortion of the each first pixel as projected onto the three-dimensional map resulting from a difference between an image capturing perspective of the image with respect to the each first pixel and a virtual viewing perspective of the three-dimensional map;

initiating, by the apparatus, a rendering of the first pixels onto the three-dimensional map from the virtual viewing perspective based on the confidence value;

determining, by the apparatus, at least one user interaction with the three-dimensional map; and

initiating, by the apparatus, a dynamic updating of the confidence value, the rendering of the first pixels onto the three-dimensional map, or a combination thereof based on the at least one user interaction,

wherein the confidence value of a respective first pixel decreases as the virtual viewing perspective moves further away from the image capturing perspective.

16. A method of claim 15, wherein the at least one user interaction results in a change in (a) the virtual viewing perspective for the three-dimensional map, (b) at least one rendering characteristic for at least one of the first pixels, the image, the second pixel, and the three-dimensional map;

or (c) a combination thereof, and wherein the at least one rendering characteristic includes a fade parameter, a toggle parameter, a size parameter, or a combination thereof.

17. A method of claim 15, further comprising:

initiating a comparison of the confidence value against at least one threshold confidence criterion to determine whether to cause the rendering of the first pixels onto the three-dimensional map.

18. A method of claim 15, further comprising:

processing metadata associated with the each first pixel and the second pixel;

determining a first camera location associated with the image and a second camera location associated with the three-dimensional map based on the metadata; and

determining an offset between the first camera location and the second camera location,

wherein the confidence value is further based on the offset.