User interfaces for interacting with top-down maps of reconstructed 3-D scences

• US 8,773,424 B2
• Filed: 02/04/2010
• Issued: 07/08/2014
• Est. Priority Date: 02/04/2010
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Chen, Billy, Ofek, Eyal, Ramos, Gonzalo Alberto, Dughi, Jonathan Robert, Gedye, David Maxwell
• Primary Examiner(s): BROOME, SAID A
• Application Number:
  US12/699,904
• Publication Number:
  US 20110187716A1
• Time in Patent Office: 1,615 Days
• Field of Search: None
• US Class Current: 345/419
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]

Abstract

Technologies are described herein for providing user interfaces through which a user may interact with a top-down map of a reconstructed structure within a 3-D scene. An application provides one or more user interfaces allowing a user to select a camera pose, a reconstruction element, a point, or a group of points on the top-down map. The application then determines at least one representative photograph from the visual reconstruction based on the selection of the user, and the displays a preview of the representative photograph on the top-down map as a thumbnail image. The provided user interfaces may further allow the user to navigate to the representative photograph in the local-navigation display of the visual reconstruction.

Claims

1. An apparatus comprising:

a processor; and

a memory configured to store computer-executable instructions that, when executed by the processor, cause the processor to

compute a three-dimensional (3-D) point cloud representing one of a structure or a scene by locating a plurality of recognizable features that appear in two or more photographs of a collection of digital photographs by calculating positions of the plurality of recognizable features in space using one of location, perspective and visibility or obscurity of the plurality of recognizable features of each photograph of the two or more photographs, each position of the plurality of recognizable features corresponding to a point in the 3-D point cloud,

generate a two-dimensional top-down map by projecting points of the 3-D point cloud into a 2-D plane, and representing each projected point of the 3-D point cloud in the top-down map with a dot, the top-down map providing a visual reconstruction of one of the structure or the scene computed from the 3-D point cloud,

receive a user interaction with the top-down map, the user interacting specifying a feature in the 3-D point cloud represented by at least one dot in the top-down map,

determine a representative digital photograph from the collection of digital photographs based on the received user interaction, the representative digital photograph corresponding to at least a located recognizable feature associated the received user interaction, and

display an image of the representative digital photograph being overlaid on the top-down map representing the one of the structure or the scene.

2. The apparatus of claim 1, wherein displaying the image of the representative digital photograph comprises displaying a thumbnail image.

3. The apparatus of claim 1, wherein the user interaction further comprises a selection of a camera pose indicator displayed on the top-down map, the camera pose indicator including a position of and an orientation of a camera used to capture a digital photograph from the collection of digital photographs, and

wherein determining the representative digital photograph from the collection of digital photographs comprises selecting a digital photograph corresponding to the selected camera pose.

4. The apparatus of claim 1, wherein the user interaction further comprises a selection of an ideal camera view on the top-down map, and

wherein determining the representative digital photograph from the collection of digital photographs comprises selecting a digital photograph corresponding to a camera pose most closely matching the ideal camera view, the camera pose including a position of and an orientation of a camera used to capture a digital photograph from the collection of digital photographs.

5. The apparatus of claim 1, wherein the user interaction further comprises a selection of a reconstruction element, and

wherein determining the representative digital photograph from the collection of digital photographs comprises selecting a digital photograph corresponding to the selected reconstruction element.

6. The apparatus of claim 1, wherein the user interaction further comprises a selection of a point in the 3-D point cloud, and

wherein determining the representative digital photograph from the collection of digital photographs further comprises selecting a digital photograph corresponding to a camera pose having a closest proximity to the selected point from among the camera poses being similarly centered on the selected point, the camera pose including a position of and an orientation of a camera used to capture a digital photograph from the collection of digital photographs.

7. The apparatus of claim 1, wherein the user interaction further comprises a selection of a point in the 3-D point cloud, and

wherein determining the representative digital photograph from the collection of digital photographs further comprises selecting a photograph corresponding to a camera pose closest to the mean position and orientation of a cluster of camera poses being similar in position and in orientation, the camera pose including a position of and an orientation of a camera used to capture a digital photograph from the collection of digital photographs.

8. The apparatus of claim 1, wherein determining the representative digital photograph from the collection of digital photographs further comprises selecting multiple digital photographs related to the user interaction, and

wherein the user interaction further determines which ones of the multiple digital photographs to display.

9. The apparatus of claim 1, wherein the user interaction comprises a selection of a group of points in the 3-D point cloud, and

wherein determining the representative digital photograph from the collection of digital photographs comprises selecting a digital photograph related to an object comprising the selected group of points.

10. A computer-implemented method comprising:

computing, on a computing device, a three-dimensional (3-D) point cloud representing one of a structure or a scene by locating a plurality of recognizable features that appear in two or more photographs of a collection of digital photographs by calculating positions of the plurality recognizable features in space using one of location, perspective and visibility or obscurity of the plurality of recognizable features of each photograph of the two or more photographs, each position of the plurality of recognizable features corresponding to a point in the 3-D point cloud,

generating, on the computing device, a two-dimensional top-down map by projecting points of the 3-D point cloud into a 2-D plane, and representing each projected point of the 3-D point cloud in the top-down map with a dot, the top-down map providing a visual reconstruction of one of the structure or the scene computed from the 3-D point cloud,

receiving, on the computing device, a selection of a feature in the 3-D point cloud represented by at least one dot in the top-down map;

determining, by the computing device, a representative digital photograph from the collection of digital photographs based on the received user interaction; and

displaying a thumbnail image of the representative digital photograph overlaid on the top-down map representing the one of the structure or the scene.

11. The computer-implemented method of claim 10, further comprising displaying the representative digital photograph in an associated local-navigation display.

12. The computer-implemented method of claim 10, wherein determining the representative digital photograph from the collection of digital photographs further comprises selecting, by the computing device, a photograph corresponding to a camera pose having a closest proximity to the selected point from among the camera poses being similarly centered on the selected point, the camera pose including a position of and an orientation of a camera used to capture a digital photograph from the collection of digital photographs.

13. The computer-implemented method of claim 10, wherein determining the representative digital photograph from the collection of digital photographs further comprises selecting, by the computing device, a photograph corresponding to a camera pose closest to the mean position and orientation of a cluster of camera poses being similar in position and in orientation, the camera pose including a position of and an orientation of a camera used to capture a digital photograph from the collection of digital photographs.

14. The computer-implemented method of claim 10, wherein determining the representative digital photograph from the collection of digital photographs further comprises selecting, by the computing device, multiple photographs related to the selected point, and

wherein a user interaction determines which ones of the multiple digital photographs to display as the thumbnail image.

15. The computer-implemented method of claim 10, further comprising:

receiving, by the computing device, a selection of a group of points in the 3-D point cloud; and

determining, by the computing device, the representative digital photograph from the collection of digital photographs related to an object comprising the selected group of points.

16. A computer-implemented system comprising:

a server computer executing a spatial processing engine that generates a three-dimensional (3-D) point cloud representing one of a structure or a scene by locating a plurality of recognizable features that appear in two or more photographs of a collection of digital photographs by calculating positions of the plurality recognizable features in space using one of location, perspective and visibility or obscurity of the plurality of recognizable features of each photograph of the two or more photographs, each position of the plurality of recognizable features corresponding to a point in the 3-D point cloud, and generates a two-dimensional top-down map by projecting points of the 3-D point cloud into a 2-D plane, and represents each projected point of the 3-D point cloud in the top-down map with a dot, the top-down map providing a visual reconstruction of one of the structure or the scene computed from the 3-D point cloud;

a visualization service executing on the server computer and configured to send the top-down map providing the visual reconstruction of one of the structure or the scene to a visualization client; and

the visualization client executing on a user computer and configured to

receive and display the top-down map providing the visual reconstruction on a display device connected to the user computer,

receive a user interaction with the top-down map providing the visual reconstruction of one of the structure or the scene,

determine a representative digital photograph from the collection of digital photographs based on the received user interaction, the representative digital photograph corresponding to at least a located recognizable feature associated with the received user interaction, and

display a thumbnail image of the representative digital photograph overlaid on the top-down map providing the visual reconstruction of the one of the structure or the scene.

17. The system of claim 16, wherein the visualization client is further configured to display the representative digital photograph in an associated local-navigation display of the visual reconstruction on the display device.

18. The system of claim 16, wherein the user interaction further comprises a camera-based interaction, and

wherein determining the representative digital photograph based on the user interaction further comprises:

determining a camera pose indicator displayed on the top-down map of the visual reconstruction of one the structure or the scene most related to the camera-based interaction, the camera pose indicator including a position of and an orientation of a camera used to capture a digital photograph from the collection of digital photographs; and

selecting the photograph corresponding to the camera pose indicator as the representative digital photograph.

19. The system of claim 16, wherein the user interaction further comprises an object-based interaction, and

wherein determining the representative photograph based on the user interaction further comprises:

determining one or more digital photographs from the visual reconstruction of one of the structure or the scene most related to the object-based interaction; and

selecting the one or more digital photographs as the representative digital photographs.