Selecting two-dimensional imagery data for display within a three-dimensional model

• US 10,163,261 B2
• Filed: 03/19/2014
• Issued: 12/25/2018
• Est. Priority Date: 03/19/2014
• Status: Active Grant

• Current Assignee: Matterport, Inc
• Original Assignee: Matterport, Inc
• Inventors: Bell, Matthew Tschudy, Gausebeck, David Alan, Coombe, Gregory William, Ford, Daniel, Brown, William John
• Primary Examiner(s): Nguyen, Phong
• Application Number:
  US14/219,906
• Publication Number:
  US 20150269785A1
• Time in Patent Office: 1,742 Days
• CPC Class Codes: G06T 13/80 2D [Two Dimensional] animation, e.g. using sprites
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images

Abstract

Systems and methods for generating three-dimensional models with correlated three-dimensional and two dimensional imagery data are provided. In particular, imagery data can be captured in two dimensions and three dimensions. Imagery data can be transformed into models. Two-dimensional data and three-dimensional data can be correlated within models. Two-dimensional data can be selected for display within a three-dimensional model. Modifications can be made to the three-dimensional model and can be displayed within a three-dimensional model or within two-dimensional data. Models can transition between two dimensional imagery data and three dimensional imagery data.

Claims

1. A system, comprising:

a memory that stores computer executable components; and

a processor that executes at least the following computer executable components stored in the memory;

an imagery component configured to receive input indicating a viewpoint from which to view a three-dimensional model of an object or environment; and

a selection component configured to determine, based on the input, a position of the viewpoint relative to the three-dimensional model, and select a two-dimensional image from a plurality of two-dimensional images captured of the object or environment for displaying based in part on the two-dimensional image having a first capture position relative to the three-dimensional model that is associated with a first navigational distance that is shorter than second navigational distances associated with second capture positions of other two-dimensional images of the plurality of two-dimensional images, wherein the first navigational distance reflects one or more first constraints associated with navigation of the three-dimensional model from the position to the first capture position, and wherein the second navigational distances reflect one or more second constraints associated with navigation of the three-dimensional model from the position to the second capture positions.

2. The system of claim 1, wherein the selection component is further configured to select the two-dimensional image based on the first navigational distance being less than a threshold distance.

3. The system of claim 1, wherein the viewpoint comprises a new viewpoint and wherein the computer executable components further comprise:

a construction component configured to generate and display a representation of the three-dimensional model from a previous viewpoint of the three-dimensional model, wherein the imagery component is configured to receive the input in association with the display of the representation from the previous viewpoint; and

a transition component configured to facilitate transitioning between displaying the representation of the three-dimensional model to displaying the two-dimensional image based on the input and selection of the two-dimensional image.

4. The system of claim 3, wherein the representation comprises a three-dimensional reconstruction of the three-dimensional model, and wherein the transition component is configured to replace the representation with the two-dimensional image or a portion of the two-dimensional image data based on the input and selection of the two-dimensional image.

5. The system of claim 3, wherein the representation comprises another two-dimensional image of the plurality of two-dimensional images, wherein the input comprises selection of a point on the other two-dimensional image, and wherein the transition component is configured to replace the representation with the two-dimensional image or a portion of the two-dimensional image data based on the input and selection of the two-dimensional image.

6. The system of claim 1, wherein the viewpoint comprises a new viewpoint and wherein the input comprises a navigational command to move a virtual camera relative to the three-dimensional model from a previous viewpoint of the three-dimensional model to the new viewpoint, and wherein the selection component is further configured to determine the position of the new viewpoint relative to the three-dimensional model based on the command, the previous viewpoint, and a navigation mode employed to view the three-dimensional model.

7. The system of claim 1, wherein the input comprises selection of a point on or within another two-dimensional image of the plurality of two-dimensional images, and wherein the selection component is further configured to determine the first position of the viewpoint relative to the three-dimensional model based on the point.

8. The system of claim 1, wherein the selection component is further configured to determine a field of view for viewing the three dimensional model from the viewpoint, and select the two-dimensional image based on a difference between the field of view and a capture field of view associated with the two-dimensional image being less than differences between the field of view associated with the viewpoint and capture fields of view associated with other two-dimensional images of the plurality of two-dimensional images.

9. The system of claim 1, wherein the position is associated with a room of the three-dimensional model and wherein the selection component is further configured to select the two-dimensional image based on the capture position of the two-dimensional image being associated with the room.

10. The system of claim 1, wherein the one or more first or second constraints comprise a barrier constraint that restricts passage through solid objects in association with the navigation of the three-dimensional model.

11. The system of claim 1, wherein the one or more first or second constraints comprise a room constraint that restricts the navigation of the three-dimensional model to be within the same room.

12. The system of claim 1, wherein the selection component is further configured to:

determine an area of the three-dimensional model for viewing based on the viewpoint;

determine image data of the three-dimensional model corresponding to the area; and

select the two-dimensional image based on a degree of correspondence between the image data and the two-dimensional image.

13. The system of claim 1, wherein the viewpoint comprises a new viewpoint and wherein the selection component is further configured to select the two-dimensional image based on a previous viewpoint from which the three-dimensional model is viewed at a time when the input is received.

14. The system of claim 13, wherein the input comprises a navigational command that directs movement of a virtual camera away from the current viewpoint in association with navigation of the three-dimensional model in a three-dimensional walkthrough mode.

15. The system of claim 1, wherein the viewpoint comprises a new viewpoint and wherein the selection component is configured to determine the position of the new viewpoint by casting a ray from a current viewpoint into the three-dimensional model in a direction determined based on the input, wherein the three-dimensional model is viewed from the current viewpoint at a time when the input is received.

16. A method, comprising

using a processor to execute the following computer executable instructions stored in a memory to perform the following acts:

receiving input indicating a viewpoint from which to view a three-dimensional model of an object or environment;

determining, based on the input, a position of the viewpoint relative to the three-dimensional model; and

selecting a two-dimensional image from a set of two-dimensional images captured of the object or environment for displaying based in part on the two-dimensional image having a first capture position relative to the three-dimensional model that is associated with a first navigational distance that is shorter than second navigational distances associated with second capture positions of other two-dimensional images of the set of two-dimensional images, wherein the first navigational distance reflects one or more first constraints associated with navigation of the three-dimensional model from the position to the first capture position, and wherein the second navigational distances reflect one or more second constraints associated with navigation of the three-dimensional model from the position to the second capture positions.

17. The method of claim 16, wherein the viewpoint comprises a new viewpoint and wherein the acts further comprise:

displaying a representation of the three-dimensional model from a previous viewpoint, wherein the receiving the input comprises receiving the input in association with interaction with the representation as displayed from the previous viewpoint; and

transitioning to displaying the two-dimensional image instead of the representation based on the selecting the two-dimensional image.

18. The method of claim 17, wherein the representation comprises a three-dimensional reconstruction of the three-dimensional model.

19. The method of claim 17, wherein the representation comprises another two-dimensional image, and wherein the input comprises selection of a point on the other two-dimensional image.

20. The method of claim 16, wherein the receiving the input comprises receiving a command to move a virtual camera relative to a three-dimensional reconstruction of the three-dimensional model from a previous viewpoint of the three-dimensional model in association with navigation of the three-dimensional model, and wherein the determining the position comprises determining the position based on the previous viewpoint.

21. The method of claim 16, wherein the acts further comprise:

determining a field of view for viewing the three-dimensional model from the viewpoint, wherein the selecting the two-dimensional image further comprises selecting the two-dimensional image based on a difference between the field of view and a capture field of view associated with the two-dimensional image being less than differences between the field of view associated with the viewpoint and capture fields of view associated with at least one other two-dimensional images of the plurality of two-dimensional images.

22. The method of claim 16, wherein the one or more first or second constraints comprise a barrier constraint that restricts passage through solid objects in association with the navigation of the three-dimensional model.

23. A system, comprising:

a memory that stores computer executable components; and

a processor that executes at least the following computer executable components stored in the memory;

an imagery component configured to receive input indicating a location of a three-dimensional model of an object or environment for viewing, wherein the input comprises selection of a region on a displayed representation of the three-dimensional model; and

a selection component configured to;

determine an area and a desired volume of the three-dimensional model for viewing based on the region; and

select a two-dimensional image from a plurality of two-dimensional images captured of the object or environment for displaying based in part on a score determined for the two-dimensional image that reflects a portion of the area and the volume of the three-dimensional model that is visible in the two-dimensional image.

24. The system of claim 23, wherein the displayed representation of the three-dimensional model was rendered from a current viewpoint relative to the three-dimensional model, wherein the selection component is configured to determine a distance between the current viewpoint and the region, and wherein the selection component is further configured to determine the area based on the distance.

25. The system of claim 23, wherein the selection component is further configured to:

determine image data of the three-dimensional model corresponding to the area; and

select the two-dimensional image based on the score being determined for the two-dimensional image based on a degree of correspondence between the image data and the two-dimensional image.

26. The system of claim 25, wherein the image data comprises an image of the three-dimensional model from a viewpoint relative to the three-dimensional model determined based in part on the location.

27. The system of claim 23, wherein the selection component is configured to determine a center point of the desired volume of the three-dimensional model for viewing based on an average position of points included in the selected region relative to a current viewpoint of the three-dimensional model, and wherein the selection component is further configured to determine the area based on the center point.

28. The system of claim 23, wherein the selection component is configured to determine a desired field of view for viewing the three-dimensional model based on a standard deviation of positions of points included in the selected region relative to a current viewpoint of the three-dimensional model, and wherein the selection component is further configured to determine the area based on the desired field of view.

29. A method comprising:

receiving, by a system comprising a processor, input selecting a region of a displayed representation of a three-dimensional model of an object or environment for viewing from a new perspective;

determining, by the system, a volume of the three-dimensional model based on the region; and

selecting, by the system, a two-dimensional image from a plurality of two-dimensional images captured of the object or environment for displaying based in part on a score determined for the two-dimensional image that reflects an amount of the volume of the three-dimensional model that is visible in the two-dimensional image.

30. The method of claim 29, wherein the displayed representation of the three-dimensional model reflects a current viewpoint relative to the three-dimensional model, and wherein the determining the volume is further based on a distance between the current viewpoint and the region.

31. The method of claim 29, further comprising:

determining, by the system, image data of the three-dimensional model corresponding to the volume, and wherein the selecting the two-dimensional image is further based on the score being determined based on a degree of correspondence between the image data and the two-dimensional image.

32. The method of claim 31, wherein the image data comprises an image of the three-dimensional model from a viewpoint relative to the three-dimensional model determined based in part on the region.

33. The method of claim 29, further comprising:

determining, by the system, a center point for the volume of the three-dimensional model based on an average position of points included in the region relative to a current viewpoint of the three-dimensional model, and wherein the determining the volume is further based on the center point.

34. The method of claim 29, further comprising:

determining, by the system, a field of view for viewing the three-dimensional model based on a standard deviation of positions of points included in the region relative to a current viewpoint of the three-dimensional model, and wherein the determining the volume is further based on the field of view.