Methods and Systems for Using a Mobile Device to Visualize a Three-Dimensional Physical Object Placed Within a Three-Dimensional Environment

• US 20150170260A1
• Filed: 02/29/2012
• Published: 06/18/2015
• Est. Priority Date: 02/29/2012
• Status: Abandoned Application

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: LEES, Jennifer, HUANG, Jonathan
• Application Number:
  US13/408,454
• Publication Number:
  US 20150170260A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06Q 30/0643 Graphical representation of items or shoppers
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Systems, methods and computer program products for using a mobile device to visualize physical objects in an environment are described herein. An embodiment includes receiving a three-dimensional model of an environment, detecting, using a sensor on the mobile device, an identifier identifying a physical object and retrieving, using the detected identifier, a three-dimensional model of the physical object. An embodiment further includes displaying, on the mobile device, the three-dimensional model of the physical object within the three-dimensional model of the environment and, in response to user gestures on the mobile device, displaying the physical object at different places within the environment.

Claims

1. A computer implemented method performed by a mobile device, comprising:

receiving from a user a length and width of an environment;

receiving a series of photographs taken front different positions in the environment by the user with a camera on a mobile device;

deriving a three-dimensional model of the environment based on the photographs, length and width;

detecting, using a sensor on the mobile device, an identifier identifying a physical object wherein the identifier and the environment are not co-located;

retrieving, using the detected identifier, a three-dimensional model of the physical object that is capable of being placed within a three-dimensional model of an environment;

displaying, on a display of the mobile device, the three-dimensional model of the physical object within the three-dimensional model of the environment; and

in response to user gestures on the display of the mobile device, displaying the three-dimensional model of the physical object at different places within the three-dimensional model of the environment.

2. The method of claim 1 wherein receiving a three-dimensional model of an environment comprises:

receiving from the user the length and width of the environment;

receiving a series of photographs taken from different positions in the environment; and

deriving a three-dimensional model of the environment based on the photographs, length and width.

3. The method of claim 2 wherein deriving a three-dimensional model comprises:

identifying features on the photographs;

matching features on adjacent photographs;

deriving a surface model for the point cloud; and

texture mapping the photographs to the surface model.

4. The method of claim 1 wherein detecting comprises:

capturing, using a camera on the mobile device, an image of a barcode for the physical object; and

scanning the image to detect an identifier identifying the physical object on a server having a three-dimensional model of the physical object, the scanning performed by an application on the mobile device.

5. The method of claim 4, wherein the scanning comprises scanning the image to detect a Uniform Resource Locator (URL) addressing the server, and wherein the retrieving comprises:

connecting to the server using the URL, the server having a three-dimensional model of the physical object;

sending, via the connection, the identifier to the server; and

downloading, from the server, the three-dimensional model of the physical object.

6. The method of claim 1 wherein displaying the physical object at different places within the environment comprises:

in response to the user touching and dragging a representation of the physical object on the display, moving the physical object within the environment.

7. The method of claim 1 wherein displaying the physical object at different places within the environment comprises:

in response to the user touching and making horizontal gestures with the mobile device, panning the physical object within the environment.

8. The method of claim 1 wherein displaying the physical object at different places within the environment comprises:

in response to the user touching either side of the display of the mobile device, panning the physical object within the environment.

9. The method of claim 1 wherein displaying the physical object comprises:

enabling a user to select a set of color choices for the physical object; and

displaying the physical object in a particular color specified by the user'"'"'s selection.

10. A computer-based system, comprising:

one or more processors;

an environment virtualizer configured to receive from a user a length and width of an environment, receive a series of photographs taken from different positions in the environment by the user with a camera on a mobile device, and derive a three-dimensional model of the environment based on the photographs, length and width;

a QR barcode application configured to detect, using a sensor on the mobile device, an identifier identifying a physical object when the identifier and the environment are not co-located;

a physical object placer configured to retrieve, using the detected identifier, a three-dimensional model of the physical object that is capable of being placed within a three-dimensional model of an environment, to display, on a display of the mobile device, the three-dimensional model of the physical object within the three-dimensional model of the environment, and in response to user gestures on the display of the mobile device, to display the three-dimensional model of the physical object at different places within the three-dimensional model of the environment.

11. The system of claim 10 wherein the environment virtualizer is further configured to receive from the user the length and width of the environment, to receive a series of photographs taken from different positions in the environment, and to derive a three-dimensional model of the environment based on the photographs, length and width.

12. The system of claim 11 wherein the environment virtualizer is further configured to identify features on the photographs, to match features on adjacent photographs, to derive a surface model for the point cloud, and to texture map the photographs to the surface model.

13. The system of claim 10 wherein the QR application is configured to capture, using a camera on the mobile device, an image of a barcode for the physical object, and to scan the image to detect an identifier identifying the physical object, the scanning performed by an application on the mobile device.

14. The system of claim 13 wherein the QR application is configured to scan the image to detect a Uniform Resource Locator (URL) addressing the server, and wherein the physical object placer is configured to connect to the server using the URL, the server having a three-dimensional model, of the physical object, to send, via the connection, the identifier to the server, and to download, from the server, the three-dimensional model of the physical object.

15. The system of claim 10 wherein the physical object placer is configured to, in response to the user touching and dragging the physical object on the display, move the physical object within the environment.

16. The system of claim 10 wherein the physical object placer is configured to, in response to the user touching and making horizontal gestures with the mobile device, pan the physical object within the environment.

17. The system of claim 10 wherein the physical object placer is configured to, in response to the user touching either side of the display of the mobile device, pan the physical object within the environment.

18. The system of claim 10 wherein the physical object placer is configured to enable a user to select a set of color choices for the physical object and display the physical object in a particular color specified by the user'"'"'s selection.

19. A non-transitory computer storage apparatus encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising:

receiving from the user a length and width of an environment;

receiving a series of photographs taken from different positions in the environment by the user with a camera on a mobile device;

deriving a three dimensional model of the Environment based on the photographs, length and width;

detecting, using a sensor on the mobile device, an identifier identifying a physical object, wherein the identifier and the environment are not co-located;

retrieving, using the detected identifier, a three-dimensional model of the physical object that is capable of being placed within a three-dimensional model of an environment;

displaying, on a display of the mobile device, the three-dimensional model of the physical object within the three-dimensional model of the environment; and

in response to user gestures on the display of the mobile device, displaying the three-dimensional model of the physical object at different places within the three-dimensional model of the environment.

20. The computer storage apparatus of claim 19, the operations further comprising:

receiving from the user the length and width of the environment;

receiving a series of photographs taken from different positions in the environment; and

deriving a three-dimensional model of the environment based on the photographs, length and width.

21. The computer storage apparatus of claim 20, the operations further comprising:

identifying features on the photographs;

matching features on adjacent photographs;

deriving a surface model for the point cloud; and

texture mapping the photographs to the surface model.

22. The computer storage apparatus of claim 19, the operations further comprising:

capturing, using a camera on the mobile device, an image of a barcode for the physical object; and

scanning the image to detect an identifier identifying the physical object on a server having a three-dimensional model of the physical object, the scanning performed by an application on the mobile device.

23. The computer storage apparatus of claim 22, wherein the scanning comprises scanning the image to detect a Uniform Resource Locator (URL) addressing the server, and wherein the retrieving comprises:

connecting to the server using the URL, the server having a three-dimensional model of the physical object;

sending, via the connection, the identifier to the server; and

downloading, from the server, the three-dimensional model of the physical object.

24. The computer storage apparatus of claim 19, the operations further comprising:

in response to the user touching and dragging a representation of the physical object on the display, moving the physical object within the environment.

25. The computer storage apparatus of claim 19, the operations further comprising:

in response to the user touching and making horizontal gestures with the mobile device, panning the physical object within the environment.

26. The computer storage apparatus of claim 19, the operations further comprising:

in response to the user touching either side of the display of the mobile device, panning the physical object within the environment.

27. The computer storage apparatus of claim 19, the operations further comprising:

enabling a user to select a set of color choices for the physical object; and

displaying the physical object in a particular color specified by the user'"'"'s selection.