Registration between actual mobile device position and environmental model

• US 9,928,652 B2
• Filed: 03/01/2013
• Issued: 03/27/2018
• Est. Priority Date: 03/01/2013
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Nicholas, Christopher G., Marti, Lukas M., van der Merwe, Rudolph, Kassebaum, John
• Primary Examiner(s): Cobb, Michael J
• Application Number:
  US13/782,921
• Publication Number:
  US 20140247279A1
• Time in Patent Office: 1,852 Days
• Field of Search: None
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A user interface enables a user to calibrate the position of a three dimensional model with a real-world environment represented by that model. Using a device'"'"'s sensor, the device'"'"'s location and orientation is determined. A video image of the device'"'"'s environment is displayed on the device'"'"'s display. The device overlays a representation of an object from a virtual reality model on the video image. The position of the overlaid representation is determined based on the device'"'"'s location and orientation. In response to user input, the device adjusts a position of the overlaid representation relative to the video image.

Claims

1. A method comprising:

determining, using at least one sensor of a device, a location and orientation of the device based on a direction and a speed of the movement of the device, a plurality of different signals from a plurality of sources over a time interval, and geographical locations of the plurality of sources;

displaying, on a display of the device, a video image of an environment of the device;

overlaying on the video image, by the device, at least one object from a virtual reality model of the environment, wherein a position at which the at least one object is overlaid on the video image is determined based at least in part on the location and orientation of the device determined by the at least one sensor of the device;

detecting, using at least one sensor of the device, positions of one or more markers physically located within the video image of the environment;

aligning the position of the at least one object from the virtual reality model within the video image based at least in part on the positions of the one or more markers in order to calibrate the virtual reality model environment with the video image of the environment of the device;

receiving a user lock-in input;

in response to receiving the user lock-in input, sending, by the device, an updated copy of the virtual reality model to match the current position of the overlaid at least one object to a central repository in which a final edition of the virtual model of the environment is stored,

wherein the central repository receives and stores a plurality of changed coordinates of the virtual model from a plurality of mobile devices over a period of time, compares the plurality of changed coordinates to each other, and creates an updated final edition of the virtual model by aggregating the changed coordinates of the virtual reality model having similar coordinate values from among the plurality of changed coordinates of the virtual model in order to adjust the coordinates of the final edition of the virtual model;

receiving, by the device, the updated final edition of the virtual model.

2. The method of claim 1, wherein aligning further comprises:

the device automatically aligning one or more virtual markers within the virtual reality model to the one or more markers physically located within the environment in accordance with the at least one sensor.

3. The method of claim 1, wherein aligning further comprises:

the device prompting a user of the device to change the orientation of the device or the location of the device in a manner that will cause the one or more markers to enter a scope of the video image.

4. The method of claim 1, wherein the markers include one or more light fixtures.

5. The method of claim 1, wherein the markers only occur at locations in the environment corresponding to locations at which one or more corresponding virtual markers occur in the virtual environment.

6. The method of claim 1, wherein the at least one object overlaid on the video image is a wire frame model, and wherein the method further comprises:

based on the determined location of the device within the environment, aligning the wire frame model of the environment over the video image.

7. The method of claim 6, further comprising:

receiving one or more vocal commands through a microphone of the device; and

in response to the one or more vocal commands, automatically adjusting a position of the wire frame model relative to the video image in a manner that corresponds to the one or more vocal commands.

8. The method of claim 6, further comprising:

moving the wire frame model of the environment in response to gestures made by a user of the device.

9. The method of claim 6, further comprising:

detecting one or more physical movements of the device using a camera of the device; and

in response to the one or more physical movements, automatically adjusting a position of the wire frame model relative to the video image in a manner that corresponds to the one or more physical movements.

10. The method of claim 6, wherein the video image includes one or more buildings, and wherein the wire frame model represents surfaces of the one or more buildings.

11. The method according to claim 6, further comprising:

modifying one of a shape or a size of the wire frame model of the environment in response to a gesture made by a user of the device.

12. The method according to claim 6, further comprising:

modifying coordinate information of the wire frame model to create a subsequent wire frame model according to the stored coordinate information, wherein the subsequent wire frame model is different from the wire frame model.

13. The method according to claim 1, further comprising:

in response to a user removing a surface of the at least one object in the virtual reality model, displaying an underlying surface of the at least one object.

14. The method of claim 1, wherein the positions of the one or more markers physically located within the environment are detected in response to the one or more markers in the environment appearing on a view of a camera of the device, and wherein the one or more detected markers are encoded into the virtual reality model.

15. The method of claim 1, further comprising modifying the stored geographic coordinate information of the at least one object to create a subsequent position of the at least one object overlaid on the video image that is different from the position at which the at least one object is overlaid on the video image.

16. The method of claim 1, wherein in response to movement of the device, obtaining the stored geographic coordinate information and determining an updated geographic coordinate information of the at least one object that is overlaid on the video image based on the stored geographic coordinate information.

17. The method of claim 1, wherein the coordinate information of the at least one object that is overlaid on the video image is based on a user input for aligning the position of the at least one object from the virtual reality model within the video image.

18. The method of claim 1, wherein the plurality of different signals comprise at least one of a cellular telephone signal, a WiFi network signal, and a Bluetooth signal.

19. The method according to claim 1, wherein the direction and speed of the movement of the device is determined using a compass and an accelerometer.

20. The method according to claim 1, wherein the one or more markers are distinctive markers physically located within the video image of the environment.

21. The method according to claim 1, wherein the one or more markers have a physical appearance so as to be visually distinguishable from the video image of the environment.

22. A device comprising:

one or more sensors;

one or more processors configured to;

determine, using at least one sensor of a device, a location and orientation of the device based on a direction and a speed of the movement of the device, a plurality of different signals from a plurality of sources over a time interval, and geographical locations of the plurality of sources;

display, on a display of the device, a video image of an environment of the device;

overlay on the video image, by the device, at least one object from a virtual reality model of the environment, wherein a position at which the at least one object is overlaid on the video image is determined based at least in part on the location and orientation of the device determined by the at least one sensor of the device;

detect, using at least one sensor of the device, positions of one or more markers physically located within the video image of the environment;

align the position of the at least one object from the virtual reality model within the video image based at least in part on the positions of the one or more markers in order to calibrate the virtual reality model environment with the video image of the environment of the device;

receive a user lock-in input;

in response to receiving the user lock-in input, send, by the device, an updated copy of the virtual reality model to match the current position of the overlaid at least one object to a central repository in which a final edition of the virtual model of the environment is stored,

wherein the central repository receives and stores a plurality of changed coordinates of the virtual model from a plurality of mobile devices over a period of time, compares the plurality of changed coordinates to each other, and creates an updated final edition of the virtual model by aggregating the changed coordinates of the virtual reality model having similar coordinate values from among the plurality of changed coordinates of the virtual model in order to adjust the coordinates of the final edition of the virtual model;

receive, by the device, the updated final edition of the virtual model.

23. The device of claim 22, wherein the at least one object overlaid on the video image is a wire frame model, and wherein the device further comprises one or more processors configured to:

based on the determined location of the device within the environment, align the wire frame model of the environment over the video image in accordance with the one or more sensors.

24. The device of claim 23, further comprising one or more processors configured to:

move the wire frame model of the environment in response to gestures made by a user of the device.

25. The device of claim 23, further comprising one or more processors configured to:

detect one or more physical movements of the device using a camera of the device; and

in response to the one or more physical movements, automatically adjust a position of the wire frame model relative to the video image in a manner that corresponds to the one or more physical movements.

26. The device of claim 23, further comprising one or more processors configured to:

modify coordinate information of a subsequent wire frame model according to the stored coordinate information.

27. A computer product comprising a non-transitory computer readable medium storing instructions that when executed causes one or more processors to perform a method comprising:

determining, using at least one sensor of a device, a location and orientation of the device based on a direction and a speed of the movement of the device, a plurality of different signals from a plurality of sources over a time interval, and geographical locations of the plurality of sources;

displaying, on a display of the device, a video image of an environment of the device;

overlaying on the video image, by the device, at least one object from a virtual reality model of the environment, wherein a position at which the at least one object is overlaid on the video image is determined based at least in part on the location and orientation of the device determined by the at least one sensor of the device;

detecting, using at least one sensor of the device, positions of one or more markers physically located within the video image of the environment;

aligning the position of the at least one object from the virtual reality model within the video image based at least in part on the positions of the one or more markers in order to calibrate the virtual reality model environment with the video image of the environment of the device;

receiving a user lock-in input;

in response to receiving the user lock-in input, sending, by the device, an updated copy of the virtual reality model to match the current position of the overlaid at least one object to a central repository in which a final edition of the virtual model of the environment is stored,

wherein the central repository receives and stores a plurality of changed coordinates of the virtual model from a plurality of mobile devices over a period of time, compares the plurality of changed coordinates to each other, and creates an updated final edition of the virtual model by aggregating the changed coordinates of the virtual reality model having similar coordinate values from among the plurality of changed coordinates of the virtual model in order to adjust the coordinates of the final edition of the virtual model;

receiving, by the device, the updated final edition of the virtual model.