Registration between actual mobile device position and environmental model

• US 10,217,290 B2
• Filed: 03/26/2018
• Issued: 02/26/2019
• Est. Priority Date: 03/01/2013
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Nicholas, Christopher G., Marti, Lukas M., van der Merwe, Rudolph, Kassebaum, John
• Primary Examiner(s): Cobb, Michael J
• Application Number:
  US15/936,286
• Publication Number:
  US 20180218542A1
• Time in Patent Office: 337 Days
• Field of Search: None
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A user interface enables a user to calibrate the position of a three dimensional model with a real-world environment represented by that model. Using a device'"'"'s sensor, the device'"'"'s location and orientation is determined. A video image of the device'"'"'s environment is displayed on the device'"'"'s display. The device overlays a representation of an object from a virtual reality model on the video image. The position of the overlaid representation is determined based on the device'"'"'s location and orientation. In response to user input, the device adjusts a position of the overlaid representation relative to the video image.

Claims

1. A method comprising:

capturing, by a camera of a mobile device, an image of an environment;

detecting a particular pattern within the image of the environment captured by the camera;

locating the particular pattern in a database of patterns;

determining a set of coordinates that are mapped to the particular pattern in the database;

based at least in part of the set of coordinates, automatically determining a location of the mobile device within the environment;

based on the determined location of the mobile device within the environment, aligning a wire frame model of a virtual model of the environment over an image of the environment presented on a display of the mobile device in order to obtain calibration information;

receiving a user lock-in input; and

in response to receiving the lock-in input, sending, by the mobile device, the calibration information to a central repository configured to store a most recently updated calibration information,

wherein the central repository is configured to receive and store a plurality of calibration information over a period of time and aggregates the plurality of calibration information to create an updated virtual model.

2. The method of claim 1, further comprising:

determining initial coordinates for a virtual model of the environment based on a global positioning system of the mobile device; and

automatically adjusting coordinates of the virtual model based on differences between a position of a marker specified within the virtual model and a position of the pattern relative to the mobile device.

3. The method of claim 1, wherein the pattern comprises a configuration of one or more components of a first type, wherein other components of the first type occur within the environment outside of the pattern, and wherein the pattern of the components occurs uniquely within the environment.

4. The method of claim 1, further comprising:

moving the wire frame model of the environment in response to gestures made by a user of the mobile device.

5. The method of claim 1, wherein the wire frame model represents objects in the virtual model.

6. The method of claim 1, further comprising displaying the wire frame model on a display;

and

in response to input gestures made on the display, modifying a position of the wire frame model.

7. The method of claim 1, further comprising:

receiving one or more vocal commands through a microphone of the mobile device; and

in response to the one or more vocal commands, automatically adjusting a position of the wire frame model relative to the image in a manner that corresponds to the one or more vocal commands.

8. A non-transitory computer readable-medium storing one or more instructions, which, when executed by one or more processors, cause the one or more processors to perform:

capturing, by a camera of a mobile device, an image of an environment;

detecting a particular pattern within the image of the environment captured by the camera;

locating the particular pattern in a database of patterns;

determining a set of coordinates that are mapped to the particular pattern in the database;

based at least in part of the set of coordinates, automatically determining a location of the mobile device within the environment;

based on the determined location of the mobile device within the environment, aligning a wire frame model of the environment over an image of the environment presented on a display of the mobile device in order to obtain calibration information, wherein the wire frame model is a virtual model of the environment;

receiving a user lock-in input; and

in response to receiving the lock-in input, sending, by the mobile device, the calibration information to a central repository configured to store a most recently updated calibration information,

wherein the central repository is configured to receive and store a plurality of calibration information over a period of time and aggregates the plurality of calibration information to create an updated virtual model.

9. The non-transitory computer readable-medium of claim 8, further comprising:

determining initial coordinates for a virtual model of the environment based on a global positioning system of the mobile device; and

automatically adjusting coordinates of the virtual model based on differences between a position of a marker specified within the virtual model and a position of the pattern relative to the mobile device.

10. The non-transitory computer readable-medium of claim 8, wherein the pattern comprises a configuration of one or more components of a first type, wherein other components of the first type occur within the environment outside of the pattern, and wherein the pattern of the components occurs uniquely within the environment.

11. The non-transitory computer readable-medium of claim 8, further comprising:

moving the wire frame model of the environment in response to gestures made by a user of the mobile device.

12. The non-transitory computer readable-medium of claim 8, wherein the wire frame model represents objects in the virtual model.

13. The non-transitory computer readable-medium of claim 8, further comprising displaying the wire frame model on a display;

and

in response to input gestures made on the display, modifying a position of the wire frame model.

14. The non-transitory computer readable-medium of claim 8, further comprising:

receiving one or more vocal commands through a microphone of the mobile device; and

in response to the one or more vocal commands, automatically adjusting a position of the wire frame model relative to the image in a manner that corresponds to the one or more vocal commands.

15. A mobile device comprising:

one or more sensors; and

one or more processors configured to;

capture, by a camera of the mobile device, an image of an environment;

detect a particular pattern within the environment captured by the camera;

locate the particular pattern in a database of patterns;

determine a set of coordinates that are mapped to the particular pattern in the database;

based at least in part of the set of coordinates, automatically determine a location of the mobile device within the environment;

based on the determined location of the mobile device within the environment, align a wire frame model of a virtual model of the environment over an image of the environment presented on a display of the mobile device in order to obtain calibration information;

receiving a user lock-in input; and

in response to receiving the lock-in input, sending, by the mobile device, the calibration information to a central repository configured to store a most recently updated calibration information,

wherein the central repository is configured to receive and store a plurality of calibration information over a period of time and aggregates the plurality of calibration information to create an updated virtual model.

16. The device of claim 15, further comprising:

determining initial coordinates for a virtual model of the environment based on a global positioning system of the mobile device; and

automatically adjusting coordinates of the virtual model based on differences between a position of a marker specified within the virtual model and a position of the pattern relative to the mobile device.

17. The device of claim 15, wherein the pattern comprises a configuration of one or more components of a first type, wherein other components of the first type occur within the environment outside of the pattern, and wherein the pattern of the components occurs uniquely within the environment.

18. The device of claim 15, further comprising:

moving the wire frame model of the environment in response to gestures made by a user of the mobile device.

19. The device of claim 15, wherein the wire frame model represents objects in the virtual model.

20. The device of claim 15, further comprising displaying the wire frame model on a display;

and

in response to input gestures made on the display, modifying a position of the wire frame model.