REGISTRATION BETWEEN ACTUAL MOBILE DEVICE POSITION AND ENVIRONMENTAL MODEL

• US 20140247279A1
• Filed: 03/01/2013
• Published: 09/04/2014
• Est. Priority Date: 03/01/2013
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Marti, Lukas M., van der Merwe, Rudolph, Nicholas, Christopher G., Kassebaum, John
• Granted Patent: US 9,928,652 B2
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A user interface enables a user to calibrate the position of a three dimensional model with a real-world environment represented by that model. Using a device'"'"'s sensor, the device'"'"'s location and orientation is determined. A video image of the device'"'"'s environment is displayed on the device'"'"'s display. The device overlays a representation of an object from a virtual reality model on the video image. The position of the overlaid representation is determined based on the device'"'"'s location and orientation. In response to user input, the device adjusts a position of the overlaid representation relative to the video image.

Claims

1. A method comprising:

determining, using at least one sensor of a device, a location and orientation of the device;

displaying, on a display of the device, a video image of an environment of the device;

overlaying on the video image, by the device, a representation of at least one object from a virtual reality model, wherein the position of the overlaid representation is determined based at least in part on the location and orientation of the device;

detecting, using at least one sensor of the device, positions of one or more markers physically located within the environment; and

aligning a position of the at least one object within the video image based at least in part on the positions of the one or more markers.

2. The method of claim 1, wherein aligning further comprises:

the device automatically aligning one or more virtual markers within the virtual reality model to the one or more markers physically located within the environment.

3. The method of claim 1, wherein aligning further comprises:

the device prompting a user of the device to change the orientation of the device or the location of the device in a manner that will cause the one or more markers to enter a scope of the video image.

4. The method of claim 1, wherein the markers include one or more light fixtures.

5. The method of claim 1, wherein the markers only occur at locations in the environment corresponding to locations at which one or more corresponding virtual markers occur in the virtual environment.

6. A method comprising:

detecting, through a camera of a mobile device, a particular pattern within an environment;

locating the particular pattern in a database of patterns;

determining a set of coordinates that are mapped to the particular pattern in the database; and

based at least in part of the set of coordinates, automatically determining a location of the mobile device within the environment.

7. The method of claim 6, further comprising:

based on the determined location of the mobile device within the environment, aligning a wire frame model of the environment over an image of the environment presented on a display of the mobile device.

8. The method of claim 6, further comprising:

determining initial coordinates for a virtual model of the environment based on a global positioning system of the mobile device; and

automatically adjusting coordinates of the virtual model based on differences between (a) a position of a marker specified within the virtual model and (b) a position of the pattern relative to the mobile device.

9. The method of claim 6, wherein the pattern comprises a configuration of one or more components of a particular kind, wherein other components of the particular kind occur within the environment outside of the pattern, and wherein the pattern of the components occurs uniquely within the environment.

10. The method of claim 6, further comprising:

moving a wire frame model of the environment in response to gestures made by a user of the mobile device.

11. A computer readable-memory storing one or more instructions, which, when executed by one or more processors, cause the one or more processors to perform:

visually searching, in an image captured by a camera of a mobile device, for a group of pixels that represent a physical marker that corresponds to a virtual marker specified in a virtual model;

locating the group of pixels within the image;

based on an orientation of the group of pixels within the image, determining an orientation of the physical marker; and

aligning an orientation of the virtual model based on the orientation of the physical marker.

12. The computer-readable memory of claim 11, wherein aligning the orientation of the virtual model comprises positioning a wire frame of a portion of the virtual model over the image in a manner that causes a representation of the virtual marker in the wire frame to overlap the group of pixels within the image.

13. The computer-readable memory of claim 11, wherein the instructions, when executed by the one or more processors, further cause the one or more processors to perform:

aligning the orientation of the virtual model based on automatically matching multiple virtual markers in the virtual model to multiple physical markers visually identified by the mobile device in an environment at least particular displayed in the image.

14. The computer-readable memory of claim 11, wherein the instructions, when executed by the one or more processors, further cause the one or more processors to perform:

transmitting, over a network, to a server, data that represents coordinate changes produced by the aligning.

15. The computer-readable memory of claim 11, wherein the physical marker is a uniquely occurring visual pattern on a surface that exists both in the virtual model and in a physical environment in which the mobile device is physically located.

16. A computer readable-memory storing one or more instructions, which, when executed by one or more processors, cause the one or more processors to perform:

estimating a first position and first orientation of a mobile device based on information obtained from at least one of a global positioning system and a compass;

based on the first position and first orientation, overlaying, over an image currently being captured by a camera of the mobile device and presented on a display of the mobile device, a wire frame model that represents one or more objects shown in the image;

after overlaying the wire frame model, automatically detecting, using optical recognition techniques, a distinct physical marker in a subsequent image captured by the camera; and

in response to detecting the distinct physical marker, estimating a second position and second orientation of the mobile device;

wherein the second position and second orientation differ from the first position and first orientation.

17. The computer-readable memory of claim 16, wherein the instructions, when executed by the one or more processors, further cause the one or more processors to perform:

automatically adjusting a position of the wire frame model relative to the image based on the second position and second orientation.

18. The computer-readable memory of claim 16, wherein the instructions, when executed by the one or more processors, further cause the one or more processors to perform:

receiving one or more vocal commands through a microphone of the mobile device; and

in response to the one or more vocal commands, automatically adjusting a position of the wire frame model relative to the image in a manner that corresponds to the one or more vocal commands.

19. The computer-readable memory of claim 16, wherein the instructions, when executed by the one or more processors, further cause the one or more processors to perform:

detecting one or more physical movements through the camera; and

in response to the one or more physical movements, automatically adjusting a position of the wire frame model relative to the image in a manner that corresponds to the one or more physical movements.

20. The computer-readable memory of claim 16, wherein estimating the second position and the second orientation of the mobile device is performed based on a comparison of a position of the distinct physical marker in the image relative to coordinates mapped to a virtual marker that corresponds uniquely to the distinct physical marker.

22. The system of claim 20, wherein determining the current position comprises visually matching an arrangement of objects visually identified in the image with an arrangement of corresponding virtual objects in a virtual model upon which the wire frame model is based.

23. The system of claim 20, wherein the enhanced reality application is configured to allow the user to manipulate pixels of the image based on an alignment of the wire frame model with the image.

24. The system of claim 20, wherein the enhanced reality application is configured to detect hand movements of the user through the camera and to move the wire frame model over the image based on the hand movements.

25. The system of claim 20, wherein the image includes one or more buildings, and wherein the wire frame model represents surfaces of the one or more buildings.

21. A system comprising:

a camera;

a display; and

a computer-readable memory storing an enhanced reality application that is configured to (a) capture an image through the camera, (b) present the image on the display, (c) visually detect a visual marker within the image, (d) determine a current position based on both the visual marker and coordinates mapped to a virtual marker that uniquely matches the visual marker, and (e) overlay the image with a wire frame model in a manner that causes a virtual marker specified in the wire frame model to be aligned with the visual marker in the image.