Augmented reality interface for interacting with displayed maps

• US 10,643,373 B2
• Filed: 05/16/2018
• Issued: 05/05/2020
• Est. Priority Date: 06/19/2017
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Fillhardt, Nathan L., Lindberg, Adrian P., Arroyo, Vincent P., Strawn, Justin M.
• Primary Examiner(s): Demeter, Hilina K
• Application Number:
  US15/981,847
• Publication Number:
  US 20180365883A1
• Time in Patent Office: 720 Days
• Field of Search: None
• CPC Class Codes: G01C 21/3638 including 3D objects and buildings three dimensional [3D] modelling, e.g. data description of 3D objects G06T17/00; geographic models G06T17/05
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80

Abstract

Various embodiments of the disclosure pertain to an augmented or virtual reality interface for interacting with maps displayed from a virtual camera perspective on a mobile device. Instead of manipulating the position of the virtual camera using a touchscreen interface, some embodiments allow a spatial location of the mobile device to control the position of the virtual camera. For example, a user can tilt the mobile device to obtain different angles of the virtual camera. As another example, the user can move the mobile device vertically to change the height of the virtual camera, e.g., a higher altitude above the ground.

Claims

1. A mobile device comprising:

a display;

a physical camera communicably coupled with the display;

a processor; and

a memory coupled to the processor, the memory storing instructions, which when executed by the processor, cause the mobile device to perform operations including;

displaying a map view on the display, the map view including a set of map objects corresponding to a location within a map from an initial virtual position of a virtual camera;

receiving user input specifying an interactive mode that uses the physical camera;

in response to the user input, capturing one or more first images of a physical environment within which the mobile device resides using the physical camera;

determining an initial position that corresponds to a physical location of the physical camera in the physical environment using the one or more first images;

specifying a set of physical positions of a set of objects of the map relative to the initial position of the physical camera, the set of objects corresponding to the set of map objects;

displaying a first virtual map image on the display, the first virtual map image being overlaid on the one or more first images;

capturing one or more second images of the physical environment within which the mobile device resides using the physical camera after movement of the mobile device;

determining a current physical position of the physical camera with respect to the initial position based on the one or more second images;

determining an updated virtual position of the virtual camera based on the current physical position of the physical camera;

rendering a second virtual map image of the set of objects based on the updated virtual position of the virtual camera; and

displaying the second virtual map image on the display, the second virtual map image being overlaid on the one or more second images.

2. The mobile device of claim 1, further comprising:

one or more sensors, wherein the current physical position of the physical camera is further determined based on one or more measurements from the one or more sensors.

3. The mobile device of claim 2, wherein the one or more sensors include at least one of:

an accelerometer, a gyroscope, and a compass.

4. The mobile device of claim 1, wherein the initial position corresponds to a physical position of the physical camera when the user input is received.

5. The mobile device of claim 1, wherein the set of physical positions of the set of objects are specified at default positions from the mobile device.

6. The mobile device of claim 5, wherein the default positions comprise a default orientation with respect to the mobile device.

7. The mobile device of claim 1, further comprising:

an orientation sensor, and wherein the operations further include;

determining an angular change between the initial position of the physical camera and the current physical position of the physical camera using the orientation sensor;

using the angular change to determine the updated virtual position, thereby rotating the virtual camera.

8. The mobile device of claim 1, wherein the operations further include:

comparing the one or more first images to the one or more second images to determine a movement vector relative to the initial position of the physical camera,

wherein determining the updated virtual position of the virtual camera based on the current physical position of the physical camera includes;

applying the movement vector to the initial virtual position of the virtual camera.

9. The mobile device of claim 8, wherein determining the updated virtual position of the virtual camera based on the current physical position of the physical camera further includes:

scaling the movement vector before applying the movement vector to the initial virtual position of the virtual camera.

10. A method of providing a view of a map on a display of a mobile device, the method comprising performing, by the mobile device having a physical camera communicably coupled with the display:

displaying a map view on the display, the map view including a set of map objects corresponding to a location within the map from an initial virtual position of a virtual camera;

receiving user input specifying an interactive mode that uses the physical camera;

in response to the user input, capturing one or more first images of a physical environment within which the mobile device resides using the physical camera;

determining an initial position that corresponds to a physical location of the physical camera in the physical environment using the one or more first images;

specifying a set of physical positions of a set of objects of the map relative to the initial position, the set of objects corresponding to the set of map objects;

displaying a first virtual map image on the display, the first virtual map image being overlaid on the one or more first images;

capturing one or more second images of the physical environment within which the mobile device resides using the physical camera after movement of the mobile device;

determining a current physical position of the physical camera with respect to the initial position based on the one or more second images;

determining an updated virtual position of the virtual camera based on the current physical position of the physical camera;

rendering a second virtual map image of the set of objects based on the updated virtual position of the virtual camera; and

displaying the second virtual map image on the display, the second virtual map image being overlaid on the one or more second images.

11. The method of claim 10, wherein the current physical position of the physical camera is further determined based on one or more measurements from one or more sensors of the mobile device.

12. The method of claim 11, wherein the one or more sensors include at least one of:

an accelerometer, a gyroscope, and a compass.

13. The method of claim 10, wherein the initial position corresponds to a physical position of the physical camera when the user input is received.

14. The method of claim 10, wherein the set of physical positions of the set of objects are specified at default positions from the mobile device.

15. The method of claim 10, further comprising:

determining an angular change between the initial position of the physical camera and the current physical position of the physical camera using an orientation sensor of the mobile device.

16. The method of claim 10, further comprising:

comparing the one or more first images to the one or more second images to determine a movement vector relative to the initial position of the physical camera,

wherein determining the updated virtual position of the virtual camera based on the current physical position of the physical camera includes applying the movement vector to the initial virtual position of the virtual camera.

17. The method of claim 16, wherein determining the updated virtual position of the virtual camera based on the current physical position of the physical camera further includes scaling the movement vector before applying the movement vector to the initial virtual position of the virtual camera.

18. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to:

display a map view on a display of a mobile device, the map view including a set of map objects corresponding to a location within the map view from an initial virtual position of a virtual camera;

receive user input specifying an interactive mode that uses a physical camera of the mobile device;

in response to the user input, capture one or more first images of a physical environment within which the mobile device resides using the physical camera;

determine an initial position that corresponds to a physical location of the physical camera in the physical environment using the one or more first images;

specify a set of physical positions of a set of objects of the map view relative to the initial position of the physical camera, the set of objects corresponding to the set of map objects;

display a first virtual map image on the display, the first virtual map image being overlaid on the one or more first images;

capture one or more second images of the physical environment within which the mobile device resides using the physical camera after movement of the mobile device;

determine a current physical position of the physical camera with respect to the initial position based on the one or more second images;

determine an updated virtual position of the virtual camera based on the current physical position of the physical camera;

render a second virtual map image of the set of objects based on the updated virtual position of the virtual camera; and

display the second virtual map image on the display, the second virtual map image being overlaid on the one or more second images.

19. The non-transitory computer-readable storage medium of claim 18, wherein the current physical position of the physical camera is further determined based on one or more measurements from one or more sensors of the mobile device.