Mobile device, server arrangement and method for augmented reality applications

• US 9,728,007 B2
• Filed: 05/18/2010
• Issued: 08/08/2017
• Est. Priority Date: 05/18/2010
• Status: Active Grant

• Current Assignee: Teknologian tutkimuskeskus VTT Oy
• Original Assignee: Teknologian tutkimuskeskus VTT Oy
• Inventors: Hakkarainen, Mika, Kantonen, Tuomas
• Primary Examiner(s): Chowdhury, Nigar
• Application Number:
  US13/698,668
• Publication Number:
  US 20130088577A1
• Time in Patent Office: 2,639 Days
• Field of Search: 348 42, 348 47- 54
• CPC Class Codes: G01S 19/42 Determining position
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 23/63 by using electronic viewfinders

Abstract

A mobile device (104, 204) comprising communications interface (324) for transferring data, a digital camera (328) for obtaining a camera view (102) of the surroundings of the mobile device at a location, a display (330) for visualizing data such as the camera view (102), and an augmented reality (AR) entity (332) configured to transmit, via the communications interface, an indication of the location of the mobile device to an external entity, obtain, by applying data transfer from said external entity via the communications interface, a representation determined on the basis of a number of 3D models (202) of one or more virtual elements (102e, 206a) deemed as visually observable from the location of the mobile device, said representation forming at least a part of at least an approximation of spherical projection (108, 208, 210) of said 3D models, wherein the associated spherical surface is configured to surround the location of the mobile device, and produce an AR view (102d) for visualization on the display on the basis of the camera view and orientation-wise matching portion, such as 2D images and/or parts thereof, of the representation. Related server arrangement and methods are presented.

Claims

1. A mobile device comprising

a communications interface for transferring data,

a digital camera for obtaining a camera view of the surroundings of the mobile device at a real world location,

a display for visualizing data such as the camera view, and

an augmented reality (AR) entity configured to

transmit, via the communications interface, an indication of said real world location of the mobile device to an external entity,

receive, by applying data transfer from said external entity via the communications interface, a representation of one or more virtual elements augmented within said camera view, each said virtual element defined by three dimensional (3D) model which is associated with a location within said camera view, each of said one or more virtual elements including a location relative to said real world location of said mobile device, said representation including at least an approximation of a spherical projection, said approximation of said spherical projection defining a virtual environment comprising an associated spherical surface of said spherical projection, which is configured to surround at least said real world location of said mobile device and at least one of said 3D models at an orientation with respect to an orientation of said mobile device corresponding to said real world location of the mobile device, and,

produce an AR view for visualization on said display on the basis of said camera view, such that said at least one of said 3D models of the representation, presents in two dimensions on said display.

2. The mobile device of claim 1, wherein said representation comprises a plurality of 2D images.

3. The mobile device of claim 1, wherein the representation includes polyhedrons, wherein each face of each polyhedron is associated with at least one image.

4. The mobile device of claim 1, wherein the spherical surface is mapped with a texture.

5. The mobile device of claim 1, comprising a satellite receiver, including a GPS (Global Positioning System) receiver, for determining the location.

6. The mobile device of claim 1, comprising an entity for network-based or assisted location determination.

7. The mobile device of claim 1, configured to utilize user input for location and/or orientation determination.

8. The mobile device of claim 1, comprising a position sensor for determining the orientation of the mobile device and/or the camera thereof.

9. The mobile device of claim 1, comprising a position sensor for determining the orientation of the mobile device or the camera thereof, wherein the position sensor comprises an element selected from the group consisting of:

an inclinometer, an accelerometer, and a gyroscope.

10. The mobile device of claim 1, configured to utilize camera image data for determining the orientation of the mobile device and/or camera thereof.

11. The mobile device of claim 1, being further configured to obtain a representation relative to an estimated future location thereof and utilize the representation upon arrival at the future location.

12. A system comprising a mobile device comprising

a communications interface for transferring data,

a digital camera for obtaining a camera view of the surroundings of the mobile device at a real world location,

a display for visualizing data such as the camera view, and

an augmented reality (AR) entity configured to

transmit, via the communications interface, an indication of the said real world location of the mobile device to an external entity,

receive, by applying data transfer from said external entity via the communications interface, a representation of one or more virtual elements augmented within said camera view, each said virtual element defined by at least one three dimensional (3D) model which is associated with a location within said camera view, each of said one or more virtual elements including a location relative to said real world location of said mobile device, said representation including at least an approximation of a spherical projection, the approximation of the spherical projection defining a virtual environment comprising an associated spherical surface of said spherical projection, which is configured to surround at least said real world location of said mobile device and at least one of said 3D models at an orientation with respect to an orientation of said mobile device corresponding to said real world location of the mobile device, and,

produce an AR view for visualization on said display on the basis of said camera view, such that said at least one of said 3D models of the representation, presents in two dimensions on said display;

said system further comprising a server arrangement comprising;

a data interface for communicating with external entities such as a network infrastructure and a mobile device connected thereto,

a model repository for maintaining a number of 3D models of one or more virtual elements and their locations relative to said real world locations, and,

a constructor entity configured to determine, on the basis of an indication of a real world location of a mobile device obtained via the data interface and on the basis of a number of 3D models of one or more virtual elements including a location relative to said real world location of the mobile device, a said representation including at least an approximation of a spherical projection, the approximation of the spherical projection configured for surrounding the mobile device at the real world location of the mobile device, and to transmit the representation via the data interface to the mobile device to facilitate producing an AR (augmented reality) view on the display of the mobile device.

13. A method for creating augmented reality (AR) view comprising:

providing an indication of a real world location of a mobile device,

obtaining a representation, comprising a plurality of two dimensional (2D) images, determined based on a plurality of three dimensional (3D) models of virtual elements including a location relative to the real world location of the mobile device, said representation including at least an approximation of a spherical projection, the approximation of the spherical projection defining a virtual environment comprising an associated spherical surface of said spherical projection, which is configured to surround at least the real world location of the mobile device and at least one of the 3D models at an orientation with respect to an orientation of said mobile device corresponding to said real world location of the mobile device, and,

obtaining a digital camera view by the mobile device at said real world location of the mobile device, and,

producing an AR view for visualization on a display on the basis of the camera view and an orientation-wise matching portion of the representation.

14. A method for providing data for augmented reality (AR) application, comprising

obtaining an indication of the real world location of a mobile device,

constructing a representation, comprising a plurality of two dimensional (2D) images, determined based on a plurality of three dimensional (3D) models of virtual elements including a location relative to the real world location of the mobile device, said representation including at least an approximation of a spherical projection, the approximation of the spherical projection defining a virtual environment comprising an associated spherical surface of said spherical projection, which is configured to surround at least the real world location of the mobile device and at least one of the 3D models at an orientation with respect to an orientation of said mobile device corresponding to said real world location of the mobile device, and,

transmitting the representation to the mobile device to facilitate it producing an AR view.

15. A method for augmenting reality, comprising

obtaining an indication of the real world location of a mobile device,

constructing a representation, comprising a plurality of two dimensional (2D) images, determined based on a plurality of three dimensional (3D) models of virtual elements including a location relative to the real world location of the mobile device, said representation including at least an approximation of a spherical projection, the approximation of the spherical projection defining a virtual environment comprising an associated spherical surface of said spherical projection, which is configured to surround at least the real world location of the mobile device and at least one of the 3D models at an orientation with respect to an orientation of said mobile device corresponding to said real world location of the mobile device, and,

producing an AR (augmented reality) view for visualization on a display on the basis of the camera view and an orientation-wise matching portion of the representation.