Distributed asynchronous localization and mapping for augmented reality

• US 8,933,931 B2
• Filed: 11/28/2012
• Issued: 01/13/2015
• Est. Priority Date: 06/02/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Balan, Alexandru, Flaks, Jason, Hodges, Steve, Isard, Michael, Williams, Oliver, Barham, Paul, Izadi, Shahram, Hiliges, Otmar, Molyneaux, David, Kim, David
• Primary Examiner(s): Nguyen, Kimbinh T
• Assistant Examiner(s): Isanians, Raffi
• Application Number:
  US13/688,093
• Publication Number:
  US 20130169626A1
• Time in Patent Office: 776 Days
• Field of Search: None
• US Class Current: 345/419
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 2340/12 Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
• CPC Class Codes: G09G 2340/125 wherein one of the images is motion video
• CPC Class Codes: G09G 2340/14 Solving problems related to the presentation of information to be displayed
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 3/003 to produce spatial visual effects

Abstract

A system and method for providing an augmented reality environment in which the environmental mapping process is decoupled from the localization processes performed by one or more mobile devices is described. In some embodiments, an augmented reality system includes a mapping system with independent sensing devices for mapping a particular real-world environment and one or more mobile devices. Each of the one or more mobile devices utilizes a separate asynchronous computing pipeline for localizing the mobile device and rendering virtual objects from a point of view of the mobile device. This distributed approach provides an efficient way for supporting mapping and localization processes for a large number of mobile devices, which are typically constrained by form factor and battery life limitations.

Claims

1. A method for providing an augmented reality environment on a mobile device, comprising:

determining whether a localization map is required, the determining whether a localization map is required is performed on the mobile device and includes acquiring a first location associated with the mobile device, the localization map is associated with a first environment;

acquiring the localization map, the localization map includes one or more image descriptors, the one or more image descriptors are associated with one or more real objects within the first environment, the acquiring the localization map includes acquiring the localization map from a mapping server, the mapping server generates the localization map based on image information received from one or more server sensors decoupled from the mobile device, the one or more server sensors move about the first environment independently of the mobile device, the localization map includes a sparse 3-D map of the first environment;

storing the localization map on the mobile device;

determining a first pose associated with the mobile device, the determining a first pose is performed on the mobile device and includes receiving one or more images, the one or more images comprise a field of view associated with the first pose, the determining a first pose includes detecting at least one of the one or more image descriptors within the one or more images;

determining whether a rendering map is required, the determining whether a rendering map is required includes determining whether a virtual object is located within the field of view associated with the first pose, the rendering map is associated with at least a portion of the first environment, the rendering map has a higher resolution than the localization map;

acquiring the rendering map;

storing the rendering map on the mobile device, the rendering map includes a dense 3-D map of the first environment, the mobile device includes a head mounted display device;

rendering the virtual object, the rendering includes registering the virtual object in relation to the rendering map; and

displaying on the mobile device a virtual image associated with the virtual object, the virtual image corresponds with a view of the virtual object such that the virtual object is perceived to exist within the field of view associated with the first pose.

2. The method of claim 1, wherein:

the rendering is performed on the mobile device, the rendering includes receiving virtual data associated with the virtual object from the mapping server, the rendering includes generating the virtual image of the virtual object based on the virtual data, the rendering includes generating virtual sounds associated with the virtual object.

3. The method of claim 1, wherein:

the determining a first pose includes determining a first position and a first orientation of the mobile device in relation to the localization map, the determining a first pose includes discarding each of the one or more image descriptors that are associated with semi-static objects.

4. The method of claim 1, wherein:

the acquiring the localization map includes transmitting a request for the localization map to the mapping server.

5. The method of claim 1, wherein:

the acquiring the localization map includes transmitting a request for the localization map to a second mobile device, the second mobile device is located within the first environment.

6. The method of claim 1, wherein:

the determining whether a virtual object is within the field of view associated with the first pose is performed on the mobile device.

7. The method of claim 1, wherein:

the one or more server sensors are associated with an autonomous vehicle.

8. An electronic device for providing an augmented reality environment, comprising:

one or more processors, the one or more processors request a localization map from a mapping server, the mapping server generates the localization map based on image information received from one or more server sensors decoupled from the electronic device, the one or more server sensors move about a first environment independently of the electronic device;

a network interface, the network interface receives the localization map, the localization map includes one or more image descriptors, the one or more image descriptors are associated with one or more real objects within the first environment, the one or more processors determine a first pose associated with a field of view of the electronic device, the network interface receives virtual data associated with the virtual object, the one or more processors determine whether a rendering map is required based on the location of the virtual object within the field of view, the network interface receives the rendering map, the rendering map has a higher resolution than the localization map;

a memory, the memory stores the rendering map, the one or more processors render the virtual object in relation to the rendering map; and

a display, the display displays a virtual image associated with the virtual object, the virtual image corresponds with a view of the virtual object such that the virtual object is perceived to exist within the field of view.

9. The electronic device of claim 8, wherein:

the electronic device is a mobile device; and

the one or more server sensors are associated with an autonomous vehicle.

10. The electronic device of claim 8, wherein:

each of the one or more real objects are characterized as one of static objects or semi-static objects;

the localization map includes a sparse 3-D map of the first environment;

the rendering map includes a dense 3-D map of the first environment; and

the electronic device includes a head mounted display device.

11. The electronic device of claim 8, wherein:

the determination of the first pose includes determining a first position and a first orientation of the electronic device in relation to the localization map.

12. The electronic device of claim 8, wherein:

the network interface receives the localization map from the mapping server.

13. The electronic device of claim 8, wherein:

the network interface receives the localization map from a second electronic device, the second electronic device is a mobile device.

14. One or more storage devices containing processor readable code for programming one or more processors to perform a method comprising the steps of:

determining whether a localization map is required, the determining whether a localization map is required is performed on a mobile device and includes acquiring a first location associated with the mobile device, the localization map is associated with a first environment, the determining whether a localization map is required includes determining whether another map stored on the mobile device is valid and associated with the first environment;

acquiring the localization map from a mapping server, the localization map includes one or more image descriptors, the one or more image descriptors are associated with one or more real objects within the first environment, the mapping server generates the localization map based on image information received from one or more server sensors decoupled from the mobile device, the one or more server sensors move about a first environment independently of the mobile device;

storing the localization map on the mobile device;

determining a first pose associated with the mobile device, the determining a first pose is performed on the mobile device and includes receiving one or more images, the one or more images comprise a field of view associated with the first pose, the determining a first pose includes detecting at least one of the one or more image descriptors within the one or more images;

determining whether a rendering map is required, the determining whether a rendering map is required includes determining whether a virtual object is located within the field of view associated with the first pose, the determining whether a virtual object is within the field of view associated with the first pose is performed on the mobile device, the rendering map is associated with at least a portion of the first environment, the rendering map has a higher resolution than the localization map;

acquiring the rendering map;

storing the rendering map on the mobile device;

rendering the virtual object, the rendering includes registering the virtual object in relation to the rendering map; and

playing on the mobile device a virtual sound associated with the virtual object, the virtual sound is perceived to originate from a location associated with the virtual object.

15. The one or more storage devices of claim 14, wherein:

the rendering is performed on the mobile device, the rendering includes receiving virtual data associated with the virtual object from the mapping server; and

the one or more server sensors are associated with an autonomous vehicle.

16. The one or more storage devices of claim 15, wherein:

the acquiring a first location includes acquiring the first location from a GPS receiver associated with the mobile device;

the localization map includes a sparse 3-D map of the first environment;

the rendering map includes a dense 3-D map of the first environment; and

the mobile device includes a head mounted display device.

17. The one or more storage devices of claim 15, wherein:

the determining a first pose includes determining a first position and a first orientation of the mobile device in relation to the localization map, the determining a first pose includes discarding each of the one or more image descriptors that are characterized as non-static objects.

18. The one or more storage devices of claim 17, wherein:

the acquiring the localization map includes transmitting a request for the localization map to the mapping server.

19. The one or more storage devices of claim 17, wherein:

the acquiring the localization map includes transmitting a request for the localization map to a second mobile device, the second mobile device is located within the first environment.