DISTRIBUTED ASYNCHRONOUS LOCALIZATION AND MAPPING FOR AUGMENTED REALITY

• US 20130169626A1
• Filed: 11/28/2012
• Published: 07/04/2013
• Est. Priority Date: 06/02/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Balan, Alexandru, Flaks, Jason, Hodges, Steve, Isard, Michael, Williams, Oliver, Barham, Paul, Izadi, Shahram, Molyneaux, David, Kim, David, Hiliges, Otmar
• Granted Patent: US 8,933,931 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 2340/12 Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
• CPC Class Codes: G09G 2340/125 wherein one of the images is motion video
• CPC Class Codes: G09G 2340/14 Solving problems related to the presentation of information to be displayed
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 3/003 to produce spatial visual effects

Abstract

A system and method for providing an augmented reality environment in which the environmental mapping process is decoupled from the localization processes performed by one or more mobile devices is described. In some embodiments, an augmented reality system includes a mapping system with independent sensing devices for mapping a particular real-world environment and one or more mobile devices. Each of the one or more mobile devices utilizes a separate asynchronous computing pipeline for localizing the mobile device and rendering virtual objects from a point of view of the mobile device. This distributed approach provides an efficient way for supporting mapping and localization processes for a large number of mobile devices, which are typically constrained by form factor and battery life limitations.

Claims

1. A method for providing an augmented reality environment on a mobile device, comprising:

determining whether a localization map is required, the step of determining whether a localization map is required is performed on the mobile device and includes acquiring a first location associated with the mobile device, the localization map is associated with a first environment;

acquiring the localization map, the localization map includes one or more image descriptors, the one or more image descriptors are associated with one or more real objects within the first environment;

storing the localization map on the mobile device;

determining a first pose associated with the mobile device, the step of determining a first pose is performed on the mobile device and includes receiving one or more images, the one or more images comprise a field of view associated with the first pose, the step of determining a first pose includes detecting at least one of the one or more image descriptors within the one or more images;

determining whether a rendering map is required, the step of determining whether a rendering map is required includes determining whether a virtual object is located within the field of view associated with the first pose, the rendering map is associated with at least a portion of the first environment, the rendering map has a higher resolution than the localization map;

acquiring the rendering map;

storing the rendering map on the mobile device;

rendering the virtual object, the step of rendering includes registering the virtual object in relation to the rendering map; and

displaying on the mobile device a virtual image associated with the virtual object, the virtual image corresponds with a view of the virtual object such that the virtual object is perceived to exist within the field of view associated with the first pose.

2. The method of claim 1, wherein:

the step of rendering is performed on the mobile device, the step of rendering includes receiving virtual data associated with the virtual object from a mapping server, the step of rendering includes generating the virtual image of the virtual object based on the virtual data, the step of rendering includes generating virtual sounds associated with the virtual object.

3. The method of claim 1, wherein:

the step of determining whether a localization map is required includes determining whether another map stored on the mobile device is valid and associated with the first environment;

the step of acquiring a first location includes acquiring the first location from a GPS receiver associated with the mobile device;

each of the one or more real objects are characterized as one of static objects or semi-static objects;

the localization map includes a sparse 3-D map of the first environment;

the rendering map includes a dense 3-D map of the first environment; and

the mobile device includes a head mounted display device.

4. The method of claim 1, wherein:

the step of determining a first pose includes determining a first position and a first orientation of the mobile device in relation to the localization map, the step of determining a first pose includes discarding each of the one or more image descriptors that are associated with semi-static objects.

5. The method of claim 1, wherein:

the step of acquiring the localization map includes transmitting a request for the localization map to a mapping server.

6. The method of claim 1, wherein:

the step of acquiring the localization map includes transmitting a request for the localization map to a second mobile device, the second mobile device is located within the first environment.

7. The method of claim 1, wherein:

the step of determining whether a virtual object is within the field of view associated with the first pose is performed on the mobile device.

8. The method of claim 1, wherein:

the step of acquiring the localization map includes transmitting the first location associated with the mobile device.

9. An electronic device for providing an augmented reality environment, comprising:

one or more processors, the one or more processors determine whether a localization map is required, the one or more processors request the localization map from a mapping server;

a network interface, the network interface receives the localization map, the localization map includes one or more image descriptors, the one or more image descriptors are associated with one or more real objects within a first environment, the one or more processors determine a first pose associated with a field of view of the electronic device, the network interface receives virtual data associated with the virtual object, the one or more processors determine whether a rendering map is required based on the location of the virtual object within the field of view, the network interface receives the rendering map, the rendering map has a higher resolution than the localization map;

a memory, the memory stores the rendering map, the one or more processors render the virtual object in relation to the rendering map; and

a display, the display displays a virtual image associated with the virtual object, the virtual image corresponds with a view of the virtual object such that the virtual object is perceived to exist within the field of view.

10. The electronic device of claim 9, wherein:

the electronic device is a mobile device; and

the network interface receives virtual data from a mapping server.

11. The electronic device of claim 9, wherein:

each of the one or more real objects are characterized as one of static objects or semi-static objects;

the localization map includes a sparse 3-D map of the first environment;

the rendering map includes a dense 3-D map of the first environment; and

the electronic device includes a head mounted display device.

12. The electronic device of claim 9, wherein:

the determination of the first pose includes determining a first position and a first orientation of the electronic device in relation to the localization map.

13. The electronic device of claim 9, wherein:

the network interface receives the localization map from a mapping server.

14. The electronic device of claim 9, wherein:

the network interface receives the localization map from a second electronic device, the second electronic device is a mobile device.

15. One or more storage devices containing processor readable code for programming one or more processors to perform a method comprising the steps of:

determining whether a localization map is required, the step of determining whether a localization map is required is performed on a mobile device and includes acquiring a first location associated with the mobile device, the localization map is associated with a first environment, the step of determining whether a localization map is required includes determining whether another map stored on the mobile device is valid and associated with the first environment;

acquiring the localization map, the localization map includes one or more image descriptors, the one or more image descriptors are associated with one or more real objects within the first environment;

storing the localization map on the mobile device;

determining a first pose associated with the mobile device, the step of determining a first pose is performed on the mobile device and includes receiving one or more images, the one or more images comprise a field of view associated with the first pose, the step of determining a first pose includes detecting at least one of the one or more image descriptors within the one or more images;

determining whether a rendering map is required, the step of determining whether a rendering map is required includes determining whether a virtual object is located within the field of view associated with the first pose, the step of determining whether a virtual object is within the field of view associated with the first pose is performed on the mobile device, the rendering map is associated with at least a portion of the first environment, the rendering map has a higher resolution than the localization map;

acquiring the rendering map;

storing the rendering map on the mobile device;

rendering the virtual object, the step of rendering includes registering the virtual object in relation to the rendering map; and

playing on the mobile device a virtual sound associated with the virtual object, the virtual sound is perceived to originate from a location associated with the virtual object.

16. The one or more storage devices of claim 15, wherein:

the step of rendering is performed on the mobile device, the step of rendering includes receiving virtual data associated with the virtual object from a mapping server.

17. The one or more storage devices of claim 16, wherein:

the step of acquiring a first location includes acquiring the first location from a GPS receiver associated with the mobile device;

the localization map includes a sparse 3-D map of the first environment;

the rendering map includes a dense 3-D map of the first environment; and

the mobile device includes a head mounted display device.

18. The one or more storage devices of claim 16, wherein:

the step of determining a first pose includes determining a first position and a first orientation of the mobile device in relation to the localization map, the step of determining a first pose includes discarding each of the one or more image descriptors that are characterized as non-static objects.

19. The one or more storage devices of claim 18, wherein:

the step of acquiring the localization map includes transmitting a request for the localization map to a mapping server.

20. The one or more storage devices of claim 18, wherein:

the step of acquiring the localization map includes transmitting a request for the localization map to a second mobile device, the second mobile device is located within the first environment.