SCALABLE 3D MAPPING SYSTEM

• US 20160179830A1
• Filed: 09/22/2015
• Published: 06/23/2016
• Est. Priority Date: 12/19/2014
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Schmalstieg, Dieter, Arth, Clemens, Pirchheim, Christian
• Granted Patent: US 10,185,775 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2264 Multidimensional index structures
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
• CPC Class Codes: G06F 16/29 Geographical information databases
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/9537 Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2219/2004 Aligning objects, relative positioning of parts

Abstract

Disclosed are a system, apparatus, and method for multiple client simultaneous localization and mapping. Tracking and mapping may be performed locally and independently by each of a plurality of clients. At configurable points in time map data may be sent to a server for stitching and fusion. In response to successful stitching and fusion to one or more maps known to the server, updated position and orientation information relative to the server'"'"'s maps may be sent back to the clients. Clients may update their local map data with the received server location data. Clients may receive additional map data from the server, which can be used for extending their maps. Clients may send queries to the server for 3D maps, and the queries may include metadata.

Claims

1. A method performed by a server for implementing a 3D map database, the method comprising:

receiving one or more keyframes and associated metadata from a client;

storing the received one or more keyframes in the 3D map database;

tagging the received one or more keyframes in the 3D map database according to the received metadata;

receiving a client request for map data;

searching the 3D map database for map data according to the client request; and

providing, in response to the searching, a resulting map data to the client requesting the map data.

2. The method of claim 1, wherein the provided map data includes one or more of:

texture data, vertex data, matrix data, or any combination thereof.

3. The method of claim 2, wherein the map data provides an anchor point to enable synchronizing of corresponding client and server maps.

4. The method of claim 1, wherein the metadata includes one or more of:

location data, client sensor data, user identification data, application identification data, temporal data, or any combination thereof.

5. The method of claim 1, further comprising:

searching the 3D map database for 3D maps to stitch with the one or more keyframes and the associated metadata;

stitching, according to the results of the search and a stitching processing schedule of the server, the one or more keyframes with one or more keyframes or 3D maps in the 3D map database; and

storing the result of stitching the one or more keyframes as a 3D map tagged with at least some of the associated metadata.

6. The method of claim 1, wherein in response to receiving the one or more keyframes, the server stitches the one or more keyframes with one or both of:

a keyframe or a 3D map from the 3D map database.

7. The method of claim 1, wherein in response to receiving the client request for map data the server selects a previously stitched map to send to the client.

8. The method of claim 1, wherein the client request includes search terms comprising one or both of metadata or a 3D map database query.

9. The method of claim 1, further comprising:

purging outdated keyframes in response to receiving a keyframe or metadata indicating a threshold level of environment changes have occurred within an area captured by the outdated keyframes.

10. A server to implement a 3D map database, the server comprising:

memory; and

a processor coupled to the memory and configured to;

receive one or more keyframes and associated metadata from a client;

store the received one or more keyframes in the 3D map database;

tag the received one or more keyframes in the 3D map database according to the received metadata;

receive a client request for map data

search the 3D map database for map data according to the client request; and

provide, in response to the searching, a resulting map data to the client requesting the map data.

11. The server of claim 10, wherein the provided map data includes one or more of:

texture data, vertex data, matrix data, or any combination thereof.

12. The server of claim 11, wherein the map data provides an anchor point to enable synchronizing of corresponding client and server maps.

13. The server of claim 10, wherein the metadata includes one or more of:

location data, client sensor data, user identification data, application identification data, temporal data, or any combination thereof.

14. The server of claim 10, further comprising instructions to:

search the 3D map database for 3D maps to stitch with the one or more keyframes and the associated metadata;

stitch, according to the results of the search and a stitching processing schedule of the server, the one or more keyframes with one or more keyframes or 3D maps in the 3D map database; and

store the result of stitching the one or more keyframes as a 3D map tagged with at least some of the associated metadata.

15. The server of claim 10, wherein in response to receiving the one or more keyframes, the server stitches the one or more keyframes with one or both of:

a keyframe or a 3D map from the 3D map database.

16. The server of claim 10, wherein in response to receiving the client request for map data the server selects a previously stitched map to send to the client.

17. The server of claim 10, wherein the client request includes search terms comprising one or both of metadata or a 3D map database query.

18. The server of claim 10, further comprising instructions to:

purge outdated keyframes in response to receiving a keyframe or metadata indicating a threshold level of environment changes have occurred within an area captured by the outdated keyframes.

19. An apparatus for implementing a 3D map database, the apparatus comprising:

means for receiving one or more keyframes and associated metadata from a client;

means for storing the received one or more keyframes in the 3D map database;

means for tagging the received one or more keyframes in the 3D map database according to the received metadata;

means for receiving a client request for map data

means for searching the 3D map database for map data according to the client request; and

means for providing, in response to the searching, a resulting map data to the client requesting the map data.

20. The apparatus of claim 19, wherein the provided map data includes one or more of:

texture data, vertex data, matrix data, or any combination thereof.

21. The apparatus of claim 20, wherein the map data provides an anchor point to enable synchronizing of corresponding client and server maps.

22. The apparatus of claim 19, wherein the metadata includes one or more of:

location data, client sensor data, user identification data, application identification data, temporal data, or any combination thereof.

23. The apparatus of claim 19, further comprising:

means for searching the 3D map database for 3D maps to stitch with the one or more keyframes and the associated metadata;

means for stitching, according to the results of the search and a stitching processing schedule of the server, the one or more keyframes with one or more keyframes or 3D maps in the 3D map database; and

means for storing the result of stitching the one or more keyframes as a 3D map tagged with at least some of the associated metadata.

24. The apparatus of claim 19, wherein in response to receiving the one or more keyframes, the server stitches the one or more keyframes with one or both of:

a keyframe or a 3D map from the 3D map database.

25. The apparatus of claim 19, further comprising:

means for purging outdated keyframes in response to receiving a keyframe or metadata indicating a threshold level of environment changes have occurred within an area captured by the outdated keyframes.

26. A machine readable non-transitory storage medium having stored therein program instructions that are executable by a server'"'"'s processor to:

receive one or more keyframes and associated metadata from a client;

store the received one or more keyframes in the 3D map database;

tag the received one or more keyframes in the 3D map database according to the received metadata;

receive a client request for map data

search the 3D map database for map data according to the client request; and

provide, in response to the searching, a resulting map data to the client requesting the map data.

27. The medium of claim 26, wherein the provided map data includes one or more of:

texture data, vertex data, matrix data, or any combination thereof.

28. The medium of claim 26, wherein the metadata includes one or more of:

location data, client sensor data, user identification data, application identification data, temporal data, or any combination thereof.

29. The medium of claim 26, further comprising instructions to:

search the 3D map database for 3D maps to stitch with the one or more keyframes and the associated metadata;

stitch, according to the results of the search and a stitching processing schedule of the server, the one or more keyframes with one or more keyframes or 3D maps in the 3D map database; and

store the result of stitching the one or more keyframes as a 3D map tagged with at least some of the associated metadata.

30. The medium of claim 26, further comprising instructions to:

purge outdated keyframes in response to receiving a keyframe or metadata indicating a threshold level of environment changes have occurred within an area captured by the outdated keyframes.