Scalable 3D mapping system

• US 10,185,775 B2
• Filed: 09/22/2015
• Issued: 01/22/2019
• Est. Priority Date: 12/19/2014
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm Technologies Incorporated (Qualcomm, Inc.)
• Inventors: Schmalstieg, Dieter, Arth, Clemens, Pirchheim, Christian
• Primary Examiner(s): Cheema, Azam M
• Application Number:
  US14/862,050
• Publication Number:
  US 20160179830A1
• Time in Patent Office: 1,218 Days
• CPC Class Codes: G06F 16/2264 Multidimensional index structures
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
• CPC Class Codes: G06F 16/29 Geographical information databases
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/9537 Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2219/2004 Aligning objects, relative positioning of parts

Abstract

Disclosed are a system, apparatus, and method for multiple client simultaneous localization and mapping. Tracking and mapping may be performed locally and independently by each of a plurality of clients. At configurable points in time map data may be sent to a server for stitching and fusion. In response to successful stitching and fusion to one or more maps known to the server, updated position and orientation information relative to the server'"'"'s maps may be sent back to the clients. Clients may update their local map data with the received server location data. Clients may receive additional map data from the server, which can be used for extending their maps. Clients may send queries to the server for 3D maps, and the queries may include metadata.

Claims

1. A method performed by a server for implementing a 3D map database, the method comprising:

receiving one or more keyframes and associated metadata from a client;

storing the received one or more keyframes in the 3D map database;

tagging the received one or more keyframes in the 3D map database according to the received metadata;

stitching, according to a stitching processing schedule of the server, the received one or more keyframes with one or more other keyframes or 3D maps in the 3D map database;

storing, in the 3D map database, a result of the stitching as a stitched 3D map tagged with at least some of the associated metadata;

receiving a client request for map data;

searching the 3D map database that includes the stitched 3D map for map data according to the client request; and

providing, in response to the searching, a resulting map data to the client requesting the map data.

2. The method of claim 1, wherein the provided map data includes one or more of:

texture data, vertex data, matrix data, or any combination thereof.

3. The method of claim 2, wherein the map data provides an anchor point to enable synchronizing of corresponding client and server maps.

4. The method of claim 1, wherein the metadata includes one or more of:

location data, client sensor data, user identification data, application identification data, temporal data, or any combination thereof.

5. The method of claim 1, further comprising:

searching the 3D map database for 3D maps to stitch with the received one or more keyframes and the associated metadata, and wherein the stitching is performed according to results of the search.

6. The method of claim 1, wherein the stitching is performed in response to receiving the one or more keyframes.

7. The method of claim 1, wherein in response to receiving the client request for map data the server selects a previously stitched map to send to the client.

8. The method of claim 1, wherein the client request includes search terms comprising one or both of metadata or a 3D map database query.

9. The method of claim 1, further comprising:

purging outdated keyframes in response to receiving a keyframe or metadata indicating a threshold level of environment changes have occurred within an area captured by the outdated keyframes.

10. A server to implement a 3D map database, the server comprising:

memory; and

a processor coupled to the memory and configured to;

receive one or more keyframes and associated metadata from a client;

store the received one or more keyframes in the 3D map database;

tag the received one or more keyframes in the 3D map database according to the received metadata;

stitch, according to a stitching processing schedule of the server, the received one or more keyframes with one or more other keyframes or 3D maps in the 3D map database;

store, in the 3D map database, a result of the stitching as a stitched 3D map tagged with at least some of the associated metadata;

receive a client request for map data;

search the 3D map database that includes the stitched 3D map for map data according to the client request; and

provide, in response to the searching, a resulting map data to the client requesting the map data.

11. The server of claim 10, wherein the provided map data includes one or more of:

texture data, vertex data, matrix data, or any combination thereof.

12. The server of claim 11, wherein the map data provides an anchor point to enable synchronizing of corresponding client and server maps.

13. The server of claim 10, wherein the metadata includes one or more of:

location data, client sensor data, user identification data, application identification data, temporal data, or any combination thereof.

14. The server of claim 10, wherein the processor is configured to:

search the 3D map database for 3D maps to stitch with the received one or more keyframes and the associated metadata; and

perform the stitching according to results of the search.

15. The server of claim 10, wherein the processor is configured to perform the stitching in response to receiving the one or more keyframes.

16. The server of claim 10, wherein in response to receiving the client request for map data the server selects a previously stitched map to send to the client.

17. The server of claim 10, wherein the client request includes search terms comprising one or both of metadata or a 3D map database query.

18. The server of claim 10, further comprising instructions to:

purge outdated keyframes in response to receiving a keyframe or metadata indicating a threshold level of environment changes have occurred within an area captured by the outdated keyframes.

19. An apparatus for implementing a 3D map database, the apparatus comprising:

means for receiving one or more keyframes and associated metadata from a client;

means for storing the received one or more keyframes in the 3D map database;

means for tagging the received one or more keyframes in the 3D map database according to the received metadata;

means for stitching, according to a stitching processing schedule of the apparatus, the received one or more keyframes with one or more other keyframes or 3D maps in the 3D map database;

means for storing, in the 3D map database, a result of the stitching as a stitched 3D map tagged with at least some of the associated metadata;

means for receiving a client request for map data;

means for searching the 3D map database that includes the stitched 3D map for map data according to the client request; and

means for providing, in response to the searching, a resulting map data to the client requesting the map data.

20. The apparatus of claim 19, wherein the provided map data includes one or more of:

texture data, vertex data, matrix data, or any combination thereof.

21. The apparatus of claim 20, wherein the map data provides an anchor point to enable synchronizing of corresponding client and server maps.

22. The apparatus of claim 19, wherein the metadata includes one or more of:

location data, client sensor data, user identification data, application identification data, temporal data, or any combination thereof.

23. The apparatus of claim 19, further comprising:

means for searching the 3D map database for 3D maps to stitch with the received one or more keyframes and the associated metadata, wherein the means for stitching comprises means for stitching according to results of the search.

24. The apparatus of claim 19, wherein the means for stitching performs the stitching in response to receiving the one or more keyframes by the means for receiving the one or more keyframes.

25. The apparatus of claim 19, further comprising:

means for purging outdated keyframes in response to receiving a keyframe or metadata indicating a threshold level of environment changes have occurred within an area captured by the outdated keyframes.

26. A machine readable non-transitory storage medium having stored therein program instructions that are executable by a server'"'"'s processor to:

receive one or more keyframes and associated metadata from a client;

store the received one or more keyframes in a 3D map database;

tag the received one or more keyframes in the 3D map database according to the received metadata;

stitch, according to a stitching processing schedule of the server, the received one or more keyframes with one or more other keyframes or 3D maps in the 3D map database;

store, in the 3D map database, a result of the stitching as a stitched 3D map tagged with at least some of the associated metadata;

receive a client request for map data;

search the 3D map database that includes the stitched 3D map for map data according to the client request; and

provide, in response to the searching, a resulting map data to the client requesting the map data.

27. The medium of claim 26, wherein the provided map data includes one or more of:

texture data, vertex data, matrix data, or any combination thereof.

28. The medium of claim 26, wherein the metadata includes one or more of:

location data, client sensor data, user identification data, application identification data, temporal data, or any combination thereof.

29. The medium of claim 26, further comprising instructions to:

search the 3D map database for 3D maps to stitch with the received one or more keyframes and the associated metadata, and wherein the instructions are executable to by the server'"'"'s processor to perform the stitching according to results of the search.

30. The medium of claim 26, further comprising instructions to:

purge outdated keyframes in response to receiving a keyframe or metadata indicating a threshold level of environment changes have occurred within an area captured by the outdated keyframes.