Client-Server Based Dynamic Search

• US 20140279860A1
• Filed: 03/15/2013
• Published: 09/18/2014
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Szalavari, Zsolt, Pan, Qi, Wagner, Daniel
• Granted Patent: US 9,495,389 B2
• Time in Patent Office: Days
• US Class Current: 707/609
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/9537 Spatial or temporal dependent retrieval, e.g. spatiotemporal queries

Abstract

Method, mobile device, computer program product and apparatus for performing a search are disclosed. The method of performing a search comprises receiving one or more images of an environment in view of a mobile device, generating a simultaneous localization and mapping of the environment using the one or more images, wherein the simultaneous localization and mapping of the environment comprises a plurality of map points representing a plurality of surfaces in a three dimensional coordinate system of the environment, sending a set of the plurality of map points as a search query to a server, receiving a query response from the server, and identifying an object in the environment based at least in part on the query response.

Claims

1. A method of performing a search, comprising:

receiving one or more images of an environment in view of a mobile device;

generating a simultaneous localization and mapping of the environment using the one or more images, wherein the simultaneous localization and mapping of the environment comprises a plurality of map points representing a plurality of surfaces in a three dimensional coordinate system of the environment;

sending a set of the plurality of map points as a search query to a server;

receiving a query response from the server; and

identifying an object in the environment based at least in part on the query response.

2. The method of claim 1, wherein the plurality of map points comprises:

views of the plurality of surfaces captured by the mobile device from different angles.

3. The method of claim 2, wherein each of a map point in the plurality of map points comprises a three dimensional location, one or more descriptors of the three dimensional location, and a corresponding surface normal of the three dimensional location.

4. The method of claim 2, wherein each of a map point in the plurality of map points comprises a representation from a direction orthogonal to a normal direction of the map point.

5. The method of claim 1, further comprising:

sending the set of the plurality of map points as a search query to the server at a first time instance; and

receiving a query response from the server at a second time instance; and

wherein the mobile device has a first pose relative to the object at the first time instance and the mobile device has a second pose relative to the object at the second time instance.

6. The method of claim 5, further comprising:

determining the second pose of the mobile device with respect to the object in the environment based at least in part on the query response received at the second time instance.

7. The method of claim 6, wherein determining the second pose of the mobile device with respect to the object in the environment comprises:

obtaining from the query response a pose of the object with respect to an origin of the environment;

tracking a pose of the mobile device with respect to the origin of the environment; and

determining the second pose of the mobile device with respect to the object using the pose of the object with respect to the origin of the environment and the pose of the mobile device with respect to the origin of the environment.

8. The method of claim 6, further comprising:

causing an augmentation to be rendered in response to the second pose of the mobile device with respect to the object in the environment.

9. The method of claim 1 further comprising:

tracking movement of the mobile device with respect to an origin of the environment; and

determining the mobile device has moved with respect to the origin of the environment.

10. The method of claim 9, wherein determining the mobile device has moved comprises at least one of:

determining the mobile device has rotated more than a predetermined angle; and

determining the mobile device has translated more than a predetermined distance.

11. The method of claim 9, further comprising:

identifying incremental map points captured by the mobile device in response to the movement of the mobile device;

generating an updated simultaneous localization and mapping of the environment with the incremental map points captured by the mobile device; and

sending the incremental map points captured by the mobile device in an updated search query to the server.

12. The method of claim 11, wherein the updated simultaneous localization and mapping of the environment includes views of the environment larger than a field of view of the environment from the mobile device;

and wherein the origin of the environment is out of the field of view of the mobile device.

13. A mobile device, comprising:

a control unit including processing logic, the processing logic comprising;

logic configured to receive one or more images of an environment in view of the mobile device;

logic configured to generate a simultaneous localization and mapping of the environment using the one or more images, wherein the simultaneous localization and mapping of the environment comprises a plurality of map points representing a plurality of surfaces in a three dimensional coordinate system of the environment;

logic configured to send a set of the plurality of map points as a search query to a server;

logic configured to receive a query response from the server; and

logic configured to identify an object in the environment based at least in part on the query response.

14. The mobile device of claim 13, wherein the plurality of map points comprises:

views of the plurality of surfaces captured by the mobile device from different angles.

15. The mobile device of claim 14, wherein each of a map point in the plurality of map points comprises a three dimensional location, one or more descriptors of the three dimensional location, and a corresponding surface normal of the three dimensional location.

16. The mobile device of claim 14, wherein each of a map point in the plurality of map points comprises a representation from a direction orthogonal to a normal direction of the map point.

17. The mobile device of claim 13, further comprising:

logic configured to send the set of the plurality of map points as a search query to the server at a first time instance; and

logic configured to receive a query response from the server at a second time instance; and

wherein the mobile device has a first pose relative to the object at the first time instance and the mobile device has a second pose relative to the object at the second time instance.

18. The mobile device of claim 17, further comprising:

logic configured to determine the second pose of the mobile device with respect to the object in the environment based at least in part on the query response received at the second time instance.

19. The mobile device of claim 18, wherein logic configured to determine the second pose of the mobile device with respect to the object in the environment comprises:

logic configured to obtain from the query response a pose of the object with respect to an origin of the environment;

logic configured to track a pose of the mobile device with respect to the origin of the environment; and

logic configured to determine the second pose of the mobile device with respect to the object using the pose of the object with respect to the origin of the environment and the pose of the mobile device with respect to the origin of the environment.

20. The mobile device of claim 18, further comprising:

logic configured to cause an augmentation to be rendered in response to the second pose of the mobile device with respect to the object in the environment.

21. The mobile device of claim 13 further comprising:

logic configured to track movement of the mobile device with respect to an origin of the environment; and

logic configured to determine the mobile device has moved with respect to the origin of the environment.

22. The mobile device of claim 21, wherein logic configured to determine the mobile device has moved comprises at least one of:

logic configured to determine the mobile device has rotated more than a predetermined angle; and

logic configured to determine the mobile device has translated more than a predetermined distance.

23. The mobile device of claim 21, further comprising:

logic configured to identify incremental map points captured by the mobile device in response to the movement of the mobile device;

logic configured to generate an updated simultaneous localization and mapping of the environment with the incremental map points captured by the mobile device; and

logic configured to send the incremental map points captured by the mobile device in an updated search query to the server.

24. The mobile device of claim 23, wherein the updated simultaneous localization and mapping of the environment includes views of the environment larger than a field of view of the environment from the mobile device;

and wherein the origin of the environment is out of the field of view of the mobile device.

25. A computer program product comprising a non-transitory medium storing instructions for execution by one or more computer systems, the instructions comprising:

instructions for receiving one or more images of an environment in view of a mobile device;

instructions for generating a simultaneous localization and mapping of the environment using the one or more images, wherein the simultaneous localization and mapping of the environment comprises a plurality of map points representing a plurality of surfaces in a three dimensional coordinate system of the environment;

instructions for sending a set of the plurality of map points as a search query to a server;

instructions for receiving a query response from the server; and

instructions for identifying an object in the environment based at least in part on the query response.

26. The computer program product of claim 25, wherein the plurality of map points comprises:

views of the plurality of surfaces captured by the mobile device from different angles.

27. The computer program product of claim 26, wherein each of a map point in the plurality of map points comprises a three dimensional location, one or more descriptors of the three dimensional location, and a corresponding surface normal of the three dimensional location.

28. The computer program product of claim 26, wherein each of a map point in the plurality of map points comprises a representation from a direction orthogonal to a normal direction of the map point.

29. The computer program product of claim 25, further comprising:

instructions for sending the set of the plurality of map points as a search query to the server at a first time instance; and

instructions for receiving a query response from the server at a second time instance; and

wherein the mobile device has a first pose relative to the object at the first time instance and the mobile device has a second pose relative to the object at the second time instance.

30. The computer program product of claim 29, further comprising:

instructions for determining the second pose of the mobile device with respect to the object in the environment based at least in part on the query response received at the second time instance.

31. The computer program product of claim 30, wherein instructions for determining the second pose of the mobile device with respect to the object in the environment comprises:

instructions for obtaining from the query response a pose of the object with respect to an origin of the environment;

instructions for tracking a pose of the mobile device with respect to the origin of the environment; and

instructions for determining the second pose of the mobile device with respect to the object using the pose of the object with respect to the origin of the environment and the pose of the mobile device with respect to the origin of the environment.

32. The computer program product of claim 30, further comprising:

instructions for causing an augmentation to be rendered in response to the second pose of the mobile device with respect to the object in the environment.

33. The computer program product of claim 25 further comprising:

instructions for tracking movement of the mobile device with respect to an origin of the environment; and

instructions for determining the mobile device has moved with respect to the origin of the environment.

34. The computer program product of claim 33, wherein instructions for determining the mobile device has moved comprises at least one of:

instructions for determining the mobile device has rotated more than a predetermined angle; and

instructions for determining the mobile device has translated more than a predetermined distance.

35. The computer program product of claim 33, further comprising:

instructions for identifying incremental map points captured by the mobile device in response to the movement of the mobile device;

instructions for generating an updated simultaneous localization and mapping of the environment with the incremental map points captured by the mobile device; and

instructions for sending the incremental map points captured by the mobile device in an updated search query to the server.

36. The computer program product of claim 35, wherein the updated simultaneous localization and mapping of the environment includes views of the environment larger than a field of view of the environment from the mobile device;

and wherein the origin of the environment is out of the field of view of the mobile device.

37. An apparatus, comprising:

means for receiving one or more images of an environment in view of a mobile device;

means for generating a simultaneous localization and mapping of the environment using the one or more images, wherein the simultaneous localization and mapping of the environment comprises a plurality of map points representing a plurality of surfaces in a three dimensional coordinate system of the environment;

means for sending a set of the plurality of map points as a search query to a server;

means for receiving a query response from the server; and

means for identifying an object in the environment based at least in part on the query response.