Detecting the location of a mobile device based on semantic indicators

• US 9,524,435 B2
• Filed: 03/20/2015
• Issued: 12/20/2016
• Est. Priority Date: 03/20/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Raynaud, Daniel, Bluntschli, Boris, Cotting, Daniel
• Primary Examiner(s): Park, Soo Jin
• Application Number:
  US14/664,099
• Publication Number:
  US 20160275350A1
• Time in Patent Office: 641 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 2201/09 Recognition of logos

Abstract

A system and computer implemented method for detecting the location of a mobile device using semantic indicators is provided. The method includes receiving, using one or more processors, a plurality of images captured by a mobile device at an area. The area is associated with a set of candidate locations. Using the one or more processors, one or more feature indicators associated with the plurality of images are detected. These feature indicators include semantic features related to the area. The semantic features are compared with a plurality of stored location features for the set of candidate locations. In accordance with the comparison, a location from the set of candidate locations is selected to identify an estimated position of the mobile device.

Claims

1. A computer implemented method, comprising:

receiving, using one or more processors, a plurality of images captured by a mobile device at an area, the area being associated with a set of candidate locations;

detecting, using the one or more processors, one or more feature indicators associated with the plurality of images, the feature indicators including a plurality of different types of semantic features related to the area, the types of semantic features including at least one of objects, colors, textures, sound, temperature, or motion;

comparing, using the one or more processors, the semantic features with a plurality of stored location features;

calculating a score for one or more of the set of candidate locations based on at least a number of matching semantic features, wherein the score is weighted based on the type of semantic feature used in the comparison, such that a first type of semantic feature is weighted differently than a second type of semantic feature;

selecting, using the one or more processors, a location from the set of candidate locations based at least in part on the comparison and the weighted score; and

estimating, using the one or more processors, a position of the mobile device based at least in part on the selected location.

2. The method of claim 1, further comprising comparing position information associated with the mobile device with position information associated with the plurality of stored location features for the set of candidate locations.

3. The method of claim 1, wherein the comparing further comprises determining whether the semantic features correspond to stored location features for a given location in the set.

4. The method of claim 1, wherein the semantic features include visual features of a food image and the selected location is associated with a menu corresponding to the food image.

5. The method of claim 1, wherein the semantic features include audio data of one or more objects at the selected location.

6. The method of claim 1, wherein the semantic features include a logo associated with the selected location.

7. The method of claim 1, wherein the semantic features include a time of day and the selected location is associated with a business open at that time of day.

8. The method of claim 1, wherein the semantic features include textures of objects located within the selected location.

9. The method of claim 1, wherein the semantic features include sensor data associated with the plurality of images and the sensor data is captured by the mobile device.

10. A non-transitory computer readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method, the method comprising:

receiving a plurality of images captured by a mobile device at an area, the area being associated with a set of candidate locations;

detecting one or more feature indicators associated with the plurality of images, the feature indicators including a plurality of different types of semantic features related to the area, the types of semantic features including at least one of objects, colors, textures, sound, temperature, or motion;

comparing the semantic features with a plurality of stored location features for the set of candidate locations;

calculating a score for one or more of the set of candidate locations based on at least a number of matching semantic features, wherein the score is weighted based on the type of semantic feature used in the comparison, such that a first type of semantic feature is weighted differently than a second type of semantic feature;

selecting a location from the set of candidate locations based at least in part on the comparison and the weighted score; and

estimating a position of the mobile device based at least in part on the selected location.

11. The non-transitory computer readable medium of claim 10, the method further comprising comparing position information associated with the mobile device with position information associated with the plurality of stored location features for the set of candidate locations.

12. A system comprising:

a memory storing a plurality of location features for a set of candidate locations; and

one or more processors in communication with the memory, the one or more processors being configured to;

receive a plurality of images captured by a mobile device at an area, the area being associated with a set of candidate locations;

detect one or more feature indicators associated with the plurality of images, the feature indicators including a plurality of different types of semantic features related to the area, the types of semantic features including at least one of objects, colors, textures, sound, temperature, or motion;

compare the semantic features with a plurality of location features stored in memory;

calculate a score for one or more of the set of candidate locations based on at least a number of matching semantic features, wherein the score is weighted based on the type of semantic feature used in the comparison, such that a first type of semantic feature is weighted differently than a second type of semantic feature;

select a location from the set of candidate locations based at least in part on the comparison and the weighted score; and

estimate a position of the mobile device based at least in part on the selected location.

13. The system of claim 12, wherein the one or more processors are further configured to compare position information associated with the mobile device with position information associated with the plurality of stored location features for the set of candidate locations.

14. The system of claim 12, wherein the comparing further comprises determining whether the semantic features correspond to stored location features for a given location in the set.

15. The system of claim 12, wherein the semantic features include visual features of a food image and the selected location is associated with a menu corresponding to the food image.

16. The system of claim 12, wherein the semantic features include audio data of one or more objects at the selected location.

17. The system of claim 12, wherein the semantic features include a logo associated with the selected location.

18. The system of claim 12, wherein the semantic features include a time of day and the selected location is associated with a business open at that time of day.

19. The system of claim 12, wherein the semantic features include textures of objects located within the selected location.

20. The system of claim 12, wherein the semantic features include sensor data associated with the plurality of images and the sensor data is captured by the mobile device.