Detecting the location of a mobile device based on semantic indicators

• US 9,842,272 B2
• Filed: 11/16/2016
• Issued: 12/12/2017
• Est. Priority Date: 03/20/2015
• Status: Expired due to Fees

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Raynaud, Daniel, Bluntschli, Boris, Cotting, Daniel
• Primary Examiner(s): PARK, SOO JIN
• Application Number:
  US15/353,304
• Publication Number:
  US 20170061606A1
• Time in Patent Office: 391 Days
• Field of Search: None
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 2201/09 Recognition of logos

Abstract

A system and computer implemented method for detecting the location of a mobile device using semantic indicators is provided. The method includes receiving, using one or more processors, a plurality of images captured by a mobile device at an area. The area is associated with a set of candidate locations. Using the one or more processors, one or more feature indicators associated with the plurality of images are detected. These feature indicators include semantic features related to the area. The semantic features are compared with a plurality of stored location features for the set of candidate locations. In accordance with the comparison, a location from the set of candidate locations is selected to identify an estimated position of the mobile device.

Claims

1. A computer implemented method, comprising:

receiving, using one or more processors, a plurality of images captured by a mobile device at an area, the area being associated with a set of candidate locations;

detecting, using the one or more processors, a first type of feature indicator and a second type of feature indicator associated with the plurality of images, the first type of feature indicator being different than the second type of feature indicator, the first and the second types of feature indicators including types of semantic features related to interior portions of the area, the types of semantic features for the first type of feature indicator including at least one of;

     1) surface textures of interior objects,

     2) patterns of interior elements,

     3) interior architecture, or

     4) interior dé

cor and the types of semantic features for the second type of feature indicator including at least one of;

     1) interior noise or

     2) interior music;

identifying, using the one or more processors, a set of feature indicators including a combination of the first type of feature indicator and the second type of feature indicator;

comparing, using the one or more processors, the set of feature indicators with a plurality of stored location feature sets;

calculating a score for one or more of the set of candidate locations based on at least a number of matching semantic features, wherein the score is weighted based on the type of semantic feature used in the comparison, such that the types of semantic features for the first type of feature indicator is weighted differently than the types of semantic features for the second type of feature indicator;

selecting, using the one or more processors, a location from the set of candidate locations based at least in part on the comparison; and

estimating, using the one or more processors, a position of the mobile device based at least in part on the selected location.

2. The method of claim 1, further comprising comparing position information associated with the mobile device with position information associated with the plurality of stored location feature sets for the set of candidate locations.

3. The method of claim 1, wherein the comparing further comprises determining whether the set of feature indicators correspond to stored location feature sets for a given location in the set of candidate locations.

4. The method of claim 1, wherein the interior dé

cor includes a particular color combination.

5. The method of claim 1, wherein the estimated position indicates a type of organization associated with the selected location.

6. The method of claim 1, wherein the semantic features further include a time of day and the selected location is associated with a business open at that time of day.

7. The method of claim 1, wherein the estimated position indicates at least one of a particular business, school, or university associated with the selected location.

8. The method of claim 1, wherein the semantic features include sensor data associated with the plurality of images and the sensor data is captured by the mobile device.

9. The method of claim 1, wherein the semantic features include at least one of visual features of a food image and audio features of a machine in operation.

10. A non-transitory computer readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method, the method comprising:

receiving a plurality of images captured by a mobile device at an area, the area being associated with a set of candidate locations;

detecting a first type of feature indicator and a second type of feature indicator associated with the plurality of images, the first type of feature indicator being different than the second type of feature indicator, the first and the second types of feature indicators including types of semantic features related to interior portions of the area, the semantic features for the first type of feature indicator including at least one of;

     1) surface textures of interior objects,

     2) patterns of interior elements,

     3) interior architecture, or

     4) interior dé

cor and the types of semantic features for the second type of feature indicator including at least one of;

     1) interior noise or

     2) interior music;

identifying a set of feature indicators including a combination of the first type of feature indicator and the second type of feature indicator;

comparing the set of feature indicators with a plurality of stored location feature sets for the set of candidate locations;

calculating a score for one or more of the set of candidate locations based on at least a number of matching semantic features, wherein the score is weighted based on the type of semantic feature used in the comparison, such that the types of semantic features for the first type of feature indicator is weighted differently than the types of semantic features for the second type of feature indicator;

selecting a location from the set of candidate locations based at least in part on the comparison; and

estimating a position of the mobile device based at least in part on the selected location.

11. The non-transitory computer readable medium of claim 10, the method further comprising comparing position information associated with the mobile device with position information associated with the plurality of stored location feature set for the set of candidate locations.

12. A system comprising:

a memory storing a plurality of location features for a set of candidate locations; and

one or more processors in communication with the memory, the one or more processors being configured to;

receive a plurality of images captured by a mobile device at an area, the area being associated with a set of candidate locations;

detect a first type of feature indicator and a second type of feature indicator associated with the plurality of images, the first type of feature indicator being different than the second type of feature indicator, the first and the second types of feature indicators including types of semantic features related to interior portions of the area, the types of semantic features for the first type of feature indicator including at least one of;

     1) surface textures of interior objects,

     2) patterns of interior elements,

     3) interior architecture, or

     4) interior dé

cor and the types of semantic features for the second type of feature indicator including at least one of;

     1) interior noise or

     2) interior music;

identify a set of feature indicators including a combination of the first type of feature indicator and the second type of feature indicator;

compare the set of feature indicators with a plurality of location feature sets stored in the memory;

calculate a score for one or more of the set of candidate locations based on at least a number of matching semantic features, wherein the score is weighted based on the type of semantic feature used in the comparison, such that the types of semantic features for the first type of feature indicator is weighted differently than the types of semantic features for the second type of feature indicator;

select a location from the set of candidate locations based at least in part on the comparison; and

estimate a position of the mobile device based at least in part on the selected location.

13. The system of claim 12, wherein the one or more processors are further configured to compare position information associated with the mobile device with position information associated with the plurality of stored location feature sets for the set of candidate locations.

14. The system of claim 12, wherein the compare further comprises determining whether the set of feature indicators correspond to stored location feature sets for a given location in the set of candidate locations.

15. The system of claim 12, wherein the interior dé

cor includes a particular color combination.

16. The system of claim 12, wherein the estimated position indicates a type of organization associated with the selected location.

17. The system of claim 12, wherein the semantic features include a time of day and the selected location is associated with a business open at that time of day.

18. The system of claim 12, wherein the estimated position indicates at least one of a particular business, school, or university associated with the selected location.

19. The system of claim 12, wherein the semantic features include at least one of motion, barometric, humidity or radiation indicators.

20. The system of claim 12, wherein the semantic features include at least one of visual features of a food image and audio features of a machine in operation.