Electronic device localization based on imagery

• US 9,953,243 B2
• Filed: 04/24/2015
• Issued: 04/24/2018
• Est. Priority Date: 04/25/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Hesch, Joel, Nerurkar, Esha, Mihelich, Patrick
• Primary Examiner(s): CONNER, SEAN M
• Application Number:
  US14/695,208
• Publication Number:
  US 20150310310A1
• Time in Patent Office: 1,096 Days
• Field of Search: None
• CPC Class Codes: G01S 19/485 whereby the further system is an optical system or imaging system
• CPC Class Codes: G06F 18/213 Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
• CPC Class Codes: G06F 18/253 of extracted features
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 2201/07 Target detection

Abstract

An electronic device includes one or more imaging cameras. After a reset of the device or other specified event, the electronic device identifies an estimate of the device'"'"'s pose based on location data such as Global Positioning System (GPS) data, cellular tower triangulation data, wireless network address location data, and the like. The one or more imaging cameras may be used to capture imagery of the local environment of the electronic device, and this imagery is used to refine the estimated pose to identify a refined pose of the electronic device. The refined pose may be used to identify additional imagery information, such as environmental features, that can be used to enhance the location based functionality of the electronic device.

Claims

1. A method comprising:

in response to a reset event at an electronic device determining, at the electronic device, an estimated pose of the electronic device based on received non-visual location data relative to a geographic frame of reference;

in response to the reset event at the electronic device, setting an origin position of the electronic device relative to a free frame of reference;

refining the estimated pose of the electronic device based on visual location data using an image captured at an imaging camera of the electronic device to identify a refined pose of the electronic device relative to the geographic frame of reference,

wherein refining the estimated pose comprises;

identifying a feature of the image;

matching the feature to a first feature descriptor of a data store of feature descriptors; and

identifying the refined pose based on a pose associated with the first feature descriptor;

tracking, concurrently with refining the estimated pose, a position change of the electronic device relative to the origin position in the free frame of reference; and

translating the refined pose in the geographic frame of reference based on the position change of the electronic device in the free frame of reference.

2. The method of claim 1, wherein the non-visual location data comprises Global Positioning System data captured at the electronic device.

3. The method of claim 1, wherein the non-visual location data comprises wireless tower triangulation information for the electronic device.

4. The method of claim 1, wherein refining the estimated pose comprises:

identifying a plurality of feature descriptors of the data store of feature descriptors based on the estimated pose; and

searching the identified plurality of feature descriptors for the first feature descriptor.

5. The method of claim 1, further comprising:

identifying the first feature descriptor based on a gravity direction of the image identified at the electronic device.

6. The method of claim 1, further comprising:

translating a plurality of intermediate poses of the electronic device between an initial pose and the refined pose of the electronic device.

7. An electronic device comprising:

an imaging camera to capture first imagery;

a processor to;

in response to a reset event, determine an estimated pose of the electronic device relative to a geographic frame of reference based on received non-visual location data;

in response to the reset event, set an origin position of the electronic device relative to a free frame of reference;

refine the estimated pose of the electronic device based on;

a first image captured at an imaging camera of the electronic device to identify a refined pose of the electronic device relative to the geographic frame of reference, and

a first feature descriptor generated based on the first imagery,

track, concurrently with refining the estimated pose, a position change of the electronic device relative to the origin position in the free frame of reference; and

translate the refined pose in geographic frame of reference based on the position change of the electronic device in the free frame of reference.

8. The electronic device of claim 7, wherein the non-visual location data comprises Global Positioning System data captured at the electronic device.

9. The electronic device of claim 7, wherein the non-visual location data comprises wireless tower triangulation information for the electronic device.

10. The electronic device of claim 7, wherein the processor is to refine the estimated pose by:

matching the first feature descriptor to a second feature descriptor; and

identifying the refined pose based on a location associated with the second feature descriptor.

11. The electronic device of claim 10, wherein the processor is to refine the estimated pose by:

identifying a plurality of feature descriptors based on the estimated pose; and

searching the identified plurality of feature descriptors for the second feature descriptor.

12. The electronic device of claim 10, wherein the processor is to:

identify the second feature descriptor based on a gravity direction of the first image identified at the electronic device.