Navigation using sensor fusion

• US 9,661,466 B1
• Filed: 08/14/2015
• Issued: 05/23/2017
• Est. Priority Date: 03/14/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Andersson, Eve Astrid, Burkhardt, Casey John, Chen, Charles L., Raman, Tiruvilwamalai Venkatram, Lundblad, John Ove, Tseng, David, Zheng, Ying
• Primary Examiner(s): Badii, Behrang
• Application Number:
  US14/826,425
• Time in Patent Office: 648 Days
• CPC Class Codes: A61B 5/6804 Garments; Clothes
• CPC Class Codes: B60C 23/0408 transmitting the signals by non-mechanical means from the wheel or tyre to a vehicle body mounted receiver
• CPC Class Codes: B60R 1/005 specially adapted for viewing traffic-lights
• CPC Class Codes: B60R 11/04 Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle
• CPC Class Codes: G01B 11/04 specially adapted for measuring length or width of objects while moving
• CPC Class Codes: G01B 11/14 for measuring distance or clearance between spaced objects or spaced apertures G01B11/26 takes precedence; rangefinders G01C3/00
• CPC Class Codes: G01B 11/2755 using photoelectric detection means
• CPC Class Codes: G01C 21/1656 with passive imaging devices, e.g. cameras
• CPC Class Codes: G01C 21/206 specially adapted for indoor navigation
• CPC Class Codes: G01C 21/34 Route searching; Route guidance
• CPC Class Codes: G01C 21/3667 Display of a road map G01C21/3614 takes precedence; guidance using 3D or perspective road maps G01C21/3635
• CPC Class Codes: G01C 22/00 Measuring distance traversed on the ground by vehicles, persons, animals or other moving solid bodies, e.g. using odometers, using pedometers
• CPC Class Codes: G05D 1/00 Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots drive control systems specially adapted for autonomous road vehicles B60W60/00
• CPC Class Codes: H04N 13/20 Image signal generators
• CPC Class Codes: H04N 23/40 Circuit details for pick-up tubes
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 5/04 Synchronising for television systems using pulse code modulation H04N7/56
• CPC Class Codes: H04N 5/067 Arrangements or circuits at the transmitter end
• CPC Class Codes: H04N 5/073 for mutually locking plural sources of synchronising signals, e.g. studios or relay stations
• CPC Class Codes: H04N 5/222 Studio circuitry; Studio devices; Studio equipment cameras or camera modules comprising electronic image sensors, or control thereof H04N23/00
• CPC Class Codes: H04N 5/265 : Mixing
• CPC Class Codes: H04N 7/18 : Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources
• CPC Class Codes: H04R 5/023 : in a chair, pillow
• CPC Class Codes: H04W 4/023 : using mutual or relative location information between multiple location based services [LBS] targets or of distance thresholds
• CPC Class Codes: H04W 4/024 : Guidance services
• CPC Class Codes: H04W 4/027 : using movement velocity, acceleration information
• CPC Class Codes: H04W 4/38 : for collecting sensor information

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for resolving erroneous movement signals and for providing navigation instructions. One of the methods includes receiving data from two or more sensors, synchronizing the received data, determining a first subset of the synchronized data from the camera for a particular period of time and a second subset of the synchronized data from the other sensor for the particular period of time, comparing the first subset with the second subset to determine whether the first subset and the second subset indicate an erroneous movement signal, based on determining that the first subset and the second subset indicate an erroneous movement signal, selecting data for one of the sensors based on a sensor priority scheme for the sensors, and resolving the erroneous movement signal based on the selected data for the one of the sensors.

Claims

1. An apparatus, comprising:

two or more sensors comprising a camera and another sensor;

a data processing apparatus; and

a non-transitory computer readable storage medium in data communication with the data processing apparatus and storing instructions executable by the data processing apparatus and upon such execution cause the data processing apparatus to perform operations comprising;

receiving data from the two or more sensors;

synchronizing the received data from the two or more sensors so that the data from the camera corresponds in time to the data from the other sensor;

determining a first subset of the synchronized data from the camera for a particular period of time and a second subset of the synchronized data from the other sensor for the particular period of time, the first subset of the synchronized data and the second subset of the synchronized data each being representative of movement of the apparatus and the first subset of the synchronized data being representative of objects in a physical area surrounding a particular physical location of the apparatus;

comparing the first subset of the synchronized data with the second subset of the synchronized data to determine whether the first subset and the second subset indicate an erroneous movement signal;

based on determining that the first subset and the second subset indicate an erroneous movement signal, selecting, from the first subset and the second subset of the synchronized data, data for one of the sensors, the selection based on a sensor priority scheme for the sensors;

determining that the selected data for the one of the sensors indicated that the apparatus is moving;

determining a plurality of feature points of the physical area using at least the first subset of the synchronized data;

determining whether the plurality of feature points indicate that the particular physical location of the apparatus corresponds with a particular location on a map;

detecting, using the feature points, an object not represented in the map; and

providing navigation instructions to a user using the plurality of feature points based on the particular location on the map and on a physical location of the object.

2. The apparatus of claim 1, the operations comprising:

determining, using at least the first subset of the synchronized data, that at least one of the feature points represents a moving object; and

providing navigation instructions to the user to prevent the user from running into the moving object.

3. The apparatus of claim 2, the operations comprising:

creating a map using the feature points and not using the at least one of the feature points that represents the moving object.

4. The apparatus of claim 1, wherein selecting the data for one of the sensors comprises:

based on determining that the first subset and the second subset indicate an erroneous movement signal, determining, using the first subset of the synchronized data and the second subset of the synchronized data, a sensor priority scheme for the sensors; and

selecting, from the first subset and the second subset of the synchronized data, data for one of the sensors using the sensor priority scheme for the sensors.

5. The apparatus of claim 4, wherein determining, using the first subset of the synchronized data and the second subset of the synchronized data, the sensor priority scheme for the sensors comprises:

identifying a first priority scheme to use when the synchronized data indicates a first type of data received from one of the two or more sensors and a second priority scheme to use when the synchronized data indicates a second type of data received from the one of the two or more sensors, the first priority scheme having a different priority for the one of the two or more sensors than the second priority scheme;

determining a particular type of data from the one of the two or more sensors included in the synchronized data; and

selecting the first priority scheme or the second priority scheme by comparing the particular type of data with the first type of data or the second type of data.

6. The apparatus of claim 5, comprising:

an accelerometer included in the two or more sensors, wherein the first type of data comprises a first speed of the apparatus determined by the accelerometer and the second type of data comprises a second speed of the apparatus determined by the accelerometer.

7. The apparatus of claim 1, wherein the two or more sensors comprise at least one sensor selected from the group consisting of an accelerometer, a gyroscope, a global positioning system receiver, and a wireless receiver.

8. A computer-implemented method comprising:

receiving, at a user device, data from two or more sensors included in the user device, the two or more sensors comprising a camera and another sensor;

synchronizing the received data from the two or more sensors so that the data from the camera corresponds in time to the data from the other sensor;

determining a first subset of the synchronized data from the camera for a particular period of time and a second subset of the synchronized data from the other sensor for the particular period of time, the first subset of the synchronized data and the second subset of the synchronized data each being representative of movement of the user device and the first subset of the synchronized data being representative of objects in a physical area surrounding a particular physical location of the user device;

comparing the first subset of the synchronized data with the second subset of the synchronized data to determine whether the first subset and the second subset indicate an erroneous movement signal;

based on determining that the first subset and the second subset indicate an erroneous movement signal, selecting, from the first subset and the second subset of the synchronized data, data for one of the sensors, the selection based on a sensor priority scheme for the sensors;

determining that the selected data for the one of the sensors indicated that the user device is moving;

determining a plurality of feature points of the physical area using at least the first subset of the synchronized data;

determining whether the plurality of feature points indicate that the particular physical location of the user device corresponds with a particular location on a map;

detecting, using the feature points, an object not represented in the map; and

providing navigation instructions to a user using the plurality of feature points based on the particular location on the map and on a physical location of the object.

9. The method of claim 8, comprising:

determining, using at least the first subset of the synchronized data, that at least one of the feature points represents a moving object; and

providing navigation instructions to the user to prevent the user from running into the moving object.

10. The method of claim 9, comprising:

creating a map using the feature points and not using the at least one of the feature points that represents the moving object.

11. The method of claim 8, wherein selecting the data for one of the sensors comprises:

based on determining that the first subset and the second subset indicate an erroneous movement signal, determining, using the first subset of the synchronized data and the second subset of the synchronized data, a sensor priority scheme for the sensors; and

selecting, from the first subset and the second subset of the synchronized data, data for one of the sensors using the sensor priority scheme for the sensors.

12. The method of claim 11, wherein determining, using the first subset of the synchronized data and the second subset of the synchronized data, the sensor priority scheme for the sensors comprises:

identifying a first priority scheme to use when the synchronized data indicates a first type of data received from one of the two or more sensors and a second priority scheme to use when the synchronized data indicates a second type of data received from the one of the two or more sensors, the first priority scheme having a different priority for the one of the two or more sensors than the second priority scheme;

determining a particular type of data from the one of the two or more sensors included in the synchronized data; and

selecting the first priority scheme or the second priority scheme by comparing the particular type of data with the first type of data or the second type of data.

13. The method of claim 12, wherein:

receiving, at the user device, data from the two or more sensors included in the user device comprises receiving, at the user device, data from an accelerometer included in the two or more sensors; and

identifying the first priority scheme to use when the synchronized data indicates the first type of data received from one of the two or more sensors and the second priority scheme to use when the synchronized data indicates the second type of data received from the one of the two or more sensors comprises identifying the first priority scheme to use when the synchronized data received from the accelerometer indicates a first speed of the user device and the second priority scheme to use when the synchronized data received from the accelerometer indicates a second speed of the user device.

14. The method of claim 8, wherein receiving data from two or more sensors included in the user device comprises receiving data from two or more sensors included in the user device, the two or more sensors comprising at least one sensor selected from the group consisting of an accelerometer, a gyroscope, a global positioning system receiver, and a wireless receiver.

15. A non-transitory computer readable storage medium storing instructions executable by a data processing apparatus and upon such execution cause the data processing apparatus to perform operations comprising:

receiving, at the data processing apparatus, data from two or more sensors included in a user device, the two or more sensors comprising a camera and another sensor;

synchronizing the received data from the two or more sensors so that the data from the camera corresponds in time to the data from the other sensor;

determining a first subset of the synchronized data from the camera for a particular period of time and a second subset of the synchronized data from the other sensor for the particular period of time, the first subset of the synchronized data and the second subset of the synchronized data each being representative of movement of the user device and the first subset of the synchronized data being representative of objects in a physical area surrounding a particular physical location of the user device;

comparing the first subset of the synchronized data with the second subset of the synchronized data to determine whether the first subset and the second subset indicate an erroneous movement signal;

based on determining that the first subset and the second subset indicate an erroneous movement signal, selecting, from the first subset and the second subset of the synchronized data, data for one of the sensors, the selection based on a sensor priority scheme for the sensors;

determining that the selected data for the one of the sensors indicated that the user device is moving;

determining a plurality of feature points of the physical area using at least the first subset of the synchronized data;

determining whether the plurality of feature points indicate that the particular physical location of the user device corresponds with a particular location on a map;

detecting, using the feature points, an object not represented in the map; and

providing navigation instructions to a user using the plurality of feature points based on the particular location on the map and on a physical location of the object.

16. The computer readable storage medium of claim 15, comprising:

determining, using at least the first subset of the synchronized data, that at least one of the feature points represents a moving object; and

providing navigation instructions to the user to prevent the user from running into the moving object.

17. The computer readable storage medium of claim 16, comprising:

creating a map using the feature points and not using the at least one of the feature points that represents the moving object.

18. The computer readable storage medium of claim 15, wherein selecting the data for one of the sensors comprises:

based on determining that the first subset and the second subset indicate an erroneous movement signal, determining, using the first subset of the synchronized data and the second subset of the synchronized data, a sensor priority scheme for the sensors; and

selecting, from the first subset and the second subset of the synchronized data, data for one of the sensors using the sensor priority scheme for the sensors.

19. The computer readable storage medium of claim 18, wherein determining, using the first subset of the synchronized data and the second subset of the synchronized data, the sensor priority scheme for the sensors comprises:

identifying a first priority scheme to use when the synchronized data indicates a first type of data received from one of the two or more sensors and a second priority scheme to use when the synchronized data indicates a second type of data received from the one of the two or more sensors, the first priority scheme having a different priority for the one of the two or more sensors than the second priority scheme;

determining a particular type of data from the one of the two or more sensors included in the synchronized data; and

selecting the first priority scheme or the second priority scheme by comparing the particular type of data with the first type of data or the second type of data.

20. The computer readable storage medium of claim 19, wherein:

receiving data from the two or more sensors included in the user device comprises receiving data from an accelerometer included in the two or more sensors; and

identifying the first priority scheme to use when the synchronized data indicates the first type of data received from one of the two or more sensors and the second priority scheme to use when the synchronized data indicates the second type of data received from the one of the two or more sensors comprises identifying the first priority scheme to use when the synchronized data received from the accelerometer indicates a first speed of the user device and the second priority scheme to use when the synchronized data received from the accelerometer indicates a second speed of the user device.

21. The computer readable storage medium of claim 15, wherein receiving data from two or more sensors included in the user device comprises receiving data from two or more sensors included in the user device, the two or more sensors comprising at least one sensor selected from the group consisting of an accelerometer, a gyroscope, a global positioning system receiver, and a wireless receiver.