LOCALISATION OF MOBILE DEVICE USING IMAGE AND NON-IMAGE SENSOR DATA IN SERVER PROCESSING

• US 20200132461A1
• Filed: 12/21/2017
• Published: 04/30/2020
• Est. Priority Date: 12/21/2016
• Status: Active Grant

• Current Assignee: Blue Vision Labs UK Limited (LYFT Incorporated)
• Original Assignee: Blue Vision Labs UK Limited (LYFT Incorporated)
• Inventors: Ondruska, Peter, Platinsky, Lukas
• Granted Patent: US 11,761,766 B2
• Time in Patent Office: Days
• CPC Class Codes: G01C 21/005 with correlation of navigation data from several sources, e.g. map or contour matching G01C21/30 takes precedence
• CPC Class Codes: G01C 21/3848 Data obtained from both position sensors and additional sensors
• CPC Class Codes: G01S 19/485 whereby the further system is an optical system or imaging system
• CPC Class Codes: G01S 5/0018 Transmission from mobile station to base station
• CPC Class Codes: G01S 5/16 using electromagnetic waves other than radio waves
• CPC Class Codes: G06T 7/0004 Industrial image inspection
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 23/00 Cameras or camera modules comprising electronic image sensors; Control thereof

Abstract

The present invention relates to method of localisation for devices. More particularly, it relates the use of both local and remote resources to provide substantially real time localisation at a device. According to an aspect, there is provided a method of determining a location of a device having one or more sensors comprising the steps of: sending a localisation request to a server system, the localisation request comprising at least a portion of data from the one or more sensors; receiving localisation data from the server system in response to the localisation request; and determining a location of the device from the received localisation data. Optionally, the method includes the further step of estimating a location of the device based on data from the one or more sensors and wherein the step of determining a location of the device includes determining the location of the device using the estimated location.

Claims

1. A method of determining a location of a device having one or more sensors comprising the steps of:

sending a localisation request to a server system, the localisation request comprising at least a portion of data from the one or more sensors;

receiving localisation data from the server system in response to the localisation request; and

determining a location of the device from the received localisation data.

2. The method of claim 1, further comprising the step of estimating a location of the device based on data from the one or more sensors and wherein the step of determining a location of the device includes determining the location of the device using the estimated location.

3. The method of claim 1, wherein the step of determining a location of the device further comprises using at least a portion of the data from the one or more sensors along with the received localisation data.

4. The method of claim 1, wherein the localisation request further comprises an estimated location.

5. The method of claim 1, wherein the step of determining a location of the device includes any of:

determining the location of the device at the point in time that the localisation request was sent to the server system;

or determining the current location of the device taking into account the point in time that the localisation request was sent to the server system.

6. The method of claim 1, wherein said one or more sensors comprises at least one visual data sensor.

7. The method of claim 6, wherein said at least one visual data sensor comprises any or a combination of:

an image camera;

a video camera;

a monocular camera;

a depth camera;

a stereo image camera;

a high dynamic range camera, a light detection and ranging sensor;

a radio detection and ranging sensor.

8. The method of claim 6, wherein said at least a portion of the data from the one or more sensors comprises visual data from the at least one visual data sensor.

9. The method of claim 8 wherein data from said one or more sensors is combined with data from an inertial measurement unit and/or a satellite positioning system.

10. The method of claim 1 wherein the step of estimating a location further comprises using previous localisation data.

11. The method of claim 10, wherein previous localisation data comprises any of:

one or more previous estimates of location;

one or more previous localisation requests;

one or more of previous said localisation data received from the server system in response to previous said localisation requests; and

previous determinations of location.

12. The method of claim 1 wherein the step of estimating a location is performed using any of:

a local pose;

or a local frame of reference.

13. The method of claim 1 wherein the localisation data received from the server system in response to said localisation request uses any of:

a global pose;

a global frame of reference;

or a portion of the master map.

14. The method of claim 13 wherein the step of estimating a location is performed using any of:

a local pose;

or a local frame of reference, the method further comprising the step of determining a global to local pose transform or relationship;

or a global frame of reference to local frame of reference transform or relationship.

15. The method of claim 1 wherein the device comprises any of a mobile phone, a robot, augmented reality or virtual reality headset, or navigation device, a self-driving vehicle, a drone or other autonomous vehicle.

16. The method of claim 1 wherein the step of determining the location of the device is accurate to within several centimetres, optionally to within a centimetre.

17. The method of claim 1 wherein the method is persistent.

18. The method of claim 1 wherein determining the location of the device can be performed inside buildings or in dense urban areas.

19. The method of claim 1 wherein the sensor data comprises data on environmental conditions such as weather conditions.

20. The method of claim 1 wherein the server system comprises global map data;

optionally wherein the global map data comprises any of;

one or more sets of local map data and/or one or more sets of global map data and/or one or more global maps of interconnected local maps.

21. The method of claim 1 wherein the server system can be any of:

a single server, a distributed server system, a cloud system, a physical system or a virtual system.

22. The method of claim 20 wherein the global map can be distributed between a plurality of server systems.

23. The method of claim 20 wherein the global map can be hosted in a peer-to-peer arrangement on any of:

one or more devices acting as a server system, or a mix of one or more server systems and one or more devices.

24. The method of claim 1 wherein the device and server system communicate over any of:

a mobile data network, or wireless data network.

25. The method of claim 1 wherein the device has a odometry system.

26. The method of claim 25, wherein the odometry system processes sensor data to determine relative position differences between successive sensor data.

27. The method of claim 26, wherein the position estimates have between three and six degrees of freedom.

28. The method of claim 25, wherein the odometry system performs full simultaneous location and mapping and/or loop closure and/or graph optimisation.

29. The method of claim 25 wherein the odometry system outputs the estimated position.

30. The method of any preceding claim 1 wherein the estimated position is relative to a local origin.

31. The method of claim 10, wherein the previous localisation data is stored with a timestamp and data from said one or more sensors is combined with data from an inertial measurement unit and/or a satellite positioning system.

32. The method of claim 1, wherein the location of the device is determined in a global co-ordinate frame.

33. The method of any preceding claim 1, wherein the localisation request is in a local coordinate frame.

34. The method of claim 33 wherein the location of the device is determined in a global co-ordinate frame, the method further comprising the step of determining a transform between the global co-ordinate frame and the local co-ordinate frame.

35. The method of claim 1, wherein the localisation data is in a global coordinate frame.

36. A method of determining localisation data at a server system comprising the steps of:

receiving a localisation request from one or more client systems, the localisation request comprising at least a portion of data from one or more sensors;

determining relevant localisation nodes from a global map and aggregating this data as localisation data; and

sending the localisation data to the relevant one or more client systems in response to the localisation request.

37. The method of claim 36 wherein the localisation request further comprises an estimated location of the respective one or more client systems.

38. The method of claim 36 wherein either the localisation data is operable to determine a location of the device at a point in time that the localisation request was sent to the server system;

or the localisation data is operable to determine the current location of the device taking into account the point in time that the localisation request was sent to the server system.

39. The method of claim 36 further comprising the step of providing feedback data to one or more client systems to send more data or less said portion of data.

40. The method of claim 36, further comprising the step of updating the global map with at least a portion of the said at least a portion of data from one or more sensors received from the one or more client systems.

41. The method of claim 36, wherein said one or more sensors comprises at least one visual data sensor.

42. The method of claim 41, wherein said at least one visual data sensor comprises any or a combination of:

an image camera;

a video camera;

a monocular camera;

a depth camera;

a stereo image camera;

a high dynamic range camera, a light detection and ranging sensor;

a radio detection and ranging sensor.

43. The method of claim 41, wherein said at least a portion of the data from the one or more sensors comprises visual data from the at least one visual data sensor.

44. The method of claim 43 wherein data from said one or more sensors is combined with data from an inertial measurement unit and/or a satellite positioning system.

45. The method of claim 36 wherein the step of estimating a location further comprises using previous localisation data.

46. The method of claim 45, wherein previous localisation data comprises any of:

one or more previous estimates of location;

one or more previous localisation requests;

one or more of previous said localisation data received from the server system in response to previous said localisation requests; and

previous determinations of location.

47. The method of claim 36 wherein the step of estimating a location is performed using any of:

a local pose;

or a local frame of reference.

48. The method of claim 36 wherein the localisation data received from the server system in response to said localisation request uses any of:

a global pose;

a global frame of reference;

or a portion of the master map.

49. The method of claim 48 wherein the step of estimating a location further comprises using previous localisation data, the method further comprising the step of determining a global to local pose transform or relationship;

or a global frame of reference to local frame of reference transform or relationship.

50. The method of claim 36 wherein the device comprises any of a mobile phone, a robot, augmented reality or virtual reality headset, or navigation device, a self-driving vehicle, a drone or other autonomous vehicle.

51. The method of claim 36 wherein the step of determining the location of the device is accurate to within several centimetres, optionally to within a centimetre.

52. The method of claim 36 wherein the method is persistent.

53. The method of claim 36 wherein determining the location of the device can be performed inside buildings or in dense urban areas.

54. The method of claim 36 wherein the sensor data comprises data on environmental conditions such as weather conditions.

55. The method of claim 36 wherein the server system comprises global map data;

optionally wherein the global map data comprises any of;

one or more sets of local map data and/or one or more sets of global map data and/or one or more global maps of interconnected local maps.

56. The method of claim 36 wherein the server system can be any of:

a single server, a distributed server system, a cloud system, a physical system or a virtual system.

57. The method of claim 55 wherein the global map can be distributed between a plurality of server systems.

58. The method of claim 55 wherein the global map can be hosted in a peer-to-peer arrangement on any of:

one or more devices acting as a server system, or a mix of one or more server systems and one or more devices.

59. The method of claim 36 wherein the device and server system communicate over any of:

a mobile data network, or wireless data network.

60. The method of claim 36 wherein the device has a odometry system.

61. The method of claim 60, wherein the odometry system processes sensor data to determine relative position differences between successive sensor data.

62. The method of claim 61, wherein the position estimates have between three and six degrees of freedom.

63. The method of claim 60, wherein the odometry system performs full simultaneous location and mapping and/or loop closure and/or graph optimisation.

64. The method of claim 60 wherein the odometry system outputs the estimated position.

65. The method of claim 36 wherein the estimated position is relative to a local origin.

66. The method of claim 45 wherein the previous localisation data is stored with a timestamp and the step of estimating a location further comprises using previous localisation data.

67. The method of claim 36, wherein the location of the device is determined in a global co-ordinate frame.

68. The method of claim 36, wherein the localisation request is in a local coordinate frame.

69. The method of claim 68 wherein the location of the device is determined in a global co-ordinate frame, the method further comprising the step of determining a transform between the global co-ordinate frame and the local co-ordinate frame.

70. The method of claim 36, wherein the localisation data is in a global coordinate frame.

71. A method of updating a global map at a server system comprising the steps of:

receiving sensor data from one or more client systems; and

adding the sensor data to the global map.

72. The method of claim 71, further comprising the step of receiving estimated location data from one or more client systems.

73. The method of claim 71, wherein the step of adding the sensor data to the global map comprises any of:

creating a new node in the global map;

updating or amending an existing node in the global map;

or deleting a node in the global map.

74. The method of claim 71 wherein the sensor data comprises any of:

environmental data;

or weather data;

satellite positioning data;

inertial measurement data;

image data;

or video data.

75. The method of claim 71 further comprising the step of optimising the global map.

76. The method of claim 75 wherein the step of optimising comprises performing global loop-closures to link positions of different localisation nodes.

77. The method of claim 71 further comprising the step of sending an instruction to the one or more client systems to provide more or less sensor data to the server system.

78. The method of claim 71 wherein updating the global map is performed at intervals, optionally at time intervals or regular time intervals, or optionally when a predetermined threshold of sensor data has been received.

79. An apparatus configured to provide a method of determining a location of a device having one or more sensors, the method comprising:

sending a localisation request to a server system, the localisation request comprising at least a portion of data from the one or more sensors;

receiving localisation data from the server system in response to the localisation request and determining a location of the device from the received localisation data.

80. (canceled)

81. A computer program product for providing a method of determining a location of a device having one or more sensors, the method comprising:

sending a localisation request to a server system, the localisation request comprising at least a portion of data from the one or more sensors;

receiving localisation data from the server system in response to the localisation request and determining a location of the device from the received localisation data.

82. (canceled)

83. (canceled)