METHOD AND APPARATUS FOR COMPARING RELEVANT INFORMATION BETWEEN SENSOR MEASUREMENTS

• US 20200130704A1
• Filed: 10/29/2018
• Published: 04/30/2020
• Est. Priority Date: 10/29/2018
• Status: Active Grant

• Current Assignee: HERE Global B.V. (Nokia Corporation)
• Original Assignee: HERE Global B.V. (Nokia Corporation)
• Inventors: VISWANATHAN, Anirudh
• Granted Patent: US 10,928,819 B2
• Time in Patent Office: Days
• CPC Class Codes: B60W 2050/0215 Sensor drifts or sensor failures
• CPC Class Codes: B60W 50/0205 Diagnosing or detecting failures; Failure detection models
• CPC Class Codes: B60W 50/023 Avoiding failures by using redundant parts
• CPC Class Codes: B60W 60/0015 specially adapted for safety
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G05D 1/0077 using redundant signals or controls
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0248 in combination with a laser lasers per se H01S

Abstract

A method is provided for comparing relevant information between observations. Methods may include: receiving first sensor data from a first sensor type; receiving second sensor data from a second sensor of a second sensor type, different from the first sensor type, where the first and second sensor data includes data associated with an environment of the sensors; generating a first binary bitmask of the first sensor data; generating a second binary bitmask of the second sensor data; applying a mutual information score function using the first binary bitmask and the second binary bitmask as inputs; generating a mutual information score from the mutual information score function, wherein the mutual information score represents a degree of similarity between the sensor data; and establishing a reliability of the at least one of the first sensor or the second sensor.

Claims

1. An apparatus to facilitate autonomous or semi-autonomous control of a vehicle comprising at least one processor and at least one non-transitory memory including computer program code instructions, the computer program code instructions configured to, when executed, cause the apparatus to at least:

receive first sensor data from a first sensor of a first sensor type, wherein the first sensor data comprises data associated with an environment of the first sensor;

receive second sensor data from a second sensor of a second sensor type, different from the first sensor type, wherein the second sensor data comprises data associated with an environment of the second sensor;

generate a first binary bitmask of the first sensor data;

generate a second binary bitmask of the second sensor data;

apply a mutual information score function using the first binary bitmask and the second binary bitmask as inputs;

generate a mutual information score from the mutual information score function, wherein the mutual information score represents a degree of similarity between the first sensor data and the second sensor data;

establish a reliability of at least one of the first sensor or the second sensor based on the mutual information score; and

provide for autonomous control of a vehicle using feedback from the at least one of the first sensor and the second sensor in response to the reliability of the at least one of the first sensor or the second sensor satisfying a predetermined reliability value.

2. The apparatus of claim 1, wherein the first sensor data corresponds to a first image, wherein the second sensor data corresponds to a second image, and wherein causing the apparatus to apply a mutual information score function using the first binary bitmask and the second binary bitmask as inputs comprises causing the apparatus to:

generate a joint binary mask between the first binary bitmask and the second binary bitmask by generating a mask value for each corresponding location between the first image and the second image, wherein the mask value includes the binary value for each of the first image and the second image at each individual corresponding location.

3. The apparatus of claim 1, wherein the first sensor type is an image sensor, and the second sensor type is a Light Distancing and Ranging (LiDAR) sensor.

4. The apparatus of claim 3, wherein the binary bitmask of the first sensor data is based on colors captured in the first sensor data, wherein the binary bitmask of the second sensor data is based on intensities captured in the second sensor data.

5. The apparatus of claim 1, wherein the first sensor and the second sensor are associated with a vehicle traveling along the road segment, wherein the apparatus is further caused to:

receive location information associated with the vehicle;

determine the road segment along which the vehicle is traveling; and

update map data associated with a road segment with at least one of the first sensor data or the second sensor data in response to the mutual information score satisfying a predetermined value.

6. The apparatus of claim 1, wherein the apparatus is further caused to provide updated map data associated with the environment to a map services provider in response to the reliability satisfying a predetermined value.

7. The apparatus of claim 1, wherein the apparatus is further caused to provide an indication to a user in response to the reliability failing to satisfy a predetermined value.

8. A computer program product comprising at least one non-transitory computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions comprising program code instructions to:

receive first sensor data from a first sensor of a first sensor type, wherein the first sensor data comprises data associated with an environment of the first sensor;

receive second sensor data from a second sensor of a second sensor type, different from the first sensor type, wherein the second sensor data comprises data associated with an environment of the second sensor;

generate a first binary bitmask of the first sensor data;

generate a second binary bitmask of the second sensor data;

apply a mutual information score function using the first binary bitmask and the second binary bitmask as inputs;

generate a mutual information score from the mutual information score function, wherein the mutual information score represents a degree of similarity between the first sensor data and the second sensor data;

establish a reliability of at least one of the first sensor or the second sensor based on the mutual information score; and

provide for autonomous control of a vehicle using feedback from the at least one of the first sensor or the second sensor in response to the reliability of the at least one of the first sensor and the second sensor satisfying a predetermined reliability value.

9. The computer program product of claim 8, wherein the first sensor data corresponds to a first image, wherein the second sensor data corresponds to a second image, and wherein the program code instructions to apply a mutual information score function using the first binary bitmask and the second binary bitmask as inputs comprise program code instructions to:

generate a joint binary mask between the first binary bitmask and the second binary bitmask by generating a mask value for each corresponding location between the first image and the second image, wherein the mask value includes the binary value for each of the first image and the second image at each individual corresponding location.

10. The computer program product of claim 8, wherein the first sensor type is an image sensor, and the second sensor type is a Light Distancing and Ranging (LiDAR) sensor.

11. The computer program product of claim 10, wherein the binary bitmask of the first sensor data is based on colors captured in the first sensor data, wherein the binary bitmask of the second sensor data is based on intensities captured in the second sensor data.

12. The computer program product of claim 8, wherein the first sensor and the second sensor are associated with the vehicle traveling along a road segment, further comprising program code instructions to:

receive location information associated with the vehicle;

determine the road segment along which the vehicle is traveling; and

update map data associated with a road segment with at least one of the first sensor data or the second sensor data in response to the mutual information score satisfying a predetermined value.

13. The computer program product of claim 8, further comprising program code instructions to provide updated map data associated with the environment to a map services provider in response to the reliability satisfying a predetermined value.

14. The computer program product of claim 8, further comprising program code instructions to provide an indication to a user in response to the reliability failing to satisfy a predetermined value.

15. A method comprising:

receiving first sensor data from a first sensor of a first sensor type, wherein the first sensor data comprises data associated with an environment of the first sensor;

receiving second sensor data from a second sensor of a second sensor type, different from the first sensor type, wherein the second sensor data comprises data associated with an environment of the second sensor;

generating a first binary bitmask of the first sensor data;

generating a second binary bitmask of the second sensor data;

applying a mutual information score function using the first binary bitmask and the second binary bitmask as inputs;

generating a mutual information score from the mutual information score function, wherein the mutual information score represents a degree of similarity between the first sensor data and the second sensor data;

establishing a reliability of at least one of the first sensor or the second sensor based on the mutual information score; and

providing for autonomous control of a vehicle using feedback from the at least one of the first sensor and the second sensor in response to the reliability of the at least one of the first sensor or the second sensor satisfying a predetermined reliability value.

16. The method of claim 15, wherein the first sensor data corresponds to a first image, wherein the second sensor data corresponds to a second image, and wherein applying a mutual information score function using the first binary bitmask and the second binary bitmask as inputs comprises:

generating a joint binary mask between the first binary bitmask and the second binary bitmask by generating a mask value for each corresponding location between the first image and the second image, wherein the mask value includes the binary value for each of the first image and the second image at each individual corresponding location.

17. The method of claim 15, wherein the first sensor type is an image sensor, and the second sensor type is a Light Distancing and Ranging (LiDAR) sensor.

18. The method of claim 17, wherein the binary bitmask of the first sensor data is based on colors captured in the first sensor data, and wherein the binary bitmask of the second sensor data is based on intensities captured in the second sensor data.

19. The method of claim 15, wherein the first sensor and the second sensor are associated with the vehicle traveling along a road segment, the method further comprising:

receiving location information associated with the vehicle;

determining the road segment along which the vehicle is traveling; and

updating map data associated with a road segment with at least one of the first sensor data or the second sensor data in response to the mutual information score satisfying a predetermined value.

20. The method of claim 15, further comprising providing updated map data associated with the environment to a map services provider in response to the reliability satisfying a predetermined value.