Method and apparatus for controlling vision sensor for autonomous vehicle

• US 10,657,387 B2
• Filed: 03/29/2017
• Issued: 05/19/2020
• Est. Priority Date: 09/01/2016
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Ji, Dae Hyun, Oh, Dokwan, Lee, Dongwook, Lee, Jaewoo, Jang, Cheolhun
• Primary Examiner(s): Hance, Robert J
• Application Number:
  US15/472,517
• Publication Number:
  US 20180060675A1
• Time in Patent Office: 1,147 Days
• CPC Class Codes: G01C 21/26 specially adapted for navigation in a road network
• CPC Class Codes: G01S 19/42 Determining position
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G06V 10/141 Control of illumination
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: H04N 23/71 Circuitry for evaluating the brightness variation
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/75 by influencing optical camera components

Abstract

A method and apparatus for controlling a vision sensor are provided. The apparatus and corresponding method are configured to predict an expected point, on a traveling path of a host vehicle, at which an illumination variation greater than or equal to a threshold is expected to occur, and determine whether the host vehicle is located within a threshold distance. The apparatus and corresponding method are also configured to control a vision sensor in the host vehicle based on the expected illumination variation in response to the host vehicle being located within the threshold distance.

Claims

1. A method of controlling a vision sensor of a host vehicle, the method comprising:

capturing an image of a traveling path of the host vehicle while the host vehicle travels on the traveling path using the vision sensor;

predicting based on analyzing shapes of objects within the image an expected point, on the traveling path of the host vehicle, at which an illumination variation of an external environment on the traveling path of the host vehicle greater than or equal to a threshold is expected to occur;

determining that the host vehicle is located within a threshold distance from the expected point; and

controlling the vision sensor in the host vehicle to adjust an image capture parameter of the vision sensor based on the expected illumination variation in response to determining that the host vehicle is located within the threshold distance from the expected point,

wherein the determining that the host vehicle is located within the threshold distance from the expected point comprises;

identifying an object associated with the expected point among the objects within the image;

obtaining global positioning system (GPS) coordinates of the object using GPS coordinates of a current location of the host vehicle and map data of the traveling path;

determining a distance between the GPS coordinates of the object and the GPS coordinates of the current location of the host vehicle; and

determining that the host vehicle is located within the threshold distance from the expected point based on the distance between the GPS coordinates of the object and the GPS coordinates of the current location of the host vehicle being less than the threshold distance.

2. The method of claim 1, wherein the predicting of the expected point comprises predicting the expected point based on any one or any combination of any two or more of the image captured by the vision sensor, the map data of the traveling path, global positioning system (GPS) coordinates of the current location of the host vehicle during traveling, and a traveling speed of the host vehicle.

3. The method of claim 1, wherein the predicting of the expected point comprises predicting the expected point based on a similarity between the image captured by the vision sensor and a pre-trained image representing the expected point.

4. The method of claim 1, wherein the predicting of the expected point comprises:

recognizing either one or both of an entry point and an exit point of a tunnel on the traveling path; and

predicting either one or both of the entry point and the exit point as the expected point.

5. The method of claim 4, wherein the recognizing of either one or both of the entry point and the exit point comprises either one or both of:

recognizing the entry point based on any one or any combination of any two or more of the map data of the traveling path and the GPS coordinates of the current location of the host vehicle during traveling; and

recognizing the exit point based on any one or any combination of any two or more of the map data and a traveling speed of the host vehicle.

6. The method of claim 5, wherein the recognizing of the entry point comprises either one or both of:

recognizing the entry point based on whether the image comprises a predetermined shape of an entrance of the tunnel; and

recognizing the entry point based on the map data and the GPS coordinates.

7. The method of claim 5, wherein the recognizing of the exit point comprises either one or both of:

recognizing the exit point based on whether the image comprises a predetermined shape of an exit of the tunnel; and

recognizing the exit point based on map data matching based on the entry point and the traveling speed.

8. The method of claim 1, wherein the determining of whether the host vehicle is located within the threshold distance comprises:

calculating a depth value of the expected point based on an image captured by the vision sensor; and

determining, based on the depth value, whether the host vehicle is located within the threshold distance.

9. The method of claim 1, wherein the determining that the host vehicle is located within the threshold distance from the expected point based on the distance between the GPS coordinates of the object and the GPS coordinates of the current location of the host vehicle being less than the threshold distance comprises determining that the host vehicle is located within the threshold distance from the expected point based on the distance between the GPS coordinates of the object and the GPS coordinates of the current location of the host vehicle being less than the threshold distance and based on a traveling speed of the host vehicle.

10. The method of claim 1, wherein the controlling of the vision sensor comprises any one or any combination of any two or more of:

controlling a sensing parameter of the vision sensor;

processing an image captured by the vision sensor; and

determining whether to use a filter for the vision sensor.

11. The method of claim 10, wherein the object is a tunnel,

wherein the expected point is an entry point of the tunnel or an exit point of the tunnel, and

wherein the controlling the vision sensor to adjust the image capture parameter comprises:

increasing, in stages, a value of a photosensitivity of the vision sensor as the distance between the GPS coordinates of the entry point of the tunnel and the GPS coordinates of the current location of the host vehicle decreases; and

reducing, in stages, the value of the photosensitivity of the vision sensor as the distance between the GPS coordinates of the exit point of the tunnel and the GPS coordinates of the current location of the host vehicle decreases.

12. The method of claim 1, wherein the controlling comprises performing high dynamic range imaging (HDRI) of the image through tone mapping.

13. The method of claim 1, wherein the object is a tunnel,

wherein the expected point comprises an entry point of the tunnel or an exit point of the tunnel, and

wherein the controlling the vision sensor to adjust the image capture parameter comprises:

detaching a filter from the vision sensor in response to determining that the host vehicle is located within the threshold distance from the entry point of the tunnel based on the distance between the GPS coordinates of the entry point of the tunnel and the GPS coordinates of the current location of the host vehicle being less than the threshold distance; and

attaching the filter to the vision sensor in response to determining that the host vehicle is located within the threshold distance from the exit point of the tunnel based on the distance between the GPS coordinates of the exit point of the tunnel and the GPS coordinates of the current location of the host vehicle being less than the threshold distance.

14. The method of claim 1, wherein the predicting of the expected point comprises:

determining whether the host vehicle is exposed to a backlight environment; and

determining a location of the host vehicle as the expected point based on a result of the determining.

15. The method of claim 14, wherein the determining of whether the host vehicle is exposed to the backlight environment comprises either one or both of:

determining that the host vehicle is exposed to the backlight environment in response to a light intensity measured by an illumination sensor being greater than or equal to a threshold of the backlight environment; and

determining that the host vehicle is exposed to the backlight environment in response to an object during daylight being recognized from an image captured by the vision sensor.

16. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1.

17. An apparatus for controlling a vision sensor of a host vehicle, the apparatus comprising:

a processor configured to;

capture an image of a traveling path of the host vehicle while the host vehicle travels on the traveling path using the vision sensor of the host vehicle;

predict based on analyzing shapes of objects within the image an expected point, on the traveling path of the host vehicle, at which an illumination variation of an external environment on the traveling path of the host vehicle greater than or equal to a threshold is expected to occur,

determine that the host vehicle is located within a threshold distance from the expected point, and

control the vision sensor in the host vehicle to adjust an image capture parameter of the vision sensor based on the expected illumination variation in response determining that to the host vehicle is located within the threshold distance from the expected point,

wherein

the processor determining that the host vehicle is located within the threshold distance from the expected point is further configured to;

identify an object associated with the expected point among the objects within the image;

obtain global positioning system (GPS) coordinates of the object using GPS coordinates of a current location of the host vehicle and map data of the traveling path;

determine a distance between the GPS coordinates of the object and the GPS coordinates of the current location of the host vehicle; and

determine that the host vehicle is located within the threshold distance from the expected point based on the distance between the GPS coordinates of the object and the GPS coordinates of the current location of the host vehicle being less than the threshold distance.

18. The apparatus of claim 17, further comprising:

a memory configured to store the map data of the traveling path;

a global positioning system (GPS) sensor configured to measure GPS coordinates of the current location of the host vehicle during traveling; and

a speed sensor configured to sense a traveling speed of the host vehicle,

wherein the processor is further configured to predict the expected point based on any one or any combination of any two or more of the map data, the GPS coordinates, and the traveling speed.

19. The apparatus of claim 17, wherein the processor is further configured to predict the expected point based on a similarity between the image captured by the vision sensor and a pre-trained image representing the expected point.

20. The apparatus of claim 17, wherein the processor is further configured to recognize either one or both of an entry point and an exit point of a tunnel on the traveling path and to predict either one or both of the entry point and the exit point as the expected point.

21. The apparatus of claim 20, wherein the processor is further configured to perform either one or both of:

recognizing the entry point based on any one or any combination of any two or more of the map data of the traveling path and GPS coordinates of the current location of the host vehicle during traveling; and

recognizing the exit point based on any one or any combination of any two or more of the map data and a traveling speed of the host vehicle.

22. The apparatus of claim 21, wherein the processor is further configured to perform either one or both of:

recognizing the entry point based on whether the image comprises a predetermined shape of an entrance of the tunnel; and

recognizing the entry point based on the map data and the GPS coordinates.

23. The apparatus of claim 21, wherein the processor is further configured to perform either one or both of:

recognizing the exit point based on whether the image comprises a predetermined shape of an exit of the tunnel; and

recognizing the exit point based on map data matching based on the entry point and the traveling speed of the host vehicle.

24. The apparatus of claim 17, wherein the processor is further configured to calculate a depth value of the expected point based on an image captured by the vision sensor, and to determine, based on the depth value, whether the host vehicle is located within the threshold distance.

25. The apparatus of claim 17, wherein the processor determining that the host vehicle is located within the threshold distance from the expected point based on the distance between the GPS coordinates of the object and the GPS coordinates of the current location of the host vehicle being less than the threshold distance is further configured to determine that the host vehicle is located within the threshold distance from the expected point based on the distance between the GPS coordinates of the object and the GPS coordinates of the current location of the host vehicle being less than the threshold distance and based on a traveling speed of the host vehicle.

26. The apparatus of claim 17, wherein the processor is further configured to perform any one or any combination of any two or more of:

controlling a sensing parameter of the vision sensor;

processing an image captured by the vision sensor; and

determining whether to use a filter for the vision sensor.

27. The apparatus of claim 17, wherein the object is a tunnel,

wherein the expected point comprises an entry point of the tunnel or an exit point of the tunnel, and

wherein the processor controlling the vision sensor in the host vehicle to adjust the image capture parameter is further configured to:

increase, in stages, a value of a photosensitivity of the vision sensor as the distance between the GPS coordinates of the entry point of the tunnel and the GPS coordinates of the current location of the host vehicle decreases; and

reduce, in stages, the value of the photosensitivity of the vision sensor of the vision sensor as the distance between the GPS coordinates of the exit point of the tunnel and the GPS coordinates of the current location of the host vehicle decreases.

28. The apparatus of claim 17, wherein the processor is further configured to perform high dynamic range imaging (HDRI) of the image through tone mapping.

29. The apparatus of claim 17, wherein the object is a tunnel,

wherein the expected point is an entry point of the tunnel or an exit point of the tunnel, and

wherein the processor controlling the vision sensor in the host vehicle to adjust the image capture parameter is further configured to:

detach a filter from the vision sensor in response to determining that the host vehicle is located within the threshold distance from the entry point of the tunnel based on the distance between the GPS coordinates of the entry point of the tunnel and the GPS coordinates of the current location of the host vehicle being less than the threshold distance; and

attach the filter to the vision sensor in response to determining that the host vehicle is located within the threshold distance from the exit point of the tunnel based on the distance between the GPS coordinates of the exit point of the tunnel and the GPS coordinates of the current location of the host vehicle being less than the threshold distance.

30. The apparatus of claim 17, wherein the processor is further configured to determine whether the host vehicle is exposed to a backlight environment, and to determine a location of the host vehicle as the expected point based on a result of the determining.

31. The apparatus of claim 30, further comprising:

an illumination sensor configured to sense a light intensity,

wherein the processor is further configured to perform either one or both of;

determining that the host vehicle is exposed to the backlight environment in response to the light intensity measured by the illumination sensor being greater than or equal to a threshold of the backlight environment; and

determining that the host vehicle is exposed to the backlight environment in response to an object during daylight being recognized from an image captured by the vision sensor.

32. A control apparatus of a host vehicle, comprising:

a vision sensor configured to capture an image of a traveling path of the host vehicle; and

a processor configured to;

predict based on analyzing shapes of objects within the image an expected point at which an illumination variation of an external environment on the traveling path of the host vehicle is greater than or equal to a threshold occurs, along the traveling path of the host vehicle,

determine that the host vehicle is located within a threshold distance from the expected point, and

control the vision sensor to adjust an image capture parameter of the vision sensor based on the illumination variation based on the processor determining that the host vehicle is located within the threshold distance from the expected point,

wherein

the processor determining that the host vehicle is located within the threshold distance from the expected point is further configured to;

identify an object associated with the expected point among the objects within the image;

obtain global positioning system (GPS) coordinates of the object using GPS coordinates of a current location of the host vehicle and map data of the traveling path;

determine the distance between the GPS coordinates of the object and the GPS coordinates of the current location of the host vehicle; and

determine that the host vehicle is located within the threshold distance from the expected point based on the distance between the GPS coordinates of the object and the GPS coordinates of the current location of the host vehicle being less than the threshold distance.

33. The control apparatus of claim 32, wherein the processor compares an image captured by the vision sensor to a pre-trained image trained on an image in a backlight state and determines that the host vehicle is exposed to a backlight environment, without a use of an illumination sensor.

34. The control apparatus of claim 33, wherein, in response to the processor determining that the host vehicle is exposed to the backlight environment, the processor applies a filter to the vision sensor or performs a backlight compensation to adjust a value of a photosensitivity of the vision sensor to adjust the image capture parameter.

35. The control apparatus of claim 32, wherein the processor adjusts a value of a photosensitivity of the vision sensor based on the distance between the GPS coordinates of the object and the GPS coordinates of the current location of the host vehicle to adjust the image capture parameter.

36. The control apparatus of claim 32, wherein the processor comprises:

a controller comprising;

a parameter adjuster configured to generate a control signal to adjust the image capture parameter of the vision sensor, and

a filter controller configured to generate a signal to control whether to attach or detach a filter for the vision sensor.

37. The control apparatus of claim 32, wherein the threshold distance is set based on weather conditions or a time of day being daytime or nighttime.