System and method to increase effective dynamic range of image sensors

• US 7,133,069 B2
• Filed: 03/15/2002
• Issued: 11/07/2006
• Est. Priority Date: 03/16/2001
• Status: Expired due to Fees

• Current Assignee: Vision Robotics Corporation
• Original Assignee: Vision Robotics Corporation
• Inventors: Wallach, Bret A., Koselka, Harvey A.
• Primary Examiner(s): Ometz, David
• Assistant Examiner(s): Hannett, James M.
• Application Number:
  US10/100,221
• Publication Number:
  US 20020176010A1
• Time in Patent Office: 1,698 Days
• Field of Search: 348/229.1, 348/362, 348/221.1, 348/218.1, 382/284
• US Class Current: 348/218.1
• CPC Class Codes: H04N 23/70 Circuitry for compensating brightness variation in the scene
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors

Abstract

A system and method of increasing the dynamic range of image sensors by controlling the exposure (either electronically or mechanically) of multiple images and optionally forming a composite image from the properly exposed areas of the multiple images of varying exposure settings. Certain embodiments of the system and method utilize image processing techniques to determine the various exposure settings and to form the optional composite image. The properly exposed areas can be used directly by machine vision applications.

Claims

1. A method of effectively increasing the dynamic range of a machine vision application, the method comprising:

capturing multiple images of a scene taken at different exposures and with multiple image sensors;

searching a selected image of the multiple images for overexposed and underexposed portions based on a pixel value indicative of luminance;

analyzing portions in other images of the multiple images taken at lower exposures corresponding spatially to the overexposed portions in the selected image;

analyzing portions in other images of the multiple images taken at higher exposures corresponding spatially to the underexposed portions in the selected image; and

applying feature extraction to at least one of the analyzed portions for distance determination based on relative locations of the features within images associated with the multiple image sensors.

2. The method of claim 1, wherein an image sensor is used to capture the multiple images.

3. The method of claim 2, wherein the image sensor is a digital camera.

4. The method of claim 1, wherein the portions can be overlapping, noncontiguous, or a combination of overlapping and noncontiguous.

5. The method of claim 1, wherein the capturing of multiple images occurs substantially simultaneously.

6. The method of claim 1, additionally comprising generating a composite image for feature extraction based on the analyzed portions that are determined to be properly exposed.

7. The method of claim 1, additionally comprising applying feature extraction to at least a portion of the selected image for distance determination based on relative locations of the features within images associated with the multiple image sensors.

8. A method of effectively increasing the dynamic range of a machine vision application, the method comprising:

obtaining an image of a scene with multiple image sensors;

analyzing the image to identify one or more regions that are either too bright or too dark, wherein any one of the regions includes a plurality of pixels;

for any identified region that is identified as being too bright, obtaining a reduced exposure image;

for any identified region that is identified as being too dark, obtaining an increased exposure image; and

analyzing one or more of the properly exposed regions of the image or one or more properly exposed regions in the reduced and/or increased exposure images, wherein analyzing comprises feature extraction for distance determination based on relative locations of the features within images associated with the multiple image sensors.

9. The method defined in claim 8, wherein an image sensor is used to obtain the images.

10. The method defined in claim 9, wherein the camera includes a CCD or CMOS sensor.

11. The method of claim 9, wherein the image sensor is a digital camera.

12. The method defined in claim 8, wherein the regions can be overlapping, noncontiguous, or a combination of overlapping and noncontiguous.

13. The method defined in claim 8, additionally comprising obtaining another image at a reduced exposure if the image sensor has a lower exposure setting available.

14. The method defined in claim 8, additionally comprising obtaining another image at an increased exposure if the image sensor has a higher exposure setting available.

15. The method defined in claim 8, wherein the obtaining a reduced exposure image and/or obtaining an increased exposure image are repeated as long as an average pixel value of any region is not within a selected limit.

16. The method defined in claim 8, additionally comprising:

analyzing the properly exposed regions in the reduced and/or increased exposure images to identify one or more subregions that are either too bright or too dark;

for any identified subregion that is too bright, obtaining another reduced exposure image;

for any identified subregion that is too dark, obtaining another increased exposure image; and

processing the properly exposed region and one or more properly exposed subregions in the reduced and/or increased exposure images.

17. The method defined in claim 8, wherein the image sensor is mounted on a mobile robot and wherein the method is repeated for a plurality of scenes looking for objects, including both landmarks and obstacles, to plan the movement of the mobile robot.

18. The method of claim 8, wherein the obtaining of the image, the obtaining of the reduced exposure image, and the obtaining of the increased exposure image occurs substantially simultaneously.

19. The method defined in claim 8, additionally comprising generating a composite image for feature extraction from the one or more properly exposed regions of the image and the one or more properly exposed regions in the reduced and/or increased exposure images.

40. The method of claim 15, additionally comprising generating a composite image for feature extraction based on the image regions being within a selected dynamic range.

20. A computer program storage medium storing instructions that when executed by a processor perform a method of increasing the dynamic range of a machine vision application, the method comprising:

obtaining a base image of a scene with multiple image sensors;

analyzing the base image to identify one or more regions that are either too bright or too dark, wherein any one of the regions includes a plurality of pixels;

for any identified region that is too bright, obtaining a reduced exposure image;

for any identified region that is too dark, obtaining an increased exposure image; and

analyzing the base image and one or more properly exposed regions in the reduced and/or increased exposure images, wherein analyzing comprises feature extraction for distance determination based on relative locations of features within images associated with the multiple image sensors.

21. The storage medium of claim 20, wherein an image sensor is used to obtain the base image, reduced exposure image, or increased exposure image.

22. The storage medium of claim 21, wherein the image sensor is a digital camera.

23. The storage medium of claim 20, wherein the regions can be overlapping, noncontiguous, or a combination of overlapping and noncontiguous.

24. The storage medium of claim 20, wherein the obtaining of the base image, the obtaining of the reduced exposure image, and the obtaining of the increased exposure image occurs substantially simultaneously.

25. The storage medium of claim 20, additionally comprising generating a composite image for feature extraction from the base image and the one or more properly exposed regions in the reduced and/or increased exposure images.

26. The method of claim 20, wherein the analyzing comprises feature extraction for object identification.

27. An image sensing system for effectively increasing the dynamic range of a machine vision application, the system comprising:

means for acquiring an image of a scene a plurality of times with different exposures using multiple image sensors to obtain a plurality of images;

means for segmenting the images into a plurality of image regions, wherein each region is indicative of a selected dynamic range; and

means for analyzing one or more of the plurality of image regions for feature extraction including distance determination based on relative locations of the features within images associated with the multiple image sensors.

28. The system of claim 27, wherein an image sensor is used to acquire the image.

29. The system of claim 28, wherein the image sensor is a digital camera.

30. The system of claim 27, wherein the image sensing system includes at least two image sensors offset from each other.

31. The system of claim 30, wherein the machine vision application utilizes corresponding image segments from the at least two image sensors to determine distance and/or orientation from the camera system to the object or feature being imaged.

32. The system of claim 31, wherein the imaging sensing system generates a composite image for feature extraction including object identification and localization, edge detection and pattern recognition.

33. The system of claim 27, wherein the imaging sensing system generates a composite image for feature extraction including object identification and localization, edge detection and pattern recognition.

34. The system of claim 27, wherein the regions can be overlapping, noncontiguous, or a combination of overlapping and noncontiguous.

35. The system of claim 27, wherein the acquiring of the image of the scene a plurality of times occurs substantially simultaneously.

36. A method of effectively increasing the dynamic range of a machine vision application, the method comprising:

acquiring an image of a scene a plurality of times with different exposures using multiple image sensors to obtain a plurality of images;

segmenting the images into a plurality of image regions, wherein each region is indicative of a selected dynamic range; and

analyzing any one or more of the plurality of image regions, wherein analyzing comprises feature extraction for distance determination based on relative locations of features within images associated with the multiple image sensors.

37. The method of claim 36, wherein an image sensor is used to acquire the image.

38. The method of claim 37, wherein the image sensor is a digital camera.

39. The method of claim 36, wherein the acquiring of the image of the scene a plurality of times occurs substantially simultaneously.

41. The method of claim 36, wherein the analyzing comprises feature extraction for object identification.

42. An image acquisition system for effectively increasing the dynamic range of a machine vision application, the system comprising:

means for acquiring a plurality of images of a scene with multiple image sensors, each image having a different exposure;

means for obtaining a plurality of image regions, wherein each region contains portions of a particular image with good dynamic range from the plurality of images; and

means for analyzing one or more of the plurality of image regions for feature extraction including distance determination based on relative locations of the features within images associated with the multiple image sensors.

43. The system of claim 42, further comprising means for moving the system from one location to another location so as to acquire images of different scenes.

44. The system of claim 42, wherein an image sensor is used to acquire the plurality of images.

45. The system of claim 44, wherein the image sensor is a digital camera.

46. The system of claim 42, wherein the image sensing system contains at least two image sensors offset from each other.

47. The system of claim 46, wherein the machine vision application utilizes corresponding image segments from the at least two image sensors to determine distance and/or orientation from the camera system to the object or feature being imaged.

48. The system of claim 42, wherein the imaging sensing system generates a composite image for feature extraction including object identification and localization, edge detection and pattern recognition.

49. The system of claim 47, wherein the imaging sensing system generates a composite image for feature extraction including object identification and localization, edge detection and pattern recognition.

50. The system of claim 42, wherein the means for acquiring the plurality of images of the scene is a means for acquiring the plurality of images of the scene substantially simultaneously.

51. A method of effectively increasing the dynamic range of a machine vision application, the method comprising:

capturing multiple images of a scene taken in different lighting conditions and with multiple image sensors;

searching an image of the multiple images for overexposed and underexposed portions based on a pixel value indicative of luminance;

reanalyzing the overexposed portions and the underexposed portions in the selected image corresponding to a spatially related portion in other images of the multiple images taken in different lighting conditions; and

analyzing at least one of the reanalyzed portions, wherein analyzing comprises feature extraction for distance determination based on relative locations of the features within images associated with the multiple image sensors.

52. The method of claim 51, wherein an image sensor is used to capture the multiple images.

53. The method of claim 52, wherein the image sensor is a digital camera.

54. The method of claim 51, wherein the portions can be overlapping, noncontiguous, or a combination of overlapping and noncontiguous.

55. The method of claim 51, wherein the different light conditions include one or more flashes of light.

56. The method of claim 51, wherein the capturing multiple images occurs substantially simultaneously.

57. The method of claim 51, additionally comprising generating a composite image for feature extraction based on the reanalyzed portions that are determined to be properly exposed.

58. The method of claim 51, additionally comprising analyzing at least a portion of the image of the multiple images, wherein analyzing comprises feature extraction for distance determination based on relative locations of the features within images associated with the multiple image sensors.

59. An image sensing system for effectively increasing the dynamic range of a machine vision application, the system comprising:

means for acquiring an image of a scene a plurality of times under different lighting conditions and with multiple image sensors so as to obtain a plurality of images;

means for segmenting the images into a plurality of image regions, wherein each region is indicative of a selected dynamic range; and

means for analyzing one or more of the plurality of image regions for feature extraction including distance determination based on relative locations of the features within images associated with the multiple image sensors.

60. The system of claim 59, wherein one or more image sensor is used to acquire the image.

61. The system of claim 60, wherein the one or more image sensor is one or more digital camera.

62. The system of claim 59, wherein the different lighting conditions include the use of one or more flashes of light.

63. The system of claim 59, wherein the image sensing system contains at least two image sensors offset from each other.

64. The system of claim 63, wherein the machine vision application utilizes corresponding image segments from the at least two image sensors to determine distance and/or orientation from the camera system to the object or feature being imaged.

65. The system of claim 64, wherein the imaging sensing system generates a composite image for feature extraction including object identification and localization, edge detection and pattern recognition.

66. The system of claim 59, wherein the imaging sensing system generates a composite image for feature extraction including object identification and localization, edge detection and pattern recognition.

67. The system of claim 59, wherein the means for acquiring the image of the scene a plurality of times is a means for acquiring the image of the scene a plurality of times substantially simultaneously.

68. A method to improve the performance of a machine vision system by increasing the amount of data captured within images of a scene, the method comprising:

obtaining multiple images of a scene taken at different exposures and with multiple image sensors, wherein one of the multiple images is designated as a base image;

analyzing properly exposed portions of the base image;

analyzing regions in higher exposure images that correspond to pixels in the base image that were underexposed;

analyzing regions in the lower exposure images that correspond to pixels in the base image that were overexposed; and

applying feature extraction to at least one of the analyzed portions or regions for distance determination based on relative locations of the features within images associated with the multiple image sensors.

69. The method of claim 68, wherein the features include landmarks for use with a vision based robot.

70. The method of claim 68, wherein subregions of the higher exposure images remain overexposed and subregions of the lower exposure images remain underexposed, and wherein the method further comprises analyzing corresponding subregions of images with still higher and lower exposures.

71. The method of claim 68, wherein the obtaining multiple images occurs substantially simultaneously.

72. The method of claim 68, additionally comprising generating a composite image for feature extraction based on the regions and portions that are determined to be properly exposed.

73. A method of effectively increasing the dynamic range of an image sensor, the method comprising:

obtaining an image of a scene;

analyzing the image to identify one or more regions that are either too bright or too dark, wherein any one of the regions includes a plurality of pixels;

for any identified region that is identified as being too bright, obtaining a reduced exposure image;

for any identified region that is identified as being too dark, obtaining an increased exposure image;

wherein the obtaining a reduced exposure image and/or obtaining an increased exposure image are repeated as long as an average pixel value of any region is not within a selected limit; and

processing one or more of the properly exposed regions of the image and one or more properly exposed regions in the reduced and/or increased exposure images.

74. A method of effectively increasing the dynamic range of an image sensor, the method comprising:

obtaining an image of a scene;

analyzing the image to identify one or more regions that are either too bright or too dark, wherein any one of the regions includes a plurality of pixels;

for any identified region that is identified as being too bright, obtaining a reduced exposure image;

for any identified region that is identified as being too dark, obtaining an increased exposure image;

processing one or more of the properly exposed regions of the image and one or more properly exposed regions in the reduced and/or increased exposure images;

analyzing the properly exposed regions in the reduced and/or increased exposure images to identify one or more subregions that are either too bright or too dark;

for any identified subregion that is too bright, obtaining another reduced exposure image;

for any identified subregion that is too dark, obtaining another increased exposure image; and

processing the properly exposed region and, one or more properly exposed subregions in the reduced and/or increased exposure images.

75. A method to improve the performance of a machine vision system by increasing the amount of data captured within images of a scene, the method comprising:

obtaining multiple images of a scene taken at different exposures, wherein one of the multiple images is designated as a base image;

analyzing features in properly exposed portions of the base image;

analyzing features in regions in higher exposure images that correspond to pixels in the base image that were underexposed;

analyzing features in regions in the lower exposure images that correspond to pixels in the base image that were overexposed, wherein subregions of the higher exposure images remain overexposed and subregions of the lower exposure images remain underexposed;

analyzing features in corresponding subregions of images with still higher and lower exposures; and

processing at least one of the analyzed features with a machine vision application associated with mapping, navigation, or localization.

76. A method to effectively improve the performance of a machine vision system, the method comprising:

capturing multiple images of a scene taken at different exposures using a plurality of image sensors;

analyzing at least a selected one of the images by applying feature extraction;

if regions of the at least selected one analyzed image are too dark or too light,

analyzing portions in other images of the multiple images taken at higher exposures corresponding spatially to the regions in the selected analyzed image that are too underexposed, and

analyzing portions in other images of the multiple images taken at lower exposures corresponding spatially to the regions in the selected analyzed image that are too overexposed; and

determining a distance from the image sensors to at least one extracted feature based on relative locations of the features within images associated with the plurality of image sensors.

77. The method of claim 76, wherein applying feature extraction includes matching at least one feature in images from one of the image sensors to a corresponding feature in images from another one of the image sensors.

78. The method of claim 76, wherein a feature is recognized when there is sufficient contrast to match the feature between images of the image sensors.

79. The method of claim 76, wherein the image sensors are offset from each other.

80. The method of claim 76, wherein capturing multiple images of a scene taken at different exposures comprises capturing the multiple images at the same time at each of the image sensors.

81. The method of claim 76, wherein capturing multiple images of a scene taken at different exposures comprises capturing the multiple images at different times at each of the image sensors.

82. The method of claim 76, wherein capturing multiple images of a scene taken at different exposures comprises capturing the multiple images for at least one exposure at one image sensor that is different from at least another exposure at another image sensor.

83. The method of claim 76, wherein the image sensors are configured to capture multiple images of a scene with the same or different exposure settings.

84. The method of claim 76, wherein the analysis of the selected image or portions in other images is performed independently for the images captured by each of the image sensors.

85. The method of claim 76, wherein the analysis of the selected image or portions in other images comprises using the same exposures for corresponding portions in the images captured from the image sensors.

86. A method of effectively increasing the dynamic range of a machine vision application, the method comprising:

capturing multiple images of a scene taken at different exposures using a plurality of image sensors;

analyzing a selected image of the multiple images for overexposed and underexposed portions based on a pixel value indicative of luminance;

analyzing portions in other images of the multiple images taken at lower exposures corresponding spatially to the overexposed portions in the selected image;

analyzing portions in other images of the multiple images taken at higher exposures corresponding spatially to the underexposed portions in the selected image; and

applying feature extraction to at least one of the analyzed portions for distance determination based on relative locations of the features within images associated with the plurality of image sensors.

87. The method of claim 86, wherein the image sensors are offset from each other.

88. The method of claim 86, wherein capturing multiple images of a scene taken at different exposures comprises capturing the multiple images at the same time at each of the image sensors.

89. The method of claim 86, wherein capturing multiple images of a scene taken at different exposures comprises capturing the multiple images at different times at each of the image sensors.

90. The method of claim 86, wherein capturing multiple images of a scene taken at different exposures comprises capturing the multiple images for at least one exposure at one image sensor that is different from at least another exposure at another image sensor.

91. The method of claim 86, wherein the image sensors are configured to capture multiple images of a scene with the same or different exposure settings.

92. The method of claim 86, wherein the analysis of the selected image or portions in other images is performed independently for the images captured by each of the image sensors.

93. The method of claim 86, wherein the analysis of the selected image or portions in other images comprises using the same exposures for corresponding portions in the images captured from the image sensors.

94. The method of claim 86, wherein applying feature extraction includes matching at least one feature in images from one of the image sensors to a corresponding feature in images from another one of the image sensors.

95. The method of claim 86, wherein the determination of distance is from the image sensors to an extracted feature.

96. The method of claim 95, wherein the extracted feature is recognized when there is sufficient contrast to match the feature between images of the image sensors.

97. An image sensing system for effectively increasing the dynamic range of a machine vision application, the system comprising:

means for acquiring an image of a scene a plurality of times with different exposures to obtain a plurality of images;

means for segmenting the images into a plurality of image regions, wherein each region is indicative of a selected dynamic range;

means for analyzing one or more of the plurality of image regions for feature extraction including object identification or distance determination; and

wherein the image sensing system includes at least two image sensors offset from each other, and wherein the machine vision application utilizes corresponding image segments from the at least two image sensors to determine distance and/or orientation from the camera system to the object or feature being imaged.

98. An image acquisition system for effectively increasing the dynamic range of a machine vision application, the system comprising:

means for acquiring a plurality of images of a scene, each image having a different exposure;

means for obtaining a plurality of image regions, wherein each region contains portions of a particular image with good dynamic range from the plurality of images;

means for analyzing one or more of the plurality of image regions for feature extraction including distance determination; and

wherein the image sensing system contains at least two image sensors offset from each other, and wherein the machine vision application utilizes corresponding image segments from the at least two image sensors to determine distance and/or orientation from the camera system to the object or feature being imaged.

99. An image sensing system for effectively increasing the dynamic range of a machine vision application, the system comprising:

means for acquiring an image of a scene a plurality of times under different lighting conditions so as to obtain a plurality of images;

means for segmenting the images into a plurality of image regions, wherein each region is indicative of a selected dynamic range;

means for analyzing one or more of the plurality of image regions for feature extraction including distance determination; and

wherein the image sensing system contains at least two image sensors offset from each other, and wherein the machine vision application utilizes corresponding image segments from the at least two image sensors to determine distance and/or orientation from the camera system to the object or feature being imaged.