Enhanced display of environmental navigation features to vehicle operator

• US 20020130953A1
• Filed: 03/12/2002
• Published: 09/19/2002
• Est. Priority Date: 03/13/2001
• Status: Abandoned Application

• Current Assignee: David Michael Geshwind, John Riconda
• Original Assignee: David Michael Geshwind, John Riconda
• Inventors: Riconda, John, Geshwind, David Michael
• Application Number:
  US10/097,029
• Publication Number:
  US 20020130953A1
• Time in Patent Office: Days
• US Class Current: 348/115
• CPC Class Codes: G01C 21/36 Input/output arrangements for on-board computers
• CPC Class Codes: G02B 19/0009 having refractive surfaces only
• CPC Class Codes: G02B 19/0014 at least one surface having optical power
• CPC Class Codes: G02B 19/0033 characterised by the use
• CPC Class Codes: G02B 2027/011 comprising device for correcting geometrical aberrations, distortion
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/01 Head-up displays

Abstract

Imaging device is trained (e.g., panned, zoomed, focussed) on environmental navigation feature, such as street sign or house number, by operator input and computer control. Optional illumination in visible, infrared, ultraviolet, or other spectrum enhances (especially nighttime) imaging. Optional processing is applied to image to increase brightness, sharpness and/or size, and/or to counter positional or other distortion or error. Computer controlled motion tracking, affected by pattern recognition algorithms with optional artificial intelligence, and/or freeze frame function, and/or optical or digital image stabilization, are used to stabilize view from moving vehicle.

Claims

23. An improved system for providing enhanced display of environmental navigation features comprising system components installed in a motor vehicle including:

a. a camera further comprising a lens and an electronic imaging subsystem;

b. a mounting holding said camera; and

,

c. a display showing the output of said electronic imaging subsystem.

72. A method for providing enhanced display of environmental navigation features comprising the steps of:

a. capturing an image by utilizing a camera further comprising a lens and electronic imaging subsystem;

b. displaying the output of said electronic imaging subsystem as an image.

73. A method as in claim 72, further comprising the step, between a. and b., of:

a1. inputting control information to transducers effecting the configuration of said lens.

74. A method as in claim 72, further comprising the step, between a. and b., of:

a2. illuminating the environment to enhance the performance of said electronic imaging subsystem.

75. A method as in claim 72, further comprising the step, between a. and b., of:

a3. enhancing, via a digital image processing subsystem, the output of said electronic imaging subsystem.

76. A system as in claim 72, wherein said enhancement comprises image brightening.

77. A system as in claim 72, wherein said enhancement comprises image sharpening.

78. A method as in claim 75, wherein said enhancement comprises displaying, in an emphasized manner, at least one recognized object of interest in said image.

79. A method as in claim 75, wherein said enhancement comprises a compensatory counter distortion applied to at least a portion of said image based upon at least some edge or rectangle recognition and a computation of the distortion inherent in the image of at least one rectangular area which has been recognized in whole or in part in said image.

80. A method as in claim 75, wherein said enhancement comprises text recognition.

81. A method as in claim 80, wherein said recognized text is compared to a database of navigational labels.

82. A method as in claim 81, wherein said database of navigational labels is keyed to the output of a GPS or similar locating subsystem.

83. A method as in claim 72, further comprising automated tracking of at least one object of interest.

84. A method as in claim 72, further comprising image stabilization.

85. A method as in claim 72, further comprising:

a. text recognition;

b. automated tracking of at least one object of interest;

c. image stabilization; and

,

d. counter distortion processing.

86. A method as in claim 72, further comprising the steps, between a. and b., of:

a1. inputting control information to transducers effecting the configuration of said lens;

a2. illuminating the environment to enhance the performance of said electronic imaging subsystem;

a3. enhancing, via a digital image processing subsystem, the output of said electronic imaging subsystem by applying a compensatory distortion to at least a portion of said image based on the recognition of said edge or rectangle; and

,

a4. enhancing, via a digital image processing subsystem, the output of said electronic imaging subsystem by displaying at least one proposed complete text selection based on partial text recognition, and the comparison of said partial text compared to a database of navigational labels keyed to the output of a GPS or similar locating subsystem.

87. A method as in claim 72, wherein said environmental navigation feature is a road sign or house number.

88. A method as in claim 86, wherein said environmental navigation feature is a road sign or house number.

90. A method for providing enhanced display of environmental navigation features comprising the steps of:

a. collecting an image comprising at least in part some environmental navigation feature;

b. enhancing said image via image processing; and

,

c. displaying said enhanced image.

1. An improved system for providing enhanced display of environmental navigation features comprising system components installed in a motor vehicle including:

a. a camera further comprising an optically configureable lens and an electronic imaging subsystem;

b. a positionable mounting holding said camera;

c. transducers effecting the configuration of said positionable mounting;

d. transducers effecting the optical configuration of said lens;

e. controls effecting said transducers;

f. an illumination source optionally used to enhance the performance of said electronic imaging subsystem;

g. a digital image processing subsystem used to optionally enhance the output of said electronic imaging subsystem; and

,

h. a display showing the output of said electronic imaging subsystem.

2. A system as in claim 1, wherein said illumination source comprises visible light.

3. A system as in claim 1, wherein said illumination source comprises ultraviolet light.

4. A system as in claim 1, wherein said illumination source comprises infrared light.

5. A system as in claim 4, wherein said infrared light is in the near-visible spectrum.

6. A system as in claim 1, wherein said illumination source is configureable to coincide with the configuration of said lens.

7. A system as in claim 1, wherein said enhancement comprises image brightening.

8. A system as in claim 1, wherein said enhancement comprises image sharpening.

9. A system as in claim 1, wherein said enhancement comprises displaying, in an emphasized manner, at least one recognized object of interest in said image.

10. A system as in claim 1, wherein said enhancement comprises a compensatory counter distortion applied to at least a portion of said image based upon at least some edge or rectangle recognition and a computation of the distortion inherent in the image of at least one rectangular area which has been recognized in whole or in part in said image.

11. A system as in claim 1, wherein said enhancement comprises further processing based upon text recognition.

12. A system as in claim 11, wherein said recognized text is compared to a database of navigational labels to obtain potential matches.

13. A system as in claim 12, wherein said recognized text is only partially recognized.

14. A system as in claim 11, wherein said database of navigational labels is keyed to the output of a GPS or similar locating subsystem.

15. A system as in claim 1, further comprising automated tracking of at least one object of interest.

16. A system as in claim 1, further comprising image stabilization.

17. A system as in claim 1, further comprising:

a. text recognition;

b. automated tracking of at least one object of interest;

c. image stabilization; and

,

d. counter distortion processing.

18. A system as in claim 1, wherein said environmental navigation feature is a road sign or house number.

19. A system as in claim 1, wherein:

a. said motor vehicle is a non-commercial civilian passenger automobile; and

,

b. said camera comprises at least one video camera and is capable of capturing both daylight and night vision images.

20. A system as in claim 1, wherein:

a. said motor vehicle is a commercial civilian passenger automobile essentially equivalent to those known as taxicab or limousine; and

,

b. said camera comprises at least one video camera and is capable of capturing both daylight and night vision images.

21. A system as in claim 1, wherein:

a. said motor vehicle is a commercial civilian passenger vehicle essentially equivalent to those known as omnibus or motor coach; and

,

b. said camera comprises at least one video camera and is capable of capturing both daylight and night vision images.

22. A system as in claim 1, wherein:

a. said motor vehicle is a commercial civilian freight vehicle essentially equivalent to those known as trucks; and

,

b. said camera comprises at least one video camera and is capable of capturing both daylight and night vision images.

24. A system as in claim 23, wherein said lens is remotely optically configureable and further comprising transducers and controls to effect the configuration of said lens.

25. A system as in claim 24, wherein said controls receive input from a user electromechanical input device.

26. A system as in claim 25, wherein said input device is a multiple axis joystick.

27. A system as in claim 26, wherein said joystick employs both translational and rotational axes.

28. A system as in claim 24, wherein said controls receive input from a digital image analysis subsystem.

29. A system as in claim 23, wherein said mounting is remotely positionable and further comprising transducers and controls to effect configuration of said mounting.

30. A system as in claim 29, wherein said controls receive input from a user electromechanical input device.

31. A system as in claim 30, wherein said input device is a multiple axis joystick.

32. A system as in claim 31, wherein said joystick employs both translational and rotational axes.

33. A system as in claim 29, wherein said controls receive input from a digital image analysis subsystem.

34. A system as in claim 23, comprising, in addition, an illumination source used to enhance the performance of said electronic imaging subsystem.

35. A system as in claim 34, wherein said illumination source comprises visible light.

36. A system as in claim 34, wherein said illumination source comprises ultraviolet light.

37. A system as in claim 34, wherein said illumination source comprises infrared light.

38. A system as in claim 34, wherein said infrared light is in the near-visible spectrum.

39. A system as in claim 34, wherein said illumination source is configureable to coincide with the configuration of said lens.

40. A system as in claim 23, comprising, in addition, a digital image processing subsystem used to enhance the output of said electronic imaging subsystem.

41. A system as in claim 40, wherein said enhancement comprises image brightening.

42. A system as in claim 40, wherein said enhancement comprises image sharpening.

43. A system as in claim 40, wherein said enhancement comprises displaying, in an emphasized manner, the image of at least one recognized object of interest in said display.

44. A system as in claim 40, wherein said enhancement comprises a compensatory counter distortion applied to at least a portion of the image in said display based upon at least some edge or rectangle recognition and a computation of the distortion inherent in the image of at least one rectangular area which has been recognized in whole or in part in said image.

45. A system as in claim 40, wherein said enhancement comprises further processing based upon text recognition.

46. A system as in claim 45, wherein said recognized text is compared to a database of navigational labels to obtain potential matches.

47. A system as in claim 46, wherein said recognized text is only partially recognized and is compared to a database of navigational labels to obtain multiple potential matches.

48. A system as in claim 46, wherein said database of navigational labels is keyed to the output of a GPS or similar locating subsystem.

49. A system as in claim 23, further comprising automated tracking of at least one object of interest.

50. A system as in claim 23, further comprising image stabilization.

51. A system as in claim 23, further comprising:

a. text recognition;

b. automated tracking of at least one object of interest;

c. image stabilization; and

,

d. counter distortion processing.

52. A system as in claim 23, wherein said environmental navigation feature is a road sign or house number.

53. A system as in claim 23, wherein said mounting and camera are mounted substantially within a side view mirror housing.

54. A system as in claim 29, wherein said mounting and camera are mounted substantially within a side view mirror housing.

55. A system as in claim 23, comprising, in addition, at least one supplementary imaging subsystem configured to provide supplementary images other than of environmental navigation features.

56. A system as in claim 55, wherein said supplementary images comprise images from within the engine compartment of said vehicle.

57. A system as in claim 55, wherein said supplementary images comprise images from within the trunk storage compartment of said vehicle.

58. A system as in claim 55, wherein said supplementary images comprise images from within the freight compartment of said vehicle.

59. A system as in claim 55, wherein said supplementary images comprise images from within at least one wheel well compartment of said vehicle.

60. A system as in claim 55, wherein said supplementary images comprise images from the area directly behind said vehicle which is normally not visible to said operator of said vehicle.

61. A system as in claim 55, wherein said supplementary images comprise images from the area directly in front of said vehicle which is normally not visible to said operator of said vehicle.

62. A system as in claim 55, wherein said supplementary images comprise images from the ‘

blind spot’

of said vehicle which is normally not visible to said operator of said vehicle.

63. A system as in claim 55, wherein said supplementary images comprise images from the view out the rear window of said vehicle.

64. A system as in claim 55, wherein said supplementary images comprise images from the rear passenger compartment of said vehicle.

65. A system as in claim 64, wherein at least one supplementary imaging subsystem is configured to provide supplementary images of children seated in the rear passenger compartment.

66. A system as in claim 23, wherein at least one supplementary display subsystem is configured to provide system images to passengers seated in the rear passenger compartment.

67. A system as in claim 55, wherein said supplementary images comprise images from a printed map or other paper-based matter.

68. A system as in claim 55, wherein said supplementary images are gathered using illumination of a spectral range limited to reduce the negative impact on the night vision of said vehicle operator.

69. A system as in claim 55, wherein said supplementary images are displayed at a scale magnified when compared to the source.

70. A system as in claim 55, wherein said supplementary images comprise images of inaccessible and/or darkened areas of said vehicle.

71. A system as in claim 55, wherein said supplementary images are gathered using a hand-positionable camera transmitting its images via a wired or wireless tether.

91. A method as in claim 90, further comprising automated tracking of at least one object of interest.

92. A method as in claim 90, further comprising image stabilization.

93. A method as in claim 90, further comprising:

a. text recognition;

b. automated tracking of at least one object of interest;

c. image stabilization; and

,

d. counter distortion processing.

94. A method as in claim 90, further comprising range finding of an object of interest via sonar.

95. A method as in claim 90, further comprising range finding of an object of interest via binocular imaging.

96. A method as in claim 90, further comprising range finding of an object of interest via comparing sequential images captured from a moving vehicle.

97. A method as in claim 90, for image processing of expected objects comprising:

a. imaging a scene limited to a first spectral range in order to locate an object;

b. imaging said scene limited to at least one spectral range distinct from said first spectral range in order to locate features within said object.

98. A method as in claim 97, wherein said object is a road sign and said features are text.

99. A method as in claim 97, wherein the limitation to spectral ranges is achieved by the use of distinct filters.

100. A method as in claim 97, wherein said first spectral range is sonic energy used for object location, and said second spectral range is suitable for visual imaging.

101. A method as in claim 90, for image processing of objects comprising:

a. imaging a scene in at least three distinct spectral ranges; and

,

b. comparing the resulting images in combination in order to locate objects and/or extract features within located objects.

102. A method as in claim 90, comprising the incorporation of knowledge of at least one quality of objects of interest expected within a specified neighborhood of the current geographic location when applying recognition software.

108. A method as in claim 90, wherein said enhancement comprises the accumulation over time of a multiplicity of partial images of an object of interest into a more complete image of said object of interest by:

a. retaining from at least some of said partial images portions that belong to said object of interest; and

,

b. discarding and replacing at least some portions that do not belong to said object of interest with information from other of said partial images that does belong to said object of interest.

109. A method as in claim 108, wherein said partial images represent an object of interest being obscured by material that is actually moving over time with respect to said object of interest.

110. A method as in claim 108, wherein said partial images represent an object of interest being obscured by material that appears to be moving over time with respect to said object of interest due to the movement of a motor vehicle with respect to said object of interest and said material.

111. A method as in claim 110, wherein, for at least some of said partial images, a counter-distortion algorithm is applied to the portions retained such that said portions retained appear to be from substantially the same perspective.

112. A method as in claim 90, further comprising object tracking via image comparison.

113. A method as in claim 112, wherein said method further comprises steps including:

a. determining an area of interest in a current image;

b. performing a series of image cross-correlation computations, utilizing different offsets, between the area of interest of a previous image and the area of interest of said current image to find the substantially minimum error offset;

c. computation of a substantially optimal physical offset for said remotely positionable mounting to compensate for the offset found by the computations performed in step b.;

d. adjusting the position of said remotely positionable camera mounting according to the computation performed in step c.

114. A method as in claim 113, wherein said different offsets of step b. are limited to a neighborhood around an area obtained by extrapolating the results of at least one previous iteration of said calculation.

115. A method as in claim 90, further comprising object tracking via computation of the change in relative position between a motor vehicle and an object of interest.

116. A method as in claim 115, wherein said method further comprises steps including:

a. determining an object of interest at a current time;

b. performing a computation to determine the change in the relative position between said motor vehicle and an object of interest between a previous time and said current time;

c. computation of a substantially optimal physical mounting offset for said remotely positionable mounting corresponding to the computation performed in step b.;

d. adjusting the position of said remotely positionable camera mounting according to the computation performed in step c.

117. A method as in claim 116, wherein the computation of step b. is limited by extrapolating the results of at least one previous iteration of said calculation.

118. A method as in claim 90, further comprising partial text recognition.

119. A method as in claim 118, wherein said vehicle operator is provided with a definite match if available, or a list of potential matches ordered by confidence.

120. A method as in claim 118, wherein for at least one identified area:

a. at least one attempt is made to recognize text within said identified object utilizing constraints based on knowledge about at least one style set expected within a specified neighborhood of the current geographic location.

121. A method as in claim 120, wherein in addition:

b. if step a. is unsuccessful, at least one attempt to recognize said text within said identified object is made utilizing more general style constraints.

122. A method as in claim 118, wherein in addition:

a. a partially recognized text fragment is matched against a database of text comprising street signs and other environmental navigation features to produce possible matches.

123. A method as in claim 122, wherein said database is restricted to a specified neighborhood of the current geographic location.

124. A method as in claim 122, wherein said database is restricted based upon at least one previous successful match.

125. A method as in claim 122, wherein:

a. said matches take into account a measurement of the size of unrecognizable material between at least one pair of recognized text fragments and the size of the material between material matching each of the fragments of said pair in a database entry; and

b. the sizes are compared for a likelihood or confidence of matching based on the closeness of the two sizes.

126. A method as in claim 122, wherein there is feedback after at least one attempt at a match and a second match is attempted based on the additional information provided during feedback.

128. A method as in claim 90, wherein:

a. recognized text only partially matches one or more database entries; and

,

b. said recognized text is re-processed for recognition utilizing knowledge about each of at least two potential partial matches in order to attempt a higher confidence for some of said potential matches with respect to others.

129. A method as in claim 90, wherein:

a. recognized text is only partially recognizable;

b. said recognized text is re-processed by determining a metric for at least one unrecognizable portion of said text; and

,

c. comparing said metric to a metric determined for the equivalent portion of at least one partial potential match of a database entry.

130. A method as in claim 90, wherein:

a. recognized text only partially matches at least one database entry; and

,

b. said recognized text is re-processed utilizing at least one more intensive or sophisticated recognition algorithm applied to at least one portion of said recognized text that does not match the equivalent portion of one or more potential matches, in order to determine if an alternative recognition target is reasonable.

131. A method as in claim 130, wherein said more intensive or sophisticated recognition algorithms employ artificial intelligence techniques.

133. A system as in claim 132, wherein:

a. said image collection subsystem is configured to point along a first axis;

b. said illumination subsystem is configured to point along a second axis;

c. said first axis and said second axis have, between them, a single rotational degree of freedom; and

,

d. said coordinated manner comprises that said first axis and said second axis are configured in a coordinated manner to improve the overlap of the area of interest comprising the image being captured and the area being illuminated.

134. A system as in claim 133, wherein said single rotational degree of freedom is adjusted to conform to the approximate center of the area being illuminated to the approximate center of the area of the subject of the image being captured based upon the distance between said system and said areas.

135. A system as in claim 134, wherein said distance is determined in response to the configuration of the focus control of the lens of said image collection subsystem

136. A system as in claim 132, wherein:

a. said image collection subsystem further comprises a configureable lens subsystem;

b. said illumination subsystem further comprises a configureable lens subsystem; and

,

c. said coordinated manner comprises that said first lens subsystem and said second lens subsystem are configured to improve the overlap of the area or interest comprising the image being captured and the area being illuminated.

137. A system as in claim 136, wherein said size of the area being illuminated is adjusted to conform to the size of the area of the subject of the image being captured.

138. A system as in claim 137, wherein said size is determined in response to the configuration of the zoom control of the lens of said image collection subsystem.

140. A system as in claim 139, wherein said area within some portion of an annulus comprises substantially an entire annular ring.

141. A system as in claim 140, further comprising a lens subsystem capable of adjusting the spread of said illumination.

142. A system as in claim 139, wherein said illumination source further comprises a multiplicity of illumination sources.

143. A system as in claim 142, wherein said multiplicity of illumination sources provide illumination in a multiplicity of distinct spectra of radiant energy.

145. A system as in claim 144, wherein said distal end comprises a configuration substantially occupying an area within some portion of an annulus surrounding the aperture of said image collection subsystem.

146. A system as in claim 145, wherein said area within some portion of an annulus comprises substantially an entire annular ring.

147. A system as in claim 144, wherein the shape of said light guide provides a path that, at least in part, surrounds said image collection subsystem creating a cavity within which said image collection subsystem resides.

148. A system as in claim 147, wherein said at least in part comprises substantially.

149. A system as in claim 148, wherein said distal end comprises a configuration occupying an area substantially within an annular ring surrounding the aperture of said image collection subsystem.

150. A system as in claim 144, wherein said light guide further comprises a multiplicity of sub-units.

151. A system as in claim 147, wherein said light guide further comprises a multiplicity of sub-units.

152. A system as in claim 149, wherein:

a. said light guide further comprises a multiplicity of sub-units; and

,

b. at least some of said sub-units comprise wedge-shaped sections of a portion of said light guide which port ion comprises a configuration substantially comprising an arc of a circularly symmetrical volume.

98-1. A method as in claim 97, wherein the limitation to spectral ranges is achieved by the use of illumination sources providing radiant energy of distinct spectra.

99-2. A method as in claim 97, wherein said first spectral range is infra red heat imaging, and said second spectral range is visible light imaging.

103. A method as in 102 wherein at least one of said at least one quality relates to shape.

104. A method as in 102 wherein at least one of said at least one quality relates to size.

105. A method as in 102 wherein at least one of said at least one quality relates to location.

106. A method as in 102 wherein at least one of said at least one quality relates to color.

107. A method as in 102 wherein at least one of said at least one quality relates to the proportion of areas of at least two distinct colors.

127. A method as in 126, wherein said feedback comprises the knowledge that a provisionally recognized letter corresponds to a geometrically similar letter in a database entry, and said second match attempts to alternatively recognize said provisionally recognized letter as said geometrically similar letter.

132. A system for capturing images comprising:

a. a configureable image collection subsystem;

b. a configureable illumination subsystem; and

,

c. a control subsystem capable of modifying the configuration of said image collection subsystem and said illumination subsystem in a coordinated manner.

139. A system for capturing images comprising:

a. an image collection subsystem;

b. an illumination source subsystem with an illumination aperture comprising a configuration substantially occupying an area within some portion of an annulus surrounding the aperture of said image collection subsystem.

144. A system for capturing images comprising:

a. an image collection subsystem;

b. an illumination source subsystem positioned such that said camera subsystem would, at least partially, obscure the path of the illumination provided by said illumination source from the subject of the image being collected;

c. a light guide positioned such that the path of said illumination progresses, at least in part, around at least some obscuring portion of said image collection subsystem, and emerges from a distal end of said light guide at a position peripheral to the obscuring portion of said image collection subsystem.