Real-time image scanning and processing

• US 20100141555A1
• Filed: 12/22/2006
• Published: 06/10/2010
• Est. Priority Date: 12/25/2005
• Status: Active Grant

• Current Assignee: Elbit Systems Limited
• Original Assignee: Elbit Systems Limited
• Inventors: Bloch, Asaf, Rorberg, Eli, Ophir, Yoav
• Granted Patent: US 8,138,991 B2
• Time in Patent Office: Days
• US Class Current: 345/8
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: G02B 2027/011 comprising device for correcting geometrical aberrations, distortion
• CPC Class Codes: G02B 2027/0112 comprising device for genereting colour display
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/648 for automatically maintaining a reference alignment, e.g. in self-levelling surveying instruments surveying instruments per se G01C
• CPC Class Codes: G06T 5/006 Geometric correction correction of chromatic aberrations adapted to be used in an image pickup device containing an electronic image sensor H04N23/10; detecting, correcting, reducing or removing artefacts resulting only from the lens unit, e.g. flare, shading, vignetting or "cos4" H04N25/61
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G09G 2340/125 wherein one of the images is motion video
• CPC Class Codes: G09G 5/377 for mixing or overlaying two or more graphic patterns G09G5/02, G09G5/397 take precedence
• CPC Class Codes: H04N 2101/00 Still video cameras
• CPC Class Codes: H04N 5/772 the recording apparatus and the television camera being placed in the same enclosure
• CPC Class Codes: H04N 5/775 between a recording apparatus and a television receiver

Abstract

An apparatus for displaying at least one image with respect to a line-of-sight (LOS), with an image source, a display processor and a displaying unit. The display processor is coupled with the image source and the displaying unit. The image source provides at least one spatially unregistered image, and the display processor spatially registers the spatially unregistered image with the LOS, thereby generating a respective at least one spatially registered image. The displaying unit displays at least one spatially registered pixel on a displaying surface. The display processor includes a storage unit, an image processor and a pixel locator, the pixel locator being coupled with the image processor, wherein the storage unit stores the spatially unregistered image. The image processor selects at least one projection pixel to be displayed and the pixel locator determines, in each spatially unregistered image, the location of the spatially registered pixel corresponding to the selected projection pixel.

Claims

1. Apparatus for displaying at least one image with respect to a line-of-sight (LOS), with substantially no image latency as perceived by a human user, the LOS being determined between the head of a user and at least one coordinate system established in space, said at least one coordinate system and said displayed at least one image being in relative motion there between, the apparatus comprising:

an image source, for providing at least one spatially unregistered image;

a display processor, coupled with said image source, for spatially registering said at least one spatially unregistered image with said LOS, thereby generating a respective at least one spatially registered image,

wherein said display processor comprises;

a storage unit, for storing said at least one spatially unregistered image;

an image processor, for selecting at least one projection pixel to be displayed;

a pixel locator, coupled with said image processor, for determining, in each of said at least one spatially unregistered image, the location of at least one spatially registered pixel corresponding to said selected projection pixel; and

a displaying unit, coupled with said display processor, for displaying said at least one spatially registered pixel on a displaying surface.

2. The apparatus according to claim 1, wherein said at least one coordinate system is established in a real space.

3. The apparatus according to claim 1, wherein said at least one coordinate system is established in a virtual space.

4. The apparatus according to claim 1, wherein said image processor adjusts the grey levels of said at least one spatially registered pixel before said displaying unit displays said at least one spatially registered pixel.

5. The apparatus according to claim 1, wherein said image processor adjusts the luminance levels of said at least one spatially registered pixel before said displaying unit displays said at least one spatially registered pixel.

6. The apparatus according to claim 1, wherein said image processor adjusts the chrominance levels of said at least one spatially registered pixel before said displaying unit displays said at least one spatially registered pixel.

7. The apparatus according to claim 4, wherein said image processor adjusts said grey levels using at least one of α

-β

filtering and at least one antialiasing algorithm.

8. The apparatus according to claim 5, wherein said image processor adjusts said luminance levels using at least one of α

-β

filtering and at least one antialiasing algorithm.

9. The apparatus according to claim 6, wherein said image processor adjusts said chrominance levels using at least one of α

-β

filtering and at least one antialiasing algorithm.

10. The apparatus according to claim 1, further comprising a recording unit, coupled with said display processor, for recording said at least one spatially registered pixel.

11. The apparatus according to claim 1, wherein said displaying surface is selected from the list consisting of:

an optical visor viewable by at least one eye of said user;

a helmet mounted display;

a heads-up display;

a heads-down display;

a virtual heads-up display; and

a virtual multi-function display.

12. The apparatus according to claim 1, wherein said pixel locator comprises a visor distortion corrector for determining said location of said at least one spatially registered pixel in accordance with distortion inherent in an optical visor used as a displaying surface.

13. The apparatus according to claim 12, wherein visor distortion corrector determines said location of said at least one spatially registered pixel, according to at least one operation selected from the list consisting of:

a known polynomial which accounts for said distortion inherent in said optical visor;

a look-up table; and

a function which allows said location of said at least one spatially registered pixel to be determined.

14. The apparatus according to claim 1, wherein said image source is selected from the list consisting of:

a camera;

an auxiliary data generator; and

an image generator.

15. The apparatus according to claim 1, wherein said pixel locator comprises a camera distortion corrector for determining said location of said at least one spatially registered pixel in said at least one spatially unregistered image received from a camera in accordance with distortion inherent in the optics of said camera.

16. The apparatus according to claim 15, wherein said camera distortion corrector determines said location of said at least one spatially registered pixel, according to at least one operation selected from the list consisting of:

a known polynomial which accounts for said distortion inherent in the optics of said camera in said at least one spatially unregistered image received from said camera;

a look-up table; and

a function which allows said location of said at least one spatially registered pixel to be determined.

17. The apparatus according to claim 14, wherein said camera is selected from the list consisting of:

an IR camera;

a visible light camera;

an ICCD camera;

an image intensified camera;

a sensor fused camera; and

an aircraft panoramic night vision system.

18. The apparatus according to claim 14, wherein said image generator generates at least one virtual reality image.

19. The apparatus according to claim 1, further comprising a head sensor, coupled with said display processor, for determining the position of the head of said user, relative to said at least one coordinate system.

20. The apparatus according to claim 19, wherein said position comprises the location and the orientation of the head of said user.

21. The apparatus according to claim 19, wherein said pixel locator comprises a head line-of-sight corrector for determining said location of said at least one spatially registered pixel in accordance with said determined position of the head of said user.

22. The apparatus according to claim 21, wherein said head line-of-sight corrector determines said location of said at least one spatially registered pixel, according to at least one operation selected from the list consisting of:

an equation using at least one scaling factor;

an equation using at least one shifting factor;

an equation using at least one rotation factor;

an equation using at least one cropping parameter;

an Euler correction;

a polynomial approximation;

an algorithm using a prediction of the position of the head of said user; and

a spatial filter of said at least one spatially unregistered image.

23. The apparatus according to claim 21, wherein said head line-of-sight corrector determines said location of said at least one spatially registered pixel in accordance with said determined position of the head of said user for at least one eye of said user.

24. The apparatus according to claim 1, wherein said at least one spatially unregistered image is selected from the list consisting of:

an IR camera image;

a visible light camera image;

an ICCD camera image;

an image intensified camera image;

an aircraft panoramic night vision system image;

a VHUD image;

a VMFD image;

a virtual reality image;

a virtual world image;

static symbology;

dynamic symbology;

a digital map;

a weapons camera image;

a weapons view image;

a target zoom camera;

an HMD day camera image;

an HMD night camera image;

three-dimensional video input; and

internal calibration patterns.

25. The apparatus according to claim 1, wherein said image processor retrieves at least one neighbor pixel of said at least one spatially registered pixel from said at least one spatially unregistered image stored is said storage unit.

26. The apparatus according to claim 25, wherein said image processor filters a retrieved at least one spatially registered pixel and said retrieved at least one neighbor pixel using a bilinear interpolation.

27. The apparatus according to claim 26, wherein said bilinear interpolation is executed using the expression

Grey_level(x″

″

, y″

″

)=A(1−

Δ

x)(1−

Δ

y)+B(Δ

x)(1−

Δ

y)+C(1−

Δ

x)(Δ

y)+D(Δ

x)(Δ

y)

wherein Grey_level(x″

″

,y″

″

) is the interpolated grey levels for said retrieved at least one spatially registered pixel, A is the current grey level of said retrieved at least one spatially registered pixel, B, C and D are respectively the grey levels of a first, second and third retrieved at least one neighbor pixel of said at least one spatially registered pixel, and Δ

x and Δ

y are subpixel information for said retrieved at least one spatially registered pixel.

28. The apparatus according to claim 25, wherein said image processor filters a retrieved at least one spatially registered pixel and said retrieved at least one neighbor pixel using antialiasing.

29. The apparatus according to claim 1, further comprising an aircraft data system, coupled with said display processor, for determining the position of an aircraft mounted with said apparatus.

30. The apparatus according to claim 29, wherein said aircraft data system monitors data determined by aviation systems related to said aircraft.

31. The apparatus according to claim 1, wherein said pixel locator comprises a canopy distortion corrector for determining said location of said at least one spatially registered pixel in accordance with distortion inherent in the canopy of a vehicle.

32. The apparatus according to claim 31, wherein canopy distortion corrector determines said location of said at least one spatially registered pixel, according to said at least one operation selected from the list consisting of:

a known polynomial which accounts for said distortion inherent in said canopy;

a look-up table; and

a function which allows said location of said at least one spatially registered pixel to be determined.

33. Method for displaying at least one image with respect to a line-of-sight (LOS), with substantially no image latency as perceived by a human user, the LOS being determined between the head of a user and at least one coordinate system established in space, said at least one coordinate system and said displayed at least one image being in relative motion there between, the method comprising the procedures of:

providing at least one spatially unregistered image, from an image source;

storing said at least one spatially unregistered image;

selecting at least one projection pixel to be displayed;

spatially registering said at least one spatially unregistered image with said LOS, thereby generating a respective at least one spatially registered image, by determining, in each of said at least one spatially unregistered image, the location of at least one spatially registered pixel corresponding to said selected projection pixel; and

displaying said at least one spatially registered pixel.

34. The method according to claim 33, wherein said at least one coordinate system is established in a real space.

35. The method according to claim 33, wherein said at least one coordinate system is established in a virtual space.

36. The method according to claim 33, further comprising the procedure of adjusting the grey levels of said at least one spatially registered pixel before said procedure of displaying.

37. The method according to claim 33, further comprising the procedure of adjusting the luminance levels of said at least one spatially registered pixel before said procedure of displaying.

38. The method according to claim 33, further comprising the procedure of adjusting the chrominance levels of said at least one spatially registered pixel before said procedure of displaying.

39. The method according to claim 36, wherein said procedure of adjusting said grey levels is executed using at least one of α

-β

filtering and at least one antialiasing algorithm.

40. The method according to claim 37, wherein said procedure of adjusting said luminance levels is executed using at least one of α

-β

filtering and at least one antialiasing algorithm.

41. The method according to claim 38, wherein said procedure of adjusting said chrominance levels is executed using at least one of α

-β

filtering and at least one antialiasing algorithm.

42. The method according to claim 33, further comprising the procedure of recording said at least one spatially registered pixel.

43. The method according to claim 33, wherein said displayed said at least one spatially registered pixel is displayed on a displaying surface selected from the list consisting of:

an optical visor viewable by at least one eye of said user;

a helmet mounted display;

a heads-up display;

a heads-down display;

a virtual heads-up display; and

a virtual multi-function display.

44. The method according to claim 33, wherein said procedure of spatially registering comprises determining said location of said at least one spatially registered pixel in accordance with at least one type of distortion, said at least one type of distortion selected from the list consisting of:

optical visor distortion inherent in an optical visor;

vehicle canopy distortion inherent in the canopy of a vehicle;

camera distortion inherent in the optics of a camera; and

head LOS distortion.

45. The method according to claim 44, wherein said procedure of spatially registering comprises determining said location of said at least one spatially registered pixel in accordance with said optical visor distortion, according to at least one operation selected from the list consisting of:

a known polynomial which accounts for said inherent distortion in said optical visor;

a look-up table; and

a function which allows said location of said at least one spatially registered pixel to be determined.

46. The method according to claim 44, wherein said procedure of spatially registering comprises determining said location of said at least one spatially registered pixel in accordance with said camera distortion in said at least one spatially unregistered image received from a camera, according to at least one operation selected from the list consisting of:

a known polynomial which accounts for said camera distortion in said at least one spatially unregistered image received from said camera;

a look-up table; and

a function which allows said location of said at least one spatially registered pixel to be determined.

47. The method according to claim 44, further comprising the procedure of determining the position of the head of said user, relative to said at least one coordinate system.

48. The method according to claim 47, wherein said position comprises the location and the orientation of the head of said user.

49. The method according to claim 47, wherein said procedure of spatially registering comprises determining said location of said at least one spatially registered pixel in accordance with said head LOS distortion and the determined position of the head of said user, according to at least one operation selected from the list consisting of:

an equation using at least one scaling factor;

an equation using at least one shifting factor;

an equation using at least one rotation factor;

an equation using at least one cropping parameter;

an Euler correction;

a polynomial approximation;

an algorithm using a prediction of the position of the head of said user; and

a spatial filter of said at least one spatially unregistered image.

50. The method according to claim 47, wherein said procedure of spatially registering comprises determining said location of said at least one spatially registered pixel in accordance with said head LOS distortion and the determined position of the head of said user for at least one eye of said user.

51. The method according to claim 33, wherein said at least one spatially unregistered image is selected from the list consisting of:

an IR camera image;

a visible light camera image;

an ICCD camera image;

an image intensified camera image;

an aircraft panoramic night vision system image;

a VHUD image;

a VMFD image;

a virtual reality image;

a virtual world image;

static symbology;

dynamic symbology;

a digital map;

a weapons camera image;

a weapons view image;

a target zoom camera;

an HMD day camera image;

an HMD night camera image;

three-dimensional video input; and

internal calibration patterns.

52. The method according to claim 33, further comprising the procedure of reading at least one neighbor pixel of said at least one spatially registered pixel from said stored at least one spatially unregistered image.

53. The method according to claim 52, further comprising the procedure of filtering a read at least one spatially registered pixel and said read at least one neighbor pixel using a bilinear interpolation.

54. The method according to claim 53, wherein said bilinear interpolation is executed using the expression

Grey_level(x″

″

, y″

″

)=A(1−

Δ

x)(1−

Δ

y)+B(Δ

x)(1−

Δ

y)+C(1−

Δ

x)(Δ

y)+D(Δ

x)(Δ

y)

wherein Grey_level(x″

″

,y″

″

) is the interpolated grey levels for said read at least one spatially registered pixel, A is the current grey level of said read at least one spatially registered pixel, B, C and D are respectively the grey levels of a first, second and third read at least one neighbor pixel of said at least one spatially registered pixel, and Δ

x and Δ

y are subpixel information for said read at least one spatially registered pixel.

55. The method according to claim 52, further comprising the procedure of filtering a read at least one spatially registered pixel and said read at least one neighbor pixel using antialiasing.

56. The method according to claim 33, further comprising the procedure of determining the position of an aircraft.

57. The method according to claim 56, further comprising the procedure of monitoring data determined by aviation systems related to said aircraft.

58. The method according to claim 44, wherein said procedure of spatially registering comprises determining said location of said at least one spatially registered pixel in accordance with said vehicle canopy distortion, according to said at least one operation selected from the list consisting of:

a known polynomial which accounts for said vehicle canopy distortion;

a look-up table; and

a function which allows said location of said at least one spatially registered pixel to be determined.