Apparatus and method for sensing and processing images

• US 5,815,608 A
• Filed: 07/29/1996
• Issued: 09/29/1998
• Est. Priority Date: 03/25/1992
• Status: Expired due to Term

• Current Assignee: Renesas Electronics Corporation
• Original Assignee: Mitsubishi Denki Kabushiki Kaisha (Mitsubishi Electric Corporation)
• Inventors: Toyoda, Takashi, Ohta, Jun, Funatsu, Eiichi, Kyuma, Kazuo, Koshiba, Yuichi, Tai, Shuichi, Lange, Eberhard, Hara, Kunihiko, Nitta, Yoshikazu
• Primary Examiner(s): Couso, Jose L.
• Application Number:
  US08/688,684
• Time in Patent Office: 792 Days
• Field of Search: 382/312, 382/100, 382/155, 382/156, 395/22, 395/25, 437/3, 348/311, 250/208.1, 350/96.15, 358/41
• US Class Current: 382/312
• CPC Class Codes: G06V 10/10 Image acquisition document image scanning and transmission H04N1/00; control of digital cameras H04N23/60

Abstract

An apparatus for sensing and processing images is provided with a photo detector array arranged in a matrix form, a control circuit for feeding a row of the array with voltage for sensitivity control, and a neural network for processing current flowing from a column of the array to the ground in order to obtain an apparatus for sensing and processing images having a simple configuration, a high frame speed, the capability of forming a focus of attention, and high throughput of data and possibility of connecting to the neural network.

Claims

1. An apparatus for sensing and processing images comprising:

an array of variable sensitivity photo detectors arranged in a matrix form, said variable sensitivity photo detectors having a conversion efficiency which is a monotone, smoothly increasing function of an applied control voltage, wherein the conversion efficiency can assume both positive and negative values;

a control circuit for feeding rows of said array with voltages for sensitivity control; and

a neural network for processing currents flowing from columns of said array to ground.

2. An apparatus for sensing and processing images according to claim 1, wherein said control circuit includes a shift register for shifting analog voltage.

3. An apparatus for sensing and processing images according to claim 2, wherein said analog voltage shift register includes a charge coupled device.

4. An apparatus for sensing and processing images according to claim 2, wherein said analog voltage shift register includes a chain of sample/hold circuits.

5. An apparatus for sensing and processing images according to claim 2, wherein a digital shift register having a width of L bits is employed as said analog voltage shift register so as to cooperate with a set of D/A convertors having a width of L bits.

6. An apparatus for sensing and processing images according to claim 2, wherein a chain of digital shift registers having a length of L and a width of one bit is employed as said analog voltage shift register so as to obtain L+1 PCM encoding analog voltage levels.

7. An apparatus for sensing and processing images according to claim 5, wherein an output terminal of said digital shift register is connected to an input terminal thereof so as to obtain a closed loop operation mode.

8. An apparatus for sensing and processing images according to claim 6, wherein an output terminal of said digital shift register is connected to an input terminal thereof so as to obtain a closed loop operation mode.

9. An apparatus for sensing and processing images according to claim 2, wherein a shift register receives input from a function generator.

10. An apparatus for sensing and processing images according to claim 3, wherein a shift register receives input from a function generator.

11. An apparatus for sensing and processing images according to claim 4, wherein a shift register receives input from a function generator.

12. An apparatus for sensing and processing images according to claim 5, wherein a shift register receives input from a function generator.

13. An apparatus for sensing and processing images according to claim 6, wherein a shift register receives input from a function generator.

14. An apparatus for sensing and processing images according to claim 1, wherein said neural network includes a resettable integrator.

15. An apparatus for sensing and processing images according to claim 1, wherein said photo detector in said array is provided with a color filter.

16. An apparatus for sensing and processing images according to claim 1, wherein said photo detectors disposed on alternate rows in said array are shifted by a distance equal to half distance between two adjacent photo detectors in said row so as to provide a hexagonal configuration of said photo detectors.

17. An apparatus for sensing and processing images according to claim 1, wherein said array is provided with a hexagonal configuration of said photo detectors by partially eliminating said photo detectors.

18. An apparatus for sensing and processing images according to claim 16, wherein said photo detector has a hexagonal form.

19. An apparatus for sensing and processing images according to claim 17, wherein said photo detector has a hexagonal form.

30. An apparatus for sensing and processing images according to claim 1, wherein a pair of semiconductor layers formed on a common semiconductor layer are provided to rise in an interdigital form, and teeth portions of said pair of interdigital semiconductor layers are combined with each other so as to form each of said photo detectors.

31. An apparatus for sensing and processing images according to claim 2, wherein a pair of semiconductor layers formed on a common semiconductor layer are provided to rise in an interdigital form, and teeth portions of said pair of interdigital semiconductor layers are combined with each other so as to form each of photo detectors.

32. An apparatus for sensing and processing images according to claim 3, wherein a pair of semiconductor layers formed on a common semiconductor layer are provided to rise in an interdigital form, and teeth portions of said pair of interdigital semiconductor layers are combined each other so as to form each of said photo detectors.

33. An apparatus for sensing and processing images according to claim 4, wherein a pair of semiconductor layers formed on a common semiconductor layer are provided to rise in an interdigital form, and teeth portions of said pair of interdigital semiconductor layers are combined with each other so as to form each of said detectors.

34. An apparatus for sensing and processing images according to claim 5, wherein a pair of semiconductor layers formed on a common semiconductor layer are provided to rise in an interdigital form, and teeth portions of said pair of interdigital semiconductor layers are combined with each other so as to form each of said photo detectors.

35. An apparatus for sensing and processing images according to claim 30, wherein a plurality of photo detectors are adjacently arranged while an electrode connected to a power source is defined as a common electrode.

36. An apparatus for sensing and processing images according to claim 31, wherein a plurality of photo detectors are adjacently arranged while an electrode connected to a power source is defined as a common electrode.

37. An apparatus for sensing and processing images according to claim 32, wherein a plurality of photo detectors are adjacently arranged while an electrode connected to a power source is defined as a common electrode.

38. An apparatus for sensing and processing images according to claim 33, wherein a plurality of photo detectors are adjacently arranged while an electrode connected to a power source is defined as a common electrode.

39. An apparatus for sensing and processing images according to claim 34, wherein said interdigital portion is embedded in said common semiconductor layer, and said electrode is also embedded in said common semiconductor layer in a wiring form.

40. An apparatus for sensing and processing images according to claim 34, wherein a plurality of photo detectors are adjacently arranged while an electrode connected to a power source is defined as a common electrode.

41. An apparatus for sensing and processing images according to claim 30, wherein a plurality of photo detectors are adjacently arranged while an electrode not connected to a power source is defined as a common electrode.

42. An apparatus for sensing and processing images according to claim 31, wherein a plurality of photo detectors are adjacently arranged while an electrode not connected to a power source is defined as a common electrode.

43. An apparatus for sensing and processing images according to claim 32, wherein a plurality of photo detectors are adjacently arranged while an electrode not connected to a power source is defined as a common electrode.

44. An apparatus for sensing and processing images according to claim 33, wherein a plurality of photo detectors are adjacently arranged while an electrode not connected to a power source is defined as a common electrode.

45. An apparatus for sensing and processing images according to claim 34, wherein a plurality of photo detectors are adjacently arranged while an electrode not connected to a power source is defined as a common electrode.

46. An apparatus for sensing and processing images according to claim 30, wherein a plurality of photo detectors are arranged in a matrix form while said two electrodes are defined as common electrodes.

47. An apparatus for sensing and processing images according to claim 31, wherein a plurality of photo detectors are arranged in a matrix form while said two electrodes are defined as common electrodes.

48. An apparatus for sensing and processing images according to claim 32, wherein a plurality of photo detectors are arranged in a matrix form while said two electrodes are defined as common electrodes.

49. An apparatus for sensing and processing images according to claim 33, wherein a plurality of photo detectors are arranged in a matrix form while said two electrodes are defined as common electrodes.

50. An apparatus for sensing and processing images according to claim 34, wherein a plurality of photo detectors are arranged in a matrix form while said two electrodes are defined as common electrodes.

51. An apparatus for sensing and processing images according to claim 30, wherein a plurality of variable sensitivity photo detectors are arranged in a matrix form while two electrodes are defined as common electrodes, and an array of light emitting diodes are arranged on said plurality of photo detectors.

52. An apparatus for sensing and processing images according to claim 31, wherein a plurality of variable sensitivity photo detectors are arranged in a matrix form while two electrodes are defined as common electrodes, and an array of light emitting diodes are arranged on said plurality of photo detectors.

53. An apparatus for sensing and processing images according to claim 32, wherein a plurality of variable sensitivity photo detectors are arranged in a matrix form while two electrodes are defined as common electrodes, and an array of light emitting diodes are arranged on said plurality of photo detectors.

54. An apparatus for sensing and processing images according to claim 33, wherein a plurality of variable sensitivity photo detectors are arranged in a matrix form while two electrodes are defined as common electrodes, and an array of light emitting diodes are arranged on said plurality of photo detectors.

55. An apparatus for sensing and processing images according to claim 34, wherein a plurality of variable sensitivity photo detectors are arranged in a matrix form while two electrodes are defined as common electrodes, and an array of light emitting diodes are arranged on said plurality of photo detectors.

56. An apparatus for sensing and processing images according to claim 30, wherein said interdigital portion is embedded in said common semiconductor layer, and said electrode is also embedded in said common semiconductor layer in a wiring form.

57. An apparatus for sensing and processing images according to claim 31, wherein said interdigital portion is embedded in said common semiconductor layer, and said electrode is also embedded in said common semiconductor layer in a wiring form.

58. An apparatus for sensing and processing images according to claim 32, wherein said interdigital portion is embedded in said common semiconductor layer, and said electrode is also embedded in said common semiconductor layer.

59. An apparatus for sensing and processing images according to claim 33, wherein said interdigital portion is embedded in said common semiconductor layer, and said electrode is also embedded in said common semiconductor layer in a wiring form.

60. An apparatus for sensing and processing images according to claim 56, wherein a plurality of photo detectors are adjacently arranged while an electrode connected to a power source is defined as a common electrode.

61. An apparatus for sensing and processing images according to claim 57, wherein a plurality of photo detectors are adjacently arranged while an electrode connected to a power source is defined as a common electrode.

62. An apparatus for sensing and processing images according to claim 58, wherein a plurality of photo detectors are adjacently arranged while an electrode connected to a power source is defined as a common electrode.

63. An apparatus for sensing and processing images according to claim 59, wherein a plurality of photo detectors are adjacently arranged while an electrode connected to a power source is defined as a common electrode.

64. An apparatus for sensing and processing images according to claim 39, wherein a plurality of photo detectors are adjacently arranged while an electrode connected to a power source is defined as a common electrode.

65. An apparatus for sensing and processing images according to claim 56, wherein a plurality of photo detectors are adjacently arranged while an electrode not connected to a power source is defined as a common electrode.

66. An apparatus for sensing and processing images according to claim 57, wherein a plurality of photo detectors are adjacently arranged while an electrode not connected to a power source is defined as a common electrode.

67. An apparatus for sensing and processing images according to claim 58, wherein a plurality of photo detectors are adjacently arranged while an electrode not connected to a power source is defined as a common electrode.

68. An apparatus for sensing and processing images according to claim 59, wherein a plurality of photo detectors are adjacently arranged while an electrode not connected to a power source is defined as a common electrode.

69. An apparatus for sensing and processing images according to claim 39, wherein a plurality of photo detectors are adjacently arranged while a plurality of photo detectors are adjacently arranged while an electrode not connected to a power source is defined as a common electrode.

70. An apparatus for sensing and processing images according to claim 56, wherein a plurality of photo detectors are arranged in a matrix form while two electrodes are defined as common electrodes.

71. An apparatus for sensing and processing images according to claim 57, wherein a plurality of photo detectors are arranged in a matrix form while two electrodes are defined as common electrodes.

72. An apparatus for sensing and processing images according to claim 58, wherein a plurality of photo detectors are arranged in a matrix form while two electrodes are defined as common electrodes.

73. An apparatus for sensing and processing images according to claim 59, wherein a plurality of photo detectors are arranged in a matrix form while two electrodes are defined as common electrodes.

74. An apparatus for sensing and processing images according to claim 39, wherein a plurality of photo detectors are arranged in a matrix form while two electrodes are defined as common electrodes.

75. An apparatus for sensing and processing images according to claim 56, wherein a plurality of variable sensitivity photo detectors are arranged in a matrix form while two electrodes are defined as common electrodes, and an array of light emitting diodes are arranged on said plurality of photo detectors.

76. An apparatus for sensing and processing images according to claim 57, wherein a plurality of variable sensitivity photo detectors are arranged in a matrix form while two electrodes are defined as common electrodes, and an array of light emitting diodes are arranged on said plurality of photo detectors.

77. An apparatus for sensing and processing images according to claim 58, wherein a plurality of variable sensitivity photo detectors are arranged in a matrix form while two electrodes are defined as common electrodes, and an array of light emitting diodes are arranged on said plurality of photo detectors.

78. An apparatus for sensing and processing images according to claim 59, wherein a plurality of variable sensitivity photo detectors are arranged in a matrix form while two electrodes are defined as common electrodes, and an array of light emitting diodes are arranged on said plurality of photo detectors.

79. An apparatus for sensing and processing images according to claim 39, wherein a plurality of variable sensitivity photo detectors are arranged in a matrix form while two electrodes are defined as common electrodes, and an array of light emitting diodes are arranged on said plurality of photo detectors.

20. A method of sensing and processing images comprising the steps of:

projecting an image on an array having a plurality of variable sensitivity photo detectors, said variable sensitivity photo detectors having a conversion efficiency which is a monotone, smoothly increasing function of an applied control voltage, wherein the conversion efficiency can assume both positive and negative values;

applying control voltages to rows of said array so as to control sensitivity of said photo detectors; and

providing a neural network with current flowing from each column of said array to the ground as an input signal so as to perform processing,

whereby said control voltages form a vector S, said projected image forms a matrix W, and said neural network employs a matrix T;

wherein said neural network generates an output V defined by V=TWS, wherein said vector S and said matrix T are selected to achieve at least one of focus of attention, horizontal and/or vertical one-dimensional convolution of a kernel, and contents of processing of one-dimensional or two-dimensional orthogonal and non-orthogonal transformation.

21. A method of sensing and processing images according to claim 20, wherein a time dependent image can be processed by using a neural network having a resettable integrator.

22. A method of sensing and processing images comprising the steps of:

projecting an image on an array having a plurality of photo detectors, said variable sensitivity photo detectors having a conversion efficiency which is a monotone, smoothly increasing function of an applied control voltage, wherein the conversion efficiency can assume both positive and negative values;

applying control voltages to rows of said array so as to control sensitivity of said photo detectors, wherein a set of said control voltages forms a matrix to be multiplied by a matrix provided by said projected image; and

classifying said projected image into a class by using a classifier actuated according to current output from each column of said array.

23. A method of sensing and processing images according to claim 22, further comprising the steps of:

performing said classifying step by using a merit function representing the degree of quality of classification in said classifier;

processing said merit function according to an optimizing algorithm so as to vary a value of said control voltage according to results of said processing, and thereby optimizing said merit function.

24. An apparatus for sensing and processing images comprising an image sensor including:

a two-dimensional array of variable sensitivity photo detectors having a conversion efficiency from light to current which is a monotone, smoothly increasing function which varies depending upon magnitude of voltage across two terminals, wherein the conversion efficiency can assume both positive and negative values;

a control circuit having one terminal of each photo detector connected to a common control electrode for each row, and the other terminal connected to a common output electrode for each column, wherein a plurality of voltage data is stored in said control circuit so as to transfer said data, and said stored voltage data is partially connected to said control electrode;

a one-dimensional memory enabling shift of said stored data; and

a fixed connecting portion for connecting between said one-dimensional memory and said output electrode or a variable connecting portion for connecting between said one-dimensional memory and said output electrode.

25. An apparatus for sensing and processing images comprising an image sensor including:

a two-dimensional array of photoreceptive elements formed by variable sensitivity photo detectors having one terminal directly connected to an output terminal and another terminal connected via a capacitor to the ground and connected via a switching gate to a control terminal, and having a conversion efficiency from light to current which is a monotone, smoothly increasing function which varies depending upon magnitude of voltage across two terminals, wherein the conversion efficiency can assume both positive and negative values, and wherein said control terminal is connected to a common control electrode for each row, and said output terminal is connected to a common output electrode for each column;

a control circuit having a terminal connected to said control electrode so as to output any voltage; and

an output circuit having a mechanism for applying current from any one of said output electrodes.

26. An apparatus for sensing and processing images comprising an image sensor including:

a two-dimensional array of photoreceptive elements formed by variable sensitivity photo detectors having one terminal grounded via a capacitor and connected to an output terminal via a switching gate, and another terminal directly connected to a control terminal, and having a conversion efficiency from light to current which is a monotone, smoothly increasing function which varies depending upon magnitude of voltage across said two terminals, wherein the conversion efficiency can assume both positive and negative values, and wherein said control terminal is connected to a common control electrode for each row, and said output terminal is connected to a common output electrode for each column;

a control circuit having a terminal connected to said control electrode so as to output any voltage; and

an output circuit having a mechanism for applying current from any one of said output electrodes.

27. An apparatus for sensing and processing images comprising an image sensor including a two-dimensional array of photoreceptive elements formed by two variable sensitivity photo detectors having a conversion efficiency from light to current which is a monotone, smoothly increasing function which varies depending upon magnitude of voltage across two terminals, wherein the conversion efficiency can assume both positive and negative values, and wherein one terminal of one photo detector is connected to a common control electrode for each row, the other terminal thereof is connected to a common output electrode for each column, and one terminal of the other photo detector is connected to a common electrode for each column, and the other terminal thereof is connected to a common output terminal for each row.

28. An apparatus for sensing and processing images comprising an image sensor including a two-dimensional array of variable sensitivity photo detectors having a conversion efficiency from light to current which is a monotone, smoothly increasing function which varies depending upon magnitude of voltage across two terminals so as to input light signal from both sides of a surface formed by said array, wherein the conversion efficiency can assume both positive and negative values.

29. An apparatus for sensing and processing images comprising an image sensor including:

a two-dimensional array of variable sensitivity photo detectors having a conversion efficiency from light to current which is a monotone, smoothly increasing function which varies depending upon magnitude of voltage across two terminals, wherein the conversion efficiency can assume both positive and negative values, and wherein one terminal is connected to a common control electrode for each row, and the other terminal is connected to a common output electrode for each column;

a control circuit optionally varying a position of a voltage vector having voltage other than zero volts with respect to voltage vectors in which only one voltage vector has a voltage other than zero volts and other voltage vectors are all equal to zero volts; and

a mechanism for connecting said voltage to said control electrode.