Method and apparatus for generating information input using reflected light image of target object

• US 6,144,366 A
• Filed: 10/17/1997
• Issued: 11/07/2000
• Est. Priority Date: 10/18/1996
• Status: Expired due to Term

• Current Assignee: PixArt Imaging Incorporated
• Original Assignee: Kabushiki Kaisha Toshiba (Toshiba Corporation)
• Inventors: Miura, Hiroki, Umeki, Naoko, Morishita, Akira, Numazaki, Shunichi, Doi, Miwako
• Primary Examiner(s): Cuchlinski, Jr., William A.
• Assistant Examiner(s): Marc-Coleman, Marthe Y.
• Application Number:
  US08/953,667
• Time in Patent Office: 1,117 Days
• Field of Search: 345/158, 345/157, 345/156, 345/419, 345/425, 463/31, 463/32, 463/33, 463/36, 463/37, 250/200, 250/206.1, 250/208.1, 250/214 R
• US Class Current: 345/156
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042

Abstract

A method and an apparatus for generating information input which are capable of realizing a direct command type information input scheme by which the gesture or the motion can be inputted easily. The apparatus has a timing signal generation unit for generating a timing signal; a lighting unit for emitting a light whose intensity vary in time, according to the timing signal generated by the timing signal generation unit; and a reflected light extraction unit having a sensor array for detecting a reflected light from a target object resulting from the light emitted by the lighting unit, in synchronization with the timing signal generated by the timing signal generation unit, so as to obtain a spatial intensity distribution of the reflected light in a form of a reflected light image indicative of an information input related to the target object, in separation from an external light that is illuminating the target object.

Claims

1. An information input generation apparatus, comprising:

a timing signal generation unit for generating a timing signal;

a lighting unit for emitting a light whose intensity vary in time, according to the timing signal generated by the timing signal generation unit; and

a reflected light extraction unit having a sensor array for detecting a reflected light from a target object resulting from the light emitted by the lighting unit, in synchronization with the timing signal generated by the timing signal generation unit, so as to obtain a spatial intensity distribution of the reflected light in a form of a reflected light image indicative of an information input related to the target object, in separation from an external light that is illuminating the target object.

2. The apparatus of claim 1, wherein the reflected light extraction unit obtains the reflected light image in separation from the external light by detecting a plurality of image data in a state of having a light emission by the lighting unit and in a state of having no light emission by the lighting unit.

3. The apparatus of claim 2, wherein the reflected light extraction unit obtains the reflected light image as a difference between the image data detected in a state of having a light emission by the lighting unit and the image data detected in a state of having no light emission by the lighting unit.

4. The apparatus of claim 1, wherein the lighting unit emits the light which is invisible to human eyes.

5. The apparatus of claim 1, wherein the reflected light extraction unit includes:

at least one photo-electric conversion unit for generating charges corresponding to input lights of the reflected light extraction unit;

at least two charge storage units for storing the charges generated by the photo-electric conversion unit under different lighting states of the lighting unit; and

gates operated in synchronization with the timing signal generated by the timing signal generation unit, for selectively leading the charges generated by the photo-electric conversion unit to one of the charge storage units.

6. The apparatus of claim 1, wherein the reflected light extraction unit includes:

an optical filter for receiving input lights of the reflected light extraction unit, and passing those input lights which have wavelengths within a specified range and blocking those input lights which have wavelengths outside the specified range.

7. The apparatus of claim 1, wherein the target object is at least a part of a body of an operator, and the reflected light image is indicative of the information input for controlling a device to be operated by the operator.

8. The apparatus of claim 1, further comprising:

a feature data generation unit for generating feature data related to the target object from the reflected light image.

9. The apparatus of claim 8, wherein the feature data generation unit generates the feature data by extracting a distance information including a depth information of the target object.

10. The apparatus of claim 8, wherein the feature data generation unit generates the feature data by extracting an object property information including a color information or a material information of the target object.

11. The apparatus of claim 8, wherein the feature data generation unit includes:

an image extraction unit for extracting an object image of at least a part of the target object, from the reflected light image.

12. The apparatus of claim 11, wherein the image extraction unit extracts the object image according to one or both of a distance information including a depth information of the target object and an object property information including a color information or a material information of the target object.

13. The apparatus of claim 11, wherein the target object is at least a part of a body of an operator, and the object image extracted by the image extraction unit indicates the information input for a pointing operation by the operator.

14. The apparatus of claim 11, wherein the feature data generation unit also includes:

a shape interpretation unit for interpreting a shape of said at least a part of the target object in the object image extracted by the image extraction unit.

15. The apparatus of claim 14, wherein the target object is at least a part of a body of an operator, and the shape interpreted by the shape interpretation unit indicates the information input for a gesture input by the operator.

16. The apparatus of claim 14, wherein the target object is at least a part of a body of an operator, and the shape interpreted by the shape interpretation unit indicates the information input for a pointing operation by the operator.

17. The apparatus of claim 14, wherein the shape interpretation unit interprets the shape as a command to be used as the information input.

18. The apparatus of claim 14, wherein the shape interpretation unit interprets the shape as a viewpoint information to be used as the information input, the viewpoint information indicating a viewpoint from which three-dimensional objects are to be displayed on a display device.

19. The apparatus of claim 11, wherein the feature data generation unit also includes:

a motion extraction unit for extracting a change of said at least a part of the target object in the object image extracted by the image extraction unit.

20. The apparatus of claim 19, wherein the motion extraction unit extracts the change as a command to be used as the information input.

21. The apparatus of claim 19, wherein the target object is at least a part of a body of an operator, and the change extracted by the motion extraction unit indicates the information input for a motion input by the operator.

22. The apparatus of claim 11, wherein the feature data generation unit also includes:

a shape interpretation unit for interpreting a shape of said at least a part of the target object in the object image extracted by the image extraction unit;

a motion extraction unit for extracting a change of said at least a part of the target object in the object image extracted by the image extraction unit; and

a command generation unit for generating at least one command to be used as the information input from the shape interpreted by the shape interpretation unit and the change extracted by the motion extraction unit.

23. The apparatus of claim 1, further comprising:

an imaging unit for imaging a visible light image of the target object.

24. The apparatus of claim 23, further comprising:

an image extraction unit for extracting at least one specific image region from the visible light image imaged by the imaging unit, according to one or both of the reflected light image and an information extracted from the reflected light image.

25. The apparatus of claim 24, further comprising:

an image memory unit for storing an extracted image data for said at least one specific image region extracted by the image extraction unit; and

an image composition unit for composing the extracted image data stored by the image memory unit with another image data in a specified manner.

26. The apparatus of claim 23, further comprising:

a Z-value image generation unit for generating a Z-value image by setting the visible light image imaged by the imaging unit in correspondence to a distance information of the target object which Is extracted from the reflected light image.

27. The apparatus of claim 26, further comprising:

an image extraction unit for extracting image data for a specific image region from the the Z-value image generated by the Z-value image generation unit, according to the distance information contained in the Z-value image, the specific image region being a region for which the distance information is within a specified range.

28. The apparatus of claim 26, wherein the Z-value image generation unit generates the Z-value image as the information input for texture information to be mapped onto a three-dimensional model.

29. The apparatus of claim 1, further comprising:

a conversion unit for converting intensities of the reflected light defining pixel values of the reflected light image into distance values with respect to the target object so as to obtain a range image of the target object.

30. The apparatus of claim 29, wherein the conversion unit is a logarithmic amplifier.

31. The apparatus of claim 1, further comprising:

a correction unit for correcting a distortion of the reflected light image obtained by the reflected light extraction unit.

32. The apparatus of claim 1, further comprising:

a correction and conversion unit for correcting a distortion of the reflected light image obtained by the reflected light extraction unit while converting intensities of the reflected light defining pixel values of the reflected light image into distance values with respect to the target object so as to obtain a range image of the target object.

33. The apparatus of claim 1, further comprising:

a nonlinear conversion unit for converting intensities of the reflected light defining pixel values of the reflected light image into rough distance values with respect to the target object; and

a correction unit for converting the rough distance values obtained by the nonlinear conversion unit into accurate distance values with respect to the target object so as to obtain a range image of the target object.

34. The apparatus of claim 1, further comprising:

a reference object activation unit for moving a reference object with respect to the reflected light extraction unit so that the reflected light extraction unit obtains the reflected light image of the reference object; and

a correction data generation unit for generating correction data to be used in correcting a distortion of the reflected light image and/or generating a range image from the reflected light image, according to a position of the reference object moved by the reference object activation unit and the reflected light image of the reference object obtained by the reflected light extraction unit.

35. The apparatus of claim 1, further comprising:

a user commanding unit for commanding a user to place a reference object at a specified position with respect to the reflected light extraction unit so that the reflected light extraction unit obtains the reflected light image of the reference object; and

a correction data generation unit for generating correction data to be used in correcting a distortion of the reflected light image and/or generating a range image from the reflected light image, according to a position of the reference object placed by the user in response to the user commanding unit and the reflected light image of the reference object obtained by the reflected light extraction unit.

36. The apparatus of claim 35, further comprising:

a verification unit for checking whether the correction data generated by the correction data generation unit are correct or not by obtaining the reflected light image of the reference object within a prescribed distance range by using the correction data.

37. The apparatus of claim 1, further comprising:

a filter for receiving the external light and the reflected light, and blocking the reflected light while passing the external light;

an external light detection unit for detecting a light amount of the external light received through the filter; and

an external light state judgement unit for judging whether the external light is in an intolerable state that has a possibility of largely affecting the reflected light image or not according to the light amount detected by the external light detection unit, and generating an intolerable signal indicating the intolerable state when the external light is judged to be in the intolerable state.

38. The apparatus of claim 37, wherein the timing signal generation unit controls the lighting unit and the reflected light extraction unit according to the intolerable signal generated by the external light state judgement unit, so as to obtain the reflected light image while the external light is not in the intolerable state.

39. The apparatus of claim 37, further comprising:

a reflected light image processing unit for determining whether to accept or reject the reflected light image obtained by the reflected light extraction unit under the external light in the intolerable state, according to the intolerable signal generated by the external light state judgement unit.

40. The apparatus of claim 1, further comprising:

a filter for receiving the external light and the reflected light, and blocking the reflected light while passing the external light;

an external light detection unit for detecting a light amount of the external light received through the filter; and

an external light fluctuation period detection unit for detecting a fluctuation period of the external light according to the light amount detected by the external light detection unit;

wherein the timing signal generation unit controls the lighting unit and the reflected light extraction unit in synchronization with the fluctuation period detected by the external light fluctuation period detection unit.

41. The apparatus of claim 1, further comprising:

an external light control unit for controlling the external light in relation to the timing signal for controlling the lighting unit and the reflected light extraction unit.

42. The apparatus of claim 1, wherein the timing signal generation unit controls the lighting unit and the reflected light extraction unit by using a plurality of operation patterns provided in advance, according to a change of the external light.

43. The apparatus of claim 42, wherein the timing signal generation unit in an optimal operation pattern selection mode controls the lighting unit to stop emitting the light while controlling the reflected light extraction unit to obtain the reflected light images by executing each of said plurality of operation patterns sequentially, and the apparatus further comprises:

an evaluation unit for generating evaluation values for the reflected light images obtained by the reflected light extraction unit in the optimal operation pattern selection mode; and

an operation pattern selection unit for selecting an optimal operation pattern among said plurality of operation patterns according to the evaluation values generated by the evaluation unit, so that the timing signal generation unit in a normal operation mode controls the lighting unit and the reflected light extraction unit according to the optimal operation pattern selected by the operation pattern selection unit.

44. The apparatus of claim 43, wherein the operation pattern selection unit selects the optimal operation pattern by which a darkest reflected light image is obtained in the optimal operation pattern selection mode.

45. The apparatus of claim 1, further comprising:

a distance information detection unit for detecting a distance information indicating a distance of the target object with respect to the reflected light extraction unit according to the reflected light image obtained by the reflected light extraction unit;

a determination unit for determining a reflected light amount to be increased or decreased according to the distance information detected by the distance information detection unit; and

a control unit for controlling the lighting unit or the reflected light extraction unit so that the reflected light amount received at the reflected light extraction unit is increased or decreased to the reflected light amount determined by the determination unit.

46. The apparatus of claim 45, wherein the lighting unit has a plurality of lighting modes with different emission light amounts, and the control unit controls the lighting unit to be operated in one of said plurality of lighting modes according to the reflected light amount determined by the determination unit.

47. The apparatus of claim 1, wherein the lighting unit has a plurality of lighting modes with different emission light amounts, and the apparatus further comprises:

a lighting state control unit for controlling the lighting unit to sequentially carry out lighting operations in said plurality of lighting modes so that the reflected light extraction unit obtains a plurality of reflected light images under the light emitted by the lighting unit in said plurality of lighting modes; and

a reflected light image composition unit for composing said plurality of reflected light images obtained by the reflected light extraction unit.

48. A method of information input generation, comprising the steps of:

(a) generating a timing signal;

(b) emitting a light whose intensity vary in time at a lighting unit, according to the timing signal generated by the step (a); and

(c) detecting a reflected light from a target object resulting from the light emitted by the step (b), in synchronization with the timing signal generated by the step (a), so as to obtain a spatial intensity distribution of the reflected light in a form of a reflected light image indicative of an information input related to the target object, in separation from an external light that is illuminating the target object, at a reflected light extraction unit.

49. The method of claim 48, wherein the step (c) obtains the reflected light image in separation from the external light by detecting a plurality of image data in a state of having a light emission by the step (b) and in a state of having no light emission by the step (b).

50. The method of claim 49, wherein the step (a) obtains the reflected light image as a difference between the image data detected in a state of having a light emission by the step (b) and the image data detected in a state of having no light emission by the step (b).

51. The method of claim 48, wherein the step (b) emits the light which is invisible to human eyes.

52. The method of claim 48, wherein the step (c) includes steps of:

(c1) generating charges corresponding to input lights of the reflected light extraction unit;

(c2) storing the charges generated by the step (c1) under different lighting states of the step (b), in at least two charge storage units; and

(c3) operating gates in synchronization with the timing signal generated by the step (a), for selectively leading the charges generated by the step (c1) to one of the charge storage units.

53. The method of claim 48, wherein the step (c) includes the step of receiving input lights of the reflected light extraction unit by an optical filter for passing those input lights which have wavelengths within a specified range and blocking those input lights which have wavelengths outside the specified range.

54. The method of claim 48, wherein the target object is at least a part of a body of an operator, and the reflected light image is indicative of the information input for controlling a device to be operated by the operator.

55. The method of claim 48, further comprising the steps of:

(d) generating feature data related to the target object from the reflected light image.

56. The method of claim 55, wherein the step (d) generates the feature data by extracting a distance information including a depth information of the target object.

57. The method of claim 55, wherein the step (d) generates the feature data by extracting an object property information including a color information or a material information of the target object.

58. The method of claim 55, wherein the step (d) includes the step of:

(d1) extracting an object image of at least a part of the target object, from the reflected light image.

59. The method of claim 58, wherein the step (d1) extracts the object image according to one or both of a distance information including a depth information of the target object and an object property information including a color information or a material information of the target object.

60. The method of claim 58, wherein the target object is at least a part of a body of an operator, and the object image extracted by the step (d1) indicates the information input for a pointing operation by the operator.

61. The method of claim 58, wherein the step (d) also includes the step of:

(d2) interpreting a shape of said at least a part of the target object in the object image extracted by the step (d1).

62. The method of claim 61, wherein the target object is at least a part of a body of an operator, and the shape interpreted by the step (d2) indicates the information input for a gesture input by the operator.

63. The method of claim 61, wherein the target object is at least a part of a body of an operator, and the shape interpreted by the step (d2) indicates the information input for a pointing operation by the operator.

64. The method of claim 61, wherein the step (d2) interprets the shape as a command to be used as the information input.

65. The method of claim 61, wherein the step (d2) interprets the shape as a viewpoint information to be used as the information input, the viewpoint information indicating a viewpoint from which three-dimensional objects are to be displayed on a display device.

66. The method of claim 58, wherein the step (d) also includes the step of:

(d3) extracting a change of said at least a part of the target object in the object image extracted by the step (d1).

67. The method of claim 66, wherein the step (d3) extracts the change as a command to be used as the information input.

68. The method of claim 66, wherein the target object is at least a part of a body of an operator, and the change extracted by the step (d3) indicates the information input for a motion input by the operator.

69. The method of claim 58, wherein the step (d) also includes:

(d4) interpreting a shape of said at least a part of the target object in the object image extracted by the step (d1);

(d5) extracting a change of said at least a part of the target object in the object image extracted by the step (d1); and

(d6) generating at least one command to be used as the information input from the shape interpreted by the step (d4) and the change extracted by the step (d5).

70. The method of claim 48, further comprising the step of:

(e) imaging a visible light image of the target object.

71. The method of claim 70, further comprising the step of:

(f) extracting at least one specific image region from the visible light image imaged by the step (e), according to one or both of the reflected light image and an information extracted from the reflected light image.

72. The method of claim 71, further comprising the steps of:

(g) storing an extracted image data for said at least one specific image region extracted by the step (f); and

(h) composing the extracted image data stored by the step (g) with another image data in a specified manner.

73. The method of claim 70, further comprising the step of:

(i) generating a Z-value image by setting the visible light image imaged by the step (e) in correspondence to a distance information of the target object which is extracted from the reflected light image.

74. The method of claim 73, further comprising the step of:

(j) extracting image data for a specific image region from the the Z-value image generated by the step (i), according to the distance information contained in the Z-value image, the specific image region being a region for which the distance information is within a specified range.

75. The method of claim 73, wherein the step (i) generates the Z-value image as the information input for texture information to be mapped onto a three-dimensional model.

76. The method of claim 48, further comprising the steps of:

(k) converting intensities of the reflected light defining pixel values of the reflected light image into distance values with respect to the target object so as to obtain a range image of the target object.

77. The method of claim 76, wherein the step (k) converts the intensities by using a logarithmic amplifier.

78. The method of claim 48, further comprising the steps of:

(l) correcting a distortion of the reflected light image obtained by the reflected light extraction unit.

79. The method of claim 48, further comprising the step of:

(m) correcting a distortion of the reflected light image obtained by the reflected light extraction unit while converting intensities of the reflected light defining pixel values of the reflected light image into distance values with respect to the target object so as to obtain a range image of the target object.

80. The method of claim 48, further comprising the steps of:

(n) nonlinearly converting intensities of the reflected light defining pixel values of the reflected light image into rough distance values with respect to the target object; and

(o) converting the rough distance values obtained by the step (n) into accurate distance values with respect to the target object so as to obtain a range image of the target object.

81. The method of claim 48, further comprising the steps of:

(p) moving a reference object with respect to the reflected light extraction unit so that the reflected light extraction unit obtains the reflected light image of the reference object; and

(q) generating correction data to be used in correcting a distortion of the reflected light image and/or generating a range image from the reflected light image, according to a position of the reference object moved by the step (p) and the reflected light image of the reference object obtained by the reflected light extraction unit.

82. The method of claim 48, further comprising the step of:

(r) commanding a user to place a reference object at a specified position with respect to the reflected light extraction unit so that the reflected light extraction unit obtains the reflected light image of the reference object; and

(s) generating correction data to be used in correcting a distortion of the reflected light image and generating a range image from the reflected light image, according to a position of the reference object placed by the user in response to the step (r) and the reflected light image of the reference object obtained by the reflected light extraction unit.

83. The method of claim 82, further comprising the step of:

(t) checking whether the correction data generated by the step (s) are correct or not by obtaining the reflected light image of the reference object within a prescribed distance range by using the correction data.

84. The method of claim 48, further comprising the steps of:

(u) receiving the external light and the reflected light, and blocking the reflected light while passing the external light, at a filter;

(v) detecting a light amount of the external light received through the filter; and

(w) judging whether the external light is in an intolerable state that has a possibility of largely affecting the reflected light image or not according to the light amount detected by the step (v), and generating an intolerable signal indicating the intolerable state when the external light is judged to be in the intolerable state.

85. The method of claim 84, wherein the timing signal generated by the step (a) controls the lighting unit and the reflected light extraction unit according to the intolerable signal generated by the step (w), so as to obtain the reflected light image while the external light is not in the intolerable state.

86. The method of claim 84, further comprising the step of:

(x) determining whether to accept or reject the reflected light image obtained by the reflected light extraction unit under the external light in the intolerable state, according to the intolerable signal generated by the step (w).

87. The method of claim 48, further comprising the steps of:

(y) receiving the external light and the reflected light, and blocking the reflected light while passing the external light, at a filter;

(z) detecting a light amount of the external light received through the filter; and

(aa) detecting a fluctuation period of the external light according to the light amount detected by the step (z);

wherein the timing signal generated by the step (a) controls the lighting unit and the reflected light extraction unit in synchronization with the fluctuation period detected by the step (aa).

88. The method of claim 48, further comprising the steps of:

(bb) controlling the external light in relation to the timing signal for controlling the lighting unit and the reflected light extraction unit.

89. The method of claim 48, wherein the timing signal generated by the step (a) controls the lighting unit and the reflected light extraction unit by using a plurality of operation patterns provided in advance, according to a change of the external light.

90. The method of claim 89, wherein the timing signal generated by the step (a) in an optimal operation pattern selection mode controls the lighting unit to stop emitting the light while controlling the reflected light extraction unit to obtain the reflected light images by executing each of said plurality of operation patterns sequentially, and the method further comprises the steps of:

(cc) generating evaluation values for the reflected light images obtained by the reflected light extraction unit in the optimal operation pattern selection mode; and

(dd) selecting an optimal operation pattern among said plurality of operation patterns according to the evaluation values generated by the step (cc), so that the timing signal generated by the step (a) in a normal operation mode controls the lighting unit and the reflected light extraction unit according to the optimal operation pattern selected by the step (dd).

91. The method of claim 90, wherein the step (dd) selects the optimal operation pattern by which a darkest reflected light image is obtained in the optimal operation pattern selection mode.

92. The method of claim 48, further comprising the steps of:

(ee) detecting a distance information indicating a distance of the target object with respect to the reflected light extraction unit according to the reflected light image obtained by the reflected light extraction unit;

(ff) determining a reflected light amount to be increased or decreased according to the distance information detected by the step (ee); and

(gg) controlling the lighting unit or the reflected light extraction unit so that the reflected light amount received at the reflected light extraction unit is increased or decreased to the reflected light amount determined by the step (ff).

93. The method of claim 92, wherein the lighting unit has a plurality of lighting modes with different emission light amounts, and the step (gg) controls the lighting unit to be operated in one of said plurality of lighting modes according to the reflected light amount determined by the step (ff).

94. The method of claim 48, wherein the lighting unit has a plurality of lighting modes with different emission light amounts, and the method further comprises the steps of:

(hh) controlling the lighting unit to sequentially carry out lighting operations in said plurality of lighting modes so that the reflected light extraction unit obtains a plurality of reflected light images under the light emitted by the lighting unit in said plurality of lighting modes; and

(ii) composing said plurality of reflected light images obtained by the reflected light extraction unit.

95. An imaging apparatus, comprising:

a plurality of unit photo-detector cells arranged in a two-dimensional array on a semiconductor substrate, each unit photo-detector cell having;

a photo-detection unit for receiving an input light and outputting signals corresponding to a light amount of the input light;

an amplification unit for amplifying the signals outputted by the photo-detection unit;

a selection unit for selecting said each unit photo-detector cell;

a reset unit for resetting the photo-detection unit; and

at least two storage units for respectively storing the signals outputted by the photo-detection unit under different states and amplified by the amplification unit; and

a difference circuit, provided on the semiconductor substrate, for detecting a difference between the signals stored in said at least two storage units of each unit photo-detector cell.

96. The apparatus of claim 95, wherein the photo-detection unit is formed by a photo-diode, the amplification unit is formed by an amplifying transistor, the resent unit is formed by a reset transistor, said at least two storage units are formed by capacitors, said photo-diode is connected with a gate of said amplifying transistor, and each of said capacitors has one electrode which is connected with the amplifying transistor and another electrode which is grounded.