COMPUTER DISPLAY DEVICE MOUNTED ON EYEGLASSES

• US 20160025983A1
• Filed: 07/27/2015
• Published: 01/28/2016
• Est. Priority Date: 07/25/2014
• Status: Active Grant

• Current Assignee: Hiroyuki Ikeda
• Original Assignee: Hiroyuki Ikeda
• Inventors: Ikeda, Hiroyuki
• Granted Patent: US 10,191,282 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/0426 tracking fingers with respect to a virtual keyboard projected or printed on the surface virtual keyboards on touch screens G06F3/04886
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus

Abstract

When an imaging apparatus (30) captures an image of a finger performing an operation on a visual confirmation screen (S2), an operation determining unit (73) of a computer display device (1) mounted on eyeglasses (10) determines the contents of the operation performed by the finger based on image data obtained by the imaging apparatus (30). A positional data generating unit (74) generates positional data of the finger based on the image data. Based on data concerning the determined contents of the operation performed by the finger, the generated positional data of the finger, and reference data concerning the visual confirmation screen stored in a storage unit (90), an input control unit (76) determines the contents of an inputted instruction corresponding to the operation performed by the finger, and controls an original screen to be displayed on an display apparatus (20) in accordance with the determined contents of the inputted instruction.

Claims

1. A device configured to be worn like eyeglasses by a user, the device comprising:

an eyeglass main body;

a display apparatus provided on the eyeglass main body and configured to display an original screen corresponding to a visual confirmation screen that appears to the user to be floating in midair;

an imaging apparatus provided on the eyeglass main body and configured such that, when the user performs an operation on the visual confirmation screen using a finger or an input instructing tool, the imaging apparatus captures one or more images of the finger or the input instructing tool that has performed the operation and generates image data;

an operation determining unit configured to determine, when the imaging apparatus has captured the image of the finger or of the input instructing tool used by the user to perform the operation on the visual confirmation screen, contents of the operation performed by the finger or the input instructing tool, based on the image data;

a positional data generating unit configured to generate, when the imaging apparatus has captured the image of the finger or of the input instructing tool used by the user to perform the operation on the visual confirmation screen, positional data of the finger or of the input instructing tool in an imaging range that is a range in which the imaging apparatus can capture an image based on the image data obtained by the imaging;

a reference data generating unit configured to generate, when the user performs a reference data setting operation using a finger or the input instructing tool at one or more predetermined positions on the visual confirmation screen, data concerning the visual confirmation screen using the positional data of the finger or of the input instructing tool generated by the positional data generating unit based on the image data for which the operation determining unit has determined that the operation at each of the predetermined positions is a prescribed operation, and to store the generated data as reference data in a storage unit; and

an input control unit configured to determine, when the user performs the operation using the finger or the input instructing tool on the visual confirmation screen, contents of an inputted instruction, which corresponds to the operation performed by the finger or the input instructing tool, based on;

(i) data concerning the contents of the operation performed by the finger or the input instructing tool determined by the operation determining unit, (ii) the positional data of the finger or of the input instructing tool generated by the positional data generating unit, and (iii) the reference data concerning the visual confirmation screen that is stored in the storage unit, and to control the original screen to be displayed on the display apparatus in accordance with the determined contents of the inputted instruction.

2. The device according to claim 1, wherein:

the imaging apparatus is configured to bring into focus a subject at a position of the visual confirmation screen recognized by the user,

the device further comprises an image extracting unit configured to determine, when an image of the subject in focus is captured by the imaging apparatus, whether or not the subject is the finger or the input instructing tool based on the image data and to extract a subset of the image data in which the finger or the input instructing tool is present,

the operation determining unit is configured to determine, based on the subset of the image data extracted by the image extracting unit, the contents of the operation performed by the finger or the input instructing tool that is present in the subset of the image data, and

the positional data generating unit is configured to generate, based on the subset of the image data for which the operation determining unit has determined the operation performed by the finger or the input instructing tool is a prescribed operation, the positional data of the finger or of the input instructing tool in the imaging range.

3. The device according to claim 1, wherein:

the imaging apparatus includes an autofocus control unit configured to;

(i) automatically bring a subject into focus, (ii) calculate, when an image of the subject that has been automatically brought into focus by the autofocus control unit is captured, data concerning the distance to the captured subject and (iii) output the calculated distance data together with the image data,

the device further comprises an image data extracting unit configured to;

(i) determine, when the image of the subject that has been automatically brought into focus is captured by the imaging apparatus, whether or not the subject is the finger or the input instructing tool based on the image data, (ii) determine whether or not the subject is spaced from the imaging apparatus in a depth direction by a predetermined, approximately constant distance based on the distance data sent together with the image data, and (iii) extract a subset of the image data in which the finger or the input instructing tool is present and the finger or the input instructing tool is spaced from the imaging apparatus in the depth direction by an approximately constant distance,

the operation determining unit is configured to determine, based on the subset of the image data extracted by the image extracting unit, the contents of the operation performed by the finger or the input instructing tool that is present in the image data, and

the positional data generating unit is configured to generate, based on the subset of the image data for which the operation determining unit has determined the operation performed by the finger or the input instructing tool is the prescribed operation, the positional data of the finger or of the input instructing tool in the imaging range.

4. The device according to claim 1, wherein

the original screen contains a predetermined mark associated with at least one of four corners displayed on the display apparatus and

the reference data generating unit is configured to generate, when the user performs the operation using the finger or the input instructing tool on the mark on the visual confirmation screen corresponding to the original screen, the reference data concerning the visual confirmation screen using the positional data of the finger or of the input instructing tool generated by the positional data generating unit.

6. The device according to claim 1, wherein the original screen displayed on the display apparatus is a character input screen including a keyboard image and a display area for displaying inputted characters.

7. The device according to claim 1, wherein

the original screen displayed on the display apparatus includes a screen corresponding to an operating unit of a remote controller for a remotely-controllable apparatus, and

the device further comprises a remote control unit configured to generate, when an operation is performed on the screen corresponding to the operating unit of the remote controller, a command signal representative of the contents of the operation and to wirelessly transmit the generated command signal to the remotely-controllable apparatus.

8. The device according to claim 1, wherein

the imaging apparatus is configured to limit an in-focus range to a constant range, and

the device further comprises notifying means for notifying a third party of the fact that the in-focus range is being limited to the constant range.

9. The device according to claim 1, wherein

the display apparatus includes a projector having a display device, an optical system, and a half mirror onto which the original screen displayed on the display device is projected via the optical system, and

the half mirror is embedded in or integrated with a prism arranged in front of a lens section of the eyeglass main body or is affixed to a prism arranged in front of the lens section of the eyeglass main body.

10. The device according to claim 1, wherein

the display apparatus includes a projector having a display device, an optical system, and a half mirror onto which the original screen displayed on the display device is projected via the optical system, and

the half mirror is affixed to a lens section of the eyeglass main body or is embedded in or integrated with the lens section of the eyeglass main body.

11. The device according to claim 1, wherein

the display apparatus includes a projector having a display device, an optical system, and a holographic sheet or a holographic film onto which the original screen displayed on the display device is projected via the optical system, and

the holographic sheet or the holographic film is affixed to a lens section of the eyeglass main body or is embedded in or integrated with the lens section of the eyeglass main body.

12. The device according to claim 1, wherein

the operation determining unit is configured to determine contents of an operation performed by a finger based on movement of a predetermined mark attached to a fingertip of the finger, and

the positional data generating unit is configured to generate positional data of the mark as the positional data of the finger.

13. The device according to claim 1, wherein the device is configured to be used by the user while being mounted on any portion of the head other than the ears instead of being worn like eyeglasses.

14. The device according to claim 1, wherein the eyeglass main body comprises a frame and two temples, the display apparatus being mounted on the frame and the imaging apparatu s being mounted on one of the temples.

5. A device configured to be like eyeglasses by a user, the device comprising:

an eyeglass main body;

a display apparatus provided on the eyeglass main body and configured to display an original screen corresponding to a visual confirmation screen that appears to a user to be floating in midair;

an imaging apparatus provided on the eyeglass main body, and including an autofocus control unit configured to automatically bring a subject into focus, the imaging apparatus being configured to calculate, when an image of the subject that has been automatically brought into focus by the autofocus control unit is captured, data concerning the distance to the captured subject and to output the calculated distance data together with image data obtained by the imaging;

an image data extracting unit configured to;

(i) determine, when the image of the subject that has been automatically brought into focus is captured by the imaging apparatus, whether or not the subject is a finger or an input instructing tool based on the image data and to extract a subset of the image data in which the finger or the input instructing tool is present;

an operation determining unit configured to determine, when the imaging apparatus captures the image of the finger or the input instructing tool used by the user to perform the operation on the visual confirmation screen, contents of the operation performed by the finger or the input instructing tool, based on the subset of the image data extracted by the image data extracting unit;

a positional data generating unit configured to generate, when the imaging apparatus captures the image of the finger or of the input instructing tool used by the user to perform the operation on the visual confirmation screen, positional data of the finger or of the input instructing tool in an imaging range of the imaging apparatus e, based on the image data for which the operation determining unit has determined the operation performed by the finger or the input instructing tool is a prescribed operation;

a reference data generating unit configured to generate, when the user performs an operation using the finger or the input instructing tool at at least three predetermined positions on the visual confirmation screen, data specifying a reference screen that corresponds to the visual confirmation screen in a three-dimensional space using (i) at least three pieces of positional data of the finger or of the input instructing tool generated by the positional data generating unit based on the image data for which the operation determining unit has determined the operation at each of the prescribed positions is the prescribed operation, and (ii) distance data sent together with the image data used when generating the pieces of positional data, and to store the generated data specifying the reference screen as reference data in a storage unit;

a distance determining unit configured to determine, when the image of the subject that has been automatically brought into focus is captured by the imaging apparatus upon the user performing an operation on the visual confirmation screen using a finger or an input instructing tool after the reference data has been generated by the reference data generating unit, whether or not the finger or the input instructing tool is positioned within a range of a predetermined, approximately-constant distance from the reference screen based on;

(i) the positional data of the finger or of the input instructing tool generated by the positional data generating unit based on the image data for which the operation determining unit has determined the operation performed by the finger or the input instructing tool is the prescribed operation, (ii) the distance data associated with the image data used when generating the positional data of the finger or of the input instructing tool, and (iii) the reference data that specifies the reference screen corresponding to the visual confirmation screen and which is stored in the storage unit; and

an input control unit configured to determine, when the user performs the operation using the finger or the input instructing tool on the visual confirmation screen and the distance determining unit has determined that the finger or the input instructing tool is located within the range of the predetermined, approximately-constant distance from the reference screen, contents of an inputted instruction corresponding to the operation performed by the finger or the input instructing tool based on;

(i) the data concerning the contents of the operation performed by the finger or the input instructing tool determined by the operation determining unit, (ii) the positional data of the finger or of the input instructing tool generated by the positional data generating unit based on the image data used to make the determination, (iii) the distance data associated with the image data used to make the determination, and (iv) the reference data that specifies the reference screen corresponding to the visual confirmation screen and which is stored in the storage unit, and to control the original screen to be displayed on the display apparatus in accordance with the determined contents of the inputted instruction.

15. The device according to claim 5, wherein the original screen displayed on the display apparatus is a character input screen including a keyboard image and a display area for displaying inputted characters.

16. The device according to claim 5, wherein

the original screen displayed on the display apparatus includes a screen corresponding to an operating unit of a remote controller for a remotely-controllable apparatus, and

the device further comprises a remote control unit configured to generate, when an operation is performed on the screen corresponding to the operating unit of the remote controller, a command signal representative of the contents of the operation and to wirelessly transmit the generated command signal to the remotely-controllable apparatus.

17. The device according to claim 5, wherein

the imaging apparatus is configured to limit an in-focus range to a constant range, and

the device further comprises notifying means for notifying a third party of the fact that the in-focus range is being limited to the constant range.

18. The device according to claim 5, wherein

the display apparatus includes a projector having a display device, an optical system, and a half mirror onto which the original screen displayed on the display device is projected via the optical system, and

the half mirror is embedded in or integrated with a prism arranged in front of a lens section of the eyeglass main body or is affixed to a prism arranged in front of the lens section of the eyeglass main body.

19. The device according to claim 5, wherein

the display apparatus includes a projector having a display device, an optical system, and a half mirror onto which the original screen displayed on the display device is projected via the optical system, and

the half mirror is affixed to a lens section of the eyeglass main body or is embedded in or integrated with the lens section of the eyeglass main body.

20. The device according to claim 5, wherein

the display apparatus includes a projector having a display device, an optical system, and a holographic sheet or a holographic film onto which the original screen displayed on the display device is projected via the optical system, and

the holographic sheet or the holographic film is affixed to a lens section of the eyeglass main body or is embedded in or integrated with the lens section of the eyeglass main body.

21. The device according to claim 5, wherein

the operation determining unit is configured to determine contents of an operation performed by a finger based on movement of a predetermined mark attached to a fingertip of the finger, and

the positional data generating unit is configured to generate positional data of the mark as the positional data of the finger.

22. The device according to claim 5, wherein the device is configured to be used by the user while being mounted on any portion of the head other than the ears instead of being worn like eyeglasses.

23. The device according to claim 5, wherein the eyeglass main body comprises a frame and two temples, the display apparatus being mounted on the frame and the imaging apparatus being mounted on one of the temples.