User interface device

• US 6,911,972 B2
• Filed: 03/29/2002
• Issued: 06/28/2005
• Est. Priority Date: 04/04/2001
• Status: Expired due to Fees

• Current Assignee: Matsushita Electric Industrial Company Limited (Panasonic Holdings Corporation)
• Original Assignee: Matsushita Electric Industrial Company Limited (Panasonic Holdings Corporation)
• Inventors: Brinjes, Jonathan Charles
• Primary Examiner(s): Shalwala, Bipin
• Assistant Examiner(s): Kovalick, Vincent E.
• Application Number:
  US10/109,064
• Publication Number:
  US 20020171633A1
• Time in Patent Office: 1,187 Days
• Field of Search: 345/158, 345/175, 345/178, 178/18.01, 178/18.09, 178/18.1, 178/18.11, 348/47, 348/94, 356/375, 382/103, 382/154, 382/291
• US Class Current: 345/175
• CPC Class Codes: G05B 2219/32014 Augmented reality assists operator in maintenance, repair, programming, assembly, use of head mounted display with 2-D 3-D display and voice feedback, voice and gesture command
• CPC Class Codes: G06F 1/1626 with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 1/1673 Arrangements for projecting a virtual keyboard
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 3/0426 tracking fingers with respect to a virtual keyboard projected or printed on the surface virtual keyboards on touch screens G06F3/04886
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06V 30/228 of three-dimensional handwriting, e.g. writing in the air
• CPC Class Codes: H04M 1/23 Construction or mounting of dials or of equivalent devices; Means for facilitating the use thereof by improving visibility H04M1/22
• CPC Class Codes: H04M 2250/12 including a sensor for measuring a physical value, e.g. temperature or motion telephones including GPS signal receivers H04M2250/10
• CPC Class Codes: H04M 2250/70 methods for entering alphabetical characters, e.g. multi-tap or dictionary disambiguation methods for entering alphabetical characters per se G06F3/0237

Abstract

There is a described a user interface device having first and second imaging means for forming first and second images of a three-dimensional zone from two different viewpoints, and image data generating means for generating image data corresponding to the first and second images. A processor determines the position of an object within the three-dimensional zone from the image data for the first and second images, associates the determined position with corresponding information and generates a data signal in accordance with the information corresponding to the determined position. In an embodiment, the user interface device is in the form of a virtual keyboard. In another embodiment, the user interface device is in the form of a tablet for handwriting recognition.

Claims

1. A user interface device for determining the position of one or more objects within a three dimensional zone, the device comprising:

first imaging means for forming a first two dimensional image, from a first viewpoint, of the object(s) against a first background; and

second imaging means for forming a second two dimensional image, from a second viewpoint different from the first viewpoint, of the object(s) against a second background overlapping the first background;

wherein the first and second backgrounds are two dimensional views of the whole of the three dimensional zone;

the device further comprising position determining means for determining the position of an object within the three dimensional zone from its position relative to the two backgrounds;

means for associating one or more object positions determined by the position determining means within the three dimensional zone with corresponding information; and

means for generating a signal in accordance with the information corresponding to the one or more determined object positions within the three dimensional zone.

2. A user interface device according to claim 1, wherein:

the first imaging means comprises a first optical aperture;

the second imaging means comprises a second optical aperture which is spatially separated from the first aperture; and

the image data generating means comprises an image capturing device provided in common to the first and second imaging means.

3. A user interface device according to claim 2, wherein the image data generating means further comprises:

a first shutter operable to prevent the forming of the first image on the common image capturing device by the first imaging means; and

a second shutter operable to prevent the formation of the second image on the common image capturing device by the second imaging means.

4. A user interface device according to claim 1, wherein the image data generating means comprises first and second image capturing means which are spatially separated from each other, wherein the first imaging means is operable to form the first image on the first image capturing means and the second imaging means is operable to form the second image on the second imaging capturing means.

5. A user interface device according to claim 2, wherein the first and second imaging means are provided with a common lens for forming the first and second images on the common image capturing device.

6. A user interface device according to claim 2, further comprising:

a first light guide operable to receive light transmitted through the first optical aperture; and

a second light guide operable to receive light transmitted through the second optical aperture.

7. A user interface device according to claim 6, further comprising beam directing means operable to direct light transmitted through the first and second light guides towards the common image capturing device.

8. A user interface device according to claim 7, wherein the beam directing means comprises a pentagonal prism.

9. A user interface device according to claim 2, further comprising:

a beam splitter operable to transmit a portion of incident light and reflect another portion of incident light;

first beam directing means operable to direct light transmitted through the first optical aperture so that the light from the first aperture which is transmitted by the beam splitter is directed towards the common image capturing device; and

second beam directing means for directing light transmitted through the second optical aperture to the beam splitter so that the light from the second aperture which is reflected by the beam splitter is directed towards the image capturing device;

wherein the beam splitter and the first and second beam directing means are arranged so that light from the first optical aperture transmitted by the beam splitter travels along the same optical path as light from the second optical aperture reflected by the beam splitter.

10. A user interface device according to claim 9, wherein the beam splitter and the first and second beam directing means are arranged so that the far-field portions of the first and second images substantially overlap at the image capturing device.

11. A user interface device according to claim 9, wherein the beam splitter is operable to transmit half the incident light from the first shutter and to reflect half the incident light from the second shutter.

12. A user interface device according to claim 1, including image capturing means comprising a charge-coupled device.

13. A user interface device according to claim 1, wherein the position determining means is operable to match the image data for a predetermined portion of the first image with the image data for a predetermined portion of the second image to form a common reference.

14. A user interface device according to claim 13, wherein the predetermined portion of the first image and the predetermined portion of the second image are located at a peripheral region of the first and second images.

15. A user interface device according to claim 1, wherein said position determining means is operable to determine the outline of one or more objects within the three-dimensional zone by identifying a change in contrast between the image data for the background and the image data for the object.

16. A user interface device according to claim 15, wherein the position determining means is operable to determine a reference position on the object by performing a pattern recognition operation on the determined outline.

17. A user interface device according to claim 16, wherein the position determining means is operable to recognise the pattern corresponding to a finger, and wherein said reference position corresponds a fingertip.

18. A user interface device as claimed in claim 1 wherein the position determining means is operable to determine the position of the object in the three dimensional zone from stored data calibrating object positions within the respective two dimensional images to actual positions within the three dimensional zone.

19. A user interface device according to claim 1, wherein the associating means comprises storing means storing data associating portions of the three-dimensional zone with corresponding information.

20. A user interface device according to claim 19, wherein the storing means stores data associating portions of the three-dimensional zone with corresponding keys of a keyboard.

21. A user interface device according to claim 20, wherein the user interface device further comprises means for determining whether or not the key corresponding to the position of the object is pressed from the position of the object.

22. A user interface device according to claim 21, wherein the generating means is operable to generate a data signal conveying an indication of the key corresponding to the position of the object together with an indication whether the corresponding key has been pressed.

23. A data processing apparatus comprising:

a user interface device according to claim 22; and

a display means,

wherein said display means is operable to display a representation of said keyboard and, in response to said data signal, to indicate the key corresponding to the position of the object but not pressed.

24. A user interface device according to claim 1, wherein said associating means comprises a detachable non-volatile memory for storing a look-up table.

25. A user interface device according to claim 1 wherein the associating means comprises handwriting recognition means for determining an input character corresponding to a sequence of positions of the object within the three-dimensional zone.

26. A user interface device as claimed in any preceding claim further comprising means for determining the size of an object whose position is to be determined to assist in the correct determination of its position.

27. A communications device comprising a user interface device according to claim 1.

28. A communications device according to claim 27, wherein the communications device is a mobile phone.

29. A data processing apparatus comprising:

a user interface device according to any preceding claim; and

a display means,

wherein the display means is operable to display input data in accordance with the data signal generated by the user interface device.

30. A communication device comprising a data processing apparatus according to claim 29.

31. A palmtop computer comprising a data processing apparatus according to either claim 29.

32. A method of inputting data into a data processing apparatus relating to the position of one or more objects within a three dimensional zone, the method comprising:

forming a first two dimensional image from a first viewpoint, of the object(s) against a first background;

forming a second two dimensional image from a second viewpoint different from the first viewpoint, of the object(s) against a second background which overlaps the first background, said first and second backgrounds being two dimensional views of the whole of the three dimensional zone;

generating image data corresponding to the first and second images;

determining the position of an object within the three-dimensional zone from its position relative to the two backgrounds using the image data corresponding to the first and second images;

associating one or more object positions determined in the position determining step within the three-dimensional zone with corresponding information; and

generating a signal in accordance with the information corresponding to the one or more determined object positions within the three-dimensional zone.

33. A method as claimed in claim 32 comprising storing data calibrating object positions within the respective first and second two dimensional images to actual positions within the three dimensional zone and using the stored data to determine position of the object(s).

34. A data processing method for determining information corresponding to one or more positions of an object in a three-dimensional zone, the method comprising the steps of:

receiving image data corresponding to a first two dimensional image from a first viewpoint of the object(s) against a first background;

receiving image data corresponding to a second image, from a second viewpoint different from the first viewpoint, of the objects against a second background overlapping the first background, the first and second backgrounds being two dimensional images of the whole of the three dimensional zone;

processing the image data corresponding to the first and second images to determine the position of the object in the three-dimensional zone; and

associating one or more object positions determined in said processing step with corresponding information.

35. A method as claimed in claim 34 further comprising determining the size of an object whose position is to be determined and using determined object size information in the determination of the position of the object.

36. A computer program product for enabling a programmable apparatus to carry out a method according to claim 34.

37. A storage medium storing a computer program product according to claim 36.

38. A signal conveying a computer program product according to claim 36.