User interface device

• US 8,395,600 B2
• Filed: 01/29/2010
• Issued: 03/12/2013
• Est. Priority Date: 01/30/2009
• Status: Active Grant

• Current Assignee: DENSO Corporation
• Original Assignee: DENSO Corporation
• Inventors: Kawashima, Takeshi, Nagata, Takashi
• Primary Examiner(s): Mengistu, Amare
• Assistant Examiner(s): Zhou, Hong
• Application Number:
  US12/657,868
• Publication Number:
  US 20100194713A1
• Time in Patent Office: 1,138 Days
• Field of Search: 345156-157, 345173-175, 382103-104, 382/190, 382/284, 701/400, 701/408
• US Class Current: 345/175
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/042 by opto-electronic means
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

A user interface device detects a fingertip placed on an imaging range. The imaging range is located or arranged so that a user inserts and place a hand along a predetermined direction. A device detects a tip area corresponding to the fingertip from the imaging range. The device determines whether the detected tip area is a finger or not. The determination can be performed by evaluating a size of the tip area, for example, whether the width is in an appropriate range or not. If the tip area is verified as a finger, a position of the fingertip is calculated. According to the device, it is possible to determine whether it is a finger or not by using a relatively simple algorithm. As a result, it is possible to eliminate objects other than finger and to provide sufficient accuracy for detecting the fingertip.

Claims

1. An user interface device comprising:

an imaging device which takes an image of an imaging range in which a hand is placed along a predetermined insertion direction;

a tip area detecting module which detects a tip area in the insertion direction from the image taken by the imaging device;

a tip area locating module which locates a position of the tip area in the imaging range and sets a position of the tip area as a tip position;

a verifying module which verifies whether the tip position indicates a position of an actual fingertip based on at least one of a size of the tip area and an area of the tip area; and

an outputting module which outputs coordinates of the tip position as coordinates of the actual fingertip if the tip position is verified as the tip position of the actual fingertip;

wherein

the tip area detecting module uses a first image which is the image taken by the imaging device and a second image which is an image obtained by shifting the first image parallel to the insertion direction, and detects a non-overlapping area generated on a distal end side in the insertion direction between the first image and the second image as the tip area;

the tip area detecting module detects a plurality of non-overlapping areas as potential tip areas, when a plurality of non-overlapping areas are separately detected by inserting a plurality of fingers into the imaging range; and

the verifying module determines whether at least one of the non-overlapping areas is an actual fingertip area or not based on whether a value S/d estimated as a finger width is in a predetermined range or not, where a value S is a total area of the image, and a value d is a sum of distance between each non-overlapping area and an edge of the imaging range, the edge being located on an entering side of the insertion direction.

2. The user interface device in claim 1 wherein

the imaging device takes an image of the hand from a lower side, the hand being inserted in a forward direction along a horizontal direction while a palm is faced down onto the imaging range.

3. The user interface device in claim 2 wherein

the imaging device includes a lighting device which lights up a surface of the hand to be imaged, and the imaging device takes an image of the hand based on reflected light from the hand.

4. The user interface device in claim 1 wherein

the tip area locating module locates the position of the tip area which is determined by the non-overlapping area and sets the position of the tip area as the tip position.

5. The user interface device in claim 1, wherein

the verifying module determines whether the at least one of the non-overlapping areas is an actual fingertip area or not based on whether a size or an area of the non-overlapping area is in a predetermined range or not.

6. The user interface device in claim 5, further comprising:

a hand guide member which guides a user to insert the hand to the imaging range while restricting movement of the hand along a guiding direction which corresponds to the insertion direction, and wherein

the verifying module determines whether the at least one of the non-overlapping areas is the actual fingertip area or not based on whether a width of the non-overlapping area is in a predetermined range or not, where the width is measured in a perpendicular direction to the guiding direction.

7. The user interface device in claim 1, wherein

the verifying module determines whether the at least one of non-overlapping areas is an actual fingertip area or not based on whether a value S/N estimated as a mean finger area is in a predetermined range or not, where the value S is a total area of the image, and a value N is the number of the non-overlapping area.

8. The user interface device in claim 1, further comprising:

a display device having a display range which is related to the imaging range by a coordinates system; and

a fingertip point displaying module which displays an actual fingertip point on the display range in a visible manner.

9. The user interface device in claim 8, wherein

the fingertip point displaying module includes a pointer displaying module which displays a pointer image for indicating the actual fingertip point based on a pointer image data prepared separately from the image, and which does not display the pointer image at the tip area which is not determined as the actual fingertip.

10. The user interface device in claim 9, wherein

the pointer displaying module displays the pointer image with a shape narrower than an actual finger in the image taken by the imaging device.

11. The user interface device in claim 10, wherein

the pointer image is a simulated finger image which simulates a contour shape of the finger or an actual finger image which is taken by imaging each finger in a separated manner.

12. The user interface device in claim 11, further comprising:

a restricting member which restricts an imaging distance of the imaging device to the hand as an object in a predetermined distance, wherein

the imaging range of the imaging device has a size capable of imaging only a part of fingers on a distal end part of a hand of an average adult, when the imaging distance to the hand of the average adult is restricted and the fingers are stretched, the part of the fingers being formed by the index finger, the middle finger, the third finger, and the little finger.

13. The user interface device in claim 12, wherein

the imaging range has a size which can take an image of at least two fingers among the index finger, the middle finger, and cannot take an image of four fingers.

14. The user interface device in claim 13, wherein

the imaging range is a rectangular shape having a longitudinal length corresponding to a width direction of the hand and a lateral length, the longitudinal length being in a range equal to or longer than 60 mm and equal to or shorter than 80 mm, the lateral length being in a range equal to or longer than 30 mm and equal to or shorter than 55 mm.

15. The user interface device in claim 14, wherein

the display device displays software-key images on the display range at corresponding positions, the software-key images corresponding to key-operation areas defined on an input coordinates plane of an input device, and wherein

the software-key images are displayed with sizes and gaps so that a plurality of software-key images arranged on a width direction of the finger are overlapped with an imaginary finger projection area of the actual finger image, the imaginary finger projection area being provided by imaginary projecting the actual finger image on the imaging range onto the display range at a corresponding position and with a corresponding size based on the coordinates system.

16. The user interface device in claim 12, wherein

the restricting member is disposed in a fixed manner to satisfy a predetermined spatial relationship with respect to the imaging device, and includes a transparent support plate which is capable of supporting the hand to be imaged on a side surface thereof, the imaging device takes the image of the hand through the transparent support plate from an opposite side surface of the transparent support plate.

17. The user interface device in claim 16, wherein

the transparent support plate provides a touch input device which provides, on the side surface, an input coordinates system corresponding to the display range, and wherein

the outputting module outputs the coordinates of the tip position as the coordinates of the fingertip point at least in a condition that the touch input device does not detect any touch input, when the tip position is determined as the actual fingertip.

18. The user interface device in claim 17, wherein

the imaging range includes a display corresponding area corresponding to the display range, and a display outer rim area formed along outside of the display corresponding area with a predetermined width, and wherein

the outputting module outputs the coordinates of the tip position as the coordinates of the fingertip point if the tip position located by the tip area locating module is located within the display corresponding area.

19. The user interface device in claim 9, wherein

the pointer displaying module includes:

a finger direction base setting module which sets a finger direction base on a coordinates plane which defines a displaying position of the pointer image on the display range; and

a pointer direction determining module which determines a direction of the pointer image based on the finger direction base and the fingertip point, and wherein

the pointer displaying module displays the pointer image on the display range so that a longitudinal direction of the pointer image is placed along with the direction determined in the pointer direction determining module and a position of the distal end part of the pointer image is placed on the fingertip point.

20. The user interface device in claim 19, wherein

the finger direction base setting module sets the finger direction base on an outside of the display range, the finger direction base being a wrist position estimated as a position of a wrist.

21. The user interface device in claim 20, wherein

the pointer displaying module displays an image of a hand at least including a finger on the display range, the image of the hand being arranged to have a spatial relationship with respect to the display range in which a wrist is placed outside and below the display range, and wherein

the finger direction base setting module sets the wrist position on a position which is distanced downwardly from the bottom edge of the display range by a predetermined length in a Y-axis, where the Y-axis is a vertical direction of the display range, and an X-axis is a direction perpendicular to the vertical direction.

22. The user interface device in claim 20, wherein

the finger direction base setting module sets the wrist position to have a spatial relationship which is predetermined with respect to the fingertip point determined in the tip area locating module.

23. The user interface device in claim 22, wherein

the pointer displaying module displays an image of a hand at least including a finger on the display range, the image of the hand being arranged to have a spatial relationship with respect to the display range in which a wrist is placed outside and below the display range, and wherein

the finger direction base setting module sets the wrist position on a position which is distanced downwardly from the fingertip point by a predetermined length in a Y-axis, where the Y-axis is a vertical direction of the display range, and an X-axis is a direction perpendicular to the vertical direction.

24. The user interface device in claim 20, wherein

the finger direction base setting module sets the wrist position which is common for all of the fingertip points determined on the display range.

25. The user interface device in claim 24, wherein

an finger direction base setting module determines the X coordinate of the wrist position so that a shifting amount in an X-axis from a standard wrist position is increased as an inclination angle of a representative actual finger image with respect to a Y-axis is increased, the standard wrist position being fixedly defined below the display range, and the representative actual finger image having the fingertip point which is located on a highest position in the display range among the fingertip points of a plurality of actual finger images.

26. The user interface device in claim 19, wherein

the finger direction base setting module sets a wrist position so that an X coordinate of the wrist position is adjusted according to an X coordinate of the fingertip point.

27. The user interface device in claim 26, wherein

the finger direction base setting module determines the X coordinate of the wrist position so that a shifting amount in an X-axis from a standard wrist position is increased as an inclination angle of an actual finger image with respect to a Y-axis is increased, the standard wrist position being fixedly defined below the display range.

28. The user interface device in claim 27, wherein

the finger direction base setting module determines the X coordinate of the wrist position so that the wrist position is shifted to the left side in the X-axis for the actual finger image which is inclined in a right up shape with respect to Y-axis, and determines the X coordinate of the wrist position so that the wrist position is shifted to the right side in the X-axis for the actual finger image which is inclined in a left up shape with respect to the Y-axis.

29. An user interface device comprising:

an imaging device which takes an image of an imaging range in which a hand is placed along a predetermined insertion direction;

a tip area detecting module which detects a tip area in the insertion direction from the image taken by the imaging device;

a tip area locating module which locates a position of the tip area in the imaging range and sets a position of the tip area as a tip position;

a verifying module which verifies whether the tip position indicates a position of an actual fingertip based on at least one of a size of the tip area and an area of the tip area; and

an outputting module which outputs coordinates of the tip position as coordinates of the actual fingertip if the tip position is verified as the tip position of the actual fingertip;

a display device having a display range which is related to the imaging range by a coordinates system; and

a fingertip point displaying module which displays an actual fingertip point on the display range in a visible manner;

wherein

the fingertip point displaying module includes a pointer displaying module which displays a pointer image for indicating the actual fingertip point based on a pointer image data prepared separately from the image, and which does not display the pointer image at the tip area which is not determined as the actual fingertip, the pointer display module includes;

a finger direction base setting module which sets a finger direction base on a coordinates plane which defines a displaying position of the pointer image on the display range; and

a pointer direction determining module which determines a direction of the pointer image based on the finger direction base and the fingertip point, wherein

the pointer displaying module displays the pointer image on the display range so that a longitudinal direction of the pointer image is placed along with the direction determined in the pointer direction determining module and a position of the distal end part of the pointer image is placed on the fingertip point;

the finger direction base setting module sets the finger direction base on an outside of the display range, the finger direction base being a wrist position estimated as a position of a wrist,

the finger direction base setting module sets a plurality of wrist positions each of which corresponds to one of a plurality of fingertip points determined on the display range, and

the pointer direction determining module determines a plurality of directions for the pointer images based on a respective one of pairs of the fingertip point and the wrist position.

30. The user interface device in claim 29, wherein

the finger direction base setting module determines an X coordinate of the wrist position so that the wrist position is shifted to the right side from the standard wrist position in an X-axis for the fingertip point which is located on the right side from a standard wrist position in the X-axis, and determines the X coordinate of the wrist position so that the wrist position is shifted to the left side from the standard wrist position in the X-axis for the fingertip point which is located on the left side from the standard wrist position in the X-axis.

31. The user interface device in claim 30, wherein

the finger direction base setting module sets the wrist position so that a shifting amount of the wrist position from the standard wrist position in the X-axis is increased as an offset amount of the fingertip point from the standard wrist position in the X-axis is increased.