Manipulation input device

• US 20090102788A1
• Filed: 10/20/2008
• Published: 04/23/2009
• Est. Priority Date: 10/22/2007
• Status: Active Grant

• Current Assignee: Mitsubishi Electric Corporation
• Original Assignee: Mitsubishi Electric Corporation
• Inventors: Tsuji, Masayuki, Nishida, Yoshihiro, Matsutani, Kiyoshi, Otsuka, Isao, Okada, Reiko
• Granted Patent: US 8,378,970 B2
• Time in Patent Office: Days
• US Class Current: 345/158
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

When a vehicle navigation system is manipulated by taking pictures of a user hand motion and gesture with a camera, as the number of apparatuses and operational objects increases, the associated hand shapes and hand motions increase, thus causing a complex manipulation for a user. Furthermore, in detecting a hand with the camera, when the image of a face having color tone information similar to that of a hand appears in an image taken with a camera, or outside light rays such as sun rays or illumination rays vary, detection accuracy is reduced. To overcome such problems, a manipulation input device is provided that includes a limited hand manipulation determination unit and a menu representation unit, whereby a simple manipulation can be achieved and manipulation can accurately be determined. In addition, detection accuracy can be improved by a unit that selects a single result from results determined by a plurality of determination units, based on images taken with a plurality of cameras.

Claims

1. A manipulation input device, comprising:

a picture-taking apparatus;

a hand region detector that detects a region of a human hand from a motion image taken with the picture-taking apparatus;

a hand manipulation determination unit that determines a hand manipulation from a shape and a motion of the detected hand region; and

a selection menu representation unit that notifies a user of a menu selected based on the determined manipulation.

3. The manipulation input device of claim 1, wherein the hand region detector comprises:

a skin tone region extraction unit that extracts a region of a skin tone from a motion image taken with a visible light camera;

a difference region extraction unit that extracts region data differing between current motion image data and background image data in an image taken with the visible light camera;

a binary correction unit that digitizes image data of the region extracted by the skin tone region extraction unit and the difference region extraction unit, to then correct the digitized image data;

a distance calculator that calculates minimum distances from inner points of the binary image region obtained by the binary correction unit to an edge thereof, to detect as the center of a hand a distance maximum point where the calculated values become a maximum one;

a center weighted correction unit that adds to the distance value of the distance maximum point a correction value such that the closer the position of the distance maximum point is to the center of an imaging screen, the larger the correction value, and the closer the position thereof is to the end portion of the imaging screen, the smaller the correction value;

a candidate hand region detector that detects a hand region from a binary image region that includes a point where the distance value of the distance maximum point to which the correction value is added is a maximum;

an edge length and area calculator that calculates an edge length, and an area, of the candidate hand region; and

a hand region determination unit that determines whether or not the candidate hand region is a hand from the calculated edge length and area.

4. The manipulation input device of claim 1, wherein the hand region detector comprises:

a difference region extraction unit that extracts region data differing between current motion image data and background image data, in an image taken with an infrared camera;

a binary correction unit that digitizes image data of the region extracted by the difference region extraction unit, to then correct the digitized image data;

a distance calculator that calculates minimum distances from inner points of the binary image region obtained by the binary correction unit to an edge thereof, to detect as the center of a hand a distance maximum point where the calculated values become a maximum one;

a center weighted correction unit that adds to the distance value of the distance maximum point a correction value such that the closer the position of the distance maximum point is to the center of an imaging screen, the larger the correction value, and the closer the position thereof is to the end portion of the imaging screen, the smaller the correction value;

a candidate hand region detector that detects a hand region from a binary image region that includes a point where the distance value of the distance maximum point to which the correction value is added is a maximum;

an edge length and area calculator that calculates an edge length, and an area, of the candidate hand region; and

a hand region determination unit that determines whether or not the candidate hand region is a hand from the calculated edge length and area.

5. The manipulation input device of claim 1, wherein the hand manipulation determination unit comprises:

an edge direction histogram generator that generates a histogram representing frequency distribution of edge directions in a detected hand region;

a hand direction determination unit that determines a hand direction according to the size of the histogram obtained by the edge direction histogram generator;

a determination result storage unit that stores the result determined by the hand direction determination unit; and

a time-series data determination unit that determines the current hand manipulation from the result for a predetermined time, stored in the determination result storage unit.

6. The manipulation input device of claim 1, wherein the hand manipulation determination unit comprises:

a center-to-edge distance calculator that calculates distances from the center of the hand region detected by the hand region detector to all the points on the edge of the hand region;

a hand direction determination unit that determines a hand direction from distances and directions of two lines connecting the hand region center with two points where calculated distances are the maximum one and the minimum one;

a determination result storage unit that stores the result determined by the hand direction determination unit; and

a time-series data determination unit that determines the current hand manipulation from the result for a predetermined time, stored in the determination result storage unit.

7. The manipulation input device of claim 1, wherein based on the detection result from the hand region detector, the hand manipulation determination unit determines at least change, execution, and cancellation, of a candidate menu.

8. The manipulation input device of claim 1, wherein the picture-taking apparatus is an infrared camera, and an infrared LED is switched on when an average infrared level of images taken with the infrared camera is lower than a predetermined value and the infrared LED is switched off when the level thereof is higher than the predetermined value.

2. A manipulation input device, comprising:

a plurality of different kinds of picture-taking apparatus

a plurality of hand region detector that detects a region of a human hand from a motion image taken with each of the plurality of picture-taking apparatus;

a plurality of hand manipulation determination unit that determines a hand manipulation from a shape and a motion of the detected hand region;

a hand manipulation determination and selection unit that selects a single result from results determined by the plurality of hand manipulation determination unit; and

a selection menu representation unit that notifies a user of a menu selected based on the manipulation selected by the hand manipulation determination and selection unit.

9. The manipulation input device of claim 2, wherein the hand region determination unit comprises

a skin tone region extraction unit that extracts a region of a skin tone from a motion image taken with a visible light camera;

a difference region extraction unit that extracts region data differing between current motion image data and background image data in an image taken with the visible light camera;

a binary correction unit that digitizes image data of the region extracted by the skin tone region extraction unit and the difference region extraction unit, to then correct the digitized image data;

a distance calculator that calculates minimum distances from inner points of the binary image region obtained by the binary correction unit to an edge thereof, to detect as the center of a hand a distance maximum point where the calculated values become a maximum one;

a center weighted correction unit that adds to the distance value of the distance maximum point a correction value such that the closer the position of the distance maximum point is to the center of an imaging screen, the larger the correction value, and the closer the position thereof is to the end portion of the imaging screen, the smaller the correction value;

a candidate hand region detector that detects a hand region from a binary image region that includes a point where the distance value of the distance maximum point to which the correction value is added is a maximum;

an edge length and area calculator that calculates an edge length, and an area, of the candidate hand region; and

a hand region determination unit that determines whether or not the candidate hand region is a hand from the calculated edge length and area.

10. The manipulation input device of claim 2, wherein the hand region detector comprises:

a difference region extraction unit that extracts region data differing between current motion image data and background image data in an image taken with an infrared camera;

a binary correction unit that digitizes images data of the region extracted by the difference region extraction unit, to then correct the digitized image data;

a distance calculator that calculates minimum distances from inner points of the binary image region obtained by the binary correction unit to an edge thereof, to detect as the center of a hand a distance maximum point where the calculated values become a maximum one;

a center weighted correction unit that adds to the distance value of the distance maximum point a correction value such that the closer the position of the distance maximum point is to the center of an imaging screen, the larger the correction value, and the closer the position thereof is to the end portion of the imaging screen, the smaller the correction value;

a candidate hand region detector that detects a hand region from a binary image region that includes a point where the distance value of the distance maximum point to which the correction value is added is a maximum;

an edge length and area calculator that calculates an edge length, and an area, of the candidate hand region; and

a hand region determination unit that determines whether or not the candidate hand region is a hand from the calculated edge length and area.

11. The manipulation input device of claim 2, wherein the hand manipulation determination unit comprises:

an edge direction histogram generator that generates a histogram representing frequency distribution of edge directions in a detected hand region;

a hand direction determination unit that determines a hand direction according to the size of the histogram obtained by the edge direction histogram generator;

a determination result storage unit that stores the result determined by the hand direction determination unit; and

a time-series data determination unit that determines the current hand manipulation from the result for a predetermined time, stored in the determination result storage unit.

12. The manipulation input device of claim 2, wherein the hand manipulation determination unit comprises:

a center-to-edge distance calculator that calculates distances from the center of the hand region detected by the hand region detector to all the points on the edge of the hand region;

a hand direction determination unit that determines a hand direction from distances and directions of two lines connecting the hand region center with two points where calculated distances are the maximum one and the minimum one;

a determination result storage unit that stores the result determined by the hand direction determination unit; and

a time-series data determination unit that determines the current hand manipulation from the result for a predetermined time, stored in the determination result storage unit.

13. The manipulation input device of claim 2, wherein based on the detection result from the hand region detector, the hand manipulation determination unit determines at least change, execution, and cancellation, of a candidate menu.

14. The manipulation input device of claim 2, wherein the picture-taking apparatus is an infrared camera, and an infrared LED is switched on when average infrared level of images taken with the infrared camera is lower than a predetermined value and the infrared LED is switched off when the level thereof is higher than the predetermined value.

15. The manipulation input device of claim 2, in which the different kinds of the plurality of picture-taking apparatus includes two different kinds of cameras, a visible light camera and an infrared camera, the manipulation input device, further comprising:

a hand manipulation determination and selection unit that determines, in each picture-taking apparatus, a hand manipulation using the plurality of hand manipulation determination units, to then select a result having the smaller difference between results determined by the hand manipulation determination units, based on images taken with the same picture-taking unit.

16. The manipulation input device of claim 2, wherein the plurality of picture-taking apparatus includes a visible light camera and an infrared camera, the manipulation input device further comprising:

a hand manipulation determination and selection unit that selects a hand manipulation result determined based on an image taken using the infrared camera when an average infrared level of the image using the infrared camera is lower than a predetermined value, and selects a hand manipulation result determined based on an image taken using the visible light camera when the level thereof is higher than the predetermined value.