Control device

• US 20100079413A1
• Filed: 09/29/2009
• Published: 04/01/2010
• Est. Priority Date: 09/29/2008
• Status: Abandoned Application

• Current Assignee: DENSO Corporation
• Original Assignee: DENSO Corporation
• Inventors: Itoh, Masahiro, Kawashima, Takeshi
• Application Number:
  US12/586,914
• Publication Number:
  US 20100079413A1
• Time in Patent Office: Days
• US Class Current: 345/175
• CPC Class Codes: B60K 2360/11 Instrument graphical user interfaces or menu aspects
• CPC Class Codes: B60K 2370/11 Graphical user interfaces or menu aspects
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60K 35/10 Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
• CPC Class Codes: B60K 37/06 of controls, e.g. controls knobs
• CPC Class Codes: G01C 21/3664 Details of the user input interface, e.g. buttons, knobs or sliders, including those provided on a touch screen; remote controllers; input using gestures
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/0416 Control or interface arrangements specially adapted for digitisers
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/0483 Interaction with page-structured environments, e.g. book metaphor
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus

Abstract

There is provided a control device for an in-vehicle electronic apparatus. The control device includes: a manipulation input element that has a manipulation input region having a predetermined area; an imaging device that has a photographing range covering the manipulation input region, and that captures an image including a hand image region representative of the hand; a hand image region identification section that identifies the hand image region in the image; an area ratio calculation section that calculates a value of hand image area ratio, which is area ratio of the hand image region to the manipulation input region; and an operation input information generation section that generates and outputs operation input information based on the calculated value of the hand image area ratio and a manipulation state of the manipulation input region, the operation input information being directed to the in-vehicle electronic apparatus.

Claims

1. A control device comprising:

a touch input device that has a manipulation surface adapted to receive a touch manipulation made by a finger of a user, and detects and outputs an input location of the touch manipulation;

an imaging device that has a photographing range having one-to-one coordinate relationship to the manipulation surface, and captures an image of a hand of the user getting access to the manipulation surface;

a fingertip specifying section that specifies a fingertip of the hand based on data of the image of the hand;

a display device that includes a display screen having one-to-one coordinate relationship to the photographing range and the manipulation surface;

a pointer image display control section that causes the display device to display a pointer image on the display screen, the pointer image pointing to a place corresponding to the fingertip;

a selection reception region setting section that sets a selection reception region on the display screen so that the selection reception region is located at a predetermined place on the display screen;

a move target image selection section that switches a move target image prepared on the selection reception region into a selected state when the touch input device detects that the touch manipulation is performed at the input location corresponds to the move target image item; and

an image movement display section that

(i) detects a target fingertip, which is the fingertip that makes the touch manipulation at the input location corresponding to the move target image item,

(ii) causes the display device to display the move target image in the selected state and the pointer image at a place corresponding to position of the target fingertip, and

(iii) causes the move target image in the selected state and the pointer image to move together on the display screen in response to movement of the target fingertip in the photographing range, in such manner that a trajectory of movement of the selected move target image and the pointer image corresponds to a trajectory of the movement of the target fingertip.

2. The control device according to claim 1, wherein:

the pointer image display control section uses a actual finger image as the pointer image, the actual finger image being extracted from the image of the hand.

3. The control device according to claim 1, wherein:

the pointer image display control section uses a pre-prepared image item as the pointer image, the pre-prepared image item being different form a actual finger image extracted from the image of the hand.

4. The control device according to claim 3, wherein:

the pre-prepared image item is a simulated finger image; and

the simulated finger image is smaller in width than the actual finger image.

5. The control device according to claim 1, wherein:

the touch manipulation includes a first touch manipulation, which is the touch manipulation that is performed by the target fingertip at the input location corresponding to the selection reception region;

the first touch manipulation switches the move target image into the selected state;

when the target fingertip is spaced apart form the manipulation surface and is moved after the first touch manipulation is performed, the image movement display section switches display mode into a coupling movement mode, in which the move target image in the selected state and the pointer image are moved together in response to the movement of the target fingertip;

the touch manipulation further includes a second touch manipulation, which is the touch manipulation that is performed, after the target fingertip is moved in the coupling movement mode, at the input location corresponding to the position of the target fingertip; and

the image movement display section switches off the coupling movement mode when the touch input device detects that the second touch manipulation is performed.

6. The control device according to claim 5, wherein:

the move target image is a marking image that highlights position of the target fingertip.

7. The control device according to claim 6, further comprising:

an operation button image display control section that causes the display device to display an operation button image on the selection reception region of the display screen, the operation button image containing the marking image as design display.

8. The control device according to claim 6, further comprising:

a marking image pasting section that causes the display device to display the marking image on the display screen, such that the marking image is fixedly pasted at a place corresponding to the input location of the second touch manipulation when the coupling movement mode is switched off.

9. The control device according to claim 6, further comprising:

a marking image deletion section that deletes the marking image, which has been displayed together with the pointier image, from the place corresponding to the input location of the second touch manipulation when the coupling movement mode is switched off.

10. The control device according to claim 7, wherein:

the marking image has a one-to-tone correspondence to a predetermined function of an electronic apparatus, which is a control target of the subject control device,

the control device further comprising;

a control command activation section that activates a control command of the predetermined function corresponding to the marking image when the touch input device detects that the second touch manipulation is performed.

11. The control device according to claim 10, wherein:

the selection reception region is a plurality of selection reception regions;

the predetermined function of the electronic apparatus is a plurality of predetermined functions;

the marking image is a plurality of marking images; and

the plurality of marking images respectively corresponds to the plurality of predetermined functions of the electronic apparatus;

the control device further comprising;

an operation button image display control section that causes the display device to respectively display a plurality of operation button images on the polarity of selection reception regions, so that the plurality of operation button images respectively contains the plurality of marking images as design display,

wherein;

when the first touch manipulation is performed at the input location corresponding to one operation button images of the operation button images, the image movement display section (i) switches one marking image of the marking images that corresponds to the one operation button image in the selected state, and (ii) switches the display mode into the coupling movement mode; and

when the touch input device detects that the second touch manipulation is performed, the control command activation section activates the control command of one of the predetermined functions corresponding to the one marking image being in the selected state.

12. The control device according to claim 10, wherein:

a part of the manipulation surface is a command activation valid region;

a part of the display screen is a window outside part, which corresponds to the command activation enablement part;

the operation button image is displayed on the window outside part of the display screen;

the control command activation section activates the control command of the predetermined function when the touch input device detects that the second touch manipulation is performed on the command activation enablement part of the manipulation surface; and

the control command activation section does not activate the control command of the predetermined function when the touch input device detects that the second touch manipulation is performed outside the command activation enablement part.

13. The control device according to claim 10, wherein:

the electronic apparatus is an in-vehicle electronic apparatus; and

the touch input device and the display device are arranged in a vehicle compartment such that the display screen of the display device is out of a filed of sight of the user who is sitting in a seat in the vehicle compartment and who is looking straight at the manipulation surface of the touch input device.

14. The control device according to claim 10, wherein:

the in-vehicle electronic apparatus is a car navigation system.

15. The control device according to claim 14, wherein:

a part of the display screen is a map display region for displaying a map for use in the car navigation system;

the operation button image is displayed on the selection reception region and is displayed on an outside of the map display region;

the control command enables a user to specify a point on the map displayed on the map display region;

the control command is assigned to correspond to the operation button image;

the control command activation section activates the control command when the touch input device detected that the second touch manipulation is performed inside the map display region; and

the control command activation section does not activates the control command when the touch input device detects that the second touch manipulation is performed inside the map display region.

16. The control device according to claim 15 wherein:

the control command is one of

(i) a destination setting command to set a destination on the map display region,

(ii) a stopover point setting command to set a stopover point on the map display region,

(iii) a peripheral facilities search command, and

(iv) a map enlargement command.

17. The control device according to claim 6, wherein:

the display screen has a pointer displayable part, in which the pointer image is displayable; and

when the target fingertip escapes from the pointer displayable part in the coupling movement mode, the image movement display section switches off the coupling movement mode and switches the marking image in an unselected state.

18. The control device according to claim 6, wherein:

the display screen has a pointer displayable part, in which the pointer image is displayable;

when the target fingertip escapes from the pointer displayable part in the coupling movement mode, the image movement display section maintains the selected state of the marking image; and

when one of the escaped target fingertip and a substitution fingertip, which is a substation fingertip of the escaped target fingertip, is detected in the pointer displayable part after the target fingertip has escaped from the pointer displayable part, the image move display section keeps the coupling movement mode

by newly setting the target fingertip to the one of the escaped target fingertip and the substitution fingertip and

by using the marking image being in the selected state.

19. The control device according to claim 6 further comprising:

a target fingertip movement detection section that detects the movement of the target fingertip in the coupling movement mode,

wherein;

when the detected movement of the target fingertip in the coupling movement mode corresponds to a predetermined mode switch off movement, the image movement display section switches off the coupling movement mode and switches the marking image in an unselected state.

20. A control device for a user to operate an in-vehicle electronic apparatus in a vehicle by manipulating the control device, the control device comprising:

a manipulation input element that is located so as to be within reach of the user who is sitting in a seat of the vehicle, and that has a manipulation input region having a predetermined area;

an imaging device that has a photographing range covering the manipulation input region, and that captures an image including a hand image region representative of the hand of the user getting access to the manipulation input element;

a hand image region identification section that identifies the hand image region in the image;

an area ratio calculation section that calculates a value of hand image area ratio, which is area ratio of the hand image region to the manipulation input region; and

an operation input information generation section that generates and outputs operation input information based on the calculated value of the hand image area ratio and a manipulation state of the manipulation input region, the operation input information being directed to the in-vehicle electronic apparatus.

21. The control device according to claim 20, wherein:

the operation input information generation section determines content of the operation input information, based on a predetermined correspondence relationship between the content of the operation input information and the value of the hand image area ratio; and

the operation input information section generates and outputs the operation input information having the content that corresponds to the calculated value of the hand image area ratio.

22. The control device according to claim 21, wherein:

when the calculated value of the hand image area ratio exceeds a predetermined threshold, the operation input information generation section outputs predetermined-function activation request information as the operation input information to request a predetermined-function of the in-vehicle electronic apparatus to be activated.

23. The control device according to claim 22, wherein:

the predetermined threshold is larger than 0.6; and

the value of the hand image area ratio larger the predetermined threshold corresponds to an occurrence of a hand cover state in the manipulation input region.

24. The control device according to claim 23, wherein:

when the manipulation input element receives a predetermined manipulation input after the predetermined-function of the in-vehicle electronic apparatus is activated, the operation input information generation section outputs operation change request information to request a change in operation state of the predetermined-function of the in-vehicle apparatus.

25. The control device according to claim 24, wherein:

the operation change request information is operation recover request information that requests deactivation of the predetermined-function of the in-vehicle electronic apparatus to recover the in-vehicle electronic apparatus into a pre-activation stage of the predetermined-function.

26. The control device according to claim 20, further comprising:

an area ratio variation detection section that detects a time variation in value of the hand image area ratio, the time variation being caused by a predetermined input hand movement in the manipulation input region,

wherein;

when the detected time variation matches a predetermined time variation, the operation input information generation section generates and outputs the operation input information having the content that corresponds to the predetermined time variation.

27. The control device according to claim 26, wherein:

the manipulation input region is divided into multiple sub-regions;

the hand image area ratio calculation section calculates the hand image area ratio in each of the multiple sub-regions;

the hand image area ratio variation detection section detects the time variation in value of the hand image area ratio in each of the multiple sub-regions.

28. The control device according to claim 27, wherein:

the hand image area ratio variation detection section detects a first state sub-region, which is the sub-region whose value of the hand image area ratio is grater than or equal to the predetermined threshold;

the hand image area ratio variation detection section detects

(i) a number of first state sub-regions and

(ii) a change in appearance location of the first sub-region in the multiple sub-regions over time

as a transition behavior; and

the operation input information generation section generates and outputs the operation input information when the detected transition behavior matches a predetermined transition behavior.

29. The control device according to claim 28, wherein:

the multiple sub-regions are arranged adjacent to each other in a row extending in a predetermined direction;

the hand image area ratio variation detection section detects a second state sub-region, which is the sub-region whose value of the hand image area ratio is less than the predetermined threshold;

the hand image area ratio variation detection section detects a state distribution change, which includes a change in distribution of the first state sub-region and the second state sub-region on the manipulation input region over time; and

the operation input information generation section generates and outputs the operation input information when the detected state distribution change matches a predetermined state distribution change.

30. The control device according to claim 29, wherein:

the state distribution change further includes a change in appearance location distribution of the first state sub-region and the second state sub-region on the manipulation input region over time.

31. The control device according to claim 29, wherein:

the hand image area ratio variation detection section determines the state distribution change by detecting one of;

a change of the number of first state sub-regions over time; and

a change of the number of second state sub-regions over time.

32. The control device according to claim 31, wherein:

the manipulation input region has a first end and a second end opposite to each other in the predetermined direction;

the multiple sub-region are aligned in the predetermined direction so as to be arranged between the first end and the end;

the predetermined input hand movement is movement of the hand across the multiple sub-regions in the predetermined direction; and

the hand image area ratio variation detection section determine the state distribution change caused by the predetermined input hand movement, by detecting movement behavior of appearance location of the first state sub-region.

33. The control device according to claim 20, wherein:

the manipulation input element is a location input device;

the location input device includes a transparent input reception plate;

one surface of the transparent input reception plate is included in the manipulation input region and is adapted to receive a touch manipulation made by a finger of the user;

the location input device sets an input coordinate system on the manipulation input region;

the location input device detects a location of the touch manipulation on the input coordinate system and outputs coordinate information on the location of the touch manipulation on the input coordinate system; and

the imaging device is located on an opposite side of the transparent input reception plate from the manipulation input region, so that the imaging device captures, through the transparent input reception plate, the image of the hand;

the control device further comprising;

a display device that displays an input window, which provides a reference for the user to perform an input operation on the location input device; and

a pointer image display section that superimposes a pointer image on the input window,

wherein;

the pointer image is based on image information on the hand image region; and

on the input window, the pointer image is located at a place corresponding to place of the hand image region in the captured image.

34. The control device according to claim 33 further comprising:

an illumination light source that is located on the opposite side of the transparent input reception light from the manipulation input region,

wherein;

the illumination source irradiates the manipulation input region with illumination light; and

the imaging device captures the image including the hand image region, based on the illumination light reflected from the hand.

35. The control device according to claim 34, wherein:

each of the display device and the display control section is a component of the in-vehicle electronic apparatus;

the operation input information generation section outputs window content change command information as the operation input information to the display control section, based on the calculated value of the hand image area ratio; and

the display control section causes the display device to change content of the input window when the display control section receives the window content change command information.

36. The control device according to claim 35, wherein:

when the hand image area ratio increases from a value lower than the predetermined threshold to a value larger than the predetermined threshold, the operation input information generation section outputs window switch command information as the outputs window content change command information;

the outputs window content change command information requests the display device to switch the input window

from a first widow that is presently displayed;

into a second window different from the first window.

37. The control device according to claim 36, wherein

when the location input device receives a predetermined touch manipulation after the input window is switched into the second input window, the operation input information generation section outputs window recovery request information;

and

the window recovery request information request the display device to recover the input window into the first window.