Operation input device and method of operation input

• US 7,893,920 B2
• Filed: 04/13/2005
• Issued: 02/22/2011
• Est. Priority Date: 05/06/2004
• Status: Active Grant

• Current Assignee: Alpine Electronics Incorporated (Alps Alpine Co., Ltd.)
• Original Assignee: Alpine Electronics Incorporated (Alps Alpine Co., Ltd.)
• Inventors: Endoh, Kenjiro
• Primary Examiner(s): Awad; Amr
• Assistant Examiner(s): Pervan; Michael
• Application Number:
  US11/104,872
• Publication Number:
  US 20050248529A1
• Time in Patent Office: 2,141 Days
• Field of Search: 345156-158, 345/159, 345/173, 345/175
• US Class Current: 345/156
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042

Abstract

A three-dimensional image sensor for detecting the three-dimensional position of an object, a coordinate detection unit for detecting the coordinate position of an objective point on a display screen based on the position of the object in a virtual pointing space, a movement detection unit for detecting the three-dimensional movement of the object in the pointing space, and an operation input contents determination unit for determining an operation based on the three-dimensional movement or the like of the object are provided, so that the inconvenience of moving a pointing device such as a finger to a fixed spatial plane is avoided by setting the pointing space which can be operated by the pointing device as a pyramid shape defined by connecting the display screen and a reference point.

Claims

1. An operation input device comprising:

position detection means for detecting the three-dimensional position of an object in a space;

pointing device recognizing means for recognizing whether an object in the pointing space, which is detected by the position detection means, is a predetermined pointing device;

pointing space setting means for setting a virtual pointing space of a shape defined by connecting an area of a display screen as a base of the pointing space and a predetermined reference point which is located at a position spaced apart from the display screen as an apex of the pointing space;

coordinate detection means for detecting a coordinate position of an objective point on the display screen using a mapping relation based on the position in the pointing space of the object which is recognized as the pointing device by the pointing device recognizing means;

movement detection means for detecting the three-dimensional movement in the pointing space of the object which is recognized as the pointing device by the pointing device recognizing means; and

determination means for determining the contents of the operation input based on the coordinate position of the objective point on the display screen, which is detected by the coordinate detection means, and the three-dimensional movement of the object detected by the movement detection means and generating a corresponding operation instruction;

wherein when the pointing device moves toward the display screen at a speed faster than a predetermined speed within a range of a predetermined coordinate position of the display screen in the pointing space, the determination means determines that a pointing start action is performed.

2. The operation input device according to claim 1, wherein when the pointing device moves away from the display screen at a speed faster than a predetermined speed within a predetermined coordinate position of the display screen in the pointing space, the determination means determines that a pointing release action is performed.

3. The operation input device according to claim 1, wherein when the pointing device moves toward the display screen at a speed faster than a predetermined speed within a predetermined coordinate position of the display screen, and then within a predetermined time period the pointing device moves away from the display screen at a speed faster than a predetermined speed within a predetermined coordinate position of the display screen in the pointing space, the determination means determines that a click action is performed.

4. The operation input device according to claim 1, wherein, when the pointing device moves toward the display screen at a speed faster than a predetermined speed within a predetermined coordinate position of the display screen and then moves so as to move the coordinate position of the display screen, which is detected by the coordinate detection means, the determination means determines that a drag action is performed.

5. The operation input device according to claim 4, comprising display control means for controlling a display in such a manner that when the determination means determines that the drag action is performed, the object located at the coordinate position of the objective point on the display screen, which is detected by the coordinate detection means, is moved in a dragged direction by a dragged amount.

6. The operation input device according to claim 1, wherein the reference point is a position of the operator'"'"'s eye, which is detected by the position detection means.

7. The operation input device according to claim 1, wherein the reference point is a position of the operator'"'"'s dominant eye, which is detected by the position detection means.

8. The operation input device according to claim 1, wherein the reference point corresponds to an apex of a pyramid containing both of the display screen and a plane obtained by connecting the positions of the pointing device when pointing at the corners of the display screen in a space as cross-sections.

9. The operation input device according to claim 1, wherein the pointing space setting means sets the position corresponding to the apex of a pyramid containing both of the display screen and a plane obtained by connecting the positions of the pointing device when pointing at the corners of the display screen in a space as cross-sections as an initial value, and causes the reference point to move from the initial value according to the three-dimensional movement of the operator'"'"'s eye, which is detected by the movement detection means, based on the position of the operator'"'"'s eye, which is detected by the position detection means.

10. The operation input device according to claim 1, wherein the pointing device recognizing means recognizes a rod-shaped projecting object as the pointing device.

11. The operation input device according to claim 1, comprising a display control means for displaying a predetermined mark at the coordinate position of the objective point on the display screen, which is detected by the coordinate detection means.

12. An operation input device comprising:

position detection means for detecting the three-dimensional position of an object in a space;

pointing space setting means for setting a virtual pointing space of a shape defined by connecting an area of a display screen as a base of the pointing space and a predetermined reference point located at a position spaced apart from the display screen as an apex of the pointing space, and setting a pointing plane obtained by mapping the coordinate position of the display screen at a position in the pointing space where the object exists;

coordinate detection means for detecting the coordinate position of the objective point on the display screen using a mapping relation based on the position of the object on the pointing plane, which is detected by the position detection means;

movement detection means for detecting the three-dimensional movement of the object in the pointing space; and

determination means for determining the contents of the operation input based on the coordinate position of the objective point on the display screen, which is detected by the coordinate detection means, and the three-dimensional movement of the object detected by the movement detection means and generating a corresponding operation instruction;

wherein the pointing plane is located between the display screen and the operator, includes the position of the pointing device when the operator performs the pointing action, and is a plane defined by the pointing device when the operator moves the pointing device in a natural manner, and the pointing space setting means sets the pointing plane every time when the pointing action is performed.

13. A method of operation input comprising:

detecting the three-dimensional position of an object in a virtual pointing space of a shape defined by connecting an area of a display screen as a base of the pointing space and a predetermined reference point located at a position spaced apart from the display screen as an apex of the pointing space wherein the reference point is a detected position of an operator'"'"'s eye;

detecting the three-dimensional movement of the object in the pointing space and calculating the coordinate position of the objective point on the display screen using a mapping relation based on the position of the detected object; and

determining the contents of the operation input based on the three-dimensional movement of the object and the coordinate position of the objective point on the display screen corresponding to the position of the object, and generating a corresponding operation instruction;

wherein the method further comprises;

displaying predetermined marks at four corners of the display screen to prompt the operator to perform the pointing action, and setting the pointing space by setting a position corresponding to an apex of a pyramid containing as cross-sections both of a plane defined by connecting the positions of the pointing device when the operator performs the pointing action for the respective predetermined marks in the space and the display screen as the initial value of the reference point;

detecting the position of the operator'"'"'s eye in the space and detecting the three-dimensional movement of the operator; and

moving the reference point from the initial value according to the three-dimensional movement of the operator'"'"'s eye and resetting the pyramid defined by the moved reference point and the display screen as the pointing space.

14. The method of operation input according to claim 13, comprising:

recognizing that the detected object in the pointing space is a predetermined pointing device, and

calculating the coordinate position of the objective point on the display screen based on the position of the detected object which is recognized as a predetermined pointing device.

15. An operation input device comprising:

pointing space setting means for setting a virtual pointing space of a shape defined by connecting, as a base of the pointing space, an area of a deployed plane including a display screen and a coplanar, extended plane outside the display screen and, as an apex of the pointing space, a predetermined reference point located at a position spaced apart from the deployed plane where an operator is expected to be located; and

pointing action detection means for detecting a pointing action performed in the pointing space and generating a corresponding operation instruction;

wherein the pointing space setting means divides at least part of the extended plane into one or more areas and sets the respective areas as pointing areas, and the pointing action detection means generates a command execution instruction corresponding to the pointing area pointed to by the pointing action

wherein the pointing action detection means comprises;

position detection means for detecting the three-dimensional position of an object in the pointing space;

pointing device recognizing means for recognizing whether the object in the pointing space, which is detected by the position detection means, is a predetermined pointing device;

coordinate detection means for detecting the coordinate position of an objective point on the deployed plane using a mapping relation based on the detected position of the object in the pointing space, where the coordinate detection means detects the coordinate position of the objective point on the deployed plane based on the position of the object recognized as the pointing device by the pointing device recognizing means in the pointing space;

movement detection means for detecting the three-dimensional movement of the object in the pointing space, where the movement detection means detects the three-dimensional movement of the object recognized as the pointing device by the pointing device recognizing means; and

determination means for determining the contents of the operation input based on the detected coordinate position of the objective point on the deployed plane and the detected three-dimensional movement of the object and generating a corresponding operation instruction; and

wherein the pointing space setting means sets a position corresponding to an apex of a pyramid containing both of the deployed plane and the plane obtained by connecting the positions of the pointing device when pointing to the corners of the display screen in a space as cross-sections as an initial value, and causes the reference point to move from the initial value according to the three-dimensional movement of the operator'"'"'s eye, which is detected by the movement detection means.

16. The operation input device according to claim 15, comprising display control means for scrolling the screen to move a point pointed to by the pointing action in a dragged direction by a dragged amount when the pointing action detection means detects that a drag action is performed.

17. The operation input device according to claim 15, comprising display control means for scrolling the screen to move a point pointed to by the pointing action to a predetermined position in the display screen when the pointing action detection means detects that a click action is performed.

18. An operation input device comprising:

position detection means for detecting the three-dimensional position of an object in a space;

pointing device recognizing means for recognizing whether an object in the space, which is detected by the position detection means, is a predetermined pointing device;

pointing space setting means for setting a virtual pointing space of a pyramid shape defined by connecting a deployed plane including a display screen and a coplanar, extended plane outside the display screen as a base of the pointing space and a predetermined reference point located at a position spaced apart from the deployed plane where an operator is expected to be located as an apex of the pointing space;

coordinate detection means for detecting a coordinate position of an objective point on the deployed plane using a mapping relation based on the position in the pointing space of the object which is recognized as the pointing device by the pointing device recognizing means;

movement detection means for detecting the three-dimensional movement in the pointing space of the object which is recognized as the pointing device by the pointing device recognizing means;

pointing plane setting means for setting as a pointing plane a spatial plane obtained by mapping the coordinates of the deployed plane to a position in the pointing space that is spaced apart from the deployed plane where the object exists;

pointing action detection means for detecting a pointing action performed on the pointing plane and generating a corresponding operation instruction; and

determination means for determining the contents of the operation input based on the coordinate position of the objective point on the deployed plane, which is detected by the coordinate detection means, and the three-dimensional movement of the object detected by the movement detection means;

wherein when the pointing device moves toward the display screen at a speed faster than a predetermined speed within a range of a predetermined coordinate position of the deployed plane in the pointing space, the determination means determines that a pointing start action is performed.

19. The operation input device according to claim 18, wherein the pointing plane is located between the deployed plane and the operator, includes the position of a pointing device when the operator performs the pointing action, and is a plane along which the track of the pointing device may follow when the operator moves the pointing device in a natural manner, and

the pointing plane setting means sets the pointing plane every time the pointing action is performed.

20. A method of operation input comprising:

setting a virtual pointing space of a shape defined by connecting, as a base of the pointing space, an area of a deployed plane including a display screen and a coplanar, extended plane outside the display screen and, as an apex of the pointing space, a predetermined reference point located at a position spaced apart from the deployed plane where an operator is expected to be located; and

detecting a pointing action performed in the pointing space and generating a corresponding operating instruction;

wherein the setting act divides at least part of the extended plane into one or more areas and sets the respective areas as pointing areas, and the detecting act generates a command execution instruction corresponding to the pointing area pointed to by the pointing action;

wherein the detecting act comprises;

detecting the three-dimensional position of an object in the pointing space;

recognizing whether the object detected in the pointing space is a predetermined pointing device;

detecting the coordinate position of an objective point on the deployed plane using a mapping relation based on the detected position of the object recognized as the pointing device in the pointing space;

detecting the three-dimensional movement of the object recognized as the pointing device in the pointing space; and

determining the contents of the operation input based on the detected coordinate position of the objective point on the deployed plane and the detected three-dimensional movement of the object and generating a corresponding operation instruction; and

wherein the setting act sets a position corresponding to an apex of a pyramid containing both of the deployed plane and the plane obtained by connecting the positions of the pointing device when pointing to the corners of the deployed plane in a space as cross-sections as an initial value, and causes the reference point to move from the initial value according to the detected three-dimensional movement of the operator'"'"'s eye.

21. The method of operation input according to claim 20, further comprising:

detecting the three-dimensional position of an object located in the pointing space;

detecting the three-dimensional movement of the object in the pointing space and calculating the coordinate position of an objective point on the deployed plane using a mapping relation based on the position of the object in the pointing space; and

determining the contents of the operation input based on the three-dimensional movement of the object and the coordinate position of the objective point on the deployed surface corresponding to the position of the object, and generating a corresponding operating instruction.

22. A method of operation input comprising:

detecting the three-dimensional position of an object in a space;

setting a virtual pointing space of a shape defined by connecting a deployed plane including a display screen and a coplanar, extended plane outside the display screen as a base of the pointing space and a predetermined reference point located at a position spaced apart from the deployed plane where an operator is expected to be located as an apex of the pointing space,

detecting the coordinate position of an objective point on the deployed plane using a mapping relation based on the detected position of the object in the pointing space;

detecting the three-dimensional movement of the object in the pointing space;

setting as a pointing plane a spatial plane obtained by mapping the coordinates of the deployed plane to a position in the pointing space that is spaced apart from the deployed plane where the object exists;

detecting a pointing action performed on the pointing plane and generating a corresponding operation instruction; and

determining the contents of the operation input based on the detected coordinate position of the objective point on the deployed plane and the detected three-dimensional movement of the object;

wherein the pointing plane is located between the deployed plane and the operator, includes the position of the object when the operator performs the pointing action, and is a plane defined by the object when the operator moves the object in a natural manner, and the pointing plane setting act sets the pointing plane every time when the pointing action is performed.