Apparatus for operating objects and a method for identifying markers from digital image frame data

• US 20090122010A1
• Filed: 10/16/2008
• Published: 05/14/2009
• Est. Priority Date: 10/18/2007
• Status: Active Grant

• Current Assignee: Jun Maeda
• Original Assignee: Jun Maeda
• Inventors: MURAI, Yukiro
• Granted Patent: US 8,232,964 B2
• Time in Patent Office: Days
• US Class Current: 345/158
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

An input apparatus other than mice is provided that supports an input to a computer by fingers or a mouth. An apparatus for operating objects according to the present invention operates an object on a screen based on an imaged operating element. The apparatus includes a computing apparatus; a display apparatus connected to the computing apparatus; and an imaging apparatus connected to the computing apparatus. The imaging apparatus images a predetermined operating element. The computing apparatus displays, as a marker, an image of the imaged operating element on the display apparatus. The object on the screen of the display apparatus is operated by movement of the marker.

Claims

1. An apparatus for operating objects that operates an object on a screen based on an imaged operating element, the apparatus comprising:

a computing apparatus;

a display apparatus connected to the computing apparatus; and

an imaging apparatus connected to the computing apparatus, wherein

the imaging apparatus images a predetermined operating element and the computing apparatus displays, as a marker, an image of the imaged operating element on the display apparatus, and

the object on the screen of the display apparatus is operated by movement of the marker.

2. The apparatus for operating objects according to claim 1, wherein

the computing apparatus generates a bitmap B(x, y) from frame data imaged by the imaging apparatus, and

the computing apparatus further extracts a matter element from the frame data, represents the matter element as a set of segment elements represented by a starting point and a width, and processes the set of segment elements, and thereby generating the marker.

3. The apparatus for operating objects according to claim 2, wherein the computing apparatus displays, as a marker, a mirror image of the image of the operating element on the display apparatus.

4. The apparatus for operating objects according to claim 3, wherein the imaging apparatus images a plurality of operating elements, and markers respectively corresponding to the plurality of operating elements are caused to act on the object, whereby the object is operated.

5. The apparatus for operating objects according to claim 4, wherein

the object is a window, and

when the plurality of markers by the plurality of operating elements are brought close to each other inside the window,

if the window is not at front of a display screen then the window is brought to the front, and if the window is at the front of the display screen then the window is dragged.

6. The apparatus for operating objects according to claim 4, wherein

the object is a window, and

when the plurality of markers by the plurality of operating elements are brought close to one side of the window from an outside of the window,

the window moves in parallel to a coordinate axis of a display screen.

7. The apparatus for operating objects according to claim 4, wherein

the object is a window, and

when, among the plurality of markers by the plurality of operating elements, one or more markers are brought close to a center from an outside of each of a pair of opposite sides of the window,

the window is reduced in a direction intersecting the pair of opposite sides and toward the center.

8. The apparatus for operating objects according to claim 4, wherein

the object is a window, and

when, among the plurality of markers by the plurality of operating elements, one or more markers are caused to extend from an inside of each of a pair of opposite sides of the window, so as to be in contact with each of the sides,

the window extends in a direction intersecting the pair of sides and going away from a center.

9. The apparatus for operating objects according to claim 4, wherein

the object is a window, and

when, among three or more of the markers by three or more of the operating elements, one or more markers are caused to be present in contact with an outside of each of three or four sides of the window and the markers are caused to move with a same velocity vector,

the window moves in parallel.

10. The apparatus for operating objects according to claim 4, wherein

the object is a window, and

when, among three or more of the markers by three or more of the operating elements, one or more markers are caused to be present in contact with an inside of each of three or four sides of the window and the markers are caused to move in a direction going away from a center,

a width and a height of the window extend in accordance with movement of the markers.

11. The apparatus for operating objects according to claim 4, wherein

in the computing apparatus, a relationship between the markers and the object is defined in advance for each action content, and furthermore, the plurality of action contents are ordered in advance from basic to complex actions, and

in frame data processing performed by the computing apparatus, it is checked along the order whether the markers and the object satisfy the relationships defined in advance or not, and when a number of terms in the ordering obtained as it is recognized that they do not satisfy the relationships is put as “

k”

, an action content corresponding to a (k−

1)th one in the ordering is an action performed on the object by the markers.

12. The apparatus for operating objects according to claim 11, wherein

in the frame data processing performed by the computing apparatus, a plurality of objects on the screen of the display apparatus are ordered from front to back of the screen, and

which object an action of the markers targets for is checked along the ordering.

13. The apparatus for operating objects according to claim 12, wherein a number of types of action performed on an object other than an object in forefront on the screen of the display apparatus is less than a number of types of action performed on the object in the forefront on the screen of the display apparatus.

14. A method for identifying markers from digital image frame data, the method comprising the steps of:

generating a bitmap B (x, y) in a pixel unit, focusing on a predetermined color, using digital image frame data;

representing presence of a bit at each y (vertical axis) value by a segment element (x, y, w) having a starting point (x) and a width (w);

determining whether there is a contact relationship between the segment elements and putting together segment elements having a contact relationship into a matter element; and

removing noise from matter elements.

15. The method according to claim 14, wherein

the step of generating a bitmap B(x, y) includes the steps of:

converting, for each pixel, the digital image frame data from RGB values to HSV values to obtain H (Hue), S (Saturation), and V (Value) values; and

comparing preset HSV values (H₀, S₀, V₀) with the HSV values of individual pixels (x, y) for each of H, S, and V and extracting only those pixels whose each difference in H, S, and V is not more than their respective threshold values as marker candidate portions and turning on bits for the pixels and turning off bits for other pixels.

16. The method according to claim 14, wherein

the step of removing noise from matter elements includes the steps of:

removing a matter element whose area has a larger value than a first predetermined value or a smaller value than a second predetermined value;

removing a matter element having a lower precise circle rate than a predetermined value;

removing beaded noise;

contrasting each matter element on a current frame with each recorded matter element on an immediately preceding frame and connecting, as identical matter elements, matter elements whose center of gravity coordinates are same or whose center of gravity coordinate difference is not more than a predetermined value; and

obtaining a velocity vector of each matter element from a difference in center of gravity coordinates between identical matter elements in an immediately preceding frame or several preceding frames and in a current frame.