Operation recognition system enabling operator to give instruction without device operation

• US 7,343,026 B2
• Filed: 02/24/2004
• Issued: 03/11/2008
• Est. Priority Date: 02/24/2003
• Status: Expired due to Fees

• Current Assignee: Toshiba Medical Systems Corporation (Canon Inc.)
• Original Assignee: Toshiba Medical Systems Corporation (Canon Inc.), Kabushiki Kaisha Toshiba (Toshiba Corporation)
• Inventors: Sato, Kei, Niwa, Kenichi
• Primary Examiner(s): AZARIAN, SEYED H
• Application Number:
  US10/784,164
• Publication Number:
  US 20040242988A1
• Time in Patent Office: 1,477 Days
• Field of Search: 382/103, 382/106, 382/107, 382/117, 382/118, 382/154, 382/168, 382/181, 382/203, 382/232, 382/274, 382/276, 382/285, 382/291, 382/305, 382/318, 382/100, 382/128, 382/129, 382/130, 382/131, 382/132, 382/133, 382/189, 382/295, 382/256, 715/862, 715/757, 345/427, 345/419, 345/7, 348/419, 348/419.1, 433/24, 378/20, 378/21
• US Class Current: 382/103
• CPC Class Codes: A61B 6/467 characterised by special input means
• CPC Class Codes: A61B 6/469 for selecting a region of interest [ROI]
• CPC Class Codes: G01R 33/28 Details of apparatus provided for in groups G01R33/44 - G01R33/64
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386

Abstract

An operation recognition system including an object to be operated, at least one camera, a processor, and a controller. At least one camera is configured to acquire image data of an operator in a predetermined period of time, and a processor is configured to perform recognition processing on the acquired image data. Upon processing the image data, the processor is configured to define a virtual plane in between the object to be operated and the operator. The processor is configured to then perform a process to determine if a predetermined part of the operator penetrates the virtual plane based on the recognition processing. Based on this determination, the controller is configured to control the object to be operated.

Claims

1. An operation recognition system, comprising:

an object to be operated;

at least one camera configured to acquire image data of an operator, at least a predetermined number of image data being acquired in a predetermined time period;

a processor configured to perform recognition processing on the acquired image data, so as to define a first virtual plane in between the object and the operator, and to determine if a predetermined part of the operator penetrates the first virtual plane by more than a variable predetermined length; and

a controller configured to control the object based on the determination.

2. The system according to claim 1, wherein the processor is configured to recognize a predetermined motion of the operator and the controller is operative when the processor recognizes the predetermined motion of the operator.

3. The system according to claim 1, wherein the processor is configured to recognize a predetermined voice of the operator and the controller is operative when the processor recognizes the predetermined voice of the operator.

4. The system according to claim 1, wherein the processor is configured to recognize an operational authority of the operator based on the recognition processing.

5. The system according to claim 1, wherein the processor is further configured to detect a part of the first virtual plane penetrated by the predetermined part of the operator, and wherein the controller controls the object based on the detection.

6. The system according to claim 1, wherein, when the processor is configured to recognize an eye of the operator as a view point, and to define the first virtual plane between the object and the view point.

7. The system according to claim 1, wherein, when the processor is configured to recognize a length of an operator'"'"'s arm, and to define the first virtual plane at a position within the recognized length from the operator.

8. The system according to claim 1, wherein the first virtual plane is defined at a variable position based on the recognition.

9. The system according to claim 1, wherein the processor is configured to determine a penetration when the predetermined part of the operator penetrates the first virtual plane for more then a predetermined period.

10. The system according to claim 9, wherein the predetermined period is variable.

11. The system according to claim 1, wherein the controller is configured to control the object based on a length by and a time period in which the predetermined part of the operator penetrates the first virtual plane.

12. The system according to claim 1, wherein the object includes a display configured to display one or more operation items, and wherein the controller is configured to control the implementation of one of the operation items based on the recognition and the determination.

13. The system according to claim 12, wherein the processor is configured to recognize a predetermined motion of the operator, and the processor is configured to determine that the predetermined motion represents a click operation.

14. The system according to claim 12, wherein the processor is configured to recognize a predetermined motion of the operator, and the processor is configured to determine that the predetermined motion represents a double-click operation.

15. The system according to claim 11, wherein the processor is configured to recognize a predetermined motion of the operator, and the processor is configured to determine that the predetermined motion represents a drag operation.

16. The system according to claim 1, further comprising a second object configured to operate in response to the operation of the object.

17. The system according to claim 1, wherein the processor is further configured to define a virtual keyboard in the first virtual plane, and wherein the controller is configured to control the object in relation to a keyboard input via the virtual keyboard based on the determination.

18. The system according to claim 17, wherein the object includes a display configured to display a keyboard image corresponding to the virtual keyboard, and wherein the keyboard image distinguishes a key corresponding to the keyboard input through the virtual keyboard.

19. The system according to claim 1, wherein the processor is further configured to define a second virtual plane different from the first virtual plane and a virtual keyboard in the second virtual plane and to determine if a predetermined part of the operator penetrates the second virtual plane based on the recognition, and wherein the controller is configured to control the object in relation to a keyboard input via the virtual keyboard based on the determination.

20. The system according to claim 19, wherein the object includes a display configured to display a keyboard image corresponding to the virtual keyboard, and wherein the keyboard image distinguishes a key corresponding to the keyboard input through the virtual keyboard from the other keys.

21. The system according to claim 1, wherein the object includes a display configured to display one or more operation items, the display including a projector and a screen, wherein the first virtual plane is defined in between the screen and the operator.

22. The system according to claim 1, wherein the object includes a medical apparatus.

23. The system according to claim 22, wherein the medical apparatus comprises:

a display monitor configured to display one or more operation items; and

a radiography device configured to radiograph a specimen, wherein the controller is configured to control the implementation of one of the operation items based on the determination, and wherein the radiography device is configured to be controlled in response to the implementation of the said one of the operation items.

24. The system according to claim 23, wherein a position of the radiography device is configured to be controlled when a catheter is operated by the operator.

25. The system according to claim 1, wherein the object includes an appliance.

26. The system according to claim 1, further comprising a display configured to display an indicator indicating how much the predetermined part of the operator penetrates the first virtual plane.

27. The system according to claim 26, wherein the indicator varies in accordance with a motion of the predetermined part of the operator.

28. The system according to claim 1, wherein the object includes a display configured to display a virtual plane position change window and a position of the first virtual plane is changed in accordance with the determination.

29. An image display apparatus for displaying an image, the apparatus comprising:

a display configured to display the image in a screen;

a position detector configured to detect a position of an operator and a position of an operator'"'"'s hand;

a virtual plane defining mechanism configured to define a virtual plane in between the screen and the operator based on the position of the operator;

a determining mechanism configured to determine a virtual contact between the virtual plane and the operator'"'"'s hand based on the position of the operator'"'"'s hand and a position of the virtual plane when the operator'"'"'s hand penetrates the virtual plane by more than a variable predetermined length; and

a display control mechanism configured to control a display of the image based on determination by the determining element.

30. The apparatus according to claim 29, further comprising a shape detector configured to detect a shape of the operator'"'"'s hand;

and

a recognition mechanism configured to recognize that the operator has an operation authority when the detected shape is identical to a predetermined shape.

31. An image display apparatus for displaying an image, comprising:

a display configured to display the image in a screen;

imaging elements provided at both sides of the screen and configured to acquire image data from different directions;

a position detector configured to detect a position of an operator relative to the screen and a position of an operator part based on the acquired image data;

a virtual plane defining mechanism configured to define a virtual plane in between the screen and the operator based on the position of the operator;

a determining mechanism configured to determine a virtual contact manner between the virtual plane and the operator part based on the position of the operator part and a position of the virtual plane when the operator part penetrates the virtual plane by more than a variable predetermined length; and

a display control mechanism configured to determine an operation item corresponding to the contact manner based on determination by the determining mechanism and to control a display of the image according to the operation item.

32. A method of controlling an object, the method comprising steps of:

acquiring a predetermined number of image data of an operator in a predetermined time period;

performing recognition processing on the acquired image data;

defining a virtual plane in between the object and the operator;

determining if a predetermined part of the operator penetrates the virtual plane by more than a variable predetermined length; and

based

on the recognition processing; and

controlling the object based on the determination.

33. The method according to claim 32, wherein the acquired image data is processed to recognize an operational authority of the operator based on the recognition processing.

34. A method of controlling an object, the method comprising steps of:

acquiring a predetermined number of image data of an operator in a predetermined time period;

performing recognition processing on the acquired image data;

defining a virtual plane in between the object and the operator;

determining if a predetermined part of the operator penetrates the virtual plane based on the recognition processing when the predetermined part of operator penetrates the first virtual plane for more than a variable predetermined period; and

controlling the object based on the determination.