Interactive information processing system responsive to user manipulation of physical objects and displayed images

• US 5,732,227 A
• Filed: 07/05/1995
• Issued: 03/24/1998
• Est. Priority Date: 07/05/1994
• Status: Expired due to Fees

• Current Assignee: Hitachi, Ltd.
• Original Assignee: Hitachi, Ltd.
• Inventors: Arai, Toshifumi, Shojima, Hiroshi, Kuzunuki, Soshiro, Kitamura, Tadaaki
• Primary Examiner(s): Powell, Mark R.
• Assistant Examiner(s): KATBAB, ABDOLLAH
• Application Number:
  US08/498,238
• Time in Patent Office: 993 Days
• Field of Search: 395/155-161, 395/333, 395/334, 395/350, 395/358, 395/352, 345/163
• US Class Current: 715/775
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/033 Pointing devices displaced or positioned by the user , e.g. mice, trackballs, pens or joysticks; Accessories therefor digitisers characterised by the transducing means G06F3/041
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386

Abstract

An information processing system includes a desk with a planar display unit which also accommodates physical objects to be manipulated by the operator. The physical objects are representative of physical objects normally used in office environments when handling paper files. A camera is provided which continuously monitors the desk display surface and is coupled to an electronic computer which can be controlled by the operator by moving the physical objects and the operator hands to simulate paper processing tasks, such as mailing, faxing, filing, opening files, editing and the like. An operator driven menu is provided on the desk display surface for facilitating operator classification and storage of physical object shapes of those physical objects which are to be used.

Claims

1. An interactive information processing system responsive to human operator movement of physical objects comprising:

a discriminator for discriminating physical objects and generating an electronic physical object image signal representative of physical attributes of a physical object; and

a menu driven physical object attribute definition system for accommodating operator storage of physical attributes of a plurality of physical objects utilizing the electronic physical object image signal from the discriminator.

2. A system according to claim 1, wherein a plurality of different types of physical objects are provided which correspond to physical objects normally used in handling paper files in an office, and wherein the attribute definition system includes means for accommodating operator indication of the type of physical object represented by the physical object image signal.

3. A system according to claim 2, wherein said physical objects include objects representative of at least two of the following types of physical objects:

(a) file of papers;

(b) mail tray;

(c) fax machine;

(d) computer keyboard;

(e) computer mouse;

(f) pen;

(g) memo;

(h) marker pen;

(i) eraser;

(j) operator'"'"'s hands in an open release position; and

(k) operator'"'"'s hands in a closed condition.

4. A system according to claim 2, wherein said types of physical objects include the operator'"'"'s hands in open and closed condition, and the following other types of physical objects:

(a) file of paper;

(b) pen;

(c) eraser;

(d) mail tray;

(e) fax machine;

(f) keyboard; and

(g) mouse.

5. An information processing system according to claim 2, comprising a display surface in the form of a desk top, said physical objects being selectively placed on the display surface by the operator.

6. An information processing system according to claim 5, wherein said discriminator is a camera which continuously views the display surface during information processing operations.

7. An information processing system according to claim 5, wherein said attribution definition system includes an operator selectable menu at the desk top which menu includes operator selection areas for operator designation of a plurality of types of physical objects to be used by the operator, whereby the operator can sequentially place the physical objects to be used on the display surface and store their physical attributes, preliminarily to conducting actual information processing operations using the physical objects.

8. An information processing system according to claim 2, comprising

a file storage unit containing electronically stored information to be processed, including information from the physical attribute definition system reflective of the physical objects to be used with the system;

a link storage unit containing information reflective of a physical object being concurrently discriminated by the discriminator; and

a file operator for processing information from the file storage unit in response to operator movement of at least one of the physical objects with respect to an image display reflective of at least one other of the physical objects.

9. An information processing system according to claim 1, comprising:

a file storage unit containing electronically stored information to be processed, including information from the physical attribute definition system reflective of the physical objects to be used with the system;

a link storage unit containing information reflective of a physical object being concurrently discriminated by the discriminator; and

a file operator for processing information from the file storage unit in response to operator movement of at least one of the physical objects with respect to an image display reflective of at least one other of the physical objects.

10. An information processing system according to claim 9, wherein said file storage unit includes information reflective of predetermined distances between respective ones of the physical objects and image display reflection of the physical objects, and wherein said file operator includes means for initiating predetermined processing steps in response to movement of a physical object and an image display to positions within and without said predetermined distances.

11. An information processing system according to claim 10, wherein said information reflective of predetermined distances is indicative of characteristic point coordinates on said respective physical objects and image displays.

12. An information processing system according to claim 11, wherein said characteristic point coordinates correspond to predetermined geometric locations on said physical objects and said image displays.

13. An information processing system according to claim 11, wherein said characteristic point coordinates includes a point on a hand of the operator.

14. An information processing system according to claim 5, wherein said file operator includes hand gesture recognizing means for recognizing operator hand gestures reflecting release and hold operations for associated electronic files to be manipulated.

15. An information processing system according to claim 14, wherein said hand gesture recognizing means recognizes a closed operator hand as a hold gesture and a hand gesture with open fingers as a release gesture.

16. An information processing system according to claim 14, wherein said hand gesture recognizing means recognizes the release and hold gestures by the size of the image reflective of the distance of the hand from a camera of the discriminator.

17. An information processing apparatus comprising:

a display unit;

a camera unit for photographing an image object displayed on said display unit, and a physical object and the hands of an operator placed on said display unit;

recognition means for recognizing the physical object from an image from said camera unit;

attribute definition means for operator-interfaceable definition of a physical object based on physical attributes recognized by said recognition means and for defining an associated motion between the physical object recognized by said recognition means and said image object;

attribute association processing means for executing an associated motion defined by said attribute definition means for said image object when a comparison value between a characteristic point coordinate of the physical object recognized by said recognition means and another characteristic point coordinate of another physical object, or between characteristic point coordinates in said image object is within a predetermined value; and

means for automatically starting said attribute association processing means when said image object or said physical object is recognized to be a hold-state by gesture recognition means.

18. An information processing apparatus provided with a display unit with a display surface on a horizontal arrangement in which data can be displayed in a document sheet format, and a camera unit for photographing the hands of an operator placed on the display surface of said display unit as well as the data displayed on said display surface of said display unit, comprising:

definition file store means for storing various information processing motions each corresponding to each of various states of images representative of hand gestures; and

process execution means for executing a particular information processing operation for said data upon recognition that a distance between an image of the hand and said data is judged to be within a predetermined value according to images from said camera unit, and that an association for carrying out said particular information process operation intended by the operator is identified between the image of the hand and said definition file store means.

19. The information processing apparatus according to claim 18, wherein a set of information processing operations corresponding to an image of a physical object which is physically placed on said display surface are stored in said definition file store means, and wherein said process execution means is caused to execute a designated information processing operation corresponding to the physical object with respect to said data when said data is moved through manipulation of the image of the hand toward the physical object on the display surface as much as to come into a predetermined distance.

20. The information processing apparatus according to claim 19, wherein an icon instead of said physical object displayed on said display unit is used.

21. The information processing apparatus according to claim 19, wherein said data moved within said predetermined distance is highlighted.

22. The information processing apparatus according to claim 20, wherein said data moved within said predetermined distance is highlighted.

23. An information processing apparatus provided with a flat panel display capable of displaying a document object including plural pages each being partially overlapped comprising:

layout coordinate designation means for designating a start point coordinate and an end point coordinate for laying out said document object;

page layout coordinate calculation means for obtaining respective coordinates for laying out respective pages of said document from said start and said end point coordinates; and

layout display means for displaying respective pages of said document on respective coordinates obtained by said page layout coordinate calculation means.

24. The information processing apparatus according to claim 23, wherein a layout coordinate point obtained by said page layout coordinate calculation means is an interpolated coordinate point which is obtained by interpolating a start point coordinate, a coordinate, and between said start point and said end point coordinates.

25. The information processing apparatus according to claim 23, further comprising:

means for simultaneously and dynamically displaying a curve which links between respective layout coordinates of the document object during layout operation by said layout coordinate designation means, and said document object as well.

26. The information processing apparatus according to claim 23, wherein said layout coordinate designation means monitors and detects characteristic points of the both hands of the operator by means of a monitor camera thereby to designate respective coordinates.

27. A method of information processing using an electronic computing system and a plurality of operator manipulated physical objects corresponding to traditionally used objects for handling paper files in an office, comprising:

selecting a plurality of physical objects;

electronically imaging a first one of the physical objects, and generating a first electronic image signal reflective of the first physical object, utilizing an object attribute definition system menu to manually define the first physical object by object type and to store the first electronic image signal with its object type;

electronically imaging a second one of the physical objects and generating a second electronic image signal reflective of the second physical object, utilizing the object attribute definition system menu to manually classify the second physical object type and to store the second electronic image signal with its object type; and

subsequently processing information in a file storage unit of the electronic computing system by moving at least one of the physical objects with respect to an image display reflection of at least one other of the physical objects.

28. A method of information processing using an electronic computing system according to claim 27, wherein said physical objects include the operator'"'"'s hands in open and closed conditions and at least one of the following types of physical objects:

(a) file of papers;

(b) mail tray;

(c) fax machine;

(d) computer keyboard;

(e) computer mouse;

(f) pen;

(g) memo;

(h) marker pen;

(i) eraser; and

comprising electronically imaging and manually classifying each of the physical objects and storing their images along with their selected object type.

29. An information processing system using a user manipulated object, comprising:

a display device,

file storage means for storing electronic information,

discrimination means for discriminating an object indicated by a user,

link definition means for generating link information for displaying electronic information corresponding to the discrimination result of the object on said display device,

link storage means for storing said link information which designates the correspondence between said object and the corresponding electronic information stored in said file storage means, and

file operation means for reading the electronic information corresponding to the discrimination result of the object by said discrimination means from said file storage means on the basis of said link information and for displaying it on said display device.

30. A system according to claim 29, wherein said discriminator is operable to discriminate user hand movements.

31. A system according to claim 30, wherein said discriminator is operable to discriminate a plurality of different hand movements and to provide the link storage unit with a corresponding plurality of discrete signals which are processed by the operator to display respective different displays.

32. A system according to claim 29, wherein said discriminator includes a camera.

33. A system according to claim 29, wherein said file operator is operable to retrieve electronic files in response to placement of a physical object in a position to be discriminated by the discriminator.

34. An information processing system according to claim 29, wherein the link definition means includes:

means for displaying the discrimination result of the object by said discrimination means and the electronic information stored in said file storage means, and

means for designating the correspondence between the selected electronic information among said displayed electronic information and said discrimination result.

35. An information processing system according to claim 34, wherein the discrimination result by said discrimination means includes at least information indicating whether the indicated object identification number and said object are indicated or not.

36. An information processing system using an object according to claim 35, wherein said link definition means operates when the link information for designating the correspondence between said indicated object discrimination result and electronic information is not stored in said link storage means.

37. An information processing system according to claim 29, wherein the file operation means changes appearance of displayed information on the display device when the link definition means operates for accepting the user'"'"'s request to designate correspondence between the displayed information and the object discriminated by the discrimination means.

38. An information processing system according to claim 29, wherein the file operation means displays dialog screens for accepting the user'"'"'s confirmation when the user operates using the electronic information, said electronic information corresponding to at least one object.

39. An information processing system according to claim 29, wherein the file operation means changes appearance of electronic information on the display when the information displayed corresponds to at least one object.

40. An information processing system using a user manipulated object comprising:

a display device,

file storage means for storing electronic information,

discrimination means for discriminating an object indicated by a user,

link storage means for storing link information which designates the correspondence between said object and the electronic information stored in said file storage means, and

link definition means for displaying the electronic information corresponding to the discrimination result of the object on said display device and for displaying a dialogue screen for user-interactive acceptance of a change in said stored link information on said display device.

41. An information processing system using an object wherein said system comprises:

a display device,

file storage means for storing electronic information,

discrimination means for discriminating an object indicated by a user,

link storage means for storing link information which designates the correspondence between said object and the electronic information stored in said file storage means,

attribute discrimination means for discriminating the attribute information added to said object, and

file operation means for executing the operation and for user-interactive modification of said link information on the basis of said discriminated attribute information for the electronic information having the correspondence with said discriminated object.

42. An information processing system using an object according to claim 41, wherein the attribute information added to said object includes at least information for designating whether the electronic information corresponding to said object is allowed to be modified or not.

43. An interactive information processing system responsive to human operator movement of physical objects comprising:

a discriminator which discriminates physical objects and generates an electronic physical object image signal representative of physical attributes of a physical object; and

a menu driven physical object attribute definition system allowing operator storage of physical attributes of a plurality of physical objects utilizing the electronic physical object image signal from the discriminator.

44. An information processing system according to claim 43, further comprising:

a hand gesture recognition system which recognizes operator hand gestures reflecting operations for manipulating associated electronic files; and

a file operator which manipulates said electronic files in response to said hand gestures.

45. An interactive information processing system according to claim 44, wherein said hand gesture recognition system recognizes at least one of the hand gestures selected from the group of:

(i) a holding gesture, (ii) a releasing gesture, (iii) a movement while holding gesture, (iv) a wrist rotation while holding gesture, and (v) a thumb swing while holding gesture.

46. An information processing apparatus comprising:

a display unit;

a camera unit for photographing an object image displayed on the display unit and the hands of an operator placed on or adjacent said display unit;

a file storage unit containing image manipulation operations corresponding to respective operator hand gestures; and

a file operator which executes a respective of said image manipulation operations on said object image in response to a respective operator hand gesture photographed by said camera unit.

47. An interactive information processing system according to claim 46, wherein said image manipulation operations correspond to at least one of the hand gestures selected from the group of:

(i) a holding gesture, (ii) a releasing gesture, (iii) a movement while holding gesture, (iv) a wrist rotation while holding gesture, and (v) a thumb swing while holding gesture.

48. An interactive information processing system according to claim 46, wherein said image manipulation operations include a holding gesture where at least one of the operator'"'"'s hands is in a closed position, and a releasing gesture where at least one of the operator'"'"'s hands is in an open position.

49. A method of information processing, comprising:

displaying an object image on a display unit;

photographing said object image displayed on the display unit and the hands of an operator placed adjacent the object image;

comparing an operator hand gesture with stored image manipulation operations; and

executing a respective of said image manipulation operations on said object image in response to said operator hand gesture photographed by said camera unit.

50. An interactive information processing system according to claim 49, wherein said stored image manipulation operations correspond to at least one of the hand gestures selected from the group of:

(i) a holding gesture, (ii) a releasing gesture, (iii) a movement while holding gesture, (iv) a wrist rotation while holding gesture, and (v) a thumb swing while holding gesture.

51. An interactive information processing system according to claim 49, wherein said stored image manipulation operations include a holding gesture where at least one of the operator'"'"'s hands is in a closed position, and a releasing gesture where at least one of the operator'"'"'s hands is in an open position.

52. A method of information processing using an electronic computing system and an operator manipulated physical object, comprising:

selecting a physical object;

electronically imaging the physical object;

generating an electronic image signal reflective of the physical object; and

utilizing an object attribute definition system menu to manually define the physical object by object type and to store the electronic image signal with its object type.