Haptic information presentation system and method

• US 10,216,278 B2
• Filed: 10/14/2016
• Issued: 02/26/2019
• Est. Priority Date: 11/20/2003
• Status: Active Grant

• Current Assignee: National Institute of Advanced Industrial Science and Technology (Government of Japan), Murata Manufacturing Co Limited
• Original Assignee: National Institute of Advanced Industrial Science and Technology (Government of Japan), Miraisens, Inc. (Murata Manufacturing Co Limited)
• Inventors: Nakamura, Norio, Fukui, Yukio, Sakai, Masataka
• Primary Examiner(s): Lim, Steven
• Assistant Examiner(s): Littlejohn, Jr., Mancil
• Application Number:
  US15/293,609
• Publication Number:
  US 20170031443A1
• Time in Patent Office: 865 Days
• Field of Search: 3404701, 3404702, 345156, 345161, 345163, 345167, 345157, 345168, 345173, 345179
• CPC Class Codes: A63F 13/285 Generating tactile feedback signals via the game input device, e.g. force feedback
• CPC Class Codes: G06F 2203/013 Force feedback applied to a game
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/041 Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G08B 6/00 Tactile signalling systems, e.g. personal calling systems
• CPC Class Codes: G09G 5/00 Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T
• CPC Class Codes: G10K 15/04 Sound-producing devices G10K15/02 takes precedence
• CPC Class Codes: H04B 3/36 Repeater circuits H04B3/58 takes precedence

Abstract

A system and method are disclosed in which in a conventional non-grounding man-machine interface having no reaction base on the human body and for giving the existence of a virtual object and the impact force of a collision to a person, a haptic sensation of a torque, a force and the like can be continuously presented in the same direction, which cannot be presented by only the physical characteristic of a haptic sensation presentation device. In a haptic presentation device, the rotation velocity of at least one rotator in the haptic presentation device is controlled by a control device, and a vibration, a force or a torque as the physical characteristic is controlled, so that the user is made to conceive various haptic information of the vibration, force, torque or the like.

Claims

1. A method of controlling a motion of a virtual object to present a haptic information to (1) a medium device or a medium apparatus, and/or (2) a user, as if (1) the medium device or the medium apparatus, and/or (2) the user operates a real object, the method comprising:

applying a haptic sensory characteristic and/or an illusion of (1) the medium device or the medium apparatus, and/or (2) the user to the virtual object; and

controlling a physical quantity and/or a sensory quantity utilizing the haptic sensory characteristic and/or the illusion of (1) the medium device or the medium apparatus and/or (2) the user;

wherein the haptic sensory characteristic and/or an illusion of (1) the medium device or the medium apparatus, and/or (2) the user controls a physical quantity of a driving device provided in a haptic presentation device, and the haptic presentation device comprises at least one of an input device, an output device, a display device, video member, or the medium device, or the medium apparatus;

wherein the haptic sensory characteristic comprises at least one of a nonlinear characteristic, a hysteresis characteristic, a masking characteristic, a threshold characteristic, or a muscle reflection;

wherein the control of the motion of the virtual object comprises at least one of (1) controlling a frequency and intensity of vibration and/or vibration sensation, (2) utilizing a masking effect to continuously or intermittently control at least one haptic information of vibration sensation, force sensation, and a torque sensation in an arbitrary direction, (3) analyzing input information from a controller to control a motor so as to control a movement in a direction of haptic presentation device, (4) analyzing input information from an acceleration sensor to control a motor, (5) controlling rotation of an eccentric rotator, (6) adjusting a spatial intensity distribution of a resting force, (7) accelerating and/or decelerating a velocity of a body having an inertia moment, (8) rotating a rotation body around an axis orthogonal to the rotation axis, or (9) controlling a posture of a gyroscope.

2. The method according to claim 1, wherein the haptic sensory characteristic comprises (i) a physical quantity provided to at least one of the medium device, the medium apparatus and the user by one or two way out of being given to at least one of the medium device, the medium apparatus, and the user and being bought by motion of at least one of the medium device, the medium apparatus, and the user;

and (ii) a sensory quantity to be presented to at least one of the medium device, the medium apparatus, and the user, wherein the haptic sensory characteristic represents a relationship between the physical quantity to be applied to at least one of the medium device, the medium apparatus, and the user, and the sensory quantity to be perceived by at least one of the medium device, the medium apparatus, and the user.

3. The method according to claim 1, wherein the sensory quantity comprises at least one of (i) a sensory quantity where an integral value of the sensory quantity does not become zero even if a further integral value of the physical quantity becomes zero;

and/or (ii) a sensory quantity that does not exist physically.

4. The method according to claim 1, wherein the sensory quantity characteristic comprises (i) one or more of physical quantity being given to the medium device or medium apparatus, and physical quantity being brought by operation of the medium device or the medium apparatus, or (ii) a sensory quantity to be presented to the medium device or the medium apparatus.

5. The method according to claim 1, wherein the motion of the virtual object is controlled to present the haptic information on a surface of a display member.

6. The method according to claim 1, wherein the motion of the virtual object is controlled to present the haptic information at a space around a surface of a display member.

7. The method according to claim 1, wherein the motion of the virtual object is controlled to present the haptic information around a surface of a display member.

8. The method according to claim 1, wherein the motion of the virtual object is controlled to present the haptic information around of a display member.

9. The method according to claim 1, wherein the controlling of the motion of the virtual object comprises controlling at least one of a motor, a vibration, a torque, a sound, a video, an air pressure, a force, a piezoelectric element, a magnet, a speaker, a rotator, an eccentric rotator, or a rotation unit.

10. The method according to claim 1, wherein the controlling of the motion of the virtual object comprises:

analyzing input information from at least one of a posture sensor or a touch panel sensor; and

controlling a motor to give a movement in a direction such that a feeling such as to operate an actual scroll wheel or joystick is presented in spite of a virtual wheel on the touch panel.

11. The method according to claim 1, wherein the haptic information comprises at least one physical quantity of a motion due to at least one of a button operation, an input operation, a switch operation, a dial operation, an operation with respect to a display member, a scroll operation with respect to the display member, a touch operation with respect to the display member, an acceleration or deceleration operation with respect to the display member, an acceleration or deceleration operation with respect to the display member, an operation for grasping the display member, an operation for rubbing the display member, an operation for pressing the display member, an operation for pulling the display member, an operation for moving the display member, an operation with respect to a virtual object, a scroll operation with respect to the virtual object, a touch operation with respect to the virtual object, an acceleration or deceleration operation with respect to the virtual object, an operation for grasping the virtual object, an operation for rubbing the virtual object, an operation for pressing the virtual object, or an operation for pulling the virtual object based on a haptic sensory characteristic of an operator.

12. A method of controlling a motion of a virtual object to present a haptic information to (1) a medium device or a medium apparatus, and/or (2) a user, as if (1) the medium device or the medium apparatus, and/or (2) the user operates a real object, the method comprising:

applying a haptic sensory characteristic and/or an illusion of (1) the medium device or the medium apparatus, and/or (2) the user to the virtual object;

sensing by a sensor the physical quantity which operates the virtual object; and

controlling a physical quantity and/or a sensory quantity utilizing the haptic sensory characteristic and/or the illusion of (1) the medium device or the medium apparatus, and/or (2) the user;

wherein the haptic sensory characteristic and/or an illusion of (1) the medium device or medium apparatus, and/or (2) the user controls a physical quantity of a driving device provided in a haptic presentation device, and the haptic presentation device comprises at least one of an input device, an output device, a display device, video member, or the medium device or the medium apparatus;

wherein the haptic sensory characteristic comprises at least one of a nonlinear characteristic, a hysteresis characteristic, a masking characteristic, a threshold characteristic, or a muscle reflection;

wherein the control of the motion of the virtual object comprises at least one of (1) controlling a frequency and intensity of vibration and/or vibration sensation, (2) utilizing a masking effect to continuously or intermittently control at least one haptic information of vibration sensation, force sensation, and a torque sensation in an arbitrary direction, (3) analyzing input information from a controller to control a motor so as to control a movement in a direction of haptic presentation device, (4) analyzing input information from an acceleration sensor to control a motor, (5) controlling rotation of an eccentric rotator, (6) adjusting a spatial intensity distribution of a resting force, (7) accelerating and/or decelerating a velocity of a body having an inertia moment, (8) rotating a rotation body around an axis orthogonal to the rotation axis, or (9) controlling a posture of a gyroscope.

13. The method according to claim 12, wherein the haptic sensory characteristic comprises (i) a physical quantity provided to at least one of the medium device, the medium apparatus, and the user by one or two way out of being given to at least one of the medium device, the medium apparatus, and the user and being bought by motion of at least one of the medium device, the medium apparatus, and the user;

and (ii) a sensory quantity to be presented to at least one of the medium device, the medium apparatus, and the user, wherein the haptic sensory characteristic represents a relationship between the physical quantity to be applied to at least one of the medium device, the medium apparatus, and the user, and the sensory quantity to be perceived by at least one of the medium device, the medium apparatus, and the user.

14. The method according to claim 12, wherein the sensory quantity comprises at least one of (i) a sensory quantity where an integral value of the sensory quantity does not become zero even if a further integral value of the physical quantity becomes zero;

and/or (ii) a sensory quantity that does not exist physically.

15. The method according to claim 12, wherein the sensory quantity characteristic comprises (i) one or more of physical quantity being given to the medium device or the medium apparatus, and physical quantity being brought by operation of the medium device or the medium apparatus, or (ii) a sensory quantity to be presented to the medium device or the medium apparatu]s.

16. The method according to claim 12, wherein the motion of the virtual object is controlled to present the haptic information on a surface of a display member.

17. The method according to claim 12, wherein the motion of the virtual object is controlled to present the haptic information at a space around a surface of a display member.

18. The method according to claim 12, wherein the motion of the virtual object is controlled to present the haptic information around a surface of a display member.

19. The method according to claim 12, wherein the motion of the virtual object is controlled to present the haptic information around of a display member.

20. The method according to claim 12, wherein the controlling of the motion of the virtual object comprises controlling at least one of a motor, a vibration, a torque, a sound, a video, an air pressure, a force, a piezoelectric element, a magnet, a speaker, a rotator, an eccentric rotator, or a rotation unit.

21. The method according to claim 12, wherein the controlling of the motion of the virtual object comprises:

analyzing input information from at least one of a posture sensor or a touch panel sensor; and

controlling a motor to give a movement in a direction such that a feeling such as to operate an actual scroll wheel or joystick is presented in spite of a virtual wheel on the touch panel.

22. The method according to claim 12, wherein the haptic information comprises at least one physical quantity of a motion due to at least one of a button operation, an input operation, a switch operation, a dial operation, an operation with respect to a display member, a scroll operation with respect to the display member, a touch operation with respect to the display member, an acceleration or deceleration operation with respect to the display member, an operation for grasping the display member, an operation for rubbing the display member, an operation for pressing the display member, an operation for pulling the display member, an operation for moving the display member, an operation with respect to a virtual object, a scroll operation with respect to the virtual object, a touch operation with respect to the virtual object, an acceleration or deceleration operation with respect to the virtual object, an operation for grasping the virtual object, an operation for rubbing the virtual object, an operation for pressing the virtual object, or an operation for pulling the virtual object based on a haptic sensory characteristic of an operator.

23. A method of controlling a motion of a virtual object to present a haptic information to (1) a medium device or a medium apparatus, and/or (2) a user, as if (1) the medium device or the medium apparatus, and/or (2) the user operates a real object, the method comprising:

applying a haptic sensory characteristic and/or an illusion of (1) the medium device or the medium apparatus, and/or (2) the user to the virtual object;

sensing by a sensor the physical quantity which operates the virtual object;

controlling a physical quantity and/or a sensory quantity utilizing the haptic sensory characteristic and/or the illusion of (1) the medium device or the medium apparatus, and/or (2) the user;

wherein the haptic sensory characteristic and/or an illusion of (1) the medium device or the medium apparatus, and/or (2) the user controls a physical quantity of a driving device provided in a haptic presentation device, and the haptic presentation device comprises at least one of an input device, an output device, a display device, video member, or the medium device or the medium apparatus;

driving a data processing device by utilizing the haptic sensory characteristic and/or an illusion of (1) the medium device or the medium apparatus, and/or (2) the user;

wherein the driving of the data processing device is performed by a controller built in a haptic presentation device in response to receiving a signal from an information processing device;

wherein the haptic sensory characteristic comprises at least one of a nonlinear characteristic, a hysteresis characteristic, a masking characteristic, a threshold characteristic, or a muscle reflection;

wherein the control of the motion of the virtual object comprises at least one of (1) controlling a frequency and intensity of vibration and/or vibration sensation, (2) utilizing a masking effect to continuously or intermittently control at least one haptic information of vibration sensation, force sensation, and a torque sensation in an arbitrary direction, (3) analyzing input information from a controller to control a motor so as to control a movement in a direction of haptic presentation device, (4) analyzing input information from an acceleration sensor to control a motor, (5) controlling rotation of an eccentric rotator, (6) adjusting a spatial intensity distribution of a resting force, (7) accelerating and/or decelerating a velocity of a body having an inertia moment, (8) rotating a rotation body around an axis orthogonal to the rotation axis, or (9) controlling a posture of a gyroscope.

24. The method according to claim 23, wherein the haptic sensory characteristic comprises (i) a physical quantity provided to at least one of the medium device or the medium apparatus, and the user by one or two way out of being given to at least one of the medium device or the medium apparatus, and the user and being bought by motion of at least one of the medium device or the medium apparatus, and the user;

and (ii) a sensory quantity to be presented to at least one of the medium device or the medium apparatus, and the user, wherein the haptic sensory characteristic represents a relationship between the physical quantity to be applied to at least one of the medium device or the medium apparatus, and the user, and the sensory quantity to be perceived by at least one of the medium device or the medium apparatus, and the user.

25. The method according to claim 23, wherein the sensory quantity comprises at least one of (i) a sensory quantity where an integral value of the sensory quantity does not become zero even if a further integral value of the physical quantity becomes zero;

and/or (ii) a sensory quantity that does not exist physically.

26. The method according to claim 23, wherein the sensory quantity characteristic comprises (i) one or more of physical quantity being given to the medium device or the medium apparatus, and physical quantity being brought by operation of the medium device or the medium apparatus, or (ii) a sensory quantity to be presented to the medium device or the medium apparatus.

27. The method according to claim 23, wherein the motion of the virtual object is controlled to present the haptic information on a surface of a display member.

28. The method according to claim 23, wherein the motion of the virtual object is controlled to present the haptic information at a space around a surface of a display member.

29. The method according to claim 23, wherein the motion of the virtual object is controlled to present the haptic information around a surface of a display member.

30. The method according to claim 23, wherein the motion of the virtual object is controlled to present the haptic information around of a display member.

31. The method according to claim 23, wherein the controlling of the motion of the virtual object comprises controlling at least one of a motor, a vibration, a torque, a sound, a video, an air pressure, a force, a piezoelectric element, a magnet, a speaker, a rotator, an eccentric rotator, or a rotation unit.

32. The method according to claim 23, wherein the controlling of the motion of the virtual object comprises:

analyzing input information from at least one of a posture sensor or a touch panel sensor; and

controlling a motor to give a movement in a direction such that a feeling such as to operate an actual scroll wheel or joystick is presented in spite of a virtual wheel on the touch panel.

33. The method according to claim 23, wherein the haptic information comprises at least one physical quantity of a motion due to at least one of a button operation, an input operation, a switch operation, a dial operation, an operation with respect to a display member, a scroll operation with respect to the display member, a touch operation with respect to the display member, an acceleration or deceleration operation with respect to the display member, an operation for grasping the display member, an operation for rubbing the display member, an operation for pressing the display member, an operation for pulling the display member, an operation for moving the display member, an operation with respect to a virtual object, a scroll operation with respect to the virtual object, a touch operation with respect to the virtual object, an acceleration or deceleration operation with respect to the virtual object, an operation for grasping the virtual object, an operation for rubbing the virtual object, an operation for pressing the virtual object, or an operation for pulling the virtual object based on a haptic sensory characteristic of an operator.

34. A method for presenting a haptic information comprising haptic sensory characteristic and/or a force that does not exist physically to (1) a medium device or a medium apparatus, and/or (2) a user by controlling a physical quantity and/or a sensory quantity utilizing the haptic sensory characteristic and/or an illusion of (1) the medium device or the medium apparatus, and/or (2) the user, the method comprising:

sensing by a sensor at least one physical quantity of (1) a finger, (2) a palm, (3) a hand, (4) a leg, (5) a head, (6) a body, (7) an operating device, and (8) a man-machine interface which operates a virtual object; and

controlling a motion of the virtual object to present the haptic information to (1) the medium device or the medium apparatus, and/or (2) the user as if (1) the medium device or the medium apparatus, and/or (2) the user operates a real object by applying the haptic sensory characteristic of (1) the medium device or the medium apparatus, and/or (2) the user to the virtual object;

wherein the haptic sensory characteristic and/or an illusion of (1) the medium device or medium apparatus, and/or (2) the user is obtained by using at least one of a nonlinear characteristic, a hysteresis characteristic, a masking characteristic, a threshold characteristic, or a muscle reflection;

wherein the haptic sensory characteristic and/or an illusion of (1) the medium device or the medium apparatus, and/or (2) the user controls a physical quantity of a driving device provided in a haptic presentation device, and the haptic presentation device comprises at least one of an input device, an output device, a display device, video member, or the medium device or the medium apparatus;

wherein the control of the motion of the virtual object comprises at least one of (1) controlling a frequency and intensity of vibration and/or vibration sensation, (2) utilizing a masking effect to continuously or intermittently control at least one haptic information of vibration sensation, force sensation, and a torque sensation in an arbitrary direction, (3) analyzing input information from a controller to control a motor so as to control a movement in a direction of haptic presentation device, (4) analyzing input information from an acceleration sensor to control a motor, (5) controlling rotation of an eccentric rotator, (6) adjusting a spatial intensity distribution of a resting force, (7) accelerating and/or decelerating a velocity of a body having an inertia moment, (8) rotating a rotation body around an axis orthogonal to the rotation axis, or (9) controlling a posture of a gyroscope.

35. The method according to claim 34, wherein the haptic sensory characteristic comprises (i) a physical quantity provided to at least one of the medium device or the medium apparatus, and the user by one or two way out of being given to at least one of the medium device or the medium apparatus, and the user and being bought by motion of at least one of the medium device or the medium apparatus, and the user;

and (ii) a sensory quantity to be presented to at least one of the medium device or the medium apparatus, and the user, wherein the haptic sensory characteristic represents a relationship between the physical quantity to be applied to at least one of the medium device or the medium apparatus, and the user, and the sensory quantity to be perceived by at least one of the medium device or the medium apparatus, and the user.

36. The method according to claim 34, wherein the sensory quantity comprises at least one of (i) a sensory quantity where an integral value of the sensory quantity does not become zero even if a further integral value of the physical quantity becomes zero;

and/or (ii) a sensory quantity that does not exist physically.

37. The method according to claim 34, wherein the sensory quantity characteristic comprises (i) one or more of physical quantity being given to the medium device or the medium apparatus, and physical quantity being brought by operation of the medium device or the medium apparatus, or (ii) a sensory quantity to be presented to the medium device or the medium apparatus.

38. The method according to claim 34, wherein the motion of the virtual object is controlled to present the haptic information on a surface of a display member.

39. The method according to claim 34, wherein the motion of the virtual object is controlled to present the haptic information at a space around a surface of a display member.

40. The method according to claim 34, wherein the motion of the virtual object is controlled to present the haptic information around a surface of a display member.

41. The method according to claim 34, wherein the motion of the virtual object is controlled to present the haptic information around of a display member.

42. The method according to claim 34, wherein the controlling of the motion of the virtual object comprises controlling at least one of a motor, a vibration, a torque, a sound, a video, an air pressure, a force, a piezoelectric element, a magnet, a speaker, a rotator, an eccentric rotator, or a rotation unit.

43. The method according to claim 34, wherein the controlling of the motion of the virtual object comprises:

analyzing input information from at least one of a posture sensor or a touch panel sensor; and

controlling a motor to give a movement in a direction such that a feeling such as to operate an actual scroll wheel or joystick is presented in spite of a virtual wheel on the touch panel.

44. The method according to claim 34, wherein the haptic information comprises at least one physical quantity of a motion due to at least one of a button operation, an input operation, a switch operation, a dial operation, an operation with respect to a display member, a scroll operation with respect to the display member, a touch operation with respect to the display member, an acceleration or deceleration operation with respect to the display member, an operation for grasping the display member, an operation for rubbing the display member, an operation for pressing the display member, an operation for pulling the display member, an operation for moving the display member, an operation with respect to a virtual object, a scroll operation with respect to the virtual object, a touch operation with respect to the virtual object, an acceleration or deceleration operation with respect to the virtual object, an operation for grasping the virtual object, an operation for rubbing the virtual object, an operation for pressing the virtual object, or an operation for pulling the virtual object based on a haptic sensory characteristic of an operator.