METHOD FOR PROVIDING HUMAN INPUT TO A COMPUTER

• US 20090300531A1
• Filed: 07/10/2009
• Published: 12/03/2009
• Est. Priority Date: 06/29/1995
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Pryor, Timothy R.
• Granted Patent: US 8,228,305 B2
• Time in Patent Office: Days
• US Class Current: 715/764
• CPC Class Codes: G06F 2203/04109 FTIR in optical digitiser, i.e. touch detection by frustrating the total internal reflection within an optical waveguide due to changes of optical properties or deformation at the touch location
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386

Abstract

The invention provides a method for providing human input to a computer which allows a user to interact with a display connected to the computer. The method includes the steps of placing a first target on a first portion of the user'"'"'s body, using an electro-optical sensing means, sensing data related to the location of the first target and data related to the location of a second portion of the user'"'"'s body, the first and second portions of the user'"'"'s body being movable relative to each other, providing an output of the electro-optical sensing means to the input of the computer, determining the location of the first target and the location of the second portion of the user'"'"'s body, and varying the output of the computer to the display based upon the determined locations for contemporaneous viewing by the user.

Claims

1. A computer implemented method for performing a control action on a computer system, the method comprising:

displaying a first virtual control object and a second virtual control object on a touch screen;

identifying a first touch input on the touch screen at the first control object and a second touch input on the touch screen at the second control object, the touch inputs being simultaneous;

detecting a change in the location on the touch screen of the first touch input and in the location on the touch screen of the second touch input corresponding to desired control actions of the first virtual control object and the second virtual control object, respectively, wherein the first touch input and the second touch input remain in simultaneous contact with the touch screen; and

performing an operation in response to said detecting step.

2. The method according to claim 1 wherein the operation includes displaying a change in the orientation of at least one of the first virtual control object and the second virtual control object.

3. The method according to claim 1 wherein the change in the location of the first and the second touch inputs includes a linear movement of one of the first and second touch inputs along the display screen.

4. The method according to claim 1 wherein:

the first touch input includes at least one of a finger and a thumb in contact with the touch screen; and

the second touch input includes at least one of a finger and a thumb in contact with the touch screen.

5. The method according to claim 1 wherein the first virtual control object and the second virtual control object are a first virtual throttle and a second virtual throttle, respectively.

6. The method according to claim 1 wherein the first control object controls a function of a vehicle.

7. The method according to claim 6 wherein the vehicle control panel is part of an aircraft cockpit display.

8. A computer implemented method for controlling a computer with a touch screen adapted to display visually observable data, the method comprising:

displaying a first controller and a second controller on the touch screen;

receiving a first touch input and a second touch input simultaneously, wherein the first touch input corresponds to at least one of a finger and a thumb in contact with the first controller and the second touch input corresponds to at least one of a finger and a thumb in contact with the second controller;

detecting a time-based change in the location of the first touch input and in the location of the second touch input along the touch screen corresponding to simultaneous actuation of the first controller and the second controller;

displaying a change in at least one of the position and the orientation of the first controller and the second controller in response to said detecting step; and

performing a function in response to said detecting step.

9. The method according to claim 8 wherein the time-based change in the location of the first touch input includes a linear movement along the touch screen.

10. The method according to claim 8 wherein the time-based change in the location of the first touch input includes an arcuate movement along the touch screen.

11. The method according to claim 8 wherein the first controller and the second controller are a first virtual throttle and a second virtual throttle, respectively.

12. The method according to claim 8 wherein the first control object controls a function of a vehicle.

13. The method according to claim 8 wherein the vehicle control panel is part of an aircraft cockpit display.

14. A touch interface for a computer comprising:

a display screen adapted to simultaneously display a plurality of virtual controllers on the display screen; and

a control circuit adapted to detect a plurality simultaneous touch inputs, to identify time-based changes in the locations of the plurality of simultaneous touch inputs along the display screen, and to cause the display screen to move the plurality of virtual controllers on the touch screen in response to the changing locations of the plurality of simultaneous touch inputs.

15. The touch interface of claim 14 wherein the time-based change in the location of the plurality of simultaneous touch inputs include a linear movement of one of the plurality of simultaneous touch inputs along the display screen.

16. The touch interface of claim 15 wherein the time-based change in the location of the plurality of simultaneous touch inputs includes an arcuate movement of one of the plurality of simultaneous touch inputs along the display screen.

17. The touch interface of claim 14 wherein the plurality of simultaneous touch inputs include at least one of a finger and a thumb in contact with the display screen.

18. The touch interface of claim 14 wherein the display screen is further adapted to display a change in the orientation of the plurality of virtual controllers.

20. The touch interface of claim 14 further comprising a vehicle control panel, the control circuit adapted to control a function of the vehicle control panel.

21. The touch interface of claim 20 wherein the vehicle control panel is part of an aircraft cockpit display.

19. The touch interface of claim 19 wherein the plurality of virtual controllers includes a first throttle and a second throttle.

22. A computer implemented method for providing feedback to a user in response to a touch input, the method comprising:

displaying a first virtual object on a touch screen display;

receiving a first touch input on a portion of the touch screen display overlying the first virtual object;

detecting a change in the location of the first touch input along the touch screen display; and

providing feedback to the user in response to the detecting step, wherein the feedback includes animating the first virtual object and actuating the touch screen display with a force feedback signal.

23. The computer implemented method of claim 22 wherein the first touch input corresponds to at least one of a finger and a thumb in contact with the touch screen display.

24. The computer implemented method of claim 22 wherein:

the first touch input corresponds to at least a portion of a hand in contact with the touch screen display; and

the change in the location of the first touch input corresponds to a hand stroke along a portion of the touch screen display.

25. The computer implemented method of claim 22 wherein animating the first virtual object includes depicting movement of the first virtual object along the touch screen display.

26. The computer implemented method of claim 22 wherein the first virtual object includes a virtual depiction of an animal.

27. The computer implemented method of claim 22 wherein data corresponding to the first virtual object is transmitted to a processor associated with the touch screen display via the internet.

28. The computer implemented method of claim 22 wherein the feedback includes an audible signal.

29. The computer implemented method of claim 22 wherein actuating the touch screen display includes vibrating the surface of the touch screen display.

30. The computer implemented method of claim 22 wherein the force feedback signal is directed to a portion of the touch screen display underlying the first touch input.

31. The computer implemented method of claim 22 wherein the force feedback signal is directed to a portion of the touch screen display overlying the first virtual object.

32. The computer implemented method of claim 22 further comprising:

displaying a second virtual object on a touch screen display;

receiving a second touch input on a portion of the touch screen display overlying the second virtual object, the first and second touch inputs being simultaneously in contact with the touch screen display; and

detecting a change in the location of the second touch input along the touch screen display.

33. The computer implemented method of claim 22 wherein the second touch input corresponds to at least one of a finger and a thumb in contact with the touch screen display.

34. The computer implemented method of claim 22 further including:

calculating a first centroid corresponding to the first touch input; and

determining the location of the first touch input based on the first centroid.

35. The computer implemented method of claim 34 further comprising:

receiving a second touch input on the touch screen display, the first and second touch inputs being simultaneously in contact with the touch screen display;

calculating a second centroid corresponding to the second touch input; and

determining the location of the second touch input based on the second centroid.

36. A computer implemented method for providing feedback to a user in response to a touch input, the method comprising:

displaying a first virtual object on a first portion of a touch screen display and a second virtual object on a second portion of the touch screen display;

receiving a first touch input on a portion of the touch screen display overlying the first virtual object;

receiving a second touch input on a portion of the touch screen overlying the second virtual object, the first and second touch inputs being simultaneously in contact with the touch screen display;

determining the location of the first touch input and the location of the second touch input; and

providing feedback to the user in response to the determining step, wherein the feedback includes animating at least one of the first virtual object and the second virtual object.

37. The method according to claim 36 wherein:

the first touch input corresponds to one of a finger and a thumb in contact with the touch screen display; and

the second touch input corresponds to one of a finger and a thumb in contact with the touch screen display.

38. The method according to claim 36 wherein the first touch input corresponds to a portion of a hand in contact with the touch screen display.

39. The method according to claim 36 further comprising detecting movement along the touch screen display of at least one of the first touch input and the second touch input.

40. The method according to claim 36 wherein:

the first virtual object includes a first virtual depiction of an animal; and

the second virtual object includes a second virtual depiction of an animal.

41. The method according to claim 36 further including:

calculating a first centroid corresponding to the first touch input;

determining the location of the first touch input based on the first centroid;

calculating a second centroid corresponding to the second touch input; and

determining the location of the second touch input based on the second centroid.

42. The method according to claim 36 wherein the feedback further includes an audible signal.

43. The method according to claim 36 wherein the feedback further includes actuating the touch screen display with a force feedback signal.

44. The method according to claim 43 wherein the force feedback signal is generated with a piezoelectric crystal.

45. The method according to claim 43 wherein actuating the touch screen display includes vibrating the surface of the touch screen display.

46. The computer implemented method of claim 43 wherein the force feedback signal is directed to a portion of the touch screen display underlying one of the first touch input and the second touch input.

47. The computer implemented method of claim 43 wherein the force feedback signal is directed to a portion of the touch screen display overlying one of the first virtual object and the second virtual object.

48. The computer implemented method of claim 36 wherein animating includes depicting movement of at least one of the first virtual object and the second virtual object along the touch screen display.

49. The computer implemented method of claim 36 wherein:

data corresponding to the first virtual object is transmitted to a processer associated with the touch screen display via the internet; and

data corresponding to the second virtual object is transmitted to a processor associated with the touch screen display via the internet.

50. A method for controlling a computer system comprising:

providing a display surface adapted to display visually observable data;

receiving a first touch input on the display surface and a second touch input on the display surface, the first and second touch inputs being simultaneously in contact with the display surface;

determining a location of the first touch input in two dimensions based on a centroid corresponding to the first touch input;

determining a location of the second touch input in two dimensions based on a centroid corresponding to the second touch input; and

performing an operation on the computer system based on the location of the first touch input and the location of the second touch input.

51. The method according to claim 50 wherein:

the first touch input includes one of a finger and a thumb in contact with the display surface; and

the second touch input includes one of a finger and a thumb in contact with the display surface.

52. The method according to claim 50 further comprising:

detecting a change in the location of the first touch input along the display surface; and

detecting a change in the location of the second touch input along the display surface.

53. The method according to claim 52 further comprising detecting a rate of the change in the location of at least one of the first touch input and the second touch input, the rate of change of the location indicating a desired value of a variable.

54. The method according to claim 50 further comprising providing an electro-optical sensor oriented toward a rear portion of the display surface.

55. The method according to claim 54 further comprising imaging the rear portion of the touch screen with the electro-optical sensor to detect a region of disturbed light energy, wherein the region of disturbed light energy corresponds to a touch input.

56. The method according to claim 50 further comprising analyzing the output of the electro-optical sensor with reference to a threshold intensity level.

57. The method according to claim 54 further comprising providing a bandpass filter between the electro-optical sensor and the rear portion of the display surface.

58. The method according to claim 54 wherein the electro-optical sensor includes a camera operating at a frame rate equal to or above 30 Hz.

59. The method according to claim 50 further comprising providing a light source oriented toward the rear portion of the display surface, wherein the light source illuminates the rear portion of the display surface.

60. The method according to claim 59 wherein the light source is an infrared light source.

61. The method according to claim 50 wherein the operation includes actuating the display screen with a force feedback signal.

62. The method according to claim 50 wherein the operation includes determining a desired value of a variable.

63. The method according to claim 50 wherein the operation includes displaying one of a knob and a slider bar on the display surface.

64. The method according to claim 50 wherein:

the centroid corresponding to the first touch input is determined with reference to a first data array; and

the centroid corresponding to the second touch input is determined with reference to a second data array.

65. A method for controlling a computer system using a touch screen comprising the steps of:

displaying visually observable data on a display surface of the touch screen;

receiving a first touch input corresponding to one of a finger and a thumb in contact with the display surface;

receiving a second touch input corresponding to one of a finger and a thumb in contact with the display surface, the first and second touch inputs being simultaneously in contact with the display surface;

calculating a first centroid corresponding to the first touch input;

calculating a second centroid corresponding to the second touch input;

determining the location of the first touch input and the location of the second touch input based on the first and second centroids, respectively; and

in response to the determining step, performing an operation on the computer system.

66. The method according to claim 65 wherein the operation includes actuating the display screen with a force feedback signal.

67. The method according to claim 65 wherein the operation includes determining a desired value of a variable.

68. The method according to claim 65 wherein the operation includes displaying one of a knob and a slider bar on the display surface.

69. The method according to claim 65 further comprising:

detecting a change in the location of at least one of the first touch input and the second touch input along the display surface.

70. The method according to claim 69 wherein the visually observable data includes an object rotated in response to the change in the location of at least one of the first touch input and the second touch input along the display surface.

71. The touch surface of claim 69 wherein the visually observable data includes an object moved in response to the change in the location of the at least one of the first touch input and the second touch input along the display surface.

72. The method according to claim 65 further comprising providing an electro-optical sensor oriented toward a rear portion of the display surface.

73. The method according to claim 72 further comprising analyzing the output of the electro-optical sensor with reference to a threshold intensity level.

74. The method according to claim 72 further comprising providing a bandpass filter to filter light received by the electro-optical sensor.

75. The method according to claim 72 wherein the electro-optical sensor includes a camera operating at a frame rate equal to or above 30 Hz.

76. The method according to claim 65 further comprising providing a light source oriented toward a rear portion of the display surface, wherein the light source illuminates the rear portion of the display surface.

77. The method according to claim 76 wherein the light source is an infrared light source.

78. The method according to claim 65 wherein:

the first centroid is calculated from a first data array corresponding to the first touch input; and

the second centroid is calculated from a second data array corresponding to the second touch input.