MULTITOUCH DATA FUSION

• US 20080211766A1
• Filed: 12/21/2007
• Published: 09/04/2008
• Est. Priority Date: 01/07/2007
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Westerman, Wayne Carl, Elias, John Greer
• Granted Patent: US 10,437,459 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 2203/011 Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: G06V 40/1365 Matching; Classification

Abstract

A method for performing multi-touch (MT) data fusion is disclosed in which multiple touch inputs occurring at about the same time are received to generating first touch data. Secondary sense data can then be combined with the first touch data to perform operations on an electronic device. The first touch data and the secondary sense data can be time-aligned and interpreted in a time-coherent manner. The first touch data can be refined in accordance with the secondary sense data, or alternatively, the secondary sense data can be interpreted in accordance with the first touch data. Additionally, the first touch data and the secondary sense data can be combined to create a new command.

Claims

1. A multi-touch (MT) data fusion system, comprising:

a MT sensing device configured for receiving multiple touch inputs occurring at about the same time and generating first touch data;

one or more MT data fusion devices configured for providing secondary data; and

a processor coupled to the MT sensing device and the one or more MT data fusion devices, the processor programmed for combining both the first touch data and the secondary data to perform operations on an electronic device.

2. The MT data fusion system of claim 1, the processor further programmed for performing temporal fusion by time-aligning the first touch data and the secondary data and interpreting the combined data in a time-coherent manner.

3. The MT data fusion system of claim 1, the processor further programmed for refining the first touch data in accordance with the secondary data.

4. The MT data fusion system of claim 1, the processor further programmed for interpreting the secondary data in accordance with the first touch data.

5. The MT data fusion system of claim 1, the processor further programmed for combining both the first touch data and the secondary data to create a new command.

6. The MT data fusion system of claim 1, further comprising memory configured for storing a database, the processor further programmed for referring to the database in view of the combined first touch data and secondary data to determine a course of action.

7. The MT data fusion system of claim 1, the processor further programmed for:

associating the first touch data with a displayed object; and

modifying a characteristic of the displayed object in accordance with secondary data received from an audio sensor.

8. The MT data fusion system of claim 7, the secondary data received from the audio sensor comprising a voice command.

9. The MT data fusion system of claim 1, the secondary data including gaze recognition data, the processor further programmed for performing an operation on a displayed object in accordance with the gaze recognition data.

10. The MT data fusion system of claim 1, the one or more MT data fusion devices including an imaging device configured for generating a hand profile, the processor further programmed for correlating the hand profile with the first touch data to identify fingers associated with contacts established by the first touch data.

11. The MT data fusion system of claim 1, the secondary data including motion data, the processor further programmed for filtering electronic device motion from the first touch data in accordance with the motion data.

12. The MT data fusion system of claim 3, the one or more MT data fusion devices including an orientation sensor for generating orientation data, the processor further programmed for interpreting the first touch data as first data or second data in accordance with the orientation data.

13. The MT data fusion system of claim 1, the one or more MT data fusion devices including a biometric sensor for determining hand characteristics, the processor further programmed for recognizing a trajectory signature from the first touch data and identifying a user based on the hand characteristics and the trajectory signature.

14. The MT data fusion system of claim 1, the secondary data including emotional data, the processor further programmed for analyzing the first touch data and initiating a corrective action in accordance with the emotional data.

15. The MT data fusion system of claim 1, the one or more MT data fusion devices including a force sensor for generating force data, the processor further programmed for modifying the first touch data in accordance with the force data.

16. The MT data fusion system of claim 1, the one or more MT data fusion devices including a second MT sensing device for generating second touch data, the processor further programmed for comparing the first touch data and the second touch data in a time domain and determining third touch data from the comparison.

17. The MT data fusion system of claim 1, the one or more MT data fusion devices comprising one or more sensors selected from the group consisting of biometric sensors, audio sensors, optical sensors, sonar sensors, image sensors, acoustic sensors, electric field sensors, shock sensors, environmental sensors, orientation sensors, pressure sensors, force sensors, and temperature sensors.

18. An electronic device comprising the MT data fusion system of claim 1.

19. A method for performing touch data fusion, comprising:

receiving a touch input and generating first touch data;

providing secondary sense data; and

combining both the first touch data and the secondary sense data to perform operations on an electronic device.

20. The method of claim 19, wherein the touch input comprises multiple touch inputs occurring at about the same time.

21. The method of claim 19, further comprising performing temporal fusion by time-aligning the first touch data and the secondary sense data and interpreting the combined data in a time-coherent manner.

22. The method of claim 19, further comprising refining the first touch data in accordance with the secondary sense data.

23. The method of claim 19, further comprising interpreting the secondary sense data in accordance with the first touch data.

24. The method of claim 19, further comprising combining both the first touch data and the secondary sense data to create a new command.

25. The method of claim 19, further comprising referring to a database in view of the combined first touch data and secondary sense data to determine a course of action.

26. The method of claim 19, the secondary sense data including audio data, the method further comprising:

associating the first touch data with a displayed object; and

modifying a characteristic of the object in accordance with the audio data.

27. The method of claim 26, the audio data comprising a voice command.

28. The method of claim 19, the secondary sense data including gaze recognition data, the method further comprising performing an operation on a displayed object in accordance with the gaze recognition data.

29. The method of claim 19, further comprising generating a hand profile as the secondary sense data and correlating the hand profile with the first touch data to identify fingers associated with contacts established by the first touch data.

30. The method of claim 19, the secondary sense data including motion data, the method further comprising filtering electronic device motion from the first touch data in accordance with the motion data.

31. The method of claim 19, further comprising generating orientation data as the secondary sense data and interpreting the first touch data as first data or second data in accordance with the orientation data.

32. The method of claim 19, further comprising determining hand characteristics as the secondary sense data and recognizing a trajectory signature from the first touch data and identifying a user based on the hand characteristics and the trajectory signature.

33. The method of claim 19, the secondary sense data including emotional data, the method further comprising analyzing the first touch data and initiating a corrective action in accordance with the emotional data.

34. The method of claim 19, further comprising generating force data as the secondary sense data and modifying the first touch data in accordance with the force data.

35. The method of claim 19, further comprising generating second touch data as the secondary sense data and comparing the first touch data and the second touch data in a time domain and determining third touch data from the comparison.

36. A computer-readable medium comprising program code for performing multi-touch (MT data fusion, the program code for causing performance of a method comprising:

receiving multiple touch inputs occurring at about the same time and generating first touch data;

providing secondary sense data; and

combining both the first touch data and the secondary sense data to perform operations on an electronic device.

37. The computer-readable medium of claim 36, the program code further for causing performance of a method comprising performing temporal fusion by time-aligning the first touch data and the secondary sense data and interpreting the combined data in a time-coherent manner.

38. The computer-readable medium of claim 36, the program code further for causing performance of a method comprising refining the first touch data in accordance with the secondary sense data.

39. The computer-readable medium of claim 36, the program code further for causing performance of a method comprising interpreting the secondary sense data in accordance with the first touch data.

40. The computer-readable medium of claim 36, the program code further for causing performance of a method comprising combining both the first touch data and the secondary sense data to create a new command.

41. The computer-readable medium of claim 36, the program code further for causing performance of a method comprising referring to a database in view of the combined first touch data and secondary sense data to determine a course of action.

42. The computer-readable medium of claim 36, the secondary sense data including audio data, the program code further for causing performance of a method comprising:

associating the first touch data with a displayed object; and

modifying a characteristic of the object in accordance with the audio data.

43. The computer-readable medium of claim 42, the audio data comprising a voice command.

44. The computer-readable medium of claim 36, the secondary sense data including gaze recognition data, the program code further for causing performance of a method comprising performing an operation on an object being displayed in accordance with the gaze recognition data.

45. The computer-readable medium of claim 36, the program code further for causing performance of a method comprising generating a hand profile as the secondary sense data and correlating the hand profile with the first touch data to identify fingers associated with contacts established by the first touch data.

46. The computer-readable medium of claim 36, the secondary sense data including motion data, the program code further for causing performance of a method comprising filtering electronic device motion from the first touch data in accordance with the motion data.

47. The computer-readable medium of claim 36, the program code further for causing performance of a method comprising generating orientation data as the secondary sense data and interpreting the first touch data as first data or second data in accordance with the orientation data.

48. The computer-readable medium of claim 36, the program code further for causing performance of a method comprising determining hand characteristics as the secondary sense data and recognizing a trajectory signature from the first touch data and identifying a user based on the hand characteristics and the trajectory signature.

49. The computer-readable medium of claim 36, the secondary sense data including emotional data, the program code further for causing performance of a method comprising analyzing the first touch data and initiating a corrective action in accordance with the emotional data.

50. The computer-readable medium of claim 36, the program code further for causing performance of a method comprising generating force data as the secondary sense data and modifying the first touch data in accordance with the force data.

51. The computer-readable medium of claim 36, the program code further for causing performance of a method comprising generating second touch data as the secondary sense data and comparing the first touch data and the second touch data in a time domain and determining third touch data from the comparison.

52. A mobile telephone including a multi-touch (MT) data fusion system, the MT data fusion system comprising:

a MT sensing device configured for receiving multiple touch inputs occurring at about the same time and generating first touch data;

one or more MT data fusion devices configured for providing secondary data; and

a processor coupled to the MT sensing device and the one or more MT data fusion devices, the processor programmed for combining both the first touch data and the secondary data to perform operations on an electronic device.

53. A digital audio player including a multi-touch (MT) data fission system, the MT data fusion system comprising:

a MT sensing device configured for receiving multiple touch inputs occurring at about the same time and generating first touch data;

one or more MT data fusion devices configured for providing secondary data; and

a processor coupled to the MT sensing device and the one or more MT data fusion devices, the processor programmed for combining both the first touch data and the secondary data to perform operations on an electronic device.