Multitouch data fusion

• US 10,437,459 B2
• Filed: 12/21/2007
• Issued: 10/08/2019
• Est. Priority Date: 01/07/2007
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Westerman, Wayne Carl, Elias, John Greer
• Primary Examiner(s): Lam, Vinh T
• Application Number:
  US11/963,516
• Publication Number:
  US 20080211766A1
• Time in Patent Office: 4,309 Days
• Field of Search: 345156, 345173-175, 704275, 178 1801- 1809
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 2203/011 Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: G06V 40/1365 Matching; Classification

Abstract

A method for performing multi-touch (MT) data fusion is disclosed in which multiple touch inputs occurring at about the same time are received to generating first touch data. Secondary sense data can then be combined with the first touch data to perform operations on an electronic device. The first touch data and the secondary sense data can be time-aligned and interpreted in a time-coherent manner. The first touch data can be refined in accordance with the secondary sense data, or alternatively, the secondary sense data can be interpreted in accordance with the first touch data. Additionally, the first touch data and the secondary sense data can be combined to create a new command.

Claims

1. A data processing system, comprising:

a first sensing device configured for receiving a multi-finger gesture input indicative of multiple touch inputs occurring at about the same time and generating first gesture data;

one or more second sensing devices configured for providing secondary data; and

a processor coupled to the first sensing device and the one or more second sensing devices, the processor capable of combining both the first gesture data and the secondary data to perform one or more operations on an electronic device,

wherein the secondary data includes gaze recognition data, and the processor is further capable of

processing the gaze recognition data to determine when a gaze is directed to an object displayed on a screen, and

when the gaze is directed to the object, processing the first gesture data in accordance with the gaze recognition data to perform the one or more operations on the electronic device, the one or more operations related to the object at which the gaze is directed.

2. The data processing system of claim 1, the processor further capable of time-aligning the first gesture data and the secondary data and interpreting the combined data in a time-coherent manner.

3. The data processing system of claim 1, the processor further capable of refining the first gesture data in accordance with the secondary data.

4. The data processing system of claim 1, the processor further capable of interpreting the secondary data in accordance with the first gesture data.

5. The data processing system of claim 1, the processor further capable of combining both the first gesture data and the secondary data to create a new command.

6. The data processing system of claim 1, further comprising memory configured for storing a database, the processor further capable of referring to the database in view of the combined first gesture data and secondary data to determine a course of action.

7. The data processing system of claim 1, the processor further capable of:

associating the first gesture data with the object being displayed; and

modifying a characteristic of the displayed object in accordance with the secondary data, the secondary data including data received from an audio sensor.

8. The data processing system of claim 7, the secondary data received from the audio sensor comprising a voice command.

9. The data processing system of claim 1, the one or more second sensing devices including an imaging device configured for generating a hand profile, the processor further capable of correlating the hand profile with the first gesture data to identify fingers associated with the multi-finger gesture input.

10. The data processing system of claim 1, the processor further capable of filtering electronic device motion from the first gesture data.

11. The data processing system of claim 1, the one or more second sensing devices including an orientation sensor for generating orientation data, the processor further capable of interpreting the first gesture data as first data or second data in accordance with the orientation data.

12. The data processing system of claim 1, the one or more second sensing devices including a biometric sensor for determining hand characteristics, the processor further capable of recognizing a trajectory signature from the first gesture data and identifying a user based on the hand characteristics and the trajectory signature.

13. The data processing system of claim 1, the secondary data including emotional data, the processor further capable of analyzing the first gesture data and initiating a corrective action in accordance with the emotional data.

14. The data processing system of claim 1, the one or more second sensing devices including a force sensor for generating force data, the processor further capable of modifying the first gesture data in accordance with the force data.

15. The data processing system of claim 1, the one or more second sensing devices further configured for generating second gesture data, the processor further capable of comparing the first gesture data and the second gesture data in a time domain and determining third gesture data from the comparison.

16. The data processing system of claim 1, the one or more second sensing devices comprising optical sensors.

17. An electronic device comprising the data processing system of claim 1.

18. A method for performing data processing, comprising:

receiving a multi-finger gesture input indicative of multiple touch inputs occurring at about the same time and generating first gesture data;

providing secondary sense data, the secondary sense data including gaze recognition data;

processing the gaze recognition data to determine when a gaze is directed to an object displayed on a screen; and

when the gaze is directed to the object, combining both the first gesture data and the secondary sense data by processing the first gesture data in accordance with the gaze recognition data to perform one or more operations on the electronic device, the one or more operations related to the object at which the gaze is directed.

19. The method of claim 18, further comprising time-aligning the first gesture data and the secondary sense data and interpreting the combined data in a time-coherent manner.

20. The method of claim 18, further comprising refining the first gesture data in accordance with the secondary sense data.

21. The method of claim 18, further comprising interpreting the secondary sense data in accordance with the first gesture data.

22. The method of claim 18, further comprising combining both the first gesture data and the secondary sense data to create a new command.

23. The method of claim 18, further comprising referring to a database in view of the combined first gesture data and secondary sense data to determine a course of action.

24. The method of claim 18, the secondary sense data including audio data, the method further comprising:

associating the first gesture data with the object being displayed; and

modifying a characteristic of the displayed object in accordance with the audio data.

25. The method of claim 24, the audio data comprising a voice command.

26. The method of claim 18, the method further comprising performing the one or more operations on the displayed object in accordance with the gaze recognition data.

27. The method of claim 18, further comprising generating a hand profile as the secondary sense data and correlating the hand profile with the first gesture data to identify fingers associated with contacts established by the first gesture data.

28. The method of claim 18, further comprising filtering electronic device motion from the first gesture data.

29. The method of claim 18, further comprising generating orientation data as part of the secondary sense data and interpreting the first gesture data in accordance with the orientation data.

30. The method of claim 18, further comprising determining hand characteristics as part of the secondary sense data and recognizing a trajectory signature from the first gesture data, and identifying a user based on the hand characteristics and the trajectory signature.

31. The method of claim 18, the secondary sense data including emotional data, the method further comprising analyzing the first gesture data and initiating a corrective action in accordance with the emotional data.

32. The method of claim 18, further comprising generating force data as part of the secondary sense data and modifying the first gesture data in accordance with the force data.

33. The method of claim 18, further comprising generating second gesture data as part of the secondary sense data and comparing the first gesture data and the second gesture data in a time domain and determining third gesture data from the comparison.

34. A non-transitory computer-readable medium comprising program code for performing data processing, the program code for causing performance of a method comprising:

receiving a multi-finger gesture input indicative of multiple touch inputs occurring at about the same time and generating first gesture data;

providing secondary sense data, the secondary sense data including gaze recognition data;

processing the gaze recognition data to determine when a gaze is directed to an object displayed on a screen; and

when the gaze is directed to the object, combining both the first gesture data and the secondary sense data by processing the first gesture data in accordance with the gaze recognition data to perform one or more operations on an electronic device, the one or more operations related to the object at which the gaze is directed.

35. The non-transitory computer-readable medium of claim 34, the program code further for causing performance of a method comprising time-aligning the first gesture data and the secondary sense data and interpreting the combined data in a time-coherent manner.

36. The non-transitory computer-readable medium of claim 34, the program code further for causing performance of a method comprising refining the first gesture data in accordance with the secondary sense data.

37. The non-transitory computer-readable medium of claim 34, the program code further for causing performance of a method comprising interpreting the secondary sense data in accordance with the first gesture data.

38. The non-transitory computer-readable medium of claim 34, the program code further for causing performance of a method comprising combining both the first gesture data and the secondary sense data to create a new command.

39. The non-transitory computer-readable medium of claim 34, the program code further for causing performance of a method comprising referring to a database in view of the combined first gesture data and secondary sense data to determine a course of action.

40. The non-transitory computer-readable medium of claim 34, the secondary sense data including audio data, the program code further for causing performance of a method comprising:

associating the first gesture data with the object being displayed; and

modifying a characteristic of the displayed object in accordance with the audio data.

41. The non-transitory computer-readable medium of claim 40, the audio data comprising a voice command.

42. The non-transitory computer-readable medium of claim 34, the program code further for causing performance of a method comprising performing the one or more operations on the displayed object in accordance with the gaze recognition data.

43. The non-transitory computer-readable medium of claim 34, the program code further for causing performance of a method comprising generating a hand profile as part of the secondary sense data and correlating the hand profile with the first gesture data to identify fingers associated with contacts established by the first gesture data.

44. The non-transitory computer-readable medium of claim 34, the program code further for causing performance of a method comprising filtering electronic device motion from the first gesture data.

45. The non-transitory computer-readable medium of claim 34, the program code further for causing performance of a method comprising generating orientation data as part of the secondary sense data and interpreting the first gesture data as first data or second data in accordance with the orientation data.

46. The non-transitory computer-readable medium of claim 34, the program code further for causing performance of a method comprising determining hand characteristics as part of the secondary sense data and recognizing a trajectory signature from the first gesture data and identifying a user based on the hand characteristics and the trajectory signature.

47. The non-transitory computer-readable medium of claim 34, the secondary sense data including emotional data, the program code further for causing performance of a method comprising analyzing the first gesture data and initiating a corrective action in accordance with the emotional data.

48. The non-transitory computer-readable medium of claim 34, the program code further for causing performance of a method comprising generating force data as part of the secondary sense data and modifying the first gesture data in accordance with the force data.

49. The non-transitory computer-readable medium of claim 34, the program code further for causing performance of a method comprising generating second gesture data as part of the secondary sense data and comparing the first gesture data and the second gesture data in a time domain and determining third gesture data from the comparison.

50. A mobile telephone including a data processing system, the data processing system comprising:

a first sensing device configured for receiving a multi-finger gesture input indicative of multiple touch inputs occurring at about the same time and generating first gesture data;

one or more second sensing devices configured for providing secondary data, the secondary data including gaze recognition data; and

a processor coupled to the first sensing device and the one or more second sensing devices, the processor capable of combining both the first gesture data and the secondary data to perform one or more operations on an electronic device;

wherein the secondary data includes gaze recognition data, and the processor is further capable of

processing the gaze recognition data to determine when a gaze is directed to an object displayed on a screen, and

when the gaze is directed to the object, processing the first gesture data in accordance with the gaze recognition data to perform the one or more operations on the electronic device, the one or more operations related to the object at which the gaze is directed.

51. A digital audio player including a data processing system, the data processing system comprising:

a first sensing device configured for receiving a multi-finger gesture input indicative of multiple touch inputs occurring at about the same time and generating first gesture data;

one or more second sensing devices configured for providing secondary data, the secondary data including gaze recognition data; and

a processor coupled to the first sensing device and the one or more second sensing devices, the processor capable of combining both the first gesture data and the secondary data to perform one or more operations on an electronic device;

wherein the secondary data includes gaze recognition data, and the processor is further capable of

processing the gaze recognition data to determine when a gaze is directed to an object displayed on a screen, and

when the gaze is directed to the object, processing the first gesture data in accordance with the gaze recognition data to perform the one or more operations on the electronic device, the one or more operations related to the object at which the gaze is directed.