Devices, methods, and graphical user interfaces for haptic mixing

• US 9,864,432 B1
• Filed: 09/20/2016
• Issued: 01/09/2018
• Est. Priority Date: 09/06/2016
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Moussette, Camille, Gleeson, Brian T., Morrell, John B., Scott, Douglas A., Williams, Joseph M.
• Primary Examiner(s): Cheng, Joe H
• Application Number:
  US15/270,885
• Time in Patent Office: 476 Days
• CPC Class Codes: G06F 1/1684 Constructional details or arrangements related to integrated I/O peripherals not covered by groups G06F1/1635 - G06F1/1675
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback

Abstract

An electronic device receives a first set of one or more inputs corresponding to user interface elements displayed on the display and a first set of one or more tactile outputs, and also receives a second set of one or more inputs corresponding to one or more hardware elements and a second set of one or more tactile outputs. In response, in accordance with a determination that the first set of one or more tactile outputs and the second set of one or more tactile outputs overlap, the device outputs, with one or more tactile output generators, a modified tactile output sequence that is modified so as to emphasize the second set of one or more tactile outputs relative to the first set of one or more tactile outputs.

Claims

1. A method, comprising:

at an electronic device with a display, a touch-sensitive surface, one or more tactile output generators, and one or more hardware elements;

receiving a first set of one or more inputs corresponding to user interface elements displayed on the display, wherein the first set of one or more inputs corresponds to a first set of one or more tactile outputs; and

receiving a second set of one or more inputs corresponding to the one or more hardware elements, wherein the second set of one or more inputs corresponds to a second set of one or more tactile outputs;

in response to the second set of one or more inputs;

in accordance with a determination that the first set of one or more tactile outputs and the second set of one or more tactile outputs do not overlap, outputting, with the one or more tactile output generators, a tactile output sequence that includes the first set of one or more tactile outputs and the second set of one or more tactile outputs; and

in accordance with a determination that the first set of one or more tactile outputs and the second set of one or more tactile outputs overlap, outputting, with the one or more tactile output generators, a modified tactile output sequence that is modified so as to emphasize the second set of one or more tactile outputs relative to the first set of one or more tactile outputs.

2. The method of claim 1, wherein the second set of one or more inputs corresponds to a click gesture, or a portion of a click gesture, performed using a respective hardware element.

3. The method of claim 1, wherein outputting the modified tactile output sequence includes performing one or more of:

increasing an amplitude of the second set of one or more tactile outputs and decreasing an amplitude of the first set of one or more tactile outputs.

4. The method of claim 1, wherein the modified tactile output sequence includes a first portion during which output of the first set of one or more tactile outputs is gradually reduced from a first output level to a second output level.

5. The method of claim 4, wherein the modified tactile output sequence includes a second portion during which output of the first set of one or more tactile outputs remains reduced.

6. The method of claim 5, wherein a duration of the second portion is selected based on a magnitude of the first portion.

7. The method of claim 5, wherein the modified tactile output sequence includes a third portion that is subsequent to the second portion and includes the second set of one or more tactile outputs, and a duration of the third portion is longer than a duration of the second portion during which output of the first set of one or more tactile outputs remains reduced.

8. The method of claim 4, wherein the modified tactile output sequence includes:

a third portion that includes the second set of one or more tactile outputs; and

a fourth portion during which the first set of one or more tactile outputs is output using an output level that is gradually increased from a third output level to a fourth output level, wherein the fourth portion is subsequent to the third portion.

9. The method of claim 8, wherein the modified tactile output sequence includes:

a fifth portion during which output of the first set of one or more tactile outputs remains reduced, wherein the fifth portion is subsequent to the third portion and prior to the fourth portion.

10. The method of claim 1, including:

playing an audio output that is synchronized with the first set of one or more tactile outputs and is triggered by a same condition that triggered the first set of one or more tactile outputs; and

continuing to play the audio output that is synchronized with the first set of one or more tactile outputs without modification independently of whether or not the second set of one or more tactile outputs is emphasized relative to the first set of one or more tactile outputs.

11. The method of claim 1, including:

playing a first audio output that is synchronized with the first set of one or more tactile outputs and is triggered by a same condition that triggered the first set of one or more tactile outputs; and

while playing the first audio output, playing a second audio output that is synchronized with the second set of one or more tactile outputs and is triggered by a same condition that triggered the second set of one or more tactile outputs, wherein the first audio output is modified to emphasize the second audio output while playing the second audio output.

12. The method of claim 11, including, while emphasizing the second set of one or more tactile outputs relative to the first set of one or more tactile outputs, continuing to process the first set of one or more inputs, including mixing together tactile outputs for the first set of one or more inputs.

13. The method of claim 11, wherein a magnitude of the first audio output is user-selected and/or a magnitude of the second audio output is user-selected.

14. The method of claim 1, including:

in response to the second set of one or more inputs, playing an audio output that is triggered by a same condition that triggered the second set of one or more tactile outputs, wherein the audio output starts after the second set of one or more tactile outputs starts.

15. The method of claim 1, wherein outputting the modified tactile output sequence includes reducing a scale of at least a portion of the first set of one or more tactile outputs that overlaps with the second set of one or more tactile outputs.

16. The method of claim 1, including:

in accordance with a determination that a first portion of the modified tactile output sequence meets output limiting criteria for the one or more tactile output generators;

reducing a scale of the modified tactile output sequence during at least a second portion of the modified tactile output sequence immediately preceding the first portion of the modified tactile output sequence; and

outputting, with the one or more tactile output generators, a tactile output sequence based on the modified tactile output sequence that includes the second portion of the modified tactile output sequence with reduced scale.

17. An electronic device, comprising:

a display;

a touch-sensitive surface;

one or more tactile output generators;

one or more processors;

memory; and

one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for;

receiving a first set of one or more inputs corresponding to user interface elements displayed on the display, wherein the first set of one or more inputs corresponds to a first set of one or more tactile outputs; and

receiving a second set of one or more inputs corresponding to the one or more hardware elements, wherein the second set of one or more inputs corresponds to a second set of one or more tactile outputs;

in response to the second set of one or more inputs;

in accordance with a determination that the first set of one or more tactile outputs and the second set of one or more tactile outputs do not overlap, outputting a tactile output sequence that includes the first set of one or more tactile outputs and the second set of one or more tactile outputs; and

in accordance with a determination that the first set of one or more tactile outputs and the second set of one or more tactile outputs overlap, outputting, with the one or more tactile output generators, a modified tactile output sequence that is modified so as to emphasize the second set of one or more tactile outputs relative to the first set of one or more tactile outputs.

18. The electronic device of claim 17, wherein the second set of one or more inputs corresponds to a click gesture, or a portion of a click gesture, performed using a respective hardware element.

19. The electronic device of claim 17, wherein outputting the modified tactile output sequence includes performing one or more of:

increasing an amplitude of the second set of one or more tactile outputs and decreasing an amplitude of the first set of one or more tactile outputs.

20. The electronic device of claim 17, wherein the modified tactile output sequence includes a first portion during which output of the first set of one or more tactile outputs is gradually reduced from a first output level to a second output level.

21. The electronic device of claim 20, wherein the modified tactile output sequence includes a second portion during which output of the first set of one or more tactile outputs remains reduced.

22. The electronic device of claim 21, wherein a duration of the second portion is selected based on a magnitude of the first portion.

23. The electronic device of claim 21, wherein the modified tactile output sequence includes a third portion that is subsequent to the second portion and includes the second set of one or more tactile outputs, and a duration of the third portion is longer than a duration of the second portion during which output of the first set of one or more tactile outputs remains reduced.

24. The electronic device of claim 20, wherein the modified tactile output sequence includes:

a third portion that includes the second set of one or more tactile outputs; and

a fourth portion during which the first set of one or more tactile outputs is output using an output level that is gradually increased from a third output level to a fourth output level, wherein the fourth portion is subsequent to the third portion.

25. The electronic device of claim 24, wherein the modified tactile output sequence includes:

a fifth portion during which output of the first set of one or more tactile outputs remains reduced, wherein the fifth portion is subsequent to the third portion and prior to the fourth portion.

26. The electronic device of claim 17, wherein the one or more programs include instructions for:

playing an audio output that is synchronized with the first set of one or more tactile outputs and is triggered by a same condition that triggered the first set of one or more tactile outputs; and

continuing to play the audio output that is synchronized with the first set of one or more tactile outputs without modification independently of whether or not the second set of one or more tactile outputs is emphasized relative to the first set of one or more tactile outputs.

27. The electronic device of claim 17, wherein the one or more programs include instructions for:

playing a first audio output that is synchronized with the first set of one or more tactile outputs and is triggered by a same condition that triggered the first set of one or more tactile outputs; and

while playing the first audio output, playing a second audio output that is synchronized with the second set of one or more tactile outputs and is triggered by a same condition that triggered the second set of one or more tactile outputs, wherein the first audio output is modified to emphasize the second audio output while playing the second audio output.

28. The electronic device of claim 27, wherein the one or more programs include instructions for, while emphasizing the second set of one or more tactile outputs relative to the first set of one or more tactile outputs, continuing to process the first set of one or more inputs, including mixing together tactile outputs for the first set of one or more inputs.

29. The electronic device of claim 27, wherein a magnitude of the first audio output is user-selected and/or a magnitude of the second audio output is user-selected.

30. The electronic device of claim 17, wherein the one or more programs include instructions for:

in response to the second set of one or more inputs, playing an audio output that is triggered by a same condition that triggered the second set of one or more tactile outputs, wherein the audio output starts after the second set of one or more tactile outputs starts.

31. The electronic device of claim 17, wherein outputting the modified tactile output sequence includes reducing a scale of at least a portion of the first set of one or more tactile outputs that overlaps with the second set of one or more tactile outputs.

32. The electronic device of claim 17, wherein the one or more programs include instructions for:

in accordance with a determination that a first portion of the modified tactile output sequence meets output limiting criteria for the one or more tactile output generators;

reducing a scale of the modified tactile output sequence during at least a second portion of the modified tactile output sequence immediately preceding the first portion of the modified tactile output sequence; and

outputting, with the one or more tactile output generators, a tactile output sequence based on the modified tactile output sequence that includes the second portion of the modified tactile output sequence with reduced scale.

33. A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by an electronic device with a display, a touch-sensitive surface, and one or more tactile output generators, cause the device to:

receive a first set of one or more inputs corresponding to user interface elements displayed on the display, wherein the first set of one or more inputs corresponds to a first set of one or more tactile outputs;

receive a second set of one or more inputs corresponding to the one or more hardware elements, wherein the second set of one or more inputs corresponds to a second set of one or more tactile outputs; and

in response to the second set of one or more inputs;

in accordance with a determination that the first set of one or more tactile outputs and the second set of one or more tactile outputs do not overlap, output a tactile output sequence that includes the first set of one or more tactile outputs and the second set of one or more tactile outputs; and

in accordance with a determination that the first set of one or more tactile outputs and the second set of one or more tactile outputs overlap, output, with the one or more tactile output generators, a modified tactile output sequence that is modified so as to emphasize the second set of one or more tactile outputs relative to the first set of one or more tactile outputs.

34. The non-transitory computer readable storage medium of claim 33, wherein the second set of one or more inputs corresponds to a click gesture, or a portion of a click gesture, performed using a respective hardware element.

35. The non-transitory computer readable storage medium of claim 33, wherein outputting the modified tactile output sequence includes performing one or more of:

increasing an amplitude of the second set of one or more tactile outputs and decreasing an amplitude of the first set of one or more tactile outputs.

36. The non-transitory computer readable storage medium of claim 33, wherein the modified tactile output sequence includes a first portion during which output of the first set of one or more tactile outputs is gradually reduced from a first output level to a second output level.

37. The non-transitory computer readable storage medium of claim 36, wherein the modified tactile output sequence includes a second portion during which output of the first set of one or more tactile outputs remains reduced.

38. The non-transitory computer readable storage medium of claim 37, wherein a duration of the second portion is selected based on a magnitude of the first portion.

39. The non-transitory computer readable storage medium of claim 37, wherein the modified tactile output sequence includes a third portion that is subsequent to the second portion and includes the second set of one or more tactile outputs, and a duration of the third portion is longer than a duration of the second portion during which output of the first set of one or more tactile outputs remains reduced.

40. The non-transitory computer readable storage medium of claim 36, wherein the modified tactile output sequence includes:

a third portion that includes the second set of one or more tactile outputs; and

a fourth portion during which the first set of one or more tactile outputs is output using an output level that is gradually increased from a third output level to a fourth output level, wherein the fourth portion is subsequent to the third portion.

41. The non-transitory computer readable storage medium of claim 40, wherein the modified tactile output sequence includes:

a fifth portion during which output of the first set of one or more tactile outputs remains reduced, wherein the fifth portion is subsequent to the third portion and prior to the fourth portion.

42. The non-transitory computer readable storage medium of claim 33, wherein the one or more programs include instructions, which when executed by the electronic device, cause the device to:

play an audio output that is synchronized with the first set of one or more tactile outputs and is triggered by a same condition that triggered the first set of one or more tactile outputs; and

continue to play the audio output that is synchronized with the first set of one or more tactile outputs without modification independently of whether or not the second set of one or more tactile outputs is emphasized relative to the first set of one or more tactile outputs.

43. The non-transitory computer readable storage medium of claim 33, wherein the one or more programs include instructions, which when executed by the electronic device, cause the device to:

play a first audio output that is synchronized with the first set of one or more tactile outputs and is triggered by a same condition that triggered the first set of one or more tactile outputs; and

while playing the first audio output, play a second audio output that is synchronized with the second set of one or more tactile outputs and is triggered by a same condition that triggered the second set of one or more tactile outputs, wherein the first audio output is modified to emphasize the second audio output while playing the second audio output.

44. The non-transitory computer readable storage medium of claim 43, wherein the one or more programs include instructions, which when executed by the electronic device, cause the device to, while emphasizing the second set of one or more tactile outputs relative to the first set of one or more tactile outputs, continue to process the first set of one or more inputs, including mixing together tactile outputs for the first set of one or more inputs.

45. The non-transitory computer readable storage medium of claim 43, wherein a magnitude of the first audio output is user-selected and/or a magnitude of the second audio output is user-selected.

46. The non-transitory computer readable storage medium of claim 33, wherein the one or more programs include instructions, which when executed by the electronic device, cause the device to:

in response to the second set of one or more inputs, play an audio output that is triggered by a same condition that triggered the second set of one or more tactile outputs, wherein the audio output starts after the second set of one or more tactile outputs starts.

47. The non-transitory computer readable storage medium of claim 33, wherein outputting the modified tactile output sequence includes reducing a scale of at least a portion of the first set of one or more tactile outputs that overlaps with the second set of one or more tactile outputs.

48. The non-transitory computer readable storage medium of claim 33, wherein the one or more programs include instructions, which when executed by the electronic device, cause the device to:

in accordance with a determination that a first portion of the modified tactile output sequence meets output limiting criteria for the one or more tactile output generators;

reduce a scale of the modified tactile output sequence during at least a second portion of the modified tactile output sequence immediately preceding the first portion of the modified tactile output sequence; and

output, with the one or more tactile output generators, a tactile output sequence based on the modified tactile output sequence that includes the second portion of the modified tactile output sequence with reduced scale.