Devices, methods, and graphical user interfaces for haptic mixing

• US 10,528,139 B2
• Filed: 01/05/2018
• Issued: 01/07/2020
• Est. Priority Date: 09/06/2016
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Moussette, Camille, Gleeson, Brian T., Morrell, John B.
• Primary Examiner(s): Cheng, Joe H
• Application Number:
  US15/863,765
• Publication Number:
  US 20180129292A1
• Time in Patent Office: 732 Days
• CPC Class Codes: G06F 1/1684 Constructional details or arrangements related to integrated I/O peripherals not covered by groups G06F1/1635 - G06F1/1675
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback

Abstract

An electronic device receives a first set of one or more inputs corresponding to user interface elements displayed on the display and a first set of one or more tactile outputs, and also receives a second set of one or more inputs corresponding to one or more hardware elements and a second set of one or more tactile outputs. In response, in accordance with a determination that the first set of one or more tactile outputs and the second set of one or more tactile outputs overlap, the device outputs, with one or more tactile output generators, a modified tactile output sequence that is modified so as to emphasize the second set of one or more tactile outputs relative to the first set of one or more tactile outputs.

Claims

1. A method, comprising:

at an electronic device with a display, a touch-sensitive surface, and a set of one or more tactile output generators;

detecting a triggering condition for a first set of one or more tactile outputs with a first priority;

detecting a triggering condition for a second set of one or more tactile outputs with a second priority; and

in response to detecting the triggering condition for the first set of one or more tactile outputs and the triggering condition for the second set of one or more tactile outputs and in accordance with a determination that the first set of one or more tactile outputs are scheduled to at least partially overlap with the second set of one or more tactile outputs;

in accordance with a determination that the first priority is higher than the second priority, outputting the second set of one or more tactile outputs, including, in accordance with a determination that the second set of one or more tactile outputs includes a first portion that precedes a second portion that overlaps with the first set of one or more tactile outputs, reducing an output level of the first portion of the second set of one or more tactile outputs prior to outputting, at reduced scale, at least the second portion of the second set of one or more tactile outputs that overlaps with the first set of one or more tactile outputs; and

in accordance with a determination that the second priority is higher than the first priority, outputting the first set of one or more tactile outputs, including, in accordance with a determination that the first set of one or more tactile outputs includes a first portion that precedes a second portion that overlaps with the second set of one or more tactile outputs, reducing an output level of the first portion of the first set of one or more tactile outputs prior to outputting, at reduced scale, at least the portion of the first set of one or more tactile outputs that overlaps with the second set of one or more tactile outputs.

2. The method of claim 1, including:

generating combined tactile outputs by combining at least the first set of one or more tactile outputs, including any portion thereof with reduced scale, and the second set of one or more tactile outputs, including any portion thereof with reduced scale; and

outputting, with the set of one or more tactile output generators, a tactile output sequence based on the combined tactile outputs.

3. The method of claim 1, wherein the output level of the first portion of a respective set of one or more tactile outputs is gradually reduced from a first output level to a second output level.

4. The method of claim 3, wherein the second output level is greater than zero.

5. The method of claim 1, including, in accordance with a determination that the first priority is the same as the second priority, combining the first set of one or more tactile outputs with the second set of one or more tactile outputs without reducing a scale of the first set of one or more tactile outputs relative to a scale of the second set of one or more tactile outputs.

6. The method of claim 1, including, in response to detecting the triggering condition for the first set of one or more tactile outputs and the triggering condition for the second set of one or more tactile outputs and in accordance with a determination that the first set of one or more tactile outputs is not scheduled to overlap with other tactile outputs, outputting, with the set of one or more tactile output generators, the first set of one or more tactile outputs and the second set of one or more tactile outputs without reducing a scale of the first set of one or more tactile outputs and the second set of one or more tactile outputs.

7. The method of claim 1, including:

detecting a triggering condition for a third set of one or more tactile outputs with a third priority that is higher than the second priority and the first priority; and

in response to detecting the triggering condition for the third set of one or more tactile outputs and in accordance with a determination that the third set of one or more tactile outputs are scheduled to at least partially overlap with the first set of one or more tactile outputs and the second set of one or more tactile outputs;

reducing a scale of at least a portion of the first set of one or more tactile outputs that overlaps with the third set of one or more tactile outputs; and

reducing a scale of at least a portion of the second set of one or more tactile outputs that overlaps with the third set of one or more tactile outputs.

8. The method of claim 7, wherein the third set of one or more tactile outputs with the third priority corresponds to one or more hardware elements distinct from the display.

9. The method of claim 1, wherein the first set of one or more tactile outputs with the first priority corresponds to asynchronous events.

10. The method of claim 1, wherein the second set of one or more tactile outputs with the second priority corresponds to synchronous events.

11. The method of claim 1, including, in response to detecting the triggering condition for the first set of one or more tactile outputs and the triggering condition for the second set of one or more tactile outputs and in accordance with the determination that the first set of one or more tactile outputs are scheduled to at least partially overlap with the second set of one or more tactile outputs and in accordance with the determination that the first priority is higher than the second priority, in conjunction with outputting, at reduced scale, at least the second portion of the second set of one or more tactile outputs with the second priority, outputting the first set of one or more tactile outputs with the first priority.

12. The method of claim 1, including, in response to detecting the triggering condition for the first set of one or more tactile outputs and the triggering condition for the second set of one or more tactile outputs and in accordance with the determination that the first set of one or more tactile outputs are scheduled to at least partially overlap with the second set of one or more tactile outputs and in accordance with the determination that the second priority is higher than the first priority, in conjunction with outputting at reduced scale, at least the second portion of the first set of one or more tactile outputs with the first priority, outputting the second set of one or more tactile outputs with the second priority.

13. The method of claim 1, including:

playing an audio output that is synchronized with the first set of one or more tactile outputs or the second set of one or more tactile outputs; and

continuing to play the audio output without modification independently of whether or not the scale of at least the second portion of the first set of one or more tactile outputs that overlaps with the second set of one or more tactile outputs or the scale of at least the second portion of the second set of one or more tactile outputs that overlaps with the first set of one or more tactile outputs is reduced.

14. The method of claim 1, further comprising:

receiving a first set of one or more inputs corresponding to user interface elements displayed on the display, wherein the first set of one or more inputs corresponds to the first set of one or more tactile outputs with the first priority; and

receiving a second set of one or more inputs corresponding to one or more hardware elements distinct from the display, wherein the second set of one or more inputs corresponds to the second set of one or more tactile outputs with the second priority;

wherein the second priority is higher than the first priority.

15. The method of claim 14, wherein the second set of one or more inputs corresponds to a click gesture, or a portion of a click gesture, performed using a respective hardware element.

16. An electronic device, comprising:

a display;

a touch-sensitive surface;

a set of one or more tactile output generators;

one or more processors;

memory; and

one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for;

detecting a triggering condition for a first set of one or more tactile outputs with a first priority;

detecting a triggering condition for a second set of one or more tactile outputs with a second priority; and

in response to detecting the triggering condition for the first set of one or more tactile outputs and the triggering condition for the second set of one or more tactile outputs and in accordance with a determination that the first set of one or more tactile outputs are scheduled to at least partially overlap with the second set of one or more tactile outputs;

in accordance with a determination that the first priority is higher than the second priority, outputting the second set of one or more tactile outputs, including, in accordance with a determination that the second set of one or more tactile outputs includes a first portion that precedes a second portion that overlaps with the first set of one or more tactile outputs, reducing an output level of the first portion of the second set of one or more tactile outputs prior to outputting, at reduced scale, at least the second portion of the second set of one or more tactile outputs that overlaps with the first set of one or more tactile outputs; and

in accordance with a determination that the second priority is higher than the first priority, outputting the first set of one or more tactile outputs, including, in accordance with a determination that the first set of one or more tactile outputs includes a first portion that precedes a second portion that overlaps with the second set of one or more tactile outputs, reducing an output level of the first portion of the first set of one or more tactile outputs prior to outputting, at reduced scale, at least the second portion of the first set of one or more tactile outputs that overlaps with the second set of one or more tactile outputs.

17. The electronic device of claim 16, wherein the one or more programs include instructions for:

generating combined tactile outputs by combining at least the first set of one or more tactile outputs, including any portion thereof with reduced scale, and the second set of one or more tactile outputs, including any portion thereof with reduced scale; and

outputting, with the set of one or more tactile output generators, a tactile output sequence based on the combined tactile outputs.

18. The electronic device of claim 16, wherein the output level of the first portion of a respective set of one or more tactile outputs is gradually reduced from a first output level to a second output level.

19. The electronic device of claim 18, wherein the second output level is greater than zero.

20. The electronic device of claim 16, wherein the one or more programs include instructions for, in accordance with a determination that the first priority is the same as the second priority, combining the first set of one or more tactile outputs with the second set of one or more tactile outputs without reducing a scale of the first set of one or more tactile outputs relative to a scale of the second set of one or more tactile outputs.

21. The electronic device of claim 16, wherein the one or more programs include instructions for, in response to detecting the triggering condition for the first set of one or more tactile outputs and the triggering condition for the second set of one or more tactile outputs and in accordance with a determination that the first set of one or more tactile outputs is not scheduled to overlap with other tactile outputs, outputting, with the set of one or more tactile output generators, the first set of one or more tactile outputs and the second set of one or more tactile outputs without reducing a scale of the first set of one or more tactile outputs and the second set of one or more tactile outputs.

22. The electronic device of claim 16, wherein the one or more programs include instructions for:

detecting a triggering condition for a third set of one or more tactile outputs with a third priority that is higher than the second priority and the first priority; and

in response to detecting the triggering condition for the third set of one or more tactile outputs and in accordance with a determination that the third set of one or more tactile outputs are scheduled to at least partially overlap with the first set of one or more tactile outputs and the second set of one or more tactile outputs;

reducing a scale of at least a portion of the first set of one or more tactile outputs that overlaps with the third set of one or more tactile outputs; and

reducing a scale of at least a portion of the second set of one or more tactile outputs that overlaps with the third set of one or more tactile outputs.

23. The electronic device of claim 22, wherein the third set of one or more tactile outputs with the third priority corresponds to one or more hardware elements distinct from the display.

24. The electronic device of claim 16, wherein the first set of one or more tactile outputs with the first priority corresponds to asynchronous events.

25. The electronic device of claim 16, wherein the second set of one or more tactile outputs with the second priority corresponds to synchronous events.

26. The electronic device of claim 16, wherein the one or more programs include instructions for, in response to detecting the triggering condition for the first set of one or more tactile outputs and the triggering condition for the second set of one or more tactile outputs and in accordance with the determination that the first set of one or more tactile outputs are scheduled to at least partially overlap with the second set of one or more tactile outputs and in accordance with the determination that the first priority is higher than the second priority, in conjunction with outputting, at reduced scale, at least the second portion of the second set of one or more tactile outputs with the second priority, outputting the first set of one or more tactile outputs with the first priority.

27. The electronic device of claim 16, wherein the one or more programs include instructions for, in response to detecting the triggering condition for the first set of one or more tactile outputs and the triggering condition for the second set of one or more tactile outputs and in accordance with the determination that the first set of one or more tactile outputs are scheduled to at least partially overlap with the second set of one or more tactile outputs and in accordance with the determination that the second priority is higher than the first priority, in conjunction with outputting, at reduced scale, at least the second portion of the first set of one or more tactile outputs with the first priority, outputting the second set of one or more tactile outputs with the second priority.

28. The electronic device of claim 16, wherein the one or more programs include instructions for:

playing an audio output that is synchronized with the first set of one or more tactile outputs or the second set of one or more tactile outputs; and

continuing to play the audio output without modification independently of whether or not the scale of at least the second portion of the first set of one or more tactile outputs that overlaps with the second set of one or more tactile outputs or the scale of at least the second portion of the second set of one or more tactile outputs that overlaps with the first set of one or more tactile outputs is reduced.

29. The electronic device of claim 16, wherein the one or more programs include instructions for:

receiving a first set of one or more inputs corresponding to user interface elements displayed on the display, wherein the first set of one or more inputs corresponds to the first set of one or more tactile outputs with the first priority; and

receiving a second set of one or more inputs corresponding to one or more hardware elements distinct from the display, wherein the second set of one or more inputs corresponds to the second set of one or more tactile outputs with the second priority;

wherein the second priority is higher than the first priority.

30. The electronic device of claim 29, wherein the second set of one or more inputs corresponds to a click gesture, or a portion of a click gesture, performed using a respective hardware element.

31. A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by an electronic device with a display, a touch-sensitive surface, and a set of one or more tactile output generators, cause the device to:

detect a triggering condition for a first set of one or more tactile outputs with a first priority;

detect a triggering condition for a second set of one or more tactile outputs with a second priority; and

in response to detecting the triggering condition for the first set of one or more tactile outputs and the triggering condition for the second set of one or more tactile outputs and in accordance with a determination that the first set of one or more tactile outputs are scheduled to at least partially overlap with the second set of one or more tactile outputs;

in accordance with a determination that the first priority is higher than the second priority, output the second set of one or more tactile outputs, including, in accordance with a determination that the second set of one or more tactile outputs includes a first portion that precedes a second portion that overlaps with the first set of one or more tactile outputs, reducing an output level of the first portion of the second set of one or more tactile outputs prior to outputting, at reduced scale, at least the second portion of the second set of one or more tactile outputs that overlaps with the first set of one or more tactile outputs; and

in accordance with a determination that the second priority is higher than the first priority, output the first set of one or more tactile outputs, including, in accordance with a determination that the first set of one or more tactile outputs includes a first portion that precedes a second portion that overlaps with the second set of one or more tactile outputs, reducing an output level of the first portion of the first set of one or more tactile outputs prior to outputting, at reduced scale, at least the second portion of the first set of one or more tactile outputs that overlaps with the second set of one or more tactile outputs.

32. The non-transitory computer readable storage medium of claim 31, wherein the one or more programs include instructions that, when executed by the electronic device, cause the device to:

generate combined tactile outputs by combining at least the first set of one or more tactile outputs, including any portion thereof with reduced scale, and the second set of one or more tactile outputs, including any portion thereof with reduced scale; and

output, with the set of one or more tactile output generators, a tactile output sequence based on the combined tactile outputs.

33. The non-transitory computer readable storage medium of claim 31, wherein the output level of the first portion of a respective set of one or more tactile outputs is gradually reduced from a first output level to a second output level.

34. The non-transitory computer readable storage medium of claim 31, wherein the second output level is greater than zero.

35. The non-transitory computer readable storage medium of claim 31, wherein the one or more programs include instructions that, when executed by the electronic device, cause the device, in accordance with a determination that the first priority is the same as the second priority, to combine the first set of one or more tactile outputs with the second set of one or more tactile outputs without reducing a scale of the first set of one or more tactile outputs relative to a scale of the second set of one or more tactile outputs.

36. The non-transitory computer readable storage medium of claim 31, wherein the one or more programs include instructions that, when executed by the electronic device, cause the device, in response to detecting the triggering condition for the first set of one or more tactile outputs and the triggering condition for the second set of one or more tactile outputs and in accordance with a determination that the first set of one or more tactile outputs is not scheduled to overlap with other tactile outputs, to output, with the set of one or more tactile output generators, the first set of one or more tactile outputs and the second set of one or more tactile outputs without reducing a scale of the first set of one or more tactile outputs and the second set of one or more tactile outputs.

37. The non-transitory computer readable storage medium of claim 31, wherein the one or more programs include instructions that, when executed by the electronic device, cause the device to:

detect a triggering condition for a third set of one or more tactile outputs with a third priority that is higher than the second priority and the first priority; and

in response to detecting the triggering condition for the third set of one or more tactile outputs and in accordance with a determination that the third set of one or more tactile outputs are scheduled to at least partially overlap with the first set of one or more tactile outputs and the second set of one or more tactile outputs;

reduce a scale of at least a portion of the first set of one or more tactile outputs that overlaps with the third set of one or more tactile outputs; and

reduce a scale of at least a portion of the second set of one or more tactile outputs that overlaps with the third set of one or more tactile outputs.

38. The non-transitory computer readable storage medium of claim 37, wherein the third set of one or more tactile outputs with the third priority corresponds to one or more hardware elements distinct from the display.

39. The non-transitory computer readable storage medium of claim 31, wherein the first set of one or more tactile outputs with the first priority corresponds to asynchronous events.

40. The non-transitory computer readable storage medium of claim 31, wherein the second set of one or more tactile outputs with the second priority corresponds to synchronous events.

41. The non-transitory computer readable storage medium of claim 31, wherein the one or more programs include instructions that, when executed by the electronic device, cause the device, in response to detecting the triggering condition for the first set of one or more tactile outputs and the triggering condition for the second set of one or more tactile outputs and in accordance with the determination that the first set of one or more tactile outputs are scheduled to at least partially overlap with the second set of one or more tactile outputs and in accordance with the determination that the first priority is higher than the second priority, in conjunction with outputting, at reduced scale, at least the second portion of the second set of one or more tactile outputs with the second priority, to output the first set of one or more tactile outputs with the first priority.

42. The non-transitory computer readable storage medium of claim 31, wherein the one or more programs include instructions that, when executed by the electronic device, cause the device, in response to detecting the triggering condition for the first set of one or more tactile outputs and the triggering condition for the second set of one or more tactile outputs and in accordance with the determination that the first set of one or more tactile outputs are scheduled to at least partially overlap with the second set of one or more tactile outputs and in accordance with the determination that the second priority is higher than the first priority, in conjunction with outputting, at reduced scale, at least the second portion of the first set of one or more tactile outputs with the first priority, to output the second set of one or more tactile outputs with the second priority.

43. The non-transitory computer readable storage medium of claim 31, wherein the one or more programs include instructions that, when executed by the electronic device, cause the device to:

play an audio output that is synchronized with the first set of one or more tactile outputs or the second set of one or more tactile outputs; and

continue to play the audio output without modification independently of whether or not the scale of at least the second portion of the first set of one or more tactile outputs that overlaps with the second set of one or more tactile outputs or the scale of at least the second portion of the second set of one or more tactile outputs that overlaps with the first set of one or more tactile outputs is reduced.

44. The non-transitory computer readable storage medium of claim 31, wherein the one or more programs include instructions that, when executed by the electronic device, cause the device to:

receive a first set of one or more inputs corresponding to user interface elements displayed on the display, wherein the first set of one or more inputs corresponds to the first set of one or more tactile outputs with the first priority; and

receive a second set of one or more inputs corresponding to one or more hardware elements distinct from the display, wherein the second set of one or more inputs corresponds to the second set of one or more tactile outputs with the second priority;

wherein the second priority is higher than the first priority.

45. The non-transitory computer readable storage medium of claim 44, wherein the second set of one or more inputs corresponds to a click gesture, or a portion of a click gesture, performed using a respective hardware element.