User device having sequential multimodal output user interface

• US 10,133,372 B2
• Filed: 12/20/2007
• Issued: 11/20/2018
• Est. Priority Date: 12/20/2007
• Status: Active Grant

• Current Assignee: Nokia Technologies Oy (Nokia Corporation)
• Original Assignee: Nokia Technologies Oy (Nokia Corporation)
• Inventors: Makela, Kaj Markus, Uusitalo, Jussi Severi
• Primary Examiner(s): ORTIZ SANCHEZ, MICHAEL
• Application Number:
  US12/004,772
• Publication Number:
  US 20090164207A1
• Time in Patent Office: 3,988 Days
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry

Abstract

In one aspect of the exemplary embodiments of this invention an apparatus includes a user interface that contains a plurality of input modalities and a plurality of output modalities, and a data processor coupled with the user interface and configurable to present a user with a content item that includes a plurality of attributes. In response to user input that data processor is operable to partition at least some of the attributes into a plurality of presentation tokens, where an individual presentation token comprises at least one attribute. The data processor is further configurable to respond to further user input to define one of the plurality of input modalities to generate a trigger condition for individual ones of the presentation tokens, where generation of a trigger condition results in an associated presentation token being made manifest to the user. The plurality of input modalities may include two or more of physical or virtual keys, an input acoustic transducer, a speech recognition unit, and a gesture detection unit, and where the plurality of output modalities may include two or more of an output acoustic transducer, a speech synthesis unit, a vibro-tactile transducer, and a display screen.

Claims

1. A method, comprising:

operating at least one data processor of a handheld mobile device to present a user of the handheld mobile device with a notification of an event on a display screen, wherein the event comprises a plurality of attributes;

using an input modality of one of a plurality of input modalities of the mobile device and the at least one data processor to partition at least some of the attributes into a plurality of presentation tokens, where an individual presentation token comprises at least one attribute of the event;

defining one of the plurality of input modalities to generate a trigger condition for at least some individual ones of the presentation tokens, where generation of a trigger condition results in an associated presentation token being made manifest to the user via one of a plurality of output modalities of the handheld mobile device that is selectable by the user;

detecting user interaction in response to presenting the user with the notification of the event; and

generating one or more trigger conditions based on the user interaction.

2. The method of claim 1, where one trigger condition is generated by activation of a physical or virtual key of a user input device.

3. The method of claim 1, where one trigger condition is generated by recognition of a word spoken by the user.

4. The method of claim 1, where one trigger condition is generated by recognition of a gesture made by the user.

5. The method of claim 1, further comprising in response to sensing a current state of the mobile device, automatically selecting a different output modality than the output modality selected by the user to make the associated presentation token manifest to the user of the mobile device.

6. The method of claim 1, where one output modality comprises an acoustic transducer.

7. The method of claim 1, where one output modality comprises a speech synthesis unit.

8. The method of claim 1, where one output modality comprises a transducer capable of generating a haptic sensation.

9. The method of claim 1, where one output modality comprises a display screen.

10. The method of claim 1, where generation of the trigger condition by a certain input modality for a particular presentation token automatically defines one of a plurality of output modalities for making the particular presentation token manifest to the user.

11. The method of claim 1, where a plurality of the trigger conditions are generated sequentially by actions of a user for causing the corresponding plurality of presentation tokens to be made manifest to the user.

12. The method of claim 1, wherein:

the presenting a user with the notification of the event, and the partitioning at least some of the attributes into a plurality of presentation tokens are performed by the at least one data processor at the handheld mobile device.

14. The method of claim 1, wherein the plurality of presentation tokens comprises a sequence of presentation tokens, and wherein a plurality of trigger conditions are generated for causing at least some of the presentation tokens to be made manifest to the user in order of the sequence of presentation tokens.

15. The method of claim 1, wherein the notification of the event is generated by an application stored in a memory of the handheld mobile device.

16. The method of claim 1, wherein the notification of the event is at least one of:

a calendar notification;

an alarm notification; and

a call notification.

17. The method of claim 1, wherein the attributes of the event comprise at least one of:

a name, a date, a location, and a time corresponding to the event.

13. A method, comprising:

operating at least one data processor of a mobile device to present a user of the mobile device with a notification on a display screen that comprises a plurality of attributes corresponding to the notification;

in response to user input from an input modality of one of a plurality of input modalities of the mobile device, generating, using the at least one data processor, a sequence of presentation tokens by partitioning at least some of the attributes, where an individual presentation token comprises at least one attribute;

receiving a first predefined user input from an input modality of one of the plurality of input modalities in response to presenting the user with the notification;

in response to receiving the first predefined user input, providing a first presentation token of the sequence of presentation tokens to the user via one of a plurality of output modalities of the mobile device that is selectable by the user;

receiving a second predefined user input from an input modality of one of the plurality of input modalities; and

in response to the second predefined user input, providing a second presentation token of the sequence of presentation tokens to the user via one of the plurality of output modalities of the mobile device that is selectable by the user, wherein the second presentation token refines the level of detail or expands the amount of information as compared to the first presentation token.

30. The method of claim 13, wherein in response to the second predefined input, the second presentation token is provided to the user via a different one of the plurality of output modalities.

31. The method of claim 13, wherein the second predefined user input is a repeat of the first predefined user input.

18. A non-transitory computer-readable medium that stores program instructions, the execution of the program instructions by at least one data processor of a handheld mobile device results in operations that comprise:

presenting a user with a notification of an event on a display screen, wherein the event comprises a plurality of attributes;

in response to user input from an input modality of one of a plurality of input modalities of the mobile device, partitioning at least some of the attributes into a plurality of presentation tokens, where an individual presentation token comprises at least one attribute of the event;

in response to further user input, defining one of the plurality of input modalities to generate a trigger condition for at least some individual ones of the presentation tokens, where generation of a trigger condition results in an associated presentation token being made manifest to the user via one of a plurality of output modalities of the handheld mobile device that is selectable by the user; and

detecting user interaction in response to presenting the user with the notification of the event;

generating one or more trigger conditions based on the user interaction.

19. The computer-readable medium of claim 18, where one trigger condition is generated by activation of a physical or virtual key of a user input device, where another trigger condition is generated by recognition of a word spoken by the user, and where another trigger condition is generated by recognition of a gesture made by the user.

20. The computer-readable medium of claim 18, further comprising an operation of, in response to sensing a current state of the handheld mobile device, automatically selecting a different output modality than the output modality selected by the user to make the associated presentation token manifest to the user of the handheld mobile device.

21. The computer-readable medium of claim 20, where one output modality comprises an acoustic transducer, where another output modality comprises a speech synthesis unit, where another output modality comprises a transducer capable of generating a haptic sensation, and where another output modality comprises a display screen.

22. The computer-readable medium of claim 18, where generation of the trigger condition by a certain input modality for a particular presentation token automatically defines one of a plurality of output modalities for making the particular presentation token manifest to the user.

23. The computer-readable medium of claim 18, where a plurality of the trigger conditions are generated sequentially by actions of a user for causing the corresponding plurality of presentation tokens to be made manifest to the user.

24. The non-transitory computer-readable medium of claim 18, wherein the execution of the program instructions by the at least one data processor occurs at the handheld mobile device.

25. An apparatus, comprising:

a user interface comprised of a plurality of input modalities and a plurality of output modalities;

a data processor; and

a non-transitory memory including computer program code, the memory and the computer program code configured to, with the data processor, cause the apparatus at least to;

present a user with a notification of an event on a touch-sensitive display screen, wherein the event comprises a plurality of attributes and, in response to user input, to partition at least some of the attributes into a plurality of presentation tokens, where an individual presentation token comprises at least one attribute of the event;

respond to further user input to define one of said plurality of input modalities to generate a trigger condition for at least some individual ones of the presentation tokens, where generation of a trigger condition results in an associated presentation token being made manifest to the user via one of the plurality of output modalities that is selectable by the user;

detecting user interaction in response to presenting the user with the notification of the event; and

generate one or more trigger conditions based on the user interaction,

wherein the apparatus is a handheld mobile device.

26. The apparatus of claim 25, where the memory and the computer program code are configured to, with the data processor, cause the apparatus at least to:

respond to sensing a current state of the apparatus, to automatically select a different output modality than the output modality selected by the user to make the associated presentation token manifest to the user.

27. The apparatus of claim 26, where said plurality of input modalities comprise two or more of physical or virtual keys, an input acoustic transducer, and the touch-sensitive display screen, and where said plurality of output modalities comprise two or more of an output acoustic transducer, a transducer capable of generating a haptic sensation, and the touch-sensitive display screen.

28. The apparatus of claim 25, where the memory and the computer program code are configured to, with the data processor, cause the apparatus at least to:

respond to generation of a trigger condition by a certain one of said plurality of input modalities, for a particular presentation token, to automatically select one of said plurality of output modalities for making the particular presentation token manifest to the user.

29. The apparatus of claim 25, wherein:

the presenting a user with the notification of the event, and the partitioning at least some of the attributes into a plurality of presentation tokens are performed by the at least one data processor at the handheld mobile device.