Method and apparatus for providing a multimodal user interface track

• US 9,436,300 B2
• Filed: 07/10/2012
• Issued: 09/06/2016
• Est. Priority Date: 07/10/2012
• Status: Active Grant

• Current Assignee: Fixapa Sendirian Berhad
• Original Assignee: Nokia Technologies Oy (Nokia Corporation)
• Inventors: Sathish, Sailesh Kumar, Curcio, Igor Danilo Diego
• Primary Examiner(s): Pham, Linh K
• Assistant Examiner(s): NICHOLAS, WENDY K
• Application Number:
  US13/545,667
• Publication Number:
  US 20140019860A1
• Time in Patent Office: 1,519 Days
• Field of Search: 715/716, 715/723, 715/726, 715200-204, 715/210, 715/211, 715/233, 386/200, 386/232, 386/223, 386/220, 386/239, 386/240, 386/278, 386/282, 386/331, 386/353, 386/356, 725/37, 725/20, 725/10, 725/34, 725/47, 725/54
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: H04N 21/4621 Controlling the complexity of the content stream or additional data, e.g. lowering the resolution or bit-rate of the video stream for a mobile client with a small screen arrangements for using the results of monitoring on user's side in broadcast systems H04H60/65; flow control in packet networks H04L47/10

Abstract

An approach is provided for providing a multimodal user interface track. A multimodal generation platform determining one or more user interface elements for interacting with at least one media segment. The multimodal generation platform further causes, at least in part, an inclusion of the one or more user interface elements as at least one track of the at least one media segment. Accordingly, when the at least one track is processed during a presentation of the at least one media segment by at least one device, the at least one track causes, at least in part, an enablement of the one or more user interface elements.

Claims

1. A method comprising:

on a device, presenting a user interface including a user interface element of a first track associated with a first user interface modality type of a media segment;

including one or more scripts for initiating one or more actions associated with the media segment in at least one track of the media segment;

determining, with a processor, one or more detected changes in contextual information associated with at least one of the user of the device or the device, wherein the one or more changes are detected during the presentation of the media segment;

identifying an additional user interface modality type associated with the one or more detected changes in the contextual information;

identifying a template associated with the additional user interface modality type;

processing the media segment with respect to the template to modify the user interface element during the presentation based on the template and the identified additional user interface modality type associated with the one or more changes, wherein the user interface modality type comprises at least one of a visual modality, an audio modality, a tactile modality, an olfactory modality or a combination thereof;

identifying one or more semantic elements based on the user interface modality type, wherein the one or more semantic elements describe characteristics or features of the media segment; and

including the one or more semantic elements as at least one other track of the media segment.

2. The method of claim 1, further comprising:

processing any of the capability information, preference information, contextual information, or a combination thereof associated with the one device to identify the user interface modality type.

3. The method of claim 2 further comprising:

determining that the capability information, preference information, the contextual information, or a combination thereof associated with the device is at least partially incomplete; and

identifying the user interface modality type associated with the one or more changes based on one or more default criteria.

4. The method of claim 1, further comprising:

processing the media segment with respect to one or more words, one or more tokens, or a combination thereof in an audio modality template to track one or more topics across tracks.

5. The method of claim 4, wherein the one or more words, the one or more tokens, or a combination thereof are based on one or more datasets located at the device, and the one or more tokens include grammar tokens for speech.

6. The method of claim 1,

wherein the one or more scripts are processed during the presentation.

7. The method of claim 1, further comprising:

in response to receiving an indication of a contextual event, cause a cancelling or limiting of the user interface modality type that requires more battery power or is prohibited based on the presence of a software protection dongle attached to the device, or a combination thereof.

8. The method of claim 1, wherein the media segment further represents one of a plurality of views from one single location, the track is processed during the presentation of the media segment by the device to cause an enablement of the one or more user interface elements for switching among views from the one single location, and the views from the one single location include one or more wide-angle views, one or more telephotos, one or more front views, one or more rear views, one or more left views, one or more right views, one or more top views, or a combination thereof.

9. The method of claim 1, wherein the template is associated with the user interface modality.

10. The method of claim 1, wherein the template is a multi-modal template, and the presentation is modified to include at least two modalities.

11. An apparatus comprising:

at least one processor; and

at least one memory including computer program code for one or more programs, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to perform at least the following;

on a device, present a user interface including a user interface element of a first track associated with a first user interface modality type of a media segment;

include one or more scripts for initiating one or more actions associated with the media segment in at least one track of the media segment;

determine one or more detected changes in contextual information associated with at least one of a user of the device or the device, wherein the one or more changes are detected during the presentation of the media segment;

identify an additional user interface modality type associated with the one or more detected changes in the contextual information by executing the one or more scripts;

identify a template associated with the additional user interface modality type; and

process the media segment with respect to the template to modify the user interface element during the presentation based on the template and the identified user interface modality type associated with the one or more changes, wherein the user interface modality type comprises at least one of a visual modality, an audio modality, a tactile modality, an olfactory modality or a combination thereof;

identify one or more semantic elements based on the user interface modality type, wherein the one or more semantic elements describe characteristics or features of the media segment; and

include the one or more semantic elements as at least one other track of the media segment.

12. The apparatus of claim 11, wherein the at least one memory and the computer program code are further configured to cause the apparatus to:

process any of the capability information, preference information, contextual information, or a combination thereof associated with the device to identify the user interface modality type.

13. The apparatus of claim 12, wherein the at least one memory and the computer program code are further configured to cause the apparatus to:

determine that the capability information, preference information, the contextual information, or a combination thereof associated with the device is at least partially incomplete; and

identify the user interface modality type associated with the one or more changes based on one or more default criteria.

14. The apparatus of claim 11, wherein the at least one memory and the computer program code are further configured to cause the apparatus to:

process the media segment with respect to one or more words, one or more tokens, or a combination thereof in an audio modality template to modify the user interface element.

15. The apparatus of claim 14, wherein the one or more words, the one or more tokens, or a combination thereof are based on one or more datasets located at the device.

16. The apparatus of claim 11,

wherein the one or more scripts are processed during the presentation.