Multidevice session establishment for multimodal browsing

• US 8,204,995 B2
• Filed: 06/29/2005
• Issued: 06/19/2012
• Est. Priority Date: 06/29/2005
• Status: Active Grant

• Current Assignee: Nokia Technologies Oy (Nokia Corporation)
• Original Assignee: Nokia Corporation
• Inventors: Sathish, Sailesh
• Primary Examiner(s): Etienne, Ario
• Assistant Examiner(s): KIM, HEE SOO
• Application Number:
  US11/171,857
• Publication Number:
  US 20070005990A1
• Time in Patent Office: 2,547 Days
• Field of Search: 709/227, 709/228, 709/220
• US Class Current: 709/227
• CPC Class Codes: H04L 67/14 Session management for real-time applications in data packet communications networks H04L65/1066

Abstract

Systems, methods, apparatuses and computer program products for establishing a single session for processing a multimodal application with multiple devices and for distributing the multimodal application amongst the multiple devices participating in the single session based on the respective capabilities of each device are provided. A system, method and computer program product for allowing new devices operated by the same user to join in the previously established session for processing the multimodal application where the capabilities of the new device exceed that of all devices currently participating in the session are also provided.

Claims

1. A method comprising:

receiving a request to establish a session for processing a multimodal application from each of one or more devices;

determining whether said one or more devices are operated by a common user;

determining to create a composite capability profile for the devices determined to be operated by said user, based at least in part on multimodal profiles associated with each of said one or more devices, said composite capability profile comprises capabilities of one or more of said one or more devices for each modality;

determining to select, for each modality, a device from said one or more devices based on its capabilities;

determining to generate a unique session identification (ID) for a single session; and

determining to provide for transmission of said unique session ID to the selected devices,

where said request to establish as session for processing the multimodal application includes a multimodal profile associated with the device from which the request was received, said multimodal profile including at least one or more capabilities of said device, and wherein determining whether said one or more devices are operated by said user comprises determining whether said unique user IDS is the same for each of said one or more devices.

2. The method of claim 1 further comprising the steps of:

receiving the multimodal application in response to a request therefore, said multimodal application including one or more requirements for using one or more modalities in association with said application;

determining which of said multiple devices participating in said single session will use each of said one or more modalities based at least in part on said one or more requirements included in said multimodal application and said composite capability profile; and

determining to provide for the distribution of at least part of said multimodal application to said multiple devices based on the determination made in the previous step.

3. The method of claim 2, wherein said multimodal application comprises markup language for each of said one or more modalities, and wherein distributing at least part of said multimodal application to said multiple devices comprises determining to divide said multimodal application into one or more modality markups and determining to transmit each of said one or more modality markups to a respective one of said multiple devices.

4. The method of claim 2, wherein said one or more modalities used in association with said multimodal application includes speech recognition, handwriting, and keystrokes.

5. The method of claim 2, wherein one modality used in association with said multimodal application is speech recognition, and said one or more requirements for using said speech recognition in association with the multimodal application includes a specification of a required vocabulary grammar size.

6. The method of claim 2, wherein determining which of said multiple devices participating in said single session will use each of said one or more modalities further comprises:

determining whether said one or more requirements for using said one or more modalities in association with said multimodal application are supported by said multiple devices; and

in an instance in which one or more modalities are not supported, then determining to provide for transmission of a request for a second multimodal application requiring lesser modality support, and receiving said second multimodal application.

7. The method of claim 6, wherein determining to provide for the distribution of at least part of said multimodal application to said multiple devices comprises determining to provide for the distribution of at least part of said second multimodal application.

8. A method comprising:

receiving a request for a multimodal application from one of multiple devices;

receiving the multimodal application in response to the request, said multimodal application including one or more requirements for each of one or more modalities used in association with said multimodal application;

determining which of said multiple devices is configured to use each of said one or more modalities in association with said multimodal application based at least in part on said one or more requirements and a composite capability profile associated with a single session for processing the multimodal application, wherein said composite capability profile comprises a compilation of one or more capabilities associated with each of said multiple devices including capabilities of one or more of the multiple devices for each modality;

determining to select, for each modality, a device from the one or more devices based on its capabilities;

and

determining to provide for the distribution of at least part of said multimodal application to said multiple devices based on the selection.

9. The method of claim 8, wherein said multimodal application comprises markup language for each of said one or more modalities, and wherein determining to provide for the distribution of at least part of said multimodal application to said multiple devices includes determining to divide said markup language into one or more modality markups and determining to provide for the transmission of each of said one or more modality markups to a respective one of said multiple devices.

10. The method of claim 8, wherein said one or more modalities include speech recognition, handwriting and keystrokes.

11. The method of claim 8, wherein determining which of said multiple devices is configured to use each of said one or more modalities in association with said multimodal application further comprises:

determining whether said one or more requirements for using said one or more modalities in association with said multimodal application are supported by said multiple devices; and

in an instance in which said one or more modalities are not supported then determining to provide for the transmission of a request for a second multimodal application requiring lesser modality support, and receiving said second multimodal application.

12. The method of claim 11, wherein determining to provide for the distribution of at least part of said multimodal application to said multiple devices comprises determining to provide for the distribution of at least part of said second multimodal application.

13. A method comprising:

receiving a request to establish a session for processing a multimodal application from a new device, wherein said request includes a multimodal profile associated with said new device, said multimodal profile including one or more capabilities of said new device;

determining whether the new device is operated by a user that is the operator of one or more current devices in a previously established single session for processing a multimodal application;

determining whether said new device should process the multimodal application based on said one or more capabilities of said new device, wherein the determination comprises determining whether said one or more capabilities of said new device exceed a composite capability profile of the previously established single session, wherein the composite capability profile is based on a combination of the one more capabilities of said one or more current devices for each modality;

determining to establish a new session with said one or more current devices and said new device upon a determination that said new device is operated by said user and should process said multimodal application; and

determining to select, for each modality, a device from the one or more devices based on its capabilities,

wherein a multimodal profile is associated with each of said one more current devices, said multimodal profile including said one more capabilities of the current device with which the multimedia profile is associated and a unique user identification (ID) identifying said user as operator of said current device.

14. The method of claim 13 further comprising:

determining to terminate said previously established single session.

15. The method of claim 13, wherein said multimodal profile associated with said new device further includes one or more user preferences associated with said new device, and wherein said multimodal profile associated with each of said one or more current devices further includes one or more user preferences associated with said current device.

16. The method of claim 15 further comprising:

determining whether said user preferences associated with said new device provide an indication that said user would prefer to allow said new device to join said single session;

determining whether said user preferences associated with said current devices provide an indication that said user would prefer to allow said new device to join said single session; and

determining to establish a new session with said one or more current devices and said new device upon a determination that said new device is operated by said user and that said one or more user preferences of said new device or of said current devices provide an indication that said user would prefer to allow said new device to join said session.

17. The method of claim 13, wherein determining whether said new device is operated by said user comprises determining whether said unique user ID included in said multimodal profile of said new device is the same as the unique user ID included in the multimodal profiles of said current devices.

18. An apparatus comprising:

at least one processor and at least one memory including computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause an apparatus to perform at least the following;

receive a request to establish a session from each of one or more devices operated by a common user, said request including a multimodal profile associated with the device from which the request was received, said multimodal profile including one or more capabilities of said device and a unique user identification (ID) identifying said common user as the user of said device;

determine to communicate with a profile manager, wherein said profile manager uses said multimodal profile of each of said one or more devices to create a composite capability profile for a single session being established, wherein the composite capability profile comprises capabilities of one or more of the one or more devices for each modality, and to select, for each modality, a device from the one or more devices based on its capabilities; and

determine to communicate with a unique identification (ID) generator, wherein said unique ID generator generates a unique session ID for the single session being established,

wherein said one or more capabilities are applied for using said one or more modalities in association with a multimodal application and said composite capability profile to determine which of said one or more devices should use each of said one or more modalities in processing said multimodal application.

19. The apparatus of claim 18, wherein the apparatus is further caused to:

determine to communicate with an application server, wherein said application server provides said apparatus with said multimodal application.

20. The apparatus of claim 18 wherein at least one of said one or more devices communicates with a media server, and wherein said device establishes a session with said media server in order to process said multimodal application using said modality with which it was assigned by said interaction manager.

21. The apparatus of claim 18, wherein said apparatus and said profile manager comprise one device.

22. The apparatus of claim 18, wherein said apparatus and said unique ID generator comprise one device.

23. An apparatus comprising:

at least one processor and at least one memory including computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause an apparatus to perform at least the following;

determine to establish a session with one or more devices determined to be operated by a user,

determine to create a composite capability profile for said session, said composite capability profile including a compilation of one or more capabilities of each of said one or more devices for each modality, wherein a multimedia profile is associated with each of said one or more devices, said multimedia profile including said one or more capabilities of said device associated with the multimedia profile and a unique user identification (ID) identifying said user of said device,

determine to select, for each modality, a device from the one or more devices based on its capabilities,

determine to create a unique session ID for said session,

determine to query said composite capability profile to determine whether said one or more devices are configured to process a multimodal application and which of said one or more devices should process each of one or more modalities associated with said multimodal application based on the selection,

determine to provide for distribution of at least part of said multimodal application to said one or more devices based on the selection, and

determine to use said unique user IDS to determine whether said one or more devices are operated by said user.

24. The apparatus of claim 23, wherein creating the composite capability profile further causes the apparatus to:

determine to create the composite capability profile based upon said multimodal profile of each of said one or more devices.

25. The apparatus of claim 23, wherein querying said composite capability profile to determine which of said one or more devices should process each of said one or more modalities associated with said multimodal application further causes the apparatus to:

determine to analyze one or more requirements for each of said one or more modalities included in said multimodal application; and

determine, based at least in part on said one or more capabilities for each of said one or more devices included in said composite capability profile, which of said one or more devices meets each of said one or more requirements.

26. The apparatus of claim 23, wherein determining to provide for the distribution of at least part of said multimodal application to said one or more devices further causes the apparatus to:

determine to divide said multimodal application into one or more modality markups; and

determine to provide for the transmission of each of said one or more modality markups to a respective one of said one or more devices.

27. The apparatus comprising:

at least one processor and at least one memory including computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause an apparatus to perform at least the following;

receive one or more multimodal profiles associated with said multiple devices, each of said multimodal profiles including one or more capabilities of the device with which the multimodal profile is associated and including one or more user preferences associated with the device with which the multimedia profile is associated;

determine to combine said one or more capabilities from each of said multimodal profiles to create a composite capability profile associated with a single session comprising capabilities of one or more of the multiple devices for each modality;

determine to combine said one or more user preferences from each of said multimodal profiles to include in said composite capability profile associated with the single session;

determine to provide for the storage of said composite capability profile; and

determine to select, for each modality, a device from the one or more devices based on its capabilities.

28. The apparatus of claim 27, wherein the apparatus is in communication with an interaction manager, said interaction manager configured to access said composite capability profile in order to determine which of said multiple devices will use each of one or more modalities to process said multimodal application.

29. A non-transitory computer readable-storage medium carrying one or more sequences of one or more instructions which, when executed by one or more processors, cause an apparatus to perform at least the following:

receiving a request to establish a session for processing a multimodal application from each of one or more devices;

determining whether said one or more devices are operated by a common user;

determining to create a composite capability profile for the devices determined to be operated by said user, based at least in part on multimodal profiles associated with each of said one or more devices, wherein the composite capability profile comprises a compilation of one or more capabilities associated with each of said multiple devices for each modality;

determining to select, for each modality, a device from the one or more devices based on its capabilities;

determining to generate a unique session identification (ID) for said a single session; and

determining to provide for the transmission of said unique session ID to the selected devices,

wherein said request to establish a session for processing the multimodal application includes a multimodal profile associated with the device from which the request was received, said multimodal profile including one or more capabilities of said device and unique user identification (ID) identifying an operator of said user, and

wherein creating a composite capability profile causes the apparatus to further perform determining to combine the capabilities included in the multimedia profile of said one or more devices determined to be operated by said user.

30. The non-transitory computer-readable storage medium of claim 29, wherein determining whether said one or more devices are operated by said user causes the apparatus to further perform:

determining whether said unique user ID is the same for each of said one or more devices.

31. A non-transitory computer readable-storage medium carrying one or more sequences of one or more instructions which, when executed by one or more processors, cause an apparatus to perform at least the following:

receiving a request for a multimodal application from one of multiple devices;

receiving the multimodal application in response to the request, said multimodal application including one or more requirements for each of one or more modalities used in association with said multimodal application;

determining which of said multiple devices is configured to use each of said one or more modalities in association with said multimodal application based at least in part on said one or more requirements and a composite capability profile associated with a single session, wherein said composite capability profile comprises a compilation of one or more capabilities associated with each of said multiple devices including capabilities of one or more of the multiple devices for each modality;

determining to select, for each modality, a device from the one or more devices based on its capabilities, and

determining to provide for the distribution of at least part of said multimodal application to said multiple devices based on the selection.

32. The non-transitory computer-readable storage medium of claim 31, wherein said multimodal application comprises markup language for each of said one or more modalities, and wherein determining to provide for the distribution of at least part of said multimodal application to said multiple devices causes the apparatus to further perform:

determining to divide said markup language into one or more modality markups; and

determining to provide for the transmission of each of said one or more modality markups to a respective one of said multiple devices.

33. The non-transitory computer-readable storage medium of claim 31, wherein said one or more modalities include speech recognition, handwriting and keystrokes.

34. A non-transitory computer readable-storage medium carrying one or more sequences of one or more instructions which, when executed by one or more processors, cause an apparatus to perform at least the following:

receiving a request to establish a session for processing a multimodal application from a new device, wherein said request includes a multimodal profile associated with said new device, said multimodal profile including one or more capabilities of said new device;

determining whether the new device is operated by a user that is the operator of one or more current devices in a previously established single session for processing a multimodal application;

determining whether said new device should process the multimodal application based on said one or more capabilities of said new device, wherein the determination comprises determining whether said one or more capabilities of said new device exceed a composite capability profile of the previously established single session, wherein the composite capability profile is based on a combination of the one more capabilities of said one or more current devices for each modality;

determining to establish a new session with said one or more current devices and said new device upon a determination that said new device is operated by said user and should process the multimodal application; and

determining to select, for each modality, a device from the one or more devices based on its capabilities,

wherein a multimodal profile is associated with each of said one or more current devices, said multimodal profile including said one more capabilities of the current device with which the multimedia profile is associated and a unique user identification (ID) identifying said user as operator of said current device.

35. The non-transitory computer-readable storage medium of claim 34, wherein the apparatus is caused to further perform:

determining to terminate a previously established single session.

36. The non-transitory computer-readable storage medium of claim 34, wherein said multimodal profile associated with said new device further includes one or more user preferences associated with said new device, and wherein said multimodal profile associated with each of said one or more current devices further includes one or more user preferences associated with said current device.

37. The non-transitory computer-readable storage medium of claim 36, wherein said apparatus is caused to further perform:

determining whether said user preferences associated with said new device provide an indication that said user would prefer to allow said new device to join said single session;

determining whether said user preferences associated with said current devices provide an indication that said user would prefer to allow said new device to join said single session; and

determining to establish a new session with said one or more current devices and said new device upon a determination that said new device is operated by said user and that said one or more user preferences of said new device or of said current devices provide an indication that said user would prefer to allow said new device to join said session.

38. The non-transitory computer-readable storage medium of claim 34, wherein said multimodal profile associated with said new device includes a unique user identification (ID) identifying an operator of said new device.

39. The non-transitory computer-readable storage medium of claim 38, wherein determining whether said new device is operated by said user causes the apparatus to further perform:

determining whether said unique user ID included in said multimodal profile of said new device is the same as the unique user ID included in the multimodal profiles of said current devices.

40. A method comprising:

determining to establish a single session with the multiple devices operated by a common user;

determining to create a composite capability profile based at least in part on multimodal profiles associated with each of the multiple devices, said composite capability profile comprises a compilation of one or more capabilities associated with each of the multiple devices for each modality;

determining to select, for each modality, a device from the one or more devices based on its capabilities;

determining to provide for the distribution of at least part of a multimodal application to the multiple devices based at least in part on the composite capability profile; and

determining to process the multimodal application using the multiple devices.

41. The method of claim 40, wherein establishing a single session further comprises:

determining whether said multiple devices are operated by said user;

determining to generate a unique session identification (ID) for said single session; and

determining to provide for the transmission of said unique session ID to those devices operated by said user.

42. The method of claim 40, wherein providing for the distribution of the multimodal application to the multiple devices further comprises:

determining to receive a multimodal application, wherein said multimodal application includes one or more requirements for using each of one or more modalities in association with said multimodal application;

determining to query said composite capability profile to determine which of said multiple devices is configured to use each of said one or more modalities in association with said multimodal application, based at least in part on said one or more requirements included in said multimodal application;

determining to divide said multimodal application into one or more modality markups, wherein each modality markup is associated with a respective one of said one or more modalities used in association with said multimodal application; and

determining to provide for the distribution of said one or more modality markups to said multiple devices based on a determination of which of said multiple devices is configured to use each of said one or more modalities in association with the multimodal application.