Multidevice session establishment for multimodal browsing

• US 20070005990A1
• Filed: 06/29/2005
• Published: 01/04/2007
• Est. Priority Date: 06/29/2005
• Status: Active Grant

• Current Assignee: Nokia Technologies Oy (Nokia Corporation)
• Original Assignee: Nokia Corporation
• Inventors: Sathish, Sailesh
• Granted Patent: US 8,204,995 B2
• Time in Patent Office: Days
• US Class Current: 713/189
• CPC Class Codes: H04L 67/14 Session management for real-time applications in data packet communications networks H04L65/1066

Abstract

Systems, methods, apparatuses and computer program products for establishing a single session for processing a multimodal application with multiple devices and for distributing the multimodal application amongst the multiple devices participating in the single session based on the respective capabilities of each device are provided. A system, method and computer program product for allowing new devices operated by the same user to join in the previously established session for processing the multimodal application where the capabilities of the new device exceed that of all devices currently participating in the session are also provided.

Claims

1. A method of establishing a single session for processing a multimodal application with multiple devices operated by a user, said method comprising:

receiving a request to establish a session for processing a multimodal application from each of one or more devices;

determining whether said one or more devices are operated by said user;

generating a unique session identification (ID) for said single session; and

transmitting said unique session ID to those devices operated by said user.

2. The method of claim 1 further comprising:

creating a composite capability profile for those devices determined to be operated by said user.

3. The method of claim 2, wherein said request to establish a session for processing the multimodal application includes a multimodal profile associated with the device from which the request was received, said multimodal profile including at least one or more capabilities of said device and a unique user identification (ID) identifying an operator of said device.

4. The method of claim 3, wherein determining whether said one or more devices are operated by said user comprises determining whether said unique user ID is the same for each of said one or more devices.

5. The method of claim 3, wherein creating a composite capability profile comprises combining the capabilities included in the multimodal profiles of said one or more devices determined to be operated by said user.

6. The method of claim 2 further comprising the steps of:

receiving the multimodal application in response to a request therefore, said multimodal application including one or more requirements for using one or more modalities in association with said application;

determining which of said multiple devices participating in said single session will use each of said one or more modalities based at least in part on said one or more requirements included in said multimodal application and said composite capability profile; and

distributing at least part of said multimodal application to said multiple devices based on the determination made in the previous step.

7. The method of claim 6, wherein said multimodal application comprises markup language for each of said one or more modalities, and wherein distributing at least part of said multimodal application to said multiple devices comprises dividing said multimodal application into one or more modality markups and transmitting each of said one or more modality markups to a respective one of said multiple devices.

8. The method of claim 6, wherein said one or more modalities used in association with said multimodal application includes speech recognition, handwriting, and keystrokes.

9. The method of claim 6, wherein one modality used in association with said multimodal application is speech recognition, and said one or more requirements for using said speech recognition in association with the multimodal application includes a specification of a required vocabulary grammar size.

10. The method of claim 6, wherein said step of determining which of said multiple devices participating in said single session will use each of said one or more modalities further comprises:

determining whether said one or more requirements for using said one or more modalities in association with said multimodal application are supported by said multiple devices; and

if said one or more modalities are not supported, then transmitting a request for a second multimodal application requiring lesser modality support, and receiving said second multimodal application.

11. The method of claim 10, wherein distributing at least part of said multimodal application to said multiple devices comprises distributing at least part of said second multimodal application.

12. A method of distributing at least part of a multimodal application amongst multiple devices operated by a user and participating in a single session for processing the multimodal application, said single session having a composite capability profile associated with said single session, said composite capability profile including at least a compilation of one or more capabilities associated with each of said multiple devices, said method comprising:

receiving a request for the multimodal application from one of said multiple devices;

receiving the multimodal application in response to the request, said multimodal application including one or more requirements for each of one or more modalities used in association with said multimodal application;

determining which of said multiple devices is capable of using each of said one or more modalities in association with said multimodal application based at least in part on said one or more requirements and said composite capability profile; and

distributing at least part of said multimodal application to said multiple devices based on a determination of which of said multiple devices is capable of using each of said one or more modalities.

13. The method of claim 12, wherein said multimodal application comprises markup language for each of said one or more modalities, and wherein distributing at least part of said multimodal application to said multiple devices includes dividing said markup language into one or more modality markups and transmitting each of said one or more modality markups to a respective one of said multiple devices.

14. The method of claim 12, wherein said one or more modalities include speech recognition, handwriting and keystrokes.

15. The method of claim 12, wherein determining which of said multiple devices is capable of using each of said one or more modalities in association with said multimodal application further comprises:

determining whether said one or more requirements for using said one or more modalities in association with said multimodal application are supported by said multiple devices; and

if it is determined that said one or more modalities are not supported then transmitting a request for a second multimodal application requiring lesser modality support, and receiving said second multimodal application.

16. The method of claim 15, wherein distributing at least part of said multimodal application to said multiple devices comprises distributing at least part of said second multimodal application.

17. A method of allowing a new device to join in a previously established single session for processing a multimodal application with one or more current devices operated by a user, said previously established single session having a composite capability profile based on a combination of one more capabilities of said one or more current devices, said method comprising:

receiving a request to establish a session for processing a multimodal application from the new device, wherein said request includes a multimodal profile associated with said new device, said multimodal profile including one or more capabilities of said new device;

determining whether the new device is operated by said user;

determining whether said new device should process the multimodal application based on said one or more capabilities of said new device; and

establishing a new session with said one or more current devices and said new device upon a determination that said new device is operated by said user and should process said multimodal application.

18. The method of claim 17, wherein determining whether said new device should process the multimodal application comprises determining whether said one or more capabilities of said new device exceed the composite capability profile of the previously established single session.

19. The method of claim 17 further comprising:

terminating said previously established single session.

20. The method of claim 17, wherein a multimodal profile is associated with each of said one or more current devices, said multimodal profile including said one or more capabilities of the current device with which the multimodal profile is associated and a unique user identification (ID) identifying said user as operator of said current device.

21. The method of claim 20, wherein said multimodal profile associated with said new device further includes one or more user preferences associated with said new device, and wherein said multimodal profile associated with each of said one or more current devices further includes one or more user preferences associated with said current device.

22. The method of claim 21 further comprising:

determining whether said user preferences associated with said new device provide an indication that said user would prefer to allow said new device to join said single session;

determining whether said user preferences associated with said current devices provide an indication that said user would prefer to allow said new device to join said single session; and

establishing a new session with said one or more current devices and said new device upon a determination that said new device is operated by said user and that said one or more user preferences of said new device or of said current devices provide an indication that said user would prefer to allow said new device to join said session.

23. The method of claim 20 wherein said multimodal profile associated with said new device includes a unique user identification (ID) identifying an operator of said new device.

24. The method of claim 23, wherein determining whether said new device is operated by said user comprises determining whether said unique user ID included in said multimodal profile of said new device is the same as the unique user ID included in the multimodal profiles of said current devices.

25. A system for establishing a single session for processing a multimodal application with multiple devices operated by a user, said multimodal application including one or more requirements for using one or more modalities in association with said multimodal application, said system comprising:

an interaction manager that receives a request to establish a session from each of one or more devices operated by a common user, said request including a multimodal profile associated with the device from which the request was received, said multimodal profile including one or more capabilities of said device and a unique user identification (ID) identifying said common user as the user of said device;

a profile manager in communication with said interaction manager, wherein said profile manager uses said multimodal profile of each of said one or more devices to create a composite capability profile for the session being established; and

a unique identification (ID) generator in communication with said interaction manager, wherein said unique ID generator generates a unique session ID for the session being established,

wherein said interaction manager uses said one or more requirements for using said one or more modalities in association with said multimodal application and said composite capability profile to determine which of said one or more devices should use each of said one or more modalities in processing said multimodal application.

26. The system of claim 25 further comprising:

an application server in communication with said interaction manager, wherein said application server provides said interaction manager with said multimodal application.

27. The system of claim 25 further comprising:

a media server in communication with at least one of said one or more devices, wherein said device establishes a session with said media server in order to process said multimodal application using said modality with which it was assigned by said interaction manager.

28. The system of claim 25, wherein said interaction manager and said profile manager comprise one device.

29. The system of claim 25, wherein said interaction manager and said unique ID generator comprise one device.

30. An interaction manager used to manage a single session for processing a multimodal application with multiple devices and to distribute at least part of said multimodal application to each of said multiple devices, said interaction manager comprising:

a processor capable of establishing a session with one or more devices determined to be operated by one user,

wherein said processor is further capable of creating a composite capability profile for said session, said composite capability profile including a compilation of one or more capabilities of each of said one or more devices,

wherein said processor is further capable of creating a unique session ID for said session,

wherein said processor is further capable of querying said composite capability profile to determine whether said one or more devices are capable of processing said multimodal application and which of said one or more devices should process each of one or more modalities associated with said multimodal application, and

wherein said processor is further capable of distributing at least part of said multimodal application to said one or more devices based on a determination of which of said one or more devices should process each of said one or more modalities.

31. The interaction manager of claim 30, wherein a multimodal profile is associated with each of said one or more devices, said multimodal profile including said one or more capabilities of said device associated with the multimodal profile and a unique user identification (ID) identifying an operator of said device.

32. The interaction manager of claim 31, wherein creating the composite capability profile comprises creating the composite capability profile based upon said multimodal profile of each of said one or more devices.

33. The interaction manager of claim 31, wherein said processor is further capable of using said unique user ID to determine whether said one or more devices are operated by one user.

34. The interaction manager of claim 30, wherein querying said composite capability profile to determine which of said one or more devices should process each of said one or more modalities associated with said multimodal application comprises analyzing one or more requirements for each of said one or more modalities included in said multimodal application and determining, based at least in part on said one or more capabilities for each of said one or more devices included in said composite capability profile, which of said one or more devices meets each of said one or more requirements.

35. The interaction manager of claim 30, wherein distributing at least part of said multimodal application to said one or more devices comprises dividing said multimodal application into one or more modality markups and transmitting each of said one or more modality markups to a respective one of said one or more devices.

36. A profile manager module for creating and storing a composite capability profile associated with a single session for processing a multimodal application with multiple devices, said profile manager module comprising:

a processor capable of receiving one or more multimodal profiles associated with said multiple devices, each of said multimodal profiles including one or more capabilities of the device with which it is associated,

wherein said processor is further capable of combining said one or more capabilities from each of said multimodal profiles to create a composite capability profile associated with said single session, and

wherein said processor is further capable of storing said composite capability profile.

37. The profile manager of claim 36, wherein each of said multimodal profiles further includes one or more user preferences associated with the device with which the multimodal profile is associated, and wherein said processor is further capable of combining said one or more user preferences from each of said multimodal profiles to include in said composite capability profile associated with the single session.

38. The profile manager module of claim 37, wherein said profile manager module is in communication with an interaction manager, said interaction manager configured to access said composite capability profile in order to determine which of said multiple devices will use each of one or more modalities to process said multimodal application.

39. A computer program product for establishing a single session for processing a multimodal application with multiple devices operated by a user, wherein the computer program product comprises at least one computer-readable storage medium having computer-readable program code portions stored therein, the computer-readable program code portions comprising:

a first executable portion for receiving a request to establish a session for processing a multimodal application from each of one or more devices;

a second executable portion for determining whether said one or more devices are operated by said user;

a third executable portion for generating a unique session identification (ID) for said single session; and

a fourth executable portion for transmitting said unique session ID to those devices operated by said user.

40. The computer program product of claim 39, wherein said computer-readable program code portions further comprise:

a fifth executable portion for creating a composite capability profile for those devices determined to be operated by said user.

41. The computer program product of claim 40, wherein said request to establish a session for processing the multimodal application includes a multimodal profile associated with the device from which the request was received, said multimodal profile including one or more capabilities of said device and a unique user identification (ID) identifying an operator of said user.

42. The computer program product of claim 41, wherein determining whether said one or more devices are operated by said user comprises determining whether said unique user ID is the same for each of said one or more devices.

43. The computer program product of claim 41, wherein creating a composite capability profile comprises combining the capabilities included in the multimodal profile of said one or more devices determined to be operated by said user.

44. A computer program product for distributing at least part of a multimodal application amongst multiple devices operated by a user and participating in a single session for processing a multimodal application, said single session having a composite capability profile associated with said session, said composite capability profile including a compilation of one or more capabilities associated with each of said multiple devices, wherein the computer program product comprises at least one computer-readable storage medium having computer-readable program code portions stored therein, the computer-readable program code portions comprising:

a first executable portion for receiving a request for the multimodal application from one of said multiple devices;

a second executable portion for receiving the multimodal application in response to the request, said multimodal application including one or more requirements for each of one or more modalities used in associated with said multimodal application;

a third executable portion for determining which of said multiple devices is capable of using each of said one or more modalities in association with said multimodal application based at least in part on said one or more requirements and said composite capability profile; and

a fourth executable portion for distributing at least part of said multimodal application to said multiple devices based on a determination of which of said multiple devices is capable of using each of said one or more modalities.

45. The computer program product of claim 44, wherein said multimodal application comprises markup language for each of said one or more modalities, and wherein distributing at least part of said multimodal application to said multiple devices includes dividing said markup language into one or more modality markups and transmitting each of said one or more modality markups to a respective one of said multiple devices.

46. The computer program product of claim 44, wherein said one or more modalities include speech recognition, handwriting and keystrokes.

47. A computer program product for allowing a new device to join in a previously established single session for processing a multimodal application with one or more current devices operated by a user, said previously established single session having a composite capability profile based on a combination of one or more capabilities of said one or more current devices, wherein the computer program product comprises at least one computer-readable storage medium having computer-readable program code portions stored therein, the computer-readable program code portions comprising:

a first executable portion for receiving a request to establish a session for processing a multimodal application from the new device, wherein said request includes a multimodal profile associated with said new device, said multimodal profile including one or more capabilities of said new device;

a second executable portion for determining whether the new device is operated by said user;

a third executable portion for determining whether said new device should process the multimodal application based on said one or more capabilities of said new device; and

a fourth executable portion for establishing a new session with said one or more current devices and said new device upon a determination that said new device is operated by said user and should process the multimodal application.

48. The computer program product of claim 47, wherein determining whether said new device should process the multimodal application comprises determining whether said one or more capabilities of said new device exceed the composite capability profile of the previously established single session.

49. The computer program product of claim 47, wherein the computer-readable program code portions further comprise:

a fifth executable portion for terminating said previously established single session.

50. The computer program product of claim 47, wherein a multimodal profile is associated with each of said one or more current devices, said multimodal profile including said one or more capabilities of the current device with which the multimodal profile is associated and a unique user identification (ID) identifying said user as operated of said current device.

51. The computer program product of claim 50, wherein said multimodal profile associated with said new device further includes one or more user preferences associated with said new device, and wherein said multimodal profile associated with each of said one or more current devices further includes one or more user preferences associated with said current device.

52. The computer program product of claim 51, wherein said computer-readable program code portions further comprise:

a fifth executable portion for determining whether said user preferences associated with said new device provide an indication that said user would prefer to allow said new device to join said single session;

a sixth executable portion for determining whether said user preferences associated with said current devices provide an indication that said user would prefer to allow said new device to join said single session; and

a sixth executable portion for establishing a new session with said one or more current devices and said new device upon a determination that said new device is operated by said user and that said one or more user preferences of said new device or of said current devices provide an indication that said user would prefer to allow said new device to join said session.

53. The computer program product of claim 47, wherein said multimodal profile associated with said new device includes a unique user identification (ID) identifying an operator of said new device.

54. The computer program product of claim 53, wherein determining whether said new device is operated by said user comprises determining whether said unique user ID included in said multimodal profile of said new device is the same as the unique user ID included in the multimodal profiles of said current devices.

55. A method of conducting a single session for processing a multimodal application with multiple devices operated by a user, said method comprising:

establishing a single session with the multiple devices;

distributing at least part of the multimodal application to the multiple devices based at least in part on one or more capabilities of each of the multiple devices; and

processing the multimodal application using the multiple devices.

56. The method of claim 55, wherein establishing a single session further comprises:

determining whether said multiple devices are operated by said user;

generating a unique session identification (ID) for said single session; and

transmitting said unique session ID to those devices operated by said user.

57. The method of claim 55, wherein distributing the multimodal application to the multiple devices further comprises:

creating a composite capability profile for the single session based at least in part on one or more capabilities of each of said multiple devices;

receiving a multimodal application, wherein said multimodal application includes one or more requirements for using each of one or more modalities in association with said multimodal application;

querying said composite capability profile to determine which of said multiple devices is capable of using each of said one or more modalities in association with said multimodal application, based at least in part on said one or more requirements included in said multimodal application;

dividing said multimodal application into one or more modality markups, wherein each modality markup is associated with a respective one of said one or more modalities used in associated with said multimodal application; and

distributing said one or more modality markups to said multiple devices based on a determination of which of said multiple devices is capable of using each of said one or more modalities in association with the multimodal application.