Method, apparatus and computer program product for joining the displays of multiple devices

• US 9,213,480 B2
• Filed: 04/08/2010
• Issued: 12/15/2015
• Est. Priority Date: 04/08/2010
• Status: Active Grant

• Current Assignee: Nokia Technologies Oy (Nokia Corporation)
• Original Assignee: Nokia Technologies Oy (Nokia Corporation)
• Inventors: Kernen, Jaakko, Korhonen, Hannu, Lucero, Andres
• Primary Examiner(s): LEGGETT, ANDREA C.
• Application Number:
  US12/756,796
• Publication Number:
  US 20110252317A1
• Time in Patent Office: 2,077 Days
• Field of Search: 715/769, 715/702, 715/766
• US Class Current: 1/1
• CPC Class Codes: G06F 1/1698 the I/O peripheral being a sending/receiving arrangement to establish a cordless communication link, e.g. radio or infrared link, integrated cellular phone details of antennas disposed inside a computer H01Q1/2266
• CPC Class Codes: G06F 2200/1614 Image rotation following screen orientation, e.g. switching from landscape to portrait mode
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/1423 controlling a plurality of local displays, e.g. CRT and flat panel display
• CPC Class Codes: G06F 3/1446 display composed of modules, e.g. video walls
• CPC Class Codes: G09G 2356/00 Detection of the display position w.r.t. other display screens
• CPC Class Codes: H04M 1/72412 using two-way short-range wireless interfaces
• CPC Class Codes: H04M 1/72445 for supporting Internet browser applications
• CPC Class Codes: H04M 2250/10 including a GPS signal receiver
• CPC Class Codes: H04M 2250/22 including a touch pad, a touch sensor or a touch detector

Abstract

A method is provided for enhancing a user interface with a mobile device by joining the displays of multiple devices together to function together with one another. In particular, example methods may provide for detecting a first touch or drag event, receiving a message indicating a second touch or drag event from a nearby device occurring during the first touch or drag event, creating a join event, and providing for transmission of the join event to the nearby device. The method may also include updating a registry of nearby devices in response to receiving a message regarding a touch or drag event from a nearby device. The method may further include providing for presentation of content on a display of a device and direction of content to be presented on a display of the nearby device.

Claims

1. A method comprising:

detecting a first drag event, in a first direction, which occurs on a display of a device;

determining, from the detected first drag event and direction information included in a message transmitted wirelessly to the device by a nearby device, that a second drag event has occurred on a display of the nearby device in a second direction that is substantially opposite to the first direction, wherein the first drag event and the second drag event correspond with a user performing a pinching motion;

determining that the second drag event occurred during the first drag event;

creating a join event in response to determining at least that the second drag event occurred on the display of the nearby device in the second direction substantially opposite to the first direction and that the second drag event occurred during the first drag event; and

providing for transmission of the join event to the nearby device.

2. The method according to claim 1, wherein the join event includes join event rules and wherein the join event rules include data regarding content to be displayed on the nearby device.

3. The method according to claim 1, further comprising:

providing for presentation of content on the display of the device; and

providing for direction of content to be presented on the display of the nearby device, wherein the content to be presented on the display of the nearby device is related to the content presented on the display of the device.

4. The method according to claim 1, wherein detecting the first drag event in the first direction comprises detecting that the first drag event was directed towards a first edge of the display of the device, and determining that the second drag event occurred in the second direction substantially opposite the first direction comprises determining that the second drag event was directed towards a second edge of the display of the nearby device.

5. The method according to claim 1, wherein join event rules for the join event define an interaction between the device and the nearby device that depends upon the relative orientation of the device and the nearby device.

6. The method according to claim 5, wherein the device and the nearby device are joined according to first join event rules when the device and the nearby device have a first relative orientation and the device and the nearby device are joined according to a second join event rules when the device and the nearby device have a second relative orientation.

7. The method according to claim 6, wherein in the first relative orientation, the device and the nearby device have the same orientation, and in the second relative orientation, the nearby device is orientated perpendicularly to the device.

8. An apparatus comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to at least perform:

detecting a first drag event, in a first direction, that occurs on a display of a device;

determining, from the detected first drag event and direction information included in a message transmitted wirelessly to the device by a nearby device, that a second drag event has occurred on a display of the nearby device in a direction that is substantially opposite to the first direction, wherein the first drag event and the second drag event correspond with a user performing a pinching motion;

determining that the second drag event occurred during the first drag event;

creating a join event in response to determining at least that the second drag event occurred on the display of the nearby device in the second direction substantially opposite to the first direction and that the second drag event occurred during the first drag event; and

providing for transmission of the join event to the nearby device.

9. The apparatus according to claim 8, wherein the join event includes join event rules and wherein the join event rules include data regarding content to be displayed on the nearby device.

10. The apparatus according to claim 8, wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to further perform:

providing for presentation of content on the display of the device; and

providing for direction of content to be presented on the display of the nearby device, wherein the content to be presented on the display of the nearby device is related to the content presented on the display of the device.

11. The apparatus according to claim 8, wherein the type of information provided for display on the joined device is determined based upon the direction of the second drag event.

12. The apparatus according to claim 8, wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to expand information presented on the display of the device during the first drag event in response to the join event.

13. The apparatus according to claim 8, wherein detecting the first drag event in the first direction comprises detecting that the first drag event was directed towards a first edge of the display of the device, and determining that the second drag event occurred in the second direction substantially opposite the first direction comprises determining that the second drag event was directed towards a second edge of the display of the nearby device.

14. The apparatus according to claim 8, wherein join event rules for the join event define an interaction between the device and the nearby device that depends upon the relative orientation of the device and the nearby device.

15. The apparatus according to claim 14, wherein the device and the nearby device are joined according to first join event rules when the device and the nearby device have a first relative orientation and the device and the nearby device are joined according to a second join event rules when the device and the nearby device have a second relative orientation.

16. The apparatus according to claim 15, wherein in the first relative orientation, the device and the nearby device have the same orientation, and in the second relative orientation, the nearby device is orientated perpendicularly to the device.

17. The apparatus according to claim 8, wherein the apparatus is the device.

18. A computer program product comprising at least one non-transitory computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions comprising:

program code instructions for detecting a first drag event, in a first direction, which occurs on a display of a device;

program code instructions for determining, from the detected first drag event and direction information included in a message transmitted wirelessly to the device by a nearby device, that a second drag event has occurred on a display of the nearby device in a direction that is substantially opposite to the first direction, wherein the first drag event and the second drag event correspond with a user performing a pinching motion;

program code for determining that the second drag event occurred during the first drag event;

program code instructions for creating a join event in response to determining at least that the second drag event occurred on the display of the nearby device in the second direction substantially opposite to the first direction and that the second drag event occurred during the first drag event;

program code instructions for providing for transmission of the join event to the nearby device.

19. The computer program product according to claim 18, wherein the join event includes join event rules and wherein the join event rules include data regarding content to be displayed on the nearby device.

20. The computer program product according to claim 18, further comprising program code instructions for providing for presentation of content on the display of the device and program code instructions for directing of content to be presented on the display of the nearby device, wherein the content to be presented on the display of the nearby device is related to the content presented on the display of the device.

21. The computer program product according to claim 18, wherein detecting the first drag event in the first direction comprises detecting the first drag event was directed towards a first edge of the display of the device, and determining that the second drag event occurred in the second direction, substantially opposite the first direction, comprises determining that the second drag event was directed towards a second edge of the display of the nearby device.