Dynamically merging multiple screens into one view port

• US 9,965,155 B2
• Filed: 03/27/2015
• Issued: 05/08/2018
• Est. Priority Date: 03/27/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Keysers, Daniel M., Deselaers, Thomas, Carbune, Victor
• Primary Examiner(s): VU, KIEU D
• Assistant Examiner(s): Basom, Blaine T.
• Application Number:
  US14/671,789
• Publication Number:
  US 20160283084A1
• Time in Patent Office: 1,138 Days
• CPC Class Codes: G01S 5/0072 Transmission between mobile stations, e.g. anti-collision systems
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/1423 controlling a plurality of local displays, e.g. CRT and flat panel display
• CPC Class Codes: H04L 67/52 specially adapted for the location of the user terminal
• CPC Class Codes: H04M 1/72412 using two-way short-range wireless interfaces
• CPC Class Codes: H04M 2250/16 including more than one display unit
• CPC Class Codes: H04M 2250/22 including a touch pad, a touch sensor or a touch detector
• CPC Class Codes: H04W 4/80 Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing dynamic, stroke-based alignment of touch displays. In one aspect, a method include providing, for output by a first mobile computing device that (i) has a first proximity sensitive display and (ii) has been designated a primary display device, a primary alignment user interface. The methods also includes transmitting, by the first mobile computing device to a second mobile computing device that (i) has a second proximity sensitive display and (ii) has been designated a secondary display device, an instruction to output a secondary alignment user interface.

Claims

1. A computer-implemented method comprising:

determining that a first mobile computing device that has a first proximity sensitive display with first screen resolution characteristics is associated with a second mobile computing device that has a second proximity sensitive display with different, second screen resolution characteristics;

obtaining touch data indicating multiple user inputs, the multiple user inputs including at least (i) a first stroke segment input along a first axis that continues from a portion of the first proximity sensitive display to a portion of the second proximity sensitive display, and (ii) a second stroke segment input along a second axis that continues from a different portion of the first proximity sensitive display to a different portion of the second proximity sensitive display, the first axis intersecting with the second axis at an intersection point between the first proximity sensitive display and the second proximity sensitive display;

determining, based on the obtained touch data, one or more differences between the first resolution characteristics of the first proximity sensitive display of the first mobile computing device and the second screen resolution characteristics of the second proximity sensitive display of the second mobile computing devices,

in response to determining that the first mobile computing device is associated with the second mobile computing device, automatically determining, based on the first stroke segment input and the second stroke segment input indicated by the obtained touch data, positional information of the first mobile computing device in relation to the second mobile computing device;

in response to automatically determining the positional information of the first mobile computing device in relation to the second mobile computing device, generating, using the touch data, a transform that accounts for the determined one or more differences between the first screen resolution characteristics of the first proximity sensitive display of the first mobile computing device and the second screen resolution characteristics of the second proximity sensitive display of the second mobile computing device; and

using the transform to dynamically merge the viewports of the first proximity sensitive display and the second proximity sensitive display despite the one or more differences between the first screen resolution characteristics of the first proximity sensitive display of the first mobile computing device and the second screen resolution characteristics of the second proximity sensitive display of the second mobile computing device.

2. The method of claim 1, comprising:

providing, for output by the first mobile computing device that (i) has a first proximity sensitive display and (ii) has been designated a primary display device, a primary alignment user interface;

transmitting, by the first mobile computing device to the second mobile computing device that (i) has a second proximity sensitive display and (ii) has been designated a secondary display device, an instruction to output a secondary alignment user interface;

receiving, by the first mobile computing device, (i) data indicative of one or more stroke segments input through the primary alignment user interface that is output on the first proximity sensitive display of the first mobile computing device, and, (ii) from the second mobile computing device, data indicative of one or more stroke segments input through the secondary alignment user interface that is output on the second proximity sensitive display of the second mobile computing device;

determining, based at least on the (i) data indicative of the one or more stroke segments input through the primary alignment user interface, and, (ii) data, received from the second mobile computing device, indicative of the one or more stroke segments input through the secondary alignment user interface, one or more display alignment parameters for the second proximity sensitive display, wherein the display alignment parameters for the second proximity sensitive display indicate an offset of the second proximity sensitive display in relation to the first proximity sensitive display;

determining, based at least on the display alignment parameters for the second proximity sensitive display, a transform for the second proximity sensitive display; and

using the transform for the second proximity sensitive display to process (i) a subsequently received input received through the second proximity sensitive display, or (ii) a subsequent output for display on the second proximity sensitive display.

3. The method of claim 2, wherein the alignment parameters comprise (i) a horizontal offset in relation to a reference point on the primary display device, (ii) a vertical offset in relation to a reference point on the primary display device, and (iii) an angular offset in relation to an axis associated with the primary display device.

4. The method of claim 2, wherein using the transform for the second proximity sensitive display comprises:

receiving, by the first mobile computing device and from the second mobile computing device, data indicative of the subsequently received input received through the second proximity sensitive display;

applying, by the first mobile computing device, the data indicative of the subsequently received input to the transform to generate a transformed input; and

providing the transformed input to an application executing on the first mobile computing device.

5. The method of claim 2, wherein using the transform for the second proximity sensitive display comprises:

receiving, by the first mobile computing device, data indicative of the subsequent output from an application executing on the first mobile computing device;

providing, by the first mobile computing device, a transformed output to the second mobile computing device.

6. The method of claim 2, comprising:

obtaining, by the first mobile computing device, data that characterizes the first mobile computing device;

receiving, by the first mobile computing device, data that characterizes the second mobile computing device; and

based at least on the (i) data that characterizes the first mobile computing device, and (ii) the data that characterizes the second mobile computing device, designates the first mobile computing device as the primary display device.

7. The method of claim 2, comprising:

after using the transform for the second proximity sensitive display, determining that the offset of the second proximity sensitive display in relation to the first proximity sensitive display has likely changed;

in response to determining that the offset of the second proximity sensitive display in relation to the first proximity sensitive display has likely changed, providing, for output by the first mobile computing device, the primary alignment user interface;

transmitting, by the first mobile computing device to the second mobile computing device, a second instruction to output the secondary alignment user interface;

receiving, by the first mobile computing device, (i) data indicative of one or more additional stroke segments input through the primary alignment user interface that is output on the first proximity sensitive display of the first mobile computing device, and, (ii) from the second mobile computing device, data indicative of one or more additional stroke segments input through the secondary alignment user interface that is output on the second proximity sensitive display of the second mobile computing device;

determining, based at least on the (i) data indicative of the one or more additional stroke segments input through the primary alignment user interface, and, (ii) data, received from the second mobile computing device, indicative of the one or more additional stroke segments input through the secondary alignment user interface, one or more subsequent display alignment parameters for the second proximity sensitive display;

determining, based at least on the subsequent display alignment parameters for the second proximity sensitive display, a second transform for the second proximity sensitive display; and

using the second transform for the second proximity sensitive display to process (i) a further received input received through the second proximity sensitive display, or (ii) a further output for display on the second proximity sensitive display.

8. The method of claim 2, wherein display alignment parameters are determined further based on (iii) a slope of one or more of the stroke segments input through the primary alignment user interface or the secondary alignment user interface, (iv) an identifier of a user of the first mobile computing device or the second mobile computing device, (v) timing information relating to input of one or more of the stroke segments input through the primary alignment user interface or the secondary alignment user interface, (vi) model information of the first mobile computing device or the second mobile computing device, or (vii) one or more characteristics of the first proximity sensitive display or the second proximity sensitive display.

9. A non-transitory computer-readable storage device encoded with a computer program, the program comprising instructions that when executed by a data processing apparatus cause the data processing apparatus to perform operations comprising:

determining that a first mobile computing device that has a first proximity sensitive display with first screen resolution characteristics is associated with a second mobile computing device that has a second proximity sensitive display with different, second screen resolution characteristics;

obtaining touch data indicating multiple user inputs, the multiple user inputs including at least (i) a first stroke segment input along a first axis that continues from a portion of the first proximity sensitive display to a portion of the second proximity sensitive display, and (ii) a second stroke segment input along a second axis that continues from a different portion of the first proximity sensitive display to a different portion of the second proximity sensitive display, the first axis intersecting with the second axis at an intersection point between the first proximity sensitive display and the second proximity sensitive display;

determining, based on the obtained touch data, one or more differences between the first resolution characteristics of the first proximity sensitive display of the first mobile computing device and the second screen resolution characteristics of the second proximity sensitive display of the second mobile computing devices,

in response to determining that the first mobile computing device is associated with the second mobile computing device, automatically determining, based on the first stroke segment input and the second stroke segment input indicated by the obtained touch data, positional information of the first mobile computing device in relation to the second mobile computing device;

in response to automatically determining the positional information of the first mobile computing device in relation to the second mobile computing device, generating, using the touch data, a transform that accounts for the determined one or more differences between the first screen resolution characteristics of the first proximity sensitive display of the first mobile computing device and the second screen resolution characteristics of the second proximity sensitive display of the second mobile computing device; and

using the transform to dynamically merge the viewports of the first proximity sensitive display and the second proximity sensitive display despite the one or more differences between the first screen resolution characteristics of the first proximity sensitive display of the first mobile computing device and the second screen resolution characteristics of the second proximity sensitive display of the second mobile computing device.

10. The computer-readable storage device of claim 9, comprising:

providing, for output by the first mobile computing device that (i) has a first proximity sensitive display and (ii) has been designated a primary display device, a primary alignment user interface;

transmitting, by the first mobile computing device to the second mobile computing device that (i) has a second proximity sensitive display and (ii) has been designated a secondary display device, an instruction to output a secondary alignment user interface;

receiving, by the first mobile computing device, (i) data indicative of one or more stroke segments input through the primary alignment user interface that is output on the first proximity sensitive display of the first mobile computing device, and, (ii) from the second mobile computing device, data indicative of one or more stroke segments input through the secondary alignment user interface that is output on the second proximity sensitive display of the second mobile computing device;

determining, based at least on the (i) data indicative of the one or more stroke segments input through the primary alignment user interface, and, (ii) data, received from the second mobile computing device, indicative of the one or more stroke segments input through the secondary alignment user interface, one or more display alignment parameters for the second proximity sensitive display, wherein the display alignment parameters for the second proximity sensitive display indicate an offset of the second proximity sensitive display in relation to the first proximity sensitive display;

determining, based at least on the display alignment parameters for the second proximity sensitive display, a transform for the second proximity sensitive display; and

using the transform for the second proximity sensitive display to process (i) a subsequently received input received through the second proximity sensitive display, or (ii) a subsequent output for display on the second proximity sensitive display.

11. The computer-readable storage device of claim 10, wherein the alignment parameters comprise (i) a horizontal offset in relation to a reference point on the primary display device, (ii) a vertical offset in relation to a reference point on the primary display device, and (iii) an angular offset in relation to an axis associated with the primary display device.

12. The computer-readable storage device of claim 10, wherein using the transform for the second proximity sensitive display comprises:

receiving, by the first mobile computing device and from the second mobile computing device, data indicative of the subsequently received input received through the second proximity sensitive display;

applying, by the first mobile computing device, the data indicative of the subsequently received input to the transform to generate a transformed input; and

providing the transformed input to an application executing on the first mobile computing device.

13. The computer-readable storage device of claim 10, wherein using the transform for the second proximity sensitive display comprises:

receiving, by the first mobile computing device, data indicative of the subsequent output from an application executing on the first mobile computing device;

applying, by the first mobile computing device, the data indicative of the subsequent output to the transform to generate a transformed output;

providing, by the first mobile computing device, the transformed output to the second mobile computing device.

14. The computer-readable storage device of claim 10, comprising:

obtaining, by the first mobile computing device, data that characterizes the first mobile computing device;

receiving, by the first mobile computing device, data that characterizes the second mobile computing device; and

based at least on the (i) data that characterizes the first mobile computing device, and (ii) the data that characterizes the second mobile computing device, designates the first mobile computing device as the primary display device.

15. The computer-readable storage device of claim 10, comprising:

after using the transform for the second proximity sensitive display, determining that the offset of the second proximity sensitive display in relation to the first proximity sensitive display has likely changed;

in response to determining that the offset of the second proximity sensitive display in relation to the first proximity sensitive display has likely changed, providing, for output by the first mobile computing device, the primary alignment user interface;

transmitting, by the first mobile computing device to the second mobile computing device, a second instruction to output the secondary alignment user interface;

receiving, by the first mobile computing device, (i) data indicative of one or more additional stroke segments input through the primary alignment user interface that is output on the first proximity sensitive display of the first mobile computing device, and, (ii) from the second mobile computing device, data indicative of one or more additional stroke segments input through the secondary alignment user interface that is output on the second proximity sensitive display of the second mobile computing device;

determining, based at least on the (i) data indicative of the one or more additional stroke segments input through the primary alignment user interface, and, (ii) data, received from the second mobile computing device, indicative of the one or more additional stroke segments input through the secondary alignment user interface, one or more subsequent display alignment parameters for the second proximity sensitive display;

determining, based at least on the subsequent display alignment parameters for the second proximity sensitive display, a second transform for the second proximity sensitive display; and

using the second transform for the second proximity sensitive display to process (i) a further received input received through the second proximity sensitive display, or (ii) a further output for display on the second proximity sensitive display.

16. The computer-readable storage device of claim 10, wherein display alignment parameters are determined further based on (iii) a slope of one or more of the stroke segments input through the primary alignment user interface or the secondary alignment user interface, (iv) an identifier of a user of the first mobile computing device or the second mobile computing device, (v) timing information relating to input of one or more of the stroke segments input through the primary alignment user interface or the secondary alignment user interface, (vi) model information of the first mobile computing device or the second mobile computing device, or (vii) one or more characteristics of the first proximity sensitive display or the second proximity sensitive display.

17. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

determining that a first mobile computing device that has a first proximity sensitive display with first screen resolution characteristics is associated with a second mobile computing device that has a second proximity sensitive display with different, second screen resolution characteristics;

obtaining touch data indicating multiple user inputs, the multiple user inputs including at least (i) a first stroke segment input along a first axis that continues from a portion of the first proximity sensitive display to a portion of the second proximity sensitive display, and (ii) a second stroke segment input along a second axis that continues from a different portion of the first proximity sensitive display to a different portion of the second proximity sensitive display, the first axis intersecting with the second axis at an intersection point between the first proximity sensitive display and the second proximity sensitive display;

determining, based on the obtained touch data, one or more differences between the first resolution characteristics of the first proximity sensitive display of the first mobile computing device and the second screen resolution characteristics of the second proximity sensitive display of the second mobile computing devices,

in response to determining that the first mobile computing device is associated with the second mobile computing device, automatically determining, based on the first stroke segment input and the second stroke segment input indicated by the obtained touch data, positional information of the first mobile computing device in relation to the second mobile computing device;

in response to automatically determining the positional information of the first mobile computing device in relation to the second mobile computing device, generating, using the touch data, a transform that accounts for the determined one or more differences between the first screen resolution characteristics of the first proximity sensitive display of the first mobile computing device and the second screen resolution characteristics of the second proximity sensitive display of the second mobile computing device; and

using the transform to dynamically merge the viewports of the first proximity sensitive display and the second proximity sensitive display despite the one or more differences between the first screen resolution characteristics of the first proximity sensitive display of the first mobile computing device and the second screen resolution characteristics of the second proximity sensitive display of the second mobile computing device.

18. The system of claim 17, comprising:

providing, for output by a first mobile computing device that (i) has a first proximity sensitive display and (ii) has been designated a primary display device, a primary alignment user interface;

transmitting, by the first mobile computing device to a second mobile computing device that (i) has a second proximity sensitive display and (ii) has been designated a secondary display device, an instruction to output a secondary alignment user interface;

receiving, by the first mobile computing device, (i) data indicative of one or more stroke segments input through the primary alignment user interface that is output on the first proximity sensitive display of the first mobile computing device, and, (ii) from the second mobile computing device, data indicative of one or more stroke segments input through the secondary alignment user interface that is output on the second proximity sensitive display of the second mobile computing device;

determining, based at least on the (i) data indicative of the one or more stroke segments input through the primary alignment user interface, and, (ii) data, received from the second mobile computing device, indicative of the one or more stroke segments input through the secondary alignment user interface, one or more display alignment parameters for the second proximity sensitive display, wherein the display alignment parameters for the second proximity sensitive display indicate an offset of the second proximity sensitive display in relation to the first proximity sensitive display;

determining, based at least on the display alignment parameters for the second proximity sensitive display, a transform for the second proximity sensitive display; and

using the transform for the second proximity sensitive display to process (i) a subsequently received input received through the second proximity sensitive display, or (ii) a subsequent output for display on the second proximity sensitive display.

19. The system of claim 18, wherein using the transform for the second proximity sensitive display comprises:

receiving, by the first mobile computing device and from the second mobile computing device, data indicative of the subsequently received input received through the second proximity sensitive display;

applying, by the first mobile computing device, the data indicative of the subsequently received input to the transform to generate a transformed input; and

providing the transformed input to an application executing on the first mobile computing device.

20. The system of claim 18, wherein using the transform for the second proximity sensitive display comprises:

receiving, by the first mobile computing device, data indicative of the subsequent output from an application executing on the first mobile computing device;

applying, by the first mobile computing device, the data indicative of the subsequent output to the transform to generate a transformed output;

providing, by the first mobile computing device, the transformed output to the second mobile computing device.