Gesture visualization and sharing between electronic devices and remote displays

• US 9,152,373 B2
• Filed: 04/12/2011
• Issued: 10/06/2015
• Est. Priority Date: 04/12/2011
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: King, Nicholas V.
• Primary Examiner(s): Yang, Kwang-Su
• Application Number:
  US13/084,779
• Publication Number:
  US 20120262379A1
• Time in Patent Office: 1,638 Days
• Field of Search: 345/173, 715/740
• US Class Current: 1/1
• CPC Class Codes: G06F 3/1454 involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
• CPC Class Codes: G09G 2340/0407 Resolution change, inclusive of the use of different resolutions for different screen areas
• CPC Class Codes: G09G 2340/06 Colour space transformation
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 2370/04 Exchange of auxiliary data, i.e. other than image data, between monitor and graphics controller
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality

Abstract

The disclosed embodiments provide a system that facilitates interaction between an electronic device and a remote display. The system includes a first application and an encoding apparatus on the electronic device, and a second application and a decoding apparatus on the remote display. The encoding apparatus obtains graphical output for a display of the electronic device and a first set of touch inputs associated with the graphical output from a first touch screen. Next, the encoding apparatus encodes the graphical output, and the first application transmits the graphical output and the first set of touch inputs to the remote display. Upon receiving the graphical output and the first set of touch inputs at the remote display, the decoding apparatus decodes the graphical output. The second application then uses the graphical output and a visual representation of the first set of touch inputs to drive the remote display.

Claims

1. A computer-implemented method for interacting with a remote display, comprising:

obtaining graphical output for a display of an electronic device;

obtaining a first set of touch inputs associated with the graphical output from a touch screen associated with the electronic device; and

transmitting the graphical output and the first set of touch inputs to the remote display, wherein the graphical output and a visual representation of the first set of touch inputs are used to drive the remote display, and wherein the visual representation of the first set of touch inputs comprises a set of dots and is not displayed on the display of the electronic device.

2. The computer-implemented method of claim 1, further comprising:

encoding the graphical output prior to transmitting the graphical output to the remote display.

3. The computer-implemented method of claim 2, wherein encoding the graphical output comprises:

converting the graphical output from a first color space to a second color space.

4. The computer-implemented method defined in claim 2, wherein encoding the graphical output prior to transmitting the graphical output to the remote display comprises:

scaling the graphical output.

5. The computer-implemented method of claim 1, further comprising:

receiving a second set of touch inputs from the remote display; and

updating the graphical output based on the second set of touch inputs.

6. The computer-implemented method of claim 5, wherein updating the graphical output based on the second set of touch inputs comprises:

providing the second set of touch inputs to an application configured to generate the graphical output.

7. The computer-implemented method of claim 6, wherein updating the graphical output based on the second set of touch inputs further comprises:

identifying the remote display as a source of the second set of touch inputs, wherein the identified remote display enables modification of the graphical output by the application prior to transmitting the graphical output to the remote display.

8. The computer-implemented method of claim 7, wherein the identified remote display further enables the transmission of data from the application to the remote display based on the second set of touch inputs.

9. The computer-implemented method of claim 1, wherein transmitting the graphical output and the first set of touch inputs to the remote display comprises at least one of:

compositing the visual representation of the first set of touch inputs into the graphical output; and

transmitting the first set of touch inputs as auxiliary data associated with the graphical output to the remote display.

10. The computer-implemented method defined in claim 1, wherein transmitting the graphical output and the first set of touch inputs to the remote display comprises:

transmitting the graphical output to the remote display in a selected communication channel; and

transmitting the first set of touch inputs to the remote display in a sideband communication channel associated with the selected communication channel.

11. A computer-implemented method for interacting with an electronic device, comprising:

receiving a graphical output and a first set of touch inputs associated with the graphical output from a touch screen display in the electronic device; and

using the graphical output and a visual representation of the first set of touch inputs to drive a remote touch screen display, wherein the visual representation of the first set of touch inputs is not displayed on the touch screen display in the electronic device, and wherein using the graphical output and the visual representation of the first set of touch inputs to drive the remote touch screen display comprises;

drawing the graphical output to a first buffer;

drawing the visual representation of the first set of touch inputs to a second buffer; and

using the first and second buffers to drive the remote touch screen display.

12. The computer-implemented method of claim 11, further comprising:

decoding the graphical output prior to using the graphical output to drive the remote touch screen display.

13. The computer-implemented method of claim 11, further comprising:

obtaining a second set of touch inputs associated with the graphical output from the remote touch screen display; and

transmitting the second set of touch inputs to the electronic device, wherein the second set of touch inputs is used by the electronic device to update the graphical output.

14. The computer-implemented method of claim 13, wherein the second set of touch inputs is further used by the electronic device to transmit data to the remote touch screen display.

15. A system for facilitating interaction between an electronic device and a remote display, comprising:

a first application on the electronic device, wherein the first application is configured to;

obtain a graphical output for a display of the electronic device;

obtain a first set of touch inputs associated with the graphical output from a first touch screen on the electronic device; and

transmit the graphical output and the first set of touch inputs to the remote display, wherein transmitting the graphical output and the first set of touch inputs to the remote display comprises;

transmitting the graphical output to the remote display in a selected communication channel; and

transmitting the first set of touch inputs to the remote display in a sideband communication channel associated with the selected communication channel; and

a second application on the remote display, wherein the second application is configured to use the graphical output and a visual representation of the first set of touch inputs to drive the remote display, and wherein the visual representation of the first set of touch inputs is not displayed on the display of the electronic device.

16. The system of claim 15, further comprising:

an encoding apparatus on the electronic device, wherein the encoding apparatus is configured to encode the graphical output prior to transmitting the graphical output to the remote display; and

a decoding apparatus on the remote display, wherein the decoding apparatus is configured to decode the graphical output prior to using the graphical output to drive the remote display.

17. The system of claim 15, wherein the second application is further configured to:

obtain a second set of touch inputs associated with the graphical output from a second touch screen on the remote display; and

transmit the second set of touch inputs to the electronic device, wherein the first application is further configured to update the graphical output based on the second set of touch inputs.

18. The system of claim 17, wherein the first application is further configured to identify the remote display as a source of the second set of touch inputs.

19. The system of claim 18, wherein the identified remote display enables at least one of:

modification of the graphical output by the first application prior to transmitting the graphical output to the remote display; and

transmission of data from the first application to the remote display based on the second set of touch inputs.

20. The system of claim 15, wherein transmitting the graphical output and the first set of touch inputs to the remote display involves at least one of:

compositing the visual representation of the first set of touch inputs into the graphical output; and

transmitting the first set of touch inputs as auxiliary data associated with the graphical output to the remote display.

21. The system of claim 15, wherein the electronic device is at least one of a mobile phone, a tablet computer, and a portable media player.

22. A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for interacting with a remote display, the method comprising:

obtaining graphical output for a display of the computer;

obtaining a first set of touch inputs associated with the graphical output from a touch screen associated with the computer; and

transmitting the graphical output and the first set of touch inputs to the remote display, wherein the graphical output and a visual representation of the first set of touch inputs are used to drive the remote display, and wherein the visual representation of the first set of touch inputs comprises a set of dots and is not displayed on the display of the computer.

23. The non-transitory computer-readable storage medium of claim 22, the method further comprising:

receiving a second set of touch inputs from the remote display; and

updating the graphical output based on the second set of touch inputs.

24. The non-transitory computer-readable storage medium of claim 23, wherein updating the graphical output based on the second set of touch inputs comprises:

providing the second set of touch inputs to an application configured to generate the graphical output.

25. The non-transitory computer-readable storage medium of claim 22, wherein transmitting the graphical output and the first set of touch inputs to the remote display comprises at least one of:

compositing the visual representation of the first set of touch inputs into the graphical output; and

transmitting the first set of touch inputs as auxiliary data associated with the graphical output to the remote display.