Automatic discovery and mirroring of server-client remote user interface (RUI) session on a companion device and synchronously controlling both sessions using RUI on companion device

• US 9,173,000 B2
• Filed: 04/12/2013
• Issued: 10/27/2015
• Est. Priority Date: 04/12/2013
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Clift, Graham, Kuo, Jenke Wu
• Primary Examiner(s): Pan, Yongjia
• Application Number:
  US13/861,610
• Publication Number:
  US 20140310599A1
• Time in Patent Office: 928 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 9/452 Remote windowing, e.g. X-Window System, desktop virtualisation protocols for virtual reality H04L67/131
• CPC Class Codes: H04N 21/41265 having a remote control device for bidirectional communication between the remote control device and client device unidirectional remote control device H04N21/42204
• CPC Class Codes: H04N 21/4227 Providing Remote input by a user located remotely from the client device, e.g. at work
• CPC Class Codes: H04N 21/472 End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content end-user interfaces for retrieving video data from a database G06F16/739; network services for supporting unicast streaming H04L65/612
• CPC Class Codes: H04N 21/482 End-user interface for program selection broadcast systems using EPGs H04H60/72

Abstract

A server sends to a video device such as a TV a remote user interface (RUI) that is presented on the TV and manipulable to send control commands back to the server. A companion device such as a tablet computer discovers the RUI session and is provided by the server with its own RUI, which mirrors that on the TV, modified as appropriate for the screen of the companion device. The server maintains the two RUIs synchronized such that the RUI on the companion mirrors the RUI on the TV.

Claims

1. A device comprising:

at least one computer memory that is not a transitory signal and that comprises instructions executable by at least one processor of a companion device to an audio video display device (AVDD) to;

receive from a server information pertaining to rendering a remote user interface (RUI), a version of which is presented on the AVDD, display locations on the RUI presented on the companion device being selectable by a user to send information correlatable to control commands for the AVDD to cause the AVDD to execute respective functions, a control command generated using the RUI on the companion device causes execution of at least one of;

change a source of video for presentation of the video on the AVDD, change volume output by the AVDD, perform a trick play function on the AVDD, record a video presented on the AVDD; and

receive from the server updates to the RUI such that the RUI presented on the companion device is synchronized with the version presented on the AVDD, wherein

the instructions are executable to receive, from the server, the information pertaining to rendering the RUI in a low level graphics card protocol via a transmission control protocol/Internet Protocol (TCP/IP) socket, the information pertaining to rendering the RUI defining how to render RUI elements of the RUI but not defining functionality of the RUI elements, such that selection of an RUI element causes sending a signal back to the server indicating a screen location at which the selection occurred, which the server then correlates to the appropriate function underlying the selected RUI element.

2. The device of claim 1, wherein the instructions are executable to receive the information pertaining to rendering the RUI formatted for rendering the RUI in a hypertext markup language that supports graphics rendering commands using vector graphics, in addition to bitmap graphics commands.

3. The device of claim 1, wherein the RUI on the AVDD is not in HTML5.

4. The device of claim 1, wherein the instructions are executable to present the RUI on the companion device modified from the version presented on the AVDD as appropriate for a display of the companion device.

5. The device of claim 1, wherein the instructions are executable to establish communications with the server.

6. The device of claim 1, wherein the instructions are executable to establish communications only with the AVDD and not with the server, the instructions being executable such that the server information pertaining to rendering the RUI is received through the AVDD by the companion device.

7. The device of claim 1, wherein the instructions are executable to execute, on a base established by a hypertext markup language that supports graphics rendering commands using vector graphics, in addition to bitmap graphics commands, a protocol application which transcodes the information pertaining to rendering the RUI to graphics on the base.

8. The device of claim 1, wherein the information pertaining to rendering the RUI received by the companion device includes extensible markup language (XML) commands and/or JavaScript Object Notation (JSON) formatted commands.

9. Server comprising:

at least one computer memory comprising instructions executable by at least one processor to;

send to an audio video display device (AVDD) at least one graphics rendering command to render, on a display of the AVDD, a remote user interface (RUI) containing at least one RUI element, the graphics rendering command to render the RUI being sent in a low level graphics card protocol via a transmission control protocol/Internet Protocol (TCP/IP) socket, such that the graphics rendering command to render the RUI defines how to render RUI elements of the RUI but does not define underlying functionality of the RUI elements, such that selection of an RUI element causes sending a signal back to the server indicating a screen location at which the selection occurred, which the server then correlates to the appropriate function underlying the selected RUI element;

in synchronization with sending the graphics rendering command to the AVDD, send to a companion device at least one corresponding graphics rendering command that causes the companion device to render the RUI or a substantially similar version thereof on a display of the companion device, the substantially similar version presented on the companion device being a version of the RUI presented on the AVDD modified for presentation capabilities of the companion device;

receive from the companion device at least one user selection signal pertaining to the RUI;

correlate the user selection signal to a control command;

at least some of the time, responsive to the user selection signal from the companion device, automatically send a graphics rendering command to the AVDD to cause the AVDD to modify the RUI presented thereon such that the RUI on the AVDD is synchronized with the RUI on the companion device.

10. The server of claim 9, wherein the instructions are executable to send at least one modified RUI rendering command to the AVDD and a corresponding at least one modified RUI rendering command to the companion device such that the RUI presented on the companion device mirrors the RUI presented on the AVDD.

11. The server of claim 9, wherein the instructions are executable to send at least one graphics rendering command to the companion device in a format suitable for rendering the RUI in an HTML5 environment.

12. The server of claim 9, wherein the instructions are executable to send at least one graphics rendering command to the AVDD using a low level graphics card protocol via a transmission control protocol/Internet Protocol (TCP/IP) socket.

13. The server of claim 9, wherein the instructions are executable to send at least one graphics rendering command to the companion device in extensible markup language (XML) format or JavaScript Object Notation (JSON) format.

14. Method comprising:

receiving, at an audio video display device (AVDD) and a companion device paired with the AVDD, graphics rendering commands to render on each device a respective remote user interface (RUI), the graphics rendering command to render the RUI being sent in a low level graphics card protocol via a transmission control protocol/Internet Protocol (TCP/IP) socket, such that the graphics rendering command to render the RUI defines how to render RUI elements of the RUI but does not define underlying functionality of the RUI elements, such that selection of an RUI element causes sending a signal back to the server indicating a screen location at which the selection occurred, which the server then correlates to the appropriate function underlying the selected RUI element; and

receiving, at both the AVDD and companion device, respective RUI modification commands to change the respective RUIs in synchronization with each other, both RUIs for generating control commands to control presentation on the AVDD.

15. The method of claim 14, comprising generating the respective RUI modification commands by a server in response to user input on the companion device.

16. The method of claim 14, comprising formatting the respective RUI modification commands differently from each other.

17. The method of claim 14, comprising sending the RUI modification command to the companion device in a format suitable for rendering the RUI in an HTML5 environment.

18. The method of claim 14, comprising sending the RUI modification command to the companion device in extensible markup language (XML) format or JavaScript Object Notation (JSON) format.

19. The method of claim 14, comprising sending the RUI modification command to the AVDD in a low level graphics card protocol via a transmission control protocol/Internet Protocol (TCP/IP) socket.

20. The method of claim 14, comprising relieving the companion device of generating the RUI and sending the RUI to the AVDD since a server bears the burden of ensuring RUI graphics sent to the AVDD and RUI graphics sent to the companion device are mirrored synchronously with each other.