Control of remote applications using companion device

• US 9,690,465 B2
• Filed: 12/21/2012
• Issued: 06/27/2017
• Est. Priority Date: 06/01/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Tudor, Alexander D., Haddad, Marc V., Medak, Matthew Alan, Tuck, Jason R.
• Primary Examiner(s): Tung, David
• Application Number:
  US13/723,365
• Publication Number:
  US 20130321268A1
• Time in Patent Office: 1,649 Days
• Field of Search: 345173-178, 715863
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/235 using a wireless connection, e.g. infrared or piconet
• CPC Class Codes: A63F 13/493 Resuming a game, e.g. after pausing, malfunction or power failure
• CPC Class Codes: A63F 13/50 Controlling the output signals based on the game progress
• CPC Class Codes: A63F 13/812 Ball games, e.g. soccer or baseball
• CPC Class Codes: G06F 16/489 using time information
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/033 Pointing devices displaced or positioned by the user , e.g. mice, trackballs, pens or joysticks; Accessories therefor digitisers characterised by the transducing means G06F3/041
• CPC Class Codes: G06F 3/03547 Touch pads, in which fingers can move on a surface
• CPC Class Codes: G06F 3/041 Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• CPC Class Codes: G06F 3/0412 Digitisers structurally integrated in a display
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: H04L 65/4015 where at least one of the additional parallel sessions is real time or time sensitive, e.g. white board sharing, collaboration or spawning of a subconference
• CPC Class Codes: H04L 65/403 : Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04N 21/4122 : additional display device, e.g. video projector digital output for controlling a plurality of local displays G06F3/1423
• CPC Class Codes: H04N 21/4722 : for requesting additional data associated with the content
• CPC Class Codes: H04N 21/4781 : Games
• CPC Class Codes: H04N 21/8455 : involving pointers to the content, e.g. pointers to the I-frames of the video stream

Abstract

Embodiments of the invention provide underlying communication functionality to enable companion experiences. A companion experience allows the user to interact with content playing on a primary device through a companion device. An application on the companion device interacts with an application running on a base device (e.g., a game console, PC, or TV) to provide additional interface options on the companion that are related to a title or application playing on the base device.

Claims

1. A method of using a companion device to control an application running on a primary device, the method comprising:

outputting for display to a screen on a display device a graphical user interface generated by an application running on the primary device;

receiving from the companion device, which has a touch screen display, a normalized touch data that describes a touch event comprising an object contacting the touch screen at a plurality of touch points on the touch screen display, the normalized touch data comprising a normalized x coordinate and a normalized y coordinate for each of the touch points, wherein the normalized x coordinate is calculated using the formula x_n=x/w where x_n is the normalized x coordinate, x is an x coordinate for a touch point, and w is the width of the touch screen display in pixels;

wherein the normalized y coordinate is calculated using the formula y_n=y/h where y_n is the normalized y coordinate, y is an y coordinate for the touch point, and h is the height of the touch screen display in pixels, wherein the touch event is movement across the touch screen;

calculating a distance an object on the graphical user interface is to be moved using the normalized touch data as input and applying a dampening logic that adjusts a delta calculated between touch points using both a dampening coefficient and an acceleration component; and

outputting for display to the display device an updated graphical user interface generated by the application showing the object moved the distance.

2. The method of claim 1, wherein the method further comprises:

rescaling the normalized touch data by multiplying the normalized x coordinate by a width of the updated graphical user interface and the normalized y coordinate by the height of the updated graphical user interface to generate scaled touch data; and

interpreting the scaled touch data to determine the distance derived from the normalized touch data.

3. The method of claim 1, wherein the method further comprises:

rescaling the normalized touch data by multiplying the normalized x coordinate by a width of the updated graphical user interface and the normalized y coordinate by the height of the updated graphical user interface to generate scaled touch data;

generating an optimized touch data by multiplying the scaled touch data by a factor that is a ratio of the companion device'"'"'s screen size to a template screen size, and

interpreting the optimized touch data to determine the distance derived from the normalized touch data.

4. The method of claim 1, wherein the object is a cursor.

5. The method of claim 1, wherein a scroll function is activated when the touch event indicates that two fingers are contacting the touch screen display and remain a threshold distance apart as they move.

6. The method of claim 1, wherein the method further comprises receiving movement data from the companion device.

7. One or more computer-storage media comprising computer-executable instructions embodied thereon that when executed by a computing device performs a method for using a companion device to manipulate a graphical user interface generated by an application operating on a primary device, the method comprising:

outputting for display on a screen of a display device a graphical user interface (“

GUI”

) generated by the application running on the primary device, the GUI comprising a cursor having a present location;

receiving control information from the companion device, which has a touch screen display, the control information comprising a normalized touch data that describes a touch event comprising an object contacting the touch screen at a plurality of touch points on the touch screen display, the normalized touch data comprising a normalized x coordinate and a normalized y coordinate for each of the touch points, wherein the normalized x coordinate is calculated using the formula x_n=x/w where x_n is the normalized x coordinate, x is an x coordinate for a touch point, and w is the width of the touch screen display in pixels;

wherein the normalized y coordinate is calculated using the formula y_n=y/h where y_n is the normalized y coordinate, y is an y coordinate for the touch point, and h is the height of the touch screen display in pixels, wherein the touch event is movement across the touch screen;

calculating a distance an object on the graphical user interface is to be moved using the normalized touch data as input and applying a dampening logic that adjusts a delta calculated between touch points using both a dampening coefficient and an acceleration component;

and

outputting for display an updated GUI showing the object moved the distance.

8. The media of claim 7, wherein the GUI height is less than a height of the display device.

9. The media of claim 7, wherein the movement data indicates the companion device is tilted up from the home position and the direction of scrolling is up.

10. The media of claim 7, wherein the movement data indicates the companion device is twisted rightward from the home position and the direction of scrolling is to the right.

11. The media of claim 7, wherein the home position is a position of the companion device when a scroll function is activated in response to the movement data.

12. The media of claim 7, wherein the normalized touch data describes a finger moving down a side of the touch screen display.

13. A computing system to implement a method for using a companion device to manipulate a graphical user interface generated by an application operating on a primary device, the computing system comprising:

a processor; and

computer storage media having computer-executable instructions stored thereon which, when executed by the processor, configure the computing system to;

activate, at the companion device, a navigation function that enables touch input on a touch screen display that is integrated in the companion device to manipulate the application running on the primary device that is separate from the companion device and has a separate;

receive, from the touch screen display, touch data describing points contacted during a touch event comprising an object contacting the touch screen at a plurality of touch points;

convert the touch data to a normalized touch data;

the normalized touch data comprising a normalized x coordinate and a normalized y coordinate for each of the touch points, wherein the normalized x coordinate is calculated using the formula x_n=x/w where x_n is the normalized x coordinate, x is an x coordinate for a touch point, and w is the width of the touch screen display in pixels;

wherein the normalized y coordinate is calculated using the formula y_n=y/h where y_n is the normalized y coordinate, y is an y coordinate for the touch point, and h is the height of the touch screen display in pixels;

dynamically adjust a frame rate at which the normalized touch data is communicated to the primary device to match a frame rate of the primary device based on network latency characteristics; and

communicate the normalized touch data to the primary device that is associated with a display device having a primary screen.

14. The computing system of claim 13, wherein the computing system is further configured to, subsequent to the touch event, detect an orientation change of the companion device and communicating, prior to a subsequent touch event, an updated screen area for the touch screen display to the primary device.

15. The computing system of claim 13, wherein the computing system is further configured to receive a message from the application running on the primary device that a web page is loading and, in response, enable a stop button displayed on the touch screen display, the stop button associated with a function that instructs the application to stop loading the web page.

16. The computing system of claim 13, wherein the computing system is further configured to:

receive a message from the application running on the primary device that the web page has just finished loading and, in response, enable a refresh button displayed on the touch screen display, the refresh button associated with a function that instructs the application to reload the web page; and

disable the stop button.