Method and system for user interface for interactive devices using a mobile device

• US 9,462,210 B2
• Filed: 11/05/2012
• Issued: 10/04/2016
• Est. Priority Date: 11/04/2011
• Status: Active Grant

• Current Assignee: Remote Telepointer, LLC
• Original Assignee: Remote Telepointer, LLC
• Inventors: Dagit, Charles E. III
• Primary Examiner(s): Arevalo, Joseph
• Application Number:
  US13/669,190
• Publication Number:
  US 20130113993A1
• Time in Patent Office: 1,429 Days
• Field of Search: 455/41.1, 455/410, 455/411, 455/557, 455/39, 455/556.1, 455/420, 455/456.1, 455/456.3, 455/414.2, 455/67.11, 455/414.1, 455/427
• US Class Current: 1/1
• CPC Class Codes: G01S 19/49 whereby the further system is an inertial position system, e.g. loosely-coupled
• CPC Class Codes: G01S 5/0027 of actual mobile position, i.e. position determined on mobile
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: H04M 1/72412 using two-way short-range wireless interfaces
• CPC Class Codes: H04M 1/72415 for remote control of appliances
• CPC Class Codes: H04M 2250/10 including a GPS signal receiver
• CPC Class Codes: H04N 21/41265 having a remote control device for bidirectional communication between the remote control device and client device unidirectional remote control device H04N21/42204
• CPC Class Codes: H04N 21/4131 home appliance, e.g. lighting, air conditioning system, metering devices home automation data switching networks exchanging configuration information on appliance services H04L12/2807
• CPC Class Codes: H04N 21/42204 User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor constructive details of casings for the remote control device H01H9/0235; user interfaces for controlling a tuning device of a television receiver through a remote control H03J9/00; remote control of peripheral devices connected to a television receiver through the remote control device of the television receiver H04B1/205
• CPC Class Codes: H04N 21/42214 using alphanumerical characters
• CPC Class Codes: H04N 21/42222 Additional components integrated in the remote control device, e.g. timer, speaker, sensors for detecting position, direction or movement of the remote control, microphone or battery charging device
• CPC Class Codes: H04N 21/42224 Touch pad or touch panel provided on the remote control
• CPC Class Codes: H04N 21/47217 for controlling playback functions for recorded or on-demand content, e.g. using progress bars, mode or play-point indicators or bookmarks specific graphical features in visual interfaces H04N21/4312
• CPC Class Codes: H04N 21/485 End-user interface for client configuration
• CPC Class Codes: H04W 4/33 for indoor environments, e.g. buildings
• CPC Class Codes: H05K 999/99 dummy group

Abstract

A software application and system that enables point-and-click interaction with a TV screen. The application determines geocode positioning information for a handheld device, and uses that data to create a virtual pointer for a television display or interactive device. Some embodiments utilize motion sensing and touchscreen input for gesture recognition interacting with video content or interactive device. Motion sensing can be coupled with positioning or localization techniques the user to calibrate the location of the interactive devices and the user location to establish and maintain virtual pointer connection relationships. The system may utilize wireless network infrastructure and cloud-based calculation and storage of position and orientation values to enable the handheld device in the TV viewing area to replace or surpass the functionality of the traditional TV remote control, and also interface directly with visual feedback on the TV screen.

Claims

1. A system for remotely interacting with a video display, comprising:

non-transitory computer-readable instructions for configuring a processor of a handheld wireless device, the handheld wireless device being separate from the video display, the processor of the handheld wireless device determining position and orientation information about the handheld device based on sensors onboard the handheld wireless device and sending the position and orientation information across at least one wireless network;

and at least one remote server configured to receive the position and orientation information and to determine the orientation of the handheld wireless device relative to the video display based on the position and orientation information received from the handheld device and a model of the environment around the handheld wireless device,

wherein the position information is derived by the handheld wireless device from a distance measurement between the handheld wireless device and a plurality of beacons, including at least one fixed-position terrestrial wireless networking device, and wherein the server is configured to remotely control a user interface on the video display in response to the position and orientation information;

non-transitory computer-readable instructions for displaying, on a display of the handheld device, at least one menu that duplicates information displayed on the video display.

2. The system of claim 1, further comprising a set-top box, in communication with the at least one server, that displays the user interface on the video display and updates the user interface in response to changes in the position and orientation information.

3. The system of claim 2, wherein the at least one server comprises software operating on the set-top box.

4. The system of claim 1, further comprising non-transitory computer-readable instructions that determine motion information relating to the position and orientation information to recognize gestures made by a user.

5. The system of claim 1, further comprising non-transitory computer-readable instructions to allow a processor to recognize at least a subset of the following gestures:

Select, Move Up, Move Down, Move Left, Move Right, Back, Forward, Scroll Up, Scroll Down, Scroll Left, Scroll Right, Hover, Stop, Play, Pause, Fast Forward, Rewind, Record, Save, Last, Channel UP, Channel Down, Volume Up, Volume Down, Mute, Hide, Exit, Off, and Help.

6. The system of claim 1, wherein position information is further derived from GPS signals received by the handheld device and the position information corresponds to an indoor location.

7. The system of claim 1, wherein the user interface presents a pointer that moves in response to movement of the handheld device.

8. The system of claim 1, wherein the at least one fixed-position terrestrial wireless networking device comprises at least one of a Wi-Fi access point and a cellular base station.

9. The system of claim 1, wherein the distance measurement is determined by observing a form of radio frequency time of flight or communication time of at least one signal sent to and received from the at least one wireless fixed-position terrestrial networking device.

10. The system of claim 1, wherein the server is further configured to facilitate interaction with a plurality of video displays, and to determine which of the video displays the user interacts with based on the position and orientation information.

11. The system of claim 1, wherein the handheld wireless devices comprises one of a tablet computer and a cell-phone.

12. A system for facilitating remote interaction with an interactive device in a physical space, comprising:

a first set of non-transitory computer-readable instructions for configuring a handheld wireless device to determine position and orientation information about the handheld wireless device based on sensors onboard the handheld wireless device and transmit the position and orientation information and at least one user input across a wireless network, wherein the position information is derived by the handheld wireless device from a distance measurement between the handheld wireless device and at least one fixed-position terrestrial wireless networking device;

a second set of non-transitory computer-readable instructions for configuring at least one processor of a computer, remote from the handheld wireless device, to receive the position and motion information and to approximate the orientation of the handheld wireless device relative to a physical space, including in relation to at least one interactive device based on the position and orientation information received from the handheld device and a model of the environment around the handheld wireless device; and

a third set of non-transitory computer-readable instructions for configuring the at least one processor of the computer to facilitate control of the at least one interactive device in response to the position and orientation information and the at least one user input;

non-transitory computer-readable instructions for displaying, on a display of the handheld device, at least one menu that duplicates information displayed on the video display.

13. The system of claim 12, wherein the interactive device comprises a video display.

14. The system of claim 12, wherein the third set of non-transitory computer-readable instructions are configured to be executed by an automation server and the at least one interactive device comprises at least one of a light switch, an alarm device, and an HVAC device controlled by the automation server.

15. The system of claim 12, wherein the second set of non-transitory computer-readable instructions are configured to be executed by one or more servers in a cloud computing space configured to communicate with the handheld wireless device across the Internet.

16. The system of claim 12, wherein the first set of non-transitory computer-readable instructions comprises instructions for displaying, on a display of the handheld wireless device, selectable actions that can be taken to interact with the at least one interactive device to solicit the at least one user input.

17. The system of claim 16, wherein the selectable actions displayed on the handheld wireless device change depending on which of a plurality of types of interactive devices that the handheld wireless device points to.

18. The system of claim 12, wherein the first set of non-transitory computer-readable instructions comprise instructions to recognize gestures made by the user to determine the at least one user input.

19. The system of claim 12, wherein the at least one other wireless fixed-position terrestrial networking device comprises at least one Wi-Fi access point.

20. The system of claim 19, wherein the distance measurement is determined by observing a form of radio frequency time of flight or communication time from at least one signal sent to and received from the at least one Wi-Fi access point.

21. The system of claim 12, wherein position information is further derived from GPS signals received by the handheld device and the position information includes horizontal and vertical position information.

22. A method for facilitating interaction between a mobile device and an interactive device comprising steps of:

a. observing by a processor on a handheld wireless device, utilizing an antenna and one or more sensors on the handheld wireless device, a plurality of distances relative to a plurality of fixed-position terrestrial network devices in a physical space and an orientation of the handheld wireless device to form the location and orientation information that reflects the position and orientation of the handheld wireless device within the physical space, and transmitting the location and orientation information from the handheld wireless device across at least one network;

b. receiving at a remote server, across the at least one network, the location and orientation information from the handheld wireless device;

c. calculating, by a processor at the remote server, a position and orientation of the handheld wireless device relative to at least one interactive device from the location and orientation information and a model of the physical space to determine that a user is pointing the handheld wireless device relative to the interactive device;

d. determining, by at least one of the processors, at least one command input from a user of the handheld wireless device; and

e. transmitting the command to the at least one interactive device across at least one of the at least one networks in response to the step of determining and the position and orientation information;

f. displaying, on a display of the handheld device, at least one menu that duplicates information displayed on the video display.

23. The method of claim 22, wherein the interactive device comprises a video display.

24. The method of claim 22, wherein the at least one interactive device comprises at least one of a light switch, an alarm device, and an HVAC device controlled by an automation server.

25. The method of claim 22, wherein the step of observing further comprises observing a GPS position of the handheld wireless device.

26. The method of claim 22, further comprising a step of calibrating the position of the at least one interactive device by providing computer-readable instructions to allow the handheld wireless device to record a position of at least one portion of the at least one interactive device when the handheld wireless device is placed approximately at the position of the interactive device in the physical space.

27. The method of claim 22, wherein the step of determining at least one command input comprises detecting a motion gesture made by a user of the handheld wireless device.

28. The method of claim 22, wherein the step of determining at least one command input comprises detecting a touch screen input made by a user of the handheld wireless device.

29. The method of claim 22, further comprising a step of displaying a cursor on a video display that indicates approximately where a user is pointing the handheld wireless device.