Methods and systems for remotely controlling electronics

• US 8,766,783 B1
• Filed: 11/05/2010
• Issued: 07/01/2014
• Est. Priority Date: 11/05/2010
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Hughes, Ronald
• Primary Examiner(s): Mehmood, Jennifer
• Assistant Examiner(s): Jiang, Yong Hang
• Application Number:
  US12/940,857
• Time in Patent Office: 1,334 Days
• Field of Search: 340/12.1, 340/12.15, 340/3.1, 340/3.43, 340/3.44, 340/3.7, 340/3.71, 340/12.22, 340/12.3
• US Class Current: 340/12.3
• CPC Class Codes: G05B 19/0428 Safety, monitoring G05B19/0423 takes precedence
• CPC Class Codes: G05B 2219/23051 Remote control, enter program remote, detachable programmer
• CPC Class Codes: G05B 2219/24024 Safety, surveillance
• CPC Class Codes: G05B 2219/24097 Camera monitors controlled machine
• CPC Class Codes: H04N 17/004 for digital television systems
• CPC Class Codes: H04N 21/25833 involving client hardware characteristics, e.g. manufacturer, processing or storage capabilities allocation of resources considering hardware capabilities in multiprogramming arrangements G06F9/5044; allocation of resources considering software capabilities in multiprogramming arrangements G06F9/5055
• CPC Class Codes: H04N 21/41265 having a remote control device for bidirectional communication between the remote control device and client device unidirectional remote control device H04N21/42204
• CPC Class Codes: H04N 21/4131 home appliance, e.g. lighting, air conditioning system, metering devices home automation data switching networks exchanging configuration information on appliance services H04L12/2807
• CPC Class Codes: H04N 21/42204 User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor constructive details of casings for the remote control device H01H9/0235; user interfaces for controlling a tuning device of a television receiver through a remote control H03J9/00; remote control of peripheral devices connected to a television receiver through the remote control device of the television receiver H04B1/205
• CPC Class Codes: H04N 21/4222 Remote control device emulator integrated into a non-television apparatus, e.g. a PDA, media center or smart toy remote control device for a television receiver integrated into a mobile phone H04M1/72415
• CPC Class Codes: H04N 21/42221 Transmission circuitry, e.g. infrared [IR] or radio frequency [RF]
• CPC Class Codes: H04N 21/42222 Additional components integrated in the remote control device, e.g. timer, speaker, sensors for detecting position, direction or movement of the remote control, microphone or battery charging device
• CPC Class Codes: H04N 21/43615 Interfacing a Home Network, e.g. for connecting the client to a plurality of peripherals home Audio Video Interoperability [HAVI] data switching networks H04L12/2805
• CPC Class Codes: H04N 21/6543 for forcing some client operations, e.g. recording remote booting in general G06F9/4416
• CPC Class Codes: H04N 21/6582 Data stored in the client, e.g. viewing habits, hardware capabilities, credit card number arrangements where receivers interact with the broadcast H04H20/38

Abstract

A computer-implemented method of controlling electronics is performed at an apparatus that includes one or more processors, a camera, and a transmitter. In the method, the camera acquires a first image of an electronic device situated within view of the camera and configured for remote control. User input is received specifying a command for the electronic device. Based on the first image, a determination is made that the electronic device is not in a state corresponding to the command. In response to the determination, an instruction corresponding to the command is transmitted to the electronic device using the transmitter.

Claims

1. A computer-implemented method of controlling electronics, comprising:

at an apparatus comprising one or more processors, a camera, and a transmitter;

using the camera to acquire a first image of an electronic device situated within view of the camera, the electronic device being configured for remote control;

receiving user input specifying a command for the electronic device;

based on the first image, determining that the electronic device is not in a state corresponding to the command;

in response to determining that the electronic device is not in a state corresponding to the command, transmitting an instruction corresponding to the command to the electronic device using the transmitter;

after transmitting the instruction, acquiring a second image of the electronic device using the camera;

based on the second image, determining that the electronic device did not perform the instruction; and

in response to determining that the electronic device did not perform the instruction, retransmitting the instruction to the electronic device using the transmitter.

2. The computer-implemented method of claim 1, wherein the apparatus is a laptop computer or tablet computer.

3. The computer-implemented method of claim 1, wherein the apparatus is a cellular telephone.

4. The computer-implemented method of claim 1, wherein:

the electronic device comprises a light configured to indicate a state of the electronic device; and

determining that the electronic device is not in a state corresponding to the command comprises detecting in the first image whether the light indicates that the electronic device is in the state corresponding to the command.

5. The computer-implemented method of claim 4, wherein the electronic device is a media player.

6. The computer-implemented method of claim 4, wherein the electronic device is a set-top box.

7. The computer-implemented method of claim 4, wherein the electronic device is a switch.

8. The computer-implemented method of claim 7, wherein the switch is an HDMI switch.

9. The computer-implemented method of claim 1, wherein:

the electronic device is a switch comprising a plurality of input ports, an output port, and a light associated with a respective input port of the plurality, wherein the light can emit a plurality of distinct colors, and each color of emitted light corresponds to a distinct state of the respective input port;

the command is a command to connect the respective input port to the output port; and

determining that the electronic device is not in a state corresponding to the command comprises detecting the color of the emitted light in the first image and ascertaining from the color of the emitted light that the respective input port is not connected to the output port.

10. The computer-implemented method of claim 1, wherein:

the electronic device is a television comprising a screen;

the command is a command to turn off the television; and

determining that the electronic device is not in a state corresponding to the command comprises detecting in the first image that the screen is emitting light.

11. The computer-implemented method of claim 1, wherein:

the electronic device is a television comprising a screen;

the command is a command to turn on the television; and

determining that the electronic device is not in a state corresponding to the command comprises detecting in the first image that the screen is not emitting light.

12. The computer-implemented method of claim 1, wherein:

the electronic device comprises a light configured to indicate a state of the electronic device, wherein presence of emitted light indicates a first state and absence of emitted light indicates a second state distinct from the first state; and

determining that the electronic device did not perform the instruction comprises detecting the presence or the absence of emitted light, and ascertaining that the electronic device did not perform the instruction based on the presence or the absence of emitted light.

13. The computer-implemented method of claim 1, wherein determining that the electronic device is not in a state corresponding to the command comprises:

identifying a probable identity of the electronic device by comparing the first image to a plurality of stored images of known electronic devices;

sending one or more test instructions to the electronic device using the transmitter, wherein the one or more test instructions are selected based on the probable identity of the electronic device;

acquiring a confirmation image of the electronic device using the camera, subsequent to transmitting the one or more test instructions to the electronic device; and

determining from the confirmation image that the state of the electronic device is consistent with execution of the test instructions.

14. The computer-implemented method of claim 1, wherein determining that the electronic device is not in a state corresponding to the command comprises:

identifying a set of two or more known electronic devices that match the first image within a predefined tolerance;

presenting the identified set of known electronic devices to the user;

receiving user selection from the identified set of known electronic devices; and

comparing status lights in the first image to stored information about status lights for the user selected electronic device.

15. The commuter-implemented method of claim 14, wherein presenting the identified set of known electronic devices includes presenting images of known devices in the identified set.

16. The computer-implemented method of claim 14, wherein presenting the identified set of known electronic devices includes presenting manufacturer and model names of known devices in the identified set.

17. The computer-implemented method of claim 1, wherein determining that the electronic device is not in a state corresponding to the command comprises:

extracting a plurality of visible device characteristics from the first image;

identifying the electronic device by comparing the plurality of visible device characteristics to stored device specifications for known electronic devices; and

determining a state of the electronic device by comparing the first image to stored state information for the identified electronic device.

18. A system for controlling electronics, comprising:

a camera;

a transmitter;

one or more processors; and

memory storing one or more programs configured for execution by the one or more processors, the one or more programs comprising instructions for;

using the camera to acquire a first image of an electronic device situated within view of the camera, the electronic device being configured for remote control;

receiving user input specifying a command for the electronic device;

based on the first image, determining that the electronic device is not in a state corresponding to the command;

in response to determining that the electronic device is not in a state corresponding to the command, transmitting an instruction corresponding to the command to the electronic device using the transmitter;

after transmitting the instruction, acquiring a second image of the electronic device using the camera;

based on the second image, determining that the electronic device did not perform the instruction; and

in response to determining that the electronic device did not perform the instruction, retransmitting the instruction to the electronic device using the transmitter.

19. The system of claim 18, wherein:

the electronic device is a switch comprising a plurality of input ports, an output port, and a light associated with a respective input port of the plurality, wherein the light can emit a plurality of distinct colors, and each color of emitted light corresponds to a distinct state of the respective input port;

the command is a command to connect the respective input port to the output port; and

the instructions for determining that the electronic device is not in a state corresponding to the command comprise instructions for detecting the color of the emitted light in the first image and ascertaining from the color of the emitted light that the respective input port is not connected to the output port.

20. The system of claim 18, wherein the instructions for determining that the electronic device is not in a state corresponding to the command comprise instructions for:

identifying a probable identity of the electronic device by comparing the first image to a plurality of stored images of known electronic devices;

sending one or more test instructions to the electronic device using the transmitter, wherein the one or more test instructions are selected based on the probable identity of the electronic device;

acquiring a confirmation image of the electronic device using the camera, subsequent to transmitting the one or more test instructions to the electronic device; and

determining from the confirmation image that the state of the electronic device is consistent with execution of the test instructions.

21. The system of claim 18, wherein the instructions for determining that the electronic device is not in a state corresponding to the command comprise instructions for:

identifying a set of two or more known electronic devices that match the first image within a predefined tolerance;

presenting the identified set of known electronic devices to the user;

receiving user selection from the identified set of known electronic devices; and

comparing status lights in the first image to stored information about status lights for the user selected electronic device.

22. The system of claim 18, wherein the instructions for determining that the electronic device is not in a state corresponding to the command comprise instructions for:

extracting a plurality of visible device characteristics from the first image;

identifying the electronic device by comparing the plurality of visible device characteristics to stored device specifications for known electronic devices; and

determining a state of the electronic device by comparing the first image to stored state information for the identified electronic device.

23. A non-transitory computer readable storage medium storing one or more programs configured for execution by a system for controlling electronics, the one or more programs comprising instructions for:

at a system comprising one or more processors, a camera, and a transmitter;

using the camera to acquire a first image of an electronic device situated within view of the camera, the electronic device being configured for remote control;

receiving user input specifying a command for the electronic device;

based on the first image, determining that the electronic device is not in a state corresponding to the command;

in response to determining that the electronic device is not in a state corresponding to the command, transmitting an instruction corresponding to the command to the electronic device using the transmitter;

after transmitting the instruction, acquiring a second image of the electronic device using the camera;

based on the second image, determining that the electronic device did not perform the instruction; and

in response to determining that the electronic device did not perform the instruction, retransmitting the instruction to the electronic device using the transmitter.

24. The computer readable storage medium of claim 23, wherein:

the electronic device is a switch comprising a plurality of input ports, an output port, and a light associated with a respective input port of the plurality, wherein the light can emit a plurality of distinct colors, and each color of emitted light corresponds to a distinct state of the respective input port;

the command is a command to connect the respective input port to the output port; and

the instructions for determining that the electronic device is not in a state corresponding to the command comprise instructions for detecting the color of the emitted light in the first image and ascertaining from the color of the emitted light that the respective input port is not connected to the output port.