Methods and systems for remotely controlling electronic devices

• US 9,473,727 B2
• Filed: 06/26/2014
• Issued: 10/18/2016
• Est. Priority Date: 11/05/2010
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Hughes, Ronald
• Primary Examiner(s): Mehmood, Jennifer
• Assistant Examiner(s): Jiang, Yong Hang
• Application Number:
  US14/316,715
• Publication Number:
  US 20140307106A1
• Time in Patent Office: 845 Days
• Field of Search: 340/12.1, 340/12.15, 340/12.22, 340/12.3, 341/176
• US Class Current: 1/1
• CPC Class Codes: G05B 19/0428 Safety, monitoring G05B19/0423 takes precedence
• CPC Class Codes: G05B 2219/23051 Remote control, enter program remote, detachable programmer
• CPC Class Codes: G05B 2219/24024 Safety, surveillance
• CPC Class Codes: G05B 2219/24097 Camera monitors controlled machine
• CPC Class Codes: H04N 17/004 for digital television systems
• CPC Class Codes: H04N 21/25833 involving client hardware characteristics, e.g. manufacturer, processing or storage capabilities allocation of resources considering hardware capabilities in multiprogramming arrangements G06F9/5044; allocation of resources considering software capabilities in multiprogramming arrangements G06F9/5055
• CPC Class Codes: H04N 21/41265 having a remote control device for bidirectional communication between the remote control device and client device unidirectional remote control device H04N21/42204
• CPC Class Codes: H04N 21/4131 home appliance, e.g. lighting, air conditioning system, metering devices home automation data switching networks exchanging configuration information on appliance services H04L12/2807
• CPC Class Codes: H04N 21/42204 User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor constructive details of casings for the remote control device H01H9/0235; user interfaces for controlling a tuning device of a television receiver through a remote control H03J9/00; remote control of peripheral devices connected to a television receiver through the remote control device of the television receiver H04B1/205
• CPC Class Codes: H04N 21/4222 Remote control device emulator integrated into a non-television apparatus, e.g. a PDA, media center or smart toy remote control device for a television receiver integrated into a mobile phone H04M1/72415
• CPC Class Codes: H04N 21/42221 Transmission circuitry, e.g. infrared [IR] or radio frequency [RF]
• CPC Class Codes: H04N 21/42222 Additional components integrated in the remote control device, e.g. timer, speaker, sensors for detecting position, direction or movement of the remote control, microphone or battery charging device
• CPC Class Codes: H04N 21/43615 Interfacing a Home Network, e.g. for connecting the client to a plurality of peripherals home Audio Video Interoperability [HAVI] data switching networks H04L12/2805
• CPC Class Codes: H04N 21/6543 for forcing some client operations, e.g. recording remote booting in general G06F9/4416
• CPC Class Codes: H04N 21/6582 Data stored in the client, e.g. viewing habits, hardware capabilities, credit card number arrangements where receivers interact with the broadcast H04H20/38

Abstract

An apparatus for controlling electronic devices has a camera and a transmitter. The camera acquires a first image of an electronic device. The device is configured for remote control. The apparatus identifies a probable identity of the device by comparing the first image to a plurality of stored images of known electronic devices. Test instructions are selected based on the probable identity of the device and sent to the device. A confirmation image of the device is acquired after the test instructions are sent. The apparatus determines from the confirmation image that the state of the device is consistent with execution of the test instructions. The apparatus receives user input specifying a command for the device. Based on the first image, the device is not in a state corresponding to the command, so the apparatus transmits an instruction corresponding to the command to the device.

Claims

1. A method of controlling an electronic device, comprising:

at an apparatus comprising one or more processors, a camera, and a transmitter;

using the camera to acquire a first image of an electronic device, wherein the electronic device is configured for remote control and the electronic device comprises a plurality of lights that indicate its state and are at respective locations;

identifying a probable identity of the electronic device based on analysis of the first image, including identifying a light state specification comprising a description of a set of states of the electronic device indicated by respective colors and the locations of the lights;

sending one or more test instructions to the electronic device using the transmitter, wherein the one or more test instructions are selected based on the probable identity of the electronic device;

acquiring a second image of the electronic device using the camera, subsequent to transmitting the one or more test instructions to the electronic device;

comparing the respective colors and the locations of the lights in the second image to the light state specification;

determining from the respective colors and the locations of the lights in the second image that the state of the electronic device is consistent with execution of the test instructions by the electronic device; and

in response to the determining,

storing locally a confirmed identity of the electronic device, and an associated set of signal instructions for controlling the electronic device.

2. The method of claim 1, wherein the analysis of the first image comprises comparing the first image to a plurality of stored images of known electronic devices.

3. The method of claim 2, wherein comparing the first image to a plurality of stored images of known electronic devices comprises:

extracting a plurality of visible device characteristics from the first image; and

comparing the plurality of visible device characteristics to stored device specifications for the known electronic devices.

4. The method of claim 1, further comprising:

receiving user input specifying a command for the electronic device; and

based on the confirmed identify of the electronic device, transmitting a signal instruction corresponding to the command to the electronic device using the transmitter.

5. The method of claim 4, wherein:

the electronic device is a switch comprising a plurality of input ports, an output port, and the plurality of lights, the plurality of lights including a first light associated with a respective input port of the plurality, wherein the first light can emit a plurality of distinct colors, and each color of emitted light corresponds to a distinct state of the respective input port;

the command is a command to connect the respective input port to the output port; and

transmitting the signal instruction comprises;

using the camera to acquire a third image of the electronic device; and

determining from the color of the emitted light that the respective input port is not connected to the output port.

6. The method of claim 4, wherein:

the electronic device is a television comprising a screen;

the command is a command to turn off the television; and

transmitting the signal instruction comprises;

using the camera to acquire a third image of the electronic device; and

determining from the third image that the screen is emitting light.

7. The method of claim 4, wherein:

the electronic device is a television comprising a screen;

the command is a command to turn on the television; and

transmitting the signal instruction comprises;

using the camera to acquire a third image of the electronic device; and

determining from the third image that the screen is not emitting light.

8. A system for controlling an electronic device, comprising:

a camera;

a transmitter;

one or more processors; and

memory storing one or more programs configured for execution by the one or more processors, the one or more programs comprising instructions for;

using the camera to acquire a first image of an electronic device, wherein the electronic device is configured for remote control and the electronic device comprises a plurality of lights that indicate its state and are at respective locations;

identifying a probable identity of the electronic device based on analysis of the first image, including identifying a light state specification comprising a description of a set of states of the electronic device indicated by respective colors and the locations of the lights;

sending one or more test instructions to the electronic device using the transmitter, wherein the one or more test instructions are selected based on the probable identity of the electronic device;

acquiring a second image of the electronic device using the camera, subsequent to transmitting the one or more test instructions to the electronic device;

comparing the respective colors and the locations of the lights in the second image to the light state specification;

determining from the respective colors and the locations of the lights in the second image that the state of the electronic device is consistent with execution of the test instructions by the electronic device; and

in response to the determining,

storing locally a confirmed identity of the electronic device, and an associated set of signal instructions for controlling the electronic device.

9. The system of claim 8, wherein the analysis of the first image comprises comparing the first image to a plurality of stored images of known electronic devices.

10. The system of claim 9, wherein the instructions for comparing the first image to a plurality of stored images of known electronic devices comprise instructions for:

extracting a plurality of visible device characteristics from the first image; and

comparing the plurality of visible device characteristics to stored device specifications for the known electronic devices.

11. The system of claim 8, further comprising instructions for:

receiving user input specifying a command for the electronic device; and

based on the confirmed identify of the electronic device, transmitting a signal instruction corresponding to the command to the electronic device using the transmitter.

12. The system of claim 11, wherein:

the electronic device is a switch comprising a plurality of input ports, an output port, and the plurality of lights, the plurality of lights including a first light associated with a respective input port of the plurality, wherein the first light can emit a plurality of distinct colors, and each color of emitted light corresponds to a distinct state of the respective input port;

the command is a command to connect the respective input port to the output port; and

the instructions for transmitting the signal instruction comprise instructions for;

using the camera to acquire a third image of the electronic device; and

determining from the color of the emitted light that the respective input port is not connected to the output port.

13. The system of claim 11, wherein:

the electronic device is a television comprising a screen;

the command is a command to turn off the television; and

the instructions for transmitting the signal instruction comprise instructions for;

using the camera to acquire a third image of the electronic device; and

determining from the third image that the screen is emitting light.

14. The system of claim 11, wherein:

the electronic device is a television comprising a screen;

the command is a command to turn on the television; and

the instructions for transmitting the signal instruction comprise instructions for;

using the camera to acquire a third image of the electronic device; and

determining from the third image that the screen is not emitting light.

15. A non-transitory computer readable storage medium storing one or more programs, configured for execution by a system comprising one or more processors, a camera, and a transmitter, the one or more programs comprising instructions for:

using the camera to acquire a first image of an electronic device, wherein the electronic device is configured for remote control and the electronic device comprises a plurality of lights that indicate its state and are at respective locations;

identifying a probable identity of the electronic device based on analysis of the first image, including identifying a light state specification comprising a description of a set of states of the electronic device indicated by respective colors and the locations of the lights;

sending one or more test instructions to the electronic device using the transmitter, wherein the one or more test instructions are selected based on the probable identity of the electronic device;

acquiring a second image of the electronic device using the camera, subsequent to transmitting the one or more test instructions to the electronic device;

comparing the respective colors and the locations of the lights in the second image to the light state specification;

determining from the respective colors and the locations of the lights in the second image that the state of the electronic device is consistent with execution of the test instructions by the electronic device; and

in response to the determining,

storing locally a confirmed identity of the electronic device, and an associated set of signal instructions for controlling the electronic device.

16. The computer readable storage medium of claim 15, wherein the analysis of the first image comprises comparing the first image to a plurality of stored images of known electronic devices.

17. The computer readable storage medium of claim 16, wherein the instructions for comparing the first image to a plurality of stored images of known electronic devices comprise instructions for:

extracting a plurality of visible device characteristics from the first image; and

comparing the plurality of visible device characteristics to stored device specifications for the known electronic devices.

18. The computer readable storage medium of claim 15, wherein the one or more programs further comprising instructions for:

receiving user input specifying a command for the electronic device; and

based on the confirmed identify of the electronic device, transmitting a signal instruction corresponding to the command to the electronic device using the transmitter.

19. The computer readable storage medium of claim 18, wherein:

the electronic device is a switch comprising a plurality of input ports, an output port, and the plurality of lights, the plurality of lights including a first light associated with a respective input port of the plurality, wherein the first light can emit a plurality of distinct colors, and each color of emitted light corresponds to a distinct state of the respective input port;

the command is a command to connect the respective input port to the output port; and

the instructions for transmitting the signal instruction comprise instructions for;

using the camera to acquire a third image of the electronic device; and

determining from the color of the emitted light that the respective input port is not connected to the output port.

20. The computer readable storage medium of claim 18, wherein:

the electronic device is a television comprising a screen;

the command is a command to turn off the television; and

the instructions for transmitting the signal instruction comprise instructions for;

using the camera to acquire a third image of the electronic device; and

determining from the third image that the screen is emitting light.