Controller for electronic appliance

• US 7,629,959 B2
• Filed: 09/20/2005
• Issued: 12/08/2009
• Est. Priority Date: 09/21/2004
• Status: Active Grant

• Current Assignee: JVC Kenwood Corporation
• Original Assignee: Victor Company of Japan Limited (JVC Kenwood Corporation)
• Inventors: Kitaura, Masahiro
• Primary Examiner(s): Shalwala; Bipin
• Assistant Examiner(s): Chowdhury; Afroza Y
• Application Number:
  US11/229,506
• Publication Number:
  US 20060061548A1
• Time in Patent Office: 1,540 Days
• Field of Search: 345/156
• US Class Current: 345/156
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: H04N 21/42204 User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor constructive details of casings for the remote control device H01H9/0235; user interfaces for controlling a tuning device of a television receiver through a remote control H03J9/00; remote control of peripheral devices connected to a television receiver through the remote control device of the television receiver H04B1/205
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/4312 involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations
• CPC Class Codes: H04N 21/4316 for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window
• CPC Class Codes: H04N 21/4318 by altering the content in the rendering process, e.g. blanking, blurring or masking an image region image enhancement or restoration in general G06T5/00
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/4402 involving reformatting operations of video signals for household redistribution, storage or real-time display details of conversion of video standards at pixel level H04N7/01; video transcoding H04N19/40; adapting incoming signals to the display format of the display terminal G09G5/005; media handling at the source in data packet switching networks H04L65/764
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 21/47 End-user applications

Abstract

A controller controls an electronic appliance. The electronic appliance has a display. The controller employs a video camera to photograph an image of a user and converts the image into a mirror image. The controller generates an operational image including operation buttons, overlays the operational image on the user'"'"'s image, and displays the images on the display. In the displayed images, the user moves his or her hand onto one of the displayed operation buttons and bends the fingers of the hand to select the operation button. This selecting operation is detected by a detector of the controller, and information about the selected operation button is supplied to a processing unit, which outputs control signals corresponding to the selected operation button.

Claims

1. A controller for controlling an electronic appliance having a display, comprising:

an image pickup unit configured to photograph an operator who is in front of the display;

a mirror image converter configured to convert an image of the operator photographed by the image pickup unit into a mirror image and output a mirror image signal representative of the mirror image;

an operational image generator configured to generate an operational image signal representative of an operational image that is displayed on the display and is used by the operator to make the electronic appliance perform a desired operation;

a mixer configured to mix the mirror image signal with the operational image signal;

a detector configured to detect that the operator displayed on the display has pointed to the operational image displayed on the display, the operator and operational image being displayed on the display according to the mixed signals from the mixer; and

a controlling unit configured to detect whether or not the operator has carried out an operation with respect to the displayed operational image while the detector is indicating that the operator has pointed to the displayed operational image, and if detecting that the operator has carried out an operation with respect to the displayed operational image, control the electronic appliance according to an operation that is determined from a position of the operational image where the operator has pointed to,

wherein the detector comprises;

an extractor configured to extract an area from the mirror image, the area corresponding to a hand of the operator pointing to the displayed operational image; and

a characteristics detector configured to detect data showing changes of an image signal of the hand corresponding area to the operational image signal,

 

wherein the controlling unit generates a waveform based on the data detected by the characteristics detector and calculates a degree of matching between the waveform and a predetermined waveform, thereby determining whether or not the operator has carried out an operation with respect to the operational image.

2. The controller of claim 1, wherein the characteristics detector further comprises:

a histogram detector configured to detect a histogram showing frequencies of occurrence of gradation levels in the image signal of the hand corresponding area,

wherein the controlling unit generates the waveform based on temporal changes of the detected histogram from the histogram detector, thereby determining whether or not the operator has carried out an operation with respect to the operational image.

3. The controller of claim 1, wherein the extractor further comprises:

a color filter configured to extract a specific-color area from the mirror image, the specific-color area corresponding to the hand of the operator Pointing to the displayed operational image.

4. The controller of claim 3, wherein the extractor further comprises:

a gradation limiter configured to extract a range of predetermined gradation levels from the mirror image signal; and

a synthesizer configured to output a synthesized signal of a signal indicating that the color filter has extracted an image signal of the specific-color area and a signal indicating that the gradation limiter has extracted an image signal of the range of the predetermined gradation levels.

5. The controller of claim 4, wherein the extractor further comprises a gate configured to replace an area excluding the hand corresponding area from the mirror image with a given brightness or color if the synthesizer do not output the synthesized signal.

6. The controller of claim 4, wherein the controlling unit further comprises:

a scaler configured to scale the mirror image signal,

wherein the controlling unit calculates a time when an integral value of the waveform with respect to time takes to reach a first threshold, thereby controlling the scaler based on the calculated time.

7. A method of controlling an electronic appliance having a display, comprising:

photographing an operator who is in front of the display with an image pickup unit;

converting an image of the operator photographed by the image pickup unit into a mirror image and outputting a mirror image signal representative of the mirror image;

generating an operational image signal representative of an operational image that is displayed on the display and is used by the operator to make the electronic appliance perform a desired operation;

mixing the mirror image signal with the operational image signal;

detecting that the operator displayed on the display has pointed to the operational image displayed on the display by extracting a specific-color area from the mirror image signal, the specific-color area corresponding to a hand of the operator pointing to the displayed operational image, the operator and operational image being displayed on the display according to the mixed signals;

detecting whether or not the operator has carried out an operation with respect to the displayed operational image while it is being detected that the operator has pointed to the displayed operational image; and

controlling, if it is detected that the operator has carried out an operation with respect to the displayed operational image, the electronic appliance according to an operation that is determined from a position of the operational image where the operator has pointed to,

wherein the first detecting step further comprises;

extracting, as an extracting step, an area from the mirror image, the area corresponding to a hand of the operator pointing to the displayed operational image; and

detecting, as a change detecting step, data showing changes of an image signal of the hand corresponding area to the operational image signal,

wherein the second detecting step further comprises;

generating, as a comparing step, a waveform based on the data detected in the change detecting step and calculating a degree of matching between the waveform and a predetermined waveform; and

determining whether or not the operator has carried out an operation with respect to the operational image.

8. The method of claim 7, wherein the change detecting step further comprises:

detecting, as a histogram detecting step, a histogram showing frequencies of occurrence of gradation levels in the image signal of the hand corresponding area,

wherein the comparing step comprises;

generating the waveform based on temporal changes of the detected histogram in the histogram detecting step to determine whether or not the operator has carried out an operation with respect to the operational image.

9. The method of claim 7, wherein the extracting step further comprises:

extracting, as a specific-color area extracting step, a specific-color area from the mirror image, the specific-color area corresponding to the hand of the operator pointing to the displayed operational image.

10. The method of claim 9, wherein the extracting step further comprises:

extracting, as a gradation range extracting step, a range of predetermined gradation levels from the mirror image signal; and

outputting, as a synthesizing step, a synthesized signal of a signal indicating that an image signal of the specific-color area has been extracted in the extracting step and a signal indicating that an image signal of the range of the predetermined gradation levels has been extracted in the gradation range extracting step.

11. The method of claim 10, wherein the extracting step further comprises:

replacing an area excluding the hand corresponding area from the mirror image with a given brightness or color if the synthesized signal is not output in the synthesizing step.

12. The method of claim 7, wherein the second extracting step further comprises:

calculating, as a calculating step, a time when an integral value of the waveform with respect to time takes to reach a first threshold; and

scaling the mirror image signal output based on the calculated time in the calculating step.