Image processor, image processing method, recording medium, computer program and semiconductor device

• US 6,771,277 B2
• Filed: 10/05/2001
• Issued: 08/03/2004
• Est. Priority Date: 10/06/2000
• Status: Active Grant

• Current Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Ohba, Akio
• Primary Examiner(s): Brier, Jeffery
• Application Number:
  US09/971,962
• Publication Number:
  US 20020097247A1
• Time in Patent Office: 1,033 Days
• Field of Search: 345/629, 345/634, 345/638, 345/156, 345/157, 345/863
• US Class Current: 345/629
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06T 1/0007 Image acquisition
• CPC Class Codes: H04N 21/42204 User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor constructive details of casings for the remote control device H01H9/0235; user interfaces for controlling a tuning device of a television receiver through a remote control H03J9/00; remote control of peripheral devices connected to a television receiver through the remote control device of the television receiver H04B1/205
• CPC Class Codes: H04N 21/42222 Additional components integrated in the remote control device, e.g. timer, speaker, sensors for detecting position, direction or movement of the remote control, microphone or battery charging device
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/4312 involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 21/47 End-user applications
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

The present invention provides an image processor to use images taken by an image pickup device, etc. as an input interface to enter commands, etc. The present invention provides an image processor that includes image capturing means for capturing a mirrored moving image partially including a moving target, image generating means for generating an object image expressing a predetermined object according to the movement of the target included in the mirrored moving image captured by the image capturing means and controlling means for combining the object image generated by this image generating means with the captured mirrored moving image and displaying the combined image on a predetermined display device.

Claims

1. An image processor comprising:

image capturing means for capturing a mirrored moving image in a time series, said mirrored moving image at least partially including a moving target;

detecting means for detecting said target and a movement component thereof by detecting image features between a mirrored moving image at an actual time point and an immediately preceding mirrored moving image;

image generating means for generating an object image representing a predetermined object in such a way as to change according to said movement component of the target detected by said detecting means;

controlling means for combining said object image generated by the image generating means with said captured mirrored moving image and for displaying said combined image on a predetermined display device; and

means for comparing a combined image obtained by combining said object image generated by said image generating means and said mirrored moving image at the actual time point with a template image, which is an image including only a part of said target included in said immediately preceding mirrored moving image, for detecting a part of said combined image whose image features most resemble said template image, and for making preparations for executing required processing based on the object image when the image of said part of said combined image includes said object image.

2. The image processor according to claim 1, wherein said image generating means is operable to generate said object image in such a way as to follow said movement of said detected target.

3. The image processor according to claim 1, further comprising means for making preparations for executing required processing based on said generated object image according to said movement component of said target.

4. The image processor according to claim 1, further comprising means for executing predetermined processing which is linked to said object image when said movement component of said target detected by said detecting means satisfies predetermined conditions.

5. An image processor comprising:

image capturing means for capturing a mirrored moving image in a time series, said mirrored moving image at least partially including a moving target;

detecting means for detecting said target and a movement component thereof by detecting image features between a mirrored moving image at an actual time point and an immediately preceding mirrored moving image;

image generating means for generating an object image representing a predetermined object in such a way as to change according to said movement component of the target detected by said detecting means; and

controlling means for combining said object image generated by the image generating means with said captured mirrored moving image and for displaying said combined image on a predetermined display device, wherein;

said mirrored moving image includes a plurality of said targets,

said detecting means is operable to detect said movement components of said plurality of targets and to detect one target based on said detected movement components of said plurality of targets, and

said image generating means is operable to change said object image according to said movement component of said one target detected by said detecting means.

6. An image processing method, comprising:

capturing a mirrored moving image at least partially including a moving target in an image processor;

generating an object image representing a predetermined object using said image processor according to a movement of said target included in said captured mirrored moving image;

combining said generated object image with said captured mirrored moving image and displaying said combined image on a predetermined display device;

comparing a combined image obtained by combining said generated object image with said captured mirrored moving image at the actual time point with a template image, which is an image including only a part of said target included in said immediately preceding captured mirrored moving image;

detecting a part of said combined image whose image features most resemble said template image; and

making preparations for executing required processing based on the generated object image when the image of said part of said combined image includes said generated object image.

7. An image processing method as claimed in claim 6 wherein said mirrored moving image includes a plurality of said targets, said method further comprising

detecting movement components of said plurality of targets;

detecting one target based on said detected movement components of said plurality of targets; and

changing said object image according to said movement component of said one detected target.

8. A recording medium containing a computer program for causing a computer with a display device connected thereto to execute steps comprising:

capturing a mirrored moving image at least partially including a moving target;

generating an object image representing a predetermined object according to movement of said target included in said captured mirrored moving image;

combining said object image with said captured mirrored moving image and displaying said combined image on said display device; and

comparing a combined image obtained by combining said generated object image with said captured mirrored moving image at the actual time point with a template image, which is an image including only a part of said target included in said immediately preceding captured mirrored moving image;

detecting a part of said combined image whose image features most resemble said template image; and

making preparations for executing required processing based on the generated object image when the image of said part of said combined image includes said generated object image.

9. A recording medium as claimed in claim 8 wherein said mirrored moving image includes a plurality of said targets, said method further comprising

detecting movement components of said plurality of targets;

detecting one target based on said detected movement components of said plurality of targets; and

changing said object image according to said movement component of said one detected target.

10. A semiconductor device for causing a computer with a display device connected thereto to perform functions comprising:

capturing a mirrored moving image at least partially including a moving target;

generating an object image representing a predetermined object according to movement of the target included in said captured mirrored moving image; and

combining said object image with said captured mirrored moving image and displaying the combined image on said display device; and

comparing a combined image obtained by combining said generated object image with said captured mirrored moving image at the actual time point with a template image, which is an image including only a part of said target included in said immediately preceding captured mirrored moving image;

detecting a part of said combined image whose image features most resemble said template image; and

making preparations for executing required processing based on the generated object image when the image of said part of said combined image includes said generated object image.

11. A semiconductor device as claimed in claim 10 wherein said mirrored moving image includes a plurality of said targets, and wherein said semiconductor device is operable to cause the computer to perform further functions including:

detecting movement components of said plurality of targets;

detecting one target based on said detected movement components of said plurality of targets; and

changing said object image according to said movement component of said one detected target.

12. A graphics processor, comprising:

an image input device operable to capture image frames from a camera in a time series such that said image frames include at least some features of a user of said graphics processor;

an image inverter operable to produce mirrored image frames from said captured image frames, one or more of said features of said mirrored image frames being a target including only a part of the features of the user;

an object image generator operable to produce object frames, said object frames representing one or more predetermined objects for display;

a superimposing image generator operable to combine said mirrored image frames and said object image frames to produce combined image frames;

a detector operable to determine movement of said target from frame to frame and to quantify said movement; and

a controller operable to cause adjustments to said object frames in response to said quantification of said movement of said target and for displaying said combined image frames on a display device and to compare said combined image frame with said mirrored image frame at the actual time point with a template image, which is an image including only a part of said target included in a mirrored image frame preceding said mirrored image frame at the actual time point, to detect a part of said combined image frame whose image features most resemble said template image, and to make preparations for executing required processing based on the object image when the image of said part of said combined image includes said object image.

13. The graphics processor of claim 12, wherein said objects include menu indicia;

and said controller is operable to determine whether said movement of said target indicates a desire by said user to select said menu indicia based on said quantification of said movement of said target by said detector, and to execute one or more further graphics processes based on said menu selection made by said user.

14. The graphics processor of claim 12, wherein said objects include a cursor;

and said controller is operable to determine whether said movement of said target indicates a desire by said user to at least one of acquire and manipulate said cursor based on said quantification of said movement of said target by said detector, and to execute one or more further graphics processes based on manipulation of said cursor by said user.

15. The graphics processor of claim 14, wherein said target includes said cursor when said user has acquired said cursor and said detector is further operable to determine movement of said target with said cursor.

16. A graphics processor as claimed in claim 12 wherein said mirrored image frames include a plurality of said targets, and wherein said controller is further operable to detect movement components of said plurality of targets, to detect one target based on said detected movement components of said plurality of targets, and to change said object image according to said movement component of said one detected target.

17. A method, comprising:

capturing image frames from a camera in a time series such that said image frames include at least some features of a user of a graphics processor;

producing mirrored image frames from said captured image frames, one or more of said features of said mirrored image frames being a target including only a part of the features of the user;

producing object frames, said object frames representing one or more predetermined objects for display;

superimposing said mirrored image frames with said object frames to produce combined image frames;

determining movement of said target from frame to frame and quantifying said movement;

adjusting said object frames in response to said quantification of said movement of said target; and

displaying said combined image frames on a display device; and

comparing a combined image frame obtained by combining said object frame and said mirrored image frame at the actual time point with a template image, which is an image including only a part of said target included in said immediately preceding mirrored image frame;

detecting a part of said combined image frame whose image features most resemble said template image; and

making preparations for executing required processing based on the object frame when the image of said part of said combined image frame includes said object frame.

18. The method of claim 17, wherein said objects include menu indicia;

and said method further includes determining whether said movement of said target indicates a desire by said user to select said menu indicia based on said quantification of said movement of said target, and executing one or more further graphics processes based on said menu selection made by said user.

19. The method of claim 17, wherein said objects include a cursor;

and said method further includes determining whether said movement of said target indicates a desire by said user to at least one of acquire and manipulate said cursor based on said quantification of said movement of said target, and executing one or more further graphics processes based on manipulation of said cursor by said user.

20. The method of claim 19, wherein said target includes said cursor when said user has acquired said cursor, said cursor moving with said target when said user has acquired said cursor.

21. A method as claimed in claim 17 wherein said mirrored image frames include a plurality of said targets, said method further comprising

detecting movement components of said plurality of targets;

detecting one target based on said detected movement components of said plurality of targets; and

changing said object image according to said movement component of said one detected target.