Motion sensing interface

• US 6,498,628 B2
• Filed: 11/17/1998
• Issued: 12/24/2002
• Est. Priority Date: 10/13/1998
• Status: Expired due to Term

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.), Sony Electronics Inc. (Sony Group Corp.)
• Inventors: Iwamura, Ryuichi
• Primary Examiner(s): Miller, John
• Assistant Examiner(s): Tran, Trang U.
• Application Number:
  US09/193,594
• Publication Number:
  US 20020057383A1
• Time in Patent Office: 1,498 Days
• Field of Search: H04/N.544, H04/N.718, H04/N.5225, 348/734, 348/169, 348/700, 348/155, 345/156, 345/157, 345/158, 345/145
• US Class Current: 348/734
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G08C 2201/32 Remote control based on movements, attitude of remote control device
• CPC Class Codes: G08C 23/00 Non-electrical signal transmission systems, e.g. optical systems
• CPC Class Codes: H04N 21/47 End-user applications

Abstract

An electronic appliance remote controller which includes a display screen (which may be part of the appliance, e.g. a TV screen) for displaying icons representing possible operations of the electronic appliance, and a motion detector circuit for detecting a motion within a field of view of the motion detector circuit. The motion detector circuit either detects an image of the user'"'"'s hand or a predetermined motion of a moving hand within the field of view as an indication that a remote control operation is to be started and, thereafter, tracks the movement of the hand. The motion detector circuit outputs a cursor control signal representative of the motion of the hand. A control circuit, connected to the display screen, the electronic appliance, and the motion detector circuit and supplied with the cursor control signal, controls the display screen to display a movable visual indicator, e.g. a cursor, whose own motion tracks the movement of the moving hand and the electronic appliance to perform operations corresponding to the icons selected by the user using the visual indicator. In one embodiment, two cameras allow three dimensional movements of the user'"'"'s hand to control the electronic appliance.

Claims

1. A remote controller for an electronic appliance, comprising:

a display screen for displaying icons representing possible operations of the electronic appliance;

a motion detector for detecting a hand motion within a field of view of the motion detector, the motion detector first detecting a predetermined motion of a user'"'"'s hand within the field of view to initiate remote control operation and, thereafter, tracking the movement of the hand and outputting a cursor control signal representative of the motion of the hand; and

a control circuit, connected to the display screen, the electronic appliance, and the motion detector and supplied with the cursor control signal, for controlling the display screen to display a movable visual indicator which tracks the movement of the moving hand and for controlling the electronic appliance to perform operations corresponding to the icons selected by the user using the visual indicator.

2. A remote controller according to claim 1, wherein the motion detector detects the selection of an icon by the user by detecting a predetermined motion pattern of the hand when the visual indicator is coincident on the display screen with a particular icon.

3. A remote controller according to claim 2, wherein the predetermined motion pattern is a cessation of movement of the hand for a predetermined period of time after the visual indicator is coincident on the display screen with the particular icon.

4. A remote controller according to claim 2, wherein the predetermined motion pattern is a movement of the hand along an axis perpendicular to the display screen after the visual indicator is coincident on the display screen with the particular icon.

5. A remote controller according to claim 1, wherein the motion detector includes at least one video camera.

6. A remote controller according to claim 5, wherein the motion detector detects motion by comparing the pixel values of each corresponding macro block between two time sequential video frames in a video output signal from the video camera and determining that motion has taken place with respect to a macro block if the difference in pixel values between the two video frames for that macro block exceeds a predetermined threshold value and, if so, determining that macro block to be an active region.

7. A remote controller according to claim 6, wherein in detecting the user'"'"'s hand, the motion detector checks each macro block of the two time sequential video frames to determine if a grouping of physically proximate macro blocks in a single video frame exhibits a predetermined pattern of becoming active regions and then non-active regions.

8. A remote controller according to claim 7, wherein in detecting the user'"'"'s hand, the motion detector, after detecting that a grouping of physically proximate macro blocks in a single video frame exhibits a predetermined pattern of becoming active regions and then non-active regions, compares a portion of the video image contained in the current active region with a stored video image and determines that the current active region of the video image output by the camera is the user'"'"'s hand if there is a match with the stored video image.

9. A remote controller according to claim 5, wherein in detecting the user'"'"'s hand, the motion detector compares a user selected portion of the video image output by the camera with a stored video image and determines that the user selected portion of the video image output by the camera is the user'"'"'s hand if there is a match with the stored video image.

10. A remote controller according to claim 1, wherein the visual indicator is a cursor.

11. A remote controller according to claim 1, wherein the electronic appliance is a television set.

12. A remote controller according to claim 1, wherein the display screen is a cathode ray tube.

13. A remote controller according to claim 1, further comprising an automatic cursor sensitivity control means for automatically adjusting a length of movement of the visual indicator on the display screen relative to a corresponding length of movement of the moving hand detected by the motion detector.

14. A remote controller according to claim 13, wherein the automatic cursor sensitivity control means adjusts the length of movement of the visual indicator on the display screen to move a fixed, predetermined distance on the display screen in response to the length of the detected first predetermined motion.

15. A remote controller according to claim 1, wherein the motion detector includes two video cameras placed perpendicularly with respect to each other to view the user.

16. A remote controller according to claim 15, wherein the motion detector detects motion by comparing the pixel values of each macro block between two time sequential video frames in a video output signal from each video camera and determining that motion has taken place with respect to a macro block if the difference in pixel values between the two video frames for that macro block exceed a predetermined threshold value and, if so, determining that macro block to be an active region.

17. A remote controller according to claim 16, wherein in detecting the user'"'"'s hand, the motion detector checks each macro block of the two time sequential video frames of each camera to determine if a grouping of physically proximate macro blocks in a single video frame exhibits a predetermined pattern of becoming active regions and then non-active regions.

18. A remote controller according to claim 17, wherein in detecting the user'"'"'s hand, the motion detector, after detecting that a grouping of corresponding physically proximate macro blocks in a single video frame of the output of each camera exhibits a predetermined pattern of becoming active regions and then non-active regions, locks onto the active regions of the video images of both cameras as the active regions.

19. A remote controller according to claim 17, wherein in detecting the user'"'"'s hand, the motion detector, after detecting that a grouping of corresponding physically proximate macro blocks in a single video frame of the output of each camera exhibits a predetermined pattern of becoming active regions and then non-active regions, compares a portion of the video image contained in the current active region with a stored video image and determines that the current active region of the video image output by the cameras is the user'"'"'s hand if there is a match with the stored video image.

20. A remote controller according to claim 15, wherein the motion detector detects the selection of an icon by the user by detecting a predetermined motion pattern of the hand when the visual indicator is coincident on the display screen with a particular icon.

21. A remote controller according to claim 20, wherein the predetermined motion pattern is a cessation of movement of the hand for a predetermined period of time after the visual indicator is coincident on the display screen with the particular icon.

22. A remote controller according to claim 20, wherein the predetermined motion pattern is a movement of the hand along an axis perpendicular to the display screen after the visual indicator is coincident on the display screen with the particular icon.

23. A method of remote controlling an electronic appliance, comprising the steps of:

displaying icons on a display screen representing possible operations of the electronic appliance;

detecting a hand motion within a field of view of the motion detector by the steps of first detecting a predetermined motion of a user'"'"'s hand within the field of view to initiate remote control operation and, thereafter, tracking the movement of the hand and outputting a cursor control signal representative of the motion of the hand; and

controlling the display screen to display a movable visual indicator which tracks the movement of the moving hand and for controlling the electronic appliance to perform operations corresponding to the icons selected by the user using the visual indicator.

24. A method of remote controlling according to claim 23, wherein the motion detecting step includes detecting the selection of an icon by the user by detecting a predetermined motion pattern of the hand when the visual indicator is coincident on the display screen with a particular icon.

25. A method of remote controlling according to claim 23, wherein the predetermined motion pattern is a cessation of movement of the hand for a predetermined period of time after the visual indicator is coincident on the display screen with the particular icon.

26. A method of remote controlling according to claim 23, wherein the predetermined motion pattern is a movement of the hand along an axis perpendicular to the display screen after the visual indicator is coincident on the display screen with the particular icon.

27. A method of remote controlling according to claim 23, wherein the motion detecting step includes the step of using a video camera to generate a video image of the field of view.

28. A method of remote controlling according to claim 27, wherein the motion detecting step includes the steps of detecting motion by comparing the pixel values of each macro block between two time sequential video frames in a video output signal from the video camera and determining that motion has taken place with respect to a macro block if the difference in pixel values between the two video frames for that macro block exceed a predetermined threshold value and, if so, determining that macro block to be an active region.

29. A method of remote controlling according to claim 28, wherein in detecting the user'"'"'s hand, the motion detecting step includes the steps of checking each macro block of the two time sequential video frames to determine if a grouping of physically proximate macro blocks in a single video frame exhibits a predetermined pattern of becoming active regions and then non-active regions.

30. A remote controller according to claim 28, wherein in detecting the user'"'"'s hand, the motion detecting step includes the steps, after detecting that a grouping of corresponding physically proximate macro blocks in a single video frame of the output of each camera exhibits a predetermined pattern of becoming active regions and then non-active regions, comparing a portion of the video image contained in the current active region with a stored video image and determining that the current active region of the video image output by the cameras is the user'"'"'s hand if there is a match with the stored video image.

31. A method of remote controlling according to claim 30, wherein in detecting the user'"'"'s hand, the motion detecting step includes the steps of, after detecting that a grouping of physically proximate macro blocks in a single video frame exhibits a predetermined pattern of becoming active regions and then non-active regions, comparing a portion of the video image contained in the current active region with a stored video image and determining that the current active region of the video image output by the camera is the user'"'"'s hand if there is a match with the stored video image.

32. A method of remote controlling according to claim 23, wherein the visual indicator is a cursor.

33. A method of remote controlling according to claim 23, wherein the electronic appliance is a television set.

34. A method of remote controlling according to claim 23, wherein the display screen is a cathode ray tube.

35. A method of remote controlling according to claim 23, further comprising the steps of automatically adjusting a length of movement of the visual indicator on the display screen relative to a corresponding length of movement of the moving hand detected by the motion detecting step.

36. A method of remote controlling according to claim 35, further comprising the steps of adjusting the length of movement of the visual indicator on the display screen to move a fixed, predetermined distance on the display screen in response to the length of the detected first predetermined motion.

37. A method of remote controlling according to claim 23, wherein the motion detecting step includes generating two simultaneous video images of the field of view by using two video cameras placed perpendicularly with respect to each other to view the user.

38. A method of remote controlling according to claim 37, wherein the motion detecting step includes the steps of detecting motion by comparing the pixel values of each macro block between two time sequential video frames in a video output signal from each video camera and determining that motion has taken place with respect to a macro block if the difference in pixel values between the two video frames for that macro block exceed a predetermined threshold value and, if so, determining that macro block to be an active region.

39. A method of remote controlling according to claim 32, wherein in detecting the user'"'"'s hand, the motion detecting step includes the steps of checking each macro block of the two time sequential video frames of each camera to determine if a grouping of physically proximate macro blocks in a single video frame exhibits a predetermined pattern of becoming active regions and then non-active regions.

40. A method of remote controlling according to claim 39, wherein in detecting the user'"'"'s hand, the motion detecting step includes the steps of, after detecting that a grouping of corresponding physically proximate macro blocks in a single video frame of the output of each camera exhibits a predetermined pattern of becoming active regions and then non-active regions, comparing a portion of the video image contained in the current active region with a stored video image and determining that the current active region of the video image output by the cameras is the user'"'"'s hand if there is a match with the stored video image.

41. A method of remote controlling according to claim 37, wherein in detecting the user'"'"'s hand, the motion detecting step includes the steps of comparing a user selected portion of the video image output by the cameras with a stored video image and determining that the user selected portion of the video image output by the cameras is the user'"'"'s hand if there is a match with the stored video image.

42. A method of remote controlling according to claim 37, wherein the motion detecting step includes the steps of detecting the selection of an icon by the user by detecting a predetermined motion pattern of the hand when the visual indicator is coincident on the display screen with a particular icon.

43. A method of remote controlling according to claim 42, wherein the predetermined motion pattern is a cessation of movement of the hand for a predetermined period of time after the visual indicator is coincident on the display screen with the particular icon.

44. A method of remote controlling according to claim 42, wherein the predetermined motion pattern is a movement of the hand along an axis perpendicular to the display screen after the visual indicator is coincident on the display screen with the particular icon.