Imaging sensor based multi-dimensional remote controller with multiple input mode

• US 9,001,208 B2
• Filed: 06/06/2012
• Issued: 04/07/2015
• Est. Priority Date: 06/17/2011
• Status: Expired due to Fees

• Current Assignee: Primax Electronics Limited
• Original Assignee: Primax Electronics Limited
• Inventors: Yasutake, Taizo
• Primary Examiner(s): Torrente, Richard
• Assistant Examiner(s): MATT, MARNIE ALEXIA
• Application Number:
  US13/490,234
• Publication Number:
  US 20120320198A1
• Time in Patent Office: 1,035 Days
• Field of Search: 340/348, 340/345
• US Class Current: 348/143
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04N 21/42222 Additional components integrated in the remote control device, e.g. timer, speaker, sensors for detecting position, direction or movement of the remote control, microphone or battery charging device
• CPC Class Codes: H04N 21/42224 Touch pad or touch panel provided on the remote control

Abstract

A method for generating a TV input command using a remote controller having an imaging sensor is presented. The method identifies the corners of a TV display screen from a graphical image captured by an imaging sensor of the remote controller. The method can then perform edge detection and segmentation of the graphical image using reference TV dimensions of the TV display screen to identify the pixel coordinates of the four corners of the TV display screen in the graphical image. The method can then map a camera center position in the pixel coordinates to virtual TV coordinates using a cross ratio algorithm, and then map a location of a cursor in the virtual TV coordinates to the coordinates of the TV display screen.

Claims

1. A method for generating a TV input command using a remote controller having an imaging sensor and a TV display screen having four corners with corresponding pixel coordinates, the method comprising:

identifying, using a remote controller, the corners of the TV display screen from a graphical image captured by an imaging sensor of the remote controller;

performing edge detection and segmentation of the graphical image using reference TV dimensions of the TV display screen to identify the pixel coordinates of the four corners of the TV display screen in the graphical image;

mapping, using a remote controller support module of the TV, a camera center position in the pixel coordinates to virtual TV coordinates using a cross ratio algorithm;

mapping, using the remote controller support module, a location of a cursor in virtual TV coordinates to the coordinates of the TV display screen

identifying a quadrant of the camera center in the TV display screen in the graphical image;

calculating a position of virtual markers that avoid collinear positioning in the cross ratio algorithm; and

using the position of virtual markers in the cross ratio algorithm.

2. The method of claim 1, further comprising identifying the reference TV dimension corresponding data using a look-up table.

3. The method of claim 2, wherein the look-up table includes reference TV dimensions relating to the distance of the remote controller from the TV display screen at various three-dimensional locations and orientations.

4. The method of claim 1, further comprising sending depth data and pixel coordinates of the four corners of the TV display screen in the graphical image to the remote controller support module of the TV.

5. The method of claim 1, further comprising:

identifying when the remote controller is more than a predefined distance from the TV display screen;

defining a virtual TV frame within the graphical image around a centroid of the TV display screen in the graphical image;

calculating the position of virtual markers using the virtual TV frame; and

using the position of virtual markers in the cross ratio algorithm.

6. The method of claim 1, further comprising generating a cursor navigation command and displaying a cursor on the TV display screen.

7. The method of claim 1, further comprising repeating the step of performing edge detection and segmentation and the step of mapping the location of the cursor in the virtual TV coordinates to the coordinates of the TV display screen.

8. The method of claim 7, further comprising stopping the repetition of the step of performing edge detection and segmentation and the step of mapping the location of the cursor in the virtual TV coordinates to the coordinates of the TV display screen when a Z-angle change is being greater than a predetermined value.

9. The method of claim 1 further comprising:

defining a portion mapping area around the location of the cursor position in the coordinates of the TV display; and

moving the location of the cursor position within the portion mapping area based on input data from a touchpad of the remote controller.

10. The method of claim 9, further comprising maintaining the cursor position constant in relation to the portion mapping area when the location of the portion mapping area is changed in response to image sensor data.

11. The method of claim 9, further comprising varying the size of the portion mapping area when the distance between the remote controller and the TV display screen changes.

12. The method of claim 1, further comprising modifying the location of the cursor in the TV coordinates based on input data from a touchpad of the remote controller.

13. The method of claim 1, further comprising recognizing free space gestures made using the remote controller based on movement of the remote controller.

14. The method of claim 1, further comprising switching between input modes of the remote controller to a mode that solely utilizes data from touchpad of the remote controller and does not use data from the imaging sensor.

15. A remote controller for use with a TV display screen having four corners with corresponding pixel coordinates, the remote controller comprising:

an imaging sensor configured to perform image capturing and depth sensing;

a multi-touch touchpad;

a microprocessor configured to process imaging sensor data and multi-touch touchpad data received from the imaging sensor and the multi-touch touchpad based one of a plurality of input modes, in at least one input mode the microprocessor is configured to (i) identify the corners of the TV display screen from a graphical image captured by the imaging sensor of the remote controller, (ii) perform edge detection and segmentation of the graphical image using reference TV dimensions of the TV display screen to identify the pixel coordinates of the four corners of the TV display screen in the graphical image, (iii) identify the quadrant of the camera center in a TV display screen of the graphical image, and (iv) calculate the position of virtual markers that avoid collinear positioning in a cross ratio algorithm.

16. The remote controller of claim 15, wherein in a multi-touch input mode the microprocessor is configured to only process data from the multi-touch touchpad.

17. A remote controller input system for a TV display screen having four corners with corresponding pixel coordinates, the system comprising:

a remote controller comprising;

an imaging sensor configured to perform image capturing and depth sensing;

a multi-touch touchpad;

a microprocessor configured to (i) identify the corners of the TV display screen from a graphical image captured by the imaging sensor of the remote controller, (ii) perform edge detection and segmentation of the graphical image using reference TV dimensions of the TV display screen to identify the pixel coordinates of the four corners of the TV display screen in the graphical image, (iii) identify a quadrant of the camera center in the TV display screen of the graphical image, and (iv) calculate the position of virtual markers that avoid collinear positioning in a cross ratio algorithm; and

a remote controller support module of the TV, the remote controller support module being configured to (i) map a camera center position in the pixel coordinates to virtual TV coordinates using a cross ratio algorithm and (ii) map a location of a cursor in the virtual TV coordinates to the coordinates of the TV display screen.