Motion compensation in an interactive display system

• US 8,982,050 B2
• Filed: 10/17/2013
• Issued: 03/17/2015
• Est. Priority Date: 10/19/2012
• Status: Active Grant

• Current Assignee: Interphase Corporation
• Original Assignee: Interphase Corporation
• Inventors: Solomon, Yoram, Kisacanin, Branislav
• Primary Examiner(s): NGUYEN, KIMNHUNG T
• Application Number:
  US14/056,286
• Publication Number:
  US 20140111433A1
• Time in Patent Office: 516 Days
• Field of Search: 345156-158, 345173-183, 348/63
• US Class Current: 345/158
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0317 in co-operation with a patterned surface, e.g. absolute position or relative movement detection for an optical mouse or pen positioned with respect to a coded surface
• CPC Class Codes: G06F 3/0321 by optically sensing the absolute position with respect to a regularly patterned surface forming a passive digitiser, e.g. pen optically detecting position indicative tags printed on a paper sheet constructional details of pen-shaped pointing devices G06F3/03545, G06F3/03542, G06F3/037
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors

Abstract

An interactive display system including a wireless pointing device, and positioning circuitry capable of determining absolute and relative positions of the display at which the pointing device is aimed. The pointing device captures images displayed by the computer, including one or more human-imperceptible positioning targets. The positioning targets are presented as patterned modulation of the intensity (e.g., variation in pixel intensity) in a display frame of the visual payload, followed by the opposite modulation in a successive frame. At least two captured image frames are subtracted from one another to recover the positioning target in the captured visual data and to remove the displayed image payload. Motion of the pointing device between the two frames is detected by relative motion sensors, and used to align the positions of the positioning targets in the captured images for those frames to improve the fidelity of the recovered positioning target following the subtraction.

Claims

1. A method of operating a computer system including a handheld human interface device, comprising:

generating visual payload image frame data;

combining, with the visual payload image frame data, at least one positioning target pattern corresponding to complementary intensity variances at one or more selected pixel locations of the visual payload image frame data, applied in first and second consecutive frames;

displaying the combined visual payload image frame data and positioning target pattern on a display;

capturing, at the handheld human interface device, image data representative of at least a portion of the display including the positioning target, from each of the first and second frames;

detecting relative motion of the device between the times at which image data is captured from the first and second frames, based on measurements acquired from one or more motion sensors in the device;

aligning the image data of the first and second frames based on the detected relative motion;

subtracting the aligned image data from the first and second frames from one another to recover the positioning target pattern as viewed at the device; and

determining a location of the display at which the device is pointing at the time of one of the first and second frames, responsive to the recovered positioning target pattern.

2. The method of claim 1, wherein the one or more motion sensors comprise:

an image capture subsystem.

3. The method of claim 1, wherein the one or more motion sensors comprise:

a plurality of inertial sensors for sensing relative motion in of three orthogonal directions.

4. The method of claim 3, wherein the one or more relative motion sensors further comprise:

an image capture subsystem.

5. The method of claim 1, wherein the aligning step comprises:

shifting the position of an element in the captured image data for one of the first and second frames by a magnitude and direction corresponding to the detected relative motion.

6. The method of claim 5, wherein the aligning step further comprises:

scaling the size of the element in the captured image data for one of the first and second frames according to the detected relative motion in a direction normal to the display.

7. The method of claim 5, wherein the aligning step further comprises:

rotating the element in the captured image data for one of the first and second frames by an amount corresponding to the detected relative motion of the device about an axis normal to the display.

8. The method of claim 5, wherein the aligning step further comprises:

skewing the element of the image data for one of the first and second frames by an amount corresponding to the detected relative motion of the device about one or more axes parallel to the display.

9. The method of claim 1, wherein the combining step comprises:

inverting the captured image data from the second frame; and

adding the aligned image data from the first and second frames to recover the positioning target pattern as viewed at the device.

10. The method of claim 1, further comprising:

capturing, at the handheld human interface device, image data representative of at least a portion of the display including the positioning target from a third frame;

wherein the step of detecting relative motion of the device between the times at which image data is captured from the first and second frames comprises;

detecting relative motion of the device between the times at which image data is captured from the first and third frames; and

interpolating the relative motion of the device between the times at which image data is captured from the first and second frames, from the detected relative motion of the device between the first and third frames.

11. An interactive display system, comprising:

a computer for generating display image data to be displayed on a display;

graphics output circuitry for generating graphics output signals corresponding to the display image data in a format suitable for display;

a pointing device, comprising;

a hand-held housing;

an image sensor disposed in the housing;

one or more motion sensors; and

image capture circuitry for capturing image data obtained by the image sensor; and

positioning circuitry for determining a location at the display at which the pointing device is aimed by performing a plurality of operations comprising;

generating visual payload image frame data;

combining, with the visual payload image frame data, at least one positioning target pattern corresponding to complementary intensity variances at one or more selected pixel locations of the visual payload image frame data, applied in first and second consecutive frames;

displaying the combined visual payload image frame data and positioning target pattern on a display;

capturing, at the handheld human interface device, image data representative of at least a portion of the display including the positioning target, from each of the first and second frames;

detecting relative motion of the device between the times at which image data is captured from the first and second frames, based on measurements acquired from one or more motion sensors in the device;

aligning the image data of the first and second frames based on the detected relative motion;

subtracting the aligned image data from the first and second frames from one another to recover the positioning target pattern as viewed at the device; and

determining a location of the display at which the device is pointing at the time of one of the first and second frames, responsive to the recovered positioning target pattern.

12. The system of claim 11, wherein the one or more motion sensors are comprised by the image capture circuitry.

13. The system of claim 11, wherein the one or more motion sensors comprise:

a plurality of inertial sensors for sensing relative motion in of three orthogonal directions.

14. The system of claim 13, wherein the one or more relative motion sensors are comprised by the image capture circuitry.

15. The system of claim 11, wherein the aligning operation comprises:

shifting the position of an element in the captured image data for one of the first and second frames by a magnitude and direction corresponding to the detected relative motion.

16. The system of claim 15, wherein the aligning operation further comprises:

scaling the size of the element in the captured image data for one of the first and second frames according to the detected relative motion in a direction normal to the display.

17. The system of claim 15, wherein the aligning operation further comprises:

rotating the element in the captured image data for one of the first and second frames by an amount corresponding to the detected relative motion of the device about an axis normal to the display.

18. The system of claim 15, wherein the aligning operation further comprises:

skewing the element of the image data for one of the first and second frames by an amount corresponding to the detected relative motion of the device about one or more axes parallel to the display.

19. The system of claim 11, wherein the combining operation comprises:

inverting the captured image data from the second frame; and

adding the aligned image data from the first and second frames to recover the positioning target pattern as viewed at the device.

20. The system of claim 11, further comprising:

capturing, at the handheld human interface device, image data representative of at least a portion of the display including the positioning target from a third frame;

wherein the operation of detecting relative motion of the device between the times at which image data is captured from the first and second frames comprises;

detecting relative motion of the device between the times at which image data is captured from the first and third frames; and

interpolating the relative motion of the device between the times at which image data is captured from the first and second frames, from the detected relative motion of the device between the first and third frames.