Method and apparatus for estimating the value of an input in presence of a non-linear factor

• US 10,238,966 B2
• Filed: 08/27/2015
• Issued: 03/26/2019
• Est. Priority Date: 08/29/2014
• Status: Active Grant

• Current Assignee: Nintendo Company Limited
• Original Assignee: Nintendo Company Limited
• Inventors: Pasquier, Francois-Xavier, Guido, Sebastien, Boudet, Janos
• Primary Examiner(s): McCulloch, Jr., William H
• Assistant Examiner(s): Zhang, Yingchuan
• Application Number:
  US15/507,517
• Publication Number:
  US 20170282068A1
• Time in Patent Office: 1,307 Days
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/40 Processing input control signals of video game devices, e.g. signals generated by the player or derived from the environment
• CPC Class Codes: A63F 13/5255 according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/534 for network load management, e.g. bandwidth optimization, latency reduction
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: H04N 13/31 using parallax barriers
• CPC Class Codes: H04N 13/366 using viewer tracking
• CPC Class Codes: H04N 13/371 for tracking viewers with different interocular distances; for tracking rotational head movements around the vertical axis
• CPC Class Codes: H04N 13/398 Synchronisation thereof; Control thereof

Abstract

The present invention concerns a method and a device for estimating the value of an input in a real-time system. It concerns more particularly real time interactive systems where the input is tracked using a first sensor, typically a camera in presence of a perturbing factor. The perturbing factor is estimated using a second sensor. The component of input computed values due to the perturbing factor are cancelled based on this estimation. A next input value is then estimated from the previous computed input values where the perturbing factor has been cancelled. The estimation is then corrected by applying back the estimation of the perturbing factor.

Claims

1. A method of estimating an input value in a real-time system, wherein the method comprises by a handheld device implementing the real-time system:

computing the input value from at least a first sensor disposed in or on said handheld device, the computed input value indicating a position of at least part of a head relative to the handheld device;

storing the computed input value in an history of computed input values;

using at least a second sensor disposed in or on said handheld device to sense hand movement of said handheld device;

estimating a non-linear perturbing factor component of the input value from the hand movement the at least second sensor senses;

determining a modified history of input values by cancelling said non-linear perturbing factor component from input values stored in said history of computed input values to thereby correct for hand movement; and

estimating the position of the at least part of the head relative to the handheld device from the modified history of input values.

2. The method of claim 1, further comprising adding back the estimated non-linear perturbing factor to the estimated position.

3. The method of claim 1, wherein estimating position from the modified history of input values is based on a model of evolution over time of the input values.

4. The method of claim 3, wherein said model of evolution over time comprises a linear model.

5. The method of claim 4, wherein said estimating position from the modified history comprises determining a set of coefficients to be applied to the modified history of input values using the least square method.

6. The method according to claim 1, wherein said input value specifies a projection of at least a part of the head in front of the handheld device in an image captured by a camera embedded in the handheld device, the camera comprising the first sensor.

7. The method according to claim 6, wherein said at least a part of the head comprises at least one eye of the user of said handheld device.

8. The method of claim 1, wherein said at least a second sensor comprises at least one gyroscope providing output values.

9. The method of claim 8, wherein estimating a non-linear perturbing factor component of the input value comprises:

integrating output values of the gyroscope.

10. The method of claim 1, wherein estimating the position comprises tracking the position of the portion of the head relative to the handheld device with the determined modified history of input values, correcting the tracked position for hand movement, and based on the corrected tracked position, predicting a new position of the portion of the head.

11. A method for determining a feedback in a real-time system, wherein the method comprises by a handheld device implementing the real-time system:

computing the input value from at least a first sensor disposed in or on said handheld device, the at least first sensor sensing at least part of a head relative to the handheld device;

storing the computed input value in an history of computed input values;

using at least a second sensor disposed in or on said handheld device to sense hand movement of said handheld device;

estimating a non-linear factor component of the input value from the hand movement the at least second sensor senses;

determining a modified history of input values by cancelling said non-linear factor component from input values stored in said history of computed input values to correct the history of computed input values for sensed hand movement;

estimating the position of the at least part of a head relative to the handheld device from the modified history of input values; and

determining the feedback based on the estimated position.

12. The method of claim 11, wherein the feedback is an estimation of a next position of a user'"'"'s eyes in front of the device, the device comprising a three-dimensional screen that displays three-dimensional stereoscopic images using a parallax barrier, the method further comprising adjusting the parallax barrier based on the estimated next position of the user'"'"'s eyes.

13. A method of determining an input for adjusting a parallax barrier of a handheld device, the method comprising:

predicting a first value for adjusting the parallax barrier at a time in the future based at least upon images captured by a camera in the handheld device;

receiving a second value from a gyroscope in said handheld device, the second value indicating at least a component of hand movement of the handheld device; and

wherein predicting the first value includes cancelling the component of hand movement of the handheld device from the predicted first value to determine an updated value for adjusting the parallax barrier in the handheld device.

14. A device for estimating an input value in a real-time system, the input value indicative of position of at least a portion of a head relative to the handheld device, the handheld device comprising:

at least a first sensor embedded in said handheld device;

at least a second sensor embedded in said handheld device;

at least one processor, operatively connected to the first sensor, configured to compute the input value from the at least first sensor;

a storage device, operatively connected to the at least one processor, for storing the computed input value in an history of computed input values;

the at least one processor being further configured to;

determine, using the at least second sensor, hand movement of said handheld device;

estimate a non-linear perturbing factor component of the input value from the determined hand movement;

determine a modified history of input values by cancelling said estimated non-linear perturbing factor component from each input value stored in said history of computed input values to thereby correct for hand movement; and

estimate the input value from the modified history of input values.

15. The device of claim 14, wherein the at least one processor is further configured to determine feedback based on the estimated input value.

16. The device of claim 15, wherein said feedback comprises the adjustment of the parallax barrier of a 3-D screen and the input value is the position of the eyes of a user in front of said device.

17. A non-transitory storage device storing a computer program product for a programmable apparatus, the computer program product comprising a sequence of instructions for implementing a method, when loaded into and executed by the programmable apparatus, of estimating an input value in a real-time system comprising:

computing the input value from at least a first sensor disposed in or on a handheld device, the input value indicating position of at least part of a head relative to the handheld device;

storing the computed input value in an history of computed input values on said handheld device;

using at least a second sensor disposed in or on said handheld device to sense hand motion of said handheld device;

estimating a non-linear factor component of the input value from at least the hand motion the at least second sensor senses;

determining a modified history of input values by cancelling said non-linear factor component from input values stored in said history of computed input values to thereby correct for hand motion; and

estimating the input value from the modified history of input values.

18. A handheld device for estimating an input value in a real-time system, the handheld device comprising:

at least a first sensor embedded in said handheld device;

at least a second sensor embedded in said handheld device;

at least one storage device; and

at least one processor operatively connected to the at least first and second sensors and the at least one storage device, the at least one processor executing instructions that control the at least one processor to;

compute the input value from the at least first sensor, the input value indicating position of at least part of a head relative to the handheld device;

store, into the at least one storage device, the computed input value in an history of computed input values;

use at least a second sensor disposed in or on said handheld device to sense hand motion of said handheld device;

estimate a non-linear factor component of the input value from the hand motion the at least second sensor senses;

determine a modified history of input values by cancelling said non-linear factor component from each input value stored in said history of computed input values to thereby correct for hand motion; and

estimate the input value from the modified history of input values.

19. The device of claim 18, wherein the at least one processor executes further instructions controlling the at least one processor to determine feedback based on the estimated input value.

20. The device of claim 19, wherein the device further comprises a stereoscopic display having a parallax barrier, said feedback comprises adjusting the parallax barrier of the stereoscopic display, and the input value indicates the position of the eyes of a user relative to the stereoscopic display.

21. The device of claim 19 wherein the first sensor comprises an image sensor configured to sense position of at least a part of a user'"'"'s face relative to the device, and the second sensor comprises an inertial sensor that is configured to sense how sensing by the image sensor is perturbed by the user'"'"'s hand movement of the device.

22. A handheld stereoscopic display device comprising:

a multi-dimensional tracking sensor producing a tracking signal;

an inertial sensor;

a memory storing program instructions;

a stereoscopic display having an adjustable parallax barrier; and

a processor connected to the tracking sensor, the inertial sensor, the memory and the stereoscopic display, the processor executing the program instructions to perform the following;

(a) tracking a part of a user'"'"'s head in response to a succession of tracking signals;

(b) correcting the tracking for hand movement the inertial sensor senses;

(c) predicting a prospective position of the part of the user'"'"'s head in response to the corrected tracking; and

(d) adjusting the stereoscopic display parallax barrier based on the predicted prospective position.