PORTABLE DATA PROCESSING APPARTATUS

• US 20110157017A1
• Filed: 12/28/2010
• Published: 06/30/2011
• Est. Priority Date: 12/31/2009
• Status: Active Grant

• Current Assignee: Sony Computer Entertainment Europe Limited (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Europe Limited (Sony Group Corp.)
• Inventors: Ankers, Greame, Brooks, Matthew, Marshall, Peter John, Webb, Jonathan
• Granted Patent: US 8,477,099 B2
• Time in Patent Office: Days
• US Class Current: 345/158
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/40 Processing input control signals of video game devices, e.g. signals generated by the player or derived from the environment
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/1075 using a touch screen
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: G06F 1/1692 the I/O peripheral being a secondary touch screen used as control interface, e.g. virtual buttons or sliders
• CPC Class Codes: G06F 3/0317 in co-operation with a patterned surface, e.g. absolute position or relative movement detection for an optical mouse or pen positioned with respect to a coded surface
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 7/269 using gradient-based methods
• CPC Class Codes: H04N 2007/145 Handheld terminals
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors

Abstract

A portable data processing apparatus is provided. It has at least one data processing function which depends on detected motion of the apparatus. The apparatus comprises a video camera operable to capture successive images of a part of the real environment around the apparatus; a video motion detector operable to detect motion of the apparatus by analysis of image motion between pairs of captured images; a hardware motion detector operable to detect motion of the apparatus, whereby the data processing function depends on motion detected by the hardware motion detector; and a controller operable to adjust the operation of the hardware motion detector if the motion detected by the video motion detector and the motion detected by the hardware motion detector differ by at least a threshold difference.

Claims

1. Portable data processing apparatus having at least one data processing function which depends on detected motion of the apparatus, the apparatus comprising:

a video camera operable to capture successive images of a part of a real environment around said apparatus;

a video motion detector operable to detect motion of said apparatus by analysis of image motion between pairs of captured images;

a hardware motion detector operable to detect motion of said apparatus, whereby said data processing function depends on motion detected by said hardware motion detector; and

a controller operable to adjust operation of said hardware motion detector if the motion detected by said video motion detector and the motion detected by said hardware motion detector differ by at least a threshold difference.

2. Apparatus according to claim 1, in which said controller is operable to reset the operation of said hardware motion detector so as to force a detection of substantially zero motion, in response to a detection that the motion detected by said video motion detector is less than the motion detected by said hardware motion detector by at least said threshold difference.

3. Apparatus according to claim 1, in which said controller is operable to cause said data processing function to use motion detected by said video motion detector in place of motion detected by said hardware motion detector, if the motion detected by said video motion detector and the motion detected by said hardware motion detector differ by at least said threshold difference.

4. Apparatus according to claim 1, in which:

said video motion detector and said hardware motion detector are arranged to detect motion as corresponding sets of motion components;

said threshold difference comprises a respective threshold value applying to each motion component; and

said controller is operable to detect that the motion detected by said video motion detector and the motion detected by said hardware motion detector differ by at least said threshold difference if the difference between any pair of the corresponding motion components exceeds said respective threshold value.

5. Apparatus according to claim 1, comprising an image processor operable to apply said data processing function as an image processing operation to one or more captured images in dependence upon motion of said apparatus as detected by said hardware motion detector.

6. Apparatus according to claim 5, in which said image processor is operable to apply said image processing operation so as to compensate, at least partially, for motion of said apparatus at the time that said one or more images were captured.

7. Apparatus according to claim 5, in which said image processor is operable to detect whether two images meet criteria:

(a) that said two images relate to similar parts of the real environment;

(b) that said two images were captured within a threshold time of one another; and

(c) that said apparatus moved by at least a threshold displacement between capture of said two images;

and, in response to a detection that two images meet all of the criteria (a), (b) and (c), to associate at least parts of said two images as a stereo pair of images.

8. Apparatus according to claim 7, in which, for two images which meet all of the criteria (a), (b) and (c), the image processor is operable to detect portions of said two images which relate to respective views of substantially the same part of the real environment, and to associate those image portions as said stereo pair of images.

9. Apparatus according to claim 1, comprising:

an object detector operable to detect a marker object within said captured images; and

an object tracker operable in case said marker object is not detected with a captured image to track an expected position of said marker object with respect to changes in position and orientation of said apparatus;

in which said object detector is operable to alter its detection of said marker object in response to the expected position.

10. Apparatus according to claim 9, in which said marker object is an augmented reality marker, and said apparatus comprises an image generator operable to generate output images from respective captured images, an output image representing a combination of a respective captured image and digitally generated image material, so that the position and/or nature of the digitally generated material within an output image depends on the position and/or nature of the captured marker object within said respective captured image, or the expected position generated by said object tracker in the case that said marker object is not detected within a captured image.

11. Apparatus according to claim 1, in which said hardware motion detector comprises:

an acceleration detector; and

an integrator for integrating an output of said acceleration detector to generate velocity data indicative of velocity of movement of said apparatus.

12. Apparatus according to claim 11, in which said hardware motion detector comprises a second integrator for integrating said velocity data to generate position data indicative of a current position of said apparatus.

13. Apparatus according to claim 1, in which:

said apparatus is a games machine operable to execute a computer game providing a game environment;

motion of said apparatus as detected by said hardware motion detector forms a control input to control operation, in said game environment, of a game object.

14. A method of operation of a portable data processing apparatus having at least one data processing function which depends on detected motion of the apparatus, the method comprising:

capturing successive images of a part of a real environment around said apparatus;

detecting motion of said apparatus by analysis of image motion between pairs of captured video images;

detecting motion of said apparatus with a hardware motion detector, whereby said data processing function depends on motion detected by said hardware motion detector; and

adjusting the operation of the hardware motion detector if the motion detected by image analysis and the motion detected by said hardware motion detector differ by at least a threshold difference.

15. A tangible computer-readable storage medium on which computer readable instructions of computer software are stored, the instructions, when executed by a computer, cause the computer to perform a method of operation of a portable data processing apparatus having at least one data processing function which depends on detected motion of the apparatus, the method comprising

capturing successive images of a part of a real environment around said apparatus;

detecting motion of said apparatus by analysis of image motion between pairs of captured video images;

detecting motion of said apparatus with a hardware motion detector, whereby said data processing function depends on motion detected by said hardware motion detector; and

adjusting the operation of the hardware motion detector if the motion detected by image analysis and the motion detected by said hardware motion detector differ by at least a threshold difference.