Control of data processing based on inter-image motion detection

• US 8,654,099 B2
• Filed: 05/17/2007
• Issued: 02/18/2014
• Est. Priority Date: 05/24/2006
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Europe Limited (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Europe Limited (Sony Group Corp.)
• Inventors: Campbell, Diarmid Achibald, Seeger, Mauritius
• Primary Examiner(s): Shalwala, Bipin
• Assistant Examiner(s): HOLTON, STEVEN E
• Application Number:
  US12/301,745
• Publication Number:
  US 20090268040A1
• Time in Patent Office: 2,469 Days
• Field of Search: 345173-178, 178 1801- 1811, 715/863, 382/107
• US Class Current: 345/175
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/533 for prompting the player, e.g. by displaying a game menu
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/308 Details of the user interface
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/757 Matching configurations of points or features
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A data processing apparatus is arranged to receive successive images from a video camera. The apparatus includes means for detecting inter-image motion for selected points in an image region associated with a control function of the apparatus. The apparatus also includes means for executing the control function if inter-image motion is detected in the image region. The detecting means is configured to select points in an image region of a current image that are validated by selecting test points in a currently received image based upon their correspondence with points in a preceding received image. The points in the preceding received image themselves corresponding with the points originally selected in the currently received image. The detecting means is also configured to deselect points in the current image whose corresponding test points do not substantially coincide therewith.

Claims

1. Data processing apparatus arranged to receive successive images from a video camera, the apparatus comprising:

means for detecting inter-image motion for selected points in an image region associated with a control function of the data processing apparatus; and

means for executing the control function if inter-image motion is detected in the image region;

in which the detecting means is arranged to;

identify test points in a currently received image which are detected to have a correspondence with respective points in a preceding received image,

for each test point, detect whether a point in the current image, which correlates with the point in the preceding image corresponding to that test point, is within a threshold image distance of that test point,

select, as a selected point for use by the detecting means, a test point for which the point in the current image, which correlates with the point in the preceding image corresponding to that test point, is within a threshold image distance of that test point,

and deselect other test points; and

difference measuring means for generating difference data and for subsequently smoothing the difference data, the difference data comprising a set of values proportional to the difference in pixel values between two successive images,

in which the difference measuring means is further configured to apply a threshold to the difference data at a difference threshold level, and output resulting binary difference data, and

in which the means for detecting inter-image motion is configured to discard, prior to any selection of test points in the current image based upon correspondence with the points in the preceding image, points in the preceding image for which the equivalent position in the difference image shows a difference below the difference threshold level.

2. Apparatus according to claim 1, comprising a video camera for supplying the successive images.

3. Apparatus according to claim 1, comprising:

a video display;

means for displaying, on the video display, a movable user control;

in which the control function comprises at least moving the moveable user control.

4. Apparatus according to claim 3, in which the moveable user control is moved by an amount dependent upon the degree of inter-image motion detected by the detecting means.

5. Apparatus according to claim 1, comprising difference data storage means operable to combine the binary difference data for a sequence of images to generate a cumulative binary difference image indicating all points over a sequence of images that have shown a difference between successive images that has exceeded a difference threshold.

6. Apparatus according to claim 1, comprising means for processing image data received from the video camera by applying one or both of the following processes:

i. spatial averaging, and

ii. temporal averaging.

7. Apparatus according to claim 1, in which the difference measuring means is operable to calculate a difference threshold between successive images, based upon the variance in the difference values of selected points that are found to be stationary.

8. Apparatus according to claim 1, in which the difference measuring means is operable to calculate a difference threshold between successive images, the calculation being based upon a histogram of difference values for pixels in two or more successive images, and identifying the first histogram peak encountered from the low difference value end of the histogram as the mean camera noise value above which the difference threshold is set.

9. Apparatus according to claim 1, in which the difference measuring means is operable recursively to obtain pairs of points for detecting motion for n previous images stored in an image history store, and to compare the cumulative net motion of these pairs of points to a distance threshold, to distinguish intentional motion from image noise.

10. Apparatus according to claim 9 in which n is three or more.

11. Apparatus according to claim 9, in which the means for detecting inter-image motion is operable to use a cumulative difference image to discard those points where no difference between successive images sufficient to exceed a difference threshold level has occurred, prior to obtaining pairs of points recursively from previous images in the image history.

12. Apparatus according to claim 9,

comprising a points history storage means for storing points detected to have shown intentional motion.

13. Apparatus according to claim 12 comprising means for graphically displaying data derived from the points history, to provide information on detected motion to a user.

14. Apparatus according to claim 9, comprising motion modelling means operable to determine changes to any or all of:

i. translation;

ii. rotation; and

iii. scale;

in pairs of points for the current and preceding image that have been determined to show intentional motion.

15. Apparatus according to claim 14 in which point pairs for some or all stored images older than the current and preceding image are used by the motion modelling means.

16. Apparatus according to claim 14 comprising motion constraint means operable to constrain the modelled motion to conform to a desired behaviour.

17. Apparatus according to claim 16 in which deviation by the user from the constrained modelled motion is evaluated, and the user interface control is locked if the deviation exceeds a threshold.

18. Apparatus according to claim 14 comprising display means to update at least a first display item according to the modelled motion.

19. Apparatus according to claim 18 in which the or each display item represents a user interface control.

20. Apparatus according to claim 19 in which the user interface control is any one of:

i. a wheel control;

ii. a dial control;

iii. a scrolling control; and

iv. a scale control.

21. Apparatus according to claim 20 in which the wheel control and the dial control are scalable.

22. Apparatus according to claim 14 in which the modelled motion is interpreted as a user input to the apparatus.

23. Apparatus according to claim 1 in which the data processing apparatus is a games machine.

24. A non-transitory computer-readable recording medium recorded with computer software having program code, the program code, when executed by a computer, causes the computer to operate as a data processing apparatus according to claim 1.

25. A data processing method comprising the steps of:

receiving successive captured images from a video camera;

detecting inter-image motion for selected points in an image region associated with a control function of the data processing apparatus; and

executing the control function if inter-image motion is detected in the image region;

in which the detecting step comprises;

identifying test points in a currently received image which are detected to have a correspondence with respective points in a preceding received image,

for each test point, detecting whether a point in the current image, which correlates with the point in the preceding image corresponding to that test point, is within a threshold image distance of that test point,

selecting, as a selected point for use by the detecting means, a test point for which the point in the current image, which correlates with the point in the preceding image corresponding to that test point, is within a threshold image distance of that test point, and

deselecting other test points;

generating difference data and for subsequently smoothing the difference data, the difference data comprising a set of values proportional to the difference in pixel values between two successive images,

applying a threshold to the difference data at a difference threshold level, and outputting resulting binary difference data, and

discarding, prior to any selection of test points in the current image based upon correspondence with the points in the preceding image, points in the preceding image for which the equivalent position in the difference image shows a difference below the difference threshold level.

26. A non-transitory computer-readable recording medium recorded with computer software having program code, the program code, when executed by a computer, cause the computer to carry out a method according to claim 25.

27. A data processing apparatus arranged to receive successive images from a video camera, the apparatus comprising one or more processing devices configured to:

detect inter-image motion for selected points in an image region associated with a control function of the data processing apparatus;

execute the control function if inter-image motion is detected in the image region;

identify test points in a currently received image which are detected to have a correspondence with respective points in a preceding received image;

for each test point, detect whether a point in the current image, which correlates with the point in the preceding image corresponding to that test point, is within a threshold image distance of that test point;

select, as a selected point for use by the detecting means, a test point for which the point in the current image, which correlates with the point in the preceding image corresponding to that test point, is within a threshold image distance of that test point;

deselect other test points;

generate difference data and subsequently smooth the difference data, the difference data comprising a set of values proportional to the difference in pixel values between two successive images;

apply a threshold to the difference data at a difference threshold level, and outputting resulting binary difference data; and

discard, prior to any selection of test points in the current image based upon correspondence with the points in the preceding image, points in the preceding image for which the equivalent position in the difference image shows a difference below the difference threshold level.