Apparatus and method of interaction with a data processor

• US 8,241,125 B2
• Filed: 07/17/2007
• Issued: 08/14/2012
• Est. Priority Date: 07/25/2006
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Europe Limited (Sony Group Corp.)
• Inventors: Hughes, Colin Jonathan
• Primary Examiner(s): Richards, N Drew
• Assistant Examiner(s): CARPENTER, ROBERT K
• Application Number:
  US12/374,721
• Publication Number:
  US 20090318228A1
• Time in Patent Office: 1,855 Days
• Field of Search: 436/36, 463/36
• US Class Current: 463/36
• CPC Class Codes: G02B 13/06 Panoramic objectives; So-called "sky lenses" including panoramic objectives having reflecting surfaces
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object

Abstract

A method of interaction with a data processor such as a games machine comprises the steps of capturing a wide angle image of an environment, obtaining image features from the wide angle image, and then interpreting these image features as inputs to software running on the games machine. The step of obtaining image features from the wide angle image further comprises the step of mapping one or more regions of the wide-angle image to corresponding one or more flat image planes, and obtaining image features from one or more of said planes of the mapped wide-angle image. Advantageously, the planes can be arranged to reflect the topology of a room or the positioning of players, and allow a consistent representation of image features and consequently of input motion to a process such as a game, without position dependent distortions of the features inherent in the wide angle image.

Claims

1. A method of interaction with a games machine, comprising the steps of:

capturing, with a camera, a wide angle image of an environment, in which the wide-angle image is substantially hemispherical;

obtaining image features from the wide angle image, and

interpreting those image features as inputs to software running on the games machine,

wherein the step of obtaining said image features from the wide angle image further comprises the steps of;

mapping, with a processing device, one or more regions of the wide-angle image to corresponding one or more flat image planes, hereafter referred to as facets;

obtaining said image features from a plurality of the facets of the mapped wide angle image; and

estimating, from said image features obtained from the plurality of the facets, the positioning of at least part of the body of at least a first user;

in which the wide angle image is mapped to facets such that each of a plurality of users in the wide angle image is mapped to a separate respective facet.

2. A method according to claim 1, in which there are five facets that together form five planar surfaces of a cube.

3. A method according to claim 1, further comprising a step in which the image features are obtained by performing edge detection on the plurality of the facets.

4. A method according to claim 3, in which edge detection is based upon changes in any or all of:

i. luminance, and

ii. chrominance.

5. A method according to claim 3, in which edge detection employs any or all of:

i. the Sobel operator;

ii. the Canny operator;

iii. the pyramid operator; and

iv. the Laplace operator.

6. A method according to claim 3, further comprising a step of selecting any or all of the following edge types as image features:

i. substantially linear edges;

ii. points where two or more linear edges meet; and

iii. predetermined shapes formed by sets of edges.

7. A method according to claim 6, in which substantially linear edges are determined by identifying three or more substantially co-linear edge points.

8. A method according to claim 6, in which predetermined shapes are identified by any one of:

i. template matching;

ii. Hough'"'"'s transform; and

iii. Bresenham'"'"'s transform.

9. A method according to claim 1, comprising a step of estimating, from image features obtained from the plurality of the facets, any or all of:

i. an initial orientation; and

ii. an initial position, with respect to said features.

10. A method according to claim 9, comprising a further step of re-estimating, from image features obtained from the plurality of the facets, any or all of:

i. a current orientation; and

ii. a current position.

11. A method according to claim 10, in which relative movement of selected image features is tracked over successive images to determine changes in any or all of:

i. orientation relative to a preceding orientation, and;

ii. position relative to a preceding position.

12. A method according to claim 11, in which the relative movement of selected image features is determined by generating an image transform embodying any or all of:

i. orientation parameters; and

ii. translation parameters, and parametrically minimising the error between the current image features and a prediction of the current image features based upon previously obtained image features processed by said image transform.

13. A method according to claim 11, in which successive estimates of movement are combined as inputs to a game.

14. A method according to claim 13, in which an input may represent the movement of any one of:

i. a torch or gun;

ii. a computer mouse;

iii. a wand or gestural device;

iv. a golf club;

v. a fishing rod;

vi. a melee weapon; and

vii. a locus in 3D space.

15. A method according to claim 1, further comprising a step of tracking movements of at least a first user by comparing image features from the plurality of the facets of successive images.

16. A method according to claim 1 in which one or more of a plurality of users whose image is mapped to a separate respective facet are displayed within a game.

17. A method according to claim 1, in which a processing sequence comprises:

i. mapping a wide-angle image to the plurality of the facets;

ii. obtaining image features from the plurality of the facets; and

iii. estimating current orientation and/or position with respect to said image features, is carried out by an image processing means of a host data processor upon receipt of a wide-angle image transmitted from a peripheral control means.

18. A method according to claim 1, in which a processing sequence comprises:

i. mapping a wide-angle image to the plurality of the facets;

ii. obtaining image features from the plurality of the facets, and;

iii. estimating current orientation and/or position with respect to said image features, is carried out by an image processing means of a peripheral control means.

19. A method according to claim 18, in which orientation and/or position information is transmitted from the peripheral control means to a host data processor.

20. A method according to claim 1, in which a processing sequence comprises:

i. mapping a wide-angle image to the plurality of the facets;

ii. obtaining image features from the plurality of the facets; and

iii. estimating current orientation and/or position with respect to said image features, is shared at any point in the processing sequence between firstly a peripheral control means and secondly a host data processor.

21. A method according to claim 20, in which information relevant to the next step in the processing sequence is transmitted from the peripheral control means to a host data processor.

22. A method according to claim 1, in which the data processor is a games machine.

23. A non-transitory data carrier comprising computer readable instructions that, when loaded into a computer, cause the computer to carry out the method of claim 1.

24. A games system comprising:

a camera to capture a wide angle image of an environment, in which the wide-angle image is substantially hemispherical;

an image processor to obtain image features from the wide angle image, and to interpret those image features as inputs to software running on the games system,

wherein the image processor is configured to;

map one or more regions of the wide-angle image to corresponding one or more flat image planes, hereafter referred to as facets;

obtain said image features from a plurality of the facets of the mapped wide angle image; and

estimate, from said image features obtained from the plurality of the facets, the positioning of at least part of the body of at least a first user;

in which the wide angle image is mapped to facets such that each of a plurality of users in the wide angle image is mapped to a separate respective facet.

25. A games system according to claim 24 in which the image processor is further operable to obtain any or all of the following image features from the plurality of the facets of the mapped wide-angle image:

i. substantially linear edges;

ii. points where two or more linear edges meet, and;

iii. predetermined shapes formed by a sets of edges.

26. A games system according to claim 24 in which the image processor is further operable to estimate a current orientation and/or position with respect to said image features.

27. A games system according to claim 24 in which the image processor is further operable to estimate the positioning of at least part of the body of at least a first user based on said image features.

28. A games system according to claim 24, wherein the camera is operably coupled to the image processor by wireless communications means.

29. A games system according to claim 24, in which the camera comprises a charge coupled device operable to generate at wide-angle image comprising at least 1.5 megapixels.

30. A games machine, comprising:

a camera configured to capture a wide angle image of an environment, in which the wide-angle image is substantially hemispherical; and

an image processor configured to;

obtain image features from the wide angle image; and

interpret those image features as inputs to software running on the games machine,

wherein the image processor is configured to obtain the image features from the wide angle image by;

mapping one or more regions of the wide-angle image to corresponding one or more flat image planes, hereafter referred to as facets;

obtaining the image features from a plurality of the facets of the mapped wide angle image; and

estimating, from the image features obtained from the plurality of the facets, the positioning of at least part of the body of at least a first user;

in which the wide angle image is mapped to facets such that each of a plurality of users in the wide angle image is mapped to a separate respective facet.

31. A non-transitory data carrier comprising computer readable instructions that, when loaded into a computer, cause the computer to operate as an image processor to:

map one or more regions of a wide-angle image to corresponding one or more flat image planes, hereafter referred to as facets;

obtain image features from a plurality of the facets of the mapped wide angle image; and

estimate, from the image features obtained from the plurality of the facets, positioning of at least part of the body of at least a first user;

in which the wide angle image is mapped to facets such that each of a plurality of users in the wide angle image is mapped to a separate respective facet.