Coordinate calculating apparatus and coordinate calculating program

• US 7,786,976 B2
• Filed: 04/18/2006
• Issued: 08/31/2010
• Est. Priority Date: 03/09/2006
• Status: Active Grant

• Current Assignee: Nintendo Company Limited
• Original Assignee: Nintendo Company Limited
• Inventors: Ohta, Keizo
• Primary Examiner(s): Mengistu; Amare
• Assistant Examiner(s): PATEL, PREMAL R
• Application Number:
  US11/405,663
• Publication Number:
  US 20070211050A1
• Time in Patent Office: 1,596 Days
• Field of Search: 345/156, 345/158, 382/276, 382/286, 382/291, 382/293, 382/295
• US Class Current: 345/156
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors

Abstract

An object position coordinate computing and selection method and apparatus detects one or more areas satisfying a predetermined criterion in a captured digital image and compares an object-position coordinate point to associate with such areas. When a plurality of such areas have been detected and associated object-position coordinate points have been computed, the coordinate computing method and apparatus sets a reference value based on at least one or both of a direction of a line connecting the first two object-position coordinate points recognized in the image and a length of the line connecting the two object-position coordinate points. The coordinate computing method and apparatus then selects a single pair combination of object-position coordinate points out of the plurality of possible pair combinations of object-position coordinate points that is closest to the set reference value for use as selected object position coordinate points.

Claims

1. A coordinate calculating apparatus for calculating two object position coordinate points from an image captured by an image capturing apparatus, the two object position coordinate points indicating positions of two predetermined objects whose images to be captured, the positions being on a plane corresponding to the captured image, the apparatus comprising:

coordinate detecting means of detecting a position coordinate point of an area satisfying a predetermined condition in the captured image for each area;

reference setting means of setting a reference with respect to at least one of a direction connecting two position coordinate points in the captured image and a length between the two position coordinate points;

selection means of, when a plurality of position coordinate points are detected by the coordinate detecting means, with respect to combinations of two position coordinate points among the plurality of position coordinate points, comparing at least one of a direction connecting two position coordinate points in the combinations and a length between the two position coordinate points, with the reference, and selecting a combination of two position coordinate points in the combinations most close to the reference as the two object position coordinate points.

2. The coordinate calculating apparatus according to claim 1, wherein the reference setting means calculates a first vector indicating a direction on the plane, based on tilt detected by a means for determining a tilt of the image capturing apparatus, where the direction connecting the two position coordinate points is used as a reference, and

the selection means determines a combination of two position coordinate points to be selected, based on a similarity between a direction connecting two position coordinate points and the direction of the first vector.

3. The coordinate calculating apparatus according to claim 2, wherein the selection means calculates a similarity from an inner product value of a vector connecting two position coordinate points among the plurality of position coordinate points and the first vector.

4. The coordinate calculating apparatus according to claim 2, wherein the reference setting means calculates the first vector based on a correspondence relationship between a tilt determined by the tilt determining means in a situation that the image capturing apparatus is at a predetermined tilt, and a direction connecting the two object position coordinate points detected from an image captured by the image capturing apparatus in the situation.

5. The coordinate calculating apparatus according to claim 2, wherein the tilt determining means includes an acceleration sensor which is fixedly provided to a housing used in common with the image capturing apparatus, and detects accelerations in two axial directions perpendicular to an image capturing direction of the image capturing apparatus, and

the reference setting means calculates the first vector based on the accelerations in the axial directions detected by the acceleration sensor.

6. The coordinate calculating apparatus according to claim 1, further comprising:

object position storing means of storing object position coordinate data including at least one of a set of the two object position coordinate points selected by the selection means, and a vector indicating a direction connecting the two object position coordinate points, every time selection is performed by the selection means,

wherein the coordinate detecting means sequentially detects position coordinate points from ages sequentially captured by the image capturing apparatus,

the reference setting means calculates or obtains a second vector which is a vector indicating a direction connecting two object position coordinate points from object position coordinate data most recently stored into the object position storing means, every time the coordinate detecting means detects a position coordinate point, and sets the second vector as the reference, and

the selection means determines a combination of two position coordinate points to be selected, based on a similarity between a direction connecting two position coordinate points and the direction of the second vector.

7. The coordinate calculating apparatus according to claim 6, wherein the selection means determines a combination of two position coordinate points to be selected, based on a similarity between a direction connecting two position coordinate points and the direction of the vector and a similarity between a length between two position coordinate points and a length of the second vector.

8. The coordinate calculating apparatus according to claim 6, further comprising:

first determination means of determining whether or not the object position coordinate data has been stored into the object position storing means by selections performed from a predetermined number of times before until the previous time, before the reference setting means sets the reference,

wherein the reference setting means calculates or obtains the second vector from most recently stored object position coordinate data only when the result of the determination by the first determination means is positive.

9. The coordinate calculating apparatus according to claim 6, wherein the selection means calculates a similarity from an inner product value of a vector connecting two position coordinate points from the plurality of position coordinate points, and the second vector.

10. The coordinate calculating apparatus according to claim 1, further comprising:

object position storing means of storing object position coordinate data including at least one of a set of the two object position coordinate points selected by the selection means, and a vector indicating a direction connecting the two object position coordinate points, every time selection is performed by the selection means; and

second determination means of determining whether or not two object position coordinate points have been selected at least once among selections performed by the selection means from a predetermined number of times before,

wherein the coordinate detecting means sequentially detects position coordinate points from images sequentially captured by the image capturing apparatus,

the reference setting means sets a first vector indicating a direction on the plane based on a tilt of the image capturing apparatus, using a direction connecting two position coordinate points on the plane as a reference, when a result of the determination by the second determination means is positive; and

calculates or obtains a second vector which is a vector indicating a direction connecting two object position coordinate points from object position coordinate data most recently stored by the object position storing means when the result of the determination by the second determination means is negative, and

the selection means determines a combination of two position coordinate points to be selected, based on a similarity between a direction connecting two position coordinate points and the direction of the first vector when the result of the determination by the second determination means is positive; and

determines a combination of two position coordinate points to be selected, based on a similarity between a direction connecting two position coordinate points and the direction of the second vector when the result of the determination by the second determination means is negative.

11. The coordinate calculating apparatus according to claim 10, wherein the tilt determining means includes an acceleration sensor which is fixedly provided to a housing used in common with the image capturing apparatus, and detects accelerations in two axial directions perpendicular to an image capturing direction of the image capturing apparatus, and

the reference setting means calculates the first vector based on the accelerations in the axial directions detected by the acceleration sensor.

12. A non-transitory computer readable storage medium storing one or more sequences of one or more processing instructions which when executed by a computer, implement a method for calculating two object position coordinate points from an image captured by an image capturing apparatus, the two object position coordinate points indicating positions of two predetermined objects whose images were captured, the positions being on a plane corresponding to the captured image, the processing instructions causing the computer to execute:

a reference setting step of setting a reference with respect to at least one of a direction connecting two position coordinate points in the captured image and a length between the two position coordinate points; and

a selection step of, with respect to combinations of two position coordinate points among position coordinate points of areas satisfying a predetermined condition in the captured image, comparing at least one of a direction connecting two position coordinate points in the combinations and a length between the two position coordinate points, with the reference, and selecting a combination of two position coordinate points in the combinations most close to the reference as the two object position coordinate points.

13. The storage medium according to claim 12, wherein, in the reference setting step, the computer calculates a first vector indicating a direction on the plane, based on a tilt of the image capturing apparatus, where the direction connecting the two position coordinate points is used as a reference, and

in the selection step, the computer determines a combination of two position coordinate points to be selected, based on a similarity between a direction connecting two position coordinate points and the direction of the first vector.

14. The storage medium according to claim 13, wherein, in the selection step, the computer calculates a similarity from an inner product value of a vector connecting two position coordinate points among the plurality of position coordinate points and the first vector.

15. The storage medium according to claim 13, wherein, in the reference setting step, the computer calculates the first vector based on a correspondence relationship between a tilt determined in a situation that the image capturing apparatus is at a predetermined tilt, and a direction connecting the two object position coordinate points detected from an image captured by the image capturing apparatus in the situation.

16. The storage medium according to claim 13, wherein the tilt determining means includes an acceleration sensor which is fixedly provided to a housing used in common on with the image capturing apparatus, and detects accelerations in two axial directions perpendicular to an image capturing direction of the image capturing apparatus, and

in the reference setting step, the computer calculates the first vector based on the accelerations in the axial directions detected by the acceleration sensor.

17. The storage medium according to claim 12, wherein the program causes the computer to further execute:

an object position storing step of storing object position coordinate data including at least one of a set of the two object position coordinate points selected by the selection step, and a vector indicating a direction connecting the two object position coordinate points, into a memory of the coordinate calculating apparatus, every time selection is performed in the selection step,

wherein position coordinate points of objects to be selected in the selection step are sequentially detected from images sequentially captured by the image capturing apparatus,

in the reference setting step, the computer calculates or obtains a second vector which is a vector indicating a direction connecting two object position coordinate points from object position coordinate data most recently stored into the memory, every time the coordinate detecting means detects a position coordinate point, and sets the second vector as the reference, and

in the selection step, the computer determines a combination of two position coordinate points to be selected, based on a similarity between a direction connecting two position coordinate points and the direction of the second vector.

18. The storage medium according to claim 17, wherein, in the selection step, the computer determines a combination of two position coordinate points to be selected, based on a similarity between a direction connecting two position coordinate points and the direction of the vector and a similarity between a length between two position coordinate points and a length of the second vector.

19. The storage medium according to claim 17, wherein the program causes the computer to further execute:

a first determination step of determining whether or not the object position coordinate data has been stored into the memory by selections performed from a predetermined number of times before until the previous time, before the reference is set in the reference setting step,

wherein, in the reference setting step, the computer calculates or obtains the second vector from most recently stored object position coordinate data only when a result of the determination by the first determination step is positive.

20. The storage medium according to claim 17, wherein the selection step calculates a similarity from an inner product value of a vector connecting two position coordinate points from the plurality of position coordinate points, and the second vector.

21. The storage medium according to claim 12, wherein the program causes the computer to further execute:

an object position storing step of storing object position coordinate data including at least one of a set of the two object position coordinate points selected by the selection step, and a vector indicating a direction connecting the two object position coordinate points, every time selection is performed in the selection step; and

a second determination step of determining whether or not two object position coordinate points have been selected in selections performed in the selection step from a predetermined number of times before,

wherein position coordinate points to be selected in the selection step are sequentially detected from images sequentially captured by the image capturing apparatus,

in the reference setting step, the computer sets a first vector indicating a direction on the plane based on a tilt of the image capturing apparatus, using a direction connecting two position coordinate points on the plane as a reference, when a result of the determination in the second determination step is positive; and

calculates or obtains a second vector which is a vector indicating a direction connecting two object position coordinate points from object position coordinate data most recently stored into the memory when the result of the determination in the second determination step is negative, and

in the selection step, the computer determines a combination of two position coordinate points to be selected, based on a similarity between a direction connecting two position coordinate points and the direction of the first vector when the result of the determination in the second determination step is positive; and

determines a combination of two position coordinate points to be selected, based on a similarity between a direction connecting two position coordinate points and the direction of the second vector when the result of the determination in the second determination step is negative.

22. The storage medium according to claim 21, wherein the tilt determining means includes an acceleration sensor which is fixedly provided to a housing used in common with the image capturing apparatus, and detects accelerations in two axial directions perpendicular to an image capturing direction of the image capturing apparatus, and

in the reference setting step, the computer calculates the first vector based on the accelerations in the axial directions detected by the acceleration sensor.

23. A method of selecting two specific object-position coordinate points from among a plurality of object-position coordinate points corresponding to imaged objects in a digital image acquired by an image capturing apparatus, said object-position coordinate points being on a plane corresponding to the acquired image and being indicative of relative positions of imaged objects in the acquired image, comprising:

detecting one or more areas in an acquired digital image which satisfy a predetermined criterion and assigning each such area a relative object-position coordinate point designation within the plane of the acquired image;

setting as a reference value a direction of a line connecting object-position coordinate points corresponding to two of said areas which are initially detected in the acquired image or a length of said line;

using position coordinate points of two of said areas that are initially detected which satisfy said predetermined condition after a predetermined period of time, if no further areas are detected; and

,

whenever more than two of said areas are detected in the acquired image and assigned a relative object-position coordinate point designation, comparing the set reference value with at least one of either a direction of a line connecting two object-position coordinate points and a length of a line connecting the two object-position coordinate points for different pair combinations of object-position coordinate points out of possible different remaining combinations of pairs of assigned object-position coordinate points; and

selecting one pair of said possible different combinations of assigned object-position coordinate point pairs that has an associated line direction or line length that is closest in value to the set reference value.

24. A method according to claim 23 wherein the direction of said line connecting an object-position coordinate point pair corresponding to two of said areas which are first detected in the acquired image is used as the reference value, and wherein said selecting one pair of possible different pair combinations further comprises:

detecting a tilt of the image capturing apparatus with respect to gravity;

defining a first vector on the plane of the acquired image based upon a detected tilt of the image capturing apparatus; and

selecting a particular pair combination of assigned position coordinate points based on a degree of similarity between a direction of a line connecting that particular pair of position coordinate points and a direction of the first vector.

25. The method according to claim 24 wherein said degree of similarity is computed based on an inner product value of a vector connecting a pair of two position coordinate points and the first vector.

26. The method according to claim 24 wherein the first vector is determined based on a correspondence relationship between a tilt determined in a situation where the image capturing apparatus is at a predetermined tilt and a direction connecting the two object position coordinate points detected from an image captured by the image capturing apparatus in that situation.

27. The method according to claim 24 wherein the tilt of the image capturing apparatus is determined by using an acceleration sensor used in common with the image capturing apparatus, wherein said acceleration sensor detects accelerations in at least two axial directions perpendicular to an image capturing direction of the image capturing apparatus, and wherein the first vector is determined based on detected accelerations in the axial directions.

28. The method according to claim 23 further comprising:

storing, every time an object position coordinate point pair selection operation is performed, object position coordinate data in a data storage device including at least one of either a selected pair of object position coordinate points and a vector indicating a direction connecting the selected pair; and

determining, whenever an object position coordinate point pair is selected, a second vector that is indicative of a direction connecting two object position coordinate points from object position coordinate point pair data that was most recently stored and setting the second vector as the reference,

wherein object position coordinate points are successively assigned from successively captured images, and a pair combination of object position coordinate points is selected based on a degree of similarity between a direction connecting two object position coordinate points and the direction of the second vector.

29. The method according to claim 28 wherein a pair combination of two position coordinate points is selected based on a degree of similarity between a direction connecting two object position coordinate points and the direction of the vector and a similarity between a length between two object position coordinate points and a length of the second vector.

30. The method according to claim 28 further comprising:

determining whether object position coordinate data has been stored into the data storage device based on a predetermined number of previous selections prior to a last selection before the reference is set, and wherein when setting the reference the second vector is determined from a most recently stored object position coordinate data only when a result of a first determination is positive.

31. The method according to claim 28 wherein determining a degree of similarity is based on an inner product value of a vector connecting said two object position coordinate points and the second vector.

32. The method according to claim 23 further comprising:

storing object position coordinate data corresponding to at least one of a pair of selected object position coordinate points and a vector indicating a direction connecting the pair of object position coordinate points, said storing taking place each time a selection of a pair of said coordinate points is performed;

determining whether a pair of object position coordinate points were previously selected during a predetermined number of previous attempts, wherein position coordinate points are successively assigned from successively acquired images and wherein, during the setting of the reference, a first vector indicative of a direction on the plane based upon a tilt of the image capturing apparatus and a direction connecting two object position coordinate points is set as a reference when a result of said further determining is positive; and

determining a second vector indicating a direction connecting a pair of object position coordinate points from object position coordinate data most recently stored into the data storage device when a result of further determination is negative, wherein a particular pair combination of two position coordinate points is chosen for selection based on a degree of similarity between a direction connecting two position coordinate points and the direction of the first vector when the result of the determination is positive, and a combination of two position coordinate points is chosen for selection based on a similarity between a direction connecting a pair of position coordinate points and the direction of the second vector when a result of said determining is negative.

33. The method according to claim 32 wherein a tilt of the image capturing apparatus is determined by using an acceleration sensor which is fixedly provided to a housing used in common with the image capturing apparatus, and wherein said sensor detects accelerations in two axial directions perpendicular to an image capturing direction of the image capturing apparatus, and wherein the first vector is determined based on accelerations detected in the axial directions.

34. An apparatus for selecting specific object-position coordinate points from among a plurality of object-position coordinate points corresponding to imaged objects in an acquired digital image, said object-position coordinate points being on a plane corresponding to the acquired image and being indicative of relative positions of imaged objects in the acquired image, comprising:

object detection programmed logic circuitry configured to detect one or more areas in a digital image which satisfy a predetermined criterion and assign each such area a relative object-position coordinate point designation within the plane of the acquired image;

reference setting detection programmed logic circuitry configured to set as a reference value at least one or both of a direction of a line connecting assigned object-position coordinate points corresponding to two of said areas which are first detected in the acquired image and a length of said line; and

reference value comparison programmed logic circuitry configured to compare the set reference value, whenever more than two of said areas are detected in the acquired image and assigned a relative object-position coordinate point designation, with at least one of either a direction of a line connecting two object-position coordinate points and a length of a line connecting the two object-position coordinate points for different pair combinations of object-position coordinate points out of possible available combinations of pairs of assigned object-position coordinate points; and

object-position coordinate point pair selection programmed logic circuitry configured to select one pair of possible different combinations of object-position coordinate point pairs that has an associated line direction or line length that is closest in value to the set reference value.

35. In an apparatus according to claim 34 wherein the direction of said line connecting assigned object-position coordinate points corresponding to two of said areas which are first detected in the acquired image is used as the reference value, said apparatus further comprising:

one or more sensors for detecting a tilt with respect to gravity of an image capturing apparatus by which said digital image is acquired;

vector computing programmed logic circuitry configured to define a first vector on the plane of the acquired image based upon a detected tilt of the image capturing apparatus; and

coordinate points pair selection programmed logic circuitry configured to select a particular pair of assigned position coordinate points based on a degree of similarity between a direction connecting a selected pair of position coordinate points and a direction of the first vector.

36. The apparatus according to claim 35 wherein said degree of similarity is computed based on an inner product value of a vector connecting a pair of two position coordinate points and the first vector.

37. The apparatus according to claim 35 wherein the first vector is determined based on a correspondence relationship between a tilt determined in a situation where the image capturing apparatus is at a predetermined tilt and a direction connecting the two object position coordinate points detected from an image captured by the image capturing apparatus in that situation.

38. The apparatus according to claim 35 wherein the tilt of the image capturing apparatus is determined by using an acceleration sensor used in common with the image capturing apparatus, wherein said acceleration sensor detects accelerations in at least two axial directions perpendicular to an image capturing direction of the image capturing apparatus, and wherein the first vector is determined based on detected accelerations in the axial directions.

39. An apparatus according to claim 34 further comprising:

object position coordinate data storing programmed logic circuitry configured to store, every time an object position coordinate point pair selection operation is performed, object position coordinate data in a data storage device including at least one of either a selected pair of object position coordinate points and a vector indicating a direction connecting the selected pair; and

second vector determining programmed logic circuitry configured to determine, whenever an object position coordinate point pair is selected, a second vector that is indicative of a direction connecting two object position coordinate points from object position coordinate point pair data that was most recently stored and setting the second vector as the reference,

wherein object position coordinate points are successively assigned from successively captured images, and a pair combination of object position coordinate points is selected based on a degree of similarity between a direction connecting two object position coordinate points and the direction of the second vector.

40. The apparatus according to claim 39 wherein a pair combination of two position coordinate points is selected based on a degree of similarity between a direction connecting two object position coordinate points and the direction of the vector and a similarity between a length between two object position coordinate points and a length of the second vector.

41. The apparatus according to claim 39 further comprising:

object position coordinate data programmed logic circuitry configured to determine whether object position coordinate data has been stored into the data storage device based on a predetermined number of previous selections prior to a last selection before the reference is set, and wherein when setting the reference the second vector is determined from a most recently stored object position coordinate data only when a result of a first determination is positive.

42. The apparatus according to claim 39 wherein determining a degree of similarity is based on an inner product value of a vector connecting said two object position coordinate points and the second vector.

43. An apparatus according to claim 34 further comprising:

object position coordinate data storing programmed logic circuitry configured to store object position coordinate data corresponding to at least one of a pair of selected object position coordinate points and a vector indicating a direction connecting the pair of object position coordinate points, wherein the storing takes place each time a selection of a pair of said coordinate points is performed;

object position coordinate point pair data programmed logic circuitry configured to further determine whether a pair of object position coordinate points were previously selected during a predetermined number of previous attempts, wherein position coordinate points are successively assigned from successively acquired images and wherein, during the setting of the reference, a first vector is set indicating a direction on the plane based upon a tilt of the image capturing apparatus and a direction connecting two object position coordinate points on the plane is set as a reference when a result of the determination is positive; and

second vector determining programmed logic circuitry configured to determine a second vector indicating a direction connecting a pair of object position coordinate points from object position coordinate data most recently stored into the data storage device when a result of further determination is negative, wherein a particular pair combination of two position coordinate points is chosen for selection based on a degree of similarity between a direction connecting two position coordinate points and the direction of the first vector when the result of the determination is positive, and a combination of two position coordinate points is chosen for selection based on a similarity between a direction connecting a pair of position coordinate points and the direction of the second vector when the result of the determination is negative.

44. An apparatus according to claim 43 wherein a tilt of the image capturing apparatus is determined by using an acceleration sensor which is fixedly provided to a housing used in common with the image capturing apparatus, and wherein said sensor detects accelerations in two axial directions perpendicular to an image capturing direction of the image capturing apparatus, and wherein the first vector is determined based on accelerations detected in the axial directions.