Computer-readable storage medium, image processing apparatus, image processing system, and image processing method

• US 8,699,749 B2
• Filed: 07/29/2011
• Issued: 04/15/2014
• Est. Priority Date: 02/24/2011
• Status: Active Grant

• Current Assignee: Nintendo Company Limited
• Original Assignee: Nintendo Company Limited
• Inventors: Osako, Satoru
• Primary Examiner(s): Abdi, Amara
• Application Number:
  US13/193,876
• Publication Number:
  US 20120219177A1
• Time in Patent Office: 991 Days
• Field of Search: 345/419, 345/475, 382/103, 382/182, 382/197, 382/199
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/181 involving edge growing; involving edge linking
• CPC Class Codes: G06T 7/75 involving models

Abstract

First, a series of edge pixels representing a contour of an object or of a design represented in the object are detected from an image acquired from a capturing apparatus. Then, a plurality of straight lines are generated on the basis of the series of detected edge pixels, and vertices of the contour are detected on the basis of the plurality of straight lines. Further, relative positions and orientations of the capturing apparatus and the object relative to each other are calculated on the basis of the detected vertices, and a virtual camera in a virtual space is set on the basis of the positions and the orientations. Then, a virtual space image obtained by capturing the virtual space with the virtual camera is displayed on a display device.

Claims

1. A non-transitory computer-readable storage medium having stored thereon an image processing program which, when executed by a computer of an information processing apparatus, causes the computer to perform operations comprising:

acquiring an image from an imaging device;

detecting from the image a series of edge pixels representing a contour of an object or of a design represented in the object;

generating a plurality of straight lines on the basis of the series of edge pixels;

detecting vertices of the contour on the basis of the plurality of straight lines;

calculating positions and orientations of the imaging device and the object relative to each other on the basis of the vertices;

setting a virtual camera in a virtual space on the basis of the calculated positions and the calculated orientations; and

displaying on a display device a virtual space image obtained by capturing the virtual space with the set virtual camera,

wherein the plurality of straight lines are generated by repeatedly generating or updating a straight line clockwise or counterclockwise starting at any one of the series of edge pixels.

2. The non-transitory computer-readable storage medium according to claim 1, wherein

the generating a plurality of straight lines includes integrating a plurality of straight lines separated from one another on the same straight line into one straight line.

3. The non-transitory computer-readable storage medium according to claim 1, wherein

the generating a plurality of straight lines includes:

first determining whether or not some edge pixels among the series of edge pixels are located on the same straight line; and

calculating a straight line on the basis of the some edge pixels that are located on the same straight line based on the first determining.

4. The non-transitory computer-readable storage medium according to claim 3, wherein

the calculating a straight line includes calculating a straight line by a least squares method on the basis of a plurality of edge pixels that are located on the same straight line based on the first determining.

5. The computer-readable storage medium according to claim 1, wherein

the generating a plurality of straight lines includes determining whether or not it is possible to perform the process of generating or updating a straight line clockwise or counterclockwise, wherein

when it is not possible to perform the process of generating or updating a straight line, a process is repeatedly performed of, in a direction opposite to the clockwise or counterclockwise direction in which the process of generating or updating a straight line has been performed, generating or updating a straight line.

6. The non-transitory computer-readable storage medium according to claim 1, wherein

the generating a plurality of straight lines includes:

generating a first straight line on the basis of the edge pixel of the starting point and a predetermined number of edge pixels continuous with the edge pixel of the starting point in either one of the clockwise and counterclockwise directions;

comparing a direction of the straight line that has already been generated or updated with a direction of a provisional straight line on the basis of the predetermined number of edge pixels continuous with the straight line in the one of the clockwise and counterclockwise directions;

when the comparing determines that the straight line and the provisional straight line are located on the same straight line, updating the straight line on the basis of a plurality of edge pixels corresponding to the straight line and the provisional straight line;

when the comparing determines that the straight line and the provisional straight line are not located on the same straight line and that the vertex of the contour that corresponds to an intersection of the straight line and the provisional straight line has a convex angle, generating a new straight line on the basis of the provisional straight line; and

when the comparing determines that the straight line and the provisional straight line are not located on the same straight line and that the vertex of the contour that corresponds to the intersection of the straight line and the provisional straight line has a concave angle, starting a process of generating or updating a straight line from the edge pixel of the starting pixel in the other one of the clockwise and counterclockwise directions.

7. The non-transitory computer-readable storage medium according to claim 1, the image processing program causing the computer to perform operations further comprising:

selecting a predetermined number of straight lines from among the generated plurality of straight lines, wherein

the detecting vertices of the contour includes detecting, as the vertices of the contour, intersections of the selected predetermined number of straight lines.

8. The non-transitory computer-readable storage medium according to claim 7, wherein

the selecting the predetermined number of straight lines includes selecting the predetermined number of straight lines having longer lengths from among the generated plurality of straight lines.

9. The non-transitory computer-readable storage medium according to claim 1, wherein

the detecting vertices of the contour includes detecting intersections of the plurality of straight lines as the vertices of the contour.

10. The non-transitory computer-readable storage medium according to claim 1, wherein

repeatedly acquiring images from the imaging device, and

detecting the series of edge pixels from the repeatedly acquired images.

11. An image processing apparatus comprising:

a memory; and

a processing system communicatively coupled to the memory and comprising at least one processor configured to perform operations comprising;

acquiring an image from an imaging device;

detecting from the image a series of edge pixels representing at least one of a contour of an object or a contour of a design represented in the object;

generating a plurality of straight lines on the basis of the series of edge pixels;

detecting vertices of the contour of the object or the contour of the design represented in the object on the basis of the plurality of straight lines;

calculating aspects of positions and orientations of the imaging device and the object relative to each other on the basis of the vertices;

setting a virtual camera in a virtual space on the basis of the calculated aspects of positions and the calculated aspects of orientations; and

displaying on a display device a virtual space image obtained by capturing the virtual space with the set virtual camera,

wherein the plurality of straight lines are generated by repeatedly generating or updating a straight line clockwise or counterclockwise starting at any one of the series of edge pixels.

15. The image processing apparatus according to claim 11, wherein the generating a plurality of straight lines includes:

first determining whether or not some edge pixels among the series of edge pixels are located on the same straight line; and

calculating a straight line on the basis of the some edge pixels that are located on the same straight line based on the first determining.

16. The processing apparatus according to claim 11, wherein the generating a plurality of straight lines includes:

generating a first straight line on the basis of the edge pixel of the starting point and a predetermined number of edge pixels continuous with the edge pixel of the starting point in either one of the clockwise and counterclockwise directions;

comparing a direction of the straight line that has already been generated or updated with a direction of a provisional straight line on the basis of the predetermined number of edge pixels continuous with the straight line in the one of the clockwise and counterclockwise directions;

when the comparing determines that the straight line and the provisional straight line are located on the same straight line, updating the straight line on the basis of a plurality of edge pixels corresponding to the straight line and the provisional straight line;

when the comparing determines that the straight line and the provisional straight line are not located on the same straight line and that the vertex of the contour that corresponds to an intersection of the straight line and the provisional straight line has a convex angle, generating a new straight line on the basis of the provisional straight line; and

when the comparing determines that the straight line and the provisional straight line are not located on the same straight line and that the vertex of the contour that corresponds to the intersection of the straight line and the provisional straight line has a concave angle, starting a process of generating or updating a straight line from the edge pixel of the starting pixel in the other one of the clockwise and counterclockwise directions.

17. The image processing apparatus according to claim 11, the operations further comprising:

selecting a predetermined number of straight lines from among the generated plurality of straight lines, wherein

the detecting vertices of the contour includes detecting, as the vertices of the contour, intersections of the selected predetermined number of straight lines.

12. An image processing method comprising:

acquiring an image from an imaging device;

detecting from the image a series of edge pixels representing one or both of a contour of an object or a contour of a design represented in the object;

generating a plurality of straight lines on the basis of the series of edge pixels;

detecting vertices of the contour of the object or of the contour of the design represented in the object on the basis of the plurality of straight lines;

calculating aspects of positions and orientations of the capturing means and the object relative to each other on the basis of the vertices;

setting a virtual camera in a virtual space on the basis of the calculated aspects of positions and the calculated aspects of orientations; and

displaying on a display device a virtual space image obtained by capturing the virtual space with the set virtual camera,

wherein the plurality of straight lines are generated by repeatedly generating or updating a straight line clockwise or counterclockwise starting at any one of the series of edge pixels.

18. The image processing method according to claim 12, wherein the generating a plurality of straight lines includes:

first determining whether or not some edge pixels among the series of edge pixels are located on the same straight line; and

calculating a straight line on the basis of the some edge pixels that are located on the same straight line based on the first determining.

19. The processing method according to claim 12, wherein the generating a plurality of straight lines includes:

generating a first straight line on the basis of the edge pixel of the starting point and a predetermined number of edge pixels continuous with the edge pixel of the starting point in either one of the clockwise and counterclockwise directions;

comparing a direction of the straight line that has already been generated or updated with a direction of a provisional straight line on the basis of the predetermined number of edge pixels continuous with the straight line in the one of the clockwise and counterclockwise directions;

when the comparing determines that the straight line and the provisional straight line are located on the same straight line, updating the straight line on the basis of a plurality of edge pixels corresponding to the straight line and the provisional straight line;

when the comparing determines that the straight line and the provisional straight line are not located on the same straight line and that the vertex of the contour that corresponds to an intersection of the straight line and the provisional straight line has a convex angle, generating a new straight line on the basis of the provisional straight line; and

when the comparing determines that the straight line and the provisional straight line are not located on the same straight line and that the vertex of the contour that corresponds to the intersection of the straight line and the provisional straight line has a concave angle, starting a process of generating or updating a straight line from the edge pixel of the starting pixel in the other one of the clockwise and counterclockwise directions.

20. The image processing method according to claim 12, the method further comprising:

selecting a number of straight lines from among the generated plurality of straight lines, wherein

the detecting vertices of the contour includes detecting, as the vertices of the contour, intersections of the selected number of straight lines.

13. An image processing system comprising:

an imaging device;

a memory;

a processing system including at least one processor, the processing system communicatively coupled to the imaging device and the memory, the processing system configured to perform operations comprising;

acquiring an image from the imaging device;

detecting from the image a series of edge pixels representing a contour of an object or of a design represented in the object;

generating a plurality of straight lines on the basis of the series of edge pixels;

detecting vertices of the contour on the basis of the plurality of straight lines;

calculating positions and orientations of the imaging device and the object relative to each other on the basis of the vertices;

setting a virtual camera in a virtual space on the basis of the calculated positions and the calculated orientations; and

displaying on a display device a virtual space image obtained by capturing the virtual space with the set virtual camera,

wherein the plurality of straight lines are generated by repeatedly generating or updating a straight line clockwise or counterclockwise starting at any one of the series of edge pixels.

14. An image processing system including an image processing apparatus and a marker, the image processing apparatus comprising:

a capturing section for capturing the marker;

a processing system comprising at least one processor, the processing system communicatively coupled to the capturing section and configured to perform operations comprising;

acquiring an image from the capturing section;

detecting from the image a series of edge pixels representing a contour of the marker or of a design represented in the marker;

generating a plurality of straight lines on the basis of the series of edge pixels;

detecting vertices of the contour on the basis of the plurality of straight lines;

calculating positions and orientations of the capturing section and the marker relative to each other on the basis of the vertices;

setting a virtual camera in a virtual space on the basis of the calculated positions and the calculated orientations; and

displaying on a display device a virtual space image obtained by capturing the virtual space with the set virtual camera,

wherein the plurality of straight lines are generated by repeatedly generating or updating a straight line clockwise or counterclockwise starting at any one of the series of edge pixels.