Image processing apparatus, image processing method, and program

• US 9,626,806 B2
• Filed: 01/13/2016
• Issued: 04/18/2017
• Est. Priority Date: 10/27/2011
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Kaino, Akihiko, Fukuchi, Masaki, Kashitani, Tatsuki, Ooi, Kenichiro, Guo, Jingjing
• Primary Examiner(s): Thompson, James A
• Application Number:
  US14/994,950
• Publication Number:
  US 20160133058A1
• Time in Patent Office: 461 Days
• Field of Search: None
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2219/004 Annotating, labelling
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

An information processing system that acquires video data captured by an image pickup unit; detects an object from the video data; detects a condition corresponding to the image pickup unit; and controls a display to display content associated with the object at a position other than a detected position of the object based on the condition corresponding to the image pickup unit.

Claims

1. An information processing system comprising:

circuitry configured to;

acquire video data of a real space captured by an image pickup unit;

detect an object in the real space corresponding to the video data;

determine a position of a virtual object associated with the detected object in a virtual space based on a three-dimensional structure of a peripheral environment of the image pickup unit;

acquire stored content information of the virtual object associated with the detected object from a memory based on at least one of a distance and an angle between the virtual object and the image pickup unit in the virtual space; and

control display of the stored content information of the virtual object such that the stored content information of the virtual object continues to be displayable at the determined position of the virtual object when the associated object is partially observable or no longer observable in the video data,

wherein the stored content information of the virtual object includes first modeling data and second modeling data that are stored in the memory prior to detecting the object and that differ according to at least one of the distance and the angle between the virtual object and the image pickup unit in the virtual space, the first modeling data and the second modeling data defining two separate models of the virtual object, respectively.

2. The information processing system of claim 1, wherein the circuitry is configured to

acquire the first modeling data of the virtual object when the distance between the virtual object and the image pickup unit corresponds to a first distance, and

acquire the second modeling data of the virtual object when the distance between the virtual object and the image pickup unit corresponds to a second distance.

3. The information processing system of claim 2, wherein the first modeling data corresponds to a near situation, and the second modeling data corresponds to a far situation.

4. The information processing system of claim 3, wherein the first modeling data and the second modeling data have different display resolutions.

5. The information processing system of claim 1, wherein the circuitry is configured to

acquire the first modeling data of the virtual object when the angle between the virtual object and the image pickup unit corresponds to a first angle, and

acquire the second modeling data of the virtual object when the angle between the virtual object and the image pickup unit corresponds to a second angle.

6. The information processing system of claim 1, wherein the circuitry is further configured to download the content information of the virtual object from an exterior server to the memory.

7. The information processing system of claim 1, wherein the circuitry is further configured to acquire one or more attributes of the virtual object, wherein the one or more attributes of the virtual object include a type of the virtual object, a type of associated application, or a type of marker corresponding to the virtual object.

8. The information processing system of claim 1, wherein the circuitry is further configured to stop the display of the virtual object when a distance between the detected object and the image pickup unit exceeds a predetermined threshold.

9. The information processing system of claim 1, wherein the circuitry is further configured to control transparency of the virtual object based on the at least one of the distance and the angle between the virtual object and the image pickup unit.

10. The information processing system of claim 1, wherein the circuitry is further configured to select the virtual object from a virtual object database based on the detected object.

11. An information processing method performed by an information processing system, the method comprising:

acquiring video data of a real space captured by an image pickup unit;

detecting an object in the real space corresponding to the video data;

determining a position of a virtual object associated with the detected object in a virtual space based on a three-dimensional structure of a peripheral environment of the image pickup unit;

acquiring stored content information of the virtual object associated with the detected object from a memory based on at least one of a distance and an angle between the virtual object and the image pickup unit in the virtual space; and

controlling display of the stored content information of the virtual object such that the stored content information of the virtual object continues to be displayable at the determined position of the virtual object when the associated object is partially observable or no longer observable in the video data,

wherein the stored content information of the virtual object includes first modeling data and second modeling data that are stored in the memory prior to detecting the object and that differ according to at least one of the distance and the angle between the virtual object and the image pickup unit in the virtual space, the first modeling data and the second modeling data defining two separate models of the virtual object, respectively.

12. The information processing method of claim 11, wherein the acquiring the stored content information of the virtual object comprises:

acquiring the first modeling data of the virtual object when the distance between the virtual object and the image pickup unit corresponds to a first distance, and

acquiring the second modeling data of the virtual object when the distance between the virtual object and the image pickup unit corresponds to a second distance.

13. The information processing method of claim 12, wherein the first modeling data corresponds to a near situation, and the second modeling data corresponds to a far situation.

14. The information processing method of claim 13, wherein the first modeling data and the second modeling data have different display resolutions.

15. The information processing method of claim 11, wherein the acquiring the stored content information of the virtual object comprises:

acquiring the first modeling data of the virtual object when the angle between the virtual object and the image pickup unit corresponds to a first angle, and

acquiring the second modeling data of the virtual object when the angle between the virtual object and the image pickup unit corresponds to a second angle.

16. A non-transitory computer-readable medium including computer program instructions, which when executed by an information processing system, cause the information processing system to perform a method, the method comprising:

acquiring video data of a real space captured by an image pickup unit;

detecting an object in the real space corresponding to the video data;

determining a position of a virtual object associated with the detected object in a virtual space based on a three-dimensional structure of a peripheral environment of the image pickup unit;

acquiring stored content information of the virtual object associated with the detected object from a memory based on at least one of a distance and an angle between the virtual object and the image pickup unit in the virtual space; and

controlling display of the stored content information of the virtual object such that the stored content information of the virtual object continues to be displayable at the determined position of the virtual object when the associated object is partially observable or no longer observable in the video data;

wherein the stored content information of the virtual object includes first modeling data and second modeling data that are stored in the memory prior to detecting the object and that differ according to at least one of the distance and the angle between the virtual object and the image pickup unit in the virtual space, the first modeling data and the second modeling data defining two separate models of the virtual object, respectively.

17. The non-transitory computer-readable medium of claim 16, wherein the acquiring the stored content information of the virtual object comprises:

acquiring the first modeling data of the virtual object when the distance between the virtual object and the image pickup unit corresponds to a first distance, and

acquiring the second modeling data of the virtual object when the distance between the virtual object and the image pickup unit corresponds to a second distance.

18. The non-transitory computer-readable medium of claim 17, wherein the first modeling data corresponds to a near situation, and the second modeling data corresponds to a far situation.

19. The non-transitory computer-readable medium of claim 18, wherein the first modeling data and the second modeling data have different display resolutions.

20. The non-transitory computer-readable medium of claim 16, wherein the acquiring the stored content information of the virtual object comprises:

acquiring the first modeling data of the virtual object when the angle between the virtual object and the image pickup unit corresponds to a first angle, and

acquiring the second modeling data of the virtual object when the angle between the virtual object and the image pickup unit corresponds to a second angle.