Control of display of composite image based on depth information

• US 10,540,805 B2
• Filed: 06/27/2016
• Issued: 01/21/2020
• Est. Priority Date: 12/02/2011
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Shionozaki, Atsushi, Rekimoto, Junichi, Goto, Hiroshi
• Primary Examiner(s): Entezari, Michelle M
• Application Number:
  US15/193,388
• Publication Number:
  US 20160307361A1
• Time in Patent Office: 1,303 Days
• Field of Search: None
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 15/503 Blending, e.g. for anti-aliasing
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30196 Human being; Person

Abstract

There is provided an image processing device including a depth acquisition unit configured to acquire depth information of an object included in a captured image, an image combining unit configured to combine the object with an image object having depth information according to each piece of the depth information, and an effect processing unit configured to execute effect processing according to the depth information of the object.

Claims

1. An image-processing system, comprising:

a sensor configured to detect a real position of a first living object, wherein the real position is associated with a display device in a depth direction in a real space; and

circuitry configured to;

acquire a first living-object image representing the first living object and a second living-object image representing a second living object different from the first living object, wherein

the second living-object image is a foreground image,

first depth information of the first living-object image is based on the detected real position, and

second depth information of the second living-object image corresponds to the second living object located in a virtual space;

control the display device for display of a composite image of the first living-object image, the second living-object image, and a background image that is in the virtual space;

compare the first depth information and the second depth information to determine that a depth distance between the first living-object image and the second living-object image is larger than a threshold distance in the virtual space;

set the threshold distance based on a depth of the second living-object image;

set the threshold distance based on a depth of the foreground image in the composite image, wherein

a second value of the threshold distance for a second depth of the foreground image in the composite image is larger than a first value of the threshold distance for a first depth of the foreground image in the composite image, and

the second depth is deeper than the first depth; and

control the display device to;

display an effect image based on the determination that the depth distance is larger than the threshold distance in the virtual space; and

change the first depth information based on a movement of the first living object.

2. The image-processing system according to claim 1, wherein the effect image includes a first effect image that extends towards the second living-object image.

3. The image-processing system according to claim 2, wherein the first effect image has an arrow shape that points at the second living-object image.

4. The image-processing system according to claim 2, wherein

the effect image further includes a second effect image,

the first effect image extends from the first living-object image towards the second living-object image in a direction perpendicular to the depth direction, and

the second effect image includes a message image associated with an annotation.

5. The image-processing system according to claim 1, wherein the first living-object image and the second living-object image correspond to humans located in different spaces.

6. The image-processing system according to claim 1, wherein the circuitry is further configured to control the display device to concurrently display the composite image and the effect image.

7. The image-processing system according to claim 6, wherein the sensor comprises an imaging device configured to:

capture a sample image of the real space; and

determine the first depth information based on the captured sample image.

8. The image-processing system according to claim 1, wherein the effect image comprises an alert image associated with an annotation.

11. The image-processing system according to claim 1, wherein the effect image corresponds to at least one of a flash, contour emphasis, a hue change, or a blur of the first living-object image.

12. The image-processing system according to claim 1, wherein each of the first living object and the second living object is a human, a plant or an animal not including the human.

9. An image-processing method, comprising:

detecting, using a depth sensor, a real position of a first living object, wherein the real position is associated with a display device in a depth direction in a real space;

acquiring a first living-object image representing the first living object and a second living-object image representing a second living object different from the first living object, wherein

the second living-object image is a foreground image,

first depth information of the first living-object image is based on the detected real position, and

second depth information of the second living-object image corresponds to the second living object located in a virtual space;

controlling the display device for display of a composite image of the first living-object image, the second living-object image, and a background image that is in the virtual space;

comparing the first depth information and the second depth information to determine that a depth distance between the first living-object image and the second living-object image is larger than a threshold distance in the virtual space;

setting the threshold distance based on a depth of the second living-object image;

setting the threshold distance based on a depth of the foreground image in the composite image, wherein

a second value of the threshold distance for a second depth of the foreground image in the composite image is larger than a first value of the threshold distance for a first depth of the foreground image in the composite image, and

the second depth is deeper than the first depth; and

controlling the display device to;

display an effect image based on the determination that the depth distance is larger than the threshold distance in the virtual space; and

change the first depth information based on a movement of the first living object.

10. A non-transitory computer-readable medium having stored thereon, computer-executable instructions which, when executed by a computer, cause the computer to execute operations, the operations comprising:

detecting a real position of a first living object, wherein the real position is associated with a display device in a depth direction in a real space;

acquiring a first living-object image representing the first living object and a second living-object image representing a second living object different from the first living object,

wherein

the second living-object image is a foreground image,

first depth information of the first living-object image is based on the detected real position, and

second depth information of the second living-object image corresponds to the second living object located in a virtual space;

controlling the display device for display of a composite image of the first living-object image, the second living-object image, and a background image that is in the virtual space;

comparing the first depth information and the second depth information to determine a depth distance between the first living-object image and the second living-object image is larger than a threshold distance in the virtual space;

setting the threshold distance based on a depth of the second living-object image;

setting the threshold distance based on a depth of the foreground image in the composite image, wherein

a second value of the threshold distance for a second depth of the foreground image in the composite image is larger than a first value of the threshold distance for a first depth of the foreground image in the composite image, and the

second depth is deeper than the first depth; and

controlling the display device to;

display an effect image based on the determination that the depth distance is larger than the threshold distance in the virtual space; and

change the first depth information based on a movement of the first living object.