Image processing device, and image procesing method

• US 9,865,068 B2
• Filed: 08/26/2011
• Issued: 01/09/2018
• Est. Priority Date: 09/06/2010
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Tsurumi, Shingo, Mochizuki, Shunsuke, Maekawa, Munechika
• Primary Examiner(s): Mazumder, Saptarshi
• Application Number:
  US13/219,062
• Publication Number:
  US 20120057794A1
• Time in Patent Office: 2,328 Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 5/00 Image enhancement or restoration

Abstract

There is provided an image processing device including a recognition unit configured to recognize a plurality of users being present in an input image captured by an imaging device, an information acquisition unit configured to acquire display information to be displayed in association with each user recognized by the recognition unit, and an output image generation unit configured to generate an output image by overlaying the display information acquired by the information acquisition unit on the input image. The output image generation unit may determine which of first display information associated with a first user and second display information associated with a second user is to be overlaid on a front side on the basis of a parameter corresponding to a distance of each user from the imaging device.

Claims

1. An image processing device, comprising:

a recognition unit configured to recognize a plurality of registered users present in an input image captured by an imaging device;

an information acquisition unit configured to acquire display information to be displayed in association with each registered user of the plurality of registered users recognized by the recognition unit; and

an output image generation unit configured to generate an output image by overlaying the display information acquired by the information acquisition unit on the input image,

wherein the output image generation unit overlays first display information associated with a first registered user on a front side of second display information associated with a second registered user if a distance of the first registered user from the imaging device is shorter than a distance of the second registered user from the imaging device,

wherein the recognition unit further recognizes a gesture of the second registered user, and

wherein the output image generation unit temporarily moves the display information of the second registered user to a front side of the display information of the first registered user based on the recognition if the gesture.

2. The image processing device according to claim 1, wherein the output image generation unit sets a transparency of each piece of the display information overlaid on each other.

3. The image processing device according to claim 1, wherein

the recognition unit further recognizes a size of a face area of each registered user present on the input image, and

the output image generation unit uses the size of the face area of each registered user to determine the distance of each of the first registered user and the second registered user.

4. The image processing device according to claim 2, wherein

the output image generation unit measures a length of time for which each registered user recognized by the recognition unit is present in the input image or a moving speed of each registered user, and

the output image generation unit sets the transparency of the display information in accordance with the length of time or the moving speed measured for each registered user associated with the display information.

5. The image processing device according to claim 4, wherein the output image generation unit sets the transparency of the display information of the plurality of registered users present in the input image for a longer time to a lower level.

6. The image processing device according to claim 4, wherein the output image generation unit sets the transparency of the display information of a registered user whose moving speed is lower to a lower level.

7. The image processing device according to claim 4, wherein

the recognition unit further recognizes a gesture of each registered user, and

the output image generation unit temporarily reduces the transparency of the display information of a registered user who has made a predetermined gesture.

8. The image processing device according to claim 1, wherein

the recognition unit further recognizes a facial expression or a speaking state of each registered user, and

the output image generation unit temporarily displays display information associated with a registered user who is having a predetermined facial expression or with a registered user who is speaking on the front side regardless of the distance of the registered user from the imaging device.

9. The image processing device according to claim 1 wherein the output image generation unit determines a display size of the display information associated with each registered user in accordance with the distance of each registered user from the imaging device.

10. The image processing device according to claim 1, wherein

the information acquisition unit acquires text input via a text input device as first type display information and acquires a voice input via a voice input device as second type display information, and

the output image generation unit sets a shape of a first object for displaying the first type display information to a first shape representing a thought, and sets a shape of a second object for displaying the second type display information to a second shape representing a speech.

11. The image processing device according to claim 1, wherein

the information acquisition unit acquires information input via a text input device or a voice input device as the display information,

the output image generation unit analyzes the display information acquired by the information acquisition unit to determine whether the display information is third type display information corresponding to a thought of a registered user or fourth type display information corresponding to a speech of the registered user, and

the output image generation unit sets a shape of a third object for displaying the third type display information to a third shape representing the thought, and sets a shape of fourth object for displaying the fourth type display information to a fourth shape representing the speech.

12. The image processing device according to claim 1, wherein

the information acquisition unit acquires information input by a registered user as fifth type display information, and acquires information acquired from an external information source based on information input by the registered user or attribute information of the registered user as sixth type display information, and

the output image generation unit displays the fifth type display information and the sixth type display information using objects with different shapes.

13. A non-transitory, computer-readable medium having stored thereon, computer-executable instructions, which when executed by processor, cause an image processing device to execute operations, the operations comprising:

recognizing, by a recognition unit, a plurality of registered users present in an input image captured by an imaging device;

acquiring, by an information acquisition unit, display information to be displayed in association with each registered user of the plurality of registered users recognized by the recognition unit; and

generating, by an output image generation unit, an output image by overlaying the display information acquired by the information acquisition unit on the input image,

wherein the output image generation unit overlays first display information associated with a first registered user on a front side of second display information associated with a second registered user if a distance of the first registered user from the imaging device is shorter than a distance of the second registered user from the imaging device,

wherein the recognition unit further recognizes a gesture of the second registered user, and

wherein the output image generation unit temporarily moves the display information of the second registered user to a front side the display information of the first registered user based on the recognition of the gesture.

14. The non-transitory, computer-readable medium according to claim 13, wherein the output image generation unit sets a transparency of each piece of the display information overlaid on each other.

15. The non-transitory, computer-readable medium according to claim 13, wherein

the recognition unit further recognizes a size of a face area of each registered user being present in the input image, and

the output image generation unit uses the size of the face area if each registered user to determine the distance of each of the first registered user and the second registered user.

16. The non-transitory, computer-readable medium according to claim 14, wherein

the output image generation unit measures a length of time for which each registered user recognized by the recognition unit is present in the input image or a moving speed of each registered user, and

the output image generation unit sets the transparency of the display information in accordance with the length of time or the moving speed measured for the registered user associated with the display information.

17. The non-transitory, computer-readable medium storing a program according to claim 16, wherein the output image generation unit sets the transparency of the display information of a registered user who is present, in the input image for a longer time to a lower level.

18. The non-transitory, computer-readable medium storing a program according to claim 16, wherein the output image generation unit sets the transparency of the display information of a registered user whose moving speed is lower to a lower level.

19. An image processing method, comprising:

recognizing a plurality of registered users present in an input image captured by an imaging device;

acquiring display information to be displayed in association with each recognized registered user of the plurality of registered users;

determining which of first display information associated with a first registered user and second display information associated with a second registered user is to be overlaid on a front side based on a parameter corresponding to a distance of each registered user from the imaging device;

generating an output image by overlaying the acquired display information on the input image;

overlaying the first display information associated with the first registered user on a front side of the second display information associated with the second registered user if the distance of the first registered user from the imaging devise is shorter than a distance of the second registered user from the imaging device;

recognizing a gesture of the second registered user; and

temporarily moving the display information of the second registered user to a front side of the display information of the first registered user based on the recognition of the gesture.

20. The image processing method according to claim 19, further comprising:

setting a transparency of each piece of the display information overlaid on each other.