Three-dimensional (3D) video for two-dimensional (2D) video messenger applications

• US 20110109725A1
• Filed: 03/16/2010
• Published: 05/12/2011
• Est. Priority Date: 11/06/2009
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Hong, Seungwook, Yu, Yang
• Granted Patent: US 8,687,046 B2
• Time in Patent Office: Days
• US Class Current: 348/47
• CPC Class Codes: H04L 12/1827 Network arrangements for conference optimisation or adaptation
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/443 OS processes, e.g. booting an STB, implementing a Java virtual machine in an STB or power management in an STB
• CPC Class Codes: H04N 21/4788 communicating with other users, e.g. chatting arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q

Abstract

Three-dimensional (3D) video is provided to a legacy two-dimensional (2D) video messenger application. A request for video content from a single video camera is received at a camera interface module from the 2D video messenger application. The request for video content from the single video camera is forwarded to a video processing offload module. A first video camera and a second video camera are controlled. Merged 3D video content from the video processing offload module is received. The merged 3D video content is sent to the 2D video messenger application in response to receipt of the merged 3D video content from the video processing offload module. This abstract is not to be considered limiting, since other embodiments may deviate from the features described in this abstract.

Claims

1. A method of providing three-dimensional (3D) video to a legacy two-dimensional (2D) video messenger application, comprising:

receiving, at a camera interface module from the 2D video messenger application, a request for video content from a single video camera;

forwarding the request for video content from the single video camera to a video processing offload module;

controlling a first video camera and a second video camera;

receiving merged 3D video content from the video processing offload module; and

sending the merged 3D video content to the 2D video messenger application in response to receipt of the merged 3D video content from the video processing offload module.

2. The method according to claim 1, further comprising:

receiving the request for video content from the single video camera from the video processing offload module in response to forwarding the request to the video processing offload module; and

where controlling the first video camera and the second video camera comprises;

controlling the first video camera in response to receipt of the request for video content from the single video camera from the 2D video messenger application; and

controlling the second video camera in response to receipt of the request for video content from the single video camera from the video processing offload module.

3. The method according to claim 1, where controlling the first video camera and the second video camera comprises:

generating control signals to control video output from the first video camera; and

generating control signals to control video output from the second video camera.

4. The method according to claim 1, further comprising:

receiving first video content from the first video camera and second video content from the second video camera;

sending the first video content and the second video content to the video processing offload module; and

where receiving the merged 3D video content from the video processing offload module comprises receiving the merged 3D video content from the video processing offload module in response to sending the first video content and the second video content to the video processing offload module.

5. The method according to claim 1, further comprising:

receiving a de-queue request associated with an outgoing queue of the first video camera from the 2D video messenger application; and

where sending the merged 3D video content to the 2D video messenger application in response to receipt of the merged 3D video content from the video processing offload module comprises copying the merged 3D video content received from the video processing offload module to the outgoing queue of the first video camera.

6. The method according to claim 1, further comprising:

receiving, at the camera interface module from the 2D video messenger application, a request to open the single video camera;

generating control signals to open the first video camera;

forwarding the request to open the single video camera to the video processing offload module;

receiving the request to open the single video camera from the video processing offload module; and

generating control signals to open the second video camera in response to receipt of the request to open the single video camera from the video processing offload module.

7. The method according to claim 1, further comprising:

receiving, at the camera interface module from the 2D video messenger application, a request to close the single video camera;

generating control signals to close the first video camera;

forwarding the request to close the single video camera to the video processing offload module;

receiving the request to close the single video camera from the video processing offload module; and

generating control signals to close the second video camera in response to receipt of the request to close the single video camera from the video processing offload module.

8. The method according to claim 1, further comprising:

receiving, at the camera interface module from the 2D video messenger application, a request to set a format for output associated with the single video camera;

generating control signals to set the format for output associated with the single video camera to the first video camera;

forwarding the request to set the format for output associated with the single video camera to the video processing offload module;

receiving the request to set the format for output associated with the single video camera from the video processing offload module; and

generating control signals to set the format for output associated with the second video camera to the second video camera in response to receipt of the request to set the format for output associated with the single video camera from the video processing offload module.

9. The method according to claim 1, further comprising:

receiving, at the camera interface module from the 2D video messenger application, a request to set an operational parameter associated with the single video camera;

generating control signals to set the operational parameter associated with the first video camera;

forwarding the request to set the operational parameter associated with the single video camera to the video processing offload module;

receiving the request to set the operational parameter associated with the single video camera from the video processing offload module; and

generating control signals to set the operational parameter associated with the second video camera in response to receipt of the request to set the operation parameter associated with the single video camera from the video processing offload module.

10. The method according to claim 1, further comprising:

entering, at the camera interface module, a sleep state in response to forwarding the request for video content from the single video camera to the video processing offload module; and

entering, at the camera interface module, an active state in response to receiving the merged 3D video content from the video processing offload module.

11. The method according to claim 1, where the merged 3D video content comprises one of frame sequential 3D video content and polarized 3D video content.

12. A computer readable storage medium storing instructions which, when executed on one or more programmed processors, carry out the method according to claim 1.

13. A method of providing three-dimensional (3D) video to a legacy two-dimensional (2D) video messenger application, comprising:

receiving, at a video processing offload module from the 2D video messenger application via a camera interface module, a request for video content from a single video camera;

receiving first video content from a first video camera and second video content from a second video camera;

merging the first video content and the second video content into three-dimensional (3D) video content; and

sending the merged 3D video content to the 2D video messenger application via the camera interface module in response to the request for the video content from the single video camera.

14. The method according to claim 13, further comprising:

sending the request for video content from the single video camera to the camera interface module in response to receipt of the request for video content from the single video camera from the 2D video messenger application via the camera interface module; and

where the camera interface module controls at least one camera in response to receipt of the request for video content from the single video camera from the video processing module.

15. The method according to claim 13, further comprising performing a 3D calibration and 3D rectification of the first video content and the second video content in response to receiving the first video content from the first video camera and the second video content from the second video camera.

16. The method according to claim 15, further comprising:

receiving, at the video processing offload module, a de-queue request originated by the 2D video messenger application associated with an outgoing queue of the first video camera via the camera interface module;

determining whether the 3D calibration and the 3D rectification of the first video content and the second video content is completed;

sending the first video content from the first video camera to the 2D video messenger application via the camera interface module in response to determining that the 3D calibration and the 3D rectification of the first video content and the second video content is not completed; and

sending the merged 3D video content to the 2D video messenger application via the camera interface module in response to determining that the 3D calibration and the 3D rectification of the first video content and the second video content is completed.

17. The method according to claim 13, further comprising:

receiving, at the video processing offload module from the 2D video messenger application via the camera interface module, a request to open the single video camera; and

sending a request to open the second video camera to the camera interface module.

18. The method according to claim 13, further comprising:

receiving, at the video processing offload module from the 2D video messenger application via the camera interface module, a request to close the single video camera; and

sending a request to close the second video camera to the camera interface module.

19. The method according to claim 13, further comprising:

receiving, at the video processing offload module from the 2D video messenger application via the camera interface module, a request to set a format for output associated with the single video camera; and

sending a request to set a format for output associated with the second video camera to the camera interface module.

20. The method according to claim 13, further comprising:

receiving, at the video processing offload module from the 2D video messenger application via the camera interface module, a request to set an operational parameter associated with the single video camera; and

sending a request to set an operational parameter associated with the second video camera to the camera interface module.

21. The method according to claim 13, further comprising:

receiving, at the video processing offload module from the 2D video messenger application via the camera interface module, a request to allocate a number of frame buffers for the single video camera; and

sending a request to allocate the same number of frame buffers for the second video camera to the camera interface module.

22. The method according to claim 13, further comprising:

sending a request for an index to a frame buffer associated with each of the first video camera and the second video camera to the camera interface module;

receiving the index to the frame buffer associated with the first video camera and the index to the frame buffer associated with the second video camera from the camera interface module; and

mapping the index to the frame buffer associated with the first video camera and the index to the frame buffer associated with the second video camera to a memory associated with the video processing offload module.

23. The method according to claim 13, further comprising:

entering, at the video processing offload module, an active state in response to receiving, from the 2D video messenger application, the request for video content from the single video camera; and

entering, at the video processing offload module, a sleep state in response to starting sending the merged 3D video content to the 2D video messenger application via the camera interface module.

24. The method according to claim 13, further comprising:

receiving a request to perform a 3D calibration of the first video content received from the first video camera and the second video content received from the second video camera; and

performing the 3D calibration of the first video content received from the first video camera and the second video content received from the second video camera.

25. The method according to claim 24, where performing the 3D calibration of the first video content received from the first video camera and the second video content received from the second video camera comprises:

estimating a positional relationship between the first video camera and the second video camera; and

rectifying the first video content and the second video content.

26. The method according to claim 25, where rectifying the first video content and the second video content comprises:

obtaining warping matrixes; and

applying the warping matrixes to the first video content and the second video content.

27. The method according to claim 24, where performing the 3D calibration of the first video content received from the first video camera and the second video content received from the second video camera comprises:

detecting at least one of an edge and a corner of an image;

estimating a positional relationship between the first video camera and the second video camera; and

adjusting at least one of the first video content and the second video content based upon the estimated positional relationship between the first video camera and the second video camera.

28. The method according to claim 13, where merging the first video content and the second video content into the 3D video content comprises forming the 3D video content as one of side-by-side 3D video content, top-and-bottom 3D video content, and frame sequential 3D video content.

29. The method according to claim 13, further comprising:

receiving a request to change a portion of the merged 3D video content;

detecting the portion of the merged 3D video content within each of the first video content received from the first video camera and the second video content received from the second video camera; and

changing a portion of the first video content received from the first video camera and a portion of the second video content received from the second video camera.

30. The method according to claim 29, where:

the request to change the portion of the merged 3D video content comprises a request to smooth a facial feature of an image of a person represented within the merged 3D video content; and

changing the portion of the first video content received from the first video camera and the portion of the second video content received from the second video camera comprises;

smoothing the facial feature of the image of the person represented in the first video content received from the first video camera; and

smoothing the facial feature of the image of the person represented in the second video content received from the second video camera.

31. The method according to claim 29, where:

the request to change the portion of the merged 3D video content comprises a request to thin a body image of a person represented within the merged 3D video content; and

changing the portion of the first video content received from the first video camera and the portion of the second video content received from the second video camera comprises;

thinning the body image of the person represented in the first video content received from the first video camera; and

thinning the body image of the person represented in the second video content received from the second video camera.

32. The method according to claim 29, where:

the request to change the portion of the merged 3D video content comprises a request to stretch a portion of an image represented within the merged 3D video content; and

changing the portion of the first video content received from the first video camera and the portion of the second video content received from the second video camera comprises;

stretching the portion of the image represented in the first video content received from the first video camera; and

stretching the portion of the image represented in the second video content received from the second video camera.

33. A computer readable storage medium storing instructions which, when executed on one or more programmed processors, carry out the method according to claim 13.

34. An apparatus for providing three-dimensional (3D) video to a legacy two-dimensional (2D) video messenger application comprising:

a first video camera;

a second video camera; and

a processor programmed to;

receive, from the 2D video messenger application, a request for video content from a single video camera;

forward the request for video content from the single video camera to a video processing offload module;

control a first video camera and a second video camera;

receive merged 3D video content from the video processing offload module; and

send the merged 3D video content to the 2D video messenger application in response to receipt of the merged 3D video content from the video processing offload module.

35. The apparatus according to claim 34, where the processor is further programmed to:

receive the request for video content from the single video camera from the video processing offload module in response to forwarding the request to the video processing offload module; and

where, in being programmed to control the first video camera and the second video camera, the processor is programmed to;

control the first video camera in response to receipt of the request for video content from the single video camera from the 2D video messenger application; and

control the second video camera in response to receipt of the request for video content from the single video camera from the video processing offload module.

36. The apparatus according to claim 34, where, in being programmed to control the first video camera and the second video camera, the processor is programmed to:

generate control signals to control video output from the first video camera; and

generate control signals to control video output from the second video.

37. The apparatus according to claim 34, where the processor is further programmed to:

receive first video content from the first video camera and second video content from the second video camera;

send the first video content and the second video content to the video processing offload module; and

where, in being programmed to receive the merged 3D video content from the video processing offload module, the processor is programmed to receive the merged 3D video content from the video processing offload module in response to sending the first video content and the second video content to the video processing offload module.

38. The apparatus according to claim 34, where the processor is further programmed to:

receive a de-queue request associated with an outgoing queue of the first video camera from the 2D video messenger application; and

where, in being programmed to send the merged 3D video content to the 2D video messenger application in response to receipt of the merged 3D video content from the video processing offload module, the processor is programmed to copy the merged 3D video content received from the video processing offload module to the outgoing queue of the first video camera.

39. The apparatus according to claim 34, where the processor is further programmed to:

receive, from the 2D video messenger application, a request to open the single video camera;

generate control signals to open the first video camera;

forward the request to open the single video camera to the video processing offload module;

receive the request to open the single video camera from the video processing offload module; and

generate control signals to open the second video camera in response to receipt of the request to open the single video camera from the video processing offload module.

40. The apparatus according to claim 34, where the processor is further programmed to:

receive, from the 2D video messenger application, a request to close the single video camera;

generate control signals to close the first video camera;

forward the request to close the single video camera to the video processing offload module;

receive the request to close the single video camera from the video processing offload module; and

generate control signals to close the second video camera in response to receipt of the request to close the single video camera from the video processing offload module.

41. The apparatus according to claim 34, where the processor is further programmed to:

receive, from the 2D video messenger application, a request to set a format for output associated with the single video camera;

generate control signals to set the format for output associated with the single video camera to the first video camera;

forward the request to set the format for output associated with the single video camera to the video processing offload module;

receive the request to set the format for output associated with the single video camera from the video processing offload module; and

generate control signals to set the format for output associated with the second video camera to the second video camera in response to receipt of the request to set the format for output associated with the single video camera from the video processing offload module.

42. The apparatus according to claim 34, where the processor is further programmed to:

receive, from the 2D video messenger application, a request to set an operational parameter associated with the single video camera;

generate control signals to set the operational parameter associated with the first video camera;

forward the request to set the operational parameter associated with the single video camera to the video processing offload module;

receive the request to set the operational parameter associated with the single video camera from the video processing offload module; and

generate control signals to set the operational parameter associated with the second video camera in response to receipt of the request to set the operation parameter associated with the single video camera from the video processing offload module.

43. The apparatus according to claim 34, where the processor is further programmed to:

enter a sleep state in response to forwarding the request for video content from the single video camera to the video processing offload module; and

enter an active state in response to receiving the merged 3D video content from the video processing offload module.

44. The apparatus according to claim 34, where the merged 3D video content comprises one of frame sequential 3D video content and polarized 3D video content.

48. The apparatus according to claim 43, where the processor is further programmed to:

receive a de-queue request originated by the 2D video messenger application associated with an outgoing queue of the first video camera via the camera interface module;

determine whether the 3D calibration and the 3D rectification of the first video content and the second video content is completed;

send the first video content from the first video camera to the 2D video messenger application via the camera interface module in response to determining that the 3D calibration and the 3D rectification of the first video content and the second video content is not completed; and

send the merged 3D video content to the 2D video messenger application via the camera interface module in response to determining that the 3D calibration and the 3D rectification of the first video content and the second video content is completed.

45. An apparatus for providing three-dimensional (3D) video to a legacy two-dimensional (2D) video messenger application comprising:

a first video camera;

a second video camera; and

a processor programmed to;

receive, from the 2D video messenger application via a camera interface module, a request for video content from a single video camera;

receive first video content from a first video camera and second video content from a second video camera;

merge the first video content and the second video content into three-dimensional (3D) video content; and

send the merged 3D video content to the 2D video messenger application via the camera interface module in response to the request for the video content from the single video camera.

46. The apparatus according to claim 45, where the processor is further programmed to:

send the request for video content from the single video camera to the camera interface module in response to receipt of the request for video content from the single video camera from the 2D video messenger application via the camera interface module; and

where, the camera interface module controls at least one camera in response to receipt of the request for video content from the single video camera from the video processing module.

47. The apparatus according to claim 45, where the processor is further programmed to perform a 3D calibration and 3D rectification of the first video content and the second video content in response to receiving the first video content from the first video camera and the second video content from the second video camera.

49. The apparatus according to claim 45, where the processor is further programmed to:

receive, from the 2D video messenger application via the camera interface module, a request to open the single video camera; and

send a request to open the second video camera to the camera interface module.

50. The apparatus according to claim 45, where the processor is further programmed to:

receive, from the 2D video messenger application via the camera interface module, a request to close the single video camera; and

send a request to close the second video camera to the camera interface module.

51. The apparatus according to claim 45, where the processor is further programmed to:

receive, from the 2D video messenger application via the camera interface module, a request to set a format for output associated with the single video camera; and

send a request to set a format for output associated with the second video camera to the camera interface module.

52. The apparatus according to claim 45, where the processor is further programmed to:

receive, from the 2D video messenger application via the camera interface module, a request to set an operational parameter associated with the single video camera; and

send a request to set an operational parameter associated with the second video camera to the camera interface module.

53. The apparatus according to claim 45, where the processor is further programmed to:

receive, from the 2D video messenger application via the camera interface module, a request to allocate a number of frame buffers for the single video camera; and

send a request to allocate the same number of frame buffers for the second video camera to the camera interface module.

54. The apparatus according to claim 45, further comprising:

a memory; and

where the processor is further programmed to;

send a request for an index to a frame buffer associated with each of the first video camera and the second video camera to the camera interface module;

receive the index to the frame buffer associated with the first video camera and the index to the frame buffer associated with the second video camera from the camera interface module; and

map the index to the frame buffer associated with the first video camera and the index to the frame buffer associated with the second video camera to the memory.

55. The apparatus according to claim 45, where the processor is further programmed to:

enter an active state in response to receiving, from the 2D video messenger application, the request for video content from the single video camera; and

enter a sleep state in response to starting sending the merged 3D video content to the 2D video messenger application via the camera interface module.

56. The apparatus according to claim 45, where the processor is further programmed to:

receive a request to perform a 3D calibration of the first video content received from the first video camera and the second video content received from the second video camera; and

perform the 3D calibration of the first video content received from the first video camera and the second video content received from the second video camera.

57. The apparatus according to claim 56, where, in being programmed to perform the 3D calibration of the first video content received from the first video camera and the second video content received from the second video camera, the processor is programmed to:

estimate a positional relationship between the first video camera and the second video camera; and

rectify the first video content and the second video content.

58. The apparatus according to claim 57, where, in being programmed to rectify the first video content and the second video content, the processor is programmed to:

obtain warping matrixes; and

apply the warping matrixes to the first video content and the second video content.

59. The apparatus according to claim 56, where, in being programmed to perform the 3D calibration of the first video content received from the first video camera and the second video content received from the second video camera, the processor is programmed to:

detect at least one of an edge and a corner of an image;

estimate a positional relationship between the first video camera and the second video camera; and

adjust at least one of the first video content and the second video content based upon the estimated positional relationship between the first video camera and the second video camera.

60. The apparatus according to claim 45, where, in being programmed to merge the first video content and the second video content into the 3D video content, the processor is programmed to form the 3D video content as one of side-by-side 3D video content, top-and-bottom 3D video content, and frame sequential 3D video content.

61. The apparatus according to claim 45, where the processor is further programmed to:

receive a request to change a portion of the merged 3D video content;

detect the portion of the merged 3D video content within each of the first video content received from the first video camera and the second video content received from the second video camera; and

change a portion of the first video content received from the first video camera and a portion of the second video content received from the second video camera.

62. The apparatus according to claim 61, where:

the request to change the portion of the merged 3D video content comprises a request to smooth a facial feature of an image of a person represented within the merged 3D video content; and

in being programmed to change the portion of the first video content received from the first video camera and the portion of the second video content received from the second video camera, the processor is programmed to;

smooth the facial feature of the image of the person represented in the first video content received from the first video camera; and

smooth the facial feature of the image of the person represented in the second video content received from the second video camera.

63. The apparatus according to claim 61, where:

the request to change the portion of the merged 3D video content comprises a request to thin a body image of a person represented within the merged 3D video content; and

in being programmed to change the portion of the first video content received from the first video camera and the portion of the second video content received from the second video camera, the processor is programmed to;

thin the body image of the person represented in the first video content received from the first video camera; and

thin the body image of the person represented in the second video content received from the second video camera.

64. The apparatus according to claim 61, where:

the request to change the portion of the merged 3D video content comprises a request to stretch a portion of an image represented within the merged 3D video content; and

in being programmed to change the portion of the first video content received from the first video camera and the portion of the second video content received from the second video camera, the processor is programmed to;

stretch the portion of the image represented in the first video content received from the first video camera; and

stretch the portion of the image represented in the second video content received from the second video camera.