SYSTEMS AND METHODS FOR GENERATING A VIRTUAL CAMERA VIEWPOINT FOR AN IMAGE

• US 20110267348A1
• Filed: 04/29/2011
• Published: 11/03/2011
• Est. Priority Date: 04/29/2010
• Status: Active Grant

• Current Assignee: Personify Incorporated (Wilson Human Capital Group, Inc.)
• Original Assignee: Personify Incorporated (Wilson Human Capital Group, Inc.)
• Inventors: Lin, Dennis, Do, Minh N., Patel, Sanjay J., Nguyen, Quang H.
• Granted Patent: US 9,053,573 B2
• Time in Patent Office: Days
• US Class Current: 345/426
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 15/205 Image-based rendering

Abstract

A color image and a depth image of a live video are received. A user is extracted from the information of the color image and the depth image. Spurious depth vales may be corrected. Points or pixels of an image as seen from a viewpoint of a reference camera at a reference camera location are mapped to points of the image as would be seen from a viewpoint of a virtual camera at a virtual camera location. As such, a transformed color image is generated. Disoccluded pixels may be processed to address any gaps within the transformed color image.

Claims

1. A method for generating a second viewpoint of an image, the method comprising:

receiving, using a computer, a depth image and a color image of a video from at least one reference camera at a reference camera location comprising a first viewpoint of the color image;

receiving a virtual camera location comprising a second viewpoint of the color image; and

translating at least one pixel from the color image to a transformed color image comprising the second viewpoint of the color image from the virtual camera location, the translating based on a depth value of the at least one pixel from the depth image.

2. The method as set forth in claim 1, wherein the virtual camera location comprises a shift from the reference camera location, the shift comprises a change in at least one dimension comprising at least one of a vertical dimension and a horizontal dimension.

3. The method as set forth in claim 1, wherein the translating to the transformed color image further comprises shifting a plurality of pixels in the color image in at least one dimension, the at least one dimension comprising at least one of a horizontal dimension and a vertical dimension, the shifting of the pixels in the horizontal dimension comprises shifting pixels in a row, the shifting of the pixels in the vertical dimension comprises shifting pixels in a column.

4. The method as set forth in claim 3, wherein the translating further comprises a sequence of one dimensional transformations of the color image, the sequence comprises the shifting of the pixels in a first transformation along a first dimension followed by a shifting of the pixels in a second transformation along a second dimension, the first and second transformation along the first and second dimension may comprise shifting of the pixels of each row of the color image followed by the shifting of the pixels in each column of the color image.

5. The method as set forth in claim 1, wherein the translating comprises a three dimensional (3D) mapping of pixels of the color image to pixels of the transformed color image, the mapping further comprises reversing a projection matrix of the at least one camera.

6. The method as set forth in claim 1, further comprising correcting spurious depth values of the depth image, the correction comprising the application of a bilateral filter to regions of the depth image that comprise an unknown depth value.

7. The method as set forth in claim 1, further comprising processing disoccluded pixels of the transformed image comprising portions of the color image that were not captured by the reference camera, the processing comprising assigning a weighted average pixel color value of nearby pixels of the transformed image to each disoccluded pixel in the transformed image.

8. The method as set forth in claim 1, further comprising processing disoccluded pixels of the transformed image comprising portions of the color image that were not captured by the reference camera, the processing comprising filling in a region comprising disoccluded pixels with an appearance of a shadow such that a light source associated with the shadow comprises the reference camera location.

9. A system, comprising at least one processor and memory, for generating a second viewpoint of an image, the system comprising:

a module for receiving a depth image and a color image of a video from at least one reference camera at a reference camera location comprising a first viewpoint of the color image;

a module for receiving a virtual camera location comprising a second viewpoint of the color image; and

a module for translating at least one pixel from the color image to a transformed color image comprising the second viewpoint of the color image from the virtual camera location, the translating based on a depth value of the at least one pixel from the depth image.

10. The system as set forth in claim 9, wherein the virtual camera docation comprises a shift from the reference camera location, the shift comprises a change in at least one dimension comprising at least one of a vertical dimension and a horizontal dimension.

11. The system as set forth in claim 10, wherein the translating to the transformed color image further comprises shifting a plurality of pixels in the color image in at least one dimension, the at least one dimension comprising at least one of a horizontal dimension and a vertical dimension, the shifting of the pixels in the horizontal dimension comprises shifting pixels in a row, the shifting of the pixels in the vertical dimension comprises shifting pixels in a column.

12. The system as set forth in claim 11, wherein the translating further comprises a sequence of one dimensional transformations of the color image, the sequence comprises the shifting of the pixels in a first transformation along a first dimension followed by a shifting of the pixels in a second transformation along a second dimension, the first and second transformation along the first and second dimension may comprise shifting of the pixels of each row of the color image followed by the shifting of the pixels in each column of the color image.

13. The system as set forth in claim 9, wherein the translating comprises a three dimensional (3D) mapping of pixels of the color image to pixels of the transformed color image, the mapping further comprises reversing a projection matrix of the at least one camera.

14. The system as set forth in claim 9, further comprising a module for correcting spunous depth values of the depth image, the correction comprising the application of a bilateral filter to regions of the depth image that comprise an unknown depth value.

15. The system as set forth in claim 9, further comprising a module for processing disoccluded pixels of the transformed image comprising portions of the color image that were not captured by the reference camera, the processing comprising assigning a weighted average pixel color value of nearby pixels of the transformed image to each disoccluded pixel in the transformed image.

16. The system as set forth in claim 9, further comprising a module for processing disoccluded pixels of the transformed image comprising portions of the color image that were not captured by the reference camera, the processing comprising filling in a region comprising disoccluded pixels with an appearance of a shadow such that a light source associated with the shadow comprises the reference camera location.

17. A non-transitory computer readable medium carrying one or more instructions for generating a second viewpoint of an image, wherein the one or more instructions, when executed by one or more processors, causes the one or more processors to perform the steps of:

receiving, using a computer, a depth image and a color image of a video from at least one reference camera at a reference camera location comprising a first viewpoint of the color image;

receiving a virtual camera location comprising a second viewpoint of the color image; and

translating at least one pixel from the color image to a transformed color image comprising the second viewpoint of the color image from the virtual camera location, the translating based on a depth value of the at least one pixel from the depth image.

18. The non-transitory computer readable medium as set forth in claim 17, wherein the virtual camera location comprises a shift from the reference camera location, the shift comprises a change in at least one dimension comprising at least one of a vertical dimension and a horizontal dimension.

19. The non-transitory computer readable medium as set forth in claim 17, wherein the translating to the transformed color image further comprises shifting a plurality of pixels in the color image in at least one dimension, the at least one dimension comprising at least one of a horizontal dimension and a vertical dimension, the shifting of the pixels in the horizontal dimension comprises shifting pixels in a row, the shifting of the pixels in the vertical dimension comprises shifting pixels in a column.

20. The non-transitory computer readable medium as set forth in claim 19, wherein the translating further comprises a sequence of one dimensional transfbrmations of the color image, the sequence comprises the shifting of the pixels in a first transformation along a first dimension followed by a shifting of the pixels in a second transformation along a second dimension, the first and second transformation along the first and second dimension may comprise shifting of the pixels of each row of the color image followed by the shifting of the pixels in each column of the color image.

21. The non-transitory computer readable medium as set forth in claim 17, wherein the translating comprises a three dimensional (3D) mapping of pixels of the color image to pixels of the transformed color image, the mapping further comprises reversing a projection matrix of the at least one camera.

22. The non-transitory computer readable medium as set forth in claim 17, wherein the steps further comprise correcting spurious depth values of the depth image, the correction comprising the application of a bilateral filter to regions of the depth image that comprise an unknown depth value.

23. The non-transitory computer readable medium as set forth in claim 17, wherein the steps further comprise processing disoccluded pixels of the transformed image comprising portions of the color image that were not captured by the reference camera, the processing comprising assigning a weighted average pixel color value of nearby pixels of the transformed image to each disoccluded pixel in the transformed image.

24. The non-transitory computer readable medium as set forth in claim 17, wherein the steps further comprises processing disoccluded pixels of the transformed image comprising portions of the color image that were not captured by the reference camera, the processing comprising filling in a region comprising disoccluded pixels with an appearance of a shadow such that a light source associated with the shadow comprises the reference camera location.