Method and system for determining reference points in video image frames

• US 9,386,276 B2
• Filed: 07/26/2012
• Issued: 07/05/2016
• Est. Priority Date: 03/23/2012
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Polycom Incorporated (HP Inc.)
• Inventors: Rosenberg, Yaakov Moshe
• Primary Examiner(s): Nguyen, Joseph J
• Application Number:
  US13/558,739
• Publication Number:
  US 20130250036A1
• Time in Patent Office: 1,440 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: H04N 7/152 Multipoint control units therefor

Abstract

A videoconferencing system for determining alignment information for images captured by two or more cameras is disclosed. The videoconferencing system can include a plurality of endpoints and at least one control unit (CU) such as a multipoint control unit (MCU), for example. An endpoint can include a plurality of cameras and at least one projector. The projector is used to project a pattern at the near end site, which pattern is captured by the plurality of cameras. The image frames produced by the cameras are processed to determine the identity and location coordinates of the images of the projected patterns. The location coordinates can be used as reference points to be used by applications such as telepresence, 3D videoconferencing, and morphing.

Claims

1. A method performed by a control unit (CU) for aligning two or more video image frames received from two or more video cameras at an endpoint, the endpoint communicably connected to the CU via a communication network, comprising:

receiving two or more reference image frames from the endpoint; and

determining two or more reference points, each reference point located in at least two of the two or more video image frames, the two or more reference points derived from location coordinates of a same spot of a projected reference image captured in the two or more reference image frames,

wherein the projected reference image includes a plurality of patterns, and

wherein the same spot corresponds to one of the plurality of patterns.

2. The method of claim 1, further comprising sending a communication to the endpoint, the communication including instructions to the endpoint to project a reference image.

3. The method of claim 1, wherein the CU is a multipoint control unit (MCU).

4. The method of claim 2, wherein the communication to the endpoint further includes instructions to the endpoint to project the reference image in an invisible spectrum.

5. The method of claim 2, wherein the communication further includes instructions to the endpoint to capture the two or more reference image frames using two or more reference cameras, each of the two or more reference cameras paired with one of the two or more video cameras.

6. The method of claim 5, wherein the determining the two or more reference points comprises transforming the location coordinates of the same spot of the projected reference image from within the two or more reference image frames into location coordinates within the two or more video image frames.

7. The method of claim 6, wherein the transforming is based on lookup tables, each lookup table comprising location coordinates of at least one spot as the at least one spot appears in one of the reference image frames and paired video image frames.

8. The method of claim 7, wherein the lookup tables are received from the endpoint.

9. The method of claim 7, wherein the lookup tables are generated by the CU.

10. The method of claim 7, further comprising:

sending another communication to the endpoint, the another communication includes instructions to the endpoint to alter a camera configuration of one or more of the two or more video cameras; and

repeating the steps of sending, receiving, and determining two or more reference points.

11. The method of claim 1, wherein the location coordinates of the same spot in the two or more reference image frames are determined by correlating a mask for each of the plurality of patterns with each of the two or more reference image frames.

12. The method of claim 1, further comprising manipulating the two or more video image frames as a function of the two or more reference points.

13. The method of claim 1, wherein the communication further includes instructions to the endpoint to capture the two or more reference image frames using the two or more video cameras.

14. The method of claim 13, wherein the determining the two or more reference points comprises defining the location coordinates of the same spot of the projected reference image captured in the two or more reference image frames as the two or more reference points.

15. A non-transitory computer readable medium containing a program for performing a method comprising:

projecting a reference image in front of two or more video cameras and two or more reference cameras of an endpoint, each of the two or more reference cameras paired with one of the two or more video cameras;

capturing two or more video image frames by the two or more video cameras;

capturing two or more reference image frames by the two or more reference cameras; and

determining two or more reference points, each reference point located in one of two or more video image frames and derived by transforming location coordinates of a same portion of the projected reference image captured in the two or more reference image frames into location coordinates within the two or more video image frames,

wherein the projected reference image includes a plurality of patterns, and

wherein the same portion of the projected reference image captured in the two or more reference image frames corresponds to one of the plurality of patterns.

16. A non-transitory computer readable medium containing a program for performing a method performed by a multipoint control unit (MCU) for aligning two or more video image frames received from two or more video cameras at an endpoint, the endpoint communicably connected to the MCU via a communication network, comprising:

sending a communication to the endpoint, the communication including instructions to the endpoint to project a reference image;

receiving two or more reference image frames from the endpoint; and

determining two or more reference points, each reference point located in one of the two or more video image frames, the two or more reference points derived from transforming location coordinates of a same portion of the projected reference image captured in the two or more reference image frames into location coordinates within the two or more video image frames,

wherein the projected reference image includes a plurality of patterns, and

wherein the same portion of the projected reference image captured in the two or more reference image frames corresponds to one of the plurality of patterns.

17. An endpoint comprising:

two or more video cameras;

a projector configured to project a reference image in front of the two or more video cameras; and

a processing module communicably coupled to the two or more video cameras and the projector, the processing module programmed to;

instruct the projector to project the reference image;

receive two or more video image frames, one each from each of the two or more video cameras;

receive two or more reference image frames; and

determine two or more reference points,

wherein each reference point is located in at least two of the two or more video image frames, and

wherein the two or more reference points are derived from location coordinates of a same spot of the projected reference image captured in the two or more reference image frames; and

transform location coordinates of the same spot of the projected reference image into location coordinates with the two or more video image frames,

wherein the projected reference image includes a plurality of patterns, and

wherein the same spot corresponds to one of the plurality of patterns.

18. The endpoint of claim 17, wherein the projector is further configured to project the reference image in an invisible spectrum.

19. The endpoint of claim 17,

wherein the processing module is further programmed to determine the location coordinates of the same spot in the two or more reference image frames by correlating a mask for each spot with each of the two or more reference image frames.

20. The endpoint of claim 17, wherein the processing module is further programmed to communicate the two or more reference points to a control unit or another endpoint.

21. The endpoint of claim 20, further comprising:

two or more reference cameras,

wherein the two or more reference image frames are captured by the two or more reference cameras.

22. A method comprising:

projecting a reference image in a room having an endpoint;

capturing a plurality of reference image frames;

processing the captured plurality of reference image frames and detecting a reference point that appears in at least two of the captured plurality of reference image frames; and

transforming location coordinates of the reference point in each of the captured plurality of reference image frames into coordinates of video image frames received from video cameras of the endpoint,

wherein the reference image comprises a plurality of different reference patterns.

23. The method of claim 22, wherein capturing a plurality of reference image frames is performed by a plurality of reference cameras of the endpoint.

24. The method of claim 22, wherein projecting the reference image is performed in an invisible spectrum.

25. The method of claim 22, wherein the plurality of different reference patterns differ in time and location in the plurality of reference image frames.