SYSTEM AND METHOD FOR INTERACTIVE MARKERLESS PAPER DOCUMENTS IN 3D SPACE WITH MOBILE CAMERAS AND PROJECTORS

• US 20130033484A1
• Filed: 08/01/2011
• Published: 02/07/2013
• Est. Priority Date: 08/01/2011
• Status: Active Grant

• Current Assignee: Fujifilm Business Innovation Corp. (Fujifilm Holdings Corporation)
• Original Assignee: Fuji Xerox Company Limited (Xerox Holdings Corp.)
• Inventors: LIAO, Chunyuan, Liu, Qiong, Huber, Jochen
• Granted Patent: US 9,117,274 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30176 Document
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04N 9/3179 Video signal processing therefor

Abstract

Systems and methods directed to fine-grained interaction with ordinary markerless paper documents and projectors at flexible poses in 3D space. Systems and methods allow for the projection of images onto non-flat variable surfaces by utilizing depth detection techniques in order to project an image or video properly onto a variable surface.

Claims

1. A system, comprising:

a camera system receiving three-dimensional orientation information for an identified surface;

at least one projector projecting an image onto the identified surface, and adjusting the image to be projected within the identified surface and oriented in accordance with the three-dimensional orientation information of the identified surface.

2. The system of claim 1, wherein the three-dimensional orientation information comprises depth information.

3. The system of claim 1, wherein the camera system detects changes to the surface in real time, and wherein the image is adjusted according to said changes in real time.

4. The system of claim 2, further comprising an infrared light source projecting an infrared light onto the identified surface;

wherein the camera system comprises a depth-sensing camera receiving depth information by recording infrared light reflected from the surface.

5. The system of claim 4, wherein the camera system comprises a RGB camera capturing visual feature information of a plurality of surfaces, wherein the visual feature information is used to identify content on the identified surface.

6. The system of claim 5, wherein the system detects gestures from a user.

7. A method, comprising:

identifying a surface;

utilizing a camera system to receive three-dimensional orientation information for the identified surface; and

projecting an image onto the identified surface and adjusting the image to fit within the identified surface and oriented in accordance with the three-dimensional orientation information.

8. The method of claim 7, wherein the three-dimensional orientation information comprises depth information.

9. The method of claim 7, further comprising detecting changes in the surface in real time, and adjusting the image according to said changes in real time.

10. The method of claim 8, further comprising projecting an infrared light onto the identified surface;

and receiving depth information by recording infrared light reflected from the surface.

11. The method of claim 10, wherein utilizing the camera system further comprises utilizing a RGB camera to capture visual feature information from a plurality of surfaces, and utilizing the visual feature information to identify content on the identified surface.

12. The method of claim 11, further comprising detecting gestures from a user.

13. A camera-projector unit, comprising:

a camera system tracking an identified surface in real time and receiving three dimensional orientation information, the three dimensional orientation information comprising depth information;

a projector projecting an image onto the identified surface and adjusting the image to be projected within the identified surface and oriented in accordance with the three-dimensional orientation information;

wherein the camera system detects changes to the surface in real time, and wherein the image is adjusted according to said changes in real time.

14. The camera-projector unit of claim 13, wherein the camera system further comprises a RGB camera capturing visual feature information from a plurality of surfaces, wherein the visual feature information is utilized to identify content within the identified surface.

15. The camera-projector unit of claim 14, wherein the camera system detects gestures from a user within the identified surface.

16. The camera-projector unit of claim 15, wherein the depth information is utilized to detect a touch from the user within the identified surface.

17. The camera-projector unit of claim 13, wherein an infrared light is sent to the identified surface, and wherein the three dimensional orientation information is obtained based on reflections of the infrared light from the identified surface.

18. The camera-projector unit of claim 13, wherein the identified surface is determined by a process comprising:

creating a depth image based on images received from the camera;

analyzing the depth image for weakly connected components to isolate potential surfaces;

determining inner points of one of said potential surfaces;

mapping the inner points to outer points of the one of said potential surfaces to define a surface, the outer points determined based on preset attributes for a desired surface; and

tracking the defined surface as the identified surface.