System and method for interactive markerless paper documents in 3D space with mobile cameras and projectors

• US 9,117,274 B2
• Filed: 08/01/2011
• Issued: 08/25/2015
• Est. Priority Date: 08/01/2011
• Status: Active Grant

• Current Assignee: Fujifilm Business Innovation Corp. (Fujifilm Holdings Corporation)
• Original Assignee: Fuji Xerox Company Limited (Xerox Holdings Corp.)
• Inventors: Liao, Chunyuan, Liu, Qiong, Huber, Jochen
• Primary Examiner(s): Richer, Joni
• Assistant Examiner(s): Wang, Yi
• Application Number:
  US13/195,689
• Publication Number:
  US 20130033484A1
• Time in Patent Office: 1,485 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30176 Document
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04N 9/3179 Video signal processing therefor

Abstract

Systems and methods directed to fine-grained interaction with ordinary markerless paper documents and projectors at flexible poses in 3D space. Systems and methods allow for the projection of images onto non-flat variable surfaces by utilizing depth detection techniques in order to project an image or video properly onto a variable surface.

Claims

1. A system, comprising:

a camera system configured to detect three-dimensional orientation information for an identified surface onto which an image is to be projected, the three-dimensional orientation information comprising information about an angle of the identified surface, an orientation of the identified surface and a shape of the identified surface;

at least one projector configured to project the image onto the identified surface, and to adjust the image to be projected within the identified surface and oriented in accordance with the information about the angle of the identified surface, the orientation of the identified surface and the shape of the identified surface,

wherein the identified surface comprises a marker-less surface, and

wherein the camera system is configured to detect the identified surface by creating a depth image based on images received from the camera, analyzing the depth image for weakly connected components to isolate potential surfaces, determining inner points of one of said potential surfaces, mapping the inner points to outer points of the one of said potential surfaces to define a surface, the outer points being determined based on preset attributes for a desired surface, and tracking the defined surface as the identified surface.

2. The system of claim 1, wherein the three-dimensional orientation information comprises depth information.

3. The system of claim 2, further comprising an infrared light source projecting an infrared light onto the identified surface;

wherein the camera system comprises a depth-sensing camera receiving depth information by recording infrared light reflected from the surface.

4. The system of claim 3, wherein the camera system comprises a RGB camera capturing visual feature information of a plurality of surfaces, wherein the visual feature information is used to identify content on the identified surface.

5. The system of claim 4, wherein the system detects gestures from a user.

6. The system of claim 1, wherein the camera system detects changes to the surface in real time, and wherein the image is adjusted according to said changes in real time.

7. The system of claim 1, wherein the identified surface comprises a flexible surface.

8. The system of claim 1, wherein the projector is configured to project onto the identified surface from a plurality of angles and a plurality of distances.

9. The system of claim 8, wherein the projector and the identified surface are configured to be manipulated simultaneously.

10. A method, comprising:

identifying a surface;

utilizing a camera system to receive three-dimensional orientation information for the identified surface onto which an image is to be projected, the three dimensional orientation information comprising information about an angle of the identified surface, an orientation of the identified surface and a shape of the identified surface; and

projecting the image onto the identified surface and adjusting the image to fit within the identified surface and oriented in accordance with the information about the angle of the identified surface, the orientation of the identified surface and the shape of the identified surface,

wherein the identified surface comprises a marker-less surface, and

wherein, in utilizing the camera system, the camera system detects the identified surface by creating a depth image based on images received from the camera, analyzing the depth image for weakly connected components to isolate potential surfaces, determining inner points of one of said potential surfaces, mapping the inner points to outer points of the one of said potential surfaces to define a surface, the outer points being determined based on preset attributes for a desired surface, and tracking the defined surface as the identified surface.

11. The method of claim 10, wherein the three-dimensional orientation information comprises depth information.

12. The method of claim 11, further comprising projecting an infrared light onto the identified surface;

and receiving depth information by recording infrared light reflected from the surface.

13. The method of claim 12, wherein utilizing the camera system further comprises utilizing a RGB camera to capture visual feature information from a plurality of surfaces, and utilizing the visual feature information to identify content on the identified surface.

14. The method of claim 13, further comprising detecting gestures from a user.

15. The method of claim 10, further comprising detecting changes in the surface in real time, and adjusting the image according to said changes in real time.

16. A camera-projector unit, comprising:

a camera system configured to track an identified surface in real time and receive three dimensional orientation information, the three dimensional orientation information comprising information about an angle of the identified surface, an orientation of the identified surface onto which an image is to be projected, a shape of the identified surface and a depth of the identified surface;

a projector configured to project the image onto the identified surface and adjust the image to be projected within the identified surface and oriented in accordance with the information about the angle of the identified surface, the orientation of the identified surface, the shape of the identified surface and the depth of the identified surface;

wherein the camera system detects changes to the surface in real time, and wherein the image is adjusted according to said changes in real time, and

wherein the identified surface comprises a marker-less surface, and

wherein the identified surface is determined by a process comprising;

creating a depth image based on images received from the camera;

analyzing the depth image for weakly connected components to isolate potential surfaces;

determining inner points of one of said potential surfaces;

mapping the inner points to outer points of the one of said potential surfaces to define a surface, the outer points determined based on preset attributes for a desired surface; and

tracking the defined surface as the identified surface.

17. The camera-projector unit of claim 16, wherein the camera system further comprises a RGB camera capturing visual feature information from a plurality of surfaces, wherein the visual feature information is utilized to identify content within the identified surface.

18. The camera-projector unit of claim 17, wherein the camera system detects gestures from a user within the identified surface.

19. The camera-projector unit of claim 18, wherein the depth information is utilized to detect a touch from the user within the identified surface.

20. The camera-projector unit of claim 16, wherein an infrared light is sent to the identified surface, and wherein the three dimensional orientation information is obtained based on reflections of the infrared light from the identified surface.