Spatially-aware projection pen

• US 9,733,792 B2
• Filed: 08/06/2009
• Issued: 08/15/2017
• Est. Priority Date: 10/27/2008
• Status: Active Grant

• Current Assignee: Autodesk, Inc.
• Original Assignee: Autodesk, Inc.
• Inventors: Khan, Azam, Attar, Ramtin, Song, Hyunyoung, Guimbretiere, Francois V., Kurtenbach, Gordon, Grossman, Tovi, Fitzmaurice, George
• Primary Examiner(s): MISHLER, ROBIN J
• Application Number:
  US12/537,013
• Publication Number:
  US 20100103178A1
• Time in Patent Office: 2,931 Days
• Field of Search: 345179, 345156, 345419
• CPC Class Codes: G06F 3/03542 Light pens for emitting or receiving light
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus

Abstract

One embodiment of the present invention sets forth a technique for providing an end user with a digital pen embedded with a spatially-aware miniature projector for use in a design environment. Paper documents are augmented to allow a user to access additional information and computational tools through projected interfaces. Virtual ink may be managed in single and multi-user environments to enhance collaboration and data management. The spatially-aware projector pen provides end-users with dynamic visual feedback and improved interaction capabilities.

Claims

1. A spatially-aware projection pen system, comprising:

an input mechanism configured to be activated and deactivated by a user;

a position tracking mechanism configured to determine a position of the spatially-aware projection pen within a three-dimensional space and relative to a display surface;

a memory configured to store design data, image data, and an application program, wherein the design data comprises computer-aided design (CAD) data;

a processor configured to;

execute the application program to generate at least a portion of the image data based on the CAD data and printed content on the display surface,

receive input signals from the input mechanism and the position tracking mechanism, and

generate a projected image based on the image data, the input signals and the position of the spatially-aware projection pen, wherein the proiected image includes a plurality of visual components, and wherein a geometry and a location of each visual component corresponds to a different content portion of the printed content;

a projector configured to display the projected image overlaid on the printed content, wherein a location of the projected image is locked relative to the printed content and remains static as the position of the spatially-aware projection pen changes; and

an enclosure configured to be grasped by a human hand and house the input mechanism, the position tracking mechanism, the memory, the projector, and the processor.

2. The system of claim 1, further comprising a force sensor that is configured to detect a force applied to a tip of the spatially-aware projection pen and provide a force signal to the processor indicating whether the tip of the spatially-aware projection pen is in contact with the display surface.

3. The system of claim 2, wherein the force signal provided by the force sensor is captured as virtual ink by the processor and stored in the memory as the image data representing pen strokes input by the user.

4. The system of claim 2, wherein the projector is further configured to display virtual ink corresponding to pen strokes in the projected image when the tip of the spatially-aware projection pen is in contact with the display surface.

5. The system of claim 1, wherein the processor is configured to render a frame of pixel data using the design data and based on the position of the spatially-aware projection pen.

6. The system of claim 5, wherein the processor is further configured to store the frame of pixel data in the memory as image data.

7. The system of claim 5, wherein the processor is further configured to output the frame of pixel data to the projector and the projector is further configured to display the frame in the projected image.

8. The system of claim 5, wherein the frame of pixel data represents a two-dimensional section view of a three-dimensional model represented by the design data.

9. The system of claim 1, wherein the position tracking mechanism is further configured to determine whether the spatially-aware projection pen is within a surface input layer that includes the display surface, wherein user input received via the input mechanism when the spatially-aware projection is within the surface input layer modifies the projected image.

10. The system of claim 1, wherein the printed content comprises a second portion of the design data.

11. The system of claim 1, further comprising an ink reservoir that is configured to deposit physical ink on the display surface when a tip of the spatially-aware projection pen is in contact with the display surface.

12. The system of claim 1, further comprising a network interface that is coupled to the processor and configured to interface with a remote user.

13. The system of claim 1, wherein the projector is positioned within the enclosure to provide a field of view angle of 30 degrees.

14. The system of claim 1, wherein the processor is configured to generate a user interface that is displayed within the projected image by the projector.

15. The system of claim 1, wherein the processor is configured to render a sequence of frames of pixel data based on the design data and a path captured by the position tracking mechanism to produce a walk-through animation of a three-dimensional model represented by the design data.

16. The system of claim 1, wherein the processor is configured to generate virtual ink that is displayed in the projected image by the projector when the position tracking mechanism indicates that the spatially-aware projection pen is in contact with the display surface.

21. The system of claim 1, wherein the position tracking mechanism is further configured to determine whether the spatially-aware projection pen is within a hover input layer that is a fixed distance from the display surface, wherein user input received via the input mechanism when the spatially-aware projection is within the hover input layer manipulates a cursor displayed on the projected image.

22. The system of claim 21, wherein the position tracking mechanism is further configured to determine whether the spatially-aware projection pen is within a spatial input layer that is a second fixed distance from the hover input layer, wherein second user input received via the input mechanism when the spatially-aware projection is within the hover input layer manipulates a scale of the projected image.

17. A spatially-aware projection pen system, comprising:

an input mechanism configured to be activated and deactivated by a user;

a camera configured to capture image data and determine a position of the spatially-aware projection pen within a three-dimensional space and relative to a display surface;

a memory configured to store design data, image data, and an application program, wherein the design data comprises computer-aided design (CAD) data;

a processor configured to;

execute the application program to generate at least a portion of the image data based on the CAD data and printed content on the display surface,

receive input signals from the input mechanism and the position tracking mechanism, and

generate a projected image based on the image data, the input signals and the position of the spatially-aware projection pen, wherein the projected image includes a plurality of visual components, and wherein a geometry and a location of each visual component corresponds to a different content portion of the printed content;

a projector configured to display the projected image overlaid on the printed content, wherein a location of the projected image is locked relative to the printed content and remains static as the position of the spatially-aware projection pen changes; and

an enclosure configured to be grasped by a human hand and house the input mechanism, the position tracking mechanism, the memory, the projector, and the processor.

18. The system of claim 17, wherein the camera is further configured to determine the position of the spatially-aware projection pen in a three-dimensional space by retrieving camera calibration parameters based on traceable patterns on the display surface.

19. The system of claim 17, wherein the projector is further configured to generate virtual ink corresponding to pen strokes in the projected image when the camera indicates that the spatially-aware projection pen is in contact with the display surface.

20. The system of claim 17, wherein the camera is further configured to determine whether the spatially-aware projection pen is within a surface input layer that includes the display surface, within a hover input layer of the three-dimensional space that is directly above the display surface, or within a spatial input layer of the three-dimensional space that is directly above the hover input layer.