Creative design systems and methods

• US 9,019,239 B2
• Filed: 11/29/2010
• Issued: 04/28/2015
• Est. Priority Date: 11/29/2010
• Status: Expired due to Fees

• Current Assignee: Northrop Grumman Systems Corporation (Northrop Grumman Corporation)
• Original Assignee: Northrop Grumman Systems Corporation (Northrop Grumman Corporation)
• Inventors: Nishihara, H. Keith, Hsu, Shi-Ping, Lariviere, Donald G.
• Primary Examiner(s): Lao, Lun-Yi
• Assistant Examiner(s): Suteerawongsa, Jarurat
• Application Number:
  US12/955,563
• Publication Number:
  US 20120133616A1
• Time in Patent Office: 1,611 Days
• Field of Search: 345156-158, 345163-166, 345/173, 345/175, 345179-184, 178/19.05
• US Class Current: 345/175
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/03542 Light pens for emitting or receiving light
• CPC Class Codes: G06F 3/0428 by sensing at the edges of the touch surface the interruption of optical paths, e.g. an illumination plane, parallel to the touch surface which may be virtual sensing beam interruptions in a planar beam grid of an optical touch-screen G06F3/0421

Abstract

Creative design systems and methods are disclosed. In one embodiment, a creative design system is provided. The creative design system comprises a high resolution display, an interactive stylus that includes a transmitter at a tip of the interactive stylus that transmits an encoded signal associated with the interactive stylus and a plurality of sensors that track movement of the interactive stylus over the high resolution display by capturing and decoding the transmitted encoded signal as the stylus moves over the high resolution display. A creative design controller is configured to display sketches of context in response to the tracking of the movement of the interactive stylus over the high resolution display.

Claims

1. A creative design system comprising:

a high resolution display;

an interactive stylus that includes a transmitter at a tip of the interactive stylus that transmits an encoded signal associated with the interactive stylus;

a plurality of sensors that track movement of the interactive stylus over the high resolution display by capturing and decoding the transmitted encoded signal as the stylus moves over the high resolution display;

a calibration component configured to employ data provided from the plurality of sensors to determine and calibrate a distance between the transmitter and a surface of a plurality of different cell regions of the high resolution display to determine if the transmitter is within a threshold distance from a cell region of the plurality of different cell regions of the high resolution display, wherein the distance is measured on an axis normal to the surface of the high resolution display and the cell region of the plurality of different cell regions; and

a creative design controller configured to display sketches of context in response to the tracking of the movement of the interactive stylus over the high resolution display.

2. The system of claim 1, wherein the transmitter is an infrared transmitter and the plurality of sensors are infrared cameras that capture images of the infrared transmissions of the encoded signal of the interactive stylus and the cameras monitor movement of the transmitter to determine when to display sketched images in response to sketches by a user on the high resolution display.

3. The system of claim 2, wherein the creative design controller employ triangulation utilizing images from at least two cameras of the plurality of cameras.

4. The system of claim 3, wherein the plurality of cameras are four cameras with one of the four cameras being located in each of the four corners of the high resolution display.

5. The system of claim 4, wherein the creative design controller further comprises a jitter suppressor that tracks the component intersection point locations from prior frames of two or more of the cameras and estimates the location of a new intersection point based on weighted averages of the intersection point locations from prior frames and intersections point locations of new frames.

6. The system of claim 1, wherein the creative design controller further comprises a subpixel tracker that gains back precision lost by the plurality of sensors relative to the high resolution display by locating a bright spot of the transmitter, locating a peak in a bandpass filtered image of the bright spot, and computing a center of mass of the bandpass filtered image to determine a sub-pixel estimate for the position of the bright spot.

7. The system of claim 1, further comprising a gesture library, wherein the creative design controller compares images captures by the infrared cameras to determine if gesture movements of at least one of the interactive stylus and a user'"'"'s hand match a gesture in the gesture library, and performs an associated function if gesture movements of at least one of the interactive stylus and a user'"'"'s hand matched a given gesture in the gesture library.

8. The system of claim 1, wherein the creative design controller preloads and displays detailed text and figures on the high resolution display and the creative design controller further comprises an image analyzer that analyzes image data from the stylus sensors to determine movement of the stylus sensor to determine when to display sketched images in response to sketches by a user on the high resolution display.

9. The system of claim 1, wherein the creative design controller further comprises a calibration component that displays a sequence of targets on the high resolution display, prompts the user to touch the interactive stylus to the targets in a predetermined order and employs images captured by the plurality of sensors to determine and calibrate one or more parameters of the plurality of sensors employing the captured images.

10. The system of claim 1, wherein the calibration component is further configured to employ histograms to determine and calibrate the distance between the transmitter and the surface of the high resolution display to determine if the transmitter is within a threshold distance from the high resolution display and if the transmitter is not within the threshold distance from the high resolution display.

11. The system of claim 1, wherein the calibration component employs histograms associated with the plurality of different cell regions of the high resolution display to determine and calibrate the distance between the transmitter and the surface of the high resolution display to determine if the transmitter is within a threshold distance from the high resolution display and if the transmitter is not within the threshold distance from the high resolution display in a particular cell region of the plurality of different cell regions of the surface of the high resolution display.

12. The system of claim 1, wherein the tip is a conical tip that deflects light to the sides for better visibility by stylus cameras.

13. The system of claim 1, wherein the interactive stylus employs pressure sensing to determine the pressure exerted on the high resolution display by the tip.

14. A creative design system comprising:

a high resolution display;

an interactive stylus that includes an infrared transmitter located at its tip that generates infrared light;

an infrared camera located in each corner of the high resolution display that captures images to track movement of the infrared light over the high resolution display; and

a creative design controller configured to display preloaded detailed text and figures and generate instructions to display sketches of context in response to movement of the infrared light over the high resolution display based on a determined sub-pixel estimate of a bright spot from the infrared light in the captured images based on the pixel resolution of the infrared camera,

wherein the creative design controller comprises a jitter suppressor that tracks the intersection point locations from prior frames of two or more of the cameras and estimates the location of a new intersection point based on weighted averages of the intersection point locations from prior frames and intersections point locations of new frames.

15. The system of claim 14, wherein the creative design controller employ triangulation to determine a location of the infrared transmitter and gains back precision lost by the cameras relative to the high resolution display by locating the bright spot of the transmitter, locating a peak in a bandpass filtered image of the bright spot, and computing a center of mass of the bandpass filtered image to determine the sub-pixel estimate for the position of the bright spot.

16. The system of claim 14, further comprising a gesture library, wherein the creative design controller compares images captured by the infrared cameras to determine if gesture movements of at least one of the interactive stylus and a user'"'"'s hand match a gesture in the gesture library, and performs an associated function if gesture movements of at least one of the interactive stylus and a user'"'"'s hand matched a given gesture in the gesture library.

17. The system of claim 14, wherein the creative design controller further comprises a calibration component that displays a sequence of targets on the high resolution display, prompts the user to touch the interactive stylus to the targets in a predetermined order and employs images captured by the plurality of cameras to determine and calibrate one or more parameters of the plurality of cameras employing the captured images and further employs histograms to determine and calibrate a distance location of the transmitter of the high resolution display to determine when the transmitter is making contact with the display and when the transmitter is not making contact with the high resolution display.

18. A method for providing creative design, the method comprising:

capturing images of a transmitter of an interactive stylus with a plurality of cameras as it is moved over a high resolution display;

analyzing the captured images and employing triangulation to determine a location of the transmitter of the interactive stylus;

generating instructions to display sketches of content in response to the determined movement of the transmitter of the interactive stylus over the high resolution display; and

tracking camera intersection point locations from prior frames of two or more of the cameras and estimating the location of a new intersection point based on weighted averages of the intersection points locations from prior frames and intersections point locations of new frames in response to a camera of the plurality of cameras failing to capture an image of the transmitter of the interactive stylus.

19. The method of claim 18, further comprising locating a bright spot in the captured images, locating a peak in a bandpass filtered image of the bright spot, computing a center of mass of the bandpass filtered image to determine a sub-pixel estimate for the position of the bright spot.

20. The method of claim 18, further comprising determining if gesture movements of at least one of the interactive stylus and a user'"'"'s hand match a gesture in the gesture library, and performs an associated function if gesture movements of at least one of the interactive stylus and a user'"'"'s hand matched a given gesture in the gesture library.

21. The method of claim 18, further comprising preloading and displaying detailed text and figures on the high resolution display.

22. The method of claim 18, further comprising displaying a sequence of targets on the high resolution display, prompting the user to touch the interactive stylus to the targets in a predetermined order and employing images captured by the plurality of cameras to determine and calibrate one or more parameters of the plurality of cameras employing the captured images.

23. The method of claim 18, further comprising employing data provided from the plurality of sensors to determine and calibrate a distance between the transmitter and a surface of the high resolution display to determine if the transmitter is within a threshold distance of the high resolution display and if the transmitter is not within the threshold distance of the high resolution display, wherein the distance is measured on an axis normal to the surface.

24. The method of claim 18, further comprising employing histograms associated with different cell regions of the high resolution display to determine and calibrate the distance between the transmitter and the surface of the high resolution display to determine if the transmitter is within a threshold distance from the high resolution display and if the transmitter is not within the threshold distance from the high resolution display in a particular cell region of the surface of the high resolution display.