Virtualization of interactive computer input

• US 6,433,774 B1
• Filed: 12/04/1998
• Issued: 08/13/2002
• Est. Priority Date: 12/04/1998
• Status: Expired due to Term

• Current Assignee: Aurora Operations Incorporated
• Original Assignee: Intel Corporation
• Inventors: DʼHooge, Herman D.
• Primary Examiner(s): Nguyen, Cao H.
• Application Number:
  US09/205,895
• Time in Patent Office: 1,348 Days
• Field of Search: 345/330, 345/331, 345/334, 345/349, 345/348, 345/973, 345/968, 345/962, 345/753, 345/751, 345/765, 345/839, 345/835, 345/786, 345/764, 345/171, 345/173, 345/113, 345/419, 345/156, 345/157
• US Class Current: 345/157
• CPC Class Codes: G06F 3/0426 tracking fingers with respect to a virtual keyboard projected or printed on the surface virtual keyboards on touch screens G06F3/04886
• CPC Class Codes: G06F 3/0489 using dedicated keyboard keys or combinations thereof
• CPC Class Codes: G06F 3/04895 Guidance during keyboard input operation, e.g. prompting

Abstract

Methods of providing user input to an interactive computer application using video input in association with touch input. The methods include generating a control view having a control image, wherein the control image is associated with an application command; generating a foreground view of a controlling entity superimposed on the control view, wherein an image location of the foreground view within the control view is indicative of a physical location of the controlling entity adjacent a touch input device; and producing an output capable of selecting the application command when the touch input device is activated by the controlling entity at the physical location where the foreground view is aligned with the control image. Machine-readable medium having instruction stored thereon capable of causing a processor to carry out the methods of providing user input. Apparatus adapted to carry out the methods of providing user input.

Claims

1. A system for providing input from a physical controlling entity positionable by a user to an application executing on the system, comprising:

a physical input device to produce commands to the application program when actuated by the physical entity;

a video input device to capture at least one image of the physical input device and to capture an action video image of the physical entity;

a monitor to display to the user a control view including the action image of the physical entity superimposed upon an image of a virtual representation of at least one of the commands at locations determined by the physical location of the input device;

a processor to generate the virtual representation of the command, and to produce the action image at a location in the control view determined by the physical entity'"'"'s position with respect to the physical input device.

2. The system of claim 1 where the control view does not include an image of the physical input device.

3. The system of claim 1 where the physical entity is a hand or a foot of the user.

4. The system of claim 1 where the processor alters the image of the physical entity before presentation in the control view.

5. The system of claim 4 where the image of the physical entity is transformed to an alternative representation of the member of the user.

6. The system of claim 1 where the monitor also displays graphics from the application.

7. The system of claim 1 where the physical input device is a keyboard having a plurality of keys that the user can actuate via the physical entity.

8. The system of claim 7 where the virtual representation is aligned with a particular one of the keys.

9. The system of claim 7 where the keyboard produces the one command when the physical entity actuates the particular one of the keys.

10. The system of claim 1 where the system further comprises a machine-readable storage medium holding instructions and data to generate the virtual representation of the command, and to produce the image of the physical entity at a location in the control view determined by the entity'"'"'s physical position with respect to the physical input device.

11. The system of claim 10 where the machine-readable medium comprises a memory incorporated into the processor.

12. The system of claim 10 where the machine-readable medium comprises a removable medium.

13. A method of providing input to an interactive application from a physical entity whose position with respect to a physical input device is controlled by a user, comprising:

capturing at least one image of the physical input device;

capturing an action image of the physical entity as it is being controlled by the user;

generating a virtual representation of a particular command for the application;

displaying both the virtual representation of the command and the action image of the entity in a control view at locations determined by their respective positions relative to the physical input device;

producing the particular command when the user causes the entity to actuate the input device at a physical location on the device such that the image of the physical entity is aligned with the virtual representation of that command in the control view.

14. The method of claim 13 where the control view does not include an image of the physical input device.

15. The method of claim 14 further comprising subtracting the image of the physical input device from the captured action image of the member of the user.

16. The method of claim 13 where the physical entity includes a hand or a foot of the user.

17. The method of claim 13 further comprising altering the action image of the physical entity before displaying it.

18. The method of claim 13 further comprising displaying graphics from the interactive application on the same monitor as the control view.

19. The method of claim 18 where the control view is displayed translucently over at least a portion of the graphics.

20. The method of claim 13 where the application is a game, a word processor, a graphic presentation, or an industrial control application.

21. The method of claim 13 where the physical input device is a touch input device.

22. The method of claim 21 where the input device is a keyboard, one of whose keys produces the command when actuated by the user via the physical entity.

23. A machine-readable medium bearing instructions for causing a digital processor to carry out the method of claim 13.

24. A method of providing a command to an application from a physical touch input device actuated by a hand and/or foot of a user, comprising:

capturing at least one background image of the physical touch input device;

capturing a foreground action image of the location of the hand and/or foot during motion caused by the user;

generating a virtual representation of the command for the computer as a control image;

displaying on a monitor both the control image of the command and the action image of the hand and/or foot of the user in a control view at locations determined by their respective positions relative to the physical input device, the background image of the physical input device itself being absent from the control view;

displaying graphics from the application on the monitor;

producing the command when the hand and/or foot of the user actuates the touch input device at a physical location on the input device such that the image of the hand and/or foot is aligned with the virtual representation of the command in the control view.

25. The method of claim 24 where the touch input device is a physical keyboard having a plurality of keys.

26. The method of claim 25 where the virtual representation is associated with one of the keys.

27. The method of claim 26 where the command is produced when the hand of the user presses the one key on the physical keyboard.

28. The method of claim 24 where the control view at least partially overlaps the graphics from the application.

29. The method of claim 28 where the control view is translucent.

30. The method of claim 24 where the background image of the physical input device is captured more than once.

31. The method of claim 24 where the background image of the physical device and the foreground image of the hand and/or foot are captured together, and the background image is subtracted from the foreground image before the displaying operation.

32. A machine-readable medium bearing instructions for causing a digital processor to carry out the method of claim 24.