Process for Providing and Editing Instructions, Data, Data Structures, and Algorithms in a Computer System

• US 20110115702A1
• Filed: 07/09/2009
• Published: 05/19/2011
• Est. Priority Date: 07/08/2008
• Status: Abandoned Application

• Current Assignee: David Seaberg
• Original Assignee: David Seaberg
• Inventors: Seaberg, David
• Application Number:
  US13/003,009
• Publication Number:
  US 20110115702A1
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence

Abstract

A method and system for computer programming using speech and one or two hand gesture input is described. The system generally uses a plurality of microphones and cameras as input devices. A configurable event recognition system is described allowing various software objects in a system to respond to speech and hand gesture and other input. From this input program code is produced that can be compiled at any time. Various speech and hand gesture events invoke functions within programs to modify programs, move text and punctuation in a word processor, manipulate mathematical objects, perform data mining, perform natural language interne search, modify project management tasks and visualizations, perform 3D modeling, web page design and web page data entry, and television and DVR programming.

Claims

1. A method of computer programming comprising:

interpreting hand gestures as programming input; and

interpreting spoken utterances as programming input.

2. The method of claim 1, further comprising receiving and resolving references implied in programming input.

3. The method of claim 1, further comprising searching at least one of local objects, local libraries, and network libraries to match metadata to programming input.

4. The method of claim 1, further comprising identifying functions similar in metadata to programming input intent.

5. The method of claim 1, further comprising a disambiguation process.

6. The method of claim 1, further comprising producing instructions from programming input.

7. The method of claim 1, further comprising execution of a function corresponding to matched metadata with programming input.

8. The method of claim 1, further comprising style naming.

9. The method of claim 1, further comprising defining of inheritance relationship between entities.

10. The method of claim 1:

further comprising adding metadata to any programming language element.

11. The method of claim 1:

further comprising mapping fields between two system objects.

12. The method of claim 1:

further comprising rearranging instructions.

13. The method of claim 1:

further comprising parallelizing a set of instructions.

14. The method of claim 1:

further comprising defining a grammar.

15. The method of claim 1:

further comprising displaying speech and gesture enabled menu areas.

16. The method of claim 1:

further comprising entering and modifying operational, axiomatic, and denotational semantics.

17. The method of claim 1:

further comprising editing of instructions and data while a program is stopped, paused, or running

18. The method of claim 1:

further comprising modifying a set of instructions from the modification of the output of a set of instructions.

19. The method of claim 1:

further comprising modifying a set of properties.

20. The method of claim 1:

further comprising diagramming an executable state machine

21. The method of claim 1:

further comprising diagramming an executable sequence diagram.

22. A method of data and event processing comprising:

allocation of computer system resources to sensor input;

transforming sensor data into broadcast or narrowcast application data for event recognition;

recognizing events from transformed sensor data; and

sending of event notifications and data to a plurality of objects.

23. The method of claim 22:

further comprising facilitating the configuration of said data and event processing by means of a programming interface or a speech and hand gesture enabled graphical user interface.

24. The method of claim 22:

further comprising defining speech and hand gesture example patterns used by recognizers to generate events.

25. The method of claim 23:

further comprising selecting an interpretation method from said programming or said speech and hand gesture enabled graphical user interface.

26. The method of claim 23:

further comprising selecting of both left and right hands to be used by the recognizers.

27. The method of claim 23:

further comprising defining specific event names.

28. The method of claim 23:

further comprising selecting what data is used and routed by objects and recognizers.

29. The method of claim 23:

further comprising adding an event handler.

30. The method of claim 23:

further comprising adding a recognizer.

31. A method comprising finding parts of hands on one or more hands using light patterns from one or more cameras.

32. The method of claim 31:

further comprising determining start points for traversing individual fingers.

33. The method of claim 32:

further comprising sampling skin near a finger traversal start point.

34. The method of claim 32:

further comprising traversing a finger using a best point in presence of rings, wrinkles, tattoos, hair, or other foreign elements.

35. The method of claim 32:

further comprising identifying a finger tip by means of a configurable set of tip detectors.

36. The method of claim 32:

further comprising estimating the positions missing fingers.

37. The method of claim 35:

further comprising using a safe distance.

38. The method of claim 35:

further comprising using a look ahead distance.

39. A system comprising:

at least one image sensor and at least one microphone;

a module to transform sensor data into broadcast or narrowcast application data for event recognition;

a set of speech and hand gesture recognizers;

a set of computer applications enabled to receive speech and hand gesture event input.

40. The system of claim 39, wherein the computer application is an integrated development environment.

41. The system of claim 39, wherein the computer application has facilities determining punctuation and text location within a document from speech and hand gesture input.

42. The system of claim 39, wherein the computer application has facilities wherein speech and hand gesture input determines mathematical operations performed on an object.

43. The system of claim 42, wherein the operations are one of selection and replacement, factoring, combining, decomposing, multiplication, division, addition, subtraction, direct entry, group selection, inverse, transpose, random access, matrix row/column changes, union, intersection, difference, complement, Cartesian product, term rearrangement, and equation and visualization modification.

44. The system of claim 39, wherein the computer application manipulates spreadsheets.

45. The system of claim 44, wherein the spreadsheet application modifies spreadsheet cell data and functions through speech and hand gesture events.

46. The system of claim 39, wherein the computer application builds presentations.

47. The system of claim 39, wherein the computer application performs data mining.

48. The system of claim 39, wherein the computer application performs project management.

49. The system of claim 48, wherein the entry of task names, start and finish dates, and timeline visualizations are manipulated with speech and hand gesture input.

50. The system of claim 39, wherein the computer application performs data compression.

51. The system of claim 39, wherein the computer application performs game application design.

52. The system of claim 51, wherein the game is configured to receive speech and hand gestures for baseball signs.

53. The system of claim 39, wherein the computer application performs continuous actions from a continue hand gesture.

54. The system of claim 39, wherein the computer application performs a reversing action from a reversing hand gesture.

55. The system of claim 39, wherein the computer application performs one of control point movement, multiple control point selection, extrusion, forward and inverse kinematic limit determination.

56. The system of claim 39, wherein the computer application facilitates an internet search.

57. The system of claim 56, wherein the computer application performs natural language query from speech and hand gesture input.

58. The system of claim 39, wherein the computer application facilitates entering data on a web page.

59. The system of claim 39, wherein the computer application facilitates the entry of instructions to record audio and video, determines the channel number, and the order of media playback through speech and hand gesture events.

60. The system of claim 59, wherein the set of gestures comprise fine and course channel increment and decrement, and reverse direction.

61. The system of claim 39, wherein the computer application performs one of pausing of a dialog, or undoing an operation from speech and hand gesture input.

62. The system of claim 39, wherein the computer application facilitates an optimization hierarchical to do list.

63. The system of claim 39:

wherein the computer application displays speech and hand gesture enabled menu areas.

64. The system of claim 39:

wherein said system is embedded in one of a desktop computer, a communication enabled slate computer, a communication enabled portable computer, a communication enabled car computer, a communication enabled wall display, a communication enabled whiteboard.