SYSTEM FOR GENERATING FUNCTIONALITY REPRESENTATION, INDEXING, SEARCHING, COMPONENTIZING, AND ANALYZING OF SOURCE CODE IN CODEBASES AND METHOD THEREOF

• US 20180373507A1
• Filed: 08/03/2018
• Published: 12/27/2018
• Est. Priority Date: 02/03/2016
• Status: Active Grant

• Current Assignee: Cocycles Co.
• Original Assignee: Cocycles Co.
• Inventors: MIZRAHI, Ran, BAR-ILAN, Laly
• Granted Patent: US 10,809,984 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 40/211 Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
• CPC Class Codes: G06F 40/242 Dictionaries
• CPC Class Codes: G06F 8/36 Software reuse
• CPC Class Codes: G06F 8/433 Dependency analysis; Data or control flow analysis
• CPC Class Codes: G06F 8/75 Structural analysis for program understanding
• CPC Class Codes: G06N 7/00 Computing arrangements based on specific mathematical models

Abstract

Systems and methods are disclosed for generating functionality representation, indexing, searching, componentizing, an analyzing source code source code unit in a one or more of code repositories. The systems and method include one or more of crawling a set of data entities in a repository system; parsing said set of data entities into abstract syntax trees (ASTs) architecture; modeling said set of data entities into a code graph (CG) architecture; establishing type ontology (TO) architecture of said set of data entities; organizing and storing said set of data entities in functionality representation index (FRI) architecture; componentizing one or more projects in the repositories into code components; and making the components discoverable by functionality and analyzable for performance, usage volume, etc.

Claims

1. A computer-implemented method for organizing, functionality indexing and constructing a source code search engine, the method comprising:

a. crawling a set of data entities in a repository system, each of the data entities representing one or more of a source code units and/or subsets of the source code units;

b. parsing said set of data entities into abstract syntax trees (ASTs) architecture;

c. modeling said set of data entities into a code graph (CG) architecture such that each one or more of a source code units and/or subsets of the source code units are set as vertices and connections between said each one or more of a source code units and/or subsets of the source code units are set as edges;

d. establishing type ontology (TO) architecture of said set of data entities by processing said set of data and assigning meta-data tags to each one or more of a source code units and/or subsets of the source code units, said tags representing classification attributes;

e. generating semantic ID based on linguistic, structural and contextual analyses of said set of data entities, said semantic ID corresponding to source code functionality of said one or more of a source code units and/or subsets of the source code units, said linguistic analysis employing linguistic clues, said structural linguistic analysis employing structural clues, and said contextual analysis employing contextual clues; and

f. organizing and storing said set of data entities in functionality representation index (FRI) architecture.

2. The method of claim 1, wherein the data entity is selected from the group consisting of integers, strings, Booleans, characters, files, arrays, lists, maps, and tables and any combinations thereof.

3. The method of claim 1, further comprising storing said set of data entities in one or more storage search server systems.

4. A computer-implemented method for automatically extracting and characterizing code components from a software project, comprising steps of

a. obtaining one or more software projects stored in a non-transitory computer readable medium;

b. scraping said projects;

c. detecting one or more programming languages of each of said projects;

d. detecting one or more environments of each said project;

e. parsing each file in each said project to obtain an abstract syntax tree of each said file;

f. identifying potential components in each said abstract syntax tree, said potential components comprising single nodes and/or collectivized nodes in said abstract syntax tree;

g. parsing said project across said files of each said project to obtain a project dependency graph of said project, said files being additional potential components of said project;

h. analyzing usage patterns of said potential components in said ASTs and said project dependency graph;

i. associating each said potential component with metadata comprising said usage patterns of said component;

j. generating a functionality representation for each potential component by analyzing said usage patterns, said dependencies and linguistic, contextual, and structural information of potential component;

k. appending said functionality representation each said potential component to said metadata of said potential component;

l. classifying a subset of the potential components as code components, using a statistical model trained on componentized projects;

said statistical model employing said metadata as features;

m. creating a component dependency graph, each node of said component dependency graph associated with a said code component;

n. matching the code components with test files of the project, if any, and appending matched test files to said metadata of matched code components;

o. matching the code components with asset files of the project, if any and appending matched asset files to said metadata of matched code components;

p. assigning a unique name to each said code component;

q. classifying each said code component as an external or internal component;

r. analyzing file dependencies of each said code component to identify a main file, if any, of said code component;

s. creating edges between component nodes in the component dependency graph, said edges associated with said dependencies and with metadata regarding types of connections between said code components of said component nodes.

5. The method of claim 4, wherein said code components comprise code units associated with said metadata rendering said code components consumable and maintainable as independent software entities.

6. The method of claim 4, wherein said metadata of each component further comprises package dependencies of said project from which said component was extracted.

7. The method of claim 4, wherein said potential code components comprise folders, files, snippets, regular expressions, or any combination thereof.

8. The method of claim 4, wherein said usage statements comprise any in a group consisting of file imports, function calls, class usage, and any combination thereof.

9. The method of claim 4, wherein the project dependency graph is language agnostic.

10. The method of claim 4, wherein said usage patterns of a said potential component comprise any in a group consisting of:

how many times said potential component is required, whether said potential component is required by siblings of said potential component in a same directory as said potential component, whether said potential component is required by a potential component in a parent directory, whether said potential component is required by a potential component in a cousin directory, whether said potential component is required as a whole or one or more parts of said potential component is required, which potential components requires said potential component, in which potential component a class instance used by said potential component is created, which potential components utilizes a class instance created by said potential component, code logic, structure, documentation, variable names, namespaces, file names, directory names, naming patterns across the project, and any combination thereof.

11. The method of claim 4, wherein said statistical model employs characteristics of said potential components selected from said usage patterns, names, file type, or any combination thereof.

12. The method of claim 4, further comprising a step of using said functionality representation to ameliorate said classification.

13. The method of claim 4, wherein said internal and external components are assigned an extent, over a distributed range, to which each said code component is reusable in other projects.

14. A computer-implemented method for providing discoverability and analytics for software components in a codebase, comprising steps of

a. obtaining one or more component dependency graphs of code components stored in a code repository, said code components each associated with metadata comprising a functionality representation of said component 705;

b. indexing said functionality representations;

c. indexing language models;

d. indexing dictionaries, lexicons, and ontologies;

e. grouping code components from the component dependency graphs into functional clusters;

f. storing said functional clusters in a functionality representation index (FRI);

g. mapping a natural language query to one or more functional identifiers (FRIDs) in said FRI and producing a set of ranked results;

h. analyzing said code components; and

i. displaying results of analyzing code components and insights and analysis regarding entire projects and codebase repositories with more than one project.

15. The method of claim 14, wherein a said functionality is defined by features of a said component selected from a group comprising actions performed, entities performed on, interaction with other components and APIs, complexity, execution time, test results, and any combination thereof.

16. The method of claim 14, further comprising a step of applying a statistical n-gram model to programming terms among said semantic functionalities, thereby generating a said dictionary of said functionalities.

17. The method of claim 14, further comprising receiving a said query through a client to a storage search server system from a user.

18. The method of claim 14, further comprising a step of tagging each word or phrase in said query, each tag representing semantic role of said each word in said query.

19. The method of claim 14, further comprising a step of classifying the words in said query into one or more sets of defined query types.

20. The method of claim 14, wherein said step of analyzing comprises any in a group consisting of analysis of an organization'"'"'s:

projects, codebase, list of structurally and functionally duplicate components, components created by an individual developer, most widely used components, components best implementing a said functionality, and any combination thereof.