APPARATUS AND METHOD FOR EXECUTING AN AUTOMATED ANALYSIS OF DATA, IN PARTICULAR SOCIAL MEDIA DATA, FOR PRODUCT FAILURE DETECTION

• US 20170091289A1
• Filed: 09/15/2016
• Published: 03/30/2017
• Est. Priority Date: 09/30/2015
• Status: Active Grant

• Current Assignee: Hitachi, Ltd.
• Original Assignee: Hitachi, Ltd.
• Inventors: OHAZULIKE, Anthony Emeka, TOMATIS, Andrea, LIN, Lan
• Granted Patent: US 10,275,407 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2465 Query processing support for facilitating data mining operations in structured databases
• CPC Class Codes: G06F 40/279 Recognition of textual entities
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 7/20 Comparing separate sets of record carriers arranged in the same sequence to determine whether at least some of the data in one set is identical with that in the other set or sets
• CPC Class Codes: G06Q 10/20 Administration of product repair or maintenance
• CPC Class Codes: G06Q 50/01 Social networking
• CPC Class Codes: H04L 51/52 for supporting social networking services

Abstract

An apparatus and a method for executing an automated analysis of analysis input data (e.g. social media data and/or On-Board-Diagnosis data) for product failure detection is proposed. Data analysis processing is performed, including: word count processing to determine word count numbers indicative of occurrence frequencies for keywords of a database in user-created text documents of the social media data; correlation determination processing to determine, for each of a plurality of keyword pairs, a respective correlation coefficient; correlation-link identification processing to identify correlation-linked keyword pairs for which the determined correlation coefficient exceeds a correlation threshold; and correlation group identification processing to identify correlation groups of keywords based on the identified correlation-linked keyword pairs; and, if one or more correlation groups of keywords are identified, analysis result data indicative of at least one of the one or more identified correlation groups of keywords is output.

Claims

1. A method for an automated data analysis, comprising:

providing one or more databases indicative of a plurality of keywords;

providing analysis input data obtained from one or more data sources, and pre-processing the analysis input data to generate pre-processed analysis input data available for data analysis processing, the analysis input data including a plurality of text documents respectively being associated with at least one of a plurality of data samples;

performing data analysis processing of the pre-processed analysis input data, including;

word count processing to determine word count numbers indicative of occurrence frequencies for keywords of the one or more databases in the text documents of the pre-processed analysis input data for each of the plurality of data samples,

correlation determination processing to determine, for each of a plurality of keyword pairs, a respective correlation coefficient being associated with the respective keyword pair, the respective correlation coefficient being indicative of a quantitative measure of correlation between the determined word count numbers of the keywords of the respective keyword pair for the plurality of data samples,

correlation-link identification processing to identify correlation-linked keyword pairs, wherein keywords of a keyword pair are determined to be correlation-linked to each other based on a correlation criteria, the correlation criteria including a criteria whether the determined correlation coefficient associated with the respective keyword pair exceeds a correlation threshold, and

correlation group identification processing to identify correlation groups of keywords based on the identified correlation-linked keyword pairs, each correlation group including keywords of at least one correlation-linked keyword pair and, for each keyword included in the respective correlation group, the respective correlation group further includes the other keywords identified to be correlation-linked to the respective keyword; and

outputting, if one or more correlation groups of keywords are identified, analysis result data indicative of at least one of the one or more identified correlation groups of keywords.

2. The method according to claim 1, wherein

the data analysis processing further includes:

omitting keywords, which fail to meet a frequency criteria, prior to the correlation determination processing;

the frequency criteria including a criteria whether the determined word count number for the respective keyword for at least one of the data samples exceeds a frequency threshold and/or a criteria whether an average word count number for the respective keyword for the plurality of data samples exceeds the frequency threshold.

3. The method according to claim 2, wherein

the data analysis processing further includes:

adjusting the frequency threshold applied to determined word count numbers for a certain data sample based on a total number of text documents being associated with the certain data sample;

—

adjusting the frequency threshold applied to determined average word count numbers for the plurality of data samples based on a total number of text documents of the plurality of data samples; and

/or

adjusting the frequency threshold based on one or more credibility weighting factors indicative of a credibility level of one or more users as creators of one or more of the text documents and/or of a credibility level of one or more data sources.

4. The method according to claim 1, wherein

the data analysis processing further includes:

distance determination processing to determine, for each of the plurality of keyword pairs, a respective distance coefficient being associated with the respective keyword pair, the respective distance coefficient being indicative of a relative distance of the keywords of the respective keyword pair in each respective text document of the pre-processed analysis input data;

wherein the correlation criteria applied in the correlation-link identification processing further includes a criteria whether the determined distance coefficient associated with the respective keyword pair exceeds a distance threshold.

5. The method according to claim 4, wherein

the distance coefficient being associated with the respective keyword pair is determined in a range between a maximum distance coefficient value, which indicates that the keywords of the respective keyword pair occur together, in particular close or next to each other, in plural text documents, and a minimum distance coefficient value, which indicates that the keywords of the respective keyword pair do not occur together in any of the text documents.

6. The method according to claim 4, wherein

the providing or more databases comprises:

providing a first database indicative of a plurality of keywords for the word count processing performed for the correlation determination processing, and

providing a second database indicative of a plurality of keywords for the word count processing performed for the distance determination processing,

wherein the second database includes all keywords included in the first database and additional keywords not included in the first database;

wherein the method further includes a machine learning process which comprises;

updating the first database by adding one or more keywords from the second database to the first database, if the distance determination processing identifies one or more keyword pairs which fulfill the criteria whether the determined distance coefficient associated with the respective keyword pair exceeds a distance threshold and which include a respective keyword which is already included in the first database and another respective keyword which was previously not included in the first database; and

repeating the word count processing for the correlation determination processing and the correlation determination processing based on the one or more added keywords of the updated first database.

7. The method according to claim 1, wherein

at least one of the one or more databases includes a plurality of search terms being surjectively mapped onto the plurality of keywords, such that each search term is mapped to one of the plurality of keywords and each keyword is associated with one or more of the plurality of search terms being mapped to the respective keyword;

wherein the pre-processing to generate the pre-processed analysis input data includes identifying occurrences of search terms in the text documents and replacing the identified search terms by their respective associated keywords based on the at least one of the one or more databases;

or

wherein word count processing includes determining word count numbers indicative of occurrence frequencies for search terms of the at least one of the one or more databases in the text documents of the pre-processed analysis input data for each of the plurality of data samples, and adding, for each of the plurality of data samples, the determined word count numbers for all search terms mapped to the same keyword to determine the respective word count number for the respective keyword.

8. The method according to claim 7, further comprising

analyzing correlation groups of the analysis result data and adjusting the mapping between search terms and keywords based on machine learning on the basis of the analysis of the analysis result data.

9. The method according to claim 1, wherein

at least one of the one or more databases is grouped into plural categories, each keyword being associated with one of the plural categories;

the categories including one or more of;

a manufacturer category including keywords associated with manufacturer'"'"'s names, in particular names of automotive manufacturers,

a product category including keywords associated with products or product models, in particular automobile models,

a part category including keywords associated with parts of products or product models, in particular automobile parts,

a malfunction category including keywords associated with product malfunction or failures,

a category including keywords associated with problems being resolved and/or normal functioning of a product or product part,

a weather category including keywords associated with weather attributes,

a location category including keywords associated with locations, in particular regions, countries, states, and/or cities,

a road type category including keywords associated with road types and/or road conditions, —

a driving maneuver action category including keywords associated with driving maneuver actions performed by a vehicle and/or by a driver of the vehicle, and

a problem-solution-feedback category including keywords associated with one or more solutions to one or more problems.

10. The method according to claim 1, further comprising:

grouping at least one of the one or more databases into plural categories, each keyword being associated with one of the plural categories; and

identifying one or more correlation groups of keywords which contain at least one keyword being associated with each of a pre-determined set of two of more categories of the at least one of the one or more databases.

11. The method according to claim 1, further comprising:

adjusting the correlation criteria based on one or more of the criteria;

if no correlation-linked keyword pair is identified in the correlation-link identification processing,

if no correlation group is identified in the correlation group identification processing,

if the maximum number of keywords occurring in the identified correlation groups is lower than a pre-set threshold, and/or

if the average number of keywords in the identified correlation groups is lower than a pre-set threshold; and

repeating the correlation-link identification processing and the correlation group identification processing based on the adjusted correlation criteria.

12. The method according to claim 10, further comprising:

adjusting the correlation criteria if no correlation group, which contains at least one keyword being associated with each or at least a pre-determined number of categories of a pre-determined set of two of more categories of the database, is identified; and

repeating the correlation-link identification processing and the correlation group identification processing based on the adjusted correlation criteria.

13. The method according to according to claim 11, further comprising:

adjusting the correlation threshold, a frequency threshold, and/or a distance threshold.

14. A computer program product including computer program instructions stored on a computer-readable medium which, when executed on a computer or computer system, are configured to execute, on the computer or computer system, the steps of a method according to claim 1.

15. An apparatus for executing an automated data analysis, comprising:

a storage configured to store one or more databases indicative of a plurality of keywords;

a data input interface configured to provide analysis input data obtained from one or more data sources, the analysis input data including a plurality of text documents respectively being associated with at least one of a plurality of data samples;

a processing system configured to execute pre-processing the analysis input data to generate pre-processed analysis input data available for data analysis processing, and the processing system being configured to execute the data analysis processing of the pre-processed analysis input data, including;

word count processing to determine word count numbers indicative of occurrence frequencies for keywords of the one or more databases in the text documents of the pre-processed analysis input data for each of the plurality of data samples,

correlation determination processing to determine, for each of a plurality of keyword pairs, a respective correlation coefficient being associated with the respective keyword pair, the respective correlation coefficient being indicative of a quantitative measure of correlation between the determined word count numbers of the keywords of the respective keyword pair for the plurality of data samples,

correlation-link identification processing to identify correlation-linked keyword pairs, wherein keywords of a keyword pair are determined to be correlation-linked to each other based on a correlation criteria, the correlation criteria including a criteria whether the determined correlation coefficient associated with the respective keyword pair exceeds a correlation threshold, and

correlation group identification processing to identify correlation groups of keywords based on the identified correlation-linked keyword pairs, each correlation group including keywords of at least one correlation-linked keyword pair and, for each keyword included in the respective correlation group, the respective correlation group further includes the other keywords identified to be correlation-linked to the respective keyword; and

a data output interface configured to output, if one or more correlation groups of keywords are identified, analysis result data indicative of at least one of the one or more identified correlation groups of keywords.