Method and apparatus for customizing word segmentation model based on artificial intelligence, device and medium

• US 10,643,033 B2
• Filed: 03/30/2018
• Issued: 05/05/2020
• Est. Priority Date: 06/14/2017
• Status: Active Grant

• Current Assignee: Beijing Baidu Netcom Science and Technology Company Limited (Baidu Incorporated)
• Original Assignee: Beijing Baidu Netcom Science and Technology Company Limited (Baidu Incorporated)
• Inventors: Zheng, Liqun, Zhan, Jinbo, Xiao, Qiugen, Fu, Zhihong, He, Jingzhou, Zhou, Guyue
• Primary Examiner(s): Patel, Shreyans A
• Application Number:
  US15/941,065
• Publication Number:
  US 20180365227A1
• Time in Patent Office: 767 Days
• CPC Class Codes: G06F 40/279 Recognition of textual entities
• CPC Class Codes: G06F 40/30 Semantic analysis

Abstract

Embodiments of the present disclosure disclose a method and an apparatus for customizing a word segmentation model based on artificial intelligence, a device and a medium. The method includes the followings. A customized segmentation training corpus is acquired. A first preset word segmentation model is rectified with an increment training method or a weight intervention method, based on the customized segmentation training corpus, to obtain a customized word segmentation model corresponding to the customized segmentation training corpus.

Claims

1. A method for customizing a word segmentation model based on artificial intelligence, comprising:

acquiring a customized segmentation training corpus;

rectifying a first preset word segmentation model with an increment training method or a weight intervention method, based on the customized segmentation training corpus, to obtain a customized word segmentation model corresponding to the customized segmentation training corpus,

wherein, rectifying the first preset word segmentation model with the increment training method or the weight intervention method based on the customized segmentation training corpus to obtain the customized word segmentation model corresponding to the customized segmentation training corpus comprises;

segmenting the customized segmentation training corpus with the first preset word segmentation model to obtain a first segmentation result;

comparing the first segmentation result with the customized segmentation training corpus based on a preset rule, to obtain a first update parameter; and

updating the first preset word segmentation model based on the first update parameter, to obtain the customized word segmentation model corresponding to the customized segmentation training corpus.

2. The method according to claim 1, wherein, rectifying the first preset word segmentation model with the increment training method or the weight intervention method based on the customized segmentation training corpus to obtain the customized word segmentation model corresponding to the customized segmentation training corpus comprises:

generating a word segmentation model rectification parameter based on the customized segmentation training corpus; and

generating the customized word segmentation model corresponding to the customized segmentation training corpus based on the word segmentation model rectification parameter and a model parameter of the first preset word segmentation model.

3. The method according to claim 1, further comprising:

storing the customized segmentation training corpus and the customized word segmentation model.

4. The method according to claim 3, further comprising:

acquiring all customized segmentation training corpuses stored based on a preset time period and an acquired rectification instruction, to generate a standard segmentation training corpus; and

rectifying a second preset word segmentation model with the increment training method based on the standard segmentation training corpus.

5. The method according to claim 3, wherein storing the customized word segmentation model comprises:

storing the customized word segmentation model in a preset storage area; and

based on a storage rule, outputting the customized word segmentation model to a client terminal, and/or storing the customized word segmentation model in a preset external server.

6. A device, comprising:

one or more processors;

a storage device, configured to store one or more programs,

wherein the one or more processors are configured to execute the one or more programs by reading the one or more programs from the storage device, to perform acts of;

acquiring a customized segmentation training corpus;

rectifying a first preset word segmentation model with an increment training method or a weight intervention method, based on the customized segmentation training corpus, to obtain a customized word segmentation model corresponding to the customized segmentation training corpus,

wherein, the one or more processors are configured to rectify the first preset word segmentation model with the increment training method or the weight intervention method based on the customized segmentation training corpus to obtain the customized word segmentation model corresponding to the customized segmentation training corpus by acts of;

segmenting the customized segmentation training corpus with the first preset word segmentation model to obtain a first segmentation result;

comparing the first segmentation result with the customized segmentation training corpus based on a preset rule, to obtain a first update parameter; and

updating the first preset word segmentation model based on the first update parameter, to obtain the customized word segmentation model corresponding to the customized segmentation training corpus.

7. The device according to claim 6, wherein the one or more processors are configured to rectify the first preset word segmentation model with the increment training method or the weight intervention method based on the customized segmentation training corpus to obtain the customized word segmentation model corresponding to the customized segmentation training corpus by acts of:

generating a word segmentation model rectification parameter based on the customized segmentation training corpus; and

generating the customized word segmentation model corresponding to the customized segmentation training corpus based on the word segmentation model rectification parameter and a model parameter of the first preset word segmentation model.

8. The device according to claim 6, wherein the one or more processors are configured to perform acts of:

storing the customized segmentation training corpus and the customized word segmentation model.

9. The device according to claim 8, wherein the one or more processors are configured to perform acts of:

acquiring all customized segmentation training corpuses stored based on a preset time period and an acquired rectification instruction, to generate a standard segmentation training corpus; and

rectifying a second preset word segmentation model with the increment training method based on the standard segmentation training corpus.

10. The device according to claim 8, wherein the one or more processors are configured to store the customized word segmentation model by acts of:

storing the customized word segmentation model in a preset storage area; and

based on a storage rule, outputting the customized word segmentation model to a client terminal, and/or storing the customized word segmentation model in a preset external server.

11. A non-transitory storage medium having computer executable instructions, wherein when the computer executable instructions are executed by a processor of a computer, a method for customizing a word segmentation model based on artificial intelligence is executed, the method comprising:

acquiring a customized segmentation training corpus;

rectifying a first preset word segmentation model with an increment training method or a weight intervention method, based on the customized segmentation training corpus, to obtain a customized word segmentation model corresponding to the customized segmentation training corpus,

wherein, rectifying the first preset word segmentation model with the increment training method or the weight intervention method based on the customized segmentation training corpus to obtain the customized word segmentation model corresponding to the customized segmentation training corpus comprises;

segmenting the customized segmentation training corpus with the first preset word segmentation model to obtain a first segmentation result;

comparing the first segmentation result with the customized segmentation training corpus based on a preset rule, to obtain a first update parameter; and

updating the first preset word segmentation model based on the first update parameter, to obtain the customized word segmentation model corresponding to the customized segmentation training corpus.

12. The non-transitory storage medium according to claim 11, wherein, rectifying the first preset word segmentation model with the increment training method or the weight intervention method based on the customized segmentation training corpus to obtain the customized word segmentation model corresponding to the customized segmentation training corpus comprises:

generating a word segmentation model rectification parameter based on the customized segmentation training corpus; and

generating the customized word segmentation model corresponding to the customized segmentation training corpus based on the word segmentation model rectification parameter and a model parameter of the first preset word segmentation model.

13. The non-transitory storage medium according to claim 11, wherein, the method further comprises:

storing the customized segmentation training corpus and the customized word segmentation model.

14. The non-transitory storage medium according to claim 13, wherein, the method further comprises:

acquiring all customized segmentation training corpuses stored based on a preset time period and an acquired rectification instruction, to generate a standard segmentation training corpus; and

rectifying a second preset word segmentation model with the increment training method based on the standard segmentation training corpus.

15. The non-transitory storage medium according to claim 13, wherein, storing the customized word segmentation model comprises:

storing the customized word segmentation model in a preset storage area; and

based on a storage rule, outputting the customized word segmentation model to a client terminal, and/or storing the customized word segmentation model in a preset external server.