Introduction

Pre-trained Language Models recently gained traction in the Natural Language Processing (NLP) domain for text summarization, generation and question answering tasks. This stems from the innovation introduced in Transformer models and their overwhelming performance compared with Recurrent Neural Network Models (Long Short Term Memory (LSTM)).

In this paper, we leverage the attention mechanism of pre-trained causal language models such as Transformer model for the downstream task of generating Failure Analysis Triplets (FATs) - a sequence of steps for analyzing defected components in the semiconductor industry. We compare different transformer models for this generative task and observe that Generative Pre-trained Transformer 2 (GPT-2) outperformed other transformer models for the failure analysis triplet generation (FATG) task.

In particular, we observe that GPT-2 (trained on 1.5B parameters) outperforms pre-trained BERT, BART and GPT-3 by a large margin on ROUGE. Furthermore, we introduce Levenshtein Sequential Evaluation metric (LESE) for better evaluation of the structured FAT data and show that it compares exactly with human judgement than existing metrics.

Failure Analysis Triplet Generation overview

Figure 1: Overview of Failure Analysis Triplet Generation process

Background: Failure Analysis in Semiconductor Industry

Root cause analysis (RCA) in semiconductor industry is the process of discovering the root causes of a failure in order to identify appropriate actions to systematically prevent and solve the underlying issues. Reliability engineers (experts) in semiconductor industry are usually tasked with carrying out RCA technique known as Failure mode and effects analysis (FMEA).

FMEA involves reviewing several components, assemblies, and subsystems to identify potential failure modes in a system and their root causes. This process is done to improve product reliability and quality; cut production cost and reduce defective parts susceptible to failures. Inspection, testing, localization, failure reproduction and documentation are amongst the major steps needed for RCA.

FRACAS: Failure Reporting, Analysis and Corrective Action System

The Failure Reporting, Analysis and Corrective Action System (FRACAS) is a closed-loop feedback path by which pertinent reliability data is collected, recorded and analyzed during in-house (laboratories) and production/operation to determine where failure concentration in the design of the equipment.

The heart of FRACAS is its database management system (DBMS) which classifies failure modes into categories that are essential in identifying the processes in the product (hardware/software) life cycle requiring the most attention for reliability improvement. The report obtained from FRACAS DBMS contains information describing the type of failure and origin of detection; a set of analysis (in the form of triplets - Step type, Substep technique and Equipment) proposed to find the failure root cause; conclusion on the outcome of failure analysis.

What are Failure Analysis Triplets?

A Failure Analysis Triplet (FAT) consists of three components:

Step type: The type of analysis step (e.g., "Sample preparation", "Physical Analysis", "Nondestructive Inspection")
Substep technique: The specific technique used (e.g., "Package decap", "SEM", "X-ray")
Equipment: The specific equipment used (e.g., "PHOENIX X-RAY NANOMEX", "CRI7", "LEICA M165C")

Example Failure Analysis Triplet

Step type: Nondestructive Inspection

Substep technique: X-ray

Equipment: PK103-PHOENIX X-RAY NANOMEX

A complete failure analysis may consist of multiple triplets in sequence, with the longest FA having 23 triplets.

Failure Analysis Triplets Generation (FATG) is a scientific process of generating a series of sequential failure analysis text associated with a failure description. We model the FATG as a sequence-to-sequence data-to-text problem where the input is the Failure Description (FDR) - structured tabular data, and the output is a long sequence of failure analysis triplets.

Pre-trained Language Models for FATG

We evaluated several pre-trained transformer models for the FATG task:

GPT-2 (Generative Pre-trained Transformer 2)

GPT-2 is a type of transformer model that uses a multi-layer Transformer decoder instead of the encoder-decoder model. GPT-2 was trained on 40G of WebText data (web pages from outbound links on Reddit) excluding Wikipedia pages. The texts are tokenized using a byte-level version of Byte Pair Encoding (BPE) and contains a vocabulary size of 50,257.

We fine-tuned three versions of GPT-2:

GPT-2 (base): 117M parameters
GPT-2 Medium: 335M parameters
GPT-2 Large: 774M-1.5B parameters

GPT-3

OPENAI-GPT-3 (175B parameters and 500B tokens) was trained on BooksCorpus dataset, which contains over 7000 unique unpublished books from a variety of genres including Adventure, Fantasy, and Romance.

BART (Bidirectional Auto-Regressive Transformer)

BART is a Bidirectional Auto-Regressive Transformer trained by corrupting text with an arbitrary noising function, and learning a model to reconstruct the original text. BART uses a standard Transformer-based neural machine translation architecture (Encoder-Decoder Transformer).

Experimental Setup

Dataset

We performed experimentation using real failure analysis data from a semiconductor industry, taking into account only successful failure analysis for one year (2019). The size of the data after preprocessing for one year (2019) is 5,809 cases.

The 10-input failure description features (also called Expert features) include: Reference, Subject, Site, Requested activity, Priority level, High confidentiality, Context, Objectives / Work description, Source of failure / request, and Source of failure (Detailed).

FATs has a dimension of R⁶⁹ with the longest FA having 23 triplets. We preprocess FDR according to the scheme presented in previous work and vectorize the joint space {x,λ} using a byte-level version of Byte Pair Encoding (BPE) from GPT-2.

Training Configuration

Experimentation was carried out on a High performance computing (HPC) cluster with 80 cores (2 × Intel Xeon E5-2698 v4 2.20GHz CPU), 512G RAM size and 8 × Nvidia V100 32G GPU.

For GPT-2 models, batch size for training and evaluation is 1; weight decay is 0.05 and the number of training epochs is 100. We use top-p = 0.95 and top-k = 10 with a normalizing temperature value of 1.9 for decoder sampling.

Results

Quantitative Evaluation

Results indicate GPT-2 models trained on the WebText dataset perform considerably better than the baseline (mini-GPT), GPT-3 and BART models. The best performing GPT-2 (GPT-2 Large) outperforms the base model by 49% on the BLEU metric; 73% better than BART and GPT-3 for FATG.

Model	BLEU-1	BLEU-3	ROUGE-1 F1	ROUGE-L F1	LESE-1 F1	LESE-3 F1
mini-GPT	11.54	11.22	12.63	11.52	7.11	0.30
BART	6.14	11.24	10.16	9.43	4.08	1.28
GPT-3	6.10	7.07	8.84	7.91	5.46	0.42
GPT-2 (base)	22.18	30.25	29.67	27.75	20.97	10.49
GPT-2 Medium	22.15	29.89	29.69	27.78	21.21	10.74
GPT-2 Large	22.46	29.73	29.82	27.93	21.25	10.73

All scores are percentages. Higher values are preferred for all metrics except Levenshtein distance (Lev). GPT-2 Medium performs best on LESE-3 triplet score, closely followed by GPT-2 Large.

Figure 2: Performance comparison of different transformer models for FATG task

Key Finding

GPT-2 Medium and Large models perform best on LESE-1 and LESE-3 triplet scores, which corresponds to human judgement. The LESE metric shows that GPT-2 models are trained on correlating domain knowledge related to failure analysis and reliability engineering, enabling adaptive transfer of knowledge for failure analysis triplets generation.

Qualitative Evaluation: Short vs Long Sequence FATG

We classify generation difficulty into two classes: (i) Short-sequence FATG (SS-FATG) and (ii) Long-sequence FATG (LS-FATG).

Short Sequence FATG (SS-FATG)

Given a FDR prompt and a short human-expert failure analysis triplets (with length ranging between 3-6), Short sequence FATG is the ability of a fine-tuned PLM to generate the same order and length of the FATs given by human-expert.

We observe that all models including mini-GPT (non-pretrained baseline), BART, GPT-2 (base), GPT-2 Medium and GPT-2 Large are capable of generating short sequences. However, BART generates incorrect SS-FATs while GPT-3 generates mostly long sequences with low BLEU and LESE scores.

GPT-2 Medium achieves perfect LESE-1 and LESE-3 scores of 100% for some short sequence cases, generating highly correlating FATs as the human expert.

Long Sequence FATG (LS-FATG)

Long sequence FATG is the ability of a fine-tuned PLM to generate long-length sequences that preserve the sequential order as the original human-expert FATs. Long sequence FATs are complete failure analysis containing all sequence of FAs in the correct order, usually above 7 or more triplets.

The best LESE-1 score obtained for a FDR prompt having 7-FATs is 31% with fine-tuned GPT-2 (Base and Large). Despite being suited for long text generation, GPT-3 fails to generate triplets consistent with human FATs. In generating LS-FATs, none of the models accurately generated the correct equipment proposed by human-expert, however, the equipment generated could serve similar purpose as that proposed by expert.

We also observe that specifying the location of FA in the failure description consequently improves the accuracy of both SS-FATG and LS-FATG, as the triplets generated for different locations differ by the type of equipment used for failure analysis.

LESE: A New Evaluation Metric

For better understanding of the sequential generation of FATs and to address the drawback of ROUGE, we propose LEvenshtein Sequential Evaluation (LESE) metric. LESE is an n-gram Levenshtein distance based metric used for measuring the similarity between two sequences by computing the n-gram edits (insertions, deletions or substitutions) required to change one sequence into another.

LESE is unbiased in the computation of precision and recall as it takes into account the total number of n-grams as the denominator, in the computation of the precision and recall rather than unigrams. When compared to human evaluation, LESE-N performs comparatively similar, especially LESE-3 for FATG compared to other metrics used for quantitative evaluation.

Why LESE?

Traditional metrics like BLEU and ROUGE do not accurately convey human-expert evaluation for structured FAT data. For example, if we flip equipment in the last two triplets, BLEU-1 does not distinguish the difference since it only seeks to find the intersection between sets in hypothesis and reference. LESE-1 and LESE-3 accurately measure the difference in equipment and correlate with human evaluation.

Conclusion

We evaluate the efficiency of pre-trained transformer models for the downstream task of failure analysis triplet generation. We observe that forward only auto-regressive modelling used in GPT-2 and GPT-3 gives it excellent capabilities to generate structured data.

When adapted for FATG, GPT-2 ROUGE scores outperforms other benchmarks for generating both short and long FATs. Since ROUGE, BLEU and METEOR scores do not accurately convey human-expert evaluation, we introduce Levenshtein Sequential Evaluation (LESE) metric. LESE-N performs on par with human expert judgment for different test cases.

GPT-2 Medium and Large models perform best on LESE-1 and LESE-3 triplet scores, which corresponds to human judgement. Fine-tuned BART generates very short triplets and seeks contextual representation of triplets making it unfit for structured long-text sequences, while GPT-3 generates long story-telling-like data that do not necessarily follow known expert failure analysis FATs.

This work demonstrates the practical value of pre-trained language models for automating failure analysis workflows in semiconductor manufacturing, enabling faster and more consistent generation of failure analysis procedures.

Read the Full Paper

For detailed methodology, additional experiments, and comprehensive analysis, please refer to the full research paper available on arXiv.

Access Full Paper on arXiv