Introduction

Pre-trained large language models (LLMs) have gained significant attention in the field of natural language processing (NLP), especially for the task of text summarization, generation, and question answering. The success of LMs can be attributed to the attention mechanism introduced in Transformer models, which have outperformed traditional recurrent neural network models (e.g., LSTM) in modeling sequential data.

In this paper, we leverage pre-trained causal language models for the downstream task of Failure Analysis Triplet Generation (FATG), which involves generating a sequence of failure analysis decision steps for identifying failure root causes in the semiconductor industry. In particular, we conduct extensive comparative analysis of various transformer models for the FATG task and find that the BERT-GPT-2 Transformer (Big GCVAE), fine-tuned on a proposed Generalized-Controllable Variational AutoEncoder loss (GCVAE), exhibits superior performance in generating informative latent space by promoting disentanglement of latent factors.

Specifically, we observe that fine-tuning the Transformer style BERT-GPT2 on the GCVAE loss yields optimal representation by reducing the trade-off between reconstruction loss and KL-divergence, promoting meaningful, diverse and coherent Failure Analysis Triplets (FATs) similar to expert expectations.

Figure 1: Symbolic representation of FATG decision-making process

Failure Analysis Triplet Generation (FATG)

FATG is a scientific process that aims to generate a sequence of failure analysis texts for a given failure description. We approach FATG as a data-to-text generation task, where the input is a Failure Description Report (FDR) represented as structured tabular data, and the output is a lengthy sequence of failure analysis triplets.

Throughout the subsequent sections of this paper, the term pre-triplet will refer to failure descriptions, while triplets (Step type; Substep technique; Equipment) will denote the collection of analyses, each comprising three key components. Each triplet contributes to a failure decision, and the main objective of this paper is to generate a set of n triplets that correspond to a specific failure description.

FRACAS Variables

FRACAS stands for "Failure Reporting, Analysis, and Corrective Action System." It is a systematic approach used in industries, particularly in engineering and manufacturing, to manage the identification, analysis, and resolution of failures or defects in products, systems, or processes.

We describe the three important variables (the triplets) for decision-making as follows:

Step type: The initial stage of fault analysis entails determining the fault analysis step type, which represents the first analysis proposed by the expert once the sample is validated for analysis. At STMicroelectronics, the fault analysis process is predominantly composed of Non-destructive Inspection, Electrical Failure Verification, Sample Preparation, Physical Analysis, Global Fault Localization, and other step types, collectively accounting for 94.3% of all conducted fault analyses.
Substep technique: In the context of fault analysis, each step type is accompanied by a corresponding substep technique. A comprehensive set of 91 distinct substep techniques is employed for sub-analysis purposes. Among these techniques, certain substep techniques are frequently associated with specific step types. Notably, package decap, optical microscopy, SAM (Scanning Acoustic Microscopy), X-ray, SEM (Scanning Electron Microscopy), die delaying, FIB (Focused Ion Beam) cross-section, continuity test, electrical parametric test, and mechanical cross-section are the most commonly used substep techniques.
Equipment: Equipment are the tools used for failure analysis. The equipment are 1348 equipment types found in the data set. OM113-LEICA M165C, PK103-PHOENIX X-RAY NANOMEX, ZZ003-CRI7, ZZ003-CRI6, MICROSCOPE LEICA DM2700M, SAM, AGR XRAY01, AGR MICROXCT 200 X-RADIA 3D, AGR STEREOSCOPIC MICROSCOPE M8-FA, AGR XPREP01, AGR DUAL BEAM HELIOS 400, AGR PARALLEL LAPPER 3-FA are the top ten equipments frequently used for analysis.

Figures 2-4: Percentage distribution of Step types, Substep techniques, and Equipment used in failure analysis

Big GCVAE Architecture

Leveraging our previous understanding of variational autoencoder for learning high-quality latent representations and optimal reconstruction of objects including text and images, we propose an improved variational Large Language Model accordingly. This model, Big GCVAE, is adopted for the FATG task by tying together two different transformers architectures (Encoder and Decoder-only) and fine-tuning them using GCVAE loss function.

The model is structured like a classic Transformer model but loaded with pretrained weights. The Encoder is an unmasked BERT model while the decoder is a GPT-2 (for example, GPT-2 base, small or large) and fine-tuned on a loss function with adaptive hyperparameters.

Figure 5: An illustration of the Big GCVAE architecture. On the leftmost is the unmasked BERT weights loaded to the Encoder and GPT-2 weights loaded to the Decoder

Key Technical Innovations

Unmasking: Masking is a concept introduced in BERT model and involves selectively hiding certain tokens within an input sequence during the pre-training phase. However, when incorporating a decoder component, such as the GPT-2 model, to complete the Big GCVAE Encoder-Decoder model, we hypothesize that the exclusive use of masking limits the model's ability to learn a quality bidirectional representation. Consequently, this restriction hampers the generalization of the latent space and considerably diminishes the mutual information within the bottleneck. Therefore, we propose the omission of masking, allowing for constructive summarization of mutual information within the latent space.

Latent Injection: Following a similar approach to BERT, the initial token of each sentence in Big GCVAE is a special classification token ([CLS]). The hidden state h[CLS] in the last layer, corresponding to this token, is extracted as the sentence-level representation. To construct the latent representation z, we employ the use of the weighted matrix W_E ∈ R^R×H, where z is a P-dimensional vector and W_E is the weight matrix.

GCVAE Loss Function

The optimization framework proposed uses adaptive hyperparameters α_t, β_t, and γ_t:

L(θ,φ,α,β,γ) = (1-α_t-β_t) E_{z∼q_φ(z|x)}[ln p_θ(x|z)] - β_t E_D[D_KL(q_φ(z|x)||p_θ(z))] + γ_t D_KL(q_φ(z)||p_θ(z))

The adaptive weight α_t controls the reconstruction error while β_t ensures the posterior latent factor q_φ(z|x) does not deviate significantly from its prior p_θ(z). Varying both terms gives us better control of the degree of disentanglement and helps us understand the parameters affecting density disentanglement.

Experimentation

Dataset

In our experimentation, we use real failure analysis data obtained from a semiconductor industry, specifically focusing on successful failure analysis cases from the year 2019. To prepare the data for training the transformer model, we concatenate all input features, including the triplet data, along the horizontal axis (x-axis). After preprocessing, the size of the data for the year 2019 reduces to 5809 observation (or FA analysis) of which 70% (4066) is used for training and 30% (1743) is used for evaluation.

Experimental Setup

The experimentation is conducted on a High-Performance Computing (HPC) cluster comprising 80 cores, 2 × Intel Xeon E5-2698 v4 2.20GHz CPUs (80 cores), 512GB RAM, and 8 × Nvidia V100 32GB GPUs.

We fine-tune the medium versions of GPT-2 with 335 million parameters after downloading the pre-trained weights through the Huggingface API. The batch size for training and evaluation is 1, the weight decay is 0.05, and the number of training epochs is 100.

Results

Quantitative Evaluation

We conduct performance comparison between Big GCVAE and derivative models of GCVAE, such as ControlVAE and β-VAE with Annealing KL-divergence. We adopt two versions of Big GCVAE based on the correlation measure as follows:

Big GCVAE†: D_KL(q_φ(z)||p_θ(z)) ← Maximum Mean Discrepancy
Big GCVAE‡: D_KL(q_φ(z)||p_θ(z)) ← Squared Mahalanobis distance

Model	Evaluation loss ↓	Reconstruction loss ↓	KL divergence ↑
GPT2-M	0.19	–	–
BigVAE	1.10	128.34	6.49
BigControlVAE	1.18	1.10	9.85
BigGCVAE†	1.18	1.09	8.23
BigGCVAE‡	1.11	1.09	3.80

Table 1: Performance evaluation of Big GCVAE models and its derivatives. Both Big GCVAE† and Big GCVAE‡ have the lowest reconstruction loss compared to BigVAE.

Big GCVAE‡ model demonstrates superior performance compared to the benchmark Big variational model across various evaluation metrics. When specifically applied for the FATG task, the Big GCVAE‡ model outperforms GPT-2 medium, highlighting the efficacy of controllable Lagrangian hyperparameters in achieving optimal representation and generalizing the latent space.

Model	BLEU-1	BLEU-3	ROUGE-1 F1	ROUGE-L F1	LESE-1 F1	LESE-3 F1	METEOR
GPT2-M	11.64	7.50	16.12	14.79	8.57	0.30	16.0
BigVAE	15.10	8.55	11.79	10.06	4.04	1.28	14.0
BigControlVAE	15.10	9.05	10.39	10.01	4.08	1.28	14.0
BigGCVAE†	14.60	9.07	11.81	10.11	4.04	1.28	14.0
BigGCVAE‡	22.46	17.60	34.77	32.63	24.91	10.73	16.0

Table 3: Performance comparison of models. Observe that Big GCVAE‡ performs best for ROUGE-1 and LESE-1 metrics. High values indicate better performance for all metrics.

Qualitative Evaluation

We conduct an evaluation of the generative capabilities of the Big GCVAE models and its variants. The results reveal a notable enhancement in the distribution of BLEU-1, BLEU-3, and LESE-1, LESE-3 scores. The figure clearly demonstrates an increased frequency of accurately generated FATs by the model that closely align with the expert failure analysis.

When compared to the decoder-only transformer model (GPT-2), the Big GCVAE exhibits the potential to generate failure analysis sequences that are notably more realistic (following the order of Step type; Substep technique and Equipment). This improvement can be attributed to the Big GCVAE's ability to generalize effectively within the latent embedding space associated with the task.

Real-World Use Cases

This paper presents real-world use cases to demonstrate the generative efficacy of Big GCVAE:

Grenoble Kostal leakage on GH2: The failure analysis triplets generated by Big GCVAE for root cause analysis on product packages are generally plausible, except for the third triplet. The FATs generated by GPT-2 are also plausible, but they contain a higher number of redundant failure analysis triplets.
Grenoble B601 Face ID H9A HCMOS9A WLCSP CSP H9A HCMOS9A CSP gate oxide breakdown: The failure analysis triplets generated by both Big GCVAE and GPT2 are plausible for the respective contexts of product packages and silicon. We hypothesize that Big GCVAE generates FATs from the abundance of space it generalizes.
B601 RIGEL3 OR TAMKOR HTOL Small Leakage Consumption VINCORE2 in power down: The failure analysis triplets generated by both Big GCVAE and GPT-2 are entirely incorrect and implausible. This can be attributed to the lack of guiding keywords (or prompts), which are crucial for the models to make accurate predictions for the next words (triplets) in the sequence.

Limitations

Despite the overwhelming performance of Big GCVAE (BERT-GPT2) model for the task of failure analysis triplets generation, it still suffers significant challenge that can be addressed. It is crucial to acknowledge that the model may occasionally generate unrealistic failure analysis triplets due to the phenomenon of hallucination. This can be both a problem of overgeneralization and overfitting. However, no particular metric perfectly addresses this phenomenon, except the quantitative and domain expert evaluations.

This limitation highlights the need for further refinement and improvement by prompt engineering failure description and using reinforcement learning to mitigate the occurrence of unrealistic outputs.

Conclusion

To overcome the challenges of robust representation and high-quality generation of failure analysis triplets, we propose a new approach that involves fine-tuning a Transformer-based Variational Autoencoder (VAE) architecture using an unmasked pre-trained BERT Encoder and a GPT2 Decoder. By leveraging the Generalized-Controllable Variational AutoEncoding (GCVAE) loss, our model aims to achieve an optimized representation with a low reconstruction loss and highly disentangled latent space.

Our evaluation of the model's performance in generating failure analysis triplets yields the following key findings:

Big GCVAE can generate failure analysis triplets that are logical and reasonable, providing valuable insights for expert failure analyst engineers in the decision-making process.
The model demonstrates its ability to generate failure analysis triplets specifically tailored to root cause identification in product packages, and it can also address potential root causes within a Silicon.
Notably, Big GCVAE is a self-supervised model that operates with adaptive-controllable hyperparameters, eliminating the need for human intervention in the decision-making process.

In summary, Big GCVAE is a robust model that can generate failure analysis triplets (sequences of text-encoded steps for analyzing defective components in the semiconductor industry) that are logical, reasonable, and tailored to specific problems. The model is able to do this by learning to represent failure analysis triplets in a latent space that is both disentangled and informative.

Read the Full Paper

For detailed methodology, additional results, and comprehensive analysis, please refer to the full research paper published in the Journal of Intelligent Manufacturing.

Access Full Paper