Credit Card Fraud Detection

Course: CS 307 • Lab 03 (Fall 2025)
Author: Reference Solution
Date: November 11, 2025

1. Introduction

Credit card fraud is rare, but expensive. Every day the bank processes tens of thousands of card transactions, the vast majority of which are legitimate. A small number are fraudulent and, if not stopped quickly, lead directly to financial loss and customer frustration. The loss minimization team therefore needs a model that can flag likely fraud in real time so that suspicious transactions can be declined or routed to manual review.

The main challenge is the imbalance between classes: in our training data fewer than 1% of transactions are labeled as fraud. A trivial model that always predicts “not fraud” would be almost perfectly accurate but completely useless. Instead, our goal is to maximize recall on frauds while keeping precision high, so that:

  • most fraudulent transactions are caught, and
  • a transaction flagged as fraud is very likely to truly be fraudulent, minimizing unnecessary interventions.

For this project we build and evaluate a supervised classifier using historical transactions with known fraud labels. The model is trained on a labeled training set, tuned using a validation split, and finally evaluated on an independent test set that simulates future production data.

2. Methods

2.1 Data

Data source and structure

We work with a labeled training dataset used for model development and a held-out test dataset used only for final evaluation.

Each row in these datasets represents a single credit card transaction and includes:

  • Fraud: binary target (1 = fraud, 0 = genuine).
  • Amount: dollar amount of the transaction.
  • PC01 – PC28: 28 principal components summarizing various anonymized behavioral and contextual features (e.g., merchant, location, time patterns), produced via PCA on the original features.

Class balance

  • Train set
    • Genuine (0): 53,961 transactions (≈ 99.42%)
    • Fraud (1): 315 transactions (≈ 0.58%)
  • Test set
    • Genuine (0): 13,490 transactions (≈ 99.42%)
    • Fraud (1): 79 transactions (≈ 0.58%)

This confirms a strong but not extreme imbalance: fewer than 1 out of 170 transactions in our data is fraudulent. Any modeling approach has to explicitly account for this.

Descriptive statistics and exploratory plots

We computed standard summary statistics (mean, standard deviation, quartiles, etc.) for all numeric variables. Two simple visualizations were created to build intuition:

  1. Distribution of transaction amount by class (Figure 1)
    • Both fraud and genuine transactions are heavily right-skewed: most purchases are small, with a long tail of larger amounts.
    • Fraudulent transactions tend to concentrate in the lower amount range as well, though there are frauds at a variety of amounts.
/var/folders/ps/rl8jzgw54618p20ks3k_4x5r0000gn/T/ipykernel_65964/2232581252.py:14: UserWarning: Attempt to set non-positive xlim on a log-scaled axis will be ignored.
  ax.set_xlim(amt_df["Amount"].min(), amt_df["Amount"].max())

  1. PC01 vs PC02 scatterplot (Figure 2)
    • When plotting PC01 against PC02, fraudulent transactions (orange) occupy a slightly different region of the space compared to genuine ones (blue), especially at higher values of PC02.
    • There is still substantial overlap, which is expected: fraud patterns are subtle and cannot be perfectly separated in two dimensions. However, this plot provides evidence that the PCA features do contain signal relevant for distinguishing fraud.

2.2 Model

Feature matrix and target

We use all available features: the transaction amount and the 28 principal components (PC01–PC28) as predictors, and the Fraud indicator as the binary target.

Choice of algorithm

We use Histogram-based Gradient Boosting (HistGradientBoostingClassifier from scikit-learn). Reasons for this choice:

To counter the imbalance we set class_weight="balanced", which internally up-weights fraudulent examples in proportion to their rarity. Misclassifying a fraud therefore incurs a higher penalty than misclassifying a genuine transaction during training.

Hyperparameter tuning

We tune three key hyperparameters using a grid search with 5-fold cross-validation, optimizing for F1-score:

  • max_iter ∈ {100, 200, 400} – number of boosting iterations (trees).
  • learning_rate ∈ {0.1, 0.2, 0.3, 0.4, 0.5} – step size for boosting.
  • max_leaf_nodes ∈ {30, 60, 120} – tree complexity.

The grid search evaluates each combination using 5-fold cross-validation on the training portion and refits the best-scoring model on all training data. F1-score is chosen as the CV metric because it balances precision and recall, which is aligned with our business objective. The best param is 400 for max_iter, 0.4 for learning_rate and 120 for max_leaf nodes.

Final training

After selecting the hyperparameters using only the training and validation data, we train a final model on the full training set using the chosen hyperparameters and class weighting to address imbalance.

3. Results

3.1 Test performance at the default 0.50 threshold

Metrics

  • Precision: 0.917
  • Recall: 0.835
  • F1: 0.874
  • Average Precision (AUPRC): 0.847

Confusion matrix

  • True negatives: 13,484
  • False positives: 6
  • False negatives: 13
  • True positives: 66

In summary, with the default 0.50 decision threshold, our model flagged 72 transactions as potentially fraudulent, of which 66 were actually fraud and 6 were not. This translates to a very low false-positive rate on genuine transactions—approximately 0.045% (6 out of 13,490)—while successfully identifying 66 out of 79 total fraud cases, covering 83.5% of all actual frauds and missing 13. With precision at 0.917 and recall at 0.835, the model yields a balanced performance. Operationally, this means that analysts would have a few more alerts to review compared to the prior configuration (6 false positives versus 2), but would also miss slightly fewer fraud cases (13 false negatives compared to 15).

Test Precision: 0.917
Test Recall:    0.835
Test F1:        0.874
Test AP (AUPRC):0.847
Test ROC-AUC:   0.920

3.2 Cross-validation signal for learning rate

The “learning_rate vs CV score” plot summarizes mean F1 across the grid, grouped by learning rate:

  • Performance improves from 0.10 → 0.40 and then dips slightly at 0.50.
  • This suggests a sweet spot around 0.40 for our current ranges of max_iter and max_leaf_nodes.
  • Because the plot averages over the other hyperparameters, it should be read as a trend, not a hard choice—grid search still selects the best full combination.

4. Discussion

4.1 Business interpretation

At the chosen operating point, the model behaves as follows on the test data:

  • False positives: 6 out of 13,490 genuine transactions are incorrectly flagged as fraud, a false positive rate of roughly 0.045%. For customers, this means that denied or challenged legitimate transactions should be extremely rare. For the loss minimization team, almost all alerts correspond to real fraud, which makes manual review more efficient.

  • False negatives: 13 out of 79 fraudulent transactions slip through undetected, meaning we catch about 84% of frauds. This is not perfect, but it is a substantial improvement over rule-based systems or naive models.

From a business perspective, this configuration prioritizes high precision—when we interrupt a customer or trigger a manual review, it is usually for a good reason—while still achieving reasonably high recall. In many banking environments this is a sensible default: false positives are highly visible to customers and can damage trust, whereas a small number of missed frauds can be absorbed financially as long as the overall loss is controlled.

That said, the threshold can be adjusted depending on the bank’s risk appetite. Lowering the threshold would increase recall (catch more frauds) at the cost of more false positives; raising it would do the opposite. The precision-recall curve summarizes exactly this trade-off.

4.2 Deployment decision

We recommend deploying this model to production, subject to the implementation of appropriate monitoring and safeguards. The key factors supporting this decision are:

Reasons to deploy:

  1. Meets performance requirements. The model exceeds 0.90 (achieving 0.917) and achieve 0.835 for the recall on the independent test set. This demonstrates that the model can reliably identify fraudulent transactions while minimizing customer friction.

  2. Strong precision minimizes customer impact. With only 6 false positives out of over 13,000 genuine transactions (0.045% false positive rate), the model will very rarely interrupt legitimate customer activity. This is critical for maintaining customer trust and satisfaction.

  3. Meaningful fraud detection capability. Catching 84% of fraudulent transactions represents significant value compared to baseline approaches. At the bank’s transaction volume, this translates to preventing thousands of fraudulent transactions daily.

  4. Flexible operating point. The decision threshold can be easily adjusted post-deployment based on observed costs of false positives versus false negatives, allowing the bank to optimize the business outcome over time.

  5. Robust methodology. The model development process followed best practices: proper train/validation/test splits, cross-validation for hyperparameter tuning, explicit threshold selection, and evaluation on held-out data. This increases confidence in real-world performance.

4.4 Limitations

  1. Limited time span and scope. The original dataset covers only two days of credit card activity and has been down-sampled. Real-world fraud patterns evolve over months and years. Performance might degrade once fraudsters adapt or customer behavior changes (concept drift).

  2. Anonymized features. Because we only see principal components rather than the original, interpretable variables, it is difficult to understand why certain transactions are labeled as fraud. This limits the ability to craft targeted rules or give human-readable explanations to customers.

  3. Single static model. The current setup trains a single model and evaluates it on a single test set. A production system would need ongoing monitoring, periodic retraining, and possibly multiple models for different customer segments or transaction types.

  4. Simple cost structure. Our threshold selection uses fixed precision/recall targets rather than an explicit financial cost model. In practice, one would quantify the dollar cost of a missed fraud vs. a false alarm and choose the threshold that minimizes expected loss.

4.5 Conclusion

The histogram-based gradient boosting classifier developed in this project successfully meets the performance requirements for credit card fraud detection, achieving 0.917 precision and 0.835 recall on held-out test data. The model demonstrates strong capability to identify fraudulent transactions while maintaining an low false positive rate of 0.045%, which minimizes unnecessary customer disruption.

We recommend deploying this model to production with appropriate monitoring. While the model has limitations related to the temporal scope of training data and anonymized features, its strong test performance and flexible threshold mechanism provide a solid foundation for real-time fraud prevention. With proper operational safeguards in place, this model can deliver substantial business value by preventing fraud losses while preserving customer trust and satisfaction.

Back to top