NASA NEO: comparison of six classifiers on an imbalanced dataset

Context

Project completed by a team of three as part of the IFT712 - Learning Techniques course at Université de Sherbrooke. The goal was to select a real dataset and compare six classification methods with a common experimental protocol.

We chose the NASA Near-Earth Objects dataset available on Kaggle, which contains 90,836 near-Earth objects, of which only 9.7% are classified as potentially hazardous.

Problem

On such an imbalanced dataset, high accuracy can mask poor detection of the minority class. We therefore chose recall as the priority metric, while analyzing precision, F1 score, AUC-ROC, and execution time.

Approach

The project was structured in Python around a common object-oriented architecture for all six models, with an abstract BaseModel class to standardize training, evaluation, and optimization.

The data pipeline includes:

• selection of 4 relevant variables after dataset exploration;
• log(1+x) transformation of highly skewed variables;
• RobustScaler to limit the influence of extreme values;
• stratified 80/20 split;
• class_weight="balanced" for linear models and SVM;
• hyperparameter optimization with GridSearchCV or RandomizedSearchCV.

The six models compared are: Decision Tree, SVM, Logistic Regression, Naive Bayes, Neural Network, and Least Squares.

My contribution

I mainly worked on the OOP architecture and preprocessing pipeline, as well as implementing Least Squares and the neural network.

I also participated in model optimization, comparative results analysis, and performance visualizations.

Results

The results show a marked trade-off between detection and precision.

SVM achieves the best recall at 99.43%, but only 29.9% precision. Least Squares reaches 98.98% recall with a time of 0.01 s in the reported benchmark. Logistic regression also reaches 95.02% recall.

Conversely, Decision Tree and Neural Network achieve the best accuracy, at 91.43% and 91.45% respectively, but with only 13.57% and 15.10% recall.

The main lesson is that model choice depends on the metric and error cost considered, not solely on accuracy.

Limits

This study remains an academic comparison on a static dataset. The results do not allow concluding operational performance on new observations.

The Kaggle dataset is a subset of NASA data. The stratified 80/20 split and cross-validation used for hyperparameter tuning support comparison within this setting, but do not establish stability over time or on another data source.

Next steps

Studying decision thresholds would allow better control of the trade-off between Recall and Precision. Temporal validation or validation on external data would then verify result stability.