Skip to content
Created synthetic transaction data for fraud detectionBuilt a complete sklearn.Pipeline with:ColumnTransformer (OneHotEncoder + StandardScaler)RandomForestClassifierTracked training using MLflow, including:ParametersMetrics (ROC AUC, accuracy, etc.)Full model artifactsExported the final model as fraud_model.pkl using joblib → This file is later used for batch scoring in Databricks.
raw/Contains incoming transaction data fraud_new.csvprocessed/Will store final scored results after Databricks processing
Deployed a new Azure Databricks WorkspaceCreated a compute cluster (fraud-cluster) using Databricks Runtime MLUpload the Trained Model into workspace
Connect Databricks to Azure Blob StorageLoad new data from Blob Storagen and convert Spark to PandasApply feature engineering inside Databricks and PredictLog scoring run to MLflow, each scoring job logs:input_filemodel_sourceProcess metrics:num_recordsavg_fraud_scoremax_fraud_scorefraud_rate_pred
Example of Log scoringSave Scored Results Back to Azure Blob Storage, we can download to explore the result as csv files
Azure + Databricks + MLflow
This project simulates a real-world Machine Learning Engineer workflow.
It covers end-to-end development from model training, cloud storage, batch scoring on Databricks, MLflow experiment tracking, and saving scored results back to Azure Blob Storage.
Workflow Summary
1. Local Development – Simulated Data + Model Training + MLflow Tracking
(features such as amount, hour, country, device type, merchant category, international flag, etc.)
2. Azure Blob Storage – Raw & Processed Data
Created a dedicated Azure Storage Account with two containers
3. Azure Databricks – Workspace & Model Setup
4. Databricks Notebook – End-to-End Batch Scoring Pipeline
Want to print your doc?
This is not the way.
This is not the way.
Try clicking the ··· in the right corner or using a keyboard shortcut (
CtrlP
) instead.