Diagnosing mechanical issues in vehicles often relies on acoustic cues—the specific rattle of a loose heat shield, the squeal of a worn serpentine belt, or the deep rumble of a failing wheel bearing. While experienced mechanics recognize these sounds instantly, translating this auditory expertise into a machine learning model is a notoriously difficult task. The GitHub repository adam-s/car-diagnosis addresses this challenge directly, serving as an open-source, end-to-end audio machine learning pipeline designed to identify automotive faults from sound clips.

Rather than acting as a polished, commercial-ready consumer application, the project is structured as an honest, transparent proof of concept. It demonstrates how to build a complete machine learning workflow from scratch, tackling every stage from raw data collection to calibrated model inference. For developers and machine learning engineers, it provides a practical blueprint for handling niche audio classification tasks where pre-existing, clean datasets are virtually non-existent.

The pipeline architecture

The repository is built around a multi-stage Python pipeline that systematically transforms raw, noisy internet audio into a calibrated diagnostic prediction. Its architecture relies on several specialized libraries, including PyTorch for deep learning operations, Librosa for audio manipulation, and Scikit-learn for model evaluation and calibration.

The pipeline consists of four distinct phases:

  1. Scraping: The pipeline begins by harvesting raw audio data. It uses search queries to find and download relevant videos from YouTube, converting them into audio files to build a custom dataset of car sounds.
  2. Cleaning: Raw internet audio is notoriously noisy, often containing speech, background music, or wind interference. The project utilizes demucs, a deep learning-based music source separation library, to isolate the core mechanical sounds and strip away unwanted background noise.
  3. Feature Extraction: Instead of training a model from scratch on raw waveforms, the pipeline leverages Contrastive Language-Audio Pretraining (CLAP). Using a pretrained CLAP model, the pipeline extracts high-dimensional embeddings from the cleaned audio clips. These embeddings capture rich semantic features of the sounds, representing them in a shared space with text.
  4. Classification and Calibration: The final layer is a classification head that maps the CLAP embeddings to specific car faults. Critically, the pipeline includes a temperature scaling step. This post-processing technique calibrates the model's output probabilities, ensuring that the confidence scores returned by the classifier actually reflect the true likelihood of a specific mechanical fault.

Pipeline capabilities and workflow

The project provides a command-line interface to run the entire machine learning lifecycle. The workflow is organized into distinct Python scripts that correspond to the architectural phases, allowing users to run the pipeline sequentially or modify individual components.

Users can initiate the data gathering process by defining target search terms for specific car faults. The pipeline handles the downloading and organizes the raw audio. From there, the cleaning script processes the directory, running the audio through the Demucs separator to isolate the mechanical noise.

Once the data is clean, the embedding script processes the audio files through the CLAP model, saving the resulting vectors. Finally, the training script fits a classifier on these embeddings. Because the project emphasizes rigorous evaluation, the pipeline outputs calibrated probability scores. Instead of simply guessing a fault, the system attempts to quantify its own uncertainty, helping users understand when the model is genuinely confident versus when it is making an uncalibrated guess.

Constraints and limitations

As a proof of concept with over 200 stars on GitHub, the project has several practical limitations that developers should consider before adapting it. The most significant bottleneck is data quality. Because the pipeline relies on automated YouTube scraping, the initial dataset is only as good as the search queries and the uploader's descriptions. Noise separation via Demucs helps, but it cannot turn a highly distorted, low-quality recording into pristine diagnostic data.

Additionally, CLAP models are trained on general audio datasets. While they are highly capable of distinguishing broad sound categories, they may lack the fine-grained acoustic resolution needed to differentiate between highly similar mechanical sounds, such as two different types of engine ticks.

The pipeline is also computationally demanding. Running source separation with Demucs and extracting embeddings via CLAP requires a modern GPU to process large datasets in a reasonable timeframe. Running this pipeline on a standard CPU will result in extremely slow execution times during the cleaning and feature extraction phases.

Getting started

To explore the codebase or run the pipeline locally, you will need a Python environment with PyTorch and CUDA support if you plan to utilize GPU acceleration. The project's documentation outlines the necessary steps to set up the dependencies, configure the scraper, and execute each stage of the pipeline. Detailed setup instructions and dependency requirements are documented in the project's README.

This repository serves as an excellent educational resource and a foundation for custom audio classification projects. It is particularly valuable for developers who want to understand how to handle noisy, real-world audio data without relying on perfectly curated academic datasets. Check out the project at car-diagnosis.