Model Card for code-lang-bert-small

A fine-tuned BERT-small model for identifying programming languages from code snippets. The model classifies raw source code into one of 16 supported languages with high accuracy.

Model Details

Model Description

This model is a fine-tuned version of prajjwal1/bert-small (29M parameters) designed for the task of programming language identification. By analyzing the syntax, keywords, and structural patterns of source code, it accurately predicts the programming language of a given snippet.

• Developed by: Pankaj8922
• Model type: Encoder-only Transformer (BERT-small) for sequence classification
• Language(s): 16 programming and markup languages (see below)
• License: Apache 2.0
• Finetuned from model: prajjwal1/bert-small

Supported Languages

Rust, Java, Dart, Python, Go, HTML, JavaScript, Typescript, C, CSS, C#, Markdown, Assembly, Lua, C++, Kotlin

Uses

Direct Use

The model is intended for classifying code snippets. It can be used directly with the Hugging Face pipeline API or integrated into applications for code tagging, automated documentation, or content filtering.

from transformers import pipeline

classifier = pipeline(
    "text-classification",
    model="Pankaj8922/code-lang-bert-small"
)

code_snippet = """
def quicksort(arr):
    if len(arr) <= 1:
        return arr
    pivot = arr[len(arr) // 2]
    return quicksort(left) + mid + quicksort(right)
"""

result = classifier(code_snippet)
print(result)
# [{'label': 'Python', 'score': 0.99}]

Out-of-Scope Use

The model is trained to classify full files or substantial code snippets. It may not perform well on:

• Very short, ambiguous one-liners.
• Heavily obfuscated or minified code.
• Code containing multiple languages (e.g., a Python file with extensive embedded SQL).
• Languages not present in the 16 supported classes.

Bias, Risks, and Limitations

The model may exhibit biases present in the training data distribution. Languages with syntactically similar constructs (e.g., C and C++, JavaScript and TypeScript) are the most common sources of confusion, as reflected in the confusion matrix. Performance on code from very niche or domain-specific libraries may be lower.

Training Details

Training Data

The model was trained on the Code-Language-Classification dataset. The official train, validation, and test splits were used.

• Train samples: 1,600,000
• Validation samples: 32,000
• Test samples: 32,000
• Classes: 16 (perfectly balanced, 2000 samples per class in test set)

Training Procedure

The BERT-small model was fine-tuned on 2 x T4 GPUs with dynamic padding for efficiency. Training was configured for 5 epochs with early stopping, but was manually stopped after 4 epochs as the model had already converged.

• Batch size: 256 (128 per device x 2 GPUs)
• Learning rate: 3e-5
• Optimizer: AdamW (weight decay: 0.01)
• Max sequence length: 512 tokens
• Early stopping patience: 2 epochs
• Checkpointing: Best model based on validation accuracy saved to the Hub.

Evaluation

The evaluation was performed on the held-out test set of 32,000 samples using the official script provided in the repository.

Testing Metrics

Metric	Value
Accuracy	96.63%
Macro F1	96.62%
Weighted F1	96.62%
Macro Precision	96.63%
Macro Recall	96.63%
Eval Loss	0.1147

Per-Class Performance

Language	Precision	Recall	F1-Score
Rust	0.9885	0.9925	0.9905
Java	0.9731	0.9785	0.9758
Dart	0.9772	0.9850	0.9811
Python	0.9890	0.9880	0.9885
Go	0.9859	0.9800	0.9829
HTML	0.9279	0.8885	0.9078
JavaScript	0.8859	0.8930	0.8894
TypeScript	0.9466	0.9580	0.9523
C	0.9566	0.9375	0.9470
CSS	0.9728	0.9845	0.9786
C#	0.9895	0.9870	0.9882
Markdown	0.9671	0.9695	0.9683
Assembly	0.9935	0.9945	0.9940
Lua	0.9885	0.9915	0.9900
C++	0.9770	0.9760	0.9765
Kotlin	0.9840	0.9870	0.9855

Key Observations

• The model performs exceptionally well on most languages, with 11 of 16 classes achieving an F1-score of 97% or higher.
• JavaScript (F1: 0.89) and HTML (F1: 0.91) are the most challenging classes, commonly confused with each other and with TypeScript/CSS.
• The model is highly confident in distinguishing structurally unique languages like Assembly (F1: 0.994) and Python (F1: 0.989).

Environmental Impact

• Hardware Type: 2 x NVIDIA T4 GPUs
• Hours used: Approx. 4 epochs of training
• Cloud Provider: Not specified
• Compute Region: Not specified

Carbon emissions can be estimated using the Machine Learning Impact calculator.