| | --- |
| | base_model: distilbert-base-uncased |
| | language: |
| | - en |
| | library_name: transformers |
| | license: cc-by-sa-4.0 |
| | pipeline_tag: text-classification |
| | tags: |
| | - text-classification |
| | datasets: |
| | - SuccubusBot/incoherent-text-dataset |
| | --- |
| | |
| | # DistilBERT Incoherence Classifier |
| |
|
| | This is a fine-tuned DistilBERT model for classifying text based on its coherence. It can identify various types of incoherence. |
| |
|
| | ## Model Details |
| |
|
| | - **Model:** DistilBERT (distilbert-base-uncased) |
| | - **Task:** Text Classification (Coherence Detection) |
| | - **Fine-tuning:** The model was fine-tuned using a custom-generated dataset that features various types of incoherence. |
| | - **Training Dataset** The model was trained on the [incoherent-text-dataset](https://huggingface.co/datasets/your_huggingface_username/incoherent-text-dataset) dataset, located on Huggingface. |
| |
|
| | ## Training Metrics |
| |
|
| | | Epoch | Training Loss | Validation Loss | Accuracy | Precision | Recall | F1 | |
| | | :---- | :------------ | :-------------- | :------- | :-------- | :----- | :------- | |
| | | 1 | 0.037500 | 0.071958 | 0.984995 | 0.985002 | 0.984995 | 0.984564 | |
| | | 2 | 0.008900 | 0.068670 | 0.985995 | 0.985973 | 0.985995 | 0.985603 | |
| | | 3 | 0.008500 | 0.058111 | 0.990330 | 0.990260 | 0.990330 | 0.990262 | |
| |
|
| | ## Evaluation Metrics |
| |
|
| | The following metrics were measured on the test set: |
| |
|
| | | Metric | Value | |
| | | :---------- | :------- | |
| | | Loss | 0.049511 | |
| | | Accuracy | 0.991 | |
| | | Precision | 0.990958 | |
| | | Recall | 0.991 | |
| | | F1-Score | 0.990962 | |
| |
|
| | ## Classification Report: |
| |
|
| | ``` |
| | precision recall f1-score support |
| | |
| | coherent 0.99 0.99 0.99 1500 |
| | grammatical_errors 0.96 0.94 0.95 250 |
| | random_bytes 1.00 1.00 1.00 250 |
| | random_tokens 1.00 1.00 1.00 250 |
| | random_words 1.00 1.00 1.00 250 |
| | run_on 1.00 0.99 1.00 250 |
| | word_soup 1.00 1.00 1.00 250 |
| | |
| | accuracy 0.99 3000 |
| | macro avg 0.99 0.99 0.99 3000 |
| | weighted avg 0.99 0.99 0.99 3000 |
| | ``` |
| |
|
| | ## Confusion Matrix |
| |
|
| |  |
| |
|
| | The confusion matrix above shows the performance of the model on each class. |
| |
|
| | ## Usage |
| |
|
| | This model can be used for text classification tasks, specifically for detecting and categorizing different types of text incoherence. You can use the `inference_example` function provided in the notebook to test your own text. |
| |
|
| | ## Limitations |
| |
|
| | The model has been trained on a generated dataset, so care must be taken in evaluating it in the real world. More data may need to be collected before evaluating this model in a real-world setting. |
| |
|
| | ## License |
| |
|
| | CC-BY-SA 4.0 |
