TomasFAV/BERTInvoiceCzechV0

BERTInvoiceCzechR (V0 – Synthetic Templates Only)

This model is a fine-tuned version of google-bert/bert-base-multilingual-cased for the task of structured information extraction from Czech invoices.

It achieves the following results on the evaluation set:

• Loss: 0.3291
• Precision: 0.5188
• Recall: 0.6917
• F1: 0.5929
• Accuracy: 0.9335

---

Model description

BERTInvoiceCzechR (V0) is the baseline model in a multi-stage experimental pipeline focused on invoice understanding.

The model performs token-level classification to extract structured fields from invoice text, such as:

• supplier
• customer
• invoice number
• bank details
• totals
• dates

This version (V0) is trained exclusively on synthetically generated invoices created from predefined templates, without any layout randomization or real-world data.

---

Training data

The dataset consists purely of:

• synthetically generated invoices
• fixed template structures
• controlled field placement and formatting

Characteristics:

• consistent layout across samples
• fully controlled annotations
• no noise or OCR artifacts
• no real invoice data
• added synthetic image augmentations

This dataset represents the simplest training scenario in the pipeline and serves as a baseline for comparison with more complex data variants.

---

Role in the pipeline

This model corresponds to:

V0 – Synthetic template-based dataset only

It is used as:

• a baseline for evaluating the impact of:
  • layout variability
  • synthetic-real hybrid data
  • real annotated invoices
• a reference point for measuring generalization gap

---

Intended uses

• Baseline model for document AI experiments
• Evaluation of synthetic data usefulness
• Comparison with more advanced dataset variants (V1–V3)
• Research in Czech invoice information extraction

---

Limitations

• Strong dependency on template structure
• May have poor generalization to:
  • unseen layouts
  • real-world invoices
  • noisy OCR outputs
• Does not capture layout variability
• Trained only on clean synthetic data

---

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 16
• eval_batch_size: 2
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 0.1
• num_epochs: 10
• mixed_precision_training: Native AMP

---

Training results

Training Loss	Epoch	Step	Validation Loss	Precision	Recall	F1	Accuracy
No log	1.0	87	0.3944	0.1965	0.2233	0.2091	0.8997
No log	2.0	174	0.2951	0.4152	0.4517	0.4327	0.9241
No log	3.0	261	0.2896	0.4790	0.5810	0.5251	0.9314
No log	4.0	348	0.3295	0.4549	0.6443	0.5333	0.9226
No log	5.0	435	0.3249	0.4908	0.6866	0.5724	0.9281
0.3757	6.0	522	0.3615	0.4646	0.6827	0.5529	0.9216
0.3757	7.0	609	0.3376	0.4913	0.6579	0.5625	0.9299
0.3757	8.0	696	0.3290	0.5194	0.6924	0.5935	0.9336
0.3757	9.0	783	0.3604	0.4906	0.6858	0.5720	0.9279
0.3757	10.0	870	0.3515	0.5011	0.6944	0.5821	0.9296

---

Framework versions

• Transformers 5.0.0
• PyTorch 2.10.0+cu128
• Datasets 4.0.0
• Tokenizers 0.22.2