Issue and suggestions regarding finetuning en_PPOCRv4_rec_train #14414

l03126 · 2024-12-18T02:45:14Z

l03126
Dec 18, 2024

Im finetuning en_PPOCRv4_rec_train. I have a small dataset (1300 images for training) however the training outputs does not seem bad. I have an issue with the model output.

YML file:

Architecture:
  Backbone:
    name: PPLCNetV3
    scale: 0.95
  Head:
    head_list:
    - CTCHead:
        Head:
          fc_decay: 1.0e-05
        Neck:
          depth: 2
          dims: 120
          hidden_dims: 120
          kernel_size:
          - 1
          - 3
          name: svtr
          use_guide: true
    - NRTRHead:
        max_text_length: 25
        nrtr_dim: 384
    name: MultiHead
  Transform: null
  algorithm: SVTR_LCNet
  model_type: rec
Eval:
  dataset:
    data_dir: /kaggle/input/recognition-data/eval
    label_file_list:
    - /kaggle/input/recognition-data/RecEvalLabel_Updated.txt
    name: SimpleDataSet
    transforms:
    - DecodeImage:
        channel_first: false
        img_mode: BGR
    - MultiLabelEncode:
        gtc_encode: NRTRLabelEncode
    - RecResizeImg:
        image_shape:
        - 3
        - 48
        - 320
    - KeepKeys:
        keep_keys:
        - image
        - label_ctc
        - label_gtc
        - length
        - valid_ratio
  loader:
    batch_size_per_card: 32
    drop_last: false
    num_workers: 8
    shuffle: false
Global:
  cal_metric_during_train: true
  character_dict_path: /kaggle/working/finetune_paddle_ocr/PaddleOCR/ppocr/utils/en_dict.txt
  checkpoints: null
  debug: false
  distributed: true
  epoch_num: 50
  eval_batch_step:
  - 0
  - 100
  infer_img: doc/imgs_words/ch/word_1.jpg
  infer_mode: false
  log_smooth_window: 20
  max_text_length: 25
  pretrained_model: /kaggle/working/pretrained_models/recognition/en_PP-OCRv4_rec_train/best_accuracy
  print_batch_step: 100
  save_epoch_step: 10
  save_inference_dir: null
  save_model_dir: /kaggle/working/finetuned_model
  save_res_path: ./output/rec/predicts_ppocrv3.txt
  use_gpu: true
  use_space_char: true
  use_visualdl: false
Loss:
  loss_config_list:
  - CTCLoss: null
  - NRTRLoss: null
  name: MultiLoss
Metric:
  ignore_space: false
  main_indicator: acc
  name: RecMetric
Optimizer:
  beta1: 0.9
  beta2: 0.999
  lr:
    learning_rate: 0.0005
    name: Cosine
    warmup_epoch: 3
  name: Adam
  regularizer:
    factor: 3.0e-05
    name: L2
PostProcess:
  name: CTCLabelDecode
Train:
  dataset:
    data_dir: /kaggle/input/recognition-data/train
    ds_width: false
    ext_op_transform_idx: 1
    label_file_list:
    - /kaggle/input/recognition-data/RecTrainLabel_Updated.txt
    name: MultiScaleDataSet
    transforms:
    - DecodeImage:
        channel_first: false
        img_mode: BGR
    - RecConAug:
        ext_data_num: 2
        image_shape:
        - 48
        - 320
        - 3
        max_text_length: 25
        prob: 0.5
    - RecAug: null
    - MultiLabelEncode:
        gtc_encode: NRTRLabelEncode
    - KeepKeys:
        keep_keys:
        - image
        - label_ctc
        - label_gtc
        - length
        - valid_ratio
  loader:
    batch_size_per_card: 32
    drop_last: true
    num_workers: 8
    shuffle: true
  sampler:
    divided_factor:
    - 8
    - 16
    first_bs: 32 #changed
    fix_bs: true #changed
    is_training: true
    name: MultiScaleSampler
    scales:
    - - 320
      - 32
    - - 320
      - 48
    - - 320
      - 64
profiler_options: null

Trained with Kaggle Multi GPU using :

!python3 -m paddle.distributed.launch --gpus '0,1' /kaggle/working/finetune_paddle_ocr/PaddleOCR/tools/train.py -c /kaggle/working/yml_file/rec_configs.yml

Training output:

[2024/12/18 00:47:31] ppocr INFO: During the training process, after the 0th iteration, an evaluation is run every 100 iterations
[2024/12/18 00:47:59] ppocr INFO: epoch: [1/50], global_step: 21, lr: 0.000083, acc: 0.000000, norm_edit_dis: 0.452535, CTCLoss: 87.524666, NRTRLoss: 2.281082, loss: 89.805748, avg_reader_cost: 0.05996 s, avg_batch_cost: 0.28793 s, avg_samples: 6.72, ips: 23.33930 samples/s, eta: 0:23:30, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 00:48:02] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 00:48:29] ppocr INFO: epoch: [2/50], global_step: 42, lr: 0.000250, acc: 0.062500, norm_edit_dis: 0.829459, CTCLoss: 13.130486, NRTRLoss: 0.988201, loss: 14.118687, avg_reader_cost: 0.09495 s, avg_batch_cost: 0.29281 s, avg_samples: 6.72, ips: 22.95018 samples/s, eta: 0:23:13, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 00:48:32] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 00:48:58] ppocr INFO: epoch: [3/50], global_step: 63, lr: 0.000417, acc: 0.656250, norm_edit_dis: 0.953869, CTCLoss: 3.552220, NRTRLoss: 0.837395, loss: 4.403297, avg_reader_cost: 0.10400 s, avg_batch_cost: 0.28936 s, avg_samples: 6.72, ips: 23.22335 samples/s, eta: 0:22:43, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 00:49:00] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 00:49:26] ppocr INFO: epoch: [4/50], global_step: 84, lr: 0.000500, acc: 0.781250, norm_edit_dis: 0.978377, CTCLoss: 1.873838, NRTRLoss: 0.808326, loss: 2.680428, avg_reader_cost: 0.10801 s, avg_batch_cost: 0.28604 s, avg_samples: 6.72, ips: 23.49358 samples/s, eta: 0:22:09, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 00:49:29] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 00:49:49] ppocr INFO: epoch: [5/50], global_step: 100, lr: 0.000499, acc: 0.843750, norm_edit_dis: 0.985853, CTCLoss: 1.050535, NRTRLoss: 0.796807, loss: 1.848653, avg_reader_cost: 0.08958 s, avg_batch_cost: 0.22867 s, avg_samples: 5.12, ips: 22.39078 samples/s, eta: 0:21:55, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB

eval model::   0%|          | 0/10 [00:00<?, ?it/s]
eval model::  10%|█         | 1/10 [00:00<00:08,  1.03it/s]
eval model::  20%|██        | 2/10 [00:01<00:04,  1.95it/s]
eval model::  30%|███       | 3/10 [00:01<00:02,  2.69it/s]
eval model::  40%|████      | 4/10 [00:01<00:01,  3.30it/s]
eval model::  50%|█████     | 5/10 [00:01<00:01,  3.79it/s]
eval model::  60%|██████    | 6/10 [00:01<00:00,  4.16it/s]
eval model::  70%|███████   | 7/10 [00:02<00:00,  4.40it/s]
eval model::  80%|████████  | 8/10 [00:02<00:00,  4.57it/s]
eval model::  90%|█████████ | 9/10 [00:02<00:00,  4.73it/s]
eval model:: 100%|██████████| 10/10 [00:02<00:00,  3.63it/s]
[2024/12/18 00:49:52] ppocr INFO: cur metric, acc: 0.826666639111112, norm_edit_dis: 0.9906255838792406, fps: 160.51409773428998
[2024/12/18 00:49:53] ppocr INFO: save best model is to /kaggle/working/finetuned_model/best_accuracy
[2024/12/18 00:49:53] ppocr INFO: best metric, acc: 0.826666639111112, is_float16: False, norm_edit_dis: 0.9906255838792406, fps: 160.51409773428998, best_epoch: 5
[2024/12/18 00:49:58] ppocr INFO: epoch: [5/50], global_step: 105, lr: 0.000499, acc: 0.843750, norm_edit_dis: 0.985954, CTCLoss: 1.062377, NRTRLoss: 0.796567, loss: 1.861096, avg_reader_cost: 0.00023 s, avg_batch_cost: 0.04190 s, avg_samples: 1.6, ips: 38.18615 samples/s, eta: 0:21:24, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 00:50:00] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 00:50:26] ppocr INFO: epoch: [6/50], global_step: 126, lr: 0.000497, acc: 0.843750, norm_edit_dis: 0.988724, CTCLoss: 0.909760, NRTRLoss: 0.795288, loss: 1.704313, avg_reader_cost: 0.11447 s, avg_batch_cost: 0.28533 s, avg_samples: 6.72, ips: 23.55146 samples/s, eta: 0:20:55, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 00:50:29] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 00:50:53] ppocr INFO: epoch: [7/50], global_step: 147, lr: 0.000494, acc: 0.875000, norm_edit_dis: 0.988683, CTCLoss: 1.025994, NRTRLoss: 0.795888, loss: 1.821981, avg_reader_cost: 0.08255 s, avg_batch_cost: 0.27189 s, avg_samples: 6.72, ips: 24.71550 samples/s, eta: 0:20:18, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 00:50:56] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 00:51:21] ppocr INFO: epoch: [8/50], global_step: 168, lr: 0.000490, acc: 0.890625, norm_edit_dis: 0.989747, CTCLoss: 0.805997, NRTRLoss: 0.790904, loss: 1.598129, avg_reader_cost: 0.08454 s, avg_batch_cost: 0.27891 s, avg_samples: 6.72, ips: 24.09338 samples/s, eta: 0:19:47, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 00:51:24] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 00:51:49] ppocr INFO: epoch: [9/50], global_step: 189, lr: 0.000485, acc: 0.906250, norm_edit_dis: 0.991241, CTCLoss: 0.602570, NRTRLoss: 0.791981, loss: 1.393008, avg_reader_cost: 0.10378 s, avg_batch_cost: 0.27869 s, avg_samples: 6.72, ips: 24.11296 samples/s, eta: 0:19:17, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 00:51:52] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 00:52:09] ppocr INFO: epoch: [10/50], global_step: 200, lr: 0.000482, acc: 0.906250, norm_edit_dis: 0.991241, CTCLoss: 0.613344, NRTRLoss: 0.791047, loss: 1.404161, avg_reader_cost: 0.10879 s, avg_batch_cost: 0.20219 s, avg_samples: 3.52, ips: 17.40971 samples/s, eta: 0:19:26, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB

eval model::   0%|          | 0/10 [00:00<?, ?it/s]
eval model::  10%|█         | 1/10 [00:00<00:05,  1.56it/s]
eval model::  20%|██        | 2/10 [00:00<00:03,  2.62it/s]
eval model::  30%|███       | 3/10 [00:01<00:02,  3.35it/s]
eval model::  40%|████      | 4/10 [00:01<00:01,  3.83it/s]
eval model::  50%|█████     | 5/10 [00:01<00:01,  4.17it/s]
eval model::  60%|██████    | 6/10 [00:01<00:00,  4.40it/s]
eval model::  70%|███████   | 7/10 [00:01<00:00,  4.57it/s]
eval model::  80%|████████  | 8/10 [00:02<00:00,  4.68it/s]
eval model::  90%|█████████ | 9/10 [00:02<00:00,  4.75it/s]
eval model:: 100%|██████████| 10/10 [00:02<00:00,  4.07it/s]
[2024/12/18 00:52:12] ppocr INFO: cur metric, acc: 0.8633333045555566, norm_edit_dis: 0.9906255838792406, fps: 160.52161277956412
[2024/12/18 00:52:14] ppocr INFO: save best model is to /kaggle/working/finetuned_model/best_accuracy
[2024/12/18 00:52:14] ppocr INFO: best metric, acc: 0.8633333045555566, is_float16: False, norm_edit_dis: 0.9906255838792406, fps: 160.52161277956412, best_epoch: 10
[2024/12/18 00:52:22] ppocr INFO: epoch: [10/50], global_step: 210, lr: 0.000479, acc: 0.906250, norm_edit_dis: 0.991207, CTCLoss: 0.664687, NRTRLoss: 0.789161, loss: 1.453209, avg_reader_cost: 0.00049 s, avg_batch_cost: 0.08304 s, avg_samples: 3.2, ips: 38.53581 samples/s, eta: 0:18:50, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 00:52:24] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 00:52:26] ppocr INFO: save model in /kaggle/working/finetuned_model/iter_epoch_10
[2024/12/18 00:52:50] ppocr INFO: epoch: [11/50], global_step: 231, lr: 0.000473, acc: 0.906250, norm_edit_dis: 0.990249, CTCLoss: 0.667837, NRTRLoss: 0.789434, loss: 1.456306, avg_reader_cost: 0.09411 s, avg_batch_cost: 0.27563 s, avg_samples: 6.72, ips: 24.38053 samples/s, eta: 0:18:19, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 00:52:53] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 00:53:18] ppocr INFO: epoch: [12/50], global_step: 252, lr: 0.000465, acc: 0.906250, norm_edit_dis: 0.991849, CTCLoss: 0.629462, NRTRLoss: 0.790014, loss: 1.418765, avg_reader_cost: 0.10047 s, avg_batch_cost: 0.28140 s, avg_samples: 6.72, ips: 23.88074 samples/s, eta: 0:17:51, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 00:53:21] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 00:53:46] ppocr INFO: epoch: [13/50], global_step: 273, lr: 0.000457, acc: 0.906250, norm_edit_dis: 0.993304, CTCLoss: 0.454486, NRTRLoss: 0.788938, loss: 1.243901, avg_reader_cost: 0.10361 s, avg_batch_cost: 0.28022 s, avg_samples: 6.72, ips: 23.98145 samples/s, eta: 0:17:22, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 00:53:49] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 00:54:13] ppocr INFO: epoch: [14/50], global_step: 294, lr: 0.000448, acc: 0.906250, norm_edit_dis: 0.991849, CTCLoss: 0.635339, NRTRLoss: 0.788913, loss: 1.425121, avg_reader_cost: 0.07422 s, avg_batch_cost: 0.26765 s, avg_samples: 6.72, ips: 25.10718 samples/s, eta: 0:16:50, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 00:54:16] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 00:54:29] ppocr INFO: epoch: [15/50], global_step: 300, lr: 0.000445, acc: 0.906250, norm_edit_dis: 0.992593, CTCLoss: 0.629642, NRTRLoss: 0.788995, loss: 1.419262, avg_reader_cost: 0.08756 s, avg_batch_cost: 0.16047 s, avg_samples: 1.92, ips: 11.96466 samples/s, eta: 0:17:03, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB

eval model::   0%|          | 0/10 [00:00<?, ?it/s]
eval model::  10%|█         | 1/10 [00:01<00:11,  1.30s/it]
eval model::  20%|██        | 2/10 [00:01<00:05,  1.52it/s]
eval model::  30%|███       | 3/10 [00:01<00:03,  2.22it/s]
eval model::  40%|████      | 4/10 [00:01<00:02,  2.81it/s]
eval model::  50%|█████     | 5/10 [00:02<00:01,  3.29it/s]
eval model::  60%|██████    | 6/10 [00:02<00:01,  3.68it/s]
eval model::  70%|███████   | 7/10 [00:02<00:00,  4.06it/s]
eval model::  80%|████████  | 8/10 [00:02<00:00,  4.33it/s]
eval model::  90%|█████████ | 9/10 [00:02<00:00,  4.54it/s]
eval model:: 100%|██████████| 10/10 [00:03<00:00,  3.19it/s]
[2024/12/18 00:54:32] ppocr INFO: cur metric, acc: 0.8633333045555566, norm_edit_dis: 0.9923716155670713, fps: 160.44362980271148
[2024/12/18 00:54:34] ppocr INFO: save best model is to /kaggle/working/finetuned_model/best_accuracy
[2024/12/18 00:54:34] ppocr INFO: best metric, acc: 0.8633333045555566, is_float16: False, norm_edit_dis: 0.9923716155670713, fps: 160.44362980271148, best_epoch: 15
[2024/12/18 00:54:45] ppocr INFO: epoch: [15/50], global_step: 315, lr: 0.000438, acc: 0.906250, norm_edit_dis: 0.991567, CTCLoss: 0.558021, NRTRLoss: 0.789872, loss: 1.351896, avg_reader_cost: 0.00083 s, avg_batch_cost: 0.11463 s, avg_samples: 4.8, ips: 41.87407 samples/s, eta: 0:16:21, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 00:54:48] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 00:55:14] ppocr INFO: epoch: [16/50], global_step: 336, lr: 0.000427, acc: 0.937500, norm_edit_dis: 0.994183, CTCLoss: 0.422507, NRTRLoss: 0.788043, loss: 1.210535, avg_reader_cost: 0.08858 s, avg_batch_cost: 0.28126 s, avg_samples: 6.72, ips: 23.89217 samples/s, eta: 0:15:53, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 00:55:16] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 00:55:41] ppocr INFO: epoch: [17/50], global_step: 357, lr: 0.000415, acc: 0.906250, norm_edit_dis: 0.992593, CTCLoss: 0.607296, NRTRLoss: 0.788361, loss: 1.394953, avg_reader_cost: 0.08815 s, avg_batch_cost: 0.27093 s, avg_samples: 6.72, ips: 24.80340 samples/s, eta: 0:15:23, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 00:55:44] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 00:56:09] ppocr INFO: epoch: [18/50], global_step: 378, lr: 0.000403, acc: 0.937500, norm_edit_dis: 0.991815, CTCLoss: 0.618196, NRTRLoss: 0.788099, loss: 1.406234, avg_reader_cost: 0.10896 s, avg_batch_cost: 0.28600 s, avg_samples: 6.72, ips: 23.49687 samples/s, eta: 0:14:56, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 00:56:12] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 00:56:37] ppocr INFO: epoch: [19/50], global_step: 399, lr: 0.000391, acc: 0.937500, norm_edit_dis: 0.994048, CTCLoss: 0.400525, NRTRLoss: 0.787620, loss: 1.188145, avg_reader_cost: 0.08922 s, avg_batch_cost: 0.27596 s, avg_samples: 6.72, ips: 24.35107 samples/s, eta: 0:14:28, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 00:56:40] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 00:56:48] ppocr INFO: epoch: [20/50], global_step: 400, lr: 0.000390, acc: 0.937500, norm_edit_dis: 0.993439, CTCLoss: 0.412338, NRTRLoss: 0.787612, loss: 1.199727, avg_reader_cost: 0.09355 s, avg_batch_cost: 0.10602 s, avg_samples: 0.32, ips: 3.01826 samples/s, eta: 0:14:41, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB

eval model::   0%|          | 0/10 [00:00<?, ?it/s]
eval model::  10%|█         | 1/10 [00:01<00:13,  1.54s/it]
eval model::  20%|██        | 2/10 [00:01<00:06,  1.29it/s]
eval model::  30%|███       | 3/10 [00:02<00:04,  1.72it/s]
eval model::  40%|████      | 4/10 [00:02<00:02,  2.30it/s]
eval model::  50%|█████     | 5/10 [00:02<00:01,  2.52it/s]
eval model::  60%|██████    | 6/10 [00:03<00:01,  2.66it/s]
eval model::  70%|███████   | 7/10 [00:03<00:01,  2.80it/s]
eval model::  80%|████████  | 8/10 [00:03<00:00,  2.96it/s]
eval model::  90%|█████████ | 9/10 [00:03<00:00,  3.14it/s]
eval model:: 100%|██████████| 10/10 [00:04<00:00,  3.53it/s]
eval model:: 100%|██████████| 10/10 [00:04<00:00,  2.21it/s]
[2024/12/18 00:56:52] ppocr INFO: cur metric, acc: 0.8633333045555566, norm_edit_dis: 0.9923716155670713, fps: 111.17107973203545
[2024/12/18 00:56:54] ppocr INFO: save best model is to /kaggle/working/finetuned_model/best_accuracy
[2024/12/18 00:56:54] ppocr INFO: best metric, acc: 0.8633333045555566, is_float16: False, norm_edit_dis: 0.9923716155670713, fps: 111.17107973203545, best_epoch: 20
[2024/12/18 00:57:10] ppocr INFO: epoch: [20/50], global_step: 420, lr: 0.000377, acc: 0.937500, norm_edit_dis: 0.996276, CTCLoss: 0.307992, NRTRLoss: 0.787739, loss: 1.096902, avg_reader_cost: 0.00131 s, avg_batch_cost: 0.15509 s, avg_samples: 6.4, ips: 41.26628 samples/s, eta: 0:13:57, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 00:57:13] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 00:57:15] ppocr INFO: save model in /kaggle/working/finetuned_model/iter_epoch_20
[2024/12/18 00:57:39] ppocr INFO: epoch: [21/50], global_step: 441, lr: 0.000363, acc: 0.937500, norm_edit_dis: 0.996314, CTCLoss: 0.264274, NRTRLoss: 0.787034, loss: 1.051516, avg_reader_cost: 0.10135 s, avg_batch_cost: 0.28982 s, avg_samples: 6.72, ips: 23.18704 samples/s, eta: 0:13:30, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 00:57:42] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 00:58:06] ppocr INFO: epoch: [22/50], global_step: 462, lr: 0.000349, acc: 0.953125, norm_edit_dis: 0.995671, CTCLoss: 0.309293, NRTRLoss: 0.787237, loss: 1.096391, avg_reader_cost: 0.08825 s, avg_batch_cost: 0.26886 s, avg_samples: 6.72, ips: 24.99403 samples/s, eta: 0:13:01, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 00:58:09] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 00:58:34] ppocr INFO: epoch: [23/50], global_step: 483, lr: 0.000335, acc: 0.937500, norm_edit_dis: 0.994792, CTCLoss: 0.368578, NRTRLoss: 0.787078, loss: 1.156222, avg_reader_cost: 0.09128 s, avg_batch_cost: 0.27645 s, avg_samples: 6.72, ips: 24.30842 samples/s, eta: 0:12:33, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 00:58:36] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 00:58:57] ppocr INFO: epoch: [24/50], global_step: 500, lr: 0.000323, acc: 0.953125, norm_edit_dis: 0.997024, CTCLoss: 0.256581, NRTRLoss: 0.787186, loss: 1.044083, avg_reader_cost: 0.08893 s, avg_batch_cost: 0.23898 s, avg_samples: 5.44, ips: 22.76364 samples/s, eta: 0:12:12, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB

eval model::   0%|          | 0/10 [00:00<?, ?it/s]
eval model::  10%|█         | 1/10 [00:00<00:05,  1.52it/s]
eval model::  20%|██        | 2/10 [00:00<00:03,  2.59it/s]
eval model::  30%|███       | 3/10 [00:01<00:02,  3.33it/s]
eval model::  40%|████      | 4/10 [00:01<00:01,  3.87it/s]
eval model::  50%|█████     | 5/10 [00:01<00:01,  4.23it/s]
eval model::  60%|██████    | 6/10 [00:01<00:00,  4.48it/s]
eval model::  70%|███████   | 7/10 [00:01<00:00,  4.66it/s]
eval model::  80%|████████  | 8/10 [00:02<00:00,  4.80it/s]
eval model::  90%|█████████ | 9/10 [00:02<00:00,  4.86it/s]
eval model:: 100%|██████████| 10/10 [00:02<00:00,  3.93it/s]
[2024/12/18 00:59:00] ppocr INFO: cur metric, acc: 0.8633333045555566, norm_edit_dis: 0.9923716155670713, fps: 164.13345266126194
[2024/12/18 00:59:02] ppocr INFO: save best model is to /kaggle/working/finetuned_model/best_accuracy
[2024/12/18 00:59:02] ppocr INFO: best metric, acc: 0.8633333045555566, is_float16: False, norm_edit_dis: 0.9923716155670713, fps: 164.13345266126194, best_epoch: 24
[2024/12/18 00:59:06] ppocr INFO: epoch: [24/50], global_step: 504, lr: 0.000320, acc: 0.937500, norm_edit_dis: 0.997024, CTCLoss: 0.259899, NRTRLoss: 0.787252, loss: 1.047466, avg_reader_cost: 0.00023 s, avg_batch_cost: 0.03764 s, avg_samples: 1.28, ips: 34.00528 samples/s, eta: 0:12:05, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 00:59:08] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 00:59:33] ppocr INFO: epoch: [25/50], global_step: 525, lr: 0.000305, acc: 0.968750, norm_edit_dis: 0.997024, CTCLoss: 0.276437, NRTRLoss: 0.786924, loss: 1.063228, avg_reader_cost: 0.09855 s, avg_batch_cost: 0.27090 s, avg_samples: 6.72, ips: 24.80623 samples/s, eta: 0:11:36, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 00:59:36] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 01:00:00] ppocr INFO: epoch: [26/50], global_step: 546, lr: 0.000289, acc: 0.937500, norm_edit_dis: 0.995536, CTCLoss: 0.295021, NRTRLoss: 0.787083, loss: 1.083288, avg_reader_cost: 0.08875 s, avg_batch_cost: 0.26833 s, avg_samples: 6.72, ips: 25.04363 samples/s, eta: 0:11:07, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 01:00:03] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 01:00:28] ppocr INFO: epoch: [27/50], global_step: 567, lr: 0.000274, acc: 0.937500, norm_edit_dis: 0.994927, CTCLoss: 0.360506, NRTRLoss: 0.787029, loss: 1.148184, avg_reader_cost: 0.10612 s, avg_batch_cost: 0.28568 s, avg_samples: 6.72, ips: 23.52249 samples/s, eta: 0:10:40, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 01:00:31] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 01:00:57] ppocr INFO: epoch: [28/50], global_step: 588, lr: 0.000258, acc: 0.968750, norm_edit_dis: 0.996280, CTCLoss: 0.245300, NRTRLoss: 0.786948, loss: 1.032283, avg_reader_cost: 0.10109 s, avg_batch_cost: 0.28222 s, avg_samples: 6.72, ips: 23.81092 samples/s, eta: 0:10:12, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 01:01:00] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 01:01:18] ppocr INFO: epoch: [29/50], global_step: 600, lr: 0.000249, acc: 0.968750, norm_edit_dis: 0.997024, CTCLoss: 0.196759, NRTRLoss: 0.786720, loss: 0.984118, avg_reader_cost: 0.10117 s, avg_batch_cost: 0.21729 s, avg_samples: 3.84, ips: 17.67223 samples/s, eta: 0:10:01, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB

eval model::   0%|          | 0/10 [00:00<?, ?it/s]
eval model::  10%|█         | 1/10 [00:00<00:05,  1.70it/s]
eval model::  20%|██        | 2/10 [00:00<00:02,  2.82it/s]
eval model::  30%|███       | 3/10 [00:00<00:01,  3.56it/s]
eval model::  40%|████      | 4/10 [00:01<00:01,  4.06it/s]
eval model::  50%|█████     | 5/10 [00:01<00:01,  4.42it/s]
eval model::  60%|██████    | 6/10 [00:01<00:00,  4.64it/s]
eval model::  70%|███████   | 7/10 [00:01<00:00,  4.81it/s]
eval model::  80%|████████  | 8/10 [00:01<00:00,  4.93it/s]
eval model::  90%|█████████ | 9/10 [00:02<00:00,  5.01it/s]
eval model:: 100%|██████████| 10/10 [00:02<00:00,  4.19it/s]
[2024/12/18 01:01:21] ppocr INFO: cur metric, acc: 0.8633333045555566, norm_edit_dis: 0.9923716155670713, fps: 166.57788540319794
[2024/12/18 01:01:23] ppocr INFO: save best model is to /kaggle/working/finetuned_model/best_accuracy
[2024/12/18 01:01:23] ppocr INFO: best metric, acc: 0.8633333045555566, is_float16: False, norm_edit_dis: 0.9923716155670713, fps: 166.57788540319794, best_epoch: 29
[2024/12/18 01:01:30] ppocr INFO: epoch: [29/50], global_step: 609, lr: 0.000242, acc: 0.968750, norm_edit_dis: 0.996849, CTCLoss: 0.283799, NRTRLoss: 0.787039, loss: 1.070859, avg_reader_cost: 0.00045 s, avg_batch_cost: 0.07501 s, avg_samples: 2.88, ips: 38.39396 samples/s, eta: 0:09:46, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 01:01:33] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 01:01:58] ppocr INFO: epoch: [30/50], global_step: 630, lr: 0.000226, acc: 0.937500, norm_edit_dis: 0.996619, CTCLoss: 0.277808, NRTRLoss: 0.786822, loss: 1.064571, avg_reader_cost: 0.09589 s, avg_batch_cost: 0.28088 s, avg_samples: 6.72, ips: 23.92519 samples/s, eta: 0:09:18, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 01:02:01] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 01:02:04] ppocr INFO: save model in /kaggle/working/finetuned_model/iter_epoch_30
[2024/12/18 01:02:27] ppocr INFO: epoch: [31/50], global_step: 651, lr: 0.000211, acc: 0.968750, norm_edit_dis: 0.997024, CTCLoss: 0.233888, NRTRLoss: 0.786804, loss: 1.020577, avg_reader_cost: 0.10592 s, avg_batch_cost: 0.28460 s, avg_samples: 6.72, ips: 23.61242 samples/s, eta: 0:08:50, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 01:02:29] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 01:02:56] ppocr INFO: epoch: [32/50], global_step: 672, lr: 0.000195, acc: 0.937500, norm_edit_dis: 0.995536, CTCLoss: 0.323963, NRTRLoss: 0.786990, loss: 1.111283, avg_reader_cost: 0.11261 s, avg_batch_cost: 0.29188 s, avg_samples: 6.72, ips: 23.02308 samples/s, eta: 0:08:23, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 01:02:59] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 01:03:23] ppocr INFO: epoch: [33/50], global_step: 693, lr: 0.000180, acc: 0.953125, norm_edit_dis: 0.997091, CTCLoss: 0.202008, NRTRLoss: 0.786659, loss: 0.988647, avg_reader_cost: 0.08591 s, avg_batch_cost: 0.27126 s, avg_samples: 6.72, ips: 24.77283 samples/s, eta: 0:07:55, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 01:03:26] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 01:03:40] ppocr INFO: epoch: [34/50], global_step: 700, lr: 0.000175, acc: 0.953125, norm_edit_dis: 0.997058, CTCLoss: 0.206990, NRTRLoss: 0.786659, loss: 0.993738, avg_reader_cost: 0.10429 s, avg_batch_cost: 0.17226 s, avg_samples: 2.24, ips: 13.00367 samples/s, eta: 0:07:49, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB

eval model::   0%|          | 0/10 [00:00<?, ?it/s]
eval model::  10%|█         | 1/10 [00:00<00:07,  1.14it/s]
eval model::  20%|██        | 2/10 [00:01<00:03,  2.06it/s]
eval model::  30%|███       | 3/10 [00:01<00:02,  2.80it/s]
eval model::  40%|████      | 4/10 [00:01<00:01,  3.41it/s]
eval model::  50%|█████     | 5/10 [00:01<00:01,  3.90it/s]
eval model::  60%|██████    | 6/10 [00:01<00:00,  4.26it/s]
eval model::  70%|███████   | 7/10 [00:02<00:00,  4.53it/s]
eval model::  80%|████████  | 8/10 [00:02<00:00,  4.71it/s]
eval model::  90%|█████████ | 9/10 [00:02<00:00,  4.87it/s]
eval model:: 100%|██████████| 10/10 [00:02<00:00,  3.70it/s]
[2024/12/18 01:03:43] ppocr INFO: cur metric, acc: 0.8999999700000011, norm_edit_dis: 0.9941176472549019, fps: 161.7683332191275
[2024/12/18 01:03:45] ppocr INFO: save best model is to /kaggle/working/finetuned_model/best_accuracy
[2024/12/18 01:03:45] ppocr INFO: best metric, acc: 0.8999999700000011, is_float16: False, norm_edit_dis: 0.9941176472549019, fps: 161.7683332191275, best_epoch: 34
[2024/12/18 01:03:56] ppocr INFO: epoch: [34/50], global_step: 714, lr: 0.000165, acc: 0.968750, norm_edit_dis: 0.997024, CTCLoss: 0.230437, NRTRLoss: 0.786828, loss: 1.017888, avg_reader_cost: 0.00085 s, avg_batch_cost: 0.11206 s, avg_samples: 4.48, ips: 39.97785 samples/s, eta: 0:07:27, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 01:03:59] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 01:04:24] ppocr INFO: epoch: [35/50], global_step: 735, lr: 0.000151, acc: 0.968750, norm_edit_dis: 0.996381, CTCLoss: 0.242623, NRTRLoss: 0.786642, loss: 1.029247, avg_reader_cost: 0.08275 s, avg_batch_cost: 0.27650 s, avg_samples: 6.72, ips: 24.30403 samples/s, eta: 0:06:59, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 01:04:27] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 01:04:51] ppocr INFO: epoch: [36/50], global_step: 756, lr: 0.000137, acc: 0.968750, norm_edit_dis: 0.998512, CTCLoss: 0.135651, NRTRLoss: 0.786695, loss: 0.922399, avg_reader_cost: 0.08181 s, avg_batch_cost: 0.27399 s, avg_samples: 6.72, ips: 24.52672 samples/s, eta: 0:06:31, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 01:04:54] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 01:05:19] ppocr INFO: epoch: [37/50], global_step: 777, lr: 0.000123, acc: 0.968750, norm_edit_dis: 0.997802, CTCLoss: 0.146370, NRTRLoss: 0.786725, loss: 0.933194, avg_reader_cost: 0.08319 s, avg_batch_cost: 0.27418 s, avg_samples: 6.72, ips: 24.50903 samples/s, eta: 0:06:02, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 01:05:22] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 01:05:49] ppocr INFO: epoch: [38/50], global_step: 798, lr: 0.000109, acc: 0.937500, norm_edit_dis: 0.995536, CTCLoss: 0.242811, NRTRLoss: 0.786914, loss: 1.030073, avg_reader_cost: 0.11654 s, avg_batch_cost: 0.29901 s, avg_samples: 6.72, ips: 22.47405 samples/s, eta: 0:05:35, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 01:05:51] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 01:06:00] ppocr INFO: epoch: [39/50], global_step: 800, lr: 0.000108, acc: 0.953125, norm_edit_dis: 0.997024, CTCLoss: 0.200385, NRTRLoss: 0.786755, loss: 0.987297, avg_reader_cost: 0.08324 s, avg_batch_cost: 0.11186 s, avg_samples: 0.64, ips: 5.72134 samples/s, eta: 0:05:35, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB

eval model::   0%|          | 0/10 [00:00<?, ?it/s]
eval model::  10%|█         | 1/10 [00:02<00:18,  2.07s/it]
eval model::  20%|██        | 2/10 [00:02<00:07,  1.01it/s]
eval model::  30%|███       | 3/10 [00:02<00:04,  1.57it/s]
eval model::  40%|████      | 4/10 [00:02<00:02,  2.11it/s]
eval model::  50%|█████     | 5/10 [00:02<00:01,  2.60it/s]
eval model::  60%|██████    | 6/10 [00:03<00:01,  3.05it/s]
eval model::  70%|███████   | 7/10 [00:03<00:00,  3.49it/s]
eval model::  80%|████████  | 8/10 [00:03<00:00,  3.88it/s]
eval model::  90%|█████████ | 9/10 [00:03<00:00,  4.19it/s]
eval model:: 100%|██████████| 10/10 [00:04<00:00,  2.43it/s]
[2024/12/18 01:06:04] ppocr INFO: cur metric, acc: 0.8999999700000011, norm_edit_dis: 0.9941176472549019, fps: 152.00270980579563
[2024/12/18 01:06:06] ppocr INFO: save best model is to /kaggle/working/finetuned_model/best_accuracy
[2024/12/18 01:06:06] ppocr INFO: best metric, acc: 0.8999999700000011, is_float16: False, norm_edit_dis: 0.9941176472549019, fps: 152.00270980579563, best_epoch: 39
[2024/12/18 01:06:21] ppocr INFO: epoch: [39/50], global_step: 819, lr: 0.000097, acc: 0.937500, norm_edit_dis: 0.995536, CTCLoss: 0.332776, NRTRLoss: 0.786750, loss: 1.119401, avg_reader_cost: 0.00117 s, avg_batch_cost: 0.14394 s, avg_samples: 6.08, ips: 42.23871 samples/s, eta: 0:05:07, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 01:06:23] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 01:06:49] ppocr INFO: epoch: [40/50], global_step: 840, lr: 0.000085, acc: 0.937500, norm_edit_dis: 0.996269, CTCLoss: 0.245384, NRTRLoss: 0.786821, loss: 1.032043, avg_reader_cost: 0.09470 s, avg_batch_cost: 0.28709 s, avg_samples: 6.72, ips: 23.40735 samples/s, eta: 0:04:39, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 01:06:53] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 01:06:55] ppocr INFO: save model in /kaggle/working/finetuned_model/iter_epoch_40
[2024/12/18 01:07:17] ppocr INFO: epoch: [41/50], global_step: 861, lr: 0.000073, acc: 0.968750, norm_edit_dis: 0.998512, CTCLoss: 0.163021, NRTRLoss: 0.786599, loss: 0.949926, avg_reader_cost: 0.10334 s, avg_batch_cost: 0.28141 s, avg_samples: 6.72, ips: 23.87969 samples/s, eta: 0:04:11, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 01:07:21] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 01:07:46] ppocr INFO: epoch: [42/50], global_step: 882, lr: 0.000062, acc: 0.968750, norm_edit_dis: 0.997024, CTCLoss: 0.223136, NRTRLoss: 0.786670, loss: 1.009778, avg_reader_cost: 0.09901 s, avg_batch_cost: 0.28062 s, avg_samples: 6.72, ips: 23.94691 samples/s, eta: 0:03:43, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 01:07:48] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 01:08:10] ppocr INFO: epoch: [43/50], global_step: 900, lr: 0.000054, acc: 0.968750, norm_edit_dis: 0.997024, CTCLoss: 0.222292, NRTRLoss: 0.786605, loss: 1.008972, avg_reader_cost: 0.08479 s, avg_batch_cost: 0.24232 s, avg_samples: 5.76, ips: 23.77041 samples/s, eta: 0:03:19, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB

eval model::   0%|          | 0/10 [00:00<?, ?it/s]
eval model::  10%|█         | 1/10 [00:00<00:07,  1.26it/s]
eval model::  20%|██        | 2/10 [00:00<00:03,  2.27it/s]
eval model::  30%|███       | 3/10 [00:01<00:02,  3.06it/s]
eval model::  40%|████      | 4/10 [00:01<00:01,  3.63it/s]
eval model::  50%|█████     | 5/10 [00:01<00:01,  4.04it/s]
eval model::  60%|██████    | 6/10 [00:01<00:00,  4.37it/s]
eval model::  70%|███████   | 7/10 [00:01<00:00,  4.58it/s]
eval model::  80%|████████  | 8/10 [00:02<00:00,  4.75it/s]
eval model::  90%|█████████ | 9/10 [00:02<00:00,  4.89it/s]
eval model:: 100%|██████████| 10/10 [00:02<00:00,  3.80it/s]
[2024/12/18 01:08:13] ppocr INFO: cur metric, acc: 0.8999999700000011, norm_edit_dis: 0.9941176472549019, fps: 166.11400995265262
[2024/12/18 01:08:15] ppocr INFO: save best model is to /kaggle/working/finetuned_model/best_accuracy
[2024/12/18 01:08:15] ppocr INFO: best metric, acc: 0.8999999700000011, is_float16: False, norm_edit_dis: 0.9941176472549019, fps: 166.11400995265262, best_epoch: 43
[2024/12/18 01:08:17] ppocr INFO: epoch: [43/50], global_step: 903, lr: 0.000052, acc: 0.968750, norm_edit_dis: 0.996883, CTCLoss: 0.222292, NRTRLoss: 0.786612, loss: 1.008972, avg_reader_cost: 0.00016 s, avg_batch_cost: 0.02490 s, avg_samples: 0.96, ips: 38.55562 samples/s, eta: 0:03:15, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 01:08:20] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 01:08:45] ppocr INFO: epoch: [44/50], global_step: 924, lr: 0.000043, acc: 0.937500, norm_edit_dis: 0.996280, CTCLoss: 0.341585, NRTRLoss: 0.786872, loss: 1.128871, avg_reader_cost: 0.09677 s, avg_batch_cost: 0.27827 s, avg_samples: 6.72, ips: 24.14935 samples/s, eta: 0:02:47, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 01:08:48] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 01:09:12] ppocr INFO: epoch: [45/50], global_step: 945, lr: 0.000035, acc: 0.968750, norm_edit_dis: 0.998546, CTCLoss: 0.090673, NRTRLoss: 0.786557, loss: 0.877466, avg_reader_cost: 0.09022 s, avg_batch_cost: 0.27493 s, avg_samples: 6.72, ips: 24.44227 samples/s, eta: 0:02:19, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 01:09:15] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 01:09:40] ppocr INFO: epoch: [46/50], global_step: 966, lr: 0.000027, acc: 0.937500, norm_edit_dis: 0.994081, CTCLoss: 0.361183, NRTRLoss: 0.786825, loss: 1.148143, avg_reader_cost: 0.10879 s, avg_batch_cost: 0.27912 s, avg_samples: 6.72, ips: 24.07592 samples/s, eta: 0:01:51, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 01:09:44] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 01:10:09] ppocr INFO: epoch: [47/50], global_step: 987, lr: 0.000021, acc: 0.953125, norm_edit_dis: 0.997024, CTCLoss: 0.245037, NRTRLoss: 0.786729, loss: 1.031742, avg_reader_cost: 0.10107 s, avg_batch_cost: 0.28322 s, avg_samples: 6.72, ips: 23.72694 samples/s, eta: 0:01:23, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 01:10:12] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 01:10:30] ppocr INFO: epoch: [48/50], global_step: 1000, lr: 0.000017, acc: 0.968750, norm_edit_dis: 0.997024, CTCLoss: 0.176089, NRTRLoss: 0.786669, loss: 0.962813, avg_reader_cost: 0.07852 s, avg_batch_cost: 0.21225 s, avg_samples: 4.16, ips: 19.59917 samples/s, eta: 0:01:06, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB

eval model::   0%|          | 0/10 [00:00<?, ?it/s]
eval model::  10%|█         | 1/10 [00:00<00:06,  1.47it/s]
eval model::  20%|██        | 2/10 [00:00<00:03,  2.53it/s]
eval model::  30%|███       | 3/10 [00:01<00:02,  3.28it/s]
eval model::  40%|████      | 4/10 [00:01<00:01,  3.84it/s]
eval model::  50%|█████     | 5/10 [00:01<00:01,  4.21it/s]
eval model::  60%|██████    | 6/10 [00:01<00:00,  4.46it/s]
eval model::  70%|███████   | 7/10 [00:01<00:00,  4.65it/s]
eval model::  80%|████████  | 8/10 [00:02<00:00,  4.81it/s]
eval model::  90%|█████████ | 9/10 [00:02<00:00,  4.90it/s]
eval model:: 100%|██████████| 10/10 [00:02<00:00,  3.92it/s]
[2024/12/18 01:10:33] ppocr INFO: cur metric, acc: 0.8999999700000011, norm_edit_dis: 0.9941176472549019, fps: 162.9157290066991
[2024/12/18 01:10:35] ppocr INFO: save best model is to /kaggle/working/finetuned_model/best_accuracy
[2024/12/18 01:10:35] ppocr INFO: best metric, acc: 0.8999999700000011, is_float16: False, norm_edit_dis: 0.9941176472549019, fps: 162.9157290066991, best_epoch: 48
[2024/12/18 01:10:41] ppocr INFO: epoch: [48/50], global_step: 1008, lr: 0.000015, acc: 0.968750, norm_edit_dis: 0.997690, CTCLoss: 0.139588, NRTRLoss: 0.786574, loss: 0.926075, avg_reader_cost: 0.00042 s, avg_batch_cost: 0.06538 s, avg_samples: 2.56, ips: 39.15837 samples/s, eta: 0:00:55, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 01:10:44] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 01:11:08] ppocr INFO: epoch: [49/50], global_step: 1029, lr: 0.000010, acc: 0.937500, norm_edit_dis: 0.995831, CTCLoss: 0.220417, NRTRLoss: 0.786663, loss: 1.009310, avg_reader_cost: 0.08387 s, avg_batch_cost: 0.27039 s, avg_samples: 6.72, ips: 24.85277 samples/s, eta: 0:00:27, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 01:11:11] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 01:11:37] ppocr INFO: epoch: [50/50], global_step: 1050, lr: 0.000006, acc: 0.968750, norm_edit_dis: 0.995944, CTCLoss: 0.221743, NRTRLoss: 0.786791, loss: 1.008275, avg_reader_cost: 0.09388 s, avg_batch_cost: 0.28869 s, avg_samples: 6.72, ips: 23.27740 samples/s, eta: 0:00:00, max_mem_reserved: 12071 MB, max_mem_allocated: 7876 MB
[2024/12/18 01:11:38] ppocr INFO: save model in /kaggle/working/finetuned_model/latest
[2024/12/18 01:11:39] ppocr INFO: save model in /kaggle/working/finetuned_model/iter_epoch_50
[2024/12/18 01:11:39] ppocr INFO: best metric, acc: 0.8999999700000011, is_float16: False, norm_edit_dis: 0.9941176472549019, fps: 162.9157290066991, best_epoch: 48
I1218 01:11:40.001865   255 process_group_nccl.cc:132] ProcessGroupNCCL destruct 
I1218 01:11:40.155754   279 tcp_store.cc:289] receive shutdown event and so quit from MasterDaemon run loop
LAUNCH INFO 2024-12-18 01:11:41,068 Pod completed
LAUNCH INFO 2024-12-18 01:11:41,068 Exit code 0

After the training was completed I exported the best_accuracy to inference model with

!python3 /kaggle/working/finetune_paddle_ocr/PaddleOCR/tools/export_model.py -c /kaggle/working/yml_file/rec_configs.yml -o Global.pretrained_model=/kaggle/working/finetuned_model/best_accuracy Global.save_inference_dir=/kaggle/working/inference/finetuned_en_PP-OCRv4_rec/

and ran inference with

!python3 /kaggle/working/finetune_paddle_ocr/PaddleOCR/tools/infer/predict_rec.py --rec_model_dir /kaggle/working/inference/finetuned_en_PP-OCRv4_rec --image_dir /kaggle/input/recognition-data/test --rec_char_dict_path /kaggle/working/finetune_paddle_ocr/PaddleOCR/ppocr/utils/en_dict.txt --rec_image_shape='3,80,800'

And I got the outputs for the images as expected. It wasn't the best. But it was satisfactory.

However, when I tried the below code to directly compare the original model and my new one directly:

fine_tuned_model = PaddleOCR(rec_model_dir='/kaggle/working/inference/finetuned_en_PP-OCRv4_rec', use_gpu=True)

original_model = PaddleOCR(rec_model_dir='/kaggle/working/pretrained_models/recognition/en_PP-OCRv4_rec_train', use_gpu=True)

image_dir = '/kaggle/input/recognition-data/test'

results = []

for image_name in os.listdir(image_dir):
    image_path = os.path.join(image_dir, image_name)
    if os.path.isfile(image_path):
        try:
            fine_tuned_result = fine_tuned_model.ocr(image_path, cls=False)
            fine_tuned_text = fine_tuned_result[0][0][1][0] if fine_tuned_result and fine_tuned_result[0] else "N/A"

            original_result = original_model.ocr(image_path, cls=False)
            original_text = original_result[0][0][1][0] if original_result and original_result[0] else "N/A"

            results.append({
                'image': image_name,
                'fine_tuned': fine_tuned_text,
                'original': original_text
            })
        except Exception as e:
            print(f"Error processing {image_name}: {e}")

# Print the results
for result in results:
    print(f"Image: {result['image']}")
    print(f"Fine-tuned Model Prediction: {result['fine_tuned']}")
    print(f"Original Model Prediction: {result['original']}")
    print()

My finetuned model started predicting in Chinese

Image: augmented_20_prepro_image1_B_crop_0.jpg
Fine-tuned Model Prediction: N/A
Original Model Prediction: N/A

Image: augmented_10_prepro_image1_F_crop_0.jpg
Fine-tuned Model Prediction: 贿'娇先疗题先与娇鲶鲶疗消房
Original Model Prediction: 704-16-C410E

Image: augmented_11_prepro_image1_C_crop_0.jpg
Fine-tuned Model Prediction: 廖贿'娇先疗题先与娇鲶鲶疗消房
Original Model Prediction: 704-16410E

Image: augmented_4_prepro_image1_F_crop_0.jpg
Fine-tuned Model Prediction: 贿'娇先疗题先与娇鲶鲶疗消房
Original Model Prediction: 70416-C410E

I checked inference.yml and the character_dict: has only Eng characters and symbols.

Questions:

What possibly can be the issue with the model such that it outputs Chinese chars when running as such?
Any suggestions to change my yml file configs to further improve the training results? How would do go about using VisualDL to view real-time outputs from Kaggle Notebook?

Hey @VishyAnand28 any input is greatly appreciated. Do you have any experience working with VisualDL as well?

GreatV · 2024-12-18T02:47:45Z

GreatV
Dec 18, 2024
Maintainer

Answer

1. Why is the fine-tuned model predicting Chinese characters?

The issue of your fine-tuned model predicting Chinese characters instead of English is most likely related to a mismatch in the character dictionary (character_dict_path) or an unexpected initialization during inference. Let's break it down:

Character Dictionary Issue:
Although you mentioned using an English character dictionary (en_dict.txt), it's possible that the model during fine-tuning or inference inadvertently used a different dictionary (e.g., a Chinese dictionary). If the character set loaded during training or inference doesn't match your dataset, the model may output characters from an unintended language.
- Fine-tuning Initialization: If the pre-trained model (/kaggle/working/pretrained_models/recognition/en_PP-OCRv4_rec_train) was originally trained on a dataset with Chinese characters, but the character dictionary during fine-tuning (en_dict.txt) was not properly aligned with the model architecture or training process, this could cause confusion for the model in inference mode.
- Inference Dictionary Mismatch: When exporting the model for inference, the character dictionary specified in the inference.yml file must match the one used during training. Even if en_dict.txt was specified, there might be a fallback to a default PaddleOCR dictionary (e.g., ppocr_keys_v1.txt, which includes Chinese characters).
Mixed Training Configuration (MultiHead):
Your architecture uses a MultiHead design (CTCHead and NRTRHead). If one of the heads has a character dictionary or configuration that supports Chinese characters, it could influence the output. Specifically:
- CTCHead uses CTCLabelDecode, which might default to an incorrect character dictionary.
- NRTRHead uses NRTRLabelEncode, which should also align with the English dictionary, but any misconfiguration could lead to unexpected behavior.
Pre-trained Model Residual Bias:
The pre-trained model (en_PP-OCRv4_rec_train) may have residual bias from training on multilingual datasets. If the fine-tuning dataset is too small (1300 images), the model might not fully adapt to the new English-only domain, causing it to revert to its pre-trained behavior, such as predicting Chinese characters.

Steps to Fix the Issue:

Verify Character Dictionary Consistency:
- Ensure that en_dict.txt is correctly specified in all stages:
  - During fine-tuning (character_dict_path in the yml file).
  - During export (Global.character_dict_path in export_model.py).
  - During inference (--rec_char_dict_path argument in predict_rec.py).
Check Pre-trained Model Initialization:
- Confirm that the pre-trained model is compatible with the English-only dictionary. If the pre-trained model was trained on multilingual data, consider updating the dictionary and fine-tuning with a larger English dataset.

Force Dictionary Alignment During Inference:

Explicitly specify the character dictionary path when loading the PaddleOCR instance:

fine_tuned_model = PaddleOCR(rec_model_dir='/kaggle/working/inference/finetuned_en_PP-OCRv4_rec', 
                             rec_char_dict_path='/kaggle/working/finetune_paddle_ocr/PaddleOCR/ppocr/utils/en_dict.txt',
                             use_gpu=True)

Inspect the Exported Model:
- Verify the exported inference model files (inference.pdmodel, inference.pdiparams) to ensure they were generated using the fine-tuned weights and the English dictionary.
Validate MultiHead Configuration:
- Ensure both CTCHead and NRTRHead are configured to use the same character dictionary and are aligned with the English dataset.

2. Suggestions to Improve Training Results:

Expand the Dataset:
- Fine-tuning on only 1300 images may not be sufficient for the model to generalize well. Consider augmenting the dataset with more synthetic or real-world English text images.
Adjust Learning Rate and Warm-Up:
- The current learning rate schedule uses CosineDecay with a warm-up epoch of 3. While this is generally effective, you can experiment with a smaller initial learning rate (e.g., 0.0001) for fine-tuning, as the model is already pre-trained.
Freeze Early Layers:
- Freeze the backbone (PPLCNetV3) layers during the initial epochs of fine-tuning to focus training on the MultiHead and character-level adaptations.
Increase max_text_length:
- If your dataset contains longer text sequences, consider increasing max_text_length in both the yml file and the NRTRHead configuration (e.g., from 25 to 50).
Use Label Smoothing:
- Introduce label smoothing in the loss functions (CTCLoss, NRTRLoss) to make training more robust.
Regularization:
- Reduce overfitting with dropout layers in the MultiHead architecture or increase the L2 regularization factor (Optimizer -> regularizer -> factor).
VisualDL for Real-Time Monitoring:
- To enable VisualDL in a Kaggle notebook:
  1. Install visualdl:
```
!pip install visualdl
```
  2. Enable use_visualdl in the yml file:
```
Global:
  use_visualdl: true
  save_model_dir: ./output/rec
```
  3. Start a VisualDL server in the notebook:
```
!visualdl --logdir ./output/rec --port 8000
```
  4. Use Kaggle's built-in "Notebook Settings" to create a custom URL (e.g., localhost:8000) and monitor metrics like accuracy, loss, and learning rate in real-time.

3. Related Discussions:

Pre-trained Model Performance and Dictionary Mismatch:
This discussion highlights potential issues with pre-trained weights and dictionary alignment, particularly when the initial accuracy is unexpectedly low. It mirrors your situation where the pre-trained model may not align with the target task or dictionary.

Response generated by feifei-bot | chatgpt-4o-latest

0 replies

VishyAnand28 · 2024-12-18T15:29:26Z

VishyAnand28
Dec 18, 2024

@l03126 please share your config file as it is. I think you have modified a lot. Have you removed the GTC strategy and many other stuff?
I want to first look at your config file, not the description you shared, which we get from the training log. Also attach the training log as text document pls.

5 replies

l03126 Dec 19, 2024
Author

Hey @VishyAnand28 tqsm for the reply. But I haven't modified/ removed any info. There's 150 lines in en_PP-OCRv4_rec.yml and theres 150 in mine as well.

Sorry I dont have the log file. Since I'm training on Kaggle I can't store the outputs in the environment for long. Will make sure to link the log next time. But this is the cli outputs pasted into a text file.

Trainv1.txt

VishyAnand28 Dec 19, 2024

Please share your actual yml file @l03126 . As far as I can tell, most of the things that you are doing is fine. I think you are having trouble with export_model.py
when you use two different dictionaries to export, these issues can be expected. Also, try to reduce learning rate to various low values like 5e-4 to 5e-5. Then try changing the loss function from cosine to piecewise. Experiment with these and resend your results here with actual yml file. You can paste log file as plain text that is fine but without seeing current yml file it is difficult to comment.

l03126 Dec 20, 2024
Author

Hey @VishyAnand28 I really don't understand what you mean by 'actual' yml file. Because the lines pasted in my post above are of the original yml file, absolutely nothing changed. I'll attach it here as well. Did you mean there were some lines that are totally missing?

rec_configs.txt

Regarding your suggestions with Learning Rate and Loss function, I'll definitely experiment with them. Thanks

VishyAnand28 Dec 20, 2024

Okay try with different configs and let's keep this thread open. I will comment on your further results.

l03126 Dec 20, 2024
Author

Alrightt. Suree

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issue and suggestions regarding finetuning en_PPOCRv4_rec_train #14414

{{title}}

Replies: 2 comments 5 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Issue and suggestions regarding finetuning en_PPOCRv4_rec_train #14414

l03126 Dec 18, 2024

Replies: 2 comments · 5 replies

GreatV Dec 18, 2024 Maintainer

Answer

1. Why is the fine-tuned model predicting Chinese characters?

Steps to Fix the Issue:

2. Suggestions to Improve Training Results:

3. Related Discussions:

VishyAnand28 Dec 18, 2024

l03126 Dec 19, 2024 Author

VishyAnand28 Dec 19, 2024

l03126 Dec 20, 2024 Author

VishyAnand28 Dec 20, 2024

l03126 Dec 20, 2024 Author

l03126
Dec 18, 2024

Replies: 2 comments 5 replies

GreatV
Dec 18, 2024
Maintainer

VishyAnand28
Dec 18, 2024

l03126 Dec 19, 2024
Author

l03126 Dec 20, 2024
Author

l03126 Dec 20, 2024
Author