lora训练超参

lora训练超参

### model
model_name_or_path: pt_Qwen-Qwen2.5-7B-Instruct

### method
stage: sft
do_train: true
finetuning_type: lora
lora_target: all
lora_rank: 16
lora_alpha: 32
lora_dropout: 0.1
# additional_target: embed_tokens, lm_head
create_new_adapter: true

### dataset
dataset: v7.0_train_data
dataset_dir: data
template: qwen
cutoff_len: 4096
max_samples: 99999999
overwrite_cache: true
preprocessing_num_workers: 16

### output
output_dir: ./saves/Qwen2.5-7B-Instruct/lora/sft_2
logging_steps: 10
save_strategy: steps
save_steps: 500
plot_loss: true
overwrite_output_dir: true

### train
per_device_train_batch_size: 8
gradient_accumulation_steps: 1
learning_rate: 1.0e-4
num_train_epochs: 10
lr_scheduler_type: cosine
warmup_ratio: 0.1
#fp16: true
bf16: true
ddp_timeout: 180000000
seed: 42
optim: adamw_torch
gradient_checkpointing: true
# use_unsloth: true

### eval
do_eval: true
val_size: 0.02
per_device_eval_batch_size: 1
eval_strategy: steps
eval_steps: 500
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值