Could not use SFT Trainer in qlora_finetuning.py

[shungyantham]

I have installed trl<0.12.0 to run qlora_finetune.py in the QLoRA/trl-example but it requires transformers 4.46.2 which causes the error below.

So I downgraded trl from 0.11.4 to 0.9.6 and I got another padding error.

[qiyuangong]

I have installed trl<0.12.0 to run qlora_finetune.py in the QLoRA/trl-example but it requires transformers 4.46.2 which causes the error below.

So I downgraded trl from 0.11.4 to 0.9.6 and I got another padding error.

These errors is caused by transformers version mismatch. Can you downgrade transformers version to 4.36.0 ?

pip install transformers==4.36.0 datasets

[shungyantham]

Hi, I have also downgraded the transformers to 4.36.0 when I downgrade the trl to 0.9.6 and I got this error

[shungyantham]

https://github.com/intel-analytics/ipex-llm/blob/main/docker/llm/finetune/xpu/Dockerfile

I build this Dockerfile and then manually pip install trl==0.9.6 in the docker container. I ran the qlora_finetune.py in LLM_Finetuning/QLoRA/trl-example. Is there anything I missed?

[qiyuangong]

https://github.com/intel-analytics/ipex-llm/blob/main/docker/llm/finetune/xpu/Dockerfile

I build this Dockerfile and then manually pip install trl==0.9.6 in the docker container. I ran the qlora_finetune.py in LLM_Finetuning/QLoRA/trl-example. Is there anything I missed?

Hi @shungyantham , we have reproduced this issue in our local env.

Please modify qlora_finetune.py Line 91. Add data_collator=transformers.DataCollatorForSeq2Seq( tokenizer, pad_to_multiple_of=8, return_tensors="pt", padding=True ) to SFTTrainer.

Code should look like this:

    trainer = SFTTrainer(
        model=model,
        train_dataset=train_data,
        args=transformers.TrainingArguments(
            per_device_train_batch_size=4,
            gradient_accumulation_steps= 1,
            warmup_steps=20,
            max_steps=200,
            learning_rate=2e-5,
            save_steps=100,
            bf16=True,  # bf16 is more stable in training
            logging_steps=20,
            output_dir="outputs",
            optim="adamw_hf", # paged_adamw_8bit is not supported yet
            gradient_checkpointing=True, # can further reduce memory but slower
        ),
        dataset_text_field="instruction",
        data_collator=transformers.DataCollatorForSeq2Seq(
            tokenizer, pad_to_multiple_of=8, return_tensors="pt", padding=True
        )
    )

[qiyuangong]

#12368

[shungyantham]

Hi @qiyuangong , I have faced another issue after adding the padding to the Trainer

[qiyuangong]

Hi @qiyuangong , I have faced another issue after adding the padding to the Trainer

Please provide transformers and trl version, as well as finetune.py.

For reference, this is key lib versions in our test env. After merging that PR, it can finetune model without modification.

transformers                4.36.0
trl                         0.9.6

[shungyantham]

Hi @qiyuangong ,

I setup a docker container following this link:
https://github.com/intel-analytics/ipex-llm/tree/main/docker/llm/finetune/xpu

inside this container, I manually installed trl==0.9.6 and ran qlora_finetuning.py in /LLM-Finetuning/QLoRA/trl-example. My transformers version is 4.36.0.

[qiyuangong]

Hi @qiyuangong ,

I setup a docker container following this link: https://github.com/intel-analytics/ipex-llm/tree/main/docker/llm/finetune/xpu

inside this container, I manually installed trl==0.9.6 and ran qlora_finetuning.py in /LLM-Finetuning/QLoRA/trl-example. My transformers version is 4.36.0.

Essential lib versions are correct. Please share your launch command and qlora_finetuning.py.

[qiyuangong]

We can resolve this padding error by adding these lines into qlora_finetuning.py.

if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

I will submit a PR for this change.

[qiyuangong]

#12398