axolotl

repository

axolotl is a fine tuning framework for fine-tuning LLM type models

Alpaca Instruction Format

Documented here - you provide a table with columns instruction, input and output. These are formatted into a prompt that is fed to the model like so:

Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
{instruction}

### Input:
{input}

### Response:

After this the generated response is compared with the output and a signal generated for backprop.

flash_attn issues

Packaging

on initial install python package packaging may be missing so pip install packaging before installing requirements.txt

Missing Symbols

• Encountered an issue where flash_attn couldn’t load due to missing symbols in the c library. Running pip install --upgrade flash_attn seems to fix this.

AttributeError: ‘torch.dtype’ object has no attribute ‘itemsize’

• Issue caused by incompatibility between transformers and torch solved by downgrading transformers to v4.38.2 as per this thread