Parameter Efficient Fine Tuning (PEFT). PEFT is actually a bag of tricks for optimising machine learning model training.
An example of using some PEFT techniques to fine-tune a 40B parameter model on Colab can be found here
FSDP
A tool for sharding model weights and gradients out across multiple devices so that you can train very large models.