# BOFT

[Orthogonal Butterfly (BOFT)](https://hf.co/papers/2311.06243) is a generic method designed for finetuning foundation models. It improves the parameter efficiency of the finetuning paradigm -- Orthogonal Finetuning (OFT), by taking inspiration from Cooley-Tukey fast Fourier transform, showing favorable results across finetuning different foundation models, including large vision transformers, large language models and text-to-image diffusion models.

The abstract from the paper is:

*Large foundation models are becoming ubiquitous, but training them from scratch is prohibitively expensive. Thus, efficiently adapting these powerful models to downstream tasks is increasingly important. In this paper, we study a principled finetuning paradigm -- Orthogonal Finetuning (OFT) -- for downstream task adaptation. Despite demonstrating good generalizability, OFT still uses a fairly large number of trainable parameters due to the high dimensionality of orthogonal matrices. To address this, we start by examining OFT from an information transmission perspective, and then identify a few key desiderata that enable better parameter-efficiency. Inspired by how the Cooley-Tukey fast Fourier transform algorithm enables efficient information transmission, we propose an efficient orthogonal parameterization using butterfly structures. We apply this parameterization to OFT, creating a novel parameter-efficient finetuning method, called Orthogonal Butterfly (BOFT). By subsuming OFT as a special case, BOFT introduces a generalized orthogonal finetuning framework. Finally, we conduct an extensive empirical study of adapting large vision transformers, large language models, and text-to-image diffusion models to various downstream tasks in vision and language*.

## BOFTConfig[[peft.BOFTConfig]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class peft.BOFTConfig</name><anchor>peft.BOFTConfig</anchor><source>https://github.com/huggingface/peft/blob/v0.18.0.rc0/src/peft/tuners/boft/config.py#L28</source><parameters>[{"name": "task_type", "val": ": Optional[Union[str, TaskType]] = None"}, {"name": "peft_type", "val": ": Optional[Union[str, PeftType]] = None"}, {"name": "auto_mapping", "val": ": Optional[dict] = None"}, {"name": "peft_version", "val": ": Optional[str] = None"}, {"name": "base_model_name_or_path", "val": ": Optional[str] = None"}, {"name": "revision", "val": ": Optional[str] = None"}, {"name": "inference_mode", "val": ": bool = False"}, {"name": "boft_block_size", "val": ": int = 4"}, {"name": "boft_block_num", "val": ": int = 0"}, {"name": "boft_n_butterfly_factor", "val": ": int = 1"}, {"name": "target_modules", "val": ": Optional[Union[list[str], str]] = None"}, {"name": "exclude_modules", "val": ": Optional[Union[list[str], str]] = None"}, {"name": "boft_dropout", "val": ": float = 0.0"}, {"name": "fan_in_fan_out", "val": ": bool = False"}, {"name": "bias", "val": ": str = 'none'"}, {"name": "modules_to_save", "val": ": Optional[list[str]] = None"}, {"name": "init_weights", "val": ": bool = True"}, {"name": "layers_to_transform", "val": ": Optional[Union[list[int], int]] = None"}, {"name": "layers_pattern", "val": ": Optional[Union[list[str], str]] = None"}]</parameters><paramsdesc>- **boft_block_size** (`int`) -- BOFT block size across different layers.
- **boft_block_num** (`int`) -- Number of BOFT blocks per injected layer.
- **boft_n_butterfly_factor** (`int`) -- Number of butterfly factors across different layers.
- **target_modules** (`Union[List[str],str]`) -- The names of the modules to apply the adapter to.
- **exclude_modules** (`Optional[Union[List[str], str]]`) --
  The names of the modules to not apply the adapter. When passing a string, a regex match will be performed.
  When passing a list of strings, either an exact match will be performed or it is checked if the name of the
  module ends with any of the passed strings.
- **boft_dropout** (`float`) --
  The multiplicative dropout probability, by setting OFT blocks to identity during training, similar to the
  dropout layer in LoRA.
- **fan_in_fan_out** (`bool`) -- Set this to True if the layer to replace stores weight like (fan_in, fan_out).
  For example, gpt-2 uses `Conv1D` which stores weights like (fan_in, fan_out) and hence this should be set
  to `True`.
- **bias** (`str`) -- Bias type for BOFT. Can be 'none', 'all' or 'boft_only'. If 'all' or 'boft_only', the
  corresponding biases will be updated during training. Be aware that this means that, even when disabling
  the adapters, the model will not produce the same output as the base model would have without adaptation.
- **modules_to_save** (`List[str]`) --List of modules apart from BOFT layers to be set as trainable
  and saved in the final checkpoint.
- **layers_to_transform** (`Union[List[int],int]`) --
  The layer indexes to transform, if this argument is specified, it will apply the BOFT transformations on
  the layer indexes that are specified in this list. If a single integer is passed, it will apply the BOFT
  transformations on the layer at this index.
- **layers_pattern** (`Optional[Union[List[str], str]]`) --
  The layer pattern name, used only if `layers_to_transform` is different from `None` and if the layer
  pattern is not in the common layers pattern. This should target the `nn.ModuleList` of the model, which is
  often called `'layers'` or `'h'`.</paramsdesc><paramgroups>0</paramgroups></docstring>

This is the configuration class to store the configuration of a [BOFTModel](/docs/peft/v0.18.0.rc0/en/package_reference/boft#peft.BOFTModel).




</div>

## BOFTModel[[peft.BOFTModel]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class peft.BOFTModel</name><anchor>peft.BOFTModel</anchor><source>https://github.com/huggingface/peft/blob/v0.18.0.rc0/src/peft/tuners/boft/model.py#L31</source><parameters>[{"name": "model", "val": ""}, {"name": "peft_config", "val": ": Union[PeftConfig, dict[str, PeftConfig]]"}, {"name": "adapter_name", "val": ": str"}, {"name": "low_cpu_mem_usage", "val": ": bool = False"}, {"name": "state_dict", "val": ": Optional[dict[str, torch.Tensor]] = None"}]</parameters><paramsdesc>- **model** ([*transformers.PreTrainedModel*]) -- The model to be adapted.
- **config** ([*BOFTConfig*]) -- The configuration of the BOFT model.
- **adapter_name** (*str*) -- The name of the adapter, defaults to *"default"*.
- **low_cpu_mem_usage** (*bool*, *optional*, defaults to *False*) --
  Create empty adapter weights on meta device. Useful to speed up the loading process.</paramsdesc><paramgroups>0</paramgroups><rettype>*torch.nn.Module*</rettype><retdesc>The BOFT model.</retdesc></docstring>

Creates BOFT and OFT model from a pretrained transformers model. Paper: https://huggingface.co/papers/2311.06243
https://huggingface.co/papers/2306.07280







<ExampleCodeBlock anchor="peft.BOFTModel.example">

Example:

```python
>>> import transformers >>> from transformers import AutoModelForSeq2SeqLM, BOFTConfig >>> from peft import
BOFTConfig, get_peft_model

>>> config = BOFTConfig( ... boft_block_size=8, ... boft_n_butterfly_factor=1, ... target_modules=["query",
"value", "key", "output.dense", "mlp.fc1", "mlp.fc2"], ... boft_dropout=0.1, ... bias="boft_only", ...
modules_to_save=["classifier"], ... )

>>> model = transformers.Dinov2ForImageClassification.from_pretrained( ... "facebook/dinov2-large", ...
num_labels=100, ... ) >>> boft_model = get_peft_model(model, config)
```

</ExampleCodeBlock>

    **Attributes**:
- **model** ([*transformers.PreTrainedModel*]) -- The model to be adapted.
- **peft_config** ([*BOFTConfig*]): The configuration of the BOFT model.


</div>

<EditOnGithub source="https://github.com/huggingface/peft/blob/main/docs/source/package_reference/boft.md" />