# P-tuning

[P-tuning](https://hf.co/papers/2103.10385) adds trainable prompt embeddings to the input that is optimized by a prompt encoder to find a better prompt, eliminating the need to manually design prompts. The prompt tokens can be added anywhere in the input sequence, and p-tuning also introduces anchor tokens for improving performance.

The abstract from the paper is:

*While GPTs with traditional fine-tuning fail to achieve strong results on natural language understanding (NLU), we show that GPTs can be better than or comparable to similar-sized BERTs on NLU tasks with a novel method P-tuning -- which employs trainable continuous prompt embeddings. On the knowledge probing (LAMA) benchmark, the best GPT recovers 64\% (P@1) of world knowledge without any additional text provided during test time, which substantially improves the previous best by 20+ percentage points. On the SuperGlue benchmark, GPTs achieve comparable and sometimes better performance to similar-sized BERTs in supervised learning. Importantly, we find that P-tuning also improves BERTs' performance in both few-shot and supervised settings while largely reducing the need for prompt engineering. Consequently, P-tuning outperforms the state-of-the-art approaches on the few-shot SuperGlue benchmark.*.

## PromptEncoderConfig[[peft.PromptEncoderConfig]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class peft.PromptEncoderConfig</name><anchor>peft.PromptEncoderConfig</anchor><source>https://github.com/huggingface/peft/blob/v0.18.0.rc0/src/peft/tuners/p_tuning/config.py#L29</source><parameters>[{"name": "task_type", "val": ": Optional[Union[str, TaskType]] = None"}, {"name": "peft_type", "val": ": Optional[Union[str, PeftType]] = None"}, {"name": "auto_mapping", "val": ": Optional[dict] = None"}, {"name": "peft_version", "val": ": Optional[str] = None"}, {"name": "base_model_name_or_path", "val": ": Optional[str] = None"}, {"name": "revision", "val": ": Optional[str] = None"}, {"name": "inference_mode", "val": ": bool = False"}, {"name": "num_virtual_tokens", "val": ": int = None"}, {"name": "token_dim", "val": ": int = None"}, {"name": "num_transformer_submodules", "val": ": Optional[int] = None"}, {"name": "num_attention_heads", "val": ": Optional[int] = None"}, {"name": "num_layers", "val": ": Optional[int] = None"}, {"name": "modules_to_save", "val": ": Optional[list[str]] = None"}, {"name": "encoder_reparameterization_type", "val": ": typing.Union[str, peft.tuners.p_tuning.config.PromptEncoderReparameterizationType] = <PromptEncoderReparameterizationType.MLP: 'MLP'>"}, {"name": "encoder_hidden_size", "val": ": int = None"}, {"name": "encoder_num_layers", "val": ": int = 2"}, {"name": "encoder_dropout", "val": ": float = 0.0"}]</parameters><paramsdesc>- **encoder_reparameterization_type** (Union[`PromptEncoderReparameterizationType`, `str`]) --
  The type of reparameterization to use.
- **encoder_hidden_size** (`int`) -- The hidden size of the prompt encoder.
- **encoder_num_layers** (`int`) -- The number of layers of the prompt encoder.
- **encoder_dropout** (`float`) -- The dropout probability of the prompt encoder.</paramsdesc><paramgroups>0</paramgroups></docstring>

This is the configuration class to store the configuration of a [PromptEncoder](/docs/peft/v0.18.0.rc0/en/package_reference/p_tuning#peft.PromptEncoder).




</div>

## PromptEncoder[[peft.PromptEncoder]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class peft.PromptEncoder</name><anchor>peft.PromptEncoder</anchor><source>https://github.com/huggingface/peft/blob/v0.18.0.rc0/src/peft/tuners/p_tuning/model.py#L24</source><parameters>[{"name": "config", "val": ""}]</parameters><paramsdesc>- **config** ([PromptEncoderConfig](/docs/peft/v0.18.0.rc0/en/package_reference/p_tuning#peft.PromptEncoderConfig)) -- The configuration of the prompt encoder.</paramsdesc><paramgroups>0</paramgroups></docstring>

The prompt encoder network that is used to generate the virtual token embeddings for p-tuning.



<ExampleCodeBlock anchor="peft.PromptEncoder.example">

Example:

```py
>>> from peft import PromptEncoder, PromptEncoderConfig

>>> config = PromptEncoderConfig(
...     peft_type="P_TUNING",
...     task_type="SEQ_2_SEQ_LM",
...     num_virtual_tokens=20,
...     token_dim=768,
...     num_transformer_submodules=1,
...     num_attention_heads=12,
...     num_layers=12,
...     encoder_reparameterization_type="MLP",
...     encoder_hidden_size=768,
... )

>>> prompt_encoder = PromptEncoder(config)
```

</ExampleCodeBlock>

**Attributes**:
- **embedding** (`torch.nn.Embedding`) -- The embedding layer of the prompt encoder.
- **mlp_head** (`torch.nn.Sequential`) -- The MLP head of the prompt encoder if `inference_mode=False`.
- **lstm_head** (`torch.nn.LSTM`) -- The LSTM head of the prompt encoder if `inference_mode=False` and
`encoder_reparameterization_type="LSTM"`.
- **token_dim** (`int`) -- The hidden embedding dimension of the base transformer model.
- **input_size** (`int`) -- The input size of the prompt encoder.
- **output_size** (`int`) -- The output size of the prompt encoder.
- **hidden_size** (`int`) -- The hidden size of the prompt encoder.
- **total_virtual_tokens** (`int`): The total number of virtual tokens of the
prompt encoder.
- **encoder_type** (Union[`PromptEncoderReparameterizationType`, `str`]): The encoder type of the prompt
  encoder.


Input shape: (`batch_size`, `total_virtual_tokens`)

Output shape: (`batch_size`, `total_virtual_tokens`, `token_dim`)


</div>

<EditOnGithub source="https://github.com/huggingface/peft/blob/main/docs/source/package_reference/p_tuning.md" />