Longformer for Encoder Decoder with gradient checkpointing

LadyHangaku
Fri, 07 Jan 2022 10:41:04 +0000

I found a solution which at least helps a little:
When using EncoderDecoderModel(), it is possible to set gradient checkpointing at least on the encoder part:
model.encoder.config.gradient_checkpointing = True

Read full topic