How to handle the cache system properly?

[John6666]

Some resources for now:

Cache management

Best way to access the cached transformation arrow file

I was experimenting with the cached arrow dataset but ran into an error The steps followed Loaded the dataset from HF using load_dataset() I created a simple transformation by shuffling the data dataset = dataset["test"].shuffle(seed=42) This step produced a new cache file in the hugging face cache directory - ~/.cache/huggingface/datasets/bt-tech___ofac/dataset/1.0.0/aa542ed919df55cc5d3347f42dd4521d05ca68751f50dbc32bae2a7f1e167559/cache-c7f4daa392c5ed6a.arrow After I restart the sessio…

Load dataset from a specific cache file

Dear community, Does anyone know if there is a way to load dataset from a specific cached file? I applied very costly transformations using a map function to do some data augmentation and finally, I had my dataset. However, now in my main script, I would like to use the cached file built from another script. Is there any way to call a specific cache file from ~/.cache/huggingface/datasets/ ?