The data we upload to huggingface contains some features that have a flexible time axis. For example, for each sample it can be of shape (T, 768), T varies. According to the official doc, huggingface does support this type of 2D array by leaving the first dim as None, as I put here: features = data…

The Dataset library anticipates cases where the dimensions of data change dynamically, but the DatasetViewer library does not yet anticipate this. This is probably a ToDo rather than a bug. @lhoestq services/worker/src/worker/utils.py main dataset: str, …

Hi thanks for confirming that! Also want to know will this array2d of flexible first dim prevent huggingface from automatically generating croissant meta? I’ve made several attempts on this, and notice that if the dataset contains one such array2d then after a short delay even the viewer still show…

If the data can be converted to parquet format, I think metadata will probably be created eventually, but maybe there is an error in the converter…:thinking: [image] Generating Croissant Metadata for Custom Image Dataset 🤗Datasets The croissant metadata on HF a…

hi, if the shape is (768,) then it’s a not a Array2D (but a Array1D but it doesn’t exist, you should use Sequence instead)

When I upload dataset containing 3 such Array2D ( (None, 768) (None, 1024) (None, 1024)) by loading them locally into memory, formatting to a pandas.Dataframe, calling from_pandas() function to convert to dataset, then using push_to_hug to submit it, I do see the parquet files in /data folder, but…

No the actual shape is (T, 768) but T is not fixed. I used Sequence of Float on 1D array and it works fine, just this Array2D of shape (None, 768) causes trouble. My understanding is the shape=(768,) in the error log means huggingface processes Array2D of shape (None, 768) to a 1D array of shape 768…

@John6666 @lhoestq So my current fix is to pad all the (T, 768) to (max_T, 768), then specify that max_T in array2D. It roughly works, and huggingface can generate parquet branch & croissant meta, but the padding makes the files too big and croissant meta just skips those features and dataset view…

Dataset Viewer not available on features of type datasets.Array2D(shape=(None, 768), dtype='float64')

🤗Datasets

John6666 May 14, 2025, 2:26am 4

If the data can be converted to parquet format, I think metadata will probably be created eventually, but maybe there is an error in the converter…

Topic		Replies	Views
Streaming .arrow IterableDataset with irregular first dimension 🤗Datasets	2	38	February 14, 2025
Significant performance difference between two shapes using Array2D features 🤗Datasets	3	382	September 5, 2023
Create dataset consisting of numpy arrays, Sequence or ArrayND? 🤗Datasets	1	216	October 24, 2024
Dataset viewer issue Site Feedback	7	469	January 17, 2025
Standard getitem returns wrong data type for arrays 🤗Datasets	2	480	November 17, 2023

Dataset Viewer not available on features of type datasets.Array2D(shape=(None, 768), dtype='float64')

Related topics