How to retrieve length of a string/varchar column in metadata ?

[rferraton]

For String fields a try to get the length.

My current code does not return anything :

public static async Task<(string Name, Type ClrType,int Length, int Precision, int Scale)[]> GetParquetSchema(string parquetFilePath)
		{
			using (Stream fs = System.IO.File.OpenRead(parquetFilePath))
			{
				using (ParquetReader reader = await ParquetReader.CreateAsync(fs))
				{
					
					var schema = new (string Name, Type ClrType, int Length, int Precision, int Scale)[reader.Schema.GetDataFields().Length];
					int i = 0;					
					foreach (DataField df in reader.Schema.GetDataFields())
					{						
						
						// populate an array using Name, ClrType and Length, Precision and Scale of the Datafield
						schema[i]= (df.Name, df.ClrType, df.ThriftSchemaElement.Type_length, df.ThriftSchemaElement.Precision, df.ThriftSchemaElement.Scale);						
						i++;						
					}
					return schema;
				}
			}
		}

If the column/field length cannot be found in DataField.ThriftSchemaElement.Type_length, where can i found it ?

[aloneguid]

Strings are variable length, do you mean number of values?

[rferraton]

I would like to get the max size of a strings column (may be in a statistics info ?). Final objective is to load parquet data into SQL tables and create SQL table in not exists.

[aloneguid]

I don't think parquet provides max length info. Statistics give you max value which is different from max length. You'll have to enumerate all values I'm afraid.

[rferraton]

😭 so i will use a CONST maxlen to create the target table if not exist, load data and refactor the table after use the datalength function. But it's a bit sad that parquet don't store this kind of usefull information.

Thanks for your answer !

[aloneguid]

Correct. Unlike relational tables, strings in parquet are not limited by length. Hope you find a good solution.

[rferraton]

I switch to a const MAX_VARCHAR_LEN

public static async Task<(string Name, Type ClrType,int Length, int Precision, int Scale)[]> GetParquetSchema(string parquetFilePath)
		{
			using (Stream fs = System.IO.File.OpenRead(parquetFilePath))
			{
				using (ParquetReader reader = await ParquetReader.CreateAsync(fs))
				{					
					var schema = new (string Name, Type ClrType, int Length, int Precision, int Scale)[reader.Schema.GetDataFields().Length];
					int i = 0;					
					foreach (DataField df in reader.Schema.GetDataFields())
					{
						// populate an array using Name, ClrType , MAX_VARCHAR_LEN (before founding better alternative), Precision and Scale of the Datafield
						schema[i]= (df.Name, df.ClrType, MAX_VARCHAR_LEN , df.ThriftSchemaElement.Precision, df.ThriftSchemaElement.Scale);						
						i++;						
					}
					return schema;
				}
			}
		}