[Integration][Format] Inconsistent skips in datagen

[pitrou]

Describe the bug, including details regarding any error messages, version, and platform.

I'm looking at the current datagen.py and I see some type-specific skips that seem inconsistent with the current implementation status.

Specifically:

• arrow/dev/archery/archery/integration/datagen.py

  Lines 1933 to 1935 in 4eca507

  	generate_decimal256_case()
  	.skip_tester('JS'),

  
  (decimal256 is supported in JS?)

• arrow/dev/archery/archery/integration/datagen.py

  Lines 1936 to 1950 in 4eca507

  	generate_decimal32_case()
  	.skip_tester('Java')
  	.skip_tester('JS')
  	.skip_tester('nanoarrow')
  	.skip_tester('Ruby')
  	.skip_tester('Rust')
  	.skip_tester('Go'),
  	generate_decimal64_case()
  	.skip_tester('Java')
  	.skip_tester('JS')
  	.skip_tester('nanoarrow')
  	.skip_tester('Ruby')
  	.skip_tester('Rust')
  	.skip_tester('Go'),

  
  (decimal32/64 are supported in Go, Rust and nanoarrovw?)

• arrow/dev/archery/archery/integration/datagen.py

  Lines 2001 to 2007 in 4eca507

  	generate_run_end_encoded_case()
  	.skip_tester('.NET')
  	.skip_tester('JS')
  	# TODO(https://github.com/apache/arrow-nanoarrow/issues/618)
  	.skip_tester('nanoarrow')
  	.skip_tester('Ruby')
  	.skip_tester('Rust'),

  
  (REE is supported in nanoarrow?)

• Support dictionary-encoded types in the IPC reader and writer arrow-nanoarrow#622
  (binary view is supported in Rust and nanoarrow?)

• arrow/dev/archery/archery/integration/datagen.py

  Lines 2016 to 2022 in 4eca507

  	generate_list_view_case()
  	.skip_tester('.NET') # Doesn't support large list views
  	.skip_tester('JS')
  	# TODO(https://github.com/apache/arrow-nanoarrow/issues/618)
  	.skip_tester('nanoarrow')
  	.skip_tester('Ruby')
  	.skip_tester('Rust'),

  
  (list view in supported in Rust and nanoarrow?)

• arrow/dev/archery/archery/integration/datagen.py

  Lines 2024 to 2028 in 4eca507

  	generate_extension_case()
  	.skip_tester('nanoarrow')
  	# TODO(https://github.com/apache/arrow/issues/38045)
  	.skip_format(SKIP_FLIGHT, '.NET')
  	.skip_tester('Ruby'),

  
  (extension is supported in nanoarrow?)

Component(s)

Format, Integration

[pitrou]

@paleolimbot @alamb @tustvold @zeroshade @trxcllnt @kou @rustyconover Could you take a look at this?

[paleolimbot]

Thanks for checking!

nanoarrow supports all of these types but not in IPC yet (I believe we support decimal 32/64 in IPC currently). We support extension types but there's an example in that file with dictionary storage (which will be supported soon but isn't yet).

[pitrou]

And conversely, it seems https://arrow.apache.org/docs/dev/status.html is outdated for Javascript (which seems to support Binary View, Utf8 View, Decimal32 and Decimal64).

[alamb]

Yes, Rust has basic support for Decmal16/32 and ListView and REE

[rustyconover]

Hi All,

I've been thinking we need a "caniuse.com" site for data types and the implementations, but I haven't had my robots make it yet.

On the compatibility table we should also include https://github.com/uwdata/flechette

Rusty

[pitrou]

Yes, Rust has basic support for Decmal16/32 and ListView and REE

Could you submit a PR to update that in the doc @alamb ?

[pitrou]

Also it seems https://arrow.apache.org/docs/dev/status.html is outdated for Swift, which AFAICT supports Timestamp? @kou

[paleolimbot]

I'm happy to submit a PR for this and I can include the Rust updates (the dictionary PR merged in nanoarrow and I was about to remove the skips here to check anyway!)

[kou]

I've opened sub-issues for JavaScript and Swift.

[alamb]

Yes, Rust has basic support for Decmal16/32 and ListView and REE

Could you submit a PR to update that in the doc @alamb ?

I double checked the status page https://arrow.apache.org/docs/status.html and it appears to be up to date for Rust: