Parquet data loader in R

[espinielli]

          @Fil yes that is what I will try:

1. Write to a temporary file
2. Dump to stdout

Originally posted by @espinielli in #873 (comment)

[espinielli]

I have been able to correctly get a Parquet data loader in R as follows (on OSs with cat):

library(arrow)

df <- data.frame(
  Name = c("Alice", "Bob", "Charlie", "Diana"),
  Age = c(25,  30,  35,  40)
)

# Write the data frame to a temporary Parquet file
temp_file <- tempfile(fileext = ".parquet")
arrow::write_parquet(df, sink = temp_file)

system2('/bin/cat', args = temp_file)

[espinielli]

A little bit too convoluted IMHO

[Fil]

this upstream issue might be relevant apache/arrow-rs#1687

[chaz23]

@espinielli I'm trying to get a Parquet data loader working correctly in R as well, and have been trying to replicate what you've written above. I'm on a Windows machine, so I don't think the line system2('/bin/cat', args = temp_file) works for me?

How do I write this data to stdout?

I keep getting the error Error: Empty input provided or not a Uint8Array. when I try to access the output of the data loader.

[chaz23]

Update - so I tried cat(temp_file) but it starts complaining about a corrupted footer...

[Fil]

To debug this, try and inspect the file that must have been saved in the cache at src/.observablehq/cache/data/file.parquet
maybe something is spuriously sending logs to stdout, which would then end up in the file and "corrupt" it.

You can also run the script independently of Framework like so:

Rscript src/data/file.parquet.R > result.parquet

To avoid using "cat", you might be able to write to stdout by doing:

arrow::write_parquet(df, sink = /dev/stdout)

or

arrow::write_parquet(df, sink = '/dev/stdout')

(I'm sorry I'm currently unable to test these recommendations because my local installation of R is broken and won't compile arrow. Hope this helps anyway.)

[chaz23]

Yes, I think using cat just sent the name of the file to stdout rather than the contents. And I'm on a windows machine, so I don't think "dev/stdout" works? I tried it anyway, and it didn't 😂 any other suggestions? I'm totally stumped...

[Fil]

maybe try stdout()?
https://stat.ethz.ch/R-manual/R-devel/library/base/html/showConnections.html

[espinielli]

@chaz23 could you give a try to the following snippet?

library(arrow)

df <- data.frame(
  Name = c("Alice", "Bob", "Charlie", "Diana"),
  Age = c(25,  30,  35,  40)
)

# Write the data frame to a temporary Parquet file
arrow::write_parquet(df, sink = file("stdout"))

[chaz23]

I just tried it - I get load src/data/test.parquet.r → [fresh] success empty output in 0ms. It doesn't seem to write anything...

[chaz23]

Oh I got it working! I used system2('cat', args = temp_file)!