Implement URL retrieval for readtable

[milktrader]

This feature has been stubbed out and currently throws the error "URL retrieval not yet implemented"

milktrader/Quandl.jl#8

milktrader/Quandl.jl#1

[milktrader]

Lines 562-564 of src/io.jl is where this feature is stubbed, btw.

    # (1) Path is an HTTP or FTP URL
    if ismatch(r"^(http://)|(ftp://)", pathname)
        error("URL retrieval not yet implemented")

[StefanKarpinski]

I would advocate for not going this way and instead having some kind of generic readurl function that returns an open stream that returns the contents of a URL. The stream interface is a nice "narrow waist" – various packages support handling arbitrary data streams, while base – or other packages – provide the ability to provide streams by various means, including reading from an internet URL.

[milktrader]

readurl(filename) = readlines( curl -s $filename ) works for that purpose, but I'd really like to lever the parsing code that very nicely places this stream into a data frame.

[StefanKarpinski]

Yes, the point is that the fundamental DataFrames interface should be reading data from a stream, rather than opening a file name. Of course, readtable(fname::String) = open(readtable,fname) is an obvious default behavior that immediately gets back the current interface of passing a file name.

[milktrader]

As my friend Max would say, I'm confuzzled about how to write this. So it comes to having a readurl function that takes a parsing function and an url?

[johnmyleswhite]

I believe the idea is that you will call readtable(readurl(URL)). Look into the existing I/O code to see the arguments for the version of readtable that accepts an IOStream as an input.

[prcastro]

Bump

[quinnj]

CSV.jl can read from any IO object, including the result of a HTTP.get(url) call, so this is supported

CSV.read(HTTP.body(HTTP.get(csv_url)))

[davidanthoff]

CSVFiles.jl also supports reading from a URL directly:

df = DataFrame(load("http://someurl.org/somefile.csv"))