Support for long vectors

[joshuaulrich]

R-3.0.0 added support for vectors longer than the maximum positive value for a signed 32-bit integer (2^31-1 ~ 2.147e9). It would be nice to support these in internal C code, but it is not particularly urgent.

The smallest xts object that would need this functionality would have 2^31 rows. That object would occupy 16GB of RAM if the index and the data were integers, and 32GB of RAM if they were both doubles.

I doubt many (if any) users have xts objects that large at the moment, but would be nice to address this.

[joshuaulrich]

Well, I finally encountered this... time to fix. :-D

R$ n <- 1.1e7
   x <- replicate(200, xts::.xts(double(n), seq_len(n)), simplify = FALSE)
   y <- do.call(merge, x)
## Error in merge.xts(c(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,  : 
##   negative length vectors are not allowed

This is because 200 * 1.1e7 = 2.2e9 is larger than .Machine$integer.max (2,147,483,647). So multiplying 200 * 1.1e7 in C overflows the 32-bit integer and returns a random value (which can be negative).

The lack of long vector support also crashes na.locf.xts

x <- xts::.xts(matrix(0.0, nrow = 1.1e7, ncol = 200), seq_len(1.1e7))

# then:
#    zoo::na.locf(x)  # segfaults
##  *** caught segfault ***
## address 0x7f5deb095848, cause 'memory not mapped'
## 
## Traceback:
##  1: na.locf.xts(x)
##  2: zoo::na.locf(x)

This is likely a problem in multiple places in the C code.

[joshuaulrich]

More test cases:

n <- 1.1e9
x <- xts::.xts(matrix(seq_len(n*2), nrow = n, ncol = 2), seq_len(n))

y <- merge(x, x[,2])
y <- zoo::coredata(x)
y <- lag(x)
y <- diff(x)  # due to lag.xts()
y <- x[ seq_len(n*2),]
y <- x[-seq_len(n*2),]
xts::isOrdered(seq_len(n*2))
xts:::naCheck(seq_len(n*2))
y <- rbind(x, x)     ####  negative length vectors are not allowed
y <- xts::make.index.unique(x)

# zoo::na.locf(x)   ### segfaults
# na.omit(x)        ### killed