Repository navigation
Making the CompressedList more widely usable #11
Description
I've been playing around with the CompressedList subclasses for representing some complex data types and I've really come to like it. I've been thinking of ways to make it more generally usable by both end-users and other developers, and I've got a few wish-list elements:
Access to unlistData and partitioning. End-users would then be able to execute arbitrary unary operations on the underlying data while preserving the partitioning, like:
A <- DataFrame(X=LETTERS, Y=runif(26))
comp.list <- split(A,A$X)
# Attempt fails, for obvious reasons.
comp.list$Y <- log(comp.list$Y)
# Assuming we had a unlistData() method:
unlistData(comp.list)$Y <- log(unlistData(comp.list)$Y)unlistData<- could even be unlist<-, if one were willing to introduce that concept. I don't mind if partitioning is getter-only; this would still be very useful for downstream functions that need to be list-aware yet don't want to create an intermediate list for efficiency purposes.
Non-virtual CompressedList class. I don't understand the motivation for making CompressedList virtual. From a representation perspective, a general concrete class would be useful if we could store any vector-like entity in unlistData. In fact, I ran into the case where I wanted to store a CompressedCharacterList as unlistData, effectively making a CompressedCompressedCharacterListList! I don't expect to be able to call many methods on this thing - other than the proposed unlistData and partitioning, and maybe unlist - I just want to use it for storage without needing to write an explicit subclass. A general CompressedList class would serve this purpose, and is better than the alternative of falling back to a SimpleList (which takes a noticeable time to generate).
A more careful unlist. If we do allow a general CompressedList class, the unlist method should probably take heed of recursive=TRUE and apply unlist on the unlistData slot.
I'm happy to chip in with a PR if these sound like good ideas.
Activity
lawremi commentedon Apr 6, 2019
An
unlist<-()has been suggested in the past (e.g., by @mtmorgan). It's a probably worth having but up until now we have managed by just adding methods for functions likelog()and usingrelist()directly. We would welcome a pull request forunlist<-(), but it would also be nice to havelog()and related methods for NumericList.Making CompressedList non-virtual (and requiring Vector for
@unlistData) is an interesting idea. A separate pull request is welcome, if only to spur discussion. Agree that it should considerrecursive=.hpages commentedon Apr 6, 2019
Note that
relist()already does what the proposedunlist<-would do on a CompressedList. So IIUC basically theunlist<-proposal would be to replace well-established idiom:with
Personally I prefer to stick to
relist()for several reasons:relist(as.character(unlist(x)), x)idiom is more readable (but that's just my opinion).unlist<-to all list-like objects, but that's whatrelist()does already).lawremi commentedon Apr 6, 2019
I do like the symmetry, simplicity and safety of the
unlist<-()syntax.relist()is in base, but it's fairly obscure. If we move forward, we should definitely makeunlist<-()work on all types of lists.LTLA commentedon Apr 6, 2019
Indeed, I didn't even know about
relistuntil a few days ago when I was poking around inside IRanges.My motivation for
unlist<-is mostly driven by use withCompressedSplitDataFrameLists, where it provides a simple mechanism for switching betweenList-mode andDataFrame-mode.The
relistsyntax would require an explicit intermediateDataFrame. I guess you could argue that this is clearer, but it's inconvenient to have to split it across three lines (especially in interactive sessions).A recursive
unlist<-also provides an approach for reaching deep into nestedCompressedListobjects, if it were possible to store aCompressedListas theunlistDataof anotherCompressedList: