lazy results with with-open

[lassemaatta]

I'm very much a beginner in monger (and mongo) so apologies if I'm asking something obvious: how does monger close cursor(s) when returning lazy results from queries? Or does it?

I originally found this issue (or rather question) when reviewing some code, but a similar pattern can be found right here in monger. For example, in monger.query/exec, we can see that the function wraps the cursor with with-open and yet it returns a lazy seq operating on said cursor. To my untrained eye this looks a lot like a typical "I accidentally closed a resource before evaluating the lazy sequence and got an Exception later" situation, yet that does not occur here.

As far as I can tell, what happens is that

1. map calls (seq cursor)
2. seq (in clojure.lang.RT) calls seqFrom which calls chunkIteratorSeq(((Iterable) coll).iterator())
3. .iterator() invokes/is com.mongodb.DBCursor.iterator() , which returns this.copy (link) ("Creates a copy of this cursor object that can be iterated")
4. chunkIteratorSeq returns a lazy seq, which pulls data from the iterator, which pulls data from the copied cursor
5. with-open closes the outer cursor
6. The original function returns, the consumer consumes the seq (or not) and the copied cursor is not closed

Also note that the docstring for .iterator() in DBCursor says "Note that use of this method does not let you call close the underlying cursor in the case of either an exception or an early break. The preferred method of iteration is to use DBCursor as an Iterator, so that you can call close() on it in a finally block."

Note that if one writes (map some-fn (iterator-seq cursor)) then we will skip the .iterator() call and iterate on the original cursor and end up IIRC with a "cursor has already been closed" error.

[michaelklishin]

It does not but eventually something triggers the evaluation of such lazy collections. Even basic tutorials usually print results out, integration tests perform assertions on collections, data is passed somewhere where evaluation will be forced.

It's a philosophical question whether libraries should force the evaluation. Given that a result set can contain many thousands of documents, it can have dire operational consequences.

[michaelklishin]

Consider using discussions for questions. They have been around for a good couple of years now.

[lassemaatta]

Consider using discussions for questions.

Good point, I always forget these exists.

It does not but eventually something triggers the evaluation of such lazy collections. Even basic tutorials usually print results out, integration tests perform assertions on collections, data is passed somewhere where evaluation will be forced.

Sure, usually something eventually at least partially evaluates the results. But my question was more about closing the cursor(s): should I be worried that mapping over the cursor creates a new cursor which is never closed?