• Mongoid::Scroll
  • Compatibility
  • Demo
  • The Problem
  • Installation
  • Usage
  • Indexes and Performance
  • Cursors
    • Standard Cursor
    • Base64 Encoded Cursor
  • Contributing
  • Copyright and License

Mongoid::Scroll

Mongoid extension that enables infinite scrolling for Mongoid::Criteria and Mongo::Collection::View.

Compatibility

This gem supports Mongoid 6, 7, 8 and 9.

Demo

Take a look at this example. Try with with bundle exec ruby examples/feed.rb.

The Problem

Traditional pagination does not work when data changes between paginated requests, which makes it unsuitable for infinite scroll behaviors.

• If a record is inserted before the current page limit, items will shift right, and the next page will include a duplicate.
• If a record is removed before the current page limit, items will shift left, and the next page will be missing a record.

The solution implemented by the scroll extension paginates data using a cursor, giving you the ability to restart pagination where you left it off. This is a non-trivial problem when combined with sorting over non-unique record fields, such as timestamps.

Installation

Add the gem to your Gemfile and run bundle install.

gem 'mongoid-scroll'

Usage

A sample model.

module Feed
  class Item
    include Mongoid::Document

    field :title, type: String
    field :position, type: Integer

    index({ position: 1, _id: 1 })
  end
end

Scroll by :position and save a cursor to the last item.

saved_iterator = nil

Feed::Item.desc(:position).limit(5).scroll do |record, iterator|
  # each record, one-by-one
  saved_iterator = iterator
end

Resume iterating using the saved cursor and save the cursor to go backwards.

Feed::Item.desc(:position).limit(5).scroll(saved_iterator.next_cursor) do |record, iterator|
  # each record, one-by-one
  saved_iterator = iterator
end

Loop over the first records again.

Feed::Item.desc(:position).limit(5).scroll(saved_iterator.previous_cursor) do |record, iterator|
  # each record, one-by-one
  saved_iterator = iterator
end

Use saved_iterator.first_cursor to loop over the first records or saved_iterator.current_cursor to loop over the same records again.

The iteration finishes when no more records are available. You can also finish iterating over the remaining records by omitting the query limit.

Feed::Item.desc(:position).limit(5).scroll(saved_iterator.next_cursor) do |record, iterator|
  # each record, one-by-one
  saved_iterator = iterator
end

Indexes and Performance

A query without a cursor is identical to a query without a scroll.

# db.feed_items.find().sort({ position: 1 }).limit(7)
Feed::Item.desc(:position).limit(7).scroll

Subsequent queries use an $or to avoid skipping items with the same value as the one at the current cursor position.

# db.feed_items.find({ "$or" : [
#   { "position" : { "$gt" : 13 }},
#   { "position" : 13, "_id": { "$gt" : ObjectId("511d7c7c3b5552c92400000e") }}
# ]}).sort({ position: 1 }).limit(7)
Feed:Item.desc(:position).limit(7).scroll(cursor)

This means you need to hit an index on position and _id.

# db.feed_items.ensureIndex({ position: 1, _id: 1 })

module Feed
  class Item
    ...
    index({ position: 1, _id: 1 })
  end
end

Cursors

You can use Mongoid::Scroll::Cursor.from_record to generate a cursor. A cursor points at the last record of the iteration and unlike MongoDB cursors will not expire.

record = Feed::Item.desc(:position).limit(3).last
cursor = Mongoid::Scroll::Cursor.from_record(record, { field: Feed::Item.fields["position"] })
# cursor or cursor.to_s can be returned to a client and passed into .scroll(cursor)

You can also a field_name and field_type instead of a Mongoid field.

cursor = Mongoid::Scroll::Cursor.from_record(record, { field_type: DateTime, field_name: "position" })

When the include_current option is set to true, the cursor will include the record it points to:

record = Feed::Item.desc(:position).limit(3).last
cursor = Mongoid::Scroll::Cursor.from_record(record, { field: Feed::Item.fields["position"], include_current: true })
Feed::Item.asc(:position).limit(1).scroll(cursor).first # record

If the field_name, field_type or direction options you specify when creating the cursor are different from the original criteria, a Mongoid::Scroll::Errors::MismatchedSortFieldsError will be raised.

cursor = Mongoid::Scroll::Cursor.from_record(record, { field_type: DateTime, field_name: "position" })
Feed::Item.desc(:created_at).scroll(cursor) # Raises a Mongoid::Scroll::Errors::MismatchedSortFieldsError

Standard Cursor

The Mongoid::Scroll::Cursor encodes a value and a tiebreak ID separated by :, and does not include other options, such as scroll direction. Take extra care not to pass a cursor into a scroll with different options.

Base64 Encoded Cursor

The Mongoid::Scroll::Base64EncodedCursor can be used instead of Mongoid::Scroll::Cursor to generate a base64-encoded string (using RFC 4648) containing all the information needed to rebuild a cursor.

Feed::Item.desc(:position).limit(5).scroll(Mongoid::Scroll::Base64EncodedCursor) do |record, iterator|
   # iterator.next_cursor is of type Mongoid::Scroll::Base64EncodedCursor
end

Contributing

Fork the project. Make your feature addition or bug fix with tests. Send a pull request. Bonus points for topic branches.

Copyright and License

MIT License, see LICENSE for details.

(c) 2013-2024 Daniel Doubrovkine, based on code by Frank Macreery, Artsy Inc.