Fix: Allow filtering empty page enumerable premature reduce exit #35

dylan-chong · 2022-02-04T02:44:27Z

Because of ALLOW FILTERING, it is possible for a Scylla query to return pages of rows which are often empty, followed by non empty pages. This causes CQEx's enumerable to fail to iterate past the empty page and therefore not stream all the results

I populate my table (partitioned by id) where 5% (by random chance) of the events match the WHERE clause and 95% do not

  cqlsh:dev>   SELECT id, ad_source   FROM events_v4   WHERE ad_source = 'facebook'   ALLOW FILTERING ; # Then me holding down ENTER

---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
 id                                   | ad_source
--------------------------------------+-----------
 8c74bfc7-700f-4ab5-b513-e15c7713ac46 |  facebook

---MORE---
 id                                   | ad_source
--------------------------------------+-----------
 24c80ed1-e73e-4a3b-9687-abe91c4fb7ee |  facebook

---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
 id                                   | ad_source
--------------------------------------+-----------
 01d19351-80f6-4349-a162-14d3edaee365 |  facebook

---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
 id                                   | ad_source
--------------------------------------+-----------
 de43a3c9-92af-4cac-af38-ee0fb7857e41 |  facebook
 202bb937-9da4-4e27-9d9a-a1a3ab13b8ab |  facebook

---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
 id                                   | ad_source
--------------------------------------+-----------
 56828ab1-6c45-410a-89ed-1f0ec5d2198b |  facebook

---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
 id                                   | ad_source
--------------------------------------+-----------
 a87b7517-78e5-4103-b7e5-0ad4d3e97099 |  facebook

---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
 id                                   | ad_source
--------------------------------------+-----------
 7fa8a5f7-3631-474d-80e8-0fba1c9a07ec |  facebook

---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
 id                                   | ad_source
--------------------------------------+-----------
 432e9c1c-2e14-432c-998b-1ddce6442543 |  facebook

---MORE---
---MORE---
---MORE---
---MORE---
 id                                   | ad_source
--------------------------------------+-----------
 f6c2c9d8-96bd-4b66-8f0d-7e8b329d3815 |  facebook

---MORE---
---MORE---
---MORE---
 id                                   | ad_source
--------------------------------------+-----------
 45b119c9-ab53-49c8-bd08-9ffd51920e24 |  facebook

---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
 id                                   | ad_source
--------------------------------------+-----------
 c9132aac-af54-4550-b382-aba7a3784b99 |  facebook

---MORE---
 id                                   | ad_source
--------------------------------------+-----------
 7c6b0afd-8654-4018-886d-8c0d2648465a |  facebook

---MORE---
---MORE---
---MORE---
 id                                   | ad_source
--------------------------------------+-----------
 f53ada20-cd2f-4915-b0ba-c046d5ab897b |  facebook

---MORE---
---MORE---
 id                                   | ad_source
--------------------------------------+-----------
 d0f34c5b-d690-4d80-b345-df10d077982a |  facebook

---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
 id                                   | ad_source
--------------------------------------+-----------
 053f5200-c862-46d4-aef8-5c2162abaafc |  facebook

---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
 id                                   | ad_source
--------------------------------------+-----------
 fe5190e2-a546-4035-858b-76544c0f08b9 |  facebook

---MORE---
 id                                   | ad_source
--------------------------------------+-----------
 b6aa50f9-f165-40b7-ab6f-13375a9eecd6 |  facebook

---MORE---
---MORE---
---MORE---
---MORE---
 id                                   | ad_source
--------------------------------------+-----------
 4d369465-233d-4539-83ce-8ee3e711b0a0 |  facebook

---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
 id                                   | ad_source
--------------------------------------+-----------
 cc44dfbb-c197-4157-8b5d-ec068176724a |  facebook

---MORE---
---MORE---
 id                                   | ad_source
--------------------------------------+-----------
 7437131b-8449-47df-b013-c9afde14ae7e |  facebook

---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
 id                                   | ad_source
--------------------------------------+-----------
 2d3c6594-8505-4f10-95c6-d1579c44a21b |  facebook

---MORE---
---MORE---
---MORE---
 id                                   | ad_source
--------------------------------------+-----------
 fab43882-f7f3-4123-991d-be7941427e33 |  facebook

---MORE---
---MORE---
---MORE---
 id                                   | ad_source
--------------------------------------+-----------
 fb0ca097-29dc-4008-bd34-5d291013d42e |  facebook

---MORE---
---MORE---
---MORE---
---MORE---
 id                                   | ad_source
--------------------------------------+-----------
 892226fe-3559-42b6-b74e-8372c04b7803 |  facebook

---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
 id                                   | ad_source
--------------------------------------+-----------
 2996bba4-875b-496e-bca5-e179c68d1cbe |  facebook

---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
 id                                   | ad_source
--------------------------------------+-----------
 068a3b11-6f92-4740-ae82-41fe86f6d547 |  facebook

---MORE---
 id                                   | ad_source
--------------------------------------+-----------
 caa21abe-ed65-484b-8770-4bf9319a0e67 |  facebook

---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
 id                                   | ad_source
--------------------------------------+-----------
 df2e824b-d42e-49ab-97f9-3996b9a8112f |  facebook

---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
 id                                   | ad_source
--------------------------------------+-----------
 7f105c85-b5bd-4402-a720-9671117a91ee |  facebook

---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
 id                                   | ad_source
--------------------------------------+-----------
 43673514-2957-4a62-9127-e73154de53f6 |  facebook

---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
 id                                   | ad_source
--------------------------------------+-----------
 7dcce63f-4469-44ab-812e-ea5075250fd0 |  facebook

---MORE---
 id                                   | ad_source
--------------------------------------+-----------
 b3e8ed89-5fa4-4817-9b22-ab1c11546e53 |  facebook

---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
 id                                   | ad_source
--------------------------------------+-----------
 ac9eb32d-9c2a-4a64-853f-13d03d5b8acf |  facebook

---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
 id                                   | ad_source
--------------------------------------+-----------
 d81bc50f-04eb-4fb6-86d7-e71545acf3f0 |  facebook

---MORE---
---MORE---
 id                                   | ad_source
--------------------------------------+-----------
 6bd2fdc3-4940-4ff2-9986-5c215a237662 |  facebook

---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
 id                                   | ad_source
--------------------------------------+-----------
 e349984c-4383-4905-8601-f0625ee7bf12 |  facebook

---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
 id                                   | ad_source
--------------------------------------+-----------
 49f00387-57e5-44a9-b42d-61221587a6a3 |  facebook

---MORE---
---MORE---
---MORE---
---MORE---
---MORE---
(38 rows)
cqlsh:dev>
cqlsh:dev>

As you can see there are many empty pages by there are still several events

I now try this using CQEx and find no events

defmodule T do
  def t do
    IO.puts("START")

    query = %CQEx.Query{
      statement: """
      SELECT id, ad_source
      FROM events_v4
      WHERE ad_source = 'facebook'
      ALLOW FILTERING
      """,
      named: "fix_mmmmm_fb_event_query"
    }

    CQEx.Client.new!()
    |> CQEx.Query.call!(query)
    |> Enum.each(fn event ->
      event[:id] |> IO.inspect(label: "found event ")
    end)

    IO.puts("END")
  end
end

iex(85)> T.t
START
END
:ok

I now run this again but with redbug

iex(87)> :redbug.start('cqerl->return', msgs: 10000, time: 15_000)
{2144, 36}
iex(88)> T.t
START

% 15:24:31 <0.6279.0>({'Elixir.IEx.Evaluator',init,4})
% cqerl:get_client()

% 15:24:31 <0.6279.0>({'Elixir.IEx.Evaluator',init,4})
% cqerl:get_client/0 -> {ok,{<0.4459.0>,#Ref<0.2140264969.181403653.147479>}}

% 15:24:31 <0.6279.0>({'Elixir.IEx.Evaluator',init,4})
% cqerl:run_query({<0.4459.0>,#Ref<0.2140264969.181403653.147479>}, {cql_query,<<"SELECT id, ad_source\nFROM events_v4\nWHERE ad_source = 'facebook'\nALLOW FILTERING\n">>,
           #{},undefined,<<"fix_mmmmm_fb_event_query">>,100,undefined,1,
           undefined,undefined})

% 15:24:31 <0.6279.0>({'Elixir.IEx.Evaluator',init,4})
% cqerl:run_query({<0.4459.0>,#Ref<0.2140264969.181403653.147479>}, {cql_query,<<"SELECT id, ad_source\nFROM events_v4\nWHERE ad_source = 'facebook'\nALLOW FILTERING\n">>,
           #{},undefined,<<"fix_mmmmm_fb_event_query">>,100,undefined,1,
           undefined,undefined}, 2)

% 15:24:31 <0.4459.0>({cqerl_client,init,1})
% cqerl:get_protocol_version()

% 15:24:31 <0.4459.0>({cqerl_client,init,1})
% cqerl:get_protocol_version/0 -> 4

% 15:24:31 <0.3006.3>({cqerl_processor,process,4})
% cqerl:put_protocol_version(4)

% 15:24:31 <0.3006.3>({cqerl_processor,process,4})
% cqerl:put_protocol_version/1 -> undefined

% 15:24:31 <0.3006.3>(dead)
% cqerl:get_protocol_version()

% 15:24:31 <0.3006.3>(dead)
% cqerl:get_protocol_version/0 -> 4

% 15:24:31 <0.4459.0>({cqerl_client,init,1})
% cqerl:get_protocol_version()

% 15:24:31 <0.4459.0>({cqerl_client,init,1})
% cqerl:get_protocol_version/0 -> 4

% 15:24:31 <0.3007.3>(dead)
% cqerl:put_protocol_version(4)

% 15:24:31 <0.3007.3>(dead)
% cqerl:put_protocol_version/1 -> undefined

% 15:24:31 <0.6279.0>({'Elixir.IEx.Evaluator',init,4})
% cqerl:run_query/3 -> {ok,{cql_result,
                               [{cqerl_result_column_spec,<<"dev">>,
                                    <<"events_v4">>,id,varchar},
                                {cqerl_result_column_spec,<<"dev">>,
                                    <<"events_v4">>,ad_source,varchar}],
                               [],
                               {cql_query,
                                   <<"SELECT id, ad_source\nFROM events_v4\nWHERE ad_source = 'facebook'\nALLOW FILTERING\n">>,
                                   #{},undefined,
                                   <<"fix_mmmmm_fb_event_query">>,100,
                                   <<0,0,0,0,163,0,0,0,48,0,0,0,1,0,0,0,36,0,0,
                                     0,101,49,50,98,52,48,50,101,45,50,54,48,
                                     56,45,52,102,49,97,45,56,57,48,100,45,99,
                                     53,99,54,54,97,51,101,100,56,54,101,1,20,
                                     0,0,0,1,0,0,0,8,0,0,0,0,0,1,126,110,162,
                                     112,184,255,255,255,255,188,74,251,163,
                                     143,140,21,94,93,101,253,253,167,16,244,
                                     164,1,0,0,0,32,0,0,0,0,1,25,0,0,0,20,0,0,
                                     0,1,0,0,0,8,0,0,0,129,118,159,166,31,252,
                                     91,35,1,0,1,0,0,0,6,78,180,144,10,181,28,
                                     97,213,110,183,148,64,75,31,131,1,0,0,0,0,
                                     0,255,255,255,255,0,0,0,0>>,
                                   1,undefined,undefined},
                               {<0.4459.0>,
                                #Ref<0.2140264969.181403653.147479>}}}

% 15:24:31 <0.6279.0>({'Elixir.IEx.Evaluator',init,4})
% cqerl:run_query/2 -> {ok,{cql_result,
                               [{cqerl_result_column_spec,<<"dev">>,
                                    <<"events_v4">>,id,varchar},
                                {cqerl_result_column_spec,<<"dev">>,
                                    <<"events_v4">>,ad_source,varchar}],
                               [],
                               {cql_query,
                                   <<"SELECT id, ad_source\nFROM events_v4\nWHERE ad_source = 'facebook'\nALLOW FILTERING\n">>,
                                   #{},undefined,
                                   <<"fix_mmmmm_fb_event_query">>,100,
                                   <<0,0,0,0,163,0,0,0,48,0,0,0,1,0,0,0,36,0,0,
                                     0,101,49,50,98,52,48,50,101,45,50,54,48,
                                     56,45,52,102,49,97,45,56,57,48,100,45,99,
                                     53,99,54,54,97,51,101,100,56,54,101,1,20,
                                     0,0,0,1,0,0,0,8,0,0,0,0,0,1,126,110,162,
                                     112,184,255,255,255,255,188,74,251,163,
                                     143,140,21,94,93,101,253,253,167,16,244,
                                     164,1,0,0,0,32,0,0,0,0,1,25,0,0,0,20,0,0,
                                     0,1,0,0,0,8,0,0,0,129,118,159,166,31,252,
                                     91,35,1,0,1,0,0,0,6,78,180,144,10,181,28,
                                     97,213,110,183,148,64,75,31,131,1,0,0,0,0,
                                     0,255,255,255,255,0,0,0,0>>,
                                   1,undefined,undefined},
                               {<0.4459.0>,
                                #Ref<0.2140264969.181403653.147479>}}}

% 15:24:31 <0.6279.0>({'Elixir.IEx.Evaluator',init,4})
% cqerl:size({cql_result,
    [{cqerl_result_column_spec,<<"dev">>,<<"events_v4">>,id,varchar},
     {cqerl_result_column_spec,<<"dev">>,<<"events_v4">>,ad_source,varchar}],
    [],
    {cql_query,
        <<"SELECT id, ad_source\nFROM events_v4\nWHERE ad_source = 'facebook'\nALLOW FILTERING\n">>,
        #{},undefined,<<"fix_mmmmm_fb_event_query">>,100,
        <<0,0,0,0,163,0,0,0,48,0,0,0,1,0,0,0,36,0,0,0,101,49,50,98,52,48,50,
          101,45,50,54,48,56,45,52,102,49,97,45,56,57,48,100,45,99,53,99,54,54,
          97,51,101,100,56,54,101,1,20,0,0,0,1,0,0,0,8,0,0,0,0,0,1,126,110,162,
          112,184,255,255,255,255,188,74,251,163,143,140,21,94,93,101,253,253,
          167,16,244,164,1,0,0,0,32,0,0,0,0,1,25,0,0,0,20,0,0,0,1,0,0,0,8,0,0,
          0,129,118,159,166,31,252,91,35,1,0,1,0,0,0,6,78,180,144,10,181,28,97,
          213,110,183,148,64,75,31,131,1,0,0,0,0,0,255,255,255,255,0,0,0,0>>,
        1,undefined,undefined},
    {<0.4459.0>,#Ref<0.2140264969.181403653.147479>}})

% 15:24:31 <0.6279.0>({'Elixir.IEx.Evaluator',init,4})
% cqerl:size/1 -> 0

% 15:24:31 <0.6279.0>({'Elixir.IEx.Evaluator',init,4})
% cqerl:has_more_pages({cql_result,
    [{cqerl_result_column_spec,<<"dev">>,<<"events_v4">>,id,varchar},
     {cqerl_result_column_spec,<<"dev">>,<<"events_v4">>,ad_source,varchar}],
    [],
    {cql_query,
        <<"SELECT id, ad_source\nFROM events_v4\nWHERE ad_source = 'facebook'\nALLOW FILTERING\n">>,
        #{},undefined,<<"fix_mmmmm_fb_event_query">>,100,
        <<0,0,0,0,163,0,0,0,48,0,0,0,1,0,0,0,36,0,0,0,101,49,50,98,52,48,50,
          101,45,50,54,48,56,45,52,102,49,97,45,56,57,48,100,45,99,53,99,54,54,
          97,51,101,100,56,54,101,1,20,0,0,0,1,0,0,0,8,0,0,0,0,0,1,126,110,162,
          112,184,255,255,255,255,188,74,251,163,143,140,21,94,93,101,253,253,
          167,16,244,164,1,0,0,0,32,0,0,0,0,1,25,0,0,0,20,0,0,0,1,0,0,0,8,0,0,
          0,129,118,159,166,31,252,91,35,1,0,1,0,0,0,6,78,180,144,10,181,28,97,
          213,110,183,148,64,75,31,131,1,0,0,0,0,0,255,255,255,255,0,0,0,0>>,
        1,undefined,undefined},
    {<0.4459.0>,#Ref<0.2140264969.181403653.147479>}})

% 15:24:31 <0.6279.0>({'Elixir.IEx.Evaluator',init,4})
% cqerl:has_more_pages/1 -> true

% 15:24:31 <0.6279.0>({'Elixir.IEx.Evaluator',init,4})
% cqerl:fetch_more({cql_result,
    [{cqerl_result_column_spec,<<"dev">>,<<"events_v4">>,id,varchar},
     {cqerl_result_column_spec,<<"dev">>,<<"events_v4">>,ad_source,varchar}],
    [],
    {cql_query,
        <<"SELECT id, ad_source\nFROM events_v4\nWHERE ad_source = 'facebook'\nALLOW FILTERING\n">>,
        #{},undefined,<<"fix_mmmmm_fb_event_query">>,100,
        <<0,0,0,0,163,0,0,0,48,0,0,0,1,0,0,0,36,0,0,0,101,49,50,98,52,48,50,
          101,45,50,54,48,56,45,52,102,49,97,45,56,57,48,100,45,99,53,99,54,54,
          97,51,101,100,56,54,101,1,20,0,0,0,1,0,0,0,8,0,0,0,0,0,1,126,110,162,
          112,184,255,255,255,255,188,74,251,163,143,140,21,94,93,101,253,253,
          167,16,244,164,1,0,0,0,32,0,0,0,0,1,25,0,0,0,20,0,0,0,1,0,0,0,8,0,0,
          0,129,118,159,166,31,252,91,35,1,0,1,0,0,0,6,78,180,144,10,181,28,97,
          213,110,183,148,64,75,31,131,1,0,0,0,0,0,255,255,255,255,0,0,0,0>>,
        1,undefined,undefined},
    {<0.4459.0>,#Ref<0.2140264969.181403653.147479>}})

% 15:24:31 <0.4459.0>({cqerl_client,init,1})
% cqerl:get_protocol_version()

% 15:24:31 <0.4459.0>({cqerl_client,init,1})
% cqerl:get_protocol_version/0 -> 4

% 15:24:31 <0.3008.3>(dead)
% cqerl:put_protocol_version(4)

% 15:24:31 <0.3008.3>(dead)
% cqerl:put_protocol_version/1 -> undefined

% 15:24:31 <0.3008.3>(dead)
% cqerl:get_protocol_version()

% 15:24:31 <0.3008.3>(dead)
% cqerl:get_protocol_version/0 -> 4

% 15:24:31 <0.4459.0>({cqerl_client,init,1})
% cqerl:get_protocol_version()
END

% 15:24:31 <0.4459.0>({cqerl_client,init,1})
% cqerl:get_protocol_version/0 -> 4

% 15:24:31 <0.3009.3>(dead)
% cqerl:put_protocol_version(4)

% 15:24:31 <0.3009.3>(dead)
% cqerl:put_protocol_version/1 -> undefined
:ok

% 15:24:31 <0.6279.0>({'Elixir.IEx.Evaluator',init,4})
% cqerl:fetch_more/1 -> {ok,{cql_result,
                                [{cqerl_result_column_spec,<<"dev">>,
                                     <<"events_v4">>,id,varchar},
                                 {cqerl_result_column_spec,<<"dev">>,
                                     <<"events_v4">>,ad_source,varchar}],
                                [],
                                {cql_query,
                                    <<"SELECT id, ad_source\nFROM events_v4\nWHERE ad_source = 'facebook'\nALLOW FILTERING\n">>,
                                    #{},undefined,
                                    <<"fix_mmmmm_fb_event_query">>,100,
                                    <<0,0,0,0,9,1,0,0,48,0,0,0,1,0,0,0,36,0,0,
                                      0,48,48,56,48,53,100,55,51,45,48,51,98,
                                      99,45,52,102,48,98,45,98,101,101,50,45,
                                      100,97,48,100,55,98,49,55,55,55,101,54,1,
                                      20,0,0,0,1,0,0,0,8,0,0,0,0,0,1,126,111,
                                      182,192,210,255,255,255,255,188,74,251,
                                      163,143,140,21,94,93,101,253,253,167,16,
                                      244,164,2,0,0,0,57,0,0,0,1,25,0,0,0,20,0,
                                      0,0,1,0,0,0,8,0,0,0,133,10,128,146,212,
                                      212,104,171,0,1,25,0,0,0,20,0,0,0,1,0,0,
                                      0,8,0,0,0,134,20,223,84,240,81,85,118,1,
                                      0,1,0,0,0,6,78,180,144,10,181,28,97,213,
                                      110,183,148,64,75,31,131,57,0,0,0,1,25,0,
                                      0,0,20,0,0,0,1,0,0,0,8,0,0,0,129,118,159,
                                      166,31,252,91,35,0,1,25,0,0,0,20,0,0,0,1,
                                      0,0,0,8,0,0,0,133,10,128,146,212,212,104,
                                      171,1,0,1,0,0,0,6,78,180,144,10,181,28,
                                      97,213,110,183,148,64,75,31,131,1,0,0,0,
                                      0,0,255,255,255,255,0,0,0,0>>,
                                    1,undefined,undefined},
                                {<0.4459.0>,
                                 #Ref<0.2140264969.181403653.147479>}}}

% 15:24:31 <0.6279.0>({'Elixir.IEx.Evaluator',init,4})
% cqerl:next({cql_result,
    [{cqerl_result_column_spec,<<"dev">>,<<"events_v4">>,id,varchar},
     {cqerl_result_column_spec,<<"dev">>,<<"events_v4">>,ad_source,varchar}],
    [],
    {cql_query,
        <<"SELECT id, ad_source\nFROM events_v4\nWHERE ad_source = 'facebook'\nALLOW FILTERING\n">>,
        #{},undefined,<<"fix_mmmmm_fb_event_query">>,100,
        <<0,0,0,0,9,1,0,0,48,0,0,0,1,0,0,0,36,0,0,0,48,48,56,48,53,100,55,51,
          45,48,51,98,99,45,52,102,48,98,45,98,101,101,50,45,100,97,48,100,55,
          98,49,55,55,55,101,54,1,20,0,0,0,1,0,0,0,8,0,0,0,0,0,1,126,111,182,
          192,210,255,255,255,255,188,74,251,163,143,140,21,94,93,101,253,253,
          167,16,244,164,2,0,0,0,57,0,0,0,1,25,0,0,0,20,0,0,0,1,0,0,0,8,0,0,0,
          133,10,128,146,212,212,104,171,0,1,25,0,0,0,20,0,0,0,1,0,0,0,8,0,0,0,
          134,20,223,84,240,81,85,118,1,0,1,0,0,0,6,78,180,144,10,181,28,97,
          213,110,183,148,64,75,31,131,57,0,0,0,1,25,0,0,0,20,0,0,0,1,0,0,0,8,
          0,0,0,129,118,159,166,31,252,91,35,0,1,25,0,0,0,20,0,0,0,1,0,0,0,8,0,
          0,0,133,10,128,146,212,212,104,171,1,0,1,0,0,0,6,78,180,144,10,181,
          28,97,213,110,183,148,64,75,31,131,1,0,0,0,0,0,255,255,255,255,0,0,0,
          0>>,
        1,undefined,undefined},
    {<0.4459.0>,#Ref<0.2140264969.181403653.147479>}})

% 15:24:31 <0.6279.0>({'Elixir.IEx.Evaluator',init,4})
% cqerl:next/1 -> empty_dataset

what is suspicious to me is the last call: % cqerl:has_more_pages/1 -> true which returns true.
This I guess indicates the Enumerable recursion exits before cqerl:has_more_pages can return false.
This last line % cqerl:next/1 -> empty_dataset indicates that we prematurely exit.

    defp maybe_fetch_and_continue(result, acc, fun) do
      case R.has_more_pages?(result) do
        true ->
          next_page = R.fetch_more!(result)

          case R.next(next_page) do                  ######### returns `:empty_dataset`
            {h, t} -> reduce(t, fun.(h, acc), fun)    
            :empty_dataset -> {:done, acc}         ######### this prematurely exists on an empty page
          end

        false ->
          {:done, acc}
      end
    end

Now it works! 🥳

iex(1)> T.t
START
found event : "8c74bfc7-700f-4ab5-b513-e15c7713ac46"
found event : "24c80ed1-e73e-4a3b-9687-abe91c4fb7ee"
found event : "01d19351-80f6-4349-a162-14d3edaee365"
found event : "de43a3c9-92af-4cac-af38-ee0fb7857e41"
found event : "202bb937-9da4-4e27-9d9a-a1a3ab13b8ab"
found event : "56828ab1-6c45-410a-89ed-1f0ec5d2198b"
found event : "a87b7517-78e5-4103-b7e5-0ad4d3e97099"
found event : "7fa8a5f7-3631-474d-80e8-0fba1c9a07ec"
found event : "432e9c1c-2e14-432c-998b-1ddce6442543"
found event : "f6c2c9d8-96bd-4b66-8f0d-7e8b329d3815"
found event : "45b119c9-ab53-49c8-bd08-9ffd51920e24"
found event : "c9132aac-af54-4550-b382-aba7a3784b99"
found event : "7c6b0afd-8654-4018-886d-8c0d2648465a"
found event : "f53ada20-cd2f-4915-b0ba-c046d5ab897b"
found event : "d0f34c5b-d690-4d80-b345-df10d077982a"
found event : "053f5200-c862-46d4-aef8-5c2162abaafc"
found event : "fe5190e2-a546-4035-858b-76544c0f08b9"
found event : "b6aa50f9-f165-40b7-ab6f-13375a9eecd6"
found event : "4d369465-233d-4539-83ce-8ee3e711b0a0"
found event : "cc44dfbb-c197-4157-8b5d-ec068176724a"
found event : "7437131b-8449-47df-b013-c9afde14ae7e"
found event : "2d3c6594-8505-4f10-95c6-d1579c44a21b"
found event : "fab43882-f7f3-4123-991d-be7941427e33"
found event : "fb0ca097-29dc-4008-bd34-5d291013d42e"
found event : "892226fe-3559-42b6-b74e-8372c04b7803"
found event : "2996bba4-875b-496e-bca5-e179c68d1cbe"
found event : "068a3b11-6f92-4740-ae82-41fe86f6d547"
found event : "caa21abe-ed65-484b-8770-4bf9319a0e67"
found event : "df2e824b-d42e-49ab-97f9-3996b9a8112f"
found event : "7f105c85-b5bd-4402-a720-9671117a91ee"
found event : "43673514-2957-4a62-9127-e73154de53f6"
found event : "7dcce63f-4469-44ab-812e-ea5075250fd0"
found event : "b3e8ed89-5fa4-4817-9b22-ab1c11546e53"
found event : "ac9eb32d-9c2a-4a64-853f-13d03d5b8acf"
found event : "d81bc50f-04eb-4fb6-86d7-e71545acf3f0"
found event : "6bd2fdc3-4940-4ff2-9986-5c215a237662"
found event : "e349984c-4383-4905-8601-f0625ee7bf12"
found event : "49f00387-57e5-44a9-b42d-61221587a6a3"
END
:ok

This affects scylla 3.x and 4.x. Have not tried Cassandra

This issue will also affect the defp find function as well

dylan-chong · 2022-02-04T02:47:49Z

My god, this issue has plagued us for a few years now. I'm so happy it's fixed now 😭 😄

dylan-chong · 2022-02-22T21:49:51Z

@matehat is this something you could review+merge?

matehat · 2022-02-23T13:53:26Z

Sorry for the delay, will review today!

dylan-chong · 2022-05-10T01:07:58Z

Sorry for the bother, but would be great to get this merged

mgoo

The fix makes sense to me
@matehat any chance of getting this merged? would be really nice!

Biffo89 · 2022-05-12T06:42:16Z

Omg I would love for this to get merged. This has been bugging me for so long. Can't wait for code review

Fix: Allow filtering empty page enumerable premature reduce exit

bc4ca53

dylan-chong added 2 commits February 4, 2022 16:15

Comment

37e7b14

Simplify and remove unneeded comment

cab0591

mgoo approved these changes May 12, 2022

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix: Allow filtering empty page enumerable premature reduce exit #35

Fix: Allow filtering empty page enumerable premature reduce exit #35

dylan-chong commented Feb 4, 2022 •

edited

Loading

dylan-chong commented Feb 4, 2022

dylan-chong commented Feb 22, 2022

matehat commented Feb 23, 2022

dylan-chong commented May 10, 2022

mgoo left a comment

Biffo89 commented May 12, 2022

Fix: Allow filtering empty page enumerable premature reduce exit #35

Are you sure you want to change the base?

Fix: Allow filtering empty page enumerable premature reduce exit #35

Conversation

dylan-chong commented Feb 4, 2022 • edited Loading

dylan-chong commented Feb 4, 2022

dylan-chong commented Feb 22, 2022

matehat commented Feb 23, 2022

dylan-chong commented May 10, 2022

mgoo left a comment

Choose a reason for hiding this comment

Biffo89 commented May 12, 2022

dylan-chong commented Feb 4, 2022 •

edited

Loading