Issues with duplicate rows following zero filling

[mayamonk]

I'm having a problem with a duplication error while trying to produce zero-filled data using the complete eBird dataset to create a smaller presence-absence dataset. While following both tutorials from "Best Practices for Using eBird Data" (https://ebird.github.io/ebird-best-practices/) and "Introduction to auk" (https://cornelllabofornithology.github.io/auk/articles/auk.html#quick-start), the step to collapse the zero-filled data results in each entry being duplicated, and most seem to be 322 duplicates.

For instance, here is the code I used while following the "Introduction to Auk" tutorial:

library(auk)
library(dplyr)
library(ggplot2)
library(gridExtra)
library(lubridate)
library(readr)
library(sf)

states <- c("US-GA", "US-IL", "US-CO", "US-IN", "US-WI", "US-FL", "US-AZ", "US-NY", "US-MO", "US-WA", "US-DE")

input_file <- "/Volumes/UES_LAB/UWIN_acad_perf_analysis/ebird/ebd_US_relJun-2023.txt/ebd_US_relJun-2023.txt"
output_file <- "ebd-filtered-states.txt"
ebird_data <- input_file %>%
auk_ebd() %>%
auk_date(date = c("2011-01-01", "2012-12-31")) %>%
auk_country(country = "United States") %>%
auk_state(states) %>%
auk_complete() %>%
auk_filter(file = output_file) %>%
read_ebd()

ebird_data %>%
glimpse()

f_ebd <- output_file
f_smp <- output_file
filters <- auk_ebd(f_ebd, file_sampling = f_smp) %>%
auk_state(states) %>%
auk_complete()
filters

ebd_sed_filtered <- auk_filter(filters, file = "ebd_filteredPA.txt", file_sampling = "sampling_filteredPA.txt")
ebd_sed_filtered

read_ebd(ebd_sed_filtered)

A tibble: 1,070 × 48

read_ebd(f_ebd)

A tibble: 1,070 × 48

read_ebd(f_smp)
A tibble: 1,070 × 48

here the data shows 1,070 entries and everything had worked thus far

ebd_zf <- auk_zerofill(ebd_sed_filtered)
ebd_zf

Zero-filled EBD: 1,096 unique checklists, for 322 species.

ebd_zf_df <- collapse_zerofill(ebd_zf)
class(ebd_zf_df)
ebd_zf_df

A tibble: 352,912 × 57

After collapse_zerofill, each entry duplicates around 322 times. Using the other tutorial from "Best Practices for Using eBird Data" works the same way, in which the entries duplicate after the code:

zerofill <- auk_zerofill(observations, checklists, collapse = TRUE)

It also results in the same total number of entries: 352,912. Using code to remove duplicates is unsuccessful, such as:

unique.data.frame(zerofill)
unique.array(zerofill)
unique.matrix(zerofill)

("zerofill" is the name of the zero-filled dataset, these result in no change)

Has anyone run into this issue or knows a possible solution? Thanks!

[mstrimas]

I'm confused what's happening here:

f_ebd <- output_file
f_smp <- output_file
filters <- auk_ebd(f_ebd, file_sampling = f_smp)

you seem to be using the same file for both the observation dataset and the checklists dataset. I also don't understand why you have this second round of filtering. The idea is to filter the observations and checklists at the same time, i.e.

observations_input <- "ebd_US_relJun-2023.txt"
checklists_input <- "ebd_US_relJun-2023_sampling.txt"
observations_output <- "ebd-filtered-states_observations.txt"
checklists_output <- "ebd-filtered-states_checklists.txt"
ebird_data <- ?auk_ebd(observations_input, file_sampling = checklists_input) %>%
  auk_date(date = c("2011-01-01", "2012-12-31")) %>%
  auk_country(country = "United States") %>%
  auk_state(states) %>%
  auk_complete() %>%
  auk_filter(file = observations_output, file_sampling = checklists_output)

The checklist file (ending in _sampling) is provided when you select the "Include sampling event data" check box when downloading the data.