Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Header Transformation not working #165

Closed
antarr opened this issue Jan 3, 2021 · 1 comment
Closed

Header Transformation not working #165

antarr opened this issue Jan 3, 2021 · 1 comment

Comments

@antarr
Copy link

antarr commented Jan 3, 2021

I'm trying to use header transformations when importing data from a CSV file. When I look at the values using byebug I get a value like this

(byebug) array_of_hashes[0]
{:countycode=>37, :firstname=>"FIRSTNAME", :lastname=>"LASTNAME", :addressline1=>"739       TWIN OAKS AVE", :city=>"CHULA VISTA", :zip=>91910, :email=>"REDACTED", :dob=>"REDACTED", :precinct=>"CHULA VISTA", :precinctnumber=>528500.0, :precinctid=>19099}

but I am expecting something like

{:county_code=>37, :firstname=>"FIRSTNAME", :last_name=>"LASTNAME", :residential_address1=>"739       TWIN OAKS AVE", :residential_city=>"CHULA VISTA", :residential_zip=>91910, :email=>"REDACTED", :date_of_birth=>"REDACTED", :precinct=>"CHULA VISTA", :precinct_number=>528500.0, :precinct_id=>19099}

import task

    task :seed, [:batch_id] => :environment do |_t, _args|
      options = {
        chunk_size: 1000,
        header_transformations: [
          key_mapping: key_mapping
        ]
      }

      Dir.entries(dir_path = Rails.root.join('data_files/records')).each do |file_name|
        next unless file_name.end_with?('.csv')

        chunks = SmarterCSV.process("#{dir_path}/#{file_name}", options)
        progress = ProgressBar.create(title: "#{dir_path}/#{file_name}", total: chunks.size)
        Parallel.map(chunks, in_threads: 10) do |chunk|
          worker(chunk)
          progress.increment
        end
        File.rename("#{dir_path}/#{file_name}", "#{dir_path}/#{file_name}.done")
      end
    end

    def key_mapping
      {
        firstname: :first_name,
        lastname: :last_name,
        middlename: :middle_name,
        precint: :precinct_name,
        precinctnumber: :precinct_number,
        precintid: :precinct_id,
        dob: :date_of_birth,
        email: :email,
        addressline1: :residential_address1,
        addressline2: :residential_secondary_addr,
        city: :residential_city,
        zip: :residential_zip
      }
    end

    def worker(array_of_hashes)
      byebug
      RecordImportWorker.perform_async array_of_hashes, @model[:id]
      AddressImportWorker.perform_async array_of_hashes, @model[:id]
    end

Headers in file

CountyCode,FirstName,MiddleName,LastName,AddressLine1,AddressLine2,City,Zip,Email,DOB,Precinct,PrecinctNumber,PrecinctId
@antoinematyja
Copy link

Had a similar issue, turns out header_transformations was added in 2.x which has not been released yet, so you are probably using the 1.2.

You can find 1.2 docs here (which have a key_mapping option, just without header_transformation):
https://github.com/tilo/smarter_csv/tree/1.2-stable#deprecated-1x-options-to-be-replaced-in-20

Or if you want to use 2.0 (unstable):
#153 (comment)

@antarr antarr closed this as completed Jan 6, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants