Skip to content

Header Transformation not working #165

Closed
@antarr

Description

@antarr

I'm trying to use header transformations when importing data from a CSV file. When I look at the values using byebug I get a value like this

(byebug) array_of_hashes[0]
{:countycode=>37, :firstname=>"FIRSTNAME", :lastname=>"LASTNAME", :addressline1=>"739       TWIN OAKS AVE", :city=>"CHULA VISTA", :zip=>91910, :email=>"REDACTED", :dob=>"REDACTED", :precinct=>"CHULA VISTA", :precinctnumber=>528500.0, :precinctid=>19099}

but I am expecting something like

{:county_code=>37, :firstname=>"FIRSTNAME", :last_name=>"LASTNAME", :residential_address1=>"739       TWIN OAKS AVE", :residential_city=>"CHULA VISTA", :residential_zip=>91910, :email=>"REDACTED", :date_of_birth=>"REDACTED", :precinct=>"CHULA VISTA", :precinct_number=>528500.0, :precinct_id=>19099}

import task

    task :seed, [:batch_id] => :environment do |_t, _args|
      options = {
        chunk_size: 1000,
        header_transformations: [
          key_mapping: key_mapping
        ]
      }

      Dir.entries(dir_path = Rails.root.join('data_files/records')).each do |file_name|
        next unless file_name.end_with?('.csv')

        chunks = SmarterCSV.process("#{dir_path}/#{file_name}", options)
        progress = ProgressBar.create(title: "#{dir_path}/#{file_name}", total: chunks.size)
        Parallel.map(chunks, in_threads: 10) do |chunk|
          worker(chunk)
          progress.increment
        end
        File.rename("#{dir_path}/#{file_name}", "#{dir_path}/#{file_name}.done")
      end
    end

    def key_mapping
      {
        firstname: :first_name,
        lastname: :last_name,
        middlename: :middle_name,
        precint: :precinct_name,
        precinctnumber: :precinct_number,
        precintid: :precinct_id,
        dob: :date_of_birth,
        email: :email,
        addressline1: :residential_address1,
        addressline2: :residential_secondary_addr,
        city: :residential_city,
        zip: :residential_zip
      }
    end

    def worker(array_of_hashes)
      byebug
      RecordImportWorker.perform_async array_of_hashes, @model[:id]
      AddressImportWorker.perform_async array_of_hashes, @model[:id]
    end

Headers in file

CountyCode,FirstName,MiddleName,LastName,AddressLine1,AddressLine2,City,Zip,Email,DOB,Precinct,PrecinctNumber,PrecinctId

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions