Skip to content

Foretello data convenience utility: Import and export CSV data

License

Notifications You must be signed in to change notification settings

frangdlt/hammer-cli-csv

 
 

Repository files navigation

hammer-cli-csv Build Status Coverage Status

Introduction

Hammer is a command line interface (CLI) framework which provides a core to which modules may be added. This module, hammer-cli-csv, adds commands to interact with the following products: Foreman standalone, Katello, Red Hat's Satellite-6, and Red Hat's Subscription Asset Manager (SAM).

The purpose of this module's commands are to allow a convenient mechanism to both export to and import from CSV files (comma separated values). Each of the server's supported resource types, such as organizations and users, is handled.

Some possible uses include

  • Import demo or development data easily and consistently
  • Export another server's data and then import into elsewhere for testing and debug
  • Export for backup and auditing

The following sections will cover installation, usage, and examples. All of the resource types are follow in the order which generally is required for dependency resolution (eg. roles must exist to assign to users so the role section comes first).

Installation

gem install hammer_cli_csv
gem install hammer_cli_katello

mkdir -p ~/.hammer/cli.modules.d/

cat <<EOQ > ~/.hammer/cli.modules.d/csv.yml
:csv:
  :enable_module: true
EOQ

# to confirm things work, this should return useful output
hammer csv --help

General Usage

Supported commands and options are available by using the --help option. Additional subcommands and options may be listed here but are considered "tech preview" to indicate the lack of testing and official support.

Option Description
--export If not specified will run import.
--file FILE_NAME File to import from or export to. If not specified reads or writes to/from stdin and stdout. Note: On ruby-1.8.7 this option is required.
--prefix PREFIX Convenience method to add a prefix to all Count substituted values. See examples below.
--server SERVER The server to run against. Overrides any config file value.
--username USERNAME Username for server. Overrides any config file value.
--password PASSWORD Password for user. Overrides any config file value.
--verbose Display verbose progress information during import.
--continue-on-error Continue processing even if individual resource error

Count Substitution

Some columns of input data have a special syntax available termed "Count substitution" below.

This hammer module started out as a way to generate large amounts of test data. As such, it was convenient to be able to add a single row to an input CSV file and have it generate multiple records.

Take this Organization CSV as an example:

Name, Count, Label, Description
# Start a row with a # to indicate a comment, a row to be skipped
Organization %d, 4, testorg%d, A test organization

The single row, with a Count value of four, would generate four organizations ("Organization 0", "Organization 1", "Organization 2", and "Organization 3"). Notice that the Label column, which must be unique, also has the Count column substition.

During export, the Count column will always be one (1).

Examples

Organizations

Overview

CSV Columns

Note: % column indicates Count substituion

Column Title Column Description %
Name Name of the organization to update or create x
Count Number of times to iterate this CSV row during import, incrementing value for substitution
Label Unique organization label x
Description Organization description

Examples

Here is an example of a CSV file to create an organization

Name, Count, Label, Description
# Start a row with a # to indicate a comment, a row to be skipped
Mega Corporation, 1, megacorp, The number one mega company in the world!

If above is saved to a file such as megacorp/organizations.csv the following command will run import to create or update the organization.

$ hammer csv organizations --version --csv-file megacorp/organizations.csv
Updating organization 'Mega Corporation'... done

$ hammer csv organizations --csv-export
"Name","Count","Label","Description"
"Mega Corporation","1","megacorp","The number one mega company in the world!"

# Import but prefix all substitution columns with a string
$ hammer csv organizations --verbose --csv-file test/data/organizations.csv --prefix xyz
Creating organization 'xyzMega Corporation'... done

# Export and pipe to import with a new prefix
$ hammer csv organizations --csv-export | hammer csv organizations --verbose --csv-file test/data/organizations.csv --prefix abc
Creating organization 'abcMega Corporation'... done

Locations

Overview

CSV Columns

Note: % column indicates Count substituion

Column Title Column Description %
Name Name of the location to update or create x
Count Number of times to iterate this CSV row during import, incrementing value for substitution
Parent Location Parent location

Puppet Environments

Overview

  • Due to the implications of removing a puppet environment from an organization or location, this column only adds to what is present already.
  • Open Issues
  • Tests
  • Sample data

CSV Columns

Note: % column indicates Count substituion

Column Title Column Description %
Name Name of the puppet environments to update or create x
Count Number of times to iterate this CSV row during import, incrementing value for substitution
Organizations Comma separated list of organizations

Operating Systems

Overview

CSV Columns

Note: % column indicates Count substituion

Column Title Column Description %
Name Name of the operating systems to update or create x
Count Number of times to iterate this CSV row during import, incrementing value for substitution
Family Operating system family
Description Operating system description
Password Hash MD5, SHA256, SHA512, or Base64
Partition Tables List of partition table names
Architectures List of architectures names
Media List of media names
Provisioning Templates List of provisioning template names
Parameters List of parameters

Domains

Overview

CSV Columns

Note: % column indicates Count substituion

Column Title Column Description %
Name Name of the domains to update or create x
Count Number of times to iterate this CSV row during import, incrementing value for substitution
Full Name Full name of the domain
Organizations Comma separated list of organizations

Architectures

Overview

CSV Columns

Note: % column indicates Count substituion

Column Title Column Description %
Name Name of the architectures to update or create x
Count Number of times to iterate this CSV row during import, incrementing value for substitution
Operating Systems Comma separated list of operating system names

Partition Tables

Overview

  • Import and export of the Organizations and Locations columns does not apply to all versions and will be silently ignored when unsupported.
  • Importing Operating Systems column does not apply to all versions and will be silently ignored when unsupported.
  • Open Issues
  • Tests
  • Sample data

CSV Columns

Note: % column indicates Count substituion

Column Title Column Description %
Name Name of the partition tables to update or create x
Count Number of times to iterate this CSV row during import, incrementing value for substitution
OS Family Operating system family
Operating Systems Comma separated list of operating system names
Layout Disk layout

Lifecycle Environments

Overview

CSV Columns

Note: % column indicates Count substituion

Column Title Column Description %
Name Name of the lifecycle environments to update or create x
Count Number of times to iterate this CSV row during import, incrementing value for substitution
Organization Organization name
Prior Environment Previous organization name in path
Description Lifecycle environment description

Host Collections

Additional arguments Description
--organization Only process organization matching this name

Overview

CSV Columns

Note: % column indicates Count substituion

Column Title Column Description %
Name Name of the host collections to update or create x
Count Number of times to iterate this CSV row during import, incrementing value for substitution
Organization Organization name
Limit Usage limit
Description Host collection description

Products

Additional arguments Description
--organization Only process organization matching this name
--[no-]sync Sync product repositories (default true)

Overview

  • Due to the length of time that syncing repositories can take, the --no-sync option may be used to skip this step. To always disable syncing, ':products_sync: false' may be specified in configuration file.
  • Open Issues
  • Tests
  • Sample data

CSV Columns

Note: % column indicates Count substituion

Column Title Column Description %
Name Name of the host collections to update or create x
Count Number of times to iterate this CSV row during import, incrementing value for substitution
Label Unique label
Organization Organization name
Repository Repository name
Repository Url Repository Url
Description Repository description

Provisioning Templates

Additional arguments Description
--organization Only process organization matching this name

Overview

CSV Columns

Note: % column indicates Count substituion

Column Title Column Description %
Name Name of the provisioning templates to update or create x
Count Number of times to iterate this CSV row during import, incrementing value for substitution
Organizations Comma separated list of organizations
Locations Comma separated list of locations
Operating Systems Comma separated list of associated operating systems
Host Group / Puppet Environment Combinations Comma separated list of host group and puppet environment associations
Kind Kind of template (eg. snippet)
Template Full text of template

Subscriptions

Additional arguments Description
--organization Only process organization matching this name

Overview

Import will only process rows with a Name of "Manifest" and trigger a manifest import to run on the specified Manifest File. Subscription rows are not be processed.

Export will output a summary of the subscriptions currently imported into the organizations. These rows will have a description in the Name column as a comment (value starting with a #). For example,

Name,Organization,Manifest File,Subscription Name,Quantity,Product SKU,Contract Number,Account Number
# Subscription,Mega Corporation,,"OpenShift Enterprise Premium, 2 Cores",2,MCT2735,10999111,5700573

CSV Columns

Note: % column indicates Count substituion

Column Title Column Description %
Name "Manifest" to trigger import of manifest file x
Count Number of times to iterate this CSV row during import, incrementing value for substitution
Organization Organization name
Manifest File Path to manifest file
Subscription Name Name of subscription
Quantity Subscription quantity
SKU Subscription SKU
Contract Number Subscription contract number
Account Number Subscription account number

Activation Keys

Overview

CSV Columns

Note: % column indicates Count substituion

Column Title Column Description %
Name Name of the activation keys to update or create x
Count Number of times to iterate this CSV row during import, incrementing value for substitution
Organization Parent organization name
Description Activation key description
Limit Usage limit
Environment Lifecycle environment name
Content View Content view name
Host Collections Comma separated list of host collections
Subscriptions Comma separated list of subscriptions

Hosts

Overview

CSV Columns

Note: % column indicates Count substituion

Column Title Column Description %
Name Name of the hosts to update or create x
Count Number of times to iterate this CSV row during import, incrementing value for substitution
Organization Organization name
Environment Puppet environment name
Operating System Operating system name
Architecture Architecture name
MAC Address Unique MAC address x
Domain Domain name
Partition Table Partition table name

Content Hosts

Additional arguments Description
--organization Only process organization matching this name
--itemized-subscriptions Export one subscription per row, only process update subscriptions on import

Overview

CSV Columns

Note: % column indicates Count substituion

Column Title Column Description %
Name Name of the content hosts to update or create x
Search Alternative to Name column to update content hosts matching search results x
Count Number of times to iterate this CSV row during import, incrementing value for substitution
Organization Organization name
Environment Puppet environment name
Content View Content view name
Host Collections Comma separate list of host collectoins
Virtual Virtual (Yes or No)
Host Virtual content host name
OS Operating system name
Arch Architecture name
Sockets Number of sockets
RAM Amount of RAM with units
Cores Number of cores
SLA Service Level Agreement
Products Comma separated list of subscriptions
Subscriptions Comma separated list of subscriptions
Subscription Name Applicable for --itemized-subscriptions
Subscription Type Applicable for --itemized-subscriptions
Subscription Quantity Applicable for --itemized-subscriptions
Subscription SKU Applicable for --itemized-subscriptions
Subscription Contract Applicable for --itemized-subscriptions
Subscription Account Applicable for --itemized-subscriptions
Subscription Start Applicable for --itemized-subscriptions
Subscription End Applicable for --itemized-subscriptions

Reports

Overview

CSV Columns

Note: % column indicates Count substituion

Column Title Column Description %
Name Name of the reports to update or create x
Count Number of times to iterate this CSV row during import, incrementing value for substitution
Time Time of report
Applied
Restarted
Failed
Failed Restarts
Skipped
Pending
Metrics

Roles

Overview

CSV Columns

Note: % column indicates Count substituion

Column Title Column Description %
Name Name of the roles to update or create x
Count Number of times to iterate this CSV row during import, incrementing value for substitution
Resource Resource to apply role to
Search Search string
Permissions Role permission
Organizations Comma separated list of organizations
Locations Comma separated list of locations

Users

Overview

CSV Columns

Note: % column indicates Count substituion

Column Title Column Description %
Name Name of the users to update or create x
Count Number of times to iterate this CSV row during import, incrementing value for substitution
First Name First name of user
Last Name Last name of user
email Email address
Organizations Comma separated list of organizations
Locations Comma separated list of locations
Roles Comma separated list of role names for user

Settings

Overview

CSV Columns

Note: % column indicates Count substituion

Column Title Column Description %
Name Setting name to update or create x
Count Number of times to iterate this CSV row during import, incrementing value for substitution
Value Setting value

Examples

Name,Value
administrator,[email protected]
idle_timeout,60000

Import

Overview

Examples

% hammer csv import -v --dir test/data --settings
Importing settings from 'test/data/settings.csv'
Updating setting 'idle_timeout'...done

Development

Code style

rubocop -R <file>

Tests

To run all tests using recorded data, run:

rake test mode=none

To run all tests to record data:

rake test mode=all

To run a single test using recorded data, run:

rake test mode=none test=resources/settings

or

rake test mode=none test=./test/resources/settings_test.rb

To run tests against your live Katello without recording a new cassette set record flag to false (does not apply to mode=none):

record=false

To see RestClient logs while testing:

logging=true

Test server configuration is taken from 'test/config.yml'. If that file does not exist then https://localhost with admin / changeme is the default.

% cat test/config.yml

:foreman:
  :enable_module: true
  :host:          'http://katello:3000'
  :username:      'admin'
  :password:      'changeme'
:katello:
  :enable_module: true
:csv:
  :enable_module: true

Note: Since the subscriptions tests rely entirely on having a Red Hat manifest, re-recording these tests is more involved. It requires access to the internal "stage" customer portal and API. To run in recording mode, specify PORTALUSERNAME and PORTALPASSWORD env variables.

PORTALUSERNAME=thomasmckay PORTALPASSWORD=xyz123 rake test mode=all record=true

About

Foretello data convenience utility: Import and export CSV data

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Ruby 99.4%
  • Makefile 0.6%