Check Point R8x Export, Import, Set/Update, Rename to new-name, and Delete mgmt_cli API scripts for bash on Check Point Gaia OS management hosts, using CSV files (v00.60.xx.yyy)
Additional documentation and information will be provided in .md, .tsv, and potentially .xlsx files in the repository.
Interim update, reorganize the README.MD and add a Quick Start
The export, import, set-update, rename-to-new-name, and delete using CSV files scripts in this post, currently version 00.60.12.100 dated 2023-03-14, are intended to allow operations on an existing R80, R80.10, R80.20[|.M1|.M2], R80.30, R80.40, R81, R81.10 and R81.20 Check Point management server (SMS or MDSM MDS) from bash expert mode on the Check Point management server host or another API enabled Check Point management server host instance (Check Point Gaia OS R8X) able to authenticate and reach the target management server host. Utilization from other LINUX releases is not supported, tested, or assumed to work.
The current set of supported objects by API version is in the supported_objects.v00.60.12.100.tsv file described by the supported_objects.version.tsv.md document
- Check Point Management API documentation is here: https://sc1.checkpoint.com/documents/latest/APIs/index.html#introduction
- Details about supported Check Point API management objects are documented in the file supported_objects.{version}.tsv file, which is updated with each version
- Scrubbing of exported data prior to import is highly recommended to ensure import or other operations take place on the desired objects.
- Smart-1 Cloud MaaS support is under evaluation and development, though CLI parameter for --context was added to facilitate current use--this is not yet tested effectively.
These scripts show examples of:
- an export of objects with full and standard json output, and export to csv output that can be reused for import or other operations.
- "standard" export of all plausible object keys and their values for supported objects
- "name-only" export of object key "name" values for supported objects. This export is used with the delete operation (scrub ensure only the desired targets are deleted)
- "uid-only" export of object key "uid" values for supported objects
- "name-and-uid" export of object keys "name" and "uid" with their respective values for supported objects
- "rename-to-new-name" export of object key "name" and values for supported objects, a second "name" is provided under the "new-name" key for renaming operations
- "name-for-delete" Export of name key value for object delete also sets other settings needed for clean delete control CSV
- an import of objects from csv output generated by the export to csv operations above
- a set (or update) operation of different objects, similar to the import operation using csv output generated by export operation
- a script to delete objects using csv files created by an object export in "name-only" mode to csv for the respective items deleted. NOTE: DANGER!, DANGER!, DANGER! Use at own risk with extreme care!
- CSV manipulation scripts to add error handling columns to existing CSV files for addressing import, set-update, rename-to-new-name, and delete operations
- MDM script to document domains and output to a domains_list.txt file for reference in calls to other scripts
- Session Cleanup scripts to show and also remove zero lock sessions that may accumulate.
- Script templates for writing new scripts and utilize the existing capabilities provided for CLI parameter handling and basic API related operations, like logon, logoff, publish.
For direct questions, contact the author at ericb (at) checkpoint (dot) com or lookup information on https://community.checkpoint.com CheckMates community.
This post includes a set of script packages, which can be used independently or combined. All script files end with .sh for bash shell and are intended for Check Point Software Technologies Gaia OS bash expert implementation on release versions R80, R80.10, R80.20[including M1 & M2], R80.30, R80.40, R81, and R81.10; potentially later versions. Scripts in the packages have specific purposes and some scripts call subscripts for extensive repeated or common operations (e.g. CLI parameter handling, mgmt_cli authentication and basic operations, etc.). The packages also include specific expected default directory folders that are not created by the script action.
A set script templates is also provided to help with development of other scripts.
Releases have packages for the key script folders:
The script packages are:
- Complete Deployment Package : devops.dev.{version}.tgz - includes devops.results and devops.my_data folders
- Deployment Package : devops.dev.only.{version}.tgz
The approach to provided compressed packages was changed to facilitate quicker implementation and deployment on the management hosts.
- Check Point Gaia OS based Management host with version R80 or higher to run the mgmt_cli commands
- To properly execute the scripts successfully, the user executing must have appropriate rights to access the Check Point Gaia OS based Management host in bash expert mode as well as credentials for access agaist the target Check Point based Management host API (SmartConsole administrator user account)
- To use the Export/Import scripts under objects.wip/export_import.wip|objects/export_import folder the subscripts folder _api_subscripts is required at the same level as the objects.wip|objects folder.
Recommended installation is to use the provided devops.dev.{version}.tgz and expand that to the working folder on the Gaia OS host, which should be placed under /var/log/ folder to ensure survival during Gaia OS upgrades. The tgz file is the devops.dev folder with current scripts and tools, and a devops.results folder for results if using --RESULTS option, and the devops.my_data for holding modified csv files for operations.
- What's New
- HOW TO Guidance
- Limitations and Caveats
- Supported Objects TSV
- Explanation of the Supported Objects TSV
- File and Folder Purpose and Utilization
- Check Point Software Technologies Management API Reference
- Check Point Software Technologies APIs
- Consolidated mgmt_cli bash Script solutions (Under Development)
- Limited Export of mgmt_cli based Policy and Layers export and import scripts
- Windows Scripts for Check Point Operations
To quickly start working with the scripts, do the following.
-
Create the working __customer folder under /var/log, if that does not exist and configure
mkdir /var/log/__customerchmod 775 /var/log/__customercd /var/log/__customer -
Download the release tgz file and deploy to a work folder on the target management host, like /var/log/__customer, the folder should be under the /var/log folder to ensure survival during upgrades
-
Expand the TGZ file, e.g.
Example:
tar -xvf devops.dev.{version}.tgztar -xvf devops.dev.v00.60.12.100.tgz -
Goto to the export import folder
cd ./devops.dev/objects.wip/export_import.wip -
Execute desired script with help parameter to show command options
Example:
./cli_api_export_objects_to_csv.sh --help
The following scripts are in the root of the folder ./devops.dev and will execute a minimu export for import on the local managemnet server (SMS). Other scripts for more detailed operations and testing are located under the ./devops.dev/objects.wip/object_operations folder.
Script Files:
_minimum_exports.sh: Execute the author's bare minimum export operations to enable import for SMS, Critical Performance Impact (CPI) objects, like all application-sites are not exported._minimum_exports_with_some_do_cpi.sh: Execute the author's bare minimum export operations to enable import for SMS, with some commands utilizing the CLI parameter to execute with Critical Performance Impact (CPI) objects._minimum_system_data_exports_with_some_do_cpi.sh: Execute the author's standard export operations using domain "System Data" for SMS, with some commands utilizing the CLI parameter to execute with Critical Performance Impact (CPI) objects.
See the WHATS_NEW.v00.60.12.100.md file for latest changes to the capabilities of the scripts
As of v00.60.00.050 the approach the folder structure has changed and object specific scripts are under objects.wip folder. As of v00.60.00.045 the approach to shared scripts has changed to focus only on the current work in progress under devops.dev folder. As of v00.60.00.075 the added connection time out handling for mgmt_cli calls, added default 600 seconds and CLI parameter for external control.
As of v00.60.08 the efforts are made to expidite the operations involving generations of CSV exports, but these are dependent on up-to-date json data from the management database, to this end a JSON Repository was implemented, more details are below.
Well functioning sets of scripts shall be packaged into releases that can be downloaded as a set for quick deployment and implementation. Future effort to create an installation and update solution, similar to other scripting solutions targeting Check Point Software Technologies will be analyzed, pending method of providing sustainable locations for such downloads.
With v00.60.08.050 the folder structure was adjusted to facilitate the other project future work with policy and layers and sharing the _templates and_api_subscripts folders as well as tools.
Presumptive folder structure for R8X API Management CLI (mgmt_cli) Template based scripts
- [.wip] named folders are for development operations
| Folder | Folder Purpose |
|---|---|
/...{script_home_folder}/ |
the folder containing the script set, generally /var/log/__customer/devops[.dev] |
./_api_subscripts[.wip] |
folder for all scripts |
./_templates[.wip] |
folder for all scripts |
./tools |
folder for all scripts with additional tools not assumed on system |
./objects[.wip] |
folder for objects operations focused scripts |
./objects[.wip]/csv_tools[.wip] |
folder for objects operations for csv handling focused scripts |
./objects[.wip]/export_import[.wip] |
folder for objects operations export, import, set, rename, and delete focused scripts |
./objects[.wip]/export_import_research[.wip] |
folder for objects operations export operations research |
./objects[.wip]/object_operations |
folder for objects operations export, import, set, rename, and delete focused scripts |
./Policy_and_Layers[.wip] |
folder for policy and layers operations focused scripts |
./Session_Cleanup[.wip] |
folder for Session Cleanup operation focused scripts |
./tools.MDSM[.wip] |
folder for Tools focused on MDSM operations scripts |
As of v00.60.08.000 the efforts are made to expidite the operations involving generations of CSV exports, but these are dependent on up-to-date json data from the management database. With version v00.60.08.000 and later, additional controls are introducted to help create a "__json_objects_repository" folder with a repository of objects json data as files for fast JQ parsing in CSV exports. If, during CSV export operation a required json repository file is not found, then the normal mgmg_cli call is made instead. The "__json_objects_repository" folder is located in the normal results folder, but has CLI parameter controls for explicit setting of the json repository folder.
Adding this section regarding approach, especailly with respect to performance related limitations that are encountered on Multi-Domain Security Management (MDSM).
Specifically for MDSM it may be necessary to tweak the execution CLI parameter for --MAXOBJECTS X, which for MDSM is set for 250 objects while for SMS is set for 500, the absolute maximum value for "limit" in a mgmt_cli show call. The easiest way to check what is possible on the target MDSM Multi-Domain Server (MDS) host, is executing a few direct mgmt_cli commands looking for the first success value. Starting at a limit value of 250 objects, work down in 100, 50, or 25 increments to find where there is a success output.
Example, start at 250, check 150, then 125, 100 would have been next:
[Expert@yourhostname:0]# mgmt_cli -r true -d "Global" show application-sites limit 250 offset 0 details-level "full" -f json --conn-timeout 600
{
"code" : "generic_error",
"message" : "Error 502. The Management API service is not available. Please check that the Management API server is up and running."
}
[Expert@yourhostname:0]# mgmt_cli -r true -d "Global" show application-sites limit 150 offset 0 details-level "full" -f json --conn-timeout 600
{
"code" : "generic_error",
"message" : "Error 502. The Management API service is not available. Please check that the Management API server is up and running."
}
[Expert@yourhostname:0]# mgmt_cli -r true -d "Global" show application-sites limit 125 offset 0 details-level "full" -f json --conn-timeout 600 | tail
"iso-8601" : "2022-02-25T15:32-0600"
},
"creator" : "System"
},
"read-only" : true
} ],
"from" : 1,
"to" : 125,
"total" : 10052
}
[Expert@yourhostname:0]#
Based on the above example, adding --OVERRIDEMAXOBJECTS --MAXOBJECTS 125 to the command line execution parameters should ensure proper execution and completion; however, the execution increment will produce ome fun numbers in the files generated. Using --OVERRIDEMAXOBJECTS --MAXOBJECTS 100 may be better, but does require more execution cycles.