BRIL_BIBGenerator

Introduction

This guide gives instructions on how to set up and run Beam Induced Background (BIB, alternatively MIB for Machine Induced Background) simulations in CMSSW with a two step method (BIB particle generation + simulation). For an official introduction and manual for CMSSW please have a look at the offline workbook.

Setup

Kickstart

Generation:

ssh -Y [email protected]
mkdir cmssw_test
cd cmssw_test
cmsrel CMSSW_11_2_0_pre6
cd CMSSW_11_2_0_pre6/src
git clone https://github.com/pkicsiny/BRIL_BIBGenerator.git
ln -s BRIL_BIBGenerator/GeneratorInterface GeneratorInterface
wget https://raw.githubusercontent.com/pkicsiny/BRIL_ITsim/master/BIBGeneration/python/BH_generator.py
scram b
mkdir generator_output
vi BH_generator.py
set output path at line 44: path/to/generator_output
set input files location at line 54: path/to/input
cmsRun BH_generation.py 

Simulation:

mkdir simulation_output
wget https://raw.githubusercontent.com/pkicsiny/BRIL_ITsim/master/BIBGeneration/generatePU.sub
wget https://raw.githubusercontent.com/pkicsiny/BRIL_ITsim/master/BIBGeneration/runSimTkOnly.sh
wget https://raw.githubusercontent.com/pkicsiny/BRIL_ITsim/master/BIBGeneration/python/BH_SimTrigRec.py
mkdir batchlog
vi runSimTkOnly.sh
set input files location at line 30: path/to/generator/output/myoutput.root
set output path at line 36: path/to/simulation_output
vi generatePU.sub
set number of MIB events to simulate at line 2
set relative path of BH_SimTrigRec.py at line 10
set number of parallel splittings at line 18
vi BH_SimTrigRec.py
set number of events processed in each chunk (total number of events/queue) at line 103
change the geometry according to the usecase at lines 76-85
cd ../../
mkdir temp
cd temp
cp ../CMSSW_11_2_0_pre6 .
cd CMSSW_11_2_0_pre6/src
delete everything except the SimG4Core folder
git clone https://github.com/pkicsiny/BRIL_ITsim.git
Delete everything from the above repository except the DataProductionTkOnly folder
cd ../../
tar -czvf sandbox.tar.bz2 CMSSW_11_2_0_pre6
cp sandbox.tar.bz2 ../CMSSW_11_2_0_pre6/src/
cd ../CMSSW_11_2_0_pre6/src/
condor_submit generatePU.sub

In details

Login to lxplus.

ssh -Y [email protected]

If you don't have any CMSSW release already, create and navigate to an empty directory

mkdir cmssw_test
cd cmssw_test

and clone a CMSSW release by

cmsrel CMSSW_11_2_0_pre6

which creates a local copy of CMSSW version 11.2.pre6. You can alternatively work with another version, but it is always recommended to use one of the newest ones (currently as of version 11). To find out about a list of currently available CMSSW releases, type

scram list -a

or visit the official CMSSW github page, where you can also browse the simulation source code. Now you can navigate to the source directory within the created CMSSW release

cd CMSSW_11_2_0_pre6/src

and activate a CMSSW working environment, paths and compiler (while being in the /src directory) by typing

cmsenv

This command has to be issued only once but every time you open up a new terminal and start to work with CMSSW. In the /src directory you can now clone this repository that will be used for BIB generation:

git clone https://github.com/pkicsiny/BRIL_BIBGenerator.git

CMSSW can only find and work with the code if it is located in the /src directory. Therefore the subdirectory BRIL_BIBGenerator/GeneratorInterface should be symlinked from the /src directory as shown below:

ln -s BRIL_BIBGenerator/GeneratorInterface GeneratorInterface

In addition, you will need to clone a config file for the generation and simulation steps respectively. These config files will be used to launch CMSSW and can be found here. You can simply use wget to clone the 2 config files into your /src directory. Type the following while being in /src

 wget https://raw.githubusercontent.com/pkicsiny/BRIL_ITsim/master/BIBGeneration/python/BH_generator.py

to get the config file for the generation step.

Getting the input

Some FLUKA simulated BIB files for beam halo and three types on beam gas (H, C and O) can be downloaded from here. It is recommended to place these files to the /eos file system because of their large size. Alternatively you can use to experiment with the input files that are used by the config file by default (see at line 54)

Generation step

Before running the generation step, you need to compile and build your C++ files in the BRIL_BIBGenerator. This is done by

scram b

and it is also recommended to store the output of the generation step in a dedicated folder.

mkdir generator_output

Before launching the BIB generation, you might want to have a look at the contents of the config file.

vi BH_generation.py

(You might use the tab key for auto-filling the path name when navigating in the terminal.) The config file accepts the following user parameters from the command line:
nEvents: number of events to read from the FLUKA input files. Can be set inside the config file through the nevents variable. If set to -1, all events from the inputs will be processed. Note that this is not recommended as CMSSW will throw an error when reaching the end of the last file. Instead, it is recommended to check the number of events input file beforehand and set this parameter accordingly, e.g. to 10000 if the inputs contain let's say 10426 events.
nThreads: number of parallel computing threads to use. Default is 1.
jobId: relevant when running the generation step on lxbatch, where the simulation is split into smaller chunks each having a unique job ID. Not used if the code is run locally on lxplus. Default is 0.
tDirectory: absolute path of the directory to where the root file will be created. It will contain the same particles as in the FLUKA dump input, but in a different, CMSSW friendly format, called HEPMC.

In addition, some other parameters can be specified inside the config file:
inputPath: absolute path specifying the location of the FLUKA input files. Also do not forget the file: prefix before the path! The line options.inputFiles= [inputPath + "/" + f for f in os.listdir(inputPath) if f[:3] == "run"] automatically parses the file names in the inputPath directory and selects files whose name begins with run. This parsing can also be adapted or removed depending on the specific usecase.
nevents: number of events, alternative hard-coded variant of nEvents.
The output file name can be changed under the #specify output name comment.

Although the CMS geometry plays no role in the generation step, CMSSW always expects an input geometry to be specified. You can use the most up to date Phase 2 full CMS geometry, just as a 'placeholder':

process.load('Configuration.Geometry.GeometryExtended2026D63_cff')

You can run the generation step by running the config file BH_generation.py.

cmsRun BH_generation.py 

which invokes code from BRIL_BIBGenerator/GeneratorInterface/BeamHaloGenerator/python/MIB_generator_cff.py which in turn invokes CMSSW through BRIL_BIBGenerator/GeneratorInterface/BeamHaloGenerator/src/BeamHaloProducer.cc.

Simulation step

The second step consists of the transport of particles and the simulation of particle-matter interactions in the CMSSW geometry model. The code for this step is created for running on lxbatch as simulating a large number of generated MIB particles might take a long time. First let's get the 3 necessary config files:

wget https://raw.githubusercontent.com/pkicsiny/BRIL_ITsim/master/BIBGeneration/generatePU.sub
wget https://raw.githubusercontent.com/pkicsiny/BRIL_ITsim/master/BIBGeneration/runSimTkOnly.sh
wget https://raw.githubusercontent.com/pkicsiny/BRIL_ITsim/master/BIBGeneration/python/BH_SimTrigRec.py

generatePU.sub

The first file (generatePU.sub) will be used to submit some files to the lxbatch cluster to do the simulation, using HTCondor. Line 1 can be ignored for MIB studies and line 2 specifies the number of MIB events to simulate. The rest of the file can be left as it is, except the last line, where you can define the number of "jobs" to submit and queue on the cluster. In order to run the most efficiently, lxbatch splits up the simulation into smaller chunks or sub-simulations (=jobs) and runs them in parallel. Depending on the number of events you intend to simulate, you can change the queue parameter. For example if at line 2 NEvents is set to 200000, you can set the queue_ to 40, that tells lxbatch to split up the simulation into 40 jobs each simulating only 200000/40=50000 events. This splitting is performed by selecting a subset of the events at runtime in the BH_SimTrigRec.py config file at lines 103-105 (currently set to 5000 events but it has to be modified accordingly to NEvents/queue). Lines 7-9 generatePU.sub in define the path where the simulation output, error and log files will be saved. Currently it is set to be saved in a batchlog folder which you can either change or create the batchlog directory in /src.

mkdir batchlog

At line 10 you can see that 2 files will be transferred to the cluster if the simulations is launched. The bash script file runSimTkOnly.sh is the highest level file controlling the simulation and invokes the python config file BH_Rec.py. Its path is now set to python/runSimTkOnly.sh but it should be changed accordingly, i.e. in this case just remove the python part.

BH_SimTrigRec.py

In this config file you can change the CMSSW geometry used for the simulation at lines 76-85. As opposed to the generation step, in the simulation step this config will not be launched directly, but instead it is invoked by runSimTkOnly.sh, which is in turn invoked by generatePU.sub.

runSimTkOnly.sh

In runSimTkOnly.sh, the INFILE parameter (currently at line 30) has to be set to point to the output root file from the generation step (e.g. at /absolute/path/to/cmssw_test/CMSSW_11_2_0_pre6/src/generator_output/myoutput.root). The parameter OUTDIR at line 36 should be similarly changed to point to the desired simulation output directory (e.g. to /absolute/path/to/cmssw_test/CMSSW_11_2_0_pre6/src/simulation_output). In general, nothing else has to be changed in this file. You can notice that it takes some command line arguments (lines 17-20):
PU: refers to pileup but it can be ignored for BIB particle simulations.
NEVENTS: number of events to read from the generator step output file and simulate. This parameter is inferred from line 1 of generatePU.sub.
JOBID: this can also be ignored. It will be set automatically on lxbatch and can take a value from 0 to the value of queue, which is set in the last line of generatePU.sub. The JOBID parameter uniquely identifies each split simulation chunk on the cluster.
At line 100 you can also see:

tar -xf sandbox.tar.bz2

which refers to an important step in setting up the simulations on the cluster.

Setting up a CMSSW sandbox

By default, simulations on lxbatch do not have access to CMSSW code stored locally on lxplus, therefore we need to send them to lxbatch along with the config file, wrapped up in a tar package, which will be automatically unpacked and used on the cluster. To create sandbox.tar.bz2, first exit the CMSSW_11_2_0_pre6 folder and create a temporary directory.

cd ../../
mkdir temp
cd temp

and copy here the full CMSSW_11_2_0_pre6 directory. Now enter the CMSSW_11_2_0_pre6/src and delete everything except BRIL_BIBGenerator/GeneratorInterface/SimG4Core which contains code for handling MIB particles in CMSSW and thus necessary for such simulations. To read more about what is different when simulating MIB particles instead of regular beam collision events, see here. Then move SimG4Core back up to the /src directory so that now that is the only content there. Then you will need some code for simulating detector response and digitization, which that can depend on the usecase. In case of BRIL, simulations were done with having only the tracker geometry. To get the necessary python files for this usecase, get the directory DataProductionTkOnly from here. The easiest (and dirtiest) way is to clone the whole repository and delete everything except the DataProductionTkOnly folder.

git clone https://github.com/pkicsiny/BRIL_ITsim.git

In the end you should have only SimG4Core (MIB specific code) and DataProductionTkOnly (detector specific code) directories in your copy of CMSSW_11_2_0_pre6/src. If so, move back up into your temp directory and compress your CMSSW release by

tar -czvf sandbox.tar.bz2 CMSSW_11_2_0_pre6

The resulting sandbox.tar.bz2 you can now copy to your original CMSSW_11_2_0_pre6/src/ directory.

cp sandbox.tar.bz2 ../CMSSW_11_2_0_pre6/src/

Finally, use

condor_submit generatePU.sub

to submit the simulation to lxbatch. The result root files will be placed to the path which was set in line 36 of runSimTkOnly.sh.