Dialog is an accessible modal extension which is the default modal for Shadows. The CSS is part of /themes/%STORE_ID%/THEME_NAME/core.css. If you want to make styling changes, it is recommended you add them to your /themes/%STORE_ID%/THEME_NAME/theme.css file.
If you are using Shadows DE, you can modify the CSS in build/core/css/components/_dialog.scss and update as part of your build process.