[docs-beta] migrate - dagster+ settings

docs/docs-beta/CONTRIBUTING.md

@@ -97,9 +97,9 @@ Before:
 
 After:
 
-| Key | Value |
-|---|---|
-| `DAGSTER_CLOUD_DEPLOYMENT_NAME` | The name of the Dagster+ deployment. <br/><br/>  **Example:** `prod`. |
+| Key                                  | Value                                                                                                     |
+| ------------------------------------ | --------------------------------------------------------------------------------------------------------- |
+| `DAGSTER_CLOUD_DEPLOYMENT_NAME`      | The name of the Dagster+ deployment. <br/><br/> **Example:** `prod`.                                      |
 | `DAGSTER_CLOUD_IS_BRANCH_DEPLOYMENT` | `1` if the deployment is a [branch deployment](/dagster-plus/features/ci-cd/branch-deployments/index.md). |
 
 ### Whitespace via `{" "}`

docs/docs-beta/docs/dagster-plus/deployment/management/settings/customizing-agent-settings.md

@@ -4,8 +4,42 @@
 unlisted: true
 ---
 
-{/* TODO move from https://docs.dagster.io/dagster-plus/deployment/agents/customizing-configuration */}
+:::note
+This guide is applicable to Dagster+.
+:::
+
+The Dagster+ Agent is a special variant of the Dagster instance used in [Dagster Open Source](/deployment/dagster-instance) and is configured through the same `dagster.yaml` file. You can customize your agent with these settings.
+
+:::note
+For [Kubernetes agents](/dagster-plus/deployment/agents/kubernetes/configuring-running-kubernetes-agent) deployed with the Dagster+ Helm chart, you'll need to refer to the Helm chart's config map for customizing the agent.
+:::
+
+## Enabling user code server TTL
+
+User code servers support a configurable time-to-live (TTL). The agent will spin down any user code servers that haven’t served requests recently and will spin them back up the next time they’re needed. Configuring TTL can save compute cost because user code servers will spend less time sitting idle.
+
+To configure TTL:
+
+<CodeExample filePath="./examples/docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/management/code_server_ttl.yaml" />
+
+## Streaming compute logs
+
+You can set up streaming compute logs by configuring the log upload interval (in seconds).
+
+<CodeExample filePath="./examples/docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/management/streaming_compute_logs.yaml" />
 
 ## Disabling compute logs
 
-{/* NOTE this is a placeholder section so the Hybrid deployment index page has somewhere to link to */}
+You can disable forwarding compute logs to Dagster+ by configuring the `NoOpComputeLogManager` setting:
+
+<CodeExample filePath="./examples/docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/management/disable_compute_logs.yaml" />
+
+## Writing compute logs to AWS S3
+
+You can write compute logs to an AWS S3 bucket by configuring the [dagster_aws.s3.compute_log_manager](/api/python-api/libraries/dagster-aws#dagster_aws.s3.S3ComputeLogManager) module.
+
+You are also able to stream partial compute log files by configuring the log upload interval (in seconds) using the `upload_interval` parameter.
+
+Note: Dagster Labs will neither have nor use your AWS credentials. The Dagster+ UI will be able to show the URLs linking to the compute log files in your S3 bucket when you set the `show_url_only` parameter to `true`.
+
+<CodeExample filePath="./examples/docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/management/aws_compute_logs.yaml" />

docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md

@@ -4,4 +4,121 @@
 unlisted: true
 ---
 
-{/* TODO move from https://docs.dagster.io/dagster-plus/managing-deployments/deployment-settings-reference */}
+# Dagster+ deployment settings reference
+
+:::note
+This guide is applicable to Dagster+.
+:::
+
+This reference describes the settings that can be configured for full deployments in [Dagster+](/dagster-plus).
+
+Refer to the [Managing deployments in Dagster+ guide](/dagster-plus/managing-deployments/managing-deployments#configuring-deployment-settings) for info about configuring settings in the Dagster+ interface or using the `dagster-cloud` CLI.
+
+## Settings schema
+
+Settings are formatted using YAML. For example:
+
+```yaml
+run_queue:
+  max_concurrent_runs: 10
+  tag_concurrency_limits:
+    - key: "database"
+      value: "redshift"
+      limit: 5
+
+run_monitoring:
+  start_timeout_seconds: 1200
+  cancel_timeout_seconds: 1200
+  max_runtime_seconds: 7200
+
+run_retries:
+  max_retries: 0
+
+sso_default_role: EDITOR
+```
+
+## Settings
+
+For each deployment, you can configure settings for:
+
+- [Run queue](#run-queue-run_queue)
+- [Run monitoring](#run-monitoring-run_monitoring)
+- [Run retries](#run-retries-run_retries)
+- [SSO default role](#sso-default-role)
+- [Non-isolated runs](#non-isolated-runs)
+
+### Run queue (run_queue)
+
+The `run_queue` settings allow you to specify how many runs can execute concurrently in the deployment.
+
+```yaml
+run_queue:
+  max_concurrent_runs: 10
+  tag_concurrency_limits:
+    - key: "database"
+      value: "redshift"
+      limit: 5
+```
+
+| Property | Descripton |
+| --- | --- |
+| run_queue.max_concurrent_runs | The maximum number of runs that are allowed to be in progress at once. Set to 0 to stop any runs from launching. Negative values aren't permitted. <ul><li>**Default** - `10` (20 minutes)</li><li>**Maximum** - `500` (Hybrid)[/dagster-plus/deployment/hybrid], `50` (Serverless)[/dagster-plus/deployment/serverless]</li></ul> |
+| run_queue.tag_concurrency_limits | A list of limits applied to runs with particular tags. <ul><li>**Defaults** - `[]`</li></ul> Each list item may have the following properties: <ul><li>`key`</li><li>`value`</li><ul><li>If defined, the `limit` is applied only to the `key-value` pair.</li><li>If set to a dict with applyLimitPerUniqueValue: true, the `limit` is applied to the number of unique values for the `key`.</li><li>If set to a dict with `applyLimitPerUniqueValue: true`, the limit is applied to the number of unique values for the `key`.</li></ul><li>`limit`</li></ul> |
+
+### Run monitoring (run_monitoring)
+
+The `run_monitoring` settings allow you to define how long Dagster+ should wait for runs to start before making them as failed, or to terminate before marking them as canceled.
+
+```yaml
+run_monitoring:
+  start_timeout_seconds: 1200
+  cancel_timeout_seconds: 1200
+  max_runtime_seconds: 7200
+```
+
+| Property | Descripton |
+| --- | --- |
+| run_monitoring.start_timeout_seconds | The number of seconds that Dagster+ will wait after a run is launched for the process or container to start executing. After the timeout, the run will fail. This prevents runs from hanging in `STARTING` indefinitely when the process or container doesn't start. <ul><li>**Default** - `1200` (20 minutes)</li></ul> |
+| run_monitoring.cancel_timeout_seconds | The number of seconds that Dagster+ will wait after a run termination is initiated for the process or container to terminate. After the timeout, the run will move into a `CANCELED` state. This prevents runs from hanging in `CANCELING` indefinitely when the process or container doesn't terminate cleanly. <ul><li>**Default** - `1200` (20 minutes)</li></ul> |
+| run_monitoring.max_runtime_seconds | The number of seconds that Dagster+ will wait after a run is moved into a `STARTED` state for the run to complete. After the timeout, the run will be terminated and moved into a `FAILURE` state. This prevents runs from hanging in `STARTED` indefinitely if the process is hanging. <ul><li>**Default** - `No limit`</li></ul> |
+
+### Run retries (run_retries)
+
+The `run_retries` settings allow you to define how Dagster+ handles retrying failed runs in the deployment.
+
+```yaml
+run_retries:
+  max_retries: 0
+```
+
+| Property | Descripton |
+| --- | --- |
+| run_retries.max_retries | The maximum number of times Dagster+ should attempt to retry a failed run. Dagster+ will use the default if this setting is undefined. <ul><li>**Default** - `0`</li></ul> |
+| run_retries.retry_on_asset_or_op_failure | Whether to retry runs that failed due to assets or ops in the run failing. Set this to false if you only want to retry failures that occur due to the run worker crashing or unexpectedly terminating, and instead rely on op or asset-level retry policies to retry assert or op failures. Setting this field to false will only change retry behavior for runs on dagster version 1.6.7 or greater. <ul><li>**Default** - `0`</li></ul> |
+
+### SSO default role
+
+The `sso_default_role` setting lets you configure the default role on the deployment which is granted to new users that log in via SSO. For more information on available roles, see the [Dagster+ permissions reference](/dagster-plus/account/managing-users/managing-user-roles-permissions#user-permissions-reference).
+
+```yaml
+sso_default_role: EDITOR
+```
+
+| Property | Descripton |
+| --- | --- |
+| sso_default_role | If SAML SSO is enabled, this is the default role that will be assigned to Dagster+ users for this deployment. If SAML SSO is not enabled, this setting is ignored. <ul><li>**Default** - `Viewer`</li></ul> |
+
+### Non-isolated runs
+
+Configure [non-isolated runs](/dagster-plus/deployment/serverless) on your deployment.
+
+```yaml
+non_isolated_runs:
+  enabled: True
+  max_concurrent_non_isolated_runs: 1
+```
+
+| Property | Descripton |
+| --- | --- |
+| enabled | If enabled, the `Isolate run environment` checkbox will appear in the Launchpad. <ul><li>**Default** - `true`</li></ul> |
+| max_concurrent_non_isolated_runs | A limit for how many non-isolated runs to launch at once. Once this limit is reached, the checkbox will be greyed out and all runs will be isolated. This helps to avoid running out of RAM on the code location server. <ul><li>**Default** - `1`</li></ul> |

docs/docs-beta/docusaurus.config.ts

@@ -1,7 +1,7 @@
 import {themes as prismThemes} from 'prism-react-renderer';
 import type {Config} from '@docusaurus/types';
 import type * as Preset from '@docusaurus/preset-classic';
-import { groupCollapsed } from 'console';
+import {groupCollapsed} from 'console';
 
 const config: Config = {
   title: 'Dagster Docs - Beta',

docs/docs-beta/sidebars.ts

@@ -5,7 +5,12 @@ const sidebars: SidebarsConfig = {
       type: 'category',
       label: 'Getting Started',
       collapsed: false,
-      items: ['intro', 'getting-started/quickstart', 'getting-started/installation', 'getting-started/glossary'],
+      items: [
+        'intro',
+        'getting-started/quickstart',
+        'getting-started/installation',
+        'getting-started/glossary',
+      ],
     },
     {
       type: 'category',
@@ -21,8 +26,8 @@ const sidebars: SidebarsConfig = {
       items: [
         {
           type: 'autogenerated',
-          dirName: 'guides/build'
-        }
+          dirName: 'guides/build',
+        },
       ],
     },
     {
@@ -33,8 +38,8 @@ const sidebars: SidebarsConfig = {
       items: [
         {
           type: 'autogenerated',
-          dirName: 'guides/automate'
-        }
+          dirName: 'guides/automate',
+        },
       ],
     },
     {
@@ -45,7 +50,7 @@ const sidebars: SidebarsConfig = {
       items: [
         {
           type: 'autogenerated',
-          dirName: 'guides/monitor'
+          dirName: 'guides/monitor',
         },
       ],
     },
@@ -57,8 +62,8 @@ const sidebars: SidebarsConfig = {
       items: [
         {
           type: 'autogenerated',
-          dirName: 'guides/test'
-        }
+          dirName: 'guides/test',
+        },
       ],
     },
     {
@@ -69,8 +74,8 @@ const sidebars: SidebarsConfig = {
       items: [
         {
           type: 'autogenerated',
-          dirName: 'guides/deploy'
-        }
+          dirName: 'guides/deploy',
+        },
       ],
     },
     {
@@ -90,9 +95,7 @@ const sidebars: SidebarsConfig = {
       type: 'category',
       label: 'Guides',
       collapsed: false,
-      items: [
-        'integrations/guides/multi-asset-integration'
-      ]
+      items: ['integrations/guides/multi-asset-integration'],
     },
     {
       type: 'category',
@@ -102,10 +105,10 @@ const sidebars: SidebarsConfig = {
       items: [
         {
           type: 'autogenerated',
-          dirName: 'integrations/libraries'
-        }
-      ]
-    }
+          dirName: 'integrations/libraries',
+        },
+      ],
+    },
   ],
   dagsterPlus: [
     'dagster-plus/index',
@@ -117,9 +120,9 @@ const sidebars: SidebarsConfig = {
       items: [
         {
           type: 'autogenerated',
-          dirName: 'dagster-plus/features'
-        }
-      ]
+          dirName: 'dagster-plus/features',
+        },
+      ],
     },
     {
       type: 'category',
@@ -128,8 +131,8 @@ const sidebars: SidebarsConfig = {
       items: [
         {
           type: 'autogenerated',
-          dirName: 'dagster-plus/deployment'
-        }
+          dirName: 'dagster-plus/deployment',
+        },
       ],
     },
   ],
@@ -147,7 +150,7 @@ const sidebars: SidebarsConfig = {
           dirName: 'api/python-api',
         },
       ],
-    }
+    },
   ],
 };
 

docs/docs-beta/src/theme/MDXComponents.tsx

@@ -1,6 +1,6 @@
 // Import the original mapper
 import MDXComponents from '@theme-original/MDXComponents';
-import { PyObject } from '../components/PyObject';
+import {PyObject} from '../components/PyObject';
 import CodeExample from '../components/CodeExample';
 import Tabs from '@theme/Tabs';
 import TabItem from '@theme/TabItem';

examples/docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/management/settings/aws_compute_logs.yaml

@@ -0,0 +1,31 @@
+# dagster.yaml
+instance_class:
+  module: dagster_cloud.instance
+  class: DagsterCloudAgentInstance
+
+dagster_cloud_api:
+  agent_token:
+    env: DAGSTER_CLOUD_AGENT_TOKEN
+  deployment: prod
+
+user_code_launcher:
+  module: dagster_cloud.workspace.docker
+  class: DockerUserCodeLauncher
+
+compute_logs:
+  module: dagster_aws.s3.compute_log_manager
+  class: S3ComputeLogManager
+  config:
+    bucket: "mycorp-dagster-compute-logs"
+    local_dir: "/tmp/cool"
+    prefix: "dagster-test-"
+    use_ssl: true
+    verify: true
+    verify_cert_path: "/path/to/cert/bundle.pem"
+    endpoint_url: "http://alternate-s3-host.io"
+    skip_empty_files: true
+    upload_interval: 30
+    upload_extra_args:
+      ServerSideEncryption: "AES256"
+    show_url_only: true
+    region: "us-west-1"

examples/docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/management/settings/code_server_ttl.yaml

@@ -0,0 +1,17 @@
+# dagster.yaml
+instance_class:
+  module: dagster_cloud.instance
+  class: DagsterCloudAgentInstance
+
+dagster_cloud_api:
+  agent_token:
+    env: DAGSTER_CLOUD_AGENT_TOKEN
+  deployment: prod
+
+user_code_launcher:
+  module: dagster_cloud.workspace.docker
+  class: DockerUserCodeLauncher
+  config:
+    server_ttl:
+      enabled: true
+      ttl_seconds: 7200 #2 hours

examples/docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/management/settings/disable_compute_logs.yaml

@@ -0,0 +1,17 @@
+# dagster.yaml
+instance_class:
+  module: dagster_cloud.instance
+  class: DagsterCloudAgentInstance
+
+dagster_cloud_api:
+  agent_token:
+    env: DAGSTER_CLOUD_AGENT_TOKEN
+  deployment: prod
+
+user_code_launcher:
+  module: dagster_cloud.workspace.docker
+  class: DockerUserCodeLauncher
+
+compute_logs:
+  module: dagster.core.storage.noop_compute_log_manager
+  class: NoOpComputeLogManager