Datasource authentication updates #1289

khansaad · 2024-09-12T05:06:17Z

Description

This PR contains new feature addition of supporting different REST API authentication for Datasource, DB, cloudwatch and other services.

Fixes # (issue)

Type of change

Bug fix
New feature
Docs update
Breaking change (What changes might users need to make in their application due to this PR?)
Requires DB changes

How has this been tested?

Testing is in progress

Please describe the tests that were run to verify your changes and steps to reproduce. Please specify any test configuration required.

New Test X
Functional testsuite

Test Configuration

Kubernetes clusters tested on: Resource Hub cluster

Checklist 🎯

Additional information

Tested using this custom image: quay.io/khansaad/autotune_operator:ds-auth

Shows the failure when wrong token path is passed :

Signed-off-by: Saad Khan <[email protected]>

msvinaykumar · 2024-09-12T11:22:06Z

src/main/java/com/autotune/common/auth/AuthenticationConfig.java

+                ", credentials=" + credentials +
+                '}';
+    }
+}


EOF missing

msvinaykumar · 2024-09-12T11:23:16Z

src/main/java/com/autotune/operator/KruizeDeploymentInfo.java

@@ -38,8 +38,9 @@
 */
 public class KruizeDeploymentInfo {
    private static final Logger LOGGER = LoggerFactory.getLogger(KruizeDeploymentInfo.class);
-    public static String database_username;
-    public static String database_password;
+    public static String database_authentication_type;


why this change required ?
this change will affect ROS

Stashed for now. Will be updating later on for common auth feature

msvinaykumar · 2024-09-12T11:26:19Z

manifests/crc/default-db-included-installation/openshift/kruize-crc-openshift.yaml

@@ -75,10 +75,15 @@ data:
        "adminUsername": "admin",
        "hostname": "kruize-db-service",


I guess this PR is limited to datasource auth , But not for DB ?

msvinaykumar · 2024-09-12T11:27:23Z

manifests/crc/default-db-included-installation/openshift/kruize-crc-openshift.yaml

@@ -92,7 +97,7 @@ data:
      "savetodb": "true",
      "dbdriver": "jdbc:postgresql://",
      "plots": "true",
-      "local": "false",
+      "local": "true",


let default value set to false , Until ROS move into thanos

msvinaykumar · 2024-09-12T11:28:10Z

manifests/crc/default-db-included-installation/openshift/kruize-crc-openshift.yaml

          "logLevel": "INFO"
+          "authentication": {


Let make this PR limited scope .. Only datasource authentication

msvinaykumar · 2024-09-12T11:30:49Z

src/main/java/com/autotune/common/auth/Credentials.java

+                ", headerName='" + headerName + '\'' +
+                '}';
+    }
+}


EOF missing

msvinaykumar · 2024-09-12T11:37:36Z

src/main/java/com/autotune/common/datasource/prometheus/PrometheusDataOperatorImpl.java

-                        KruizeConstants.SupportedDatasources.PROMETHEUS
-                )
-        );
+    public JSONObject getJsonObjectForQuery(String url, String query, AuthenticationConfig authenticationConfig) {


notice that many functions are using authenticationConfig. This configuration should be set at the experiment level, similar to how experiment.setDatasource works. When retrieving the datasource using experiment.getDatasource, it should also return the associated authenticationConfig.

To achieve this, the authenticationConfig object should be linked to the datasource and accessible through it.

Updated now.
Instead of passing authenticationObject in every function, refactored the methods to take the datasource object instead so that it will help in fetching both the datasource URL as well as authenticationConfig

msvinaykumar · 2024-09-12T11:41:45Z

src/main/java/com/autotune/utils/CloudWatchAppender.java

@@ -73,7 +73,7 @@ public void append(LogEvent event) {
    }

    public static void configureLoggerForCloudWatchLog() {
-        if (cloudwatch_logs_access_key_id != null && !cloudwatch_logs_access_key_id.isEmpty() && cloudwatch_logs_secret_access_key != null && !cloudwatch_logs_secret_access_key.isEmpty() && cloudwatch_logs_region != null && !cloudwatch_logs_region.isEmpty()) {
+        if (cloudwatch_logs_authentication_credentials_access_key_id != null && !cloudwatch_logs_authentication_credentials_access_key_id.isEmpty() && cloudwatch_logs_authentication_credentials_secret_access_key != null && !cloudwatch_logs_authentication_credentials_secret_access_key.isEmpty() && cloudwatch_logs_region != null && !cloudwatch_logs_region.isEmpty()) {


lets limit this change only related to only datasource

msvinaykumar · 2024-09-12T11:44:29Z

src/main/java/com/autotune/experimentManager/handler/MetricCollectionHandler.java

                                    String queryResult = (String) ado.getValueForQuery(experimentTrial.getDatasourceInfoHashMap()
                                            .get(containerMetric.getDatasource())
-                                            .getUrl().toString(), updatedContainerQuery);
+                                            .getUrl().toString(), updatedContainerQuery, noAuthConfig);


authoconfig should get tagged to datasource obj and
datasource obj should get tagged to Experiment

so data pass using parameter should be just experiment Obj everywhere

…s passing the datasource URL and other review updates Signed-off-by: Saad Khan <[email protected]>

Signed-off-by: Saad Khan <[email protected]>

…onding updates Signed-off-by: Saad Khan <[email protected]>

msvinaykumar

LGTM

Review Checklist

Common Review

Code formatting adheres to the project’s style guide.
No unnecessary commented-out code.
Appropriate use of comments and documentation.

Logical Flow Review

Code logic is clear and easy to follow.
All branches and conditions are covered and tested.
No redundant or duplicate code.

Demo Update

Demonstration examples are updated to reflect changes.
README and documentation include updated demo instructions.
Any necessary demo files are included and tested.

Naming Conventions

Variable and function names follow the project's naming conventions.
File names are descriptive and adhere to naming standards.
No ambiguous or misleading names.

Workflow

CI/CD pipelines are updated and passing.
Tests are included and all pass.
The workflow for deploying or integrating changes is clear and documented.

Does It Affect ROS

Changes are reviewed for compatibility with ROS.
ROS-specific configurations are updated if necessary.
Any ROS dependencies or integrations are tested and verified.

chandrams · 2024-09-16T11:40:07Z

@khansaad - Can you resolve the conflicts.

Signed-off-by: Saad Khan <[email protected]>

khansaad · 2024-09-16T17:14:04Z

@khansaad - Can you resolve the conflicts.

Done

chandrams · 2024-09-17T05:54:22Z

manifests/crc/default-db-included-installation/openshift/kruize-crc-openshift.yaml

@@ -134,7 +134,13 @@ data:
          "provider": "prometheus",
          "serviceName": "prometheus-k8s",
          "namespace": "openshift-monitoring",
-          "url": ""
+          "url": "",


Please update the other openshift yaml in BYODB

Signed-off-by: Saad Khan <[email protected]>

chandrams

LGTM

chandrams · 2024-09-17T09:58:22Z

@khansaad - I see a merge conflict, can you please resolve this. Thanks.

khansaad added 2 commits September 10, 2024 17:01

add authentication support for datasources

e80d391

Signed-off-by: Saad Khan <[email protected]>

update DB,cloudwatch auth

9b81fd7

Signed-off-by: Saad Khan <[email protected]>

khansaad added enhancement New feature or request local_monitoring labels Sep 12, 2024

khansaad added this to the Kruize 0.0.25_rm Release milestone Sep 12, 2024

khansaad self-assigned this Sep 12, 2024

msvinaykumar reviewed Sep 12, 2024

View reviewed changes

khansaad added 2 commits September 12, 2024 21:19

temporarily stashed changes related to common auth, changes to method…

e4b2e4f

…s passing the datasource URL and other review updates Signed-off-by: Saad Khan <[email protected]>

add missing EOF

1e4e4c1

Signed-off-by: Saad Khan <[email protected]>

khansaad marked this pull request as ready for review September 12, 2024 15:55

khansaad added 2 commits September 12, 2024 22:40

fix issue occurring while converting authJson to authConfig object

25b30f9

Signed-off-by: Saad Khan <[email protected]>

revert changes in the Datasource table for authentication and corresp…

83f5afa

…onding updates Signed-off-by: Saad Khan <[email protected]>

msvinaykumar approved these changes Sep 13, 2024

View reviewed changes

dinogun requested a review from chandrams September 15, 2024 17:47

khansaad added 2 commits September 16, 2024 22:33

Merge remote-tracking branch 'upstream/mvp_demo' into datasource-auth

2def4bc

update manifests with missed URL change

8150c9e

Signed-off-by: Saad Khan <[email protected]>

chandrams reviewed Sep 17, 2024

View reviewed changes

rbadagandi1 linked an issue Sep 17, 2024 that may be closed by this pull request

Test scenarios for Datasource authentication #1286

Open

28 tasks

update BYODB manifests with the datasource object

1f2108b

Signed-off-by: Saad Khan <[email protected]>

chandrams approved these changes Sep 17, 2024

View reviewed changes

Merge remote-tracking branch 'upstream/mvp_demo' into datasource-auth

f0c5b16

chandrams merged commit f2b3fbd into kruize:mvp_demo Sep 17, 2024
1 of 3 checks passed

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Datasource authentication updates #1289

Datasource authentication updates #1289

khansaad commented Sep 12, 2024 •

edited

Loading

msvinaykumar Sep 12, 2024

msvinaykumar Sep 12, 2024

khansaad Sep 12, 2024

msvinaykumar Sep 12, 2024

khansaad Sep 12, 2024

msvinaykumar Sep 12, 2024

msvinaykumar Sep 12, 2024

msvinaykumar Sep 12, 2024

msvinaykumar Sep 12, 2024

khansaad Sep 12, 2024

msvinaykumar Sep 12, 2024

msvinaykumar Sep 12, 2024

msvinaykumar left a comment •

edited

Loading

chandrams commented Sep 16, 2024

khansaad commented Sep 16, 2024

chandrams Sep 17, 2024

chandrams left a comment

chandrams commented Sep 17, 2024

		@@ -75,10 +75,15 @@ data:
		"adminUsername": "admin",
		"hostname": "kruize-db-service",

Datasource authentication updates #1289

Datasource authentication updates #1289

Conversation

khansaad commented Sep 12, 2024 • edited Loading

Description

Type of change

How has this been tested?

Checklist 🎯

Additional information

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

msvinaykumar left a comment • edited Loading

Choose a reason for hiding this comment

Common Review

Logical Flow Review

Demo Update

Naming Conventions

Workflow

Does It Affect ROS

chandrams commented Sep 16, 2024

khansaad commented Sep 16, 2024

Choose a reason for hiding this comment

chandrams left a comment

Choose a reason for hiding this comment

chandrams commented Sep 17, 2024

khansaad commented Sep 12, 2024 •

edited

Loading

msvinaykumar left a comment •

edited

Loading