Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implementation backend part of plugin in TypeScript #594

Open
wants to merge 72 commits into
base: v3.3
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
72 commits
Select commit Hold shift + click to select a range
b484d6d
initial migration commit
lunaticusgreen Jul 7, 2024
515879f
initial migration commit
lunaticusgreen Jul 7, 2024
c0024d9
add binary builds
lunaticusgreen Jul 7, 2024
ef3f85b
fix tsconfig.json
lunaticusgreen Jul 8, 2024
1ecbb81
Add alerts data processing
lunaticusgreen Jul 9, 2024
13728b3
packages cleanup
lunaticusgreen Jul 10, 2024
068d4e5
packages cleanup
lunaticusgreen Jul 10, 2024
b9004f2
packages cleanup
lunaticusgreen Jul 11, 2024
8b0f07b
packages cleanup
lunaticusgreen Jul 11, 2024
46d4a64
packages cleanup
lunaticusgreen Jul 11, 2024
1ab8cbd
packages cleanup
lunaticusgreen Jul 11, 2024
250faa0
packages cleanup
lunaticusgreen Jul 12, 2024
d4ac99b
fix type
lunaticusgreen Jul 12, 2024
f3a0d2c
fix type
lunaticusgreen Jul 12, 2024
8500693
bugfixes
lunaticusgreen Jul 15, 2024
6df689b
bugfixes
lunaticusgreen Jul 15, 2024
f911620
fix data transformation
lunaticusgreen Jul 15, 2024
62af727
fix data transformation
lunaticusgreen Jul 15, 2024
80a1033
fix data transformation
lunaticusgreen Jul 15, 2024
8057687
fix data transformation
lunaticusgreen Jul 15, 2024
42d6ccb
Merge branch 'master' into be-plugin-ts
lunaticusgreen Jul 15, 2024
b3affa8
Merge branch 'be-plugin-ts' of github.com:Altinity/clickhouse-grafana…
Slach Jul 16, 2024
eb9f98f
package-lock.json after npm install
Slach Jul 16, 2024
9d64f5d
fix Invalid interpolation format for "command" option in service "ba…
Slach Jul 16, 2024
f7343ab
add PKG_CACHE_PATH to backend_builder
Slach Jul 16, 2024
8544934
return back grafana-clickhouse-datasource and upgrade external instal…
Slach Jul 16, 2024
d66ad19
try to debug testflows failures
Slach Jul 16, 2024
000bb74
added healthchecks to proper selenium-standalone works
Slach Jul 16, 2024
87cd379
we don't need grafana_external_install, return -o classic back
Slach Jul 16, 2024
ff43a37
why i need to struggle with testflows instead of develop plugin
Slach Jul 16, 2024
9daaacf
add healthcheck to mysql and postgres docker-compose.yaml services
Slach Jul 16, 2024
2dce9f0
add more logging to failed tests and add trickster to selenium-standa…
Slach Jul 17, 2024
3384e7e
change GF_UNIFIED_ALERTING_ENABLED: false, GF_ALERTING_ENABLED:…
Slach Jul 17, 2024
568ab1b
add Show machine IP
Slach Jul 17, 2024
19b1131
Merge branch 'master' of github.com:Altinity/clickhouse-grafana into …
Slach Jul 17, 2024
69a140b
continue merge with master
Slach Jul 17, 2024
487f21c
remove wrong double `-d` arg in docker-compose up
Slach Jul 17, 2024
aa0f90d
Merge branch 'master' into be-plugin-ts
Slach Aug 11, 2024
f48b182
Merge branch 'master' into be-plugin-ts
Slach Aug 11, 2024
39ae837
Merge branch 'master' into be-plugin-ts
Slach Aug 15, 2024
dd5ea47
Merge branch 'be-plugin-ts' of github.com:Altinity/clickhouse-grafana…
Slach Aug 15, 2024
5ffba51
FIx data processing
lunaticusgreen Aug 26, 2024
f18780c
FIx data processing
lunaticusgreen Aug 26, 2024
ac4349e
FIx data processing
lunaticusgreen Aug 26, 2024
08e20b2
FIx data processing
lunaticusgreen Aug 26, 2024
8dd7a20
Update.
antip00 Aug 28, 2024
fd585ed
Merge https://github.com/Altinity/clickhouse-grafana
antip00 Sep 2, 2024
da610e7
Update.
antip00 Sep 3, 2024
369fd08
merge with master
Slach Sep 5, 2024
accf84f
Merge branch 'be-plugin-ts' of github.com:Altinity/clickhouse-grafana…
Slach Sep 5, 2024
d85e3ea
sync package.json with package-lock.json
Slach Sep 5, 2024
0aa590b
Merge branch 'v3.3' into be-plugin-ts
Slach Sep 13, 2024
c8c51bc
run CI and testflows in any PR
Slach Sep 13, 2024
b9c2a4b
run CI and testflows in any PR
Slach Sep 13, 2024
0ed5e45
Update.
antip00 Sep 16, 2024
404046a
Bump dompurify from 2.5.1 to 2.5.6
dependabot[bot] Sep 16, 2024
26233b4
Adding tests for legacy alerts.
antip00 Sep 18, 2024
a1fd944
Update.
antip00 Sep 18, 2024
cd1233e
Merge https://github.com/Altinity/clickhouse-grafana
antip00 Sep 18, 2024
feddd4c
Update.
antip00 Sep 18, 2024
fdc5498
Update.
antip00 Sep 18, 2024
c14f507
Bump github.com/grafana/grafana-plugin-sdk-go from 0.228.0 to 0.250.0
dependabot[bot] Sep 19, 2024
47a0cbb
Merge pull request #629 from Altinity/dependabot/go_modules/github.co…
Slach Sep 19, 2024
7e3158d
Merge pull request #627 from Altinity/dependabot/npm_and_yarn/dompuri…
Slach Sep 19, 2024
6562289
Merge pull request #628 from antip00/master
Slach Sep 19, 2024
d7a27f7
merge master with be-plugin-ts, resolved conflicts
Slach Sep 19, 2024
e4c7e23
return ffails back
Slach Sep 20, 2024
cfd1cf1
Fix incorrect import
lunaticusgreen Sep 23, 2024
e65cb15
Adding tests for default values.
antip00 Sep 23, 2024
cc4158f
Update.
antip00 Sep 23, 2024
453760e
Merge pull request #631 from antip00/master
Slach Oct 1, 2024
68113d7
merged with master, resolve conflicts
Slach Oct 1, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .config/tsconfig.json
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,6 @@
},
"transpileOnly": true
},
"include": ["../src", "./types", "/Users/lunaticus/Documents/Work/grafana-plugin-sdk-typescript3/**/*"],
"include": ["../src", "./types"],
"extends": "@grafana/tsconfig"
}
7 changes: 4 additions & 3 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,10 @@
"e2e": "npm exec cypress install && npm exec grafana-e2e run",
"e2e:update": "npm exec cypress install && npm exec grafana-e2e run --update-screenshots",
"server": "docker-compose up --build -d grafana",
"server:old": "GRAFANA_VERSION=10.4.5 docker-compose up --build -d grafana",
"sign": "bash -xec \"source ./.release_env && npx --yes @grafana/sign-plugin@latest\"",
"backend:build": "webpack -c webpack-be.config.js --env production",
"backend:dev": "webpack -w -c webpack-be.config.js --env development",
"backend:build": "webpack -c webpack-backend.config.js --env production",
"backend:dev": "webpack -w -c webpack-backend.config.js --env development",
"backend:build:binary": "npm run build-linux-arm64 && npm run build-linux-amd64 && npm run build-macos-arm64 && npm run build-macos-amd64 && npm run build-windows-amd64",
"build-linux-arm64": "pkg -t latest-linuxstatic-arm64 dist/backend-plugin.js -o dist/altinity-clickhouse-plugin_linux_arm64",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how to install pkg in system?

"build-linux-amd64": "pkg -t latest-linuxstatic-x64 dist/backend-plugin.js -o dist/altinity-clickhouse-plugin_linux_amd64",
Expand All @@ -30,7 +31,7 @@
"@grafana/e2e": "^10.2.3",
"@grafana/e2e-selectors": "10.2.3",
"@grafana/eslint-config": "^6.0.0",
"@grafana/ts-backend": "https://github.com/lunaticusgreen/grafana-plugin-sdk-typescript.git#integration-with-frontend",
"@grafana/ts-backend": "https://github.com/lunaticusgreen/grafana-plugin-sdk-typescript.git#integration-with-frontend2",
"@grafana/tsconfig": "^1.2.0-rc1",
"@swc/core": "1.3.75",
"@swc/helpers": "^0.5.0",
Expand Down
12 changes: 4 additions & 8 deletions src/backend/DataService.ts
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
import {DataService, logger} from '@grafana/ts-backend';
import {ClickhouseClient} from "./helpers/clickhouse-client";
import {transformData} from "./helpers/transform-data";
import {createQuery, getRequestSettings} from "./helpers/query";
import {DataQueryRequest} from "@grafana/data";
import {CHQuery} from "../types/types";
import {transformResponse} from "./helpers/transform-to-timeseries";

// @ts-ignore
type BackendDataQueryRequest<T> = Omit<DataQueryRequest<T>, "app", "timezone", "scopedVars">;
Expand Down Expand Up @@ -60,13 +60,9 @@ export class ClickhouseDataService extends DataService<Request, any> {
});

const results = await Promise.all(allQueryPromise);
logger.info("QueryData result", JSON.stringify(results));

const dataFrames = results.map((result: any, index: number) => transformData(result.body, targets[index].refid))


logger.info('Data Frames', JSON.stringify(dataFrames), targets.map((target: any) => target.refid));

return dataFrames
logger.info('ResultsData',JSON.stringify(transformResponse(results[0].body, targets[0].refid)))
// results.map((result: any, index: number) => transformData(result.body, targets[index].refid))
return transformResponse(results[0].body, targets[0].refid)
}
}
8 changes: 5 additions & 3 deletions src/backend/helpers/clickhouse-client.ts
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ export class ClickhouseClient {
data: this.settings.UsePost ? query : undefined,
params: !this.settings.UsePost ? { query } : undefined,
headers: {},
httpsAgent: new https.Agent(httpsAgentOptions)
// httpsAgent: new https.Agent(httpsAgentOptions)
};

if (this.settings.UseCompression && ['gzip', 'br', 'deflate', 'zstd'].includes(this.settings.CompressionType)) {
Expand All @@ -68,8 +68,10 @@ export class ClickhouseClient {
password
};
} else if (this.settings.UseYandexCloudAuthorization) {
reqConfig.headers['X-ClickHouse-User'] = this.settings.XHeaderUser;
if (this.settings.XHeaderKey) {
reqConfig.headers['X-ClickHouse-User'] = this.settings.XHeaderUser;
logger.info('Work with settings', this.settings)

if (this.settings.XHeaderKey) {
reqConfig.headers['X-ClickHouse-Key'] = this.settings.XHeaderKey;
}
const password = this.settings.Instance.DecryptedSecureJSONData['xHeaderKey'];
Expand Down
222 changes: 222 additions & 0 deletions src/backend/helpers/parser.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,222 @@
// @ts-nocheck
const datePrefix = "Date";
const dateTimePrefix = "DateTime";
const dateTime64Prefix = "DateTime64";
const timeZonePrefix = "('";
const timeZone64Separator = ",";
const dateTZPrefix = datePrefix + timeZonePrefix;
const dateTimeTZPrefix = dateTimePrefix + timeZonePrefix;
const dateTime64TZPrefix = dateTime64Prefix + timeZonePrefix;

const dateLayout = "yyyy-MM-dd";
const dateTimeLayout = `${dateLayout} HH:mm:ss`;
const dateTime64Layout3 = `${dateTimeLayout}.SSS`;
const dateTime64Layout6 = `${dateTimeLayout}.SSSSSS`;

const dateTimeTypeRE = /(Date\([^)]+\)|DateTime\([^)]+\)|DateTime64\([^)]+\))/;

function parseTimeZone(tz) {
try {
return new Intl.DateTimeFormat('en-US', { timeZone: tz }).resolvedOptions().timeZone;
} catch (err) {
return "UTC";
}
}

function extractTimeZoneNameFromFieldType(fieldType) {
let tz = "";
if (fieldType.startsWith(dateTZPrefix)) {
tz = fieldType.slice(dateTZPrefix.length + 1, -2);
} else if (fieldType.startsWith(dateTimeTZPrefix)) {
tz = fieldType.slice(dateTimeTZPrefix.length + 1, -2);
} else if (fieldType.startsWith(dateTime64TZPrefix) && fieldType.includes(timeZone64Separator)) {
tz = fieldType.slice(fieldType.indexOf(timeZone64Separator) + 3, -2);
} else if (dateTimeTypeRE.test(fieldType)) {
const matches = fieldType.match(dateTimeTypeRE);
if (matches.length > 0) {
return extractTimeZoneNameFromFieldType(matches[0]);
}
}
return tz.trim();
}

function fetchTimeZoneFromFieldType(fieldType, tzFromServer) {
const tz = extractTimeZoneNameFromFieldType(fieldType);
return tz !== "" ? parseTimeZone(tz) : tzFromServer;
}

function newDataFieldByType(fieldName, fieldType) {
if (fieldType.startsWith("LowCardinality")) {
fieldType = fieldType.slice("LowCardinality(".length, -1);
}

const isNullable = fieldType.includes("Nullable");
fieldType = fieldType.replace("Nullable(", "").replace(")", "");

switch (fieldType) {
case "String":
case "UUID":
case "IPv6":
case "IPv4":
return newStringField(fieldName, isNullable);
case "UInt8":
case "UInt16":
case "UInt32":
case "Int8":
case "Int16":
case "Int32":
case "Float32":
case "Float64":
return newFloat64Field(fieldName, isNullable);
case "UInt64":
if (fieldName === "t" && !isNullable) {
return newTimeField(fieldName, false);
}
return isNullable ? [] : [];
case "Int64":
return isNullable ? [] : [];
default:
if (fieldType.startsWith("Decimal")) {
return newFloat64Field(fieldName, isNullable);
} else if (fieldType.startsWith("FixedString") || fieldType.startsWith("Enum")) {
return newStringField(fieldName, isNullable);
} else if (fieldType.startsWith(dateTime64Prefix) || fieldType.startsWith(dateTimePrefix) || fieldType.startsWith(datePrefix)) {
return newTimeField(fieldName, isNullable);
} else {
return newStringField(fieldName, isNullable);
}
}
}

function newTimeField(fieldName, isNullable) {
return isNullable ? [] : [];
}

function newFloat64Field(fieldName, isNullable) {
return isNullable ? [] : [];
}

function newStringField(fieldName, isNullable) {
return isNullable ? [] : [];
}

function parseFloatValue(value, isNullable) {
if (value != null) {
const floatValue = parseFloat(value);
return isNullable ? floatValue : floatValue;
}
return isNullable ? null : 0.0;
}

function parseStringValue(value, isNullable) {
if (value != null) {
const stringValue = String(value);
return isNullable ? stringValue : stringValue;
}
return isNullable ? null : "";
}

function parseMapValue(value, isNullable) {
if (value instanceof Object) {
try {
return JSON.stringify(value);
} catch (err) {
return null;
}
}
return isNullable ? null : "";
}

function parseUInt64Value(value, isNullable) {
if (value != null) {
const uint64Value = BigInt(value);
return isNullable ? uint64Value : uint64Value;
}
return isNullable ? null : BigInt(0);
}

function parseInt64Value(value, isNullable) {
if (value != null) {
const int64Value = BigInt(value);
return isNullable ? int64Value : int64Value;
}
return isNullable ? null : BigInt(0);
}

function parseTimestampValue(value, isNullable) {
if (value != null) {
const intValue = BigInt(value);
const timeValue = new Date(Number(intValue));
return isNullable ? timeValue : timeValue;
}
return isNullable ? null : new Date(0);
}

function parseDateTimeValue(value, layout, timezone, isNullable) {
if (value != null) {
const dateTimeValue = new Date(`${value} ${timezone}`);
return isNullable ? dateTimeValue : dateTimeValue;
}
return isNullable ? null : new Date(0);
}

export const parseValue = (fieldName, fieldType, tz, value, isNullable) => {
if (fieldType.startsWith("Nullable")) {
return parseValue(fieldName, fieldType.slice("Nullable(".length, -1), tz, value, true);
} else if (fieldType.startsWith("LowCardinality")) {
return parseValue(fieldName, fieldType.slice("LowCardinality(".length, -1), tz, value, isNullable);
} else if (fieldType.startsWith("Map(") && fieldType.endsWith(")")) {
return parseMapValue(value, isNullable);
} else {
switch (fieldType) {
case "String":
case "UUID":
case "IPv4":
case "IPv6":
return parseStringValue(value, isNullable);
case "UInt8":
case "UInt16":
case "UInt32":
case "Int8":
case "Int16":
case "Int32":
case "Float32":
case "Float64":
return parseFloatValue(value, isNullable);
case "UInt64":
if (fieldName === "t") {
return parseTimestampValue(value, isNullable);
}
return parseUInt64Value(value, isNullable);
case "Int64":
if (fieldName === "t") {
return parseTimestampValue(value, isNullable);
}
return parseInt64Value(value, isNullable);
default:
if (fieldType.startsWith("Decimal")) {
return parseFloatValue(value, isNullable);
} else if (fieldType.startsWith("FixedString") || fieldType.startsWith("Enum")) {
return parseStringValue(value, isNullable);
} else if (fieldType.startsWith(dateTime64Prefix) && fieldType.includes("3")) {
return parseDateTimeValue(value, dateTime64Layout3, tz, isNullable);
} else if (fieldType.startsWith(dateTime64Prefix) && fieldType.includes("6")) {
return parseDateTimeValue(value, dateTime64Layout6, tz, isNullable);
} else if (fieldType.startsWith(dateTimePrefix)) {
return parseDateTimeValue(value, dateTimeLayout, tz, isNullable);
} else if (fieldType.startsWith(datePrefix)) {
return parseDateTimeValue(value, dateLayout, tz, isNullable);
} else {
console.warn(`Value [${value}] has compound type [${fieldType}] and will be returned as string`);

try {
const byteValue = JSON.stringify(value);
return parseStringValue(byteValue, isNullable);
} catch (err) {
console.warn(`Unable to append value of unknown type ${typeof value} because of JSON encoding problem: ${err}`);
return null;
}
}
}
}
}
14 changes: 9 additions & 5 deletions src/backend/helpers/query.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,23 +4,27 @@ import { CHDataSourceOptions } from "../../types/types";
import {logger} from '@grafana/ts-backend';

export const getRequestSettings = (pluginContext: any): any => {
logger.info("getRequestSettings pluginContext", pluginContext);
logger.info("getRequestSettings pluginContext", JSON.stringify( pluginContext?.datasourceinstancesettings));
const jsonData: CHDataSourceOptions = pluginContext?.datasourceinstancesettings?.json

const decryptedSecureJsonData = pluginContext?.
datasourceinstancesettings?.decryptedsecurejsondataMap.reduce((acc: any, [key, value]: [string, string]) => {
acc[key] = value;
return acc;
},{});
return {
Instance: {
URL: pluginContext.datasourceinstancesettings.url,
BasicAuthEnabled: pluginContext.datasourceinstancesettings.basicAuthEnabled,
DecryptedSecureJSONData: pluginContext.datasourceinstancesettings.decryptedSecureJsonData,
DecryptedSecureJSONData: decryptedSecureJsonData,
BasicAuthUser: pluginContext.datasourceinstancesettings.basicAuthUser,
},
UsePost: jsonData.usePOST || false,
UseCompression: jsonData.useCompression || false,
CompressionType: jsonData.compressionType || 'gzip',
UseYandexCloudAuthorization: jsonData.useYandexCloudAuthorization || false,
XHeaderUser: jsonData.xHeaderUser || '',
XHeaderKey: '', // Optional, set as needed
TLSSkipVerify: false, // Set as needed
XHeaderKey: decryptedSecureJsonData.xHeaderKey, // Optional, set as needed
TLSSkipVerify: true, // Set as needed
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wrong default behavior rollback it

};
};
export const createQuery = (options: any, target: any, request: any) => {
Expand Down
4 changes: 0 additions & 4 deletions src/backend/helpers/transform-data.ts
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,6 @@ export const transformData = (inputData: any, refId: string): any => {
const meta = inputData.meta;
const data = inputData.data;

// console.log('meta', meta)

logger.info("transformData inputData", meta)

const fields = meta.map((metaField, index) => {
Expand All @@ -50,12 +48,10 @@ export const transformData = (inputData: any, refId: string): any => {
logger.info('-------', fieldType, fieldName)
const values = data.map(entry => {
if (fieldName === 'time' || fieldName === 't') {
console.log(1)
return new Date(Number(entry[metaField.name]));
}

if (index === 0 && metaField.type === 'UInt64') {
console.log(2)
fieldType = FieldType.time;
}

Expand Down
Loading
Loading