Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: use composite builds to speed up compilation #692

Open
wants to merge 13 commits into
base: develop
Choose a base branch
from
2 changes: 1 addition & 1 deletion .eslintrc.js
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ module.exports = {
parserOptions: {
ecmaVersion: 2018,
sourceType: 'module',
project: './tsconfig.json',
project: './tsconfig.base.json',
},
plugins: [
'@typescript-eslint',
Expand Down
5 changes: 2 additions & 3 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -29,13 +29,12 @@ lib
.env.test.local
.env.production.local
settings.json
tsconfig.tsbuildinfo

# doc
**/docs/api
/docs/api

# misc
/.idea
.vscode

# doc
/docs/api
16 changes: 8 additions & 8 deletions babel.config.json
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
{
"presets": [
[
"@babel/preset-env",
{
"modules": "commonjs"
}
]
"presets": [
[
"@babel/preset-env",
{
"modules": "commonjs"
}
]
}
]
}
19 changes: 11 additions & 8 deletions jest.config.js
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,9 @@ module.exports = {
// Parachain block time is 12s
testTimeout: 15000,
setupFilesAfterEnv: ['../jest-setup/setup.js'],
transformIgnorePatterns: ['/node_modules/(?!@polkadot|@babel/runtime/helpers/esm/)'],
transformIgnorePatterns: [
'/node_modules/(?!@polkadot|@babel/runtime/helpers/esm/)',
],
coverageThreshold: {
global: {
branches: 70,
Expand All @@ -16,8 +18,8 @@ module.exports = {
},
},
transform: {
"\\.js$": "babel-jest",
"\\.ts$": "ts-jest"
'\\.js$': 'babel-jest',
'\\.ts$': 'ts-jest',
},
collectCoverageFrom: [
'**/*/src/**/*.ts',
Expand All @@ -40,11 +42,12 @@ module.exports = {
'!did/src/Did.utils.ts',
'!utils/src/jsonabc.ts',
],
resolver: "ts-jest-resolver",
resolver: 'ts-jest-resolver',
rootDir: 'packages',
coverageDirectory: 'coverage',
moduleDirectories: [
"node_modules",
"packages/*/src"
]
globals: {
'ts-jest': {
tsconfig: 'tsconfig.base.json',
},
},
}
9 changes: 6 additions & 3 deletions package.json
Original file line number Diff line number Diff line change
@@ -1,17 +1,20 @@
{
"repository": "https://github.com/kiltprotocol/sdk-js",
"name": "root-workspace",
"private": true,
"workspaces": [
"packages/*",
"docs/*"
],
"license": "BSD-4-Clause",
"scripts": {
"check": "tsc -p tsconfig.json --noEmit",
"build": "yarn workspaces foreach -p -t --exclude '{root-workspace}' run build",
"check": "tsc -b tsconfig.json",
"build": "yarn clean && yarn workspaces foreach -pt --exclude 'root-workspace' run build",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we run yarn clean before a build, we lose the incremental compilation benefit you mention in the PR description, I guess. So we should probably not clean by default.

Copy link
Contributor Author

@rflechtner rflechtner Nov 30, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is by design; yarn build is only meant to be run for production builds essentially (it's also the command run in CI). Running a clean beforehand makes sure we don't get any weird artefacts. For standard builds and rebuilds in development you can simply use build:dev.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As mentioned below, we should then somehow advertise that build:dev is the to-go command then.

"build:dev": "yarn workspaces foreach -pt --exclude 'root-workspace' run build:cjs",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess this is to be used before unit/integration tests? Can we mention it somewhere? Otherwise I am 99% sure people will still be running yarn build.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, I haven't updated READMEs and such. Could also add comments above the respective scripts

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay so we don't have a section on development in the README and we can't add comments to the package.json, where do you think we should put that info?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess the README then? I don't see any other place where this could go, personally. Maybe @arty-name has some prior experience for this?

"build:watch": "yarn build:dev && tsc -b tsconfig.json --watch",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the benefit of this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It complains as soon as you break something and is more efficient as running yarn check or yarn build:dev repeatedly. Also I would have to double check but potentially you need to have it running if you are using jest in watch mode.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just checked, you would want to have this running as well if jest is running in watch mode to make sure packages are being rebuild on changes. Otherwise imports from another package may use out of date versions.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this is supposed to be run before and during tests? And yarn build:dev alone then when?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, this is for watch mode. As in, you keep jest running and it re-executes tests whenever you change a file. So far we didn't have a solution for that. You can also just run yarn build:dev && yarn test whenever you want to test of course. It's a question of preference.

As for the naming, I was considering making yarn build:dev the new yarn build and introducing a yarn build:production instead for release builds. Do you think that's better? I decided to go the other way because there is no harm in running the production build in a development scenario (it's just slower) but it would be bad to release a development build because the esm files are missing.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes sense. Rather use the build for development and something else for production.

"build:docs": "typedoc --theme default --out docs/api --tsconfig tsconfig.docs.json && touch docs/.nojekyll",
"bundle": "yarn workspace @kiltprotocol/sdk-js run bundle",
"clean": "rimraf tests/dist && yarn workspaces foreach -p --exclude '{root-workspace}' run clean",
"clean": "rimraf tests/dist && yarn workspaces foreach -p --exclude 'root-workspace' run clean",
"clean:docs": "rimraf docs/api",
"prepublish": "yarn workspaces foreach -p --no-private exec cp -f ../../LICENSE .",
"publish": "yarn workspaces foreach -pt --no-private npm publish",
Expand Down
15 changes: 10 additions & 5 deletions packages/asset-did/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -12,14 +12,15 @@
}
},
"files": [
"lib/**/*"
"lib/**/*.js",
"lib/**/*.d.ts",
"lib/**/*.json"
],
"scripts": {
"clean": "rimraf ./lib",
"build": "yarn clean && yarn build:ts",
"build:ts": "yarn build:cjs && yarn build:esm",
"build:cjs": "tsc --declaration -p tsconfig.build.json && echo '{\"type\":\"commonjs\"}' > ./lib/cjs/package.json",
"build:esm": "tsc --declaration -p tsconfig.esm.json && echo '{\"type\":\"module\"}' > ./lib/esm/package.json"
"build": "yarn build:cjs && yarn build:esm",
"build:cjs": "tsc -b && echo '{\"type\":\"commonjs\"}' > ./lib/cjs/package.json",
"build:esm": "tsc -p tsconfig.esm.json && echo '{\"type\":\"module\"}' > ./lib/esm/package.json"
},
"repository": "github:kiltprotocol/sdk-js",
"engines": {
Expand All @@ -32,5 +33,9 @@
"dependencies": {
"@kiltprotocol/types": "workspace:*",
"@kiltprotocol/utils": "workspace:*"
},
"devDependencies": {
"rimraf": "^3.0.2",
"typescript": "^4.8.3"
}
}
17 changes: 0 additions & 17 deletions packages/asset-did/tsconfig.build.json

This file was deleted.

6 changes: 4 additions & 2 deletions packages/asset-did/tsconfig.esm.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
{
"extends": "./tsconfig.build.json",
"extends": "./tsconfig.json",
"compilerOptions": {
"module": "ES6",
"outDir": "./lib/esm"
"outDir": "./lib/esm",
"declaration": false,
"composite": false
}
}
18 changes: 18 additions & 0 deletions packages/asset-did/tsconfig.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
{
"extends": "../../tsconfig.base.json",
"compilerOptions": {
"outDir": "./lib/cjs",
"rootDir": "./src"
},
"include": ["src/**/*.ts", "src/**/*.js"],
"exclude": ["coverage", "**/*.spec.ts"],
// indicates package dependencies; must be in sync with @kiltprotocol imports in package.json
"references": [
{
"path": "../types"
},
{
"path": "../utils"
}
]
}
11 changes: 6 additions & 5 deletions packages/augment-api/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -13,16 +13,17 @@
}
},
"files": [
"lib/**/*"
"lib/**/*.js",
"lib/**/*.d.ts",
"lib/**/*.json"
],
"scripts": {
"clean": "rimraf ./lib",
"build": "yarn clean && yarn build:ts",
"build": "yarn build:cjs && yarn build:esm",
"build:cjs": "tsc -b tsconfig.json && echo '{\"type\":\"commonjs\"}' > ./lib/cjs/package.json",
"build:esm": "tsc -p tsconfig.esm.json && echo '{\"type\":\"module\"}' > ./lib/esm/package.json",
"build:types": "yarn generate:defs && yarn generate:meta && yarn build:fixes",
"build:fixes": "node scripts/fixTypes.mjs",
"build:ts": "yarn build:cjs && yarn build:esm",
"build:cjs": "tsc --declaration -p tsconfig.build.json && echo '{\"type\":\"commonjs\"}' > ./lib/cjs/package.json",
"build:esm": "tsc --declaration -p tsconfig.esm.json && echo '{\"type\":\"module\"}' > ./lib/esm/package.json",
"generate:defs": "ts-node --skip-project ../../node_modules/.bin/polkadot-types-from-defs --package @kiltprotocol/augment-api --input ./src/interfaces --endpoint ./metadata/spiritnet.json",
"generate:meta": "ts-node --skip-project ../../node_modules/.bin/polkadot-types-from-chain --package @kiltprotocol/augment-api --endpoint ./metadata/spiritnet.json --output ./src/interfaces --strict",
"update-metadata": "node ./scripts/fetchMetadata.js -o './metadata/spiritnet.json' -e 'wss://spiritnet.kilt.io/'"
Expand Down
29 changes: 0 additions & 29 deletions packages/augment-api/tsconfig.build.json

This file was deleted.

6 changes: 4 additions & 2 deletions packages/augment-api/tsconfig.esm.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
{
"extends": "./tsconfig.build.json",
"extends": "./tsconfig.json",
"compilerOptions": {
"module": "ES6",
"outDir": "./lib/esm"
"outDir": "./lib/esm",
"declaration": false,
"composite": false
}
}
22 changes: 22 additions & 0 deletions packages/augment-api/tsconfig.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
{
"extends": "../../tsconfig.base.json",
"compilerOptions": {
"outDir": "./lib/cjs",
"rootDir": "./src",
"skipLibCheck": true,
"noUnusedLocals": false,
"paths": {
"@kiltprotocol/augment-api/*": ["./src/*"],
"@polkadot/api/augment": ["./src/interfaces/augment-api.ts"],
"@polkadot/types/augment": ["./src/interfaces/augment-types.ts"]
}
},
"include": ["src/**/*.ts", "src/**/*.js"],
"exclude": ["coverage", "**/*.spec.ts", "src/**/definitions.ts"],
// indicates package dependencies; must be in sync with @kiltprotocol imports in package.json
"references": [
{
"path": "../type-definitions"
}
]
}
11 changes: 6 additions & 5 deletions packages/chain-helpers/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -12,14 +12,15 @@
}
},
"files": [
"lib/**/*"
"lib/**/*.js",
"lib/**/*.d.ts",
"lib/**/*.json"
],
"scripts": {
"clean": "rimraf ./lib",
"build": "yarn clean && yarn build:ts",
"build:ts": "yarn build:cjs && yarn build:esm",
"build:cjs": "tsc --declaration -p tsconfig.build.json && echo '{\"type\":\"commonjs\"}' > ./lib/cjs/package.json",
"build:esm": "tsc --declaration -p tsconfig.esm.json && echo '{\"type\":\"module\"}' > ./lib/esm/package.json"
"build": "yarn build:cjs && yarn build:esm",
"build:cjs": "tsc -b tsconfig.json && echo '{\"type\":\"commonjs\"}' > ./lib/cjs/package.json",
"build:esm": "tsc -p tsconfig.esm.json && echo '{\"type\":\"module\"}' > ./lib/esm/package.json"
},
"repository": "github:kiltprotocol/sdk-js",
"engines": {
Expand Down
16 changes: 0 additions & 16 deletions packages/chain-helpers/tsconfig.build.json

This file was deleted.

6 changes: 4 additions & 2 deletions packages/chain-helpers/tsconfig.esm.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
{
"extends": "./tsconfig.build.json",
"extends": "./tsconfig.json",
"compilerOptions": {
"module": "ES6",
"outDir": "./lib/esm"
"outDir": "./lib/esm",
"declaration": false,
"composite": false
}
}
15 changes: 15 additions & 0 deletions packages/chain-helpers/tsconfig.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
{
"extends": "../../tsconfig.base.json",
"compilerOptions": {
"outDir": "./lib/cjs",
"rootDir": "./src"
},
"include": ["src/**/*.ts", "src/**/*.js"],
"exclude": ["coverage", "**/*.spec.ts"],
// indicates package dependencies; must be in sync with @kiltprotocol imports in package.json
"references": [
{ "path": "../config" },
{ "path": "../types" },
{ "path": "../utils" }
]
}
11 changes: 6 additions & 5 deletions packages/config/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -12,14 +12,15 @@
}
},
"files": [
"lib/**/*"
"lib/**/*.js",
"lib/**/*.d.ts",
"lib/**/*.json"
],
"scripts": {
"clean": "rimraf ./lib",
"build": "yarn clean && yarn build:ts",
"build:ts": "yarn build:cjs && yarn build:esm",
"build:cjs": "tsc --declaration -p tsconfig.build.json && echo '{\"type\":\"commonjs\"}' > ./lib/cjs/package.json",
"build:esm": "tsc --declaration -p tsconfig.esm.json && echo '{\"type\":\"module\"}' > ./lib/esm/package.json"
"build": "yarn build:cjs && yarn build:esm",
"build:cjs": "tsc -b tsconfig.json && echo '{\"type\":\"commonjs\"}' > ./lib/cjs/package.json",
"build:esm": "tsc -p tsconfig.esm.json && echo '{\"type\":\"module\"}' > ./lib/esm/package.json"
},
"repository": "github:kiltprotocol/sdk-js",
"engines": {
Expand Down
17 changes: 0 additions & 17 deletions packages/config/tsconfig.build.json

This file was deleted.

6 changes: 4 additions & 2 deletions packages/config/tsconfig.esm.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
{
"extends": "./tsconfig.build.json",
"extends": "./tsconfig.json",
"compilerOptions": {
"module": "ES6",
"outDir": "./lib/esm"
"outDir": "./lib/esm",
"declaration": false,
"composite": false
}
}
Loading