Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

2.4.0 beta (2.3 release) #765

Merged
merged 134 commits into from
May 21, 2024
Merged
Changes from 5 commits
Commits
Show all changes
134 commits
Select commit Hold shift + click to select a range
5ba27d2
Merge pull request #195 from skalenetwork/beta
dmytrotkk May 22, 2020
508993b
Fix exit code in calc version script
dmytrotkk May 25, 2020
2b840ea
Merge pull request #201 from skalenetwork/fix-calc-version-exit-code-…
dmytrotkk May 25, 2020
b8cc5c5
Merge pull request #307 from skalenetwork/beta
dmytrotkk Sep 16, 2020
c317225
Merge pull request #461 from skalenetwork/beta
dmytrotkk Apr 9, 2021
acc08e0
Merge pull request #499 from skalenetwork/beta
dmytrotkk Jun 18, 2021
e8d0c61
Merge pull request #511 from skalenetwork/beta
dmytrotkk Jul 19, 2021
eef39cd
Merge pull request #534 from skalenetwork/beta
dmytrotkk Sep 23, 2021
dae2330
Merge pull request #536 from skalenetwork/beta
dmytrotkk Sep 23, 2021
7bdadfb
Merge pull request #574 from skalenetwork/beta
dmytrotkk Mar 21, 2022
c6636ef
SKALE-779 Full sync node init cmd
dmytrotkk May 4, 2022
a8c037d
SKALE-779 Add schain name to required params
dmytrotkk May 9, 2022
707c85d
SKALE-779 Add sync node allocation
dmytrotkk May 11, 2022
d80cde5
SKALE-779 Fix flake8.
dmytrotkk May 13, 2022
713f32f
SKALE-779 Fix resource allocation tests
dmytrotkk May 13, 2022
96b2097
SKALE-779 Fix resource allocation tests
dmytrotkk May 13, 2022
274d97d
SKALE-779 Remove unused disk_mountpoint
dmytrotkk May 19, 2022
7f9f008
SKALE-779 Add update-sync cmd, add tests, add sync-cli pipeline, upda…
dmytrotkk May 19, 2022
4f0dfd4
SKALE-779 Fix flake8 errors
dmytrotkk May 19, 2022
8d49171
SKALE-779 Fix sync node build
dmytrotkk May 20, 2022
5bacd8b
Merge pull request #593 from skalenetwork/feature/SKALE-779-full-sync…
dmytrotkk May 20, 2022
d3e57f5
SKALE-779 Fix publish pipeline
dmytrotkk May 20, 2022
4ad5d95
Merge pull request #598 from skalenetwork/feature/SKALE-779-full-sync…
dmytrotkk May 20, 2022
88e2eb6
SKALE-779 Fix sync-node update cmd
dmytrotkk May 20, 2022
224bfa1
Merge pull request #599 from skalenetwork/feature/SKALE-779-full-sync…
dmytrotkk May 20, 2022
927f06c
SKALE-779 Add SSL CLI to sync-node-cli
dmytrotkk Jun 1, 2022
cb7c2f2
Merge pull request #602 from skalenetwork/feature/SKALE-779-full-sync…
dmytrotkk Jun 1, 2022
3a799ce
SKALE-779 Restructure SSL commands
dmytrotkk Jun 2, 2022
c9fedfb
SKALE-779 Fix configs import
dmytrotkk Jun 3, 2022
0e21a8d
Fix SSL import
dmytrotkk Jun 3, 2022
2926055
SKLAE-779 Update ssl test
dmytrotkk Jun 3, 2022
2724a3e
SKALE-779 Fix SSL upload
dmytrotkk Jun 3, 2022
21c0e19
SKALE-779 Fix SSL upload
dmytrotkk Jun 3, 2022
e1610bf
SKALE-799 Fix SSL upload
dmytrotkk Jun 6, 2022
f0adc50
SKALE-779 Fix minor issues
dmytrotkk Jun 7, 2022
25cebac
Fix flake
dmytrotkk Jun 7, 2022
d92148d
Merge pull request #603 from skalenetwork/feature/SKALE-779-full-sync…
dmytrotkk Jun 7, 2022
3bdd0eb
Hotfix: add version and build info commands to sync-node-cli
dmytrotkk Jun 8, 2022
bad57e2
Merge pull request #606 from skalenetwork/feature/SKALE-779-full-sync…
dmytrotkk Jun 8, 2022
76596ff
SKALE-779 Add firwall setup for sync node
dmytrotkk Jun 9, 2022
d9f4334
Add cryptography package, update pyOpenSSL
dmytrotkk Jun 9, 2022
1a30cc3
Mock configure_firewall_rules in tests
dmytrotkk Jun 9, 2022
900f669
Merge pull request #607 from skalenetwork/feature/SKALE-779-full-sync…
dmytrotkk Jun 9, 2022
c50c0f1
Hotfix: add nginx reload to ssl upload
dmytrotkk Jun 21, 2022
6110703
Merge pull request #626 from skalenetwork/feature/SKALE-779-full-sync…
dmytrotkk Jun 21, 2022
64c3bdc
Create btrfs volume without lvmpy for syncnode
badrogger Jul 19, 2022
395d9e6
Fix error message for malformed env config
badrogger Jul 21, 2022
08ff20a
Add command to resize block device
badrogger Jul 25, 2022
bb77dec
Improve logs
badrogger Jul 25, 2022
a414a48
Handle missing filesystem
badrogger Jul 28, 2022
1893d65
Merge branch 'beta' into hotfix-5-stable
badrogger Aug 18, 2022
25149bf
Remove redundant function
badrogger Aug 18, 2022
4f2684f
Merge pull request #644 from skalenetwork/hotfix-5-stable
badrogger Aug 19, 2022
1a7d7c0
Merge branch 'stable' into merge-stable-to-sync
badrogger Sep 22, 2022
8a37aa9
Publish builds for beta-sync-node
badrogger Sep 22, 2022
a690b1e
Merge pull request #661 from skalenetwork/merge-stable-to-sync
badrogger Sep 22, 2022
34bc52d
Merge branch 'beta-sync-node' into no-volume-limits
badrogger Sep 23, 2022
c7b3611
Merge pull request #636 from skalenetwork/no-volume-limits
badrogger Sep 23, 2022
79effcc
Fix filestorage mapping
badrogger Sep 24, 2022
0055fd9
Merge pull request #662 from skalenetwork/fix-filestorage-mapping
badrogger Sep 24, 2022
6adb53d
Pass SKALE_STATE_DIR env to docker-compose
badrogger Sep 25, 2022
3a8f8fc
Merge pull request #663 from skalenetwork/fix-filestorage-mapping
badrogger Sep 25, 2022
396a068
Add new sync node options
dmytrotkk Dec 1, 2022
6714841
Update README with new options
dmytrotkk Dec 1, 2022
c46895a
Bump python version
dmytrotkk Dec 1, 2022
329c40b
Downgrade flake8 to 5.0.4
dmytrotkk Dec 1, 2022
8e11f2a
Hotfix: Remove double types
dmytrotkk Dec 2, 2022
65bf278
Bump python version to 3.8
dmytrotkk Dec 2, 2022
adb6b1d
Fix licence header
dmytrotkk Dec 2, 2022
f39e010
Remove sync node from resource allocation
dmytrotkk Dec 2, 2022
e074e11
Remove sync node from tests
dmytrotkk Dec 2, 2022
c8c5830
Update schain_allocation.yml in tests
dmytrotkk Dec 2, 2022
a9c77c2
Minor changes
dmytrotkk Dec 6, 2022
edfcbda
Remove lock file for node options
dmytrotkk Dec 6, 2022
6cd7056
Fix tests
dmytrotkk Dec 6, 2022
5374021
Merge pull request #680 from skalenetwork/enhancement/add-sync-node-o…
dmytrotkk Dec 7, 2022
21f4278
Increase schain stop timeout to 5 minutes
badrogger Dec 8, 2022
58352f5
Fix flake8
badrogger Dec 8, 2022
a90786f
Merge pull request #685 from skalenetwork/increase-timeout-sync-node
dmytrotkk Dec 14, 2022
ccb6fbd
Fix filestorage endpoint
badrogger Jan 19, 2023
b3cdad1
Small change to trigger build
badrogger Jan 19, 2023
5332bb7
Merge pull request #695 from skalenetwork/fix-fs-endpoint
badrogger Jan 19, 2023
4b1b335
Merge pull request #696 from skalenetwork/beta-sync-node
badrogger Jan 23, 2023
cb28d81
Merge branch 'develop' into merge-develop-to-beta-sync-node
badrogger Feb 6, 2023
2d8adcd
Fix tests
badrogger Feb 9, 2023
49ed7be
Merge branch 'develop' into merge-develop-to-beta-sync-node
badrogger Feb 9, 2023
976d6e6
Remove snapshot_from from init/update sync
badrogger Feb 10, 2023
cba08c2
Merge pull request #701 from skalenetwork/merge-develop-to-beta-sync-…
badrogger Feb 24, 2023
742934e
Merge pull request #707 from skalenetwork/beta-sync-node
badrogger Mar 6, 2023
62db939
Merge 2.3.0 to beta-sync-node
badrogger Jun 16, 2023
46348f6
Fix missing VOLUME_GROUP config param
badrogger Jun 16, 2023
411e78f
Rollback to python 3.8 for 18.04 compatible build
badrogger Jun 19, 2023
76bb02f
Add missing sync param to build script intput
badrogger Jun 19, 2023
70271e8
Fix asset name
badrogger Jun 19, 2023
fb7f349
Add missing lvmpy submodule
badrogger Jun 19, 2023
d24cb30
Fix tests
badrogger Jun 19, 2023
776df9f
Merge pull request #728 from skalenetwork/beta-sync-node-2.3.0
badrogger Jun 19, 2023
a63112a
Fix sync node publish pipeline
badrogger Jun 20, 2023
91302b4
Merge pull request #730 from skalenetwork/fix-sync-node-publish
badrogger Jun 20, 2023
819a262
Fix publish.yml home dir path
badrogger Jun 20, 2023
25b2a8d
Merge pull request #731 from skalenetwork/fix-sync-node-publish
badrogger Jun 20, 2023
e635354
Remove prerelease from sync-node based releases
badrogger Jul 7, 2023
2922850
Merge pull request #732 from skalenetwork/remove-prerelease-from-sync
badrogger Jul 7, 2023
7eac870
Fix update-meta invocation
badrogger Jul 8, 2023
7eb2aae
Trigger build
badrogger Jul 8, 2023
34f9f11
Merge pull request #733 from skalenetwork/fix-update-meta-invokation
badrogger Jul 8, 2023
15069ef
Merge branch 'beta-sync-node' into merge-beta-sync-node-to-stable
badrogger Sep 19, 2023
29701ef
Merge pull request #735 from skalenetwork/merge-beta-sync-node-to-stable
badrogger Sep 19, 2023
8cd70a5
Bump GitPython to 3.1.36
badrogger Sep 19, 2023
a08e52b
Merge pull request #736 from skalenetwork/bump-gitpython
badrogger Sep 19, 2023
101bb18
Merge branch 'beta-sync-node' into bump-gitpython-sync-node-stable
badrogger Sep 19, 2023
bc9ff1c
Merge pull request #737 from skalenetwork/bump-gitpython-sync-node-st…
badrogger Sep 19, 2023
94e41f3
Merge sync-node CLI to develop
dmytrotkk Jan 24, 2024
a968b10
Merge sync-node CLI to develop - fix flake8
dmytrotkk Jan 24, 2024
7921428
Update publish pipeline - remove sync-node branches
dmytrotkk Jan 24, 2024
d6bbb34
Fix test pipeline - update binary check
dmytrotkk Jan 24, 2024
1575119
Update volume module - use regular sChain mountpoint
dmytrotkk Jan 24, 2024
fe6e479
Update test and publish pipelines
dmytrotkk Jan 25, 2024
74e1028
Update build and publish pipeline - fix mkdir
dmytrotkk Jan 25, 2024
ff59242
Minor changes - remove IMA_ENDPOINT, merge compose_up_sync
dmytrotkk Jan 26, 2024
2bb429a
Minor fix - update wallet module
dmytrotkk Jan 26, 2024
9b11594
Minor update - add DISABLE_IMA to sync node
dmytrotkk Jan 29, 2024
c78155a
Minor update - remove DISABLE_IMA from optional params
dmytrotkk Jan 29, 2024
ebb42a5
Remove DISABLE_IMA variable from sync node
dmytrotkk Jan 31, 2024
0699d42
Remove CONTAINER_CONFIGS_DIR from required params
dmytrotkk Feb 7, 2024
6c3d29c
Merge pull request #748 from skalenetwork/merge-sync-node-to-develop
dmytrotkk Mar 19, 2024
8874270
Fix build
badrogger Apr 10, 2024
fc633c1
Add optional telegraf service setup step
badrogger Apr 11, 2024
5a3a4f6
Add missing INFLUX_URL parameter
badrogger Apr 11, 2024
c828ee8
Remove INFLUX_ORG and INFLUX_BUCKET
badrogger Apr 18, 2024
fd0af66
Leave only removing telegraf during update/turn-off
badrogger Apr 23, 2024
c46f60b
Update GitPython to 3.1.41
badrogger May 14, 2024
7f1cb81
Merge pull request #750 from skalenetwork/add-telegraf
badrogger May 15, 2024
2fe74ca
Merge pull request #752 from skalenetwork/v2.4.x
DmytroNazarenko May 17, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion node_cli/cli/__init__.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
__version__ = '2.2.1'


if __name__ == "__main__":
print(__version__)
7 changes: 6 additions & 1 deletion node_cli/cli/sync_node.py
Original file line number Diff line number Diff line change
@@ -20,7 +20,12 @@
import click

from node_cli.core.node import init_sync, update_sync
from node_cli.utils.helper import abort_if_false, safe_load_texts, streamed_cmd, error_exit
from node_cli.utils.helper import (
abort_if_false,
safe_load_texts,
streamed_cmd,
error_exit
)
from node_cli.utils.exit_codes import CLIExitCodes


4 changes: 2 additions & 2 deletions node_cli/configs/__init__.py
Original file line number Diff line number Diff line change
@@ -53,8 +53,8 @@

COMPOSE_PATH = os.path.join(CONTAINER_CONFIG_PATH, 'docker-compose.yml')
SYNC_COMPOSE_PATH = os.path.join(CONTAINER_CONFIG_PATH, 'docker-compose-sync.yml')
ENVIRONMENT_PARAMS_FILEPATH = os.path.join(
CONTAINER_CONFIG_PATH, 'environment_params.yaml')
STATIC_PARAMS_FILEPATH = os.path.join(CONTAINER_CONFIG_PATH, 'static_params.yaml')

NGINX_TEMPLATE_FILEPATH = os.path.join(CONTAINER_CONFIG_PATH, 'nginx.conf.j2')
NGINX_CONFIG_FILEPATH = os.path.join(NODE_DATA_PATH, 'nginx.conf')
NGINX_CONTAINER_NAME = 'skale_nginx'
1 change: 0 additions & 1 deletion node_cli/core/node.py
Original file line number Diff line number Diff line change
@@ -164,7 +164,6 @@ def init_sync(
archive,
catchup,
historic_state

)
if not inited_ok:
error_exit(
1 change: 1 addition & 0 deletions node_cli/main.py
Original file line number Diff line number Diff line change
@@ -86,6 +86,7 @@ def get_sources_list():
logs_cli,
resources_allocation_cli,
node_cli,
sync_node_cli,
wallet_cli,
ssl_cli,
exit_cli,
15 changes: 7 additions & 8 deletions node_cli/operations/base.py
Original file line number Diff line number Diff line change
@@ -19,7 +19,7 @@

import functools
import logging
from typing import Dict, Optional
from typing import Dict

from node_cli.cli.info import VERSION
from node_cli.configs import CONTAINER_CONFIG_PATH, CONTAINER_CONFIG_TMP_PATH
@@ -29,11 +29,13 @@
from node_cli.core.nginx import generate_nginx_config
from node_cli.core.node_options import NodeOptions
from node_cli.core.resources import update_resource_allocation, init_shared_space_volume
from node_cli.core.node_options import NodeOptions

from node_cli.operations.common import (
backup_old_contracts, download_contracts, configure_filebeat,
configure_flask, unpack_backup_archive
backup_old_contracts,
download_contracts,
configure_filebeat,
configure_flask,
unpack_backup_archive
)
from node_cli.operations.volume import (
cleanup_volume_artifacts,
@@ -95,7 +97,7 @@ def wrapper(env_filepath: str, env: Dict, *args, **kwargs):


@checked_host
def update(env_filepath: str, env: dict) -> bool:
def update(env_filepath: str, env: Dict) -> None:
compose_rm(env)
remove_dynamic_containers()

@@ -162,9 +164,6 @@ def init(env_filepath: str, env: dict) -> bool:
docker_lvmpy_install(env)
init_shared_space_volume(env['ENV_TYPE'])

node_options = NodeOptions()
node_options.snapshot_from = snapshot_from

update_meta(
VERSION,
env['CONTAINER_CONFIGS_STREAM'],
11 changes: 8 additions & 3 deletions node_cli/operations/common.py
Original file line number Diff line number Diff line change
@@ -29,11 +29,16 @@
from distutils.dir_util import copy_tree

from node_cli.configs import (
CONTRACTS_PATH, BACKUP_CONTRACTS_PATH, MANAGER_CONTRACTS_FILEPATH, IMA_CONTRACTS_FILEPATH,
SRC_FILEBEAT_CONFIG_PATH, G_CONF_HOME, FILEBEAT_CONFIG_PATH, FLASK_SECRET_KEY_FILE
CONTRACTS_PATH,
BACKUP_CONTRACTS_PATH,
G_CONF_HOME,
FILEBEAT_CONFIG_PATH,
FLASK_SECRET_KEY_FILE,
IMA_CONTRACTS_FILEPATH,
MANAGER_CONTRACTS_FILEPATH,
SRC_FILEBEAT_CONFIG_PATH
)


logger = logging.getLogger(__name__)


30 changes: 30 additions & 0 deletions node_cli/utils/helper.py
Original file line number Diff line number Diff line change
@@ -370,3 +370,33 @@ def ok_result(payload: dict = None):

def err_result(msg: str = None):
return 'error', msg


class UrlType(click.ParamType):
name = 'url'

def convert(self, value, param, ctx):
try:
result = urlparse(value)
except ValueError:
self.fail(f'Some characters are not allowed in {value}',
param, ctx)
if not all([result.scheme, result.netloc]):
self.fail(f'Expected valid url. Got {value}', param, ctx)
return value


class IpType(click.ParamType):
name = 'ip'

def convert(self, value, param, ctx):
try:
ipaddress.ip_address(value)
except ValueError:
self.fail(f'expected valid ipv4/ipv6 address. Got {value}',
param, ctx)
return value


URL_TYPE = UrlType()
IP_TYPE = IpType()
15 changes: 8 additions & 7 deletions setup.py
Original file line number Diff line number Diff line change
@@ -21,15 +21,15 @@ def find_version(*file_paths):
extras_require = {
'linter': [
"flake8==5.0.4",
"isort>=4.2.15,<5.8.1",
"isort>=4.2.15,<5.10.2",
],
'dev': [
"bumpversion==0.6.0",
"pytest==7.2.0",
"pytest-cov==2.9.0",
"twine==2.0.0",
"mock==4.0.2",
"freezegun==0.3.15"
"pytest==7.1.2",
"pytest-cov==3.0.0",
"twine==4.0.1",
"mock==4.0.3",
"freezegun==1.2.2"
]
}

@@ -64,8 +64,9 @@ def find_version(*file_paths):
"packaging==21.3",
"python-debian==0.1.48",
"python-iptables==1.0.0",
"MarkupSafe==2.0.1",
"PyYAML==6.0",
"pyOpenSSL==22.0.0",
"MarkupSafe==2.1.1",
"cryptography==37.0.2",
"filelock==3.0.12"
],
53 changes: 49 additions & 4 deletions tests/conftest.py
Original file line number Diff line number Diff line change
@@ -29,14 +29,21 @@
import yaml

from node_cli.configs import (
ENVIRONMENT_PARAMS_FILEPATH, GLOBAL_SKALE_DIR, GLOBAL_SKALE_CONF_FILEPATH,
REMOVED_CONTAINERS_FOLDER_PATH, NGINX_CONTAINER_NAME
CONTAINER_CONFIG_TMP_PATH,
GLOBAL_SKALE_CONF_FILEPATH,
GLOBAL_SKALE_DIR,
META_FILEPATH,
NGINX_CONTAINER_NAME,
REMOVED_CONTAINERS_FOLDER_PATH,
STATIC_PARAMS_FILEPATH
)
from node_cli.configs.node_options import NODE_OPTIONS_FILEPATH
from node_cli.utils.global_config import generate_g_config_file
from node_cli.configs.resource_allocation import RESOURCE_ALLOCATION_FILEPATH
from node_cli.configs.ssl import SSL_FOLDER_PATH
from node_cli.configs.resource_allocation import RESOURCE_ALLOCATION_FILEPATH
from node_cli.utils.docker_utils import docker_client
from node_cli.utils.global_config import generate_g_config_file

from tests.helper import TEST_META_V1, TEST_META_V2


TEST_ENV_PARAMS = """
@@ -251,3 +258,41 @@ def nginx_container(dutils, ssl_folder):
c.remove(force=True)
except Exception:
pass


@pytest.fixture
def meta_file_v1():
with open(META_FILEPATH, 'w') as f:
json.dump(TEST_META_V1, f)
try:
yield META_FILEPATH
finally:
os.remove(META_FILEPATH)


@pytest.fixture
def meta_file_v2():
with open(META_FILEPATH, 'w') as f:
json.dump(TEST_META_V2, f)
try:
yield META_FILEPATH
finally:
os.remove(META_FILEPATH)


@pytest.fixture
def ensure_meta_removed():
try:
yield
finally:
if os.path.isfile(META_FILEPATH):
os.remove(META_FILEPATH)


@pytest.fixture
def tmp_config_dir():
os.mkdir(CONTAINER_CONFIG_TMP_PATH)
try:
yield CONTAINER_CONFIG_TMP_PATH
finally:
shutil.rmtree(CONTAINER_CONFIG_TMP_PATH)
12 changes: 12 additions & 0 deletions tests/helper.py
Original file line number Diff line number Diff line change
@@ -25,6 +25,18 @@

BLOCK_DEVICE = os.getenv('BLOCK_DEVICE')

TEST_META_V1 = {
'version': '0.1.1',
'config_stream': 'develop'
}

TEST_META_V2 = {
'version': '0.1.1',
'config_stream': 'develop',
'docker_lvmpy_stream': '1.1.2'

}


def response_mock(
status_code=0,