-
Notifications
You must be signed in to change notification settings - Fork 11
Swarm and Nomad support #62
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
sfondev
wants to merge
65
commits into
sr-lab:interactive_repair
Choose a base branch
from
sfondev:swarm-and-nomad-parsers
base: interactive_repair
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 36 commits
Commits
Show all changes
65 commits
Select commit
Hold shift + click to select a range
fb8afb7
add swarm and nomad parse exceptions
sfondev 0098d97
add Nomad parser
sfondev ae6b6b1
add Swarm parser
sfondev 5a1e69b
fix incorrect exception name
sfondev 4c0178c
format swarm.py
sfondev 10dd8e6
change Terraform parser get_element_info method visibility to protect…
sfondev 602191f
changed the Transformer for Nomad to be a new class that extends the …
sfondev 0adf8ae
format nomad.py
sfondev f5089f4
format nomad.py and fix missing abs file path issue
sfondev 759ff54
fixes swarm.py adds support for 'extends' and 'includes' extension
sfondev 899368c
fix docker_images_scraper.py: add support for the new search api and …
sfondev 9680335
remove forgotten print 😅
sfondev 7031412
fix boolean parsing
sfondev c32defa
swarm and nomad tech introduction
sfondev 4cd8574
fix add more common tar compressed with bzip2 extension, also for zst…
sfondev 602602b
adding the smells error messages
sfondev 808439c
swarm and nomad parsers fixes
sfondev 4d2fbd9
correctly escape the '%' character for copying into latex
sfondev 5c67131
adding implementation of Log Collection and some of container image s…
sfondev 253dc45
add deprecated official images smell
sfondev e178edc
added files needed for log collector/log aggregator smell
sfondev 7ad6a8d
currently useless file, will remove later
sfondev d7177d4
currently useless file, will remove later
sfondev 1e19b3d
added file needed for the unstable image smell
sfondev 3a49007
adding no api gateway smell files
sfondev 94712dc
adding priveleged containers, mounted docker socket and Nomad integri…
sfondev 8a24153
adding multiple services per deployment unit and missing healthcheck …
sfondev f804267
fix wrong version of non official image smell detection
sfondev 757d8c8
fix alternative ways of referring to an official image from Docker Hub
sfondev f8765a3
fix logic error and always make tag lower, since it can be mixed or c…
sfondev e9671bd
fix small parser mistake
sfondev 85ee60a
try to fix self include when extending from another file
sfondev 3f57889
format + extends should be fixed now...
sfondev 27b9f6b
remove unused stuff
sfondev 9a683b9
fix uniformize use of container image string element instead of whole…
sfondev 060f0e5
fix rename error key
sfondev e7cfd7c
removed unused commented code
sfondev 948e374
removed parsing of included files, was not a correct approach for sca…
sfondev 663556d
rename class and add missing detection
sfondev 52d225c
rename file
sfondev 85a8fe9
fix typo in class name
sfondev ded778e
moved the detection of nomad no integrity check together with the others
sfondev 0594562
fix: move smell checkers out of visitor.py
sfondev 5b6cd01
move container image name parser util to separated file
sfondev 8bd9632
format swarm.py
sfondev ae48806
fix matching invalid ip bindings in complete command invocations, by …
sfondev 0ade21c
fix typo
sfondev f8ce625
remove hack, fix variable parsing in swarm
sfondev 7ceb362
change to use StringChecker and add handling of arrays and hashes
sfondev 365c906
fix empty strings in env var value
sfondev aef512b
fix some of SecurityVisitor global constants not being initialized fo…
sfondev f1ccef5
add: swarm tests
sfondev 578bb59
add missing __init__.py needed for test detection
sfondev 9d12ba3
fix filenames of official docker images and deprecated official image…
sfondev e6150d4
update official docker images list
sfondev cb0e621
remove unnecessary method
sfondev 200e9dc
remove unused import
sfondev febd84d
fix: consider both list of official images. The newer list of officia…
sfondev 374c39f
format visitor.py
sfondev 8a5f19e
fix missing healthchecks smell and wobbly service interaction on nomad
sfondev fa902f5
add nomad tests
sfondev 4ee287e
fix: add hack to avoid using detection function tailored to nomad for…
sfondev 9835c68
fix name of nomad task config logging driver attribute
sfondev 7caa60e
hack: support multiple artifacts missing integrity checks in Nomad
sfondev 6d9cad3
fix: pinned container image without digest smell
sfondev File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,46 @@ | ||
| from glitch.analysis.rules import Error | ||
| from glitch.analysis.security.smell_checker import SecuritySmellChecker | ||
| from glitch.analysis.security.visitor import SecurityVisitor | ||
| from glitch.repr.inter import CodeElement, KeyValue, Hash, String | ||
| from typing import List | ||
|
|
||
|
|
||
| class DeprecatedOfficialDockerImages(SecuritySmellChecker): | ||
| def check(self, element: CodeElement, file: str) -> List[Error]: | ||
| errors: List[Error] = [] | ||
| image = "" | ||
| bad_element = element | ||
| if isinstance(element, KeyValue) and element.name == "image": | ||
| if isinstance(element.value, String): | ||
| image = element.value.value | ||
| elif ( | ||
| isinstance(element, KeyValue) | ||
| and element.name == "config" | ||
| and isinstance(element.value, Hash) | ||
| ): | ||
| for k, v in element.value.value.items(): | ||
| if isinstance(k, String) and k.value == "image": | ||
| image = v.value | ||
| bad_element = v | ||
| break | ||
| if image != "": | ||
| img_name, _, _ = SecurityVisitor.image_parser(image) | ||
| for obsolete_img in SecurityVisitor.DEPRECATED_OFFICIAL_DOCKER_IMAGES: | ||
| obsolete_img_dockerio = f"docker.io/library/{obsolete_img}" | ||
| obsolete_img_library = f"library/{obsolete_img}" | ||
| obsolete_img_complete_link = ( | ||
| f"registry.hub.docker.com/library/{obsolete_img}" | ||
| ) | ||
|
|
||
| if ( | ||
| img_name == obsolete_img | ||
| or img_name == obsolete_img_dockerio | ||
| or img_name == obsolete_img_library | ||
| or img_name == obsolete_img_complete_link | ||
| ): | ||
| errors.append( | ||
| Error("sec_depr_off_imgs", bad_element, file, repr(bad_element)) | ||
| ) | ||
| break | ||
|
|
||
| return errors |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,46 @@ | ||
| from glitch.analysis.rules import Error | ||
| from glitch.analysis.security.smell_checker import SecuritySmellChecker | ||
| from glitch.repr.inter import CodeElement, KeyValue, Hash, String,Array | ||
| from typing import List | ||
|
|
||
|
|
||
| class DockerSocketMountedInsideContainerUse(SecuritySmellChecker): | ||
| def check(self, element: CodeElement, file: str) -> List[Error]: | ||
| errors: List[Error] = [] | ||
| if isinstance(element, KeyValue): | ||
| if element.name == "volumes" and isinstance(element.value, Array): | ||
| for volume in element.value.value: | ||
| if isinstance(volume, String) and volume.value.split(":")[ | ||
| 0 | ||
| ].startswith("/var/run/docker.sock"): | ||
| errors.append( | ||
| Error( | ||
| "sec_mounted_docker_socket", volume, file, repr(volume) | ||
| ) | ||
| ) | ||
| break | ||
| elif element.name == "config" and isinstance(element.value, Hash): | ||
| found_socket_exposed = False | ||
| for k, v in element.value.value.items(): | ||
| if ( | ||
| isinstance(k, String) | ||
| and k.value == "volumes" | ||
| and isinstance(v, Array) | ||
| ): | ||
| for volume in v.value: | ||
| if isinstance(volume, String) and volume.value.split(":")[ | ||
| 0 | ||
| ].startswith("/var/run/docker.sock"): | ||
| errors.append( | ||
| Error( | ||
| "sec_mounted_docker_socket", | ||
| volume, | ||
| file, | ||
| repr(volume), | ||
| ) | ||
| ) | ||
| found_socket_exposed = True | ||
| break | ||
| if found_socket_exposed: | ||
| break | ||
| return errors |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
43 changes: 43 additions & 0 deletions
43
glitch/analysis/security/multiple_services_per_deplyment_unit.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,43 @@ | ||
| from glitch.analysis.rules import Error | ||
| from glitch.analysis.security.smell_checker import SecuritySmellChecker | ||
| from glitch.analysis.security.visitor import SecurityVisitor | ||
| from glitch.repr.inter import CodeElement, Hash, String, UnitBlock, UnitBlockType | ||
| from typing import List | ||
|
|
||
|
|
||
| class MultipleServicesPerDeploymentUnitCheck(SecuritySmellChecker): | ||
| def check(self, element: CodeElement, file: str) -> List[Error]: | ||
| # FIXME: Besides log collectors, there are other types of agents/sidecars for observability (some of which are also on the log collector list) | ||
| # and proxies which should also be allowed besides the main microservice | ||
| errors: List[Error] = [] | ||
| if isinstance(element, UnitBlock) and element.type == UnitBlockType.block: | ||
| main_service_found = False | ||
| for au in element.atomic_units: | ||
| if au.type in ["task.docker", "task.podman"]: | ||
| image_name = "" | ||
| for att in au.attributes: | ||
| if att.name == "config" and isinstance(att.value, Hash): | ||
| for k, v in att.value.value.items(): | ||
| if isinstance(k, String) and k.value == "image": | ||
| image_name, _, _ = SecurityVisitor.image_parser( | ||
| v.value | ||
| ) | ||
| break | ||
| if image_name != "": | ||
| break | ||
|
|
||
| if image_name in SecurityVisitor.LOG_AGGREGATORS_AND_COLLECTORS: | ||
| continue | ||
|
|
||
| elif main_service_found: | ||
| errors.append( | ||
| Error("arc_multiple_services", au, file, repr(au)) | ||
| ) | ||
| else: | ||
| main_service_found = True | ||
| elif main_service_found: | ||
| # when there are other types of tasks that aren't docker or podman based | ||
| # and one that is likely the main microservice has already been found | ||
| errors.append(Error("arc_multiple_services", au, file, repr(au))) | ||
|
|
||
| return errors |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,127 @@ | ||
| from glitch.analysis.rules import Error | ||
| from glitch.analysis.security.smell_checker import SecuritySmellChecker | ||
| from glitch.analysis.security.visitor import SecurityVisitor | ||
| from glitch.repr.inter import ( | ||
| CodeElement, | ||
| Hash, | ||
| Array, | ||
| VariableReference, | ||
| String, | ||
| UnitBlock, | ||
| UnitBlockType, | ||
| ) | ||
| from typing import List, Dict, Any | ||
|
|
||
|
|
||
| class NoAPIGatewayCheck(SecuritySmellChecker): | ||
| def check(self, element: CodeElement, file: str) -> List[Error]: | ||
| errors: List[Error] = [] | ||
| # Tries to follow an logic similar to the one presented for Kubernetes pods ond doi: 10.5220/0011845500003488 | ||
|
|
||
| if isinstance(element, UnitBlock) and element.type == UnitBlockType.block: | ||
| has_api_gateway = False | ||
|
|
||
| network_info: Dict[str, Any] = { | ||
| "mode": "bridge", # default network mode | ||
| "ports": [], | ||
| } | ||
|
|
||
| network_mode_element = None | ||
| for att in element.attributes: | ||
| if att.name == "network" and isinstance(att.value, Hash): | ||
| for k, v in att.value.value.items(): | ||
| if isinstance(k, String) and k.value == "mode": | ||
| if v.value == "host": | ||
| network_mode_element = v | ||
| network_info["mode"] = v.value | ||
| elif ( | ||
| isinstance(k, String) or isinstance(k, VariableReference) | ||
| ) and k.value == "port": | ||
| port_info: Dict[str, Any] = { | ||
| "name": "", | ||
| } | ||
| for _k, _v in v.value.items(): | ||
| if isinstance(_k, String) and _k.value == "port": | ||
| port_info["name"] = _v.value | ||
| elif isinstance(_k, String) and _k.value in [ | ||
| "static", | ||
| "to", | ||
| ]: | ||
| port_info[_k.value] = _v.value | ||
| network_info["ports"].append(port_info) | ||
|
|
||
| for au in element.atomic_units: | ||
| for att in au.attributes: | ||
| if att.name == "config" and isinstance(att.value, Hash): | ||
| temp_errors: List[Error] = [] | ||
| is_api_gateway = False | ||
|
|
||
| if ( | ||
| isinstance(au.name, String) | ||
| and "gateway" in au.name.value.strip().lower() | ||
| ): | ||
| is_api_gateway = True | ||
| has_api_gateway = True | ||
|
|
||
| for k, v in att.value.value.items(): | ||
| if ( | ||
| isinstance(k, String) | ||
| and k.value == "ports" | ||
| and isinstance(v, Array) | ||
| and not is_api_gateway | ||
| ): | ||
| for port in v.value: | ||
| if isinstance(port, String): | ||
| for exp_port in network_info["ports"]: | ||
| if exp_port["name"] == port.value: | ||
| if network_info["mode"] == "host": | ||
| temp_errors.append( | ||
| Error( | ||
| "arc_no_apig", | ||
| port, | ||
| file, | ||
| repr(port), | ||
| ) | ||
| ) | ||
|
|
||
| elif network_info[ | ||
| "mode" | ||
| ] == "bridge" and ( | ||
| "to" in exp_port.keys() | ||
| or "static" in exp_port.keys() | ||
| ): | ||
| temp_errors.append( | ||
| Error( | ||
| "arc_no_apig", | ||
| port, | ||
| file, | ||
| repr(port), | ||
| ) | ||
| ) | ||
|
|
||
| if isinstance(k, String) and k.value == "image": | ||
| image_name, _, _ = SecurityVisitor.image_parser(v.value) | ||
| if ( | ||
| image_name in SecurityVisitor.API_GATEWAYS | ||
| or is_api_gateway | ||
| ): | ||
| is_api_gateway = True | ||
| has_api_gateway = True | ||
| else: | ||
| errors += temp_errors | ||
| temp_errors = [] | ||
|
|
||
| if not is_api_gateway: | ||
| errors += temp_errors | ||
|
|
||
| if not has_api_gateway and network_info["mode"] == "host": | ||
| errors.append( | ||
| Error( | ||
| "arc_no_apig", | ||
| network_mode_element, | ||
| file, | ||
| repr(network_mode_element), | ||
| ) | ||
| ) | ||
|
|
||
| return errors |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why? Shouldn't this be parsed to a variable?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think so, I will try to fix it in the parser