Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fields not getting masked #14

Open
f9-abs opened this issue Jan 6, 2022 · 6 comments
Open

Fields not getting masked #14

f9-abs opened this issue Jan 6, 2022 · 6 comments

Comments

@f9-abs
Copy link

f9-abs commented Jan 6, 2022

Hi,

I am trying to test this plugin on Windows 10 using td-agent v4, but defined fields are not getting masked. I ran td-agent in debug mode as well, but no there is message on what is going wrong. During td-agent startup, plugin does get loaded and fields to be masked are also read properly.

As you can see from attached files, I am trying to mask the key "ProviderName" and "Computer". This is the output (with dummy value for "Computer" tag), as you can see both keys are not getting masked:

--- sample log entries ---
2022-01-06 12:37:55.877141700 +0530 winevt.raw: {"ProviderName":"Nexthink Collector","ProviderGUID":"","EventID":"3221225728","Level":"2","Task":"1","Opcode":"0","Keywords":"0x80000000000000","TimeCreated":"2022/01/06 07:07:55.85937600","EventRecordID":"198171","ActivityID":"","RelatedActivityID":"","ProcessID":"0","ThreadID":"0","Channel":"Application","Computer":"N-LAPTOPID.domain.net","UserID":"","Version":"0","Description":"Connection to Nexthink Appliance cannot be established: Websocket error: Host not found: [Host not found] [appliance host: engine9.domain.net:8443]","EventData":["Connection to Nexthink Appliance cannot be established: Websocket error: Host not found: [Host not found] [appliance host: engine9.domain.net:8443]"]}

2022-01-06 12:34:13.877078900 +0530 winevt.raw: {"ProviderName":"ESENT","ProviderGUID":"","EventID":"102","Level":"4","Task":"1","Opcode":"0","Keywords":"0x80000000000000","TimeCreated":"2022/01/06 07:04:12.183182000","EventRecordID":"198155","ActivityID":"","RelatedActivityID":"","ProcessID":"0","ThreadID":"0","Channel":"Application","Computer":"N-LAPTOPID.domain.net","UserID":"","Version":"0","Description":"Video.UI (18280,P,98) {F28496CB-2EF7-4366-B722-51C0D5BFB252}: The database engine (10.00.18363.0000) is starting a new instance (0).","EventData":["Video.UI","18280,P,98","{F28496CB-2EF7-4366-B722-51C0D5BFB252}: ","0","10","00","18363","0000"]}
--- sample log entries ---

--- startup log ---
2022-01-06 12:27:03 +0530 [info]: adding match pattern="winevt.raw" type="stdout"
2022-01-06 12:27:05 +0530 [info]: adding filter pattern="**" type="masking"
black list fields:
ProviderName
Computer
--- startup log ---

Any ideas on what could be going wrong here? Thanks in advance.

test-files.zip

@kohend
Copy link
Contributor

kohend commented Jan 6, 2022

You didn't configure it properly.
The blacklist fields are the fields which contain the list of fields to mask, the list itself is not in the configuration.

@f9-abs
Copy link
Author

f9-abs commented Jan 7, 2022

In config file, I have provided path to the file containing list of fields to mask. Is this incorrect?

--- clip ---
<filter "**">
@type masking
fieldsToMaskFilePath "C:/opt/td-agent/fields-to-mask-file.txt"

--- clip ---

@kohend
Copy link
Contributor

kohend commented Jan 7, 2022

Oh, that looks OK, is the log parsed as JSON in your config? If not, it won't identify the fields properly.

@f9-abs
Copy link
Author

f9-abs commented Jan 7, 2022

Yes. I tested some other plugins (e.g. https://github.com/y-ken/fluent-plugin-anonymizer), and that worked fine.

@f9-abs
Copy link
Author

f9-abs commented Jan 17, 2022

Hi. Can you please share any updates on this? BTW, does this plugin support masking based on regex instead of keys?

@shyimo
Copy link
Contributor

shyimo commented Jan 31, 2022

Hi @f9-abs. Thanks for the issue.
we will try to reproduce it the problem and let you know about the findings

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

3 participants