Welcome to piicatcher Discussions!

[vrajat]

👋 Welcome!

We’re using Discussions as a place to connect with other members of our community. We hope that you:

• Ask questions you’re wondering about.
• Share ideas.
• Engage with other community members.
• Welcome others and are open-minded. Remember that this is a community we
  build together 💪.

To get started, comment below with an introduction of yourself and tell us about what you do with this community.

[Shweta0703]

Hi,

I have a simple query, Can we add new entities and modify regex expressions as per our requirement in piicatcher?

Thanks,
Shweta

[vrajat]

Hi @Shweta0703 ,
Yes you can add new entities and modify regex expressions. You have a few options:

• Fork the project, make changes and deploy it in your environment.
• If these new entities and regex expressions are useful in general, then I am open to accepting these contributions to the main project.
• There are plans to provide plugin points to allow injecting custom regular expressions and entities. Check Support for custom regular expressions #115 for example.

Can you tell me more about new entities and regular expressions you want to add? We can continue the conversation in #115 or here.

[Shweta0703]

Thank you for the reply.

I will get back to you on the details you asked for.
But first, I have another query... is it only text based scanner? in case we have other datatypes like int, datetime. Will it handle this kind of data?

Thanks,
Shweta

[vrajat]

There are two types of scanners - column name and column content.

Column name looks at names of columns like dob, full name
etc and doesn't consider data types.

Column content uses NLP to scan text and looks for names, phone numbers, addresses etc. This scanner only scans text columns because NLP does not work on other data types.

[Shweta0703]

Thanks for the reply.

Currently, we are not adding any new entities. From the future perspective, I asked whether we can add new entities and regex or not.

Could you please let me know what can I follow to create .exe of piicatcher. I have made few changes and I want to test it using command line.

Thanks,
Shweta Tyagi

[vrajat]

I am not familiar with windows eco-system. I dont know how to create .exe files for python apps. Your best bet is to search the web on how to run python apps on windows.

[vingov]

Hi, Great tool, I was wondering can this be used to scan and detect PII on Columnar files like Parquet, ORC on HDFS?

[vrajat]

The support for scanning files is minimal right now - text files in POSIX filesystems. However, I do want to add support for all types of structured files (JSON, CSV and others like avro, parquet and ORC) on other storage platforms (HDFS, S3 etc)

One question I have is if it is easier to access these files through a query engine like Presto. That is why I prioritized support for AWS Athena for ORC on S3. Do you have data in Parquet and ORC that are not accessed by a query engine?

[diksha-dot]

Hi, Just wanted to check if piicatcher works for s3 files.If yes, could you please attach any sample demo link.

[jhecking]

It doesn't at the moment. But could you please elaborate this use case a bit more? Do you want to scan individual files hosted on S3 like e.g. CSV or parquet files? Or do you want to use piicatcher to scan an entire bucket full of files to check if any of them contain sensitive data?

[Adomkay]

Hi, thanks for this thread. I would like to scan an entire bucket full of files to check if any of them contain sensitive data.

[dikshapandey-ml]

hi , i am trying to create lambda function and trying to use piicatcher module.I've created the layer but upon running simple import piicatcher command, i am getting error nable to import module 'lambda_function': No module named 'piicatcher'.Can someone pls help.I've tried on all python version runtime 3.8 and above. any help @jhecking