Build a new transform to automate crawling and then convert to parquet

[shahrokhDaijavad]

Search before asking

• I searched the issues and found no similar issues.

Component

Transforms/Other

Feature

This is using the DPK-connector lib in the repo (https://github.com/IBM/data-prep-kit/tree/dev/data-connector-lib) which is available as a stand-alone pip install now. This is about making this a "data ingestion" transform with parquet output that can easily be fed into other DPK transforms.

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[touma-I]

@touma-I to get the code from the developers and adapt it to the DPK transform and submit a PR.

[Bytes-Explorer]

Request for consideration as we build this transform:

1. Simple API call with minimal lines
2. Explain what parameters are there in the API and how to use it in the readme
3. Request from Sujee on storing data

[touma-I]

@Bytes-Explorer Can you explain more why you need a structured/nested directory structure ? What are the motivation for it and how this will be used? There are two issues why we should NOT do it:

1. Web sites tend to have badly defined structure with cyclical graph
2. The framework is not setup to handle nested folder.
   Please provide additional detail or we can skip this requirement for now.

[Bytes-Explorer]

@touma-I That requirement is from @sujee Lets discuss in the call. It is in the issue so that requests from our users are not missed on slack.