Skip to content

Commit

Permalink
doc: update README.md
Browse files Browse the repository at this point in the history
  • Loading branch information
alimghmi committed Oct 24, 2023
1 parent 5c2342a commit 437559c
Show file tree
Hide file tree
Showing 2 changed files with 65 additions and 44 deletions.
4 changes: 2 additions & 2 deletions .env.sample
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
URL="https://www.ice.com/public-web/cds-settlement-prices/icc/single-names"
LOG_LEVEL="INFO"
URL=https://www.ice.com/public-web/cds-settlement-prices/icc/single-names
LOG_LEVEL=INFO
OUTPUT_TABLE=
INSERTER_MAX_RETRIES=2
REQUEST_MAX_RETRIES=3
Expand Down
105 changes: 63 additions & 42 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,42 +1,63 @@
# ICE Data Scraper

This project is designed to scrape the ICE (Intercontinental Exchange) webpage, specifically the Markit ICE Settlement Prices. The scraped data is then transformed and inserted into a Microsoft SQL Server database.


## Environment Setup

To set up the environment for this project, you need to have the following configurations in an `.env` file:

```plaintext
URL="https://www.ice.com/public-web/cds-settlement-prices/icc/single-names"
LOG_LEVEL="INFO"
OUTPUT_TABLE=<name_of_the_output_table>
INSERTER_MAX_RETRIES=2
REQUEST_MAX_RETRIES=3
REQUEST_BACKOFF_FACTOR=2
MSSQL_SERVER=<your_mssql_server>
MSSQL_DATABASE=<your_mssql_database>
MSSQL_USERNAME=<your_mssql_username>
MSSQL_PASSWORD=<your_mssql_password>
```

Replace the placeholders (<...>) with the appropriate values.

## How to Run
1. Ensure you have all the required libraries installed.
2. Set up your .env file with the appropriate values.
3. Run the main.py script:

```bash
python main.py
```

The script will then:

- Initialize the scraping engine
- Fetch the data from the ICE webpage
- Transform the fetched data
- Prepare the database inserter
- Insert the transformed data into the specified output table in the MS SQL Server database
- Log the process as it progresses

# ICE Data Scraper with Docker 🚀

Scrape, transform, and insert ICE (Intercontinental Exchange) settlement prices into an MSSQL database, with Docker support for easy deployment and scaling.


## 📌 Features:

- **Efficient Scraping:** Targets the Markit ICE Settlement Prices from ICE's official page.
- **Data Transformation:** Tailored data transformation for easy database insertion.
- **MSSQL Support:** Built-in support to insert data into a Microsoft SQL Server database.
- **Dockerized:** Simplified deployment and setup using Docker.
- **Robust Error Handling:** Multi-retry mechanisms and comprehensive logging.


## 🚀 Getting Started:

### Environment Setup

1. Clone the repository:
``` bash
git clone [email protected]:alimghmi/ice-client.git
cd ice-client
```
2. Create an `.env` file in the project root and configure the following:
```
URL="https://www.ice.com/public-web/cds-settlement-prices/icc/single-names"
LOG_LEVEL="INFO"
OUTPUT_TABLE=<name_of_the_output_table>
INSERTER_MAX_RETRIES=2
REQUEST_MAX_RETRIES=3
REQUEST_BACKOFF_FACTOR=2
MSSQL_SERVER=<mssql_server>
MSSQL_DATABASE=<mssql_database>
MSSQL_USERNAME=<mssql_username>
MSSQL_PASSWORD=<mssql_password>
```
Replace the placeholders (`<...>`) with the appropriate values.

### Running with Python

1. Install the required packages:
```bash
pip install -r requirements.txt
```
2. Run the `main.py` script:
```bash
python main.py
```

### Running with Docker

1. Build the Docker image:
```bash
docker build -t ice-data-scraper .
```
2. Run the Docker container:
```bash
docker run --env-file .env ice-data-scraper
```

## 🙌 Contribution

Feel free to fork the repository, make changes, and open a pull request. All contributions are welcomed!

0 comments on commit 437559c

Please sign in to comment.