- Create a
.envfile in the project directory with the following variables:
DB_USERNAME=your_db_username DB_PASSWORD=your_db_password DB_HOST=your_db_host DB_PORT=your_db_port DB_NAME=your_db_name S3_BUCKET_NAME=your_s3_bucket_name TABLE_NAME=your_table_name
- Set the chunk size in the
main.pyfile if needed:
chunk_size = 10000 # Number of rows per chunk
-
Install the dependencies:
pip install -r requirements.txt -
Run the script:
python main.py