GitHub - alphilla/Data-Preprocessing-Models: Kafka and others

Name	Name	Last commit message	Last commit date
Latest commit History 178 Commits 178 Commits
AWS Glue Job trigger from Lambda.JPG	AWS Glue Job trigger from Lambda.JPG
AWS MSK Connector for Snowflake.txt	AWS MSK Connector for Snowflake.txt
AWS Pinpoint using Boto3.ipynb	AWS Pinpoint using Boto3.ipynb
AWSS3ToSnowflake.zip	AWSS3ToSnowflake.zip
Access_s3_data_in_Spark_outside_AWS_Account.ipynb	Access_s3_data_in_Spark_outside_AWS_Account.ipynb
Analyze Apache Parquet optimized data using Amazon Kinesis Data Firehose, Amazon Athena.png	Analyze Apache Parquet optimized data using Amazon Kinesis Data Firehose, Amazon Athena.png
Architecture of the generic Job to ingest data from s3 to Snowflake.png	Architecture of the generic Job to ingest data from s3 to Snowflake.png
Athena_Table_as_source_for_Apache_Spark.ipynb	Athena_Table_as_source_for_Apache_Spark.ipynb
Athena_parameterized_queries_using_Python.ipynb	Athena_parameterized_queries_using_Python.ipynb
Automatic Mean Imputation.ipynb	Automatic Mean Imputation.ipynb
Automatic Mode Imputation.ipynb	Automatic Mode Imputation.ipynb
CSV_Sentiment.ipynb	CSV_Sentiment.ipynb
Cake preparation-Detail Architecture (1).jpg	Cake preparation-Detail Architecture (1).jpg
Cloudy-Snow.ipynb	Cloudy-Snow.ipynb
Consumer_File SQS.ipynb	Consumer_File SQS.ipynb
Create_,_Update_,_Delete_Athena_Prepared_Statement_using_boto3.ipynb	Create_,_Update_,_Delete_Athena_Prepared_Statement_using_boto3.ipynb
DLQ Redrive.drawio.png	DLQ Redrive.drawio.png
Data Analysis Practice.ipynb	Data Analysis Practice.ipynb
Data Warehouse in Snowflake.png	Data Warehouse in Snowflake.png
Data_Analysis_Play.tsv	Data_Analysis_Play.tsv
Data_Analysis_Practice.ipynb	Data_Analysis_Practice.ipynb
Demo Data used in Generic Framework for External Table.zip	Demo Data used in Generic Framework for External Table.zip
End-to-End Project on Location Based Service.txt	End-to-End Project on Location Based Service.txt
Error Notifications for Snowflake Tasks using SNS.png	Error Notifications for Snowflake Tasks using SNS.png
Error_Handler_Python_File SQS.ipynb	Error_Handler_Python_File SQS.ipynb
Feature Scaling.ipynb	Feature Scaling.ipynb
Feature Selection based on correlation.ipynb	Feature Selection based on correlation.ipynb
Flat Any JSON File PySpark.ipynb	Flat Any JSON File PySpark.ipynb
Flat any Json Using_Recursion,Filter & Map.ipynb	Flat any Json Using_Recursion,Filter & Map.ipynb
Flat_DataFrame.ipynb	Flat_DataFrame.ipynb
Flatten JSON & Harmonize Schema.ipynb	Flatten JSON & Harmonize Schema.ipynb
Fundamentals of Data Preprocessing using Python.ipynb	Fundamentals of Data Preprocessing using Python.ipynb
Glue Workflow Lab.txt	Glue Workflow Lab.txt
Go Deeper in Partitioning.ipynb	Go Deeper in Partitioning.ipynb
Incremental Crawling , Incremental ETL with Glue Workflow.png	Incremental Crawling , Incremental ETL with Glue Workflow.png
Kafka , Faust & Snowflake.txt	Kafka , Faust & Snowflake.txt
Kafka Consumer Internals with Auto Offset Commits and At Least Once Processing.png	Kafka Consumer Internals with Auto Offset Commits and At Least Once Processing.png
Kafka Faust Snowflake Python.PNG	Kafka Faust Snowflake Python.PNG
Kafka Producer Internals.png	Kafka Producer Internals.png
Kafka Retry Architecture.png	Kafka Retry Architecture.png
Kmeans.ipynb	Kmeans.ipynb
Limit_and_offset_concept_in_API.ipynb	Limit_and_offset_concept_in_API.ipynb
Long-Running Workflows with Human Interactions using Step Functions.txt	Long-Running Workflows with Human Interactions using Step Functions.txt
MSK Lab 1.txt	MSK Lab 1.txt
MSK To AWS Lambda.txt	MSK To AWS Lambda.txt
MSK project with serverless Producer & Consumer.txt	MSK project with serverless Producer & Consumer.txt
MSK with s3 sink connector and Snowflake.txt	MSK with s3 sink connector and Snowflake.txt
Manual Offset Commits and At-most Once Processing.png	Manual Offset Commits and At-most Once Processing.png
Manual Offset Commits and Exactly Once Processing.png	Manual Offset Commits and Exactly Once Processing.png
Multicollinearity Effect in Regression.ipynb	Multicollinearity Effect in Regression.ipynb
Multiple Linear Regression Code.ipynb	Multiple Linear Regression Code.ipynb
Null_Value_Handling_Pyspark.ipynb	Null_Value_Handling_Pyspark.ipynb
One Hot Encoding.ipynb	One Hot Encoding.ipynb
Outliers_&_Skewness_Handling.ipynb	Outliers_&_Skewness_Handling.ipynb
Partitioning in Athena using AWS Lambda.txt	Partitioning in Athena using AWS Lambda.txt
Percentage_of_Null_values_in_each_column_of_Dataframe.ipynb	Percentage_of_Null_values_in_each_column_of_Dataframe.ipynb
Pivoting.ipynb	Pivoting.ipynb
Practice Session Data Analysis.ipynb	Practice Session Data Analysis.ipynb
Publish Message in MSK Cluster from AWS Lambda.txt	Publish Message in MSK Cluster from AWS Lambda.txt
Publisher SQS.ipynb	Publisher SQS.ipynb
PySpark5.ipynb	PySpark5.ipynb
PySpark6.ipynb	PySpark6.ipynb
Pyspark1.ipynb	Pyspark1.ipynb
Pyspark2.ipynb	Pyspark2.ipynb
Pyspark3.ipynb	Pyspark3.ipynb
Pyspark4.ipynb	Pyspark4.ipynb
Real-Time Streaming Project with Smartphone Data.txt	Real-Time Streaming Project with Smartphone Data.txt
Receive_message_from_SQS_Queue.ipynb	Receive_message_from_SQS_Queue.ipynb
Recursion Pattern with AWS Step Funciton & Lambda.png	Recursion Pattern with AWS Step Funciton & Lambda.png
SNS_Message_Publish.ipynb	SNS_Message_Publish.ipynb
Send_message_SQS_Queue.ipynb	Send_message_SQS_Queue.ipynb
Serverless Manual Approval Steps in AWS Step Functions and Amazon API Gateway.png	Serverless Manual Approval Steps in AWS Step Functions and Amazon API Gateway.png
Shake detection using Accelerometer , Kafka & Python.txt	Shake detection using Accelerometer , Kafka & Python.txt
Simple Linear Regression (ML).ipynb	Simple Linear Regression (ML).ipynb
Simple OTP System using AWS Serverless.txt	Simple OTP System using AWS Serverless.txt
Snowflake Codes for generic s3 to snowflake loader.txt	Snowflake Codes for generic s3 to snowflake loader.txt
Snowflake External Table Partitioning.txt	Snowflake External Table Partitioning.txt
Snowflake Parallel Processing using Python Lab.txt	Snowflake Parallel Processing using Python Lab.txt
Snowflake Row Level Security.sql	Snowflake Row Level Security.sql
Snowflake Schema Detection.txt	Snowflake Schema Detection.txt
Snowflake Stored Porcedure Parallel execution (Part 1).txt	Snowflake Stored Porcedure Parallel execution (Part 1).txt
Snowflake Stored Porcedure Parallel execution (Part 2).txt	Snowflake Stored Porcedure Parallel execution (Part 2).txt
Snowflake code for External Table refresh framework.txt	Snowflake code for External Table refresh framework.txt
Snowflake logging (1).txt	Snowflake logging (1).txt
Sorting.ipynb	Sorting.ipynb
String Functions.ipynb	String Functions.ipynb
Success File in PySpark.txt	Success File in PySpark.txt
Talend with EMR & Snowflake.png	Talend with EMR & Snowflake.png
Time Traven in Snowflake.txt	Time Traven in Snowflake.txt
Trigger Airflow code via rest api.txt	Trigger Airflow code via rest api.txt
Untitled7.ipynb	Untitled7.ipynb
Using KMS for Client Side Encryption.ipynb	Using KMS for Client Side Encryption.ipynb
_Manual Offset Commits & At Least Once Processing in Kafka Consumer.drawio.png	_Manual Offset Commits & At Least Once Processing in Kafka Consumer.drawio.png
airflow_emr_s3_snowflake_setup.txt	airflow_emr_s3_snowflake_setup.txt
airflow_emr_spark_s3_snowflake.py	airflow_emr_spark_s3_snowflake.py
airflow_news_data_pipeline.py	airflow_news_data_pipeline.py
algolia_layer1.zip	algolia_layer1.zip
aws-eventbridge-kinesisfirehose-s3.drawio.png	aws-eventbridge-kinesisfirehose-s3.drawio.png
convert_json_to_csv_in_kinesis_firehose_transformation.ipynb	convert_json_to_csv_in_kinesis_firehose_transformation.ipynb
dbscan_visualization.txt	dbscan_visualization.txt

Search code, repositories, users, issues, pull requests...

alphilla/Data-Preprocessing-Models

Folders and files

Latest commit

History

About

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages