This repository accompanies the AWS big Data Blog post Batch Data Ingestion into Amazon OpenSearch with AWS Glue. This repo showcases multiple ways to leverage AWS Glue to seamlessly ingest data into Amazon OpenSearch. We’ll cover multiple batch ingestion methods, share practical examples, and discuss best practices to help you build optimized and scalable data pipelines on AWS.
The repository implements the architecture outlined in the diagram below. We will explore various ingestion methods to load data into Amazon OpenSearch using AWS Glue.
AWS provides two managed services for Apache Spark— Amazon EMR and AWS Glue — that streamline data processing and integration workflows for large-scale data applications. In this repository, we will focus on batch data ingestion into OpenSearch using AWS Glue and showcase the implementation details.
AWS Glue offers multiple integration options with Amazon OpenSearch, including:
In this repository, we’ll explore each integration method in detail, guiding you through the setup and implementation. As we progress, we will incrementally build the architecture diagram shown below, providing a clear path for creating robust data pipelines on AWS using AWS Glue and Amazon OpenSearch service.
- Access to a valid AWS account
- The AWS Command Line Interface (AWS CLI) installed on your local machine
gitinstalled on your local machine- Permission to create AWS resources
- Familiarity with Apache Spark , AWS Glue and Amazon OpenSearch Service
Please follow the implementation details provided in the AWS Big Data Blog post, Batch Data Ingestion into Amazon OpenSearch with AWS Glue
See CONTRIBUTING for more information.
See the LICENSE for more information.
This solution provides a reference implementation. AWS makes no claims regarding its security properties of any open source code used in this implementation. Please evaluate the solution according to your organization's security best practices prior implemntatation.