Data Flow

Managed infrastructure

OCI Data Flow handles infrastructure provisioning, network setup, and teardown when Spark jobs are complete. Storage and security are also managed, which means less work is required for creating and managing Spark applications for big data analysis.

Easier cluster management

With OCI Data Flow, there are no clusters to install, patch, or upgrade, which saves time and operational costs for projects.

Simplified capacity planning

OCI Data Flow runs each Spark job in private dedicated resources, eliminating the need for upfront capacity planning.

Lower costs

With OCI Data Flow, IT only needs to pay for the infrastructure resources that Spark jobs use while they are running.

Advanced streaming support capabilities

Spark Streaming with zero management, automatic fault-tolerance, and automatic patching.

Enable continuous processing

With Spark Streaming support, you gain capabilities for continuous retrieval and continuous availability of processed data. OCI Data Flow handles the heavy lifting of stream processing with Spark, along with the ability to perform machine learning on streaming data using MLLib. OCI Data Flow supports Oracle Cloud Infrastructure (OCI) Object Storage and any Kafka-compatible streaming source, including Oracle Cloud Infrastructure (OCI) Streaming as data sources and sinks.

Automatic fault tolerance

Spark handles late-arriving data due to outages and can catch up backlogged data over time with watermarking—a Spark feature that maintains, stores, and then aggregates late data—without needing to manually restart the job. OCI Data Flow automatically restarts your application when possible and your application can simply continue from the last checkpoint.

Cloud native authentication

OCI Data Flow streaming applications can use cloud native authentication via resource principals so applications can run longer than 24 hours.

Cloud native security and governance

Leverage unmatched security from Oracle Cloud Infrastructure. Authentication, isolation, and all other critical points are addressed. Protect business-critical data with the highest levels of security.

Granular security

OCI Data Flow makes native use of Oracle Cloud's Identity and Access Management system for controlled data and access, so data stays secure.

Managed resources

Set quotas and limits to manage resources available to OCI Data Flow and control costs.

Simplified operations

OCI Data Flow simplifies common operational tasks like log management and access to operational UIs, freeing up developer time to focus on building applications.

Increased visibility

OCI Data Flow makes it easy to see what Spark users are doing by aggregating operational information into a single, searchable UI.

Simple debugging and diagnostics

Tracking down logs and tools to troubleshoot a Spark job can take hours—but not with a consolidated view of log output, Spark history server, and more.

Avoid future costs

Sort, search, and filter to investigate historic applications to better address expensive jobs and avoid unnecessary expenditures.

Manage runaway Spark jobs

Administrators can easily discover and stop live Spark jobs that are running for too long or consuming too many resources and driving up costs.

Simplified development

Big data ecosystems require many moving parts and integrations—but OCI Data Flow is compatible with existing Spark investments and big data services, making it easy to manage the service and deliver its results where they’re needed.

Compatible with existing applications

Migrate existing Spark applications from Hadoop or other big data services.

Secure output management

Automatically—and securely—capture and store Spark jobs' output, and then access them through the UI or REST APIs to bring make analytics available.

Control with REST APIs

All aspects of OCI Data Flow can be managed using simple REST APIs, from application creation to execution to accessing results of Spark jobs.