DataFusion is an extensible query execution framework, written in Rust, that uses Apache Arrow as its in-memory format.
The DataFusion CLI allows SQL queries to be executed by an in-process DataFusion context.
USAGE:
datafusion-cli [OPTIONS]
OPTIONS:
-c, --batch-size <BATCH_SIZE> The batch size of each query, or use DataFusion default
-f, --file <FILE>... Execute commands from file(s), then exit
--format <FORMAT> [default: table] [possible values: csv, tsv, table, json,
nd-json]
-h, --help Print help information
-p, --data-path <DATA_PATH> Path to your data, default to current directory
-q, --quiet Reduce printing other than the results and work quietly
-r, --rc <RC>... Run the provided files on startup instead of ~/.datafusionrc
-V, --version Print version information
Create a CSV file to query.
$ echo "1,2" > data.csv$ datafusion-cli
DataFusion CLI v12.0.0
> CREATE EXTERNAL TABLE foo (a INT, b INT) STORED AS CSV LOCATION 'data.csv';
0 rows in set. Query took 0.001 seconds.
> SELECT * FROM foo;
+---+---+
| a | b |
+---+---+
| 1 | 2 |
+---+---+
1 row in set. Query took 0.017 seconds.The CLI can query data in S3 if the following environment variables are defined:
AWS_REGIONAWS_ACCESS_KEY_IDAWS_SECRET_ACCESS_KEY
Note that the region must be set to the region where the bucket exists until the following issue is resolved:
Example:
$ aws s3 cp test.csv s3://my-bucket/
upload: ./test.csv to s3://my-bucket/test.csv
$ export AWS_REGION=us-east-1
$ export AWS_SECRET_ACCESS_KEY=***************************
$ export AWS_ACCESS_KEY_ID=**************
$ ./target/release/datafusion-cli
DataFusion CLI v12.0.0
❯ create external table test stored as csv location 's3://my-bucket/test.csv';
0 rows in set. Query took 0.374 seconds.
❯ select * from test;
+----------+----------+
| column_1 | column_2 |
+----------+----------+
| 1 | 2 |
+----------+----------+
1 row in set. Query took 0.171 seconds.Build the datafusion-cli by cd into the sub-directory:
cd datafusion-cli
cargo build