This repository provides a kdb library to access Elasticsearch via its Web API. It provides the following features:
- Document retrieval and search
- Single and bulk document uploading
This library has been written for use with the kdb-common set of libraries.
Once the library is loaded, the Elasticsearch Web API URL must be configured (with .es.setTargetServer[esInstance]) before using any other library functions. If you don't, you'll see a NoElasticsearchUrlException.
Configures which Elasticsearch instance to interface with using this library. The URL should be symbol and be either HTTP or HTTPS.
Example:
q).es.setTargetInstance `:http://192.168.1.78:9200
2019.07.07 12:24:15.832 INFO pid-229 jas 0 Elasticsearch instance set [ URL: :http://192.168.1.78:9200 ]
Provides information on all the indices available in the current Elasticsearch instance
Example:
q).es.getAllIndices[]
2019.07.07 12:26:25.279 INFO pid-322 jas 0 Querying for all available indices from Elasticsearch
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
--------------------------------------------------------------------------------------------------------------------------------
"yellow" "open" "kdb-test-index-2019.07.07" "u0gG5DpCRVu56eBvPSMzOA" ,"1" ,"1" ,"2" ,"0" "3.9kb" "3.9kb"
"yellow" "open" "kdb-test-index-2019.07.05" "2UBqNcBkQaSmGQR4PYJZEQ" ,"1" ,"1" ,"1" ,"0" "3.3kb" "3.3kb"
Allows a search to be performed on the specified index.
NOTE: This search is only a URI search so not all search options are available through this function. See the URI search section of the Elasticsearch documentation for more information
Example:
/ Insert some data to a new index
q) .es.http.post[`$"kdb-test-index2-2019.07.07"; `; `time`sym`price!(.z.p; `VOD.L; 1000f)];
q) .es.http.post[`$"kdb-test-index2-2019.07.07"; `; `time`sym`price!(.z.p; `VOD.L; 2000f)];
/ Search for entries where price = 2000
q) .es.search[`$"kdb-test-index2-2019.07.07"; "price:2000"]
took | 0f
timed_out| 0b
_shards | `total`successful`skipped`failed!1 1 0 0f
hits | `total`max_score`hits!(`value`relation!(1f;"eq");1f;+`_index`_type`_id`_score`_source!(,"kdb-test-index2-2019.07.07";,"doc";,"UhBszGsBBs67mAIUf-jR";,1f;,`time`sym`price!("2019-07-07T12:3..
Retrieves a specific document from the a specific index
Example:
/ Once of the documents added in the .es.search example above
q) .es.getDocument[`$"kdb-test-index2-2019.07.07"; `URBszGsBBs67mAIUYeg1]
_index | "kdb-test-index2-2019.07.07"
_type | "doc"
_id | "URBszGsBBs67mAIUYeg1"
_version | 1f
_seq_no | 0f
_primary_term| 1f
found | 1b
_source | `time`sym`price!("2019-07-07T12:33:05.180098000";"VOD.L";1000f)
Allows any Elasticsearch URL to be queried and the results returned. The library expects all returned data to be in JSON format to be oconverted into native kdb+ types
Adds a new document to an index
index: The name of the index to add the document to. The index will be automatically appended with today's date (in yyyy.mm.dd format) if there is no date component specifiedid: The ID of the entry to add. Generally this should be left as a null symbol to allow Elasticsearch to generate its owncontent: The content to be uploaded. It must be either a dictionary or a JSON string
Example:
q) .es.http.post[`$"kdb-test-index2"; `; `time`sym`price!(.z.p; `VOD.L; 2000f)]
2019.07.07 12:33:13.060 INFO pid-382 jas 0 No date found in index name. Automatically adding today's date [ Index: kdb-test-index2 ]
_index | "kdb-test-index2-2019.07.07"
_type | "doc"
_id | "UhBszGsBBs67mAIUf-jR"
_version | 1f
result | "created"
_shards | `total`successful`failed!2 1 0f
_seq_no | 1f
_primary_term| 1f
Allows multiple documents to be uploaded to an index in one operation via the Bulk API.
index: The name of the index to add the document to. The index will be automatically appended with today's date (in yyyy.mm.dd format) if there is no date component specifiedcontentTable: The data to upload where each row of the table is a single document
If you want to specify custom IDs for each document to be added, ensure that the table provided has an id column
Example:
q) tbl:flip `col1`col2`col3!2?/:(`2; 100f; 1b)
q) .es.http.postBulk[`$"kdb-test-index4"; tbl]
2019.07.07 12:46:53.208 INFO pid-411 jas 0 No date found in index name. Automatically adding today's date [ Index: kdb-test-index4 ]
took | 3f
errors| 0b
items | +(,`index)!,(`_index`_type`_id`_version`result`_shards`_seq_no`_primary_term`status!("kdb-test-index4-2019.07.07";"doc";"VhB5zGsBBs67mAIUA-h7";1f;"created";`total`successful`failed!2 1 0f;2..