Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Commit bc79cfe

Browse filesBrowse files
fabpotfabpot
committed
feature #32360 [Monolog] Added ElasticsearchLogstashHandler (lyrixx)
This PR was merged into the 4.4 branch. Discussion ---------- [Monolog] Added ElasticsearchLogstashHandler | Q | A | ------------- | --- | Branch? | 4.4 | Bug fix? | no | New feature? | yes | BC breaks? | no | Deprecations? | no | Tests pass? | yes | Fixed tickets | | License | MIT | Doc PR | This PR was initially [submitted on Monolog](Seldaek/monolog#1334). It has been refused , but Jordi suggested to add it to the symfony bridge. So here we go :) --- ATM, there are few options to push log to Elastic Stack in order to play with them: * install logstash and use a Gelf handler. It works but you have to install logstash and configure it. Not an easy task. More over, it need an extra PHP package * use the ES handler: It does not play well with context and extra: Kibana is not able to filter on nested object. And this handler is tightly coupled to the ElasticaFormatter formater. More over, it need an extra PHP package * use something to parse file logs. This is really a bad idea since it involves a parsing... More over a daemon is needed to do that (file beat / logstash / you name it) This is why I'm introducing a new Handler. * There is not need to install anything (expect ES, of course) * It play very well with Kibana, as it uses the Logstash format * It requires symfony/http-client, but in a modern PHP application (SF 4.3) this dependency is already present * It slow down a bit the application since it trigger an HTTP request for each logs. But symfony/http-client is non-blocking. If you want to use it in production, I recommend to wrap this handler in a buffer handler or a cross-finger handle to have only one HTTP call. --- Some performance consideration en a prod env with a buffer handler + this one * with push to ES: https://blackfire.io/profiles/f94ccf35-9f9d-4df1-bfc5-7fa75a535628/graph * with push to ES commented: https://blackfire.io/profiles/6b66bc18-6b90-4341-963f-797f7a7a689c/graph As you can see, as requests are made synchronously, there is no penalty on `AppKernel::Handler()` 😍! But the PHP worker has more work to do, and it's busy much more time (about X2) I explained everything in the PHP Doc Block --- This is what you can expect **out of the box** ![image](https://user-images.githubusercontent.com/408368/59916122-9b7b7580-941e-11e9-9a22-f56bc1d1a288.png) Commits ------- 1587e9a [Monolog] Added ElasticsearchLogstashHandler
2 parents 63272d6 + 1587e9a commit bc79cfe
Copy full SHA for bc79cfe

File tree

4 files changed

+263
-0
lines changed
Filter options

4 files changed

+263
-0
lines changed

‎src/Symfony/Bridge/Monolog/CHANGELOG.md

Copy file name to clipboardExpand all lines: src/Symfony/Bridge/Monolog/CHANGELOG.md
+1Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ CHANGELOG
55
-----
66

77
* The `RouteProcessor` class has been made final
8+
* Added `ElasticsearchLogstashHandler`
89

910
4.3.0
1011
-----

‎src/Symfony/Bridge/Monolog/Handler/ElasticsearchLogstashHandler.php

Copy file name to clipboard
+142Lines changed: 142 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,142 @@
1+
<?php
2+
3+
/*
4+
* This file is part of the Symfony package.
5+
*
6+
* (c) Fabien Potencier <fabien@symfony.com>
7+
*
8+
* For the full copyright and license information, please view the LICENSE
9+
* file that was distributed with this source code.
10+
*/
11+
12+
namespace Symfony\Bridge\Monolog\Handler;
13+
14+
use Monolog\Formatter\FormatterInterface;
15+
use Monolog\Formatter\LogstashFormatter;
16+
use Monolog\Handler\AbstractHandler;
17+
use Monolog\Logger;
18+
use Symfony\Component\HttpClient\HttpClient;
19+
use Symfony\Contracts\HttpClient\Exception\ExceptionInterface;
20+
use Symfony\Contracts\HttpClient\HttpClientInterface;
21+
22+
/**
23+
* Push logs directly to Elasticsearch and format them according to Logstash specification.
24+
*
25+
* This handler dials directly with the HTTP interface of Elasticsearch. This
26+
* means it will slow down your application if Elasticsearch takes times to
27+
* answer. Even if all HTTP calls are done asynchronously.
28+
*
29+
* In a development environment, it's fine to keep the default configuration:
30+
* for each log, an HTTP request will be made to push the log to Elasticsearch.
31+
*
32+
* In a production environment, it's highly recommended to wrap this handler
33+
* in a handler with buffering capabilities (like the FingersCrossedHandler, or
34+
* BufferHandler) in order to call Elasticsearch only once with a bulk push. For
35+
* even better performance and fault tolerance, a proper ELK (https://www.elastic.co/what-is/elk-stack)
36+
* stack is recommended.
37+
*
38+
* @author Grégoire Pineau <lyrixx@lyrixx.info>
39+
*/
40+
class ElasticsearchLogstashHandler extends AbstractHandler
41+
{
42+
private $endpoint;
43+
private $index;
44+
private $client;
45+
private $responses;
46+
47+
public function __construct(string $endpoint = 'http://127.0.0.1:9200', string $index = 'monolog', HttpClientInterface $client = null, int $level = Logger::DEBUG, bool $bubble = true)
48+
{
49+
if (!interface_exists(HttpClientInterface::class)) {
50+
throw new \LogicException(sprintf('The %s handler needs an HTTP client. Try running "composer require symfony/http-client".', __CLASS__));
51+
}
52+
53+
parent::__construct($level, $bubble);
54+
$this->endpoint = $endpoint;
55+
$this->index = $index;
56+
$this->client = $client ?: HttpClient::create(['timeout' => 1]);
57+
$this->responses = new \SplObjectStorage();
58+
}
59+
60+
public function handle(array $record): bool
61+
{
62+
if (!$this->isHandling($record)) {
63+
return false;
64+
}
65+
66+
$this->sendToElasticsearch([$record]);
67+
68+
return !$this->bubble;
69+
}
70+
71+
public function handleBatch(array $records): void
72+
{
73+
$records = array_filter($records, [$this, 'isHandling']);
74+
75+
if ($records) {
76+
$this->sendToElasticsearch($records);
77+
}
78+
}
79+
80+
protected function getDefaultFormatter(): FormatterInterface
81+
{
82+
return new LogstashFormatter('application', null, null, 'ctxt_', LogstashFormatter::V1);
83+
}
84+
85+
private function sendToElasticsearch(array $records)
86+
{
87+
$formatter = $this->getFormatter();
88+
89+
$body = '';
90+
foreach ($records as $record) {
91+
foreach ($this->processors as $processor) {
92+
$record = $processor($record);
93+
}
94+
95+
$body .= json_encode([
96+
'index' => [
97+
'_index' => $this->index,
98+
'_type' => '_doc',
99+
],
100+
]);
101+
$body .= "\n";
102+
$body .= $formatter->format($record);
103+
$body .= "\n";
104+
}
105+
106+
$response = $this->client->request('POST', $this->endpoint.'/_bulk', [
107+
'body' => $body,
108+
'headers' => [
109+
'Content-Type' => 'application/json',
110+
],
111+
]);
112+
113+
$this->responses->attach($response);
114+
115+
$this->wait(false);
116+
}
117+
118+
public function __destruct()
119+
{
120+
$this->wait(true);
121+
}
122+
123+
private function wait(bool $blocking)
124+
{
125+
foreach ($this->client->stream($this->responses, $blocking ? null : 0.0) as $response => $chunk) {
126+
try {
127+
if ($chunk->isTimeout() && !$blocking) {
128+
continue;
129+
}
130+
if (!$chunk->isFirst() && !$chunk->isLast()) {
131+
continue;
132+
}
133+
if ($chunk->isLast()) {
134+
$this->responses->detach($response);
135+
}
136+
} catch (ExceptionInterface $e) {
137+
$this->responses->detach($response);
138+
error_log(sprintf("Could not push logs to Elasticsearch:\n%s", (string) $e));
139+
}
140+
}
141+
}
142+
}

‎src/Symfony/Bridge/Monolog/Tests/Handler/ElasticsearchLogstashHandlerTest.php

Copy file name to clipboard
+119Lines changed: 119 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,119 @@
1+
<?php
2+
3+
/*
4+
* This file is part of the Symfony package.
5+
*
6+
* (c) Fabien Potencier <fabien@symfony.com>
7+
*
8+
* For the full copyright and license information, please view the LICENSE
9+
* file that was distributed with this source code.
10+
*/
11+
12+
namespace Symfony\Bridge\Monolog\Tests\Handler;
13+
14+
use Monolog\Formatter\FormatterInterface;
15+
use Monolog\Formatter\LogstashFormatter;
16+
use Monolog\Logger;
17+
use PHPUnit\Framework\TestCase;
18+
use Symfony\Bridge\Monolog\Handler\ElasticsearchLogstashHandler;
19+
use Symfony\Component\HttpClient\MockHttpClient;
20+
use Symfony\Component\HttpClient\Response\MockResponse;
21+
22+
class ElasticsearchLogstashHandlerTest extends TestCase
23+
{
24+
public function testHandle()
25+
{
26+
$callCount = 0;
27+
$responseFactory = function ($method, $url, $options) use (&$callCount) {
28+
$body = <<<EOBODY
29+
{"index":{"_index":"log","_type":"_doc"}}
30+
{"@timestamp":"2020-01-01T00:00:00.000000+01:00","@version":1,"host":"my hostname","message":"My info message","type":"application","channel":"app","level":"INFO"}
31+
32+
33+
EOBODY;
34+
35+
$this->assertSame('POST', $method);
36+
$this->assertSame('http://es:9200/_bulk', $url);
37+
$this->assertSame($body, $options['body']);
38+
$this->assertSame('Content-Type: application/json', $options['normalized_headers']['content-type'][0]);
39+
++$callCount;
40+
41+
return new MockResponse();
42+
};
43+
44+
$handler = new ElasticsearchLogstashHandlerWithHardCodedHostname('http://es:9200', 'log', new MockHttpClient($responseFactory));
45+
46+
$record = [
47+
'message' => 'My info message',
48+
'context' => [],
49+
'level' => Logger::INFO,
50+
'level_name' => Logger::getLevelName(Logger::INFO),
51+
'channel' => 'app',
52+
'datetime' => new \DateTime('2020-01-01T00:00:00+01:00'),
53+
'extra' => [],
54+
];
55+
56+
$handler->handle($record);
57+
58+
$this->assertSame(1, $callCount);
59+
}
60+
61+
public function testBandleBatch()
62+
{
63+
$callCount = 0;
64+
$responseFactory = function ($method, $url, $options) use (&$callCount) {
65+
$body = <<<EOBODY
66+
{"index":{"_index":"log","_type":"_doc"}}
67+
{"@timestamp":"2020-01-01T00:00:00.000000+01:00","@version":1,"host":"my hostname","message":"My info message","type":"application","channel":"app","level":"INFO"}
68+
69+
{"index":{"_index":"log","_type":"_doc"}}
70+
{"@timestamp":"2020-01-01T00:00:01.000000+01:00","@version":1,"host":"my hostname","message":"My second message","type":"application","channel":"php","level":"WARNING"}
71+
72+
73+
EOBODY;
74+
75+
$this->assertSame('POST', $method);
76+
$this->assertSame('http://es:9200/_bulk', $url);
77+
$this->assertSame($body, $options['body']);
78+
$this->assertSame('Content-Type: application/json', $options['normalized_headers']['content-type'][0]);
79+
++$callCount;
80+
81+
return new MockResponse();
82+
};
83+
84+
$handler = new ElasticsearchLogstashHandlerWithHardCodedHostname('http://es:9200', 'log', new MockHttpClient($responseFactory));
85+
86+
$records = [
87+
[
88+
'message' => 'My info message',
89+
'context' => [],
90+
'level' => Logger::INFO,
91+
'level_name' => Logger::getLevelName(Logger::INFO),
92+
'channel' => 'app',
93+
'datetime' => new \DateTime('2020-01-01T00:00:00+01:00'),
94+
'extra' => [],
95+
],
96+
[
97+
'message' => 'My second message',
98+
'context' => [],
99+
'level' => Logger::WARNING,
100+
'level_name' => Logger::getLevelName(Logger::WARNING),
101+
'channel' => 'php',
102+
'datetime' => new \DateTime('2020-01-01T00:00:01+01:00'),
103+
'extra' => [],
104+
],
105+
];
106+
107+
$handler->handleBatch($records);
108+
109+
$this->assertSame(1, $callCount);
110+
}
111+
}
112+
113+
class ElasticsearchLogstashHandlerWithHardCodedHostname extends ElasticsearchLogstashHandler
114+
{
115+
protected function getDefaultFormatter(): FormatterInterface
116+
{
117+
return new LogstashFormatter('application', 'my hostname', null, 'ctxt_', LogstashFormatter::V1);
118+
}
119+
}

‎src/Symfony/Bridge/Monolog/composer.json

Copy file name to clipboardExpand all lines: src/Symfony/Bridge/Monolog/composer.json
+1Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@
2323
},
2424
"require-dev": {
2525
"symfony/console": "^3.4|^4.0|^5.0",
26+
"symfony/http-client": "^4.4|^5.0",
2627
"symfony/security-core": "^3.4|^4.0|^5.0",
2728
"symfony/var-dumper": "^3.4|^4.0|^5.0"
2829
},

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.