Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

Conversation

mmehnert
Copy link
Contributor

@mmehnert mmehnert commented May 11, 2024

When using the dovecot-fts-xapian plugin as described in the "Full-Text Search" article of the projects' documentation, the daily logwatch emails get cluttered with output from the indexing of emails.
This pull request adds an ignore.conf to the logwatch configuration with a regular expression to filter out the corresponding messages.

Type of change

  • Improvement (non-breaking change that does improve existing functionality)

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • New and existing unit tests pass locally with my changes
  • I have added information about changes made in this PR to CHANGELOG.md

@georglauterbach georglauterbach self-requested a review May 11, 2024 14:08
@georglauterbach georglauterbach added service/dovecot kind/improvement Improve an existing feature, configuration file or the documentation labels May 11, 2024
@georglauterbach georglauterbach added this to the v14.0.0 milestone May 11, 2024
@georglauterbach
Copy link
Member

Thank you for the PR, this looks like a sensible change.

However, the checklist is not supposed to be shortened, it's supposed to be completed! I have re-added the missing points, please make sure to complete all the items and check them in the list. I have already ticked the items in the list that can be ticked.

Copy link
Member

@georglauterbach georglauterbach left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍🏼 One small change and we're good I think.

target/logwatch/ignore.conf Show resolved Hide resolved
@mmehnert
Copy link
Contributor Author

Thank you for the PR, this looks like a sensible change.

However, the checklist is not supposed to be shortened, it's supposed to be completed! I have re-added the missing points, please make sure to complete all the items and check them in the list. I have already ticked the items in the list that can be ticked.

I'm sorry that I mishandled that. I can't see the missing points / check them in the list. Should I copy them from somewhere again and modify the pull request text myself?

@polarathene polarathene changed the title add logwatch ignore.conf and ignore "dovecot: indexer-worker" output of successfuly indexed messages chore(logwatch): Add ignore.conf to ignore logs from Dovecot index-worker May 11, 2024
@polarathene
Copy link
Member

polarathene commented May 11, 2024

I can't see the missing points / check them in the list. Should I copy them from somewhere again and modify the pull request text myself?

I've restored these.

I'm not sure if @georglauterbach is wanting them all addressed though (tests/docs), they don't seem necessary.

You should add a changelog entry, this seems suitable for the "Updates" section:

- **Environment Variables:**
- `ONE_DIR` has been removed (legacy ENV) ([#3840](https://github.com/docker-mailserver/docker-mailserver/pull/3840))

Just add something above the referenced like, such as:

- **Dovecot:**
  - `logwatch` now filters out non-error logs related to the status of the `index-worker` process for FTS indexing. ([#4012](https://github.com/docker-mailserver/docker-mailserver/pull/4012))

Regarding this contribution, do you have some samples of the logs you're preventing? I have seen here that there can be error logs that may be relevant to not ignore if they appear?

Jan 21 06:35:24 mailserver dovecot: indexer-worker: Error: terminate called after throwing an instance of 'std::bad_alloc'
Jan 21 06:35:24 mailserver dovecot: indexer-worker: Error: what(): std::bad_alloc
Jan 21 06:35:24 mailserver dovecot: indexer: Error: Indexer worker disconnected, discarding 1 requests for email@example.com
Jan 21 06:35:24 mailserver dovecot: indexer-worker(email@example.com)<16330><rgw7KfF/AJsqCqVDQq0AAD1inuN9Ly/l:UJw9Kz5ZRVzKPwAAhwicDQ>: Fatal: master: service(indexer-worker): child 16330 killed with signal 6 (core dumps disabled - https://dovecot.org/bugreport.html#coredumps)
Jan 21 06:35:24 mailserver dovecot: imap(email@example.com)<16326><rgw7KfF/AJsqCqVDQq0AAD1inuN9Ly/l>: Error: indexer failed to index mailbox INBOX

Whereas the goal seems to be more targeted around filtering these kind of logs out:

Jan 21 06:31:34 mailserver dovecot: indexer-worker(email@example.com)<16330><ZN42KfF//poqCqVDQq0AAD1inuN9Ly/l:Wut/Mf9YRVzKPwAAhwicDQ>: Indexed 5969 messages in Gesendet (UIDs 1..5972)
Jan 21 06:31:34 mailserver dovecot: indexer-worker(email@example.com)<16330><ZN42KfF//poqCqVDQq0AAD1inuN9Ly/l:8BYpLzZZRVzKPwAAhwicDQ>: Start indexing 'Papierkorb' (/srv/mailstore/example.com/email/xapian-indexes/db_90dda8344dfc3158af39000087089c0d)
Jan 21 06:31:42 mailserver dovecot: indexer-worker(email@example.com)<16330><ZN42KfF//poqCqVDQq0AAD1inuN9Ly/l:8BYpLzZZRVzKPwAAhwicDQ>: Done indexing 'Papierkorb'
Jan 21 06:31:42 mailserver dovecot: indexer-worker(email@example.com)<16330><ZN42KfF//poqCqVDQq0AAD1inuN9Ly/l:8BYpLzZZRVzKPwAAhwicDQ>: Indexed 1012 messages in Papierkorb (UIDs 1..3588)
Jan 21 06:31:42 mailserver dovecot: indexer-worker(email@example.com)<16330><rgw7KfF/AJsqCqVDQq0AAD1inuN9Ly/l:UJw9Kz5ZRVzKPwAAhwicDQ>: Start indexing 'INBOX' (/srv/mailstore/example.com/email/xapian-indexes/db_28c518021bee30580b02000087089c0d)

Keywords in these logs are dovecot: indexer-worker + Indexed / Start indexing (or xapian-indexes, possibly too specific) / Done indexing.

I see the regex is targeting one keyword pair there but will still include the start/done log lines, is that still desired? The error logs should remain, so no problem there.


For maintainers - Reference links related to the Dovecot indexer process and FTS:

@georglauterbach
Copy link
Member

Indeed, we don't no l need most of the points, so I ticked them, too. The CHANGELOG update is the most important one :)

@polarathene
Copy link
Member

so I ticked them, too

That'd be misleading as no such changes were done. Removing those lines is fine, I've edited to drop those redundant checklist items.

@mmehnert just update CHANGELOG.md as per suggestion above and consider the feedback on the regex, you should be good to go.

@mmehnert
Copy link
Contributor Author

Regarding this contribution, do you have some samples of the logs you're preventing? I have seen here that there can be error logs that may be relevant to not ignore if they appear?

I was explicitly only targeting messages of the form

Jan 21 06:31:34 mailserver dovecot: indexer-worker(email@example.com)<16330><ZN42KfF//poqCqVDQq0AAD1inuN9Ly/l:Wut/Mf9YRVzKPwAAhwicDQ>: Indexed 5969 messages in Gesendet (UIDs 1..5972)

Interestingly in my setup I don't have the messages containing "Start indexing" and "Done indexing". So I did not include a regexp for those. @polarathene do you encounter those messages in your installation? If so, I'm happy to add the necessary two lines to ignore.conf

The error messages like

Jan 21 06:35:24 mailserver dovecot: indexer-worker: Error: terminate called after throwing an instance of 'std::bad_alloc'

should not be excluded in my opinion and are not matched by the regexp that I've put into ignore.conf

@polarathene
Copy link
Member

Thanks for the contribution! 😁

I'll leave this for another maintainer to approve before merging.

@polarathene do you encounter those messages in your installation?

I don't actually deploy DMS (ironic I know), and I can't configure a local instance with FTS to try it out myself any time soon 😓

If those log lines aren't present, I wouldn't worry about it. Either something changed since that report or we have a less verbose log from Dovecot configured. If someone is affected and it bothers them they're welcome to contribute another PR 👍

@polarathene polarathene enabled auto-merge (squash) May 12, 2024 06:54
@polarathene polarathene merged commit dab3d9f into docker-mailserver:master May 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/improvement Improve an existing feature, configuration file or the documentation service/dovecot

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

Morty Proxy This is a proxified and sanitized view of the page, visit original site.