Testing is in progress

Dang it, I accidentally added my other PR into this one. Will fix as soon as the tests finish.

I have x failed tests:

mail_with_relays.bats
 ✗ [Relay Host] default mapping is added from ENV variables for new virtual user (alias) entry [93]
   (from function `assert_success' in file test/test_helper/bats-assert/src/assert_success.bash, line 42,
    in test file test/tests/serial/mail_with_relays.bats, line 52)
     `assert_success' failed
   
   -- command failed --
   status : 127
   output : /home/jirka/Projects/docker-mailserver/test/bats/lib/bats-core/test_functions.bash: line 161: ./setup.sh: cannot execute: required file not found
   --

no_container.bats
 ✗ [No Existing Container] 'setup.sh -p <PATH> -i <IMAGE>' should correctly use options [28]
   (from function `assert_success' in file test/test_helper/bats-assert/src/assert_success.bash, line 42,
    in test file test/tests/serial/no_container.bats, line 26)
     `assert_success' failed
   
   -- command failed --
   status : 127
   output : /home/jirka/Projects/docker-mailserver/test/bats/lib/bats-core/test_functions.bash: line 161: ./setup.sh: cannot execute: required file not found
   --                                                                                                                                                                                                     

The whole test process can be seen here:

DMS-Solar_tests.cast.gz

I have added information about changes made in this PR to CHANGELOG.md

You forgot to update the CHANGELOG.md.

Dang it. I messed up some commits and reverted that one by accident. Forgot to add it back. Will do in the next 40 minutes

The wording isn't the best but it gets the message across (keep in mind that it's 11pm here so I'm tired). I'm open to suggestions.

Documentation preview for this PR is ready! 🎉

Built with commit: e51869d

Too bad, really loved this feature. With large mailboxes this litterly makes the difference between snappy mail search and no mail search at all :/ Guess I will have a look at Xapian then :)

Guess I will have a look at Xapian then :)

Does Xapian not accomplish the same?

The future successor (flatcurve I think?) is also meant to be a better replacement AFAIK.

Solr support can be restored if someone is willing to contribute the requirements (as per the changelog entry).

Realistically though, if you look at what was removed, it really is just a single package dovecot-solr support wise. You could leverage user-patches.sh to install that during each container startup, or via your own Dockerfile build that uses FROM on one of our DMS releases. That can even be inlined into your compose.yaml file config which should minimize any friction.

The docs are not helpful, so they've been dropped. No maintainer is able to assist / support with Solr issues either, so it makes sense for us to drop it officially.

Guess I will have a look at Xapian then :)

Does Xapian not accomplish the same?

The future successor (flatcurve I think?) is also meant to be a better replacement AFAIK.

Solr support can be restored if someone is willing to contribute the requirements (as per the changelog entry).

Realistically though, if you look at what was removed, it really is just a single package dovecot-solr support wise. You could leverage user-patches.sh to install that during each container startup, or via your own Dockerfile build that uses FROM on one of our DMS releases. That can even be inlined into your compose.yaml file config which should minimize any friction.

The docs are not helpful, so they've been dropped. No maintainer is able to assist / support with Solr issues either, so it makes sense for us to drop it officially.

Thx! Used Xapian before and the non server part is a bonus I guess. Seems single threaded though and is indexing for more then a day now, however it does seem to work. Will look in getting solr to work again if I can find some time for that :)

In the end xapian took like 3 days (!) to finish and start offering fts on clients like gmail. Its index had grew to 20+ GB ... so I decided to give solr one more go as I could recall it to grow that big or take such an insane amount of time to finish the initial indexing.

This post proved to be rather useful: https://cyclops.nettrends.nl/blog/2023/09/dovecot-with-apache-solr-in-docker-with-ssl/

Of course it wasn't as simple as just using solr 9 solrconfig.xml and schema.xml from the official dovecot repo, I also had to copy /opt/solr/contrib/analysis-extras/lib to the lib dir for the dovecot core and copy in some defaults like protwords.txt. But after that I had a working solr 9 dovecot core.

Building Mailserver with dovecot-solr with podman was also not hassle free, but my guess is people with real docker probably won't have that issue(?).

In the end mailserver+solr took a couple of minutes and about 1.6GB to index my insanely big inbox. After that I could search on my mobile for specific keywords in the big pile of e-mail near instantly where searching without dovecot fts is simply not possible, it just timeouts.

I strongly urge you to reconsider this decision, still consider myself a novice on docker container building and all of this just took my an hour or so, mostly on figuring out the missing parts.

I strongly urge you to reconsider this decision, still consider myself a novice on docker container building and all of this just took my an hour or so, mostly on figuring out the missing parts.

You can contribute a community guide to our docs. The only integration DMS really did was include a plugin. Since we already include a Dovecot plugin for a Lua-based community guide, I'd be open to accepting this. The only rule we have then is that provided the plugin does not negatively affect maintenance, it should be fine to include it.

My main concern is we can't officially support it, since no maintainer has a need for Solr, nor interest in keeping documentation relevant (which is why we dropped it with v14, docs were terribly outdated).

For the few users that do bother to setup Solr because they have a need for it, it's a few lines for you to add the plugin in compose.yaml (or a separate Dockerfile / user-patches.sh). If you would like others to benefit, you are welcome to contribute a guide, but that is often more effort to get iterated on through review than users have time to spare.

Show your support / interest for dovecot-solr in this issue: #4052

Another user shared their approach to getting Solr working with DMS. I wasn't fond of it as it did not seem like it would age well as a guide in the docs, so you contribute something to the docs and that becomes outdated like the previous guide that no one updated or raised issue with for years, we would likely drop it again 🤷‍♂️

If enough users show interest for dovecot-solr package on that issue I referenced, it can get the package included without any guide in docs. For now the workaround to include the package is simple enough, considering all the additional effort you require to get Solr configured and compatible across DMS updates that I don't see how including the package in DMS is that much of an improvement?

It would be great if you could just include dovecot-solr in your build process and nothing more. As long as one does not load the solr plugin via config it should not touch or affect anything. Imho the method I used now is future proof:

• I just used the official solr docker container
• I used the official dovecot solr config which is up2date with current atm. I have no reason to susupect they wouldn't update this in the future.

Will write a support doc then. After all it didn't took that much.

I have tried setting up Solar in a Kubernetes cluster yesterday, and I can confidently say this is nothing we can maintain at all [1]. I appreciate the documentation PR, and we will definitely merge it (after all the feedback has been addressed).

Installing dovecot-solr is problematic for the ARM64 builds, though; hence, I would stick with user-patches.sh.

[1]

• Changing up configs in such a manner is the first coffin in the nail for me; why do I need to fiddle with wrong default configs?
• Next is state management: I have to create the configs and cores manually on first start, but the way this is being done is neither easily handleable in Kubernetes nor is it actually nice, due to the manual overhead. While this project primarily targets Docker and Docker Compose, I can image people run into the same pitfalls and weird error messages as I did.
• I can also imagine that enabling Flatcurve with Dovecot 2.4 is way easier.
• I acknowledge that Solr 9 may be more performant, and this is the reason I think we should document it, but I am not willing nor do I have the time to support people with Solr issues. This sounds harsh, but please keep in mind that the most precious resource in this project, by far, is maintainers' time. And we need to make sure we can properly maintain DMS. Hence, I still support the removal of Solr.

I understand, not being able to provide an arm64 build these days is problematic. Pitty.
However imho solr does have legacy, but the need to describe how the content you want to index and search atually looks like is a feature not a bug ;) Stuff like elastic search which does seem to have the coolaid these days has the exact same requirement. However the need to constantly review your scheme when the software changes, which it does quite regularly, is kind of stupid. I can only agree on that part.

It would be great if you could just include dovecot-solr in your build process and nothing more

Is there a reason that my suggestion to add the package via compose.yaml or user-patches.sh was not adequate?

It's really not much friction for a user that wants to setup Solr?:

services:
  dms:
    hostname: mail.example.test
    # Do not use `image` anymore, unless referring to the tag below
    # Add this build section to your real `compose.yaml`:
    build:
      tags:
        - local/dms:14.0
      dockerfile_inline: |
        FROM docker.io/mailserver/docker-mailserver:14.0
        RUN apt-get update && apt-get install dovecot-solr

Running docker compose up like normal will then pull the DMS image just like when you'd reference it in the image property, but it'll extend that image with the additional RUN directive and store this new image locally as local/dms:14.0.

When you want to update, instead of changing the image property, you'd now just bump the FROM line of the base DMS image to use. I've omitted the patch version, so forcing a rebuild will use the last published major.minor version match. Otherwise this will work the same as you're used to, and you can add any additional packages you may want?

user-patches.sh is an alternative to the above, but since you should recreate the container every time you restart DMS, it'll add that package each time delaying start up. That shouldn't be frequent, but if it were an issue the compose.yaml image extending approach above works better.

Great tip about the compose.yml edit, just tried is add it works like a charm. Learned a new trick today 👍 :D