Tags: osm2pgsql-dev/osm2pgsql
Tags
Release notes 2.2.0 We are excited to publish version 2.2.0 of osm2pgsql. This release adds two powerful building blocks that can be used to implement all sorts of new features. The new "Locators" and the "deleted callbacks" featuere seem simple by themselves, but they can be used as basis for many features that have been requested for years. This release introduces a new feature: Locators. They help with a common problem, namely how to quickly find out in which region an OSM object is. The region can be a country or any other area. A locator is initialized with one or more regions, each region has a name and a polygon or bounding box. A geometry of an OSM object can then be checked against this region list to figure out in which region(s) it is located. This check is much faster than it would be to do this inside the database after import. Locators can be used for all sorts of interesting features, for instance: * Read larger OSM file but import only data inside some area. * Annotate each OSM object with the country (or other region) it is in. This can then, for instance, be used to show special highway shields for each country. * Use the information which region the data is in for further processing, for instance setting of default values for the speed limit or using special language transliterations rules based on country. For details see the [manual](https://osm2pgsql.org/doc/manual.html#locators) and look at the example config files provided in the flex-config/locator directory. The Lua callback functions `process_node()`, `process_way()`, and `process_relation()` (and their `untagged` companions) only get called for new or changed OSM objects. Deleted objects are usually handled behind the scenes and for most use cases this is enough. But sometimes it is useful to also know when an object was deleted. That's why we have new `process_deleted_node/way/relation()` callbacks now. These open up a lot of new processing options for working with changes. Some users do advanced processing of OSM data inside the database after the data is imported with osm2pgsql. This is much easier now because you don't need database triggers anymore to detect and process deleted objects. We have added a osm2pgsql-expire command. It is currently marked as experimental so it might change without notice. You can use it to turn entries in expire files (that look like `ZOOM/X/Y`) into a GeoJSON file visualizing the tiles. When given an OSM data file it can generate the tile output (in expire file format or as GeoJSON). This tool is intended as a debugging aid. * Fix: Ways and relations that didn't change themselves but were processed due to some of their members changing were always send to the normal `process_node/way/relation()` callback even if they had no tags. This is now fixed, the `process_untagged_*()` callbacks are called instead. * Fix problem when detecting PostGIS version at the start of osm2pgsql. If the PostGIS version can not be detected, we report that properly. * Allow both `--username` and `--user` command line options in all commands, there was an inconcistency between osm2pgsql and osm2pgsql-replication before. * When using the RAM middle, untagged nodes that were members of a relation were not found when building the geometry for that relation. This is now fixed and MultiPoint and GeometryCollection geometries are built correctly. * Fixed bounding box calculation for relations with node members. * Reworked README.md, CONTRIBUTING.md and Contribution Guide on the website. * Refactored a lot of the boilerplate code handling the Lua integration. This code is now shorter and easier to understand and maintain. * Fixed bugs in wildcard matching of old C transforms. * Fixed and improved tests around index creation. * Finally fixed the last warnings reported by clang-tidy, see https://osm2pgsql.org/news/2025/08/21/zero-warnings.html, from now on new code is only allowed in if it checks cleanly with clang-tidy. * Various cleanups in flex example config files. * Lots of code cleanups, mainly to bring naming and formatting to a consistent state. * As always, some code was refactored and there were many small fixes to the code and docs.
Release 2.1.0 This is a bugfix and maintenance release. The main visible change is that osm2pgsql now works better together with connection poolers and similar database middleware: prepared statements now use protocol level prepare instead of PREPARE sql commands and there is a new option to explicitly create a primary key on flex tables. Other fixes and features: * check flat node file format for forwards compatibility to prepare for future changes of the flatnode file format * fix two-stage processing when flat node file is used * fix getting node locations in slim node with two-stage processing * updated included libosmium, protozero and fmt libraries
Release 2.0.1 This release fixes some small issues with 2.0.0: * fix forwarding of `--schema` parameter from osm2pgsql-replication to osm2pgsql * install `osm2pgsql_find_changed_ways` function in the right schema (thanks @falko17) * install osm2pgsql-gen binary by default
Release 2.0.0 This release marks a milestone in modernizing osm2pgsql. We removed a lot of accumulated cruft from more than a decade of development. This makes osm2pgsql easier to understand for users and for developers. And it has allowed us to solve some long-standing issues and will allow further improvements in the future. Major breaking changes: * The legacy format for the middle tables has been removed as well as the old non-bucket way node index. * The gazetteer output (used by Nominatim) has been removed. * Several command line options have been removed and others are checked more strictly. * There are some new library requirements and Lua is not optional any more. * The `add_row()` function in the flex Lua config has been removed, use `insert()` instead. * Check that Lua functions on OSM object are called correctly using the colon-syntax. * Handling of untagged objects and object attributes has changed. Please see the [Upgrading appendix](https://osm2pgsql.org/doc/manual.html#upgrading) for the details of these changes. We suggest you update to version 1.11.0 first and resolve any issues you see there before upgrading to version 2.0.0. This is the first release that deprecates the "pgsql" output, please start moving towards the "flex" output instead. See [this FAQ entry](https://osm2pgsql.org/doc/faq.html#the-pgsql-output-is-deprecated-what-does-that-mean) for the details. New features in the flex output: * Two-stage processing now also supports node members of relations, not only way members * Optionally build id index as unique index * Allow setting the names of indexes in the database * New after_nodes|ways|relations() processing functions in flex Lua config files * Make osm2pgsql properties available in Lua in osm2pgsql.properties table * Add get_bbox() function to geometries in Lua * Allow empty config file in flex output, useful for some corner uses Changes in the generalization code: * Add tile-based generalizer calling SQL commands * Fix: Do not run ANALYZE in append mode, autovacuum will do that for us * Fix: Handle errors in threads correctly stopping the program with an error message Other fixes and features: * Make --flat-nodes also work in non-slim mode, useful if memory is tight * Fix off-by-one error in expire code generating out of bounds tiles * Property changes are stored later to database to avoid changing the database if anything breaks * Report (up to 100) missing nodes in the input file (in debug log) * Simplified code for area assembly from multipolygon relations * Replication: guess state from file when state info is not available * Flush and close COPYs after nodes, ways, and relations in flex output avoid COPYs that are open for a long time * Remove special case for old PostGIS versions when clustering * Avoid looking for parents of new nodes and ways in the database middle speeding up changes * As always: Lots of code cleanups, refactorings and small fixes
Release 1.11.0 This release makes the [new middle database format](https://blog.jochentopf.com/2023-07-25-improving-the-middle-of-osm2pgsql.html) the default. If you have not switched already, you need to reimport your database to take advantage of that. We have changed the way we are parsing the command line options. The new code uses the [CLI11](https://github.com/CLIUtils/CLI11) library (a copy of which is included in the repository) and is much cleaner and also much stricter. You now get warnings (and sometimes errors) for many combinations of options that don't make sense. Please check the output from osm2pgsql and osm2pgsql-replication for such messages and fix your command lines accordingly. Note especially that duplicated options are not allowed any more. This can happen, for instance, when using osm2pgsql-replication which adds the database connection parameters (such as `-d`) when it calls osm2pgsql. If all goes well this will be the last release starting with a 1. We are planning for a version 2.0.0 in the second quarter of 2024. In that release we will remove all the functionality that has been deprecated. We will also remove support for the legacy database middle format and only support the new format introduced in version 1.9.0. Further changes: * The number of database connections that osm2pgsql was opening could be quite large as it was depending on the number of tables. This is no longer the case. Osm2pgsql is opening far fewer connections now, usually you will not need to change the PostgreSQL `max_connections` settings any more. * Osm2pgsql now adds the context (the part of osm2pgsql responsible for a database connection) and the connection number to the application name used in the database connection. This allows you to better monitor what osm2pgsql is doing using the `pg_stat_activity` table in the database. * Bugfix: Using the new database format with `-x, --extra-attributes` did not work due to a wrong SQL command. This is fixed now. Many thanks to [Thunderforest](https://www.thunderforest.com/) who supported development of the features in this release.
Release 1.10.0 This is a relatively small but still important release. The new middle table format has changed slightly: the `tags` field can now be `NULL`. This makes storage more efficient and indexing faster. The new middle format is now declared stable and production ready. To use it, use the command line option `--middle-database-format=new`, in a future version of osm2pgsql this will become the new default. If you have used this option already with one of the 1.9.x versions of osm2pgsql you have to reload your database or use this SQL command to update the table: `ALTER TABLE <name> ALTER COLUMN tags DROP NOT NULL;`, for `<name>` use `planet_osm_nodes`, `planet_osm_ways`, and `planet_osm_rels` or the equivalents if you are using a different table name prefix. Other changes: * Emit a warning that the flex output `area` type and the `add_row()` functions are deprecated if you use them. If you get this warning, read https://osm2pgsql.org/doc/tutorials/switching-from-add-row-to-insert/ * Add first/last timestamps to expire tables. Having these timestamps allows various expire/updating strategies. * The `docs` directory is now called `man`, because it only contains the man pages. All other docs are on the project web site. * Various improvements on the (still experimental) generalization code. The biggest change is that we switch from using the CImg to the OpenCV library which makes the code an order of magnitude faster.
Release 1.9.2 This release fixes a bug introduced in 1.9.0 with two-stage processing that will lead to crashes. If you are using any 1.9.x version, please upgrade to 1.9.2. In one case we had some performance problems updating an osm2pgsql database with 1.9.1 due to the PostgreSQL query planning choosing a bad plan. This release contains a workaround for that problem. We also improved the (experimental) generalizer code a bit: * More information is shown in log level 'info', including some timing information. * The Lua config `run_sql()` command now can have either a single SQL statement in the `sql` field (as before) or a list of SQL commands. * For convenience, the Lua config `run_sql()` command now has an optional `transaction` field which can be set to `true` to wrap the SQL commands in BEGIN/COMMIT. * The new `if_has_rows` fields on the `run_sql()` command can be set to string with an SQL query. If that field is set, the SQL statement(s) in the `sql` field is only run, if the SQL query returns at least one row. * Some performance improvements in low-level code in the generalizer.
Release 1.9.0 This release brings three new major features: * a new osm2pgsql_properties table that saves command line options and reuses them on updates * a new database middle saves raw OSM data in JSONB format and is explicitly designed to be queried by the user * the new (and still experimental) osm2pgsql-gen adds geometry generalization to osm2pgsql Other changes include: * cleanup of schema handling * tile expiry output into database tables * a new `spherical_area()` function for flex config files to calculate the area of a (multi)polygon on the sphere. * when using the new database middle, the `--middle-with-nodes` option allows you to store all tagged nodes in the database (with their tags and location). * several improvements to osm2pgsql-replication to make it more flexible and better tested (thanks to @amandasaurus and @JakobMiksch) * don't do multi-statement SQL queries to be compatible with the PgPool-II connection pooler. Please note that this version drops support for implicit DB schema other than public. If you rely on implict user schemas or custom schema paths, you now must configure the schema to be used with the `--schema` option. For more information on all new features and changes read the more extensive release notes for 1.9.0 at https://osm2pgsql.org/news/.
PreviousNext