Importing regions is taking too long

[manideep0610]

Hi everyone,
I am adding new databases with nominatim add-data. I first added luxemberg and now I am adding France. Through this method ("add -data"). It is taking too long it's been a week now I initiated through "nominatim import --osm-file file1.pbf --osm-file file2.pbf". It is still processing, Is there any other method or any reference document I can refer to get regions added to nominatim.

Any help/suggestions appreciated.
Following below doc to add:
https://nominatim.org/release-docs/latest/admin/Advanced-Installations/
Thank you

[mtmail]

France data is 100x as big as Luxembourg data, so it will take much much longer. How much depends a lot on the hardware you use.

Software Environment (please complete the following information):

• Nominatim version:
• Postgresql version:
• Postgis version:
• OS:

Hardware Configuration (please complete the following information):

• RAM:
• number of CPUs:
• type and size of disks:

Did you add any further parameters to nominatim import, for example --threads? Or osm2pgsql cache? https://nominatim.org/release-docs/latest/admin/Import/#notes-on-memory-usage

What does 'still processing' mean? What are the last lines printed on screen or in the logfile?

[manideep0610]

@mtmail Please find below details:

Software Environment (please complete the following information):

Nominatim version: 4.4
Postgresql version: PostgreSQL 12.18 (Ubuntu 12.18-0ubuntu0.20.04.1)
Postgis version: no postgis
OS: "20.04.6 LTS (Focal Fossa)" Ubuntu

Hardware Configuration (please complete the following information):

RAM: 64G
number of CPUs: 8
type and size of disks: Balanced persistent disk/Boot disk 1000G

We have tried with "--threads"
Please find below image which is running from more than a week, Could you please suggest accordingly. If there is any other process or document provided would be great.

Thanks

[mtmail]

Ok, so it's still importing but very very slow.

• Change the postgresql settings, see https://nominatim.org/release-docs/latest/admin/Installation/#tuning-the-postgresql-database. You can also disable fsync (https://www.postgresql.org/docs/12/runtime-config-wal.html#GUC-FSYNC) but don't forget to enable it again after the import or you risk data loss.

• It's not clear what hardware you use for the harddrives. If you have spinning discs (HDD, not SSD) then those will be the bottleneck.

• Use --threads 6. This makes no difference in the 'Processing:' (osm2pgsql) step but it will help in later steps of the import and indexing.

• Use --osm2pgsql-cache 20000 (https://nominatim.org/release-docs/latest/admin/Import/#notes-on-memory-usage).

Is the machine running inside a virtual machine, any other process that might access the harddrive or database other than Nominatim?

[manideep0610]

@mtmail Thanks for the response, Please find below details as suggested :

1)Change the postgresql settings, see https://nominatim.org/release-docs/latest/admin/Installation/#tuning-the-postgresql-database. You can also disable fsync (https://www.postgresql.org/docs/12/runtime-config-wal.html#GUC-FSYNC) but don't forget to enable it again after the import or you risk data loss.

We already have the same config set and few params with higher config and fsync is off.

**shared_buffers = 20GB
maintenance_work_mem = 10GB
autovacuum_work_mem = 2GB
work_mem = 50MB
effective_cache_size = 24GB
synchronous_commit = off
max_wal_size = 1GB
min_wal_size = 80MB
checkpoint_completion_target = 0.9
checkpoint_timeout = 10min

fsync is off.**

2)It's not clear what hardware you use for the harddrives. If you have spinning discs (HDD, not SSD) then those will be the bottleneck.

We are using SSD

3. Use --threads 6. This makes no difference in the 'Processing:' (osm2pgsql) step but it will help in later steps of the import and indexing.
   4)Use --osm2pgsql-cache 20000 (https://nominatim.org/release-docs/latest/admin/Import/#notes-on-memory-usage).

We are using in below format to import/add region:

"nominatim add-data --file france-******.osm.pbf --threads 8 --osm2pgsql-cache 24000"

Please suggest what else can be done as we are with high config and the most importantly the job is still running and adding region is still in progress:

It is running in GCP-VM.

Could you please suggest and do the needful.

Thanks,
Manideep.

[mtmail]

Great, you already do all the best practices. But it's still too slow. Check postgresql if any process is blocking the insert commands (SELECT * FROM pg_stat_activity;). I'm not confident you'll find anything. Can you upgrade the VM in GCP to use faster harddrives? With AWS EC2 there are something like 5 different types of harddrives (confusing to me)

[lonvia]

nominatim add-data will always be orders of magnitude slower than nominatim import. You can speed up the process somewhat, by running nominatim refresh --functions --no-diff-updates before adding the data, but it will still be slow.

The faster way is to just drop the database and rebuild it from scratch by running nominatim import --osm-file luxemburg.odm.pbf --osm-file france.osm.pbf.