Importing the entire planet

[aviNrich]

Hi.
Is it possible to import the entire world to the database?
If so, can you share the correct docker commands?

Thanks

[leonardehrenfried]

Yes, it's possible but will require a machine with 64GB of memory: https://nominatim.org/release-docs/latest/admin/Installation/#hardware

The original planet file from https://planet.openstreetmap.org/ is throttled so you want to use a mirror from: https://wiki.openstreetmap.org/wiki/Planet.osm#Planet.osm_mirrors

Choose a mirror near your server for the best download speed.

If you're in Europe you could use the German mirror. I haven't tested it myself but i think it's this command:

docker run -it \
  -e PBF_URL=https://ftp5.gwdg.de/pub/misc/openstreetmap/planet.openstreetmap.org/pbf/planet-latest.osm.pbf \
  -e REPLICATION_URL=https://ftp5.gwdg.de/pub/misc/openstreetmap/planet.openstreetmap.org/replication/day/ \
  -p 8080:8080 \
  --name nominatim \
  mediagis/nominatim:3.7

You might need to also play around with the tuning parameters documented in the wiki.

Please report back if it works.

[aviNrich]

Thank you.

1. Can you please estimate the load time 4 cores + 64GB?
2. Once imported, can I reduce the memory?

Thanks

[mtmail]

Prepare for 5-10 days of importing. It's easier to estimate once the first couple of ranks are imported.

Disc speed is also a factor, use the fastest discs (or network mounted storage) you can find.

After the import you can lower the RAM.

[yikakabu]

After the import, the server running normally with ocuppied about 10 GB memory. But the memory usage starts to increase from a certain point, and now takes up 30GB.
What happened about the sudden growth? And How much memory should supply to daily use.

[MichaelKorn]

Thanks for the information here! Some recent insights:
I did the import on GCP with n2-highmem-16 (16 vCPUs, 128 GB) and SSD Disk. It took about 62 hours (+download time for pbf file).
Directly after the import 958GB were occupied on the nominatim disk (including 64GB for pbf file) (separate boot disk).

After the imported I scaled down to n2-custom-6-49152 (6 vCPUs, 48 GB) and "balanced persistent disk", it is running fine! It seems a smaller machine could also be fine, PostgreSQL is doing a great job.

The downloaded pbf file was a little bit outdated (June 11th) and daily updates ware applied (until today, June 24th). After the updates the nominatim disk is with 976GB occupied.

[leonardehrenfried]

Great update, thanks!

[pgassmann]

I started Nominatim 4.1 with planet import on a Hetzner Root Server AX101 with AMD Ryzen™ 9 5950X (16 cores) and 128 GB DDR4 ECC

It took around 2 days to import everything. plus ~1 night to get up-to-date with nominatim replication.

It uses now 428G on ZFS (zfs with lz4 compression)

zfsdata/container  referenced            428G                   -
zfsdata/container  compressratio         2.12x                  -

[leonardehrenfried]

Thanks for that!

Did you use the defaut command line options or did you customise it a bit?

[mtmail]

Open a new question for your specific setup

[meyerovb]

Why would you run an rds? You're already paying for an ec2 to host the software, why not just run postgres on it? U use rds ur now paying data transfer fee (2 cents per gb but still) and the server spec for the postgres would probably rival the ec2, doubling your cost. Unless I'm guessing wrong here.

[darkBuddha]

Import of Germany is possible on a M1 MacBook, volume size is 99 GB after import. It takes 8-12 hours, I used 12 GB RAM for the podman VM.

[yikakabu]

After the import, the server running normally with ocuppied about 10 GB memory. But the memory usage starts to increase from a certain point, and now takes up 30GB.
What happened about the sudden growth? And How much memory should supply to daily use.

[meyerovb]

Would using a 128gb ram 32 core cpu make it substantially faster for the initial load than 64gb ram 16 cores?

[mtmail]

duplicate of #428

[nikitadtastewise]

Upload OSM data North America 12.3 GB + US Postcode.

Infrastructure configuration for initialization:

1. AWS RDS Postgres 14.6, db.r6g.xlarge (4vCPU, 32GB, arm), 300GB, 3000 IOPS, 125Mb/s throughput, volume autoscaling. Default AWS Postgres DB parameters.
2. EKS Job pod request 3vCPU + 6GB memory.
3. Enable flatnode feature to speed up process.

Total time initialize: 37 hours.
Total db size: 182 GB.

[ligoo]

The whole planet import on:

• AMD Ryzen 7 7700, 8 cores, 16 threads
• 64 GB DDR5 RAM
• 2x1 TB Gen4 NVMe SSDs

Total time: ~38 hours.
Total DB size: 884G + 82G flatnode = 966G

The import was done with the IMPORT_WIKIPEDIA, the IMPORT_US_POSTCODES, IMPORT_GB_POSTCODES and IMPORT_TIGER_ADDRESSES options to true and shm_size to 16gb.

[sungwoo-lifeoverflow]

The whole planet import ( IMPORT_STYLE=admin )

Docker image : mediagis/nominatim:4.3
Machine : aws ec2 m5.2xlarge ( 32GB RAM & vCPU 8 ) & EBS 1TB

At EC2, before run docker

1. make folder mkdir -p /nominatim-data/share (for docker mount)
2. cd /nominatim-data/share & manual download planet-latest.osm.pbf (71.x GB) ~1.5hr (delete after init)
3. mkdir -p /nominatim-data/postgresql (for docker mount)
4. mkdir -p /nominatim-data/flatnode (for docker mount)

"init command (init.sh)"

#!/bin/sh

HOST_CACHE_DIR_PATH="/nominatim-data"
NOMINATIM_PASSWORD="nominatim_password"

docker run -it \
  --shm-size=8g \
  -e THREADS=8 \
  -e PBF_PATH=/nominatim/share/planet-latest.osm.pbf \
  -e REPLICATION_URL=https://ftp5.gwdg.de/pub/misc/openstreetmap/planet.openstreetmap.org/replication/day \
  -e NOMINATIM_PASSWORD=$NOMINATIM_PASSWORD \
  -e IMPORT_STYLE=admin \
  -v $HOST_CACHE_DIR_PATH/postgresql:/var/lib/postgresql/14/main \
  -v $HOST_CACHE_DIR_PATH/flatnode:/nominatim/flatnode \
  -v $HOST_CACHE_DIR_PATH/share:/nominatim/share \
  -p 8080:8080 \
  --name nominatim \
  mediagis/nominatim:4.3

• run init script & simple logs (~24 hours)

2023-10-11 09:51:59: (run init.sh)
2023-10-12 05:01:42 - rank 20
2023-10-12 06:27:57: Done 2285867 in 7330 @ 311.833 per second - rank 20 ETA (seconds): 2301.67
2023-10-12 06:48:54: Done 2681964 in 8586 @ 312.332 per second - rank 20 ETA (seconds): 1029.80
2023-10-12 09:09:29: Done 1344800 in 1445 @ 930.395 per second - rank 0 ETA (seconds): 465.01
2023-10-12 09:31:34: Done 125874/125874 in 60 @ 2074.746 per second - FINISHED postcodes (location_postcode)
2023-10-12 09:33:26: Post-process tables
2023-10-12 09:58:27: Setting up website directory at /nominatim/website

• Final Size ( DB+flatnode = 321G + 84G = 405G )

# Terminal
[... nominatim-data] sudo du -d 1 -h
84G     ./flatnode
72G     ./share ---> (planet-latest.osm.pbf, not necessary)
321G    ./postgresql
476G    .

[leonardehrenfried]

Thanks for the report. These are valuable for others wanting to import the entire planet.

[willb0]

bless up bro ty

[Curiosity007]

The whole planet import ( IMPORT_STYLE=admin )

Docker image : mediagis/nominatim:4.3
Machine : aws ec2 m5.2xlarge ( 32GB RAM & vCPU 8 ) & EBS 1TB

At EC2, before run docker

1. make folder mkdir -p /nominatim-data/share (for docker mount)
2. cd /nominatim-data/share & manual download planet-latest.osm.pbf (71.x GB) ~1.5hr (delete after init)
3. mkdir -p /nominatim-data/postgresql (for docker mount)
4. mkdir -p /nominatim-data/flatnode (for docker mount)

"init command (init.sh)"

#!/bin/sh

HOST_CACHE_DIR_PATH="/nominatim-data"
NOMINATIM_PASSWORD="nominatim_password"

docker run -it \
  --shm-size=8g \
  -e THREADS=8 \
  -e PBF_PATH=/nominatim/share/planet-latest.osm.pbf \
  -e REPLICATION_URL=https://ftp5.gwdg.de/pub/misc/openstreetmap/planet.openstreetmap.org/replication/day \
  -e NOMINATIM_PASSWORD=$NOMINATIM_PASSWORD \
  -e IMPORT_STYLE=admin \
  -v $HOST_CACHE_DIR_PATH/postgresql:/var/lib/postgresql/14/main \
  -v $HOST_CACHE_DIR_PATH/flatnode:/nominatim/flatnode \
  -v $HOST_CACHE_DIR_PATH/share:/nominatim/share \
  -p 8080:8080 \
  --name nominatim \
  mediagis/nominatim:4.3

• run init script & simple logs (~24 hours)

2023-10-11 09:51:59: (run init.sh)
2023-10-12 05:01:42 - rank 20
2023-10-12 06:27:57: Done 2285867 in 7330 @ 311.833 per second - rank 20 ETA (seconds): 2301.67
2023-10-12 06:48:54: Done 2681964 in 8586 @ 312.332 per second - rank 20 ETA (seconds): 1029.80
2023-10-12 09:09:29: Done 1344800 in 1445 @ 930.395 per second - rank 0 ETA (seconds): 465.01
2023-10-12 09:31:34: Done 125874/125874 in 60 @ 2074.746 per second - FINISHED postcodes (location_postcode)
2023-10-12 09:33:26: Post-process tables
2023-10-12 09:58:27: Setting up website directory at /nominatim/website

• Final Size ( DB+flatnode = 321G + 84G = 405G )

# Terminal
[... nominatim-data] sudo du -d 1 -h
84G     ./flatnode
72G     ./share ---> (planet-latest.osm.pbf, not necessary)
321G    ./postgresql
476G    .

Is there a way to export the entire docker structure or the folders inside the docker, and replicate in local Ubuntu box?

I would require ec2 to create the initial configuration, but once it is done, I can comfortably run it in my local machine. Is there a way to do this?

[mikeresnik]

I ran into out of memory / out of space issues with the simple m5.2xlarge ec2 instance at 1TB.
Currently am trying a m5.4xlarge (64gb) with 1.5TB storage since I remember seeing that 64gb - 128gb the minimum recommended.
Hoping to make this into an AMI and spin up on machines with lower RAM once running.

I am also importing US postcodes, GB postcodes, and TIGER data which adds to the data size some.

[leonardehrenfried]

Thanks for the report. They are useful because both nominatim and the data evolves over time and hardware requirements change.

[lnbs97]

For reference: importing Germany took me 10hrs and reverse geocoding 55k coordinate pairs took 16mins.

Container Size: 108GB
System: AMD Ryzen 5 3600, 32GB RAM, 1TB NVMe SSD

[nikitadtastewise]

Import multiple regions version 4.3.0:

• North America 14.6 GB + US and GB Postcode + TIGER data
• Australia and Oceania 1.2 GB
• France 4.5 GB
• Germany 4.2 GB
• India 1.46 GB
• Spain 1.2 GB
• United Kingdom 1.78 GB

Hardware:

• RDS Postgres 14.10, db.r6g.xlarge 4vCPU, 32GB, 550GB
• EC2 r5.xlarge 4vCPU, 32GB, 200GB root, 150GB flatnode file

ETA

• Import multiple regions + US and GB postcodes extra ~ 3 days 18 hours
• Import tigger data ~ 9 hours 15 mins

[vovavili]

Did anybody try USA only import here? Is this feasible to do on a 16GB RAM Windows laptop, with 12 GB dedicated to WSL2? I am thinking of running the following command:

docker run -it -v nominatim-flatnode:/nominatim/flatnode -e PBF_URL=https://download.geofabrik.de/north-america/us-latest.osm.pbf -e REPLICATION_URL=https://download.geofabrik.de/north-america/us-updates/ -e IMPORT_US_POSTCODES=true -e work_mem=30MB -e maintenance_work_mem=600MB -e effective_cache_size=6GB -e autovacuum_max_workers=1 -e max_connections=100 -p 8080:8080 --name nominatim mediagis/nominatim:4.4

[0sha-dow0]

this should work as i have usa data installed in a vm with 16gb ram im not quite sure about the work_mem=30MB -e maintenance_work_mem=600MB -e effective_cache_size=6GB but i did run the normal docker run command and it work with a load of approx 200tps

[stevetracvc]

I'll update with the time once it's done. UTC times

2024-10-06 01:33:06: Using project directory: /nominatim
...
2024-10-06 11:49:36  Reading input files done in 36987s (10h 16m 27s).
2024-10-06 11:49:36    Processed 9416831157 nodes in 2648s (44m 8s) - 3556k/s
2024-10-06 11:49:36    Processed 1053257706 ways in 29565s (8h 12m 45s) - 36k/s
2024-10-06 11:49:36    Processed 12565219 relations in 4774s (1h 19m 34s) - 3k/s
...
2024-10-06 17:26:34: Load data into placex table
2024-10-06 19:50:33: Setting up tokenizer
2024-10-06 19:50:35: Calculate postcodes
...
2024-10-06 21:08:07: Starting rank 1 (using batch size 1)
...
2024-10-07 01:42:25: Starting rank 22 (using batch size 1)
...
2024-10-09 06:44:32: Done 1718333/1718333 in 2948 @ 582.811 per second - FINISHED rank 0
... (ran out of drive space at start of postcode indexing, took a day for AWS to expand and snapshot so I wouldn't mess up continuing the init)
2024-10-10 13:27:00: Starting indexing postcodes using 1 threads
...
2024-10-10 14:53:18: Done 2775266/2775266 in 5178 @ 535.932 per second - FINISHED postcodes (location_postcode)

I pre-downloaded the pbf via S3.

x8g.xlarge, 64GB RAM and 4 vCPUs

HOST_CACHE_DIR_PATH="/opt/nominatim"

docker run -it \
  --shm-size=16g \
  -e THREADS=8 \
  -e PBF_PATH=/nominatim/share/planet-240930.osm.pbf \
  -e REPLICATION_URL=https://planet.openstreetmap.org/replication/day/ \
  -e IMPORT_STYLE=address \
  -e IMPORT_US_POSTCODES=true \
  -e IMPORT_GB_POSTCODES=true \
  -e IMPORT_TIGER_ADDRESSES=true \
  -e IMPORT_WIKIPEDIA=true \
  -v $HOST_CACHE_DIR_PATH/postgresql:/var/lib/postgresql/14/main \
  -v $HOST_CACHE_DIR_PATH/flatnode:/nominatim/flatnode \
  -v $HOST_CACHE_DIR_PATH/share:/nominatim/share \
  -p 8080:8080 \
  --name nominatim \
  mediagis/nominatim:4.4

[mikeresnik]

So attempting this again with an m5.4xlarge (64gb) with 1.5TB storage failed just now.
Not sure if extratags requires a beefier box.

Post-process tables
/app/init.sh: line 92:   111 Killed                  sudo -E -u nominatim nominatim import --osm-file $OSMFILE --threads $THREADS

this was the command I used to start docker:

docker run -it \
  --shm-size=32g \
  -e THREADS=8 \
  -e PBF_PATH=/nominatim/share/planet-latest.osm.pbf \
  -e REPLICATION_URL=https://ftp5.gwdg.de/pub/misc/openstreetmap/planet.openstreetmap.org/replication/day \
  -v $HOST_CACHE_DIR_PATH/postgresql:/var/lib/postgresql/14/main \
  -v $HOST_CACHE_DIR_PATH/flatnode:/nominatim/flatnode \
  -v $HOST_CACHE_DIR_PATH/share:/nominatim/share \
  -e IMPORT_STYLE=extratags \
  -e IMPORT_US_POSTCODES=true \
  -e IMPORT_GB_POSTCODES=true \
  -e IMPORT_TIGER_ADDRESSES=true \
  -p 8080:8080 \
  -p 5432:5432 \
  --name nominatim \
  mediagis/nominatim:4.4

Going to reattempt with a larger box (more memory, same disk space), m5.8xlarge (128gb RAM).
Will bump shm-size to 64g as well and report back here in about a week when it finishes or crashes again.

[stevetracvc]

you DON'T have to start from scratch, I started mine back up where it crashed. Just make sure you don't do anything yet

[mikeresnik]

That's nice to know, how do you start it back up from where it crashed?

[stevetracvc]

I made a snapshot before doing anything, just in case I screwed something up. But that took about 12 hours on my 1TB drive. So I dunno if you want to wait for that.

These are the important scripts to read
https://github.com/mediagis/nominatim-docker/blob/master/4.4/start.sh
https://github.com/mediagis/nominatim-docker/blob/master/4.4/init.sh

start.sh is the one called when the container starts up. So what I did was replace it with one that just ran cat so that the docker container would run indefinitely. Then, I ran docker exec -it nominatim bash to drop into a bash shell. From there, I manually continued the import. I also moved the init.sh script to init.sh.orig so it wouldn't accidentally run (I did this from my host machine, docker cp files back and forth to the existing container).

also, copy an empty file to /var/lib/postgresql/14/main/import-finished inside the container, just in case, before restarting the container.

The most important part is whether it got to line 31 in start.sh or not. If it finishes the init.sh script, then it makes a file /var/lib/postgresql/14/main/import-finished and will skip init the next time the container is started. This prevents it from deleting the postgres and flatnode stuff

The really long parts of importing are the indexing, line 102 in init.sh
sudo -E -u nominatim nominatim index --threads $THREADS
My issue was that it crashed on the last step of that, so I had to search the github repo for what the index command actually did. It looks like yours is done with that, you're in

https://github.com/osm-search/Nominatim/blob/6299afcad75e57ef28375201a9078cfaad057546/src/nominatim_db/clicmd/setup.py#L56

if you look a couple lines above, there's

if args.continue_at in \
            ('import-from-file', 'load-data', 'indexing', None):

which means that you can actually call nominatim with the --continue db-postprocess flag. You'll need to manually do that from the command line. Here's what I ran once I got to where you are, from inside the docker container:

# from the top of the init.sh script
tailpid=0
replicationpid=0
topServices() {   service apache2 stop;   service postgresql stop
  if [ $replicationpid -ne 0 ]; then     echo "Shutting down replication process";     kill $replicationpid;   fi;   kill $tailpid
  exit 0; }
trap stopServices SIGTERM TERM INT

######## new stuff here, to pick up where yours died
cd /nominatim/
sudo -E -u nominatim nominatim import --continue db-postprocess --threads $THREADS

####### continue init.sh
sudo -E -u nominatim nominatim admin --check-database

if [ "$REPLICATION_URL" != "" ]; then
  sudo -E -u nominatim nominatim replication --init
  if [ "$FREEZE" = "true" ]; then
    echo "Skipping freeze because REPLICATION_URL is not empty"
  fi
else
  if [ "$FREEZE" = "true" ]; then
    echo "Freezing database"
    sudo -E -u nominatim nominatim freeze
  fi
fi

export NOMINATIM_QUERY_TIMEOUT=600
export NOMINATIM_REQUEST_TIMEOUT=3600
if [ "$REVERSE_ONLY" = "true" ]; then
  sudo -H -E -u nominatim nominatim admin --warm --reverse
else
  sudo -H -E -u nominatim nominatim admin --warm
fi
export NOMINATIM_QUERY_TIMEOUT=10
export NOMINATIM_REQUEST_TIMEOUT=60

# gather statistics for query planner to potentially improve query performance
# see, https://github.com/osm-search/Nominatim/issues/1023
# and  https://github.com/osm-search/Nominatim/issues/1139
sudo -E -u nominatim psql -d nominatim -c "ANALYZE VERBOSE"

sudo service postgresql stop

# Remove slightly unsafe postgres config overrides that made the import faster
rm /etc/postgresql/14/main/conf.d/postgres-import.conf

echo "Deleting downloaded dumps in ${PROJECT_DIR}"
rm -f ${PROJECT_DIR}/*sql.gz
rm -f ${PROJECT_DIR}/*csv.gz
rm -f ${PROJECT_DIR}/tiger-nominatim-preprocessed.csv.tar.gz

if [ "$PBF_URL" != "" ]; then
  rm -f ${OSMFILE}
fi

Now you should be able to stop the container, replace the start.sh file with the original, and restart the container to get it to finish.

You should double check everything before running what I said, but AFAIK that's all the steps I took. Saved me about 3 days of reindexing.

[stevetracvc]

Again, you should follow the bash scripts and the python code a bit just to make sure it's not missing anything, but I'm pretty sure yours died AFTER this part

        if args.continue_at in \
            ('import-from-file', 'load-data', 'indexing', None):
            LOG.warning('Indexing places')
            indexer = Indexer(args.config.get_libpq_dsn(), tokenizer, num_threads)
            await indexer.index_full(analyse=not args.index_noanalyse)

which means it should be done with that index_full function