Continuous updates

[acerioni]

I am struggling to make Photon continuously update, despite the fact that "my" Nominatim regularly does. As a matter of facts, I end up having Photon exposing features with no geometry.

Can you confirm that this script, https://github.com/komoot/photon/blob/master/continuously_update_from_nominatim.sh, which was last committed more than 4 years ago, is still providing a valid recipe?

Thanks in advance for your help.

[systemed]

Is #369 relevant?

[acerioni]

In order to benefit from the latest PRs, I compiled the source code myself, instead of using the latest binaries. Nevertheless, I still have the problem that I mentioned.

I also have the impression that Nominatim's update.php script should be executed with the ' --index' parameter soon after Photon is done with the update, otherwise Nominatim updates don't work properly. Something like these:

<some_path>/update.php --no-npi --import-osmosis --no-index

curl http://<photon_host>:<photon_port>/nominatim-update

<sleep for - let's say - half an hour>

<some_path>/update.php --index

By the way, the documentation of Nominatim says that the "--no-npi" parameter is deprecated: what was that for?

Could anyone please confirm/oppose my impression?

Thanks in advance for your help,
Alessandro

[lonvia]

The script should work as is. @simonpoole did some debugging and figured out that the photon script should be setting the 'indexed' flag in Nominatim's database to signal that it is done. Otherwise Nominatim will refuse to continue with the updates.

[simonpoole]

See osm-search/Nominatim#1284 too.

I'm having a general look at the way updating works, in particular splitting NominatimUpdater so that it outputs a file with the required actions and can read such a file back in, to get rid of the requirement that you need to have a local nominatim DB for updates. But no promises, I'm at the exploratory/prototyping stage and there may be (well very likely there definitely will be) some hurdles I'm not aware of yet.

[hendrikmoree]

We've done something similar in our version of Photon / Nomitatim

We changed the update process of Nominatim to the following:

./utils/update.php --no-index --import-osmosis
sleep 10

REVISION=`psql nominatim -c "SELECT sequence_id FROM import_status WHERE indexed = false" -t -A`
psql nominatim -c "SELECT place_id, indexed_status FROM placex WHERE indexed_status > 0" -t -A -F"," > $UPDATES_DIR/places_$REVISION

sleep 10
./utils/update.php --index --import-osmosis

We've create our own updater in Photon, which just accepts an array of the changed ids in the Nominatim instance: https://github.com/flitsmeister/photon/blob/master/src/main/java/de/komoot/photon/nominatim/FMNominatimUpdater.java

No we remember the last revision of Photon and check's every x hours on the nominatim service if there's a new file with updates available.

This works perfectly for months already. Maybe it will help you find a solution

[acerioni]

Thanks a lot for your reply, @hendrikmoree.

Could you please provide me with an example of how you invoke the endpoint /fm-nominatim-update that you set up? In particular, I don't understand how to produce the JSON which has to be sent as the body of the request...

[simonpoole]

See #385 for what I've been working on. As it says it is WIP and I've only tested it running against a Nominatim instance with a small OSM extract.

[lonvia]

Should be fixed in the latest release.