Search API: Non-latin place tokens/keywords are wrongly used when no accept-language.

[stouch]

Following to #3132, I open this new discussion because I think that the specific case I previously described can occur for different queries and this need a specific thread.

As a reminder, this is the original issue :
Aisne is a french county. Aisne is translated in Greek : Αιν
So, once this name is tokenized in keywords, one of the token is : ain (because ν matches with the latin char n)
Problem: When I query "Ain", consequently it gives me Aisne... while I'm expecting Ain which is actually another french county.
If I set accept-language to en_GB, then "Ain" well returns Ain, and not Aisne.

So, I have several questions about this issue, that might occur for many places translated with different alphabets around the world and which tokens are wrongly mapped with the latin charset :

1. "Ain" is not equal to "Αιν", could the tokenizer use all the chars available in utf8 including non-latin chars ?

2. The keywords (+importance) seem to have more prominance in queries than exact match, is this OK ? (= the quick fix you suggested in the last thread is something realistic ?)

3. When there is no accept-language defined, should it not be the original-country translation (=default language name) that'd have the more prominance ? So for example, without accept-language, If I type Ain, the matching should first use all the original language/default name, find the most probable which is Ain. Only if this first search does not give any result, it would try another search with all the keywords. I guess this has not be done because we all write places with our own chars instead of local chars and so for example, looking for Moscou, a query as I described would first look for Moscou in all the default names and so Москва would not be found, then the search would find for Москва thanks to its full list of keywords. But anyway we could have several sets of keywords, I'm not sure of which would be the best approach to this.

[lonvia]

As already said in the other thread, this has already been fixed in the new Python frontend. Mismatching scripts now result in a penalty. You can try that on nominatim.openstreetmap.org which now runs the Python frontend.

Penalties for lesser known names is a different story and needs a completely different search index. I have ideas but an implementation is still far away in the future.

[stouch]

I'm not sure to get it . Can I use this new python API search with https://github.com/mediagis/nominatim-docker ?

[mtmail]

Possibly if you change the Dockerfile to point to the current master branch. But that's a question for the nominatim-docker repository maintainers, it's a separate project.

[stouch]

Hi, as this is possible now, I upgraded to 4.3 my docker container, and it seems that "Ain" still return "Aisne" using the /search endpoint. This is still fixed in https://nominatim.openstreetmap.org/ui/search.html?q=Ain so I guess I got a cache. Or do I need to set the accept-language always to en (Actually when I try /search endpoint with accept-language=en, it well returns "Ain", but with accept-language=fr it still return "Aisne"). @lonvia

[mtmail]

In an earlier thread mediagis/nominatim-docker#470 I wrote Nominatim 4.3 will have the Python frontend, that got delayed. It was possible to enable it for testing but afaik nobody with the Docker setup did this. Python frontend became default in Nominatim 4.4. https://github.com/osm-search/Nominatim/releases/tag/v4.4.0

[stouch]

Ah! As I was able to request the /search endpoint, I thought I had access to this new frontend. I gonna upgrade to 4.4 and I'll tell you if everything is OK , thanks a lot

[stouch]

Following this thread, and as mentioned below, @mtmail, I still dont the expected results.
I was asking if there is some cache (or does the last docker nominatim image 4.4 uses well the python backend?).

Indeed, in official API: https://nominatim.openstreetmap.org/ui/search.html?q=Ain&accept-language=fr gives well "Ain" in first result, while my API with docker nominatim still results in "Aisne" :

/search?q=Ain&accept-language=fr :

[
	{
		"place_id": 9273468,
		"licence": "Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright",
		"osm_type": "relation",
		"osm_id": 7411,
		"boundingbox": [
			"48.8372137",
			"50.0693878",
			"2.9579507",
			"4.2557496"
		],
		"lat": "49.453285449999996",
		"lon": "3.606899003594057",
		"display_name": "Aisne, Hauts-de-France, France métropolitaine, France",
		"place_rank": 12,
		"category": "boundary",
		"type": "administrative",
		"importance": 0.735482477945902
	},

and for en language , I got the good results :
/search?q=Ain&accept-language=en

[
	{
		"place_id": 9285955,
		"licence": "Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright",
		"osm_type": "relation",
		"osm_id": 7387,
		"boundingbox": [
			"45.6107563",
			"46.5199006",
			"4.7281684",
			"6.1700273"
		],
		"lat": "46.06551335",
		"lon": "5.28478031423462",
		"display_name": "Ain, Auvergne-Rhône-Alpes, Metropolitan France, France",
		"place_rank": 12,
		"category": "boundary",
		"type": "administrative",
		"importance": 0.7289198530506933
	},

My setup is

docker run -d --shm-size=2g  \
 -e PBF_URL=https://ftp5.gwdg.de/pub/misc/openstreetmap/planet.openstreetmap.org/pbf/planet-latest.osm.pbf  \
 -e REPLICATION_URL=https://ftp5.gwdg.de/pub/misc/openstreetmap/planet.openstreetmap.org/replication/day/  \
 -e IMPORT_STYLE=/usr/local/etc/nominatim/styles/import-custom.lua   \
 -e IMPORT_WIKIPEDIA=/nominatim/extras/wikimedia-importance.sql.gz   \
 -e UPDATE_MODE=catch-up   \
 -v ./styles:/usr/local/etc/nominatim/styles  \
 -v /mnt/nominatim/extras:/nominatim/extras  \
 -v /mnt/nominatim/flatnode:/nominatim/flatnode  \
 -v /mnt/nominatim/postgres-data:/var/lib/postgresql/14/main \
 -v ./postgres-config:/etc/postgresql/14/main/conf.d  \
 -p 8080:8080  \
 --name nominatim  
 mediagis/nominatim:4.4

(with a recent catch-up update),

and with import-custom.lua:

local flex = require('flex-base')

flex.set_main_tags{
    boundary = {administrative = 'named'},
    landuse = 'fallback',
    place = 'always'
}

flex.set_prefilters{delete_keys = {'building', 'source', 'highway',
                                   'addr:housenumber', 'addr:street', 'addr:city',
                                   'source', '*source', 'type',
                                   'is_in:postcode', '*:wikidata', '*:wikipedia',
                                   '*:prefix', '*:suffix', 'name:prefix:*', 'name:suffix:*',
                                   'name:etymology', 'name:signed', 'name:botanical',
                                   'addr:street:name', 'addr:street:type'},
                    delete_tags = {landuse = {'cemetry', 'no'},
                                   boundary = {'place'}},
                    extratag_keys = {'wikipedia', 'wikipedia:*', 'wikidata', 'capital', 'openGeoDB:postal_codes', 'fixme:addr:postcode'}
                   }

flex.set_name_tags{main = {'name', 'name:*',
                          'int_name', 'int_name:*',
                          'nat_name', 'nat_name:*',
                          'reg_name', 'reg_name:*',
                          'loc_name', 'loc_name:*',
                          'old_name', 'old_name:*',
                          'alt_name', 'alt_name:*', 'alt_name_*',
                          'official_name', 'official_name:*',
                          'place_name', 'place_name:*',
                          'short_name', 'short_name:*'},
                   extra = {'ref', 'int_ref', 'nat_ref', 'reg_ref',
                            'loc_ref', 'old_ref',
                            'iata', 'icao', 'pcode', 'pcode:*', 'ISO3166-2'}
                  }

flex.set_address_tags{extra = {'addr:*', 'is_in:*'},
                      postcode = {'postal_code', 'postcode', 'addr:postcode'},
                      country = {'country_code', 'ISO3166-1',
                                 'addr:country_code', 'is_in:country_code',
                                 'addr:country', 'is_in:country'},
                      postcode_fallback = false
                     }

flex.set_unused_handling{extra_keys = {'place'}}

and with wikimedia importance downloaded at :
Nov 17 2019 wikimedia-importance.sql.gz

[lonvia]

mediagis/nominatim-docker#507 has an open PR to add support for the Python frontend. Please try with this. Questions about it should go to the docker repository.

[mtmail]

Oh no, I was sure the Docker project switched to the Python frontend implementation already :/

[stouch]

I upgraded to 4.4 and I still got that :

curl --request GET \
  --url 'http://<MY_NOMINATIM_HOST>:8080/search?q=Ain&accept-language=fr'

returns:

[
	{
		"place_id": 9273468,
		"licence": "Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright",
		"osm_type": "relation",
		"osm_id": 7411,
		"boundingbox": [
			"48.8372137",
			"50.0693878",
			"2.9579507",
			"4.2557496"
		],
		"lat": "49.453285449999996",
		"lon": "3.606899003594057",
		"display_name": "Aisne, Hauts-de-France, France métropolitaine, France",
		"place_rank": 12,
		"category": "boundary",
		"type": "administrative",
		"importance": 0.735482477945902
	},
	{

@mtmail @lonvia

It must have some cache no?

[lonvia]

Have you switched to the Python frontend?

[stouch]

I use mediagis/nominatim:4.4 as it comes (@mtmail mentioned above that Python frontend is the default one) :

docker run -d --shm-size=2g  \
 -e PBF_URL=https://ftp5.gwdg.de/pub/misc/openstreetmap/planet.openstreetmap.org/pbf/planet-latest.osm.pbf  \
 -e REPLICATION_URL=https://ftp5.gwdg.de/pub/misc/openstreetmap/planet.openstreetmap.org/replication/day/  \
 -e IMPORT_STYLE=/usr/local/etc/nominatim/styles/import-custom.lua   \
 -e IMPORT_WIKIPEDIA=/nominatim/extras/wikimedia-importance.sql.gz   \
 -e UPDATE_MODE=catch-up   \
 -v ./styles:/usr/local/etc/nominatim/styles  \
 -v /mnt/nominatim/extras:/nominatim/extras  \
 -v /mnt/nominatim/flatnode:/nominatim/flatnode  \
 -v /mnt/nominatim/postgres-data:/var/lib/postgresql/14/main \
 -v ./postgres-config:/etc/postgresql/14/main/conf.d  \
 -p 8080:8080  \
 --name nominatim  
 mediagis/nominatim:4.4