feat: Switch geodata providers #7393

camdecoster · 2025-03-20T03:12:59Z

Description

Adds scripts to build topojson from UN sourced data and updates build process to run these scripts.

Closes #7334

Changes

Adds script for downloading shapefiles, geojson
Adds script for converting shapefiles, geojson into topojson
Updates topojson path references

Testing

Run npm run build_topojson and make sure the script completes successfully
Try loading the maps in ./dist/topojson in Mapshaper
Open the test dashboard
Try loading some of the geo plots and look for errors

Notes

The new maps are built from this UN data source for countries, coastlines, and lands layers. The oceans, lakes, rivers, and subunits layers are built from Natural Earth Data.
The current maps come from sane-topojson, which is entirely derived from Natural Earth Data
I haven't been able to verify the resolution of the data from the UN. There's a reference here to the "UNGeodata 25 million scale", but I haven't been able to confirm what that means. As such, the labels '50m' and '110m' probably aren't accurate. But they need to remain as they are to not break references.
Mapshaper can't handle antimeridian cutting, so features that cross the antimeridian can look weird (source)
I want to credit sane-topojson for some of the design of these scripts. I'm attempting to create a (mostly) drop in replacement, so it made sense to follow those conventions.

TODO

devtools/test_dashboard/devtools.js

tasks/topojson/process_geodata.mjs

ndrezn · 2025-03-20T13:43:09Z

tasks/topojson/get_geodata.mjs

+import config from './config.json' assert { type: 'json' };
+
+try {
+    // Download data from UN


As these origin URLs may change (link rot), I worry that this piece of the script could become out of date quickly. I'm not sure there's too much we can do here but I might recommend committing the current source file (but not adding it to the dist/. Maybe someone else has a better idea.

That seems reasonable. The script could be updated to look for the file first and only download if it doesn't exist.

@marthacryan do you have an opinion or ideas on this? What's the ideal dev UX here?

FYI, in 3b47c67 I added the archive from the UN.

Co-authored-by: Nathan Drezner <[email protected]>

gvwilson · 2025-03-25T14:05:52Z

Sync with this branch.
npm i to install/update packages.
npm run build_topojson to build JSON files.
Go to mapshaper.org.
Load europe_50m.json.
Initially looks like the lakes are misplaced (for example, the Caspian Sea seems to overlap the Russian border in a weird way in the screenshot).
On closer inspection, maybe this is valid: I'm unable to find a map showing how national borders cross the Caspian Sea, and Russia might very well have a claim to the northwest section.
The three lakes in the north check out: one is in Sweden, the other two are near St. Petersburg.

gvwilson · 2025-03-25T14:08:10Z

Q for @camdecoster : did the old usa map have the states? as you say, it's not much use without that…

gvwilson · 2025-03-25T14:50:48Z

Trying to get npm run test-jasmine to work with the new topojson files, it looks like I'm tripping over a configuration issue.

Original error message complained about not being able to find ../../../dist/topojsonworld_110m.json (note the missing '/' between 'topojson' and 'world').
I modified getTopojsonPath in src/lib/topojson_utils.js as follows to insert the '/' if required:

 topojsonUtils.getTopojsonPath = function(topojsonURL, topojsonName) {
-    return topojsonURL + topojsonName + '.json';
+    if (topojsonURL.endsWith('/')) {
+       return topojsonURL + topojsonName + '.json';
+    } else {
+       return topojsonURL + '/' + topojsonName + '.json';
+    }
 };

I also modified test/jasmine/karma.conf.js as follows because we're getting the JSON files from the local dist directory not the installed sane-topojson package:

-var pathToSaneTopojsonDist = path.join(__dirname, '..', '..', 'node_modules', 'sane-topojson', 'dist');
+var pathToTopojsonDist = path.join(__dirname, '..', '..', 'dist', 'topojson');

// ... and further down, edit to match the new variable name

-        {pattern: pathToSaneTopojsonDist + '/**', included: false, watched: false, served: true}
+        {pattern: pathToTopojsonDist + '/**', included: false, watched: false, served: true}

This still produces error messages like the one shown below saying that the JSON files can't be found:

	Failed: plotly.js could not find topojson file at ../../../dist/topojson/world_110m.json. Make sure the *topojsonURL* plot config option is set properly.
	Error: plotly.js could not find topojson file at ../../../dist/topojson/world_110m.json. Make sure the *topojsonURL* plot config option is set properly.
	    at Object.<anonymous> (/Users/gvwilson/plotly/plotly.js/src/plots/geo/geo.js:137:35 <- /private/var/folders/w2/l51fjbjd25n9zbwkz9fw9jp00000gn/T/0c95a0b81672d3a2e3f59fc0534657ef-bundle.js:147433:31)
	    at Object.event (/Users/gvwilson/plotly/plotly.js/node_modules/@plotly/d3/d3.js:504:42 <- /private/var/folders/w2/l51fjbjd25n9zbwkz9fw9jp00000gn/T/0c95a0b81672d3a2e3f59fc0534657ef-bundle.js:4875:48)
	    at XMLHttpRequest.respond (/Users/gvwilson/plotly/plotly.js/node_modules/@plotly/d3/d3.js:1951:24 <- /private/var/folders/w2/l51fjbjd25n9zbwkz9fw9jp00000gn/T/0c95a0b81672d3a2e3f59fc0534657ef-bundle.js:6326:30)
25 03 2025 10:40:03.170:WARN [web-server]: 404: /dist/topojson/world_110m.json
[note: the message above is then repeated several times]

I've tried variations on the path in test/jasmine/tests/geo_test.js specified by this line:

Plotly.setPlotConfig({ topojsonURL: '../../../dist/topojson' });

Adding an extra .. or removing one of the ones that's there doesn't affect the outcome.

camdecoster · 2025-03-25T14:59:26Z

@gvwilson yes the old maps had 'subunits' layers for a few regions (USA and Brazil did for sure). The UN maps don't include that information. Thanks for looking into the tests. I'm working on an update to get the tests working but I haven't pushed that yet. I'll try to get that out later today.

ndrezn · 2025-03-25T15:05:11Z

@etpinard when you have a chance could you give this PR a look? Would love your eyes. Thank you!

In the short term what was the source for US states/Canadian provinces in the old dataset?

Refactor to simplify scripts Switch to UN/NE geodata hybrid Save UN geodata to archive Remove extra info from topojson Add centroids to geojson Use 'simplify' to create 110m maps

…data-providers

This is necessary because Mapshaper applies changes to a child layer to the parent as well (for some commands).

…data-providers

archmoj · 2025-07-07T20:20:55Z

devtools/test_dashboard/devtools.js

@@ -17,8 +17,7 @@ var Tabs = {
        Plotly.setPlotConfig({

            // use local topojson files
-            topojsonURL: '../../node_modules/sane-topojson/dist/',
-
+            topojsonURL: "../../dist/topojson/",


During development (e.g. using test dashboard), it would be better to depend on a file in the build folder instead of the dist folder.
The changes to the dist folder should be limited to releases.

I'll look into this in combination with your comment in preprocess.js.

archmoj · 2025-07-07T20:22:59Z

draftlogs/7393_change.md

@@ -0,0 +1 @@
+- Switch to United Nations/Natural Earth geodata for building topojson used in geo plot


Do we still depend of Natural Earth?

We do for subunit maps (US states, Canadian provinces, etc.).

Thanks for clarification.
Let's mention that in the draftlog please.

archmoj · 2025-07-07T20:30:53Z

package.json

@@ -130,6 +136,7 @@
    "check-node-version": "^4.2.1",
    "chttps": "^1.0.6",
    "css-loader": "^7.1.2",
+    "d3-geo": "^3.1.1",


So d3-geo is upgraded from v1 to v3.
This looks fine. But I'm wondering if it was necessary?
Also if we could do that in a separate PR?

Also d3-geo is used in the source code. So it should be listed in dependencies not dev-dependencies.

I missed that it was included previously. I'll revert this change.

FYI - I tested bumping d3-geo to v3 on this branch:
https://github.com/plotly/plotly.js/compare/d3-geo-v3
Several baselines need small changes there which are not significant.
To avoid merge conflict, we could be fine with bumping d3-geo and updating the source of topojson in one PR as is. But please add a note in the draftlog of the PR mentioning the d3-geo bump.
Thank you 🙏

I'd like to do that in a separate PR, so I'll remove it from this one.

src/lib/topojson_utils.js

archmoj · 2025-07-07T20:40:57Z

tasks/preprocess.js

@@ -72,13 +71,3 @@ function writeLibFiles(obj) {
        );
    }
 }
-
-// copy topojson files from sane-topojson to dist/


Perhaps we could keep this process to copy topojson files from topojson or build folder to dist?

I'll look into that in combination with your comment on devtools.js

archmoj · 2025-07-07T20:44:53Z

tasks/topojson/config.mjs

@@ -0,0 +1,109 @@
+const config = {
+    resolutions: [50, 110],


sane-topojson used to provide 50 and 110 resolutions.
Is there any difference between 50 and 110 resolutions now that the UN dataset is used?

The 110m maps are a simplified version of the 50m maps. I haven't been able to get any confirmation as to what the UN data resolution actually is, so these labels aren't necessarily correct. In absence of a better solution, I left them as is.

Noting that the topojson files are overwritten on the CDN during the release process here:

plotly.js/tasks/cdn_publish.sh

Line 32 in 5e2163b

cp $dist/topojson/* $sync

this PR could have sudden impact on users using old and new versions.
Is that what we really want? Or should we rename new topojson files to avoid this issue?
cc: @gvwilson @emilykl @ndrezn @alexcjohnson

archmoj · 2025-07-07T20:47:19Z

dist/topojson/africa_110m.json

The dist files should be replaced during the release process only.

Does that mean that I should remove all of the new topojson files until I generate a release PR?

I think they could be generated inside the build folder as part of pretest or preprocess.
Alternatively we could move all these extra steps and dev dependencies to a folder e.g. topojson. Similar to stackgl_modules also test building & potential changes on the CI:

plotly.js/.circleci/config.yml

Lines 473 to 490 in 5e2163b

test-stackgl-bundle:

docker:

- image: cimg/node:18.20.4

working_directory: ~/plotly.js

steps:

- checkout

- run:

name: Set up build environment

command: cd stackgl_modules && npm ci

- run:

name: Bundle bundle-stackgl/index.js

command: cd stackgl_modules && cp index.js INDEX.js && npm run bundle-stackgl

- run:

name: Test stackgl_modules/index.js diff - If failed please remember this file in auto generated and you should not modify it directly until a dependeny change. To suggest changes please submit pull request to the relevant dependency.

command: diff --unified --color stackgl_modules/INDEX.js stackgl_modules/index.js

- store_artifacts:

path: stackgl_modules/index.js

destination: stackgl_modules/index.js

The dist folder should be updated when calling npm run build and npm run cibuild.

archmoj · 2025-07-07T21:21:50Z

@camdecoster Great work 💪 🥇
There is a notable jump between long horizontal border between Canada and US when comparing the diff of test/image/baselines/geo_text_chart_arrays.png.
Wondering if adding more vertices on such segments (as part of the post-process script) could help improve the results?

camdecoster added 4 commits March 19, 2025 18:16

Add new dev dependencies

e8d763d

Add build scripts

7f7b690

Update some topojson references

6b596c0

Add npm build scripts

353bf94

gvwilson assigned ndrezn Mar 20, 2025

gvwilson added feature something new P1 needed for current cycle labels Mar 20, 2025

camdecoster added 2 commits March 20, 2025 07:13

Add 'usa' config

fc6022f

Add clipping bounds for Antarctica

2910eea

camdecoster changed the title ~~Switch geodata providers~~ feat: Switch geodata providers Mar 20, 2025

ndrezn reviewed Mar 20, 2025

View reviewed changes

ndrezn requested a review from emilykl March 20, 2025 17:03

camdecoster and others added 6 commits March 20, 2025 11:40

Apply suggestions from code review

5d66dad

Co-authored-by: Nathan Drezner <[email protected]>

Add geodata archive, update get_geodata script paths

3b47c67

Switch config file to JS

b434656

Switch coastlines layer source

a4d0b21

Add Oceania clipping bounds

e257b7b

Switch layer name to 'ocean'

1154ff1

camdecoster added 7 commits March 25, 2025 20:42

Update test config

86d5b00

Switch to UN/NE geodata hybrid

86f048c

Refactor to simplify scripts Switch to UN/NE geodata hybrid Save UN geodata to archive Remove extra info from topojson Add centroids to geojson Use 'simplify' to create 110m maps

Merge remote-tracking branch 'origin/master' into cam/7334/switch-geo…

ac61095

…data-providers

Fix zip/unzip of UN geojson

f4c3d48

Fix saving 110m maps

22a8e0a

Update dist with new maps

247f925

Update geo tests

3cf9730

camdecoster added 11 commits June 25, 2025 14:55

Fix merge issues with package-lock

d7d7d0b

Update required dev dependencies again

eb1e966

Split out antimeridian fixes

60171d7

This is necessary because Mapshaper applies changes to a child layer to the parent as well (for some commands).

Update topojson

90554a2

Update test baseline images

d836db9

Update geo tests

3ddc6e0

Fix comment

f952173

Further update geo tests

75e9f52

Update geo test with values from CircleCI

3a7bd66

Fix linting/formatting of new JS files

a6efc84

Merge remote-tracking branch 'origin/master' into cam/7334/switch-geo…

4f4bbfc

…data-providers

archmoj reviewed Jul 7, 2025

View reviewed changes

src/lib/topojson_utils.js Outdated Show resolved Hide resolved

archmoj reviewed Jul 7, 2025

View reviewed changes

camdecoster added 11 commits July 7, 2025 15:33

Revert to previous version of 'd3-geo'

434017f

Refactor helper function

9a4a966

Update draftlog

1f336e1

Update test baselines

1bd2fc2

Move topojson build script files

6dae5aa

Update paths

665c892

Add topojson package files

ae3c471

Add maps to topojson dist

a2cce65

Update library paths

7112d33

Restore previous topojson

6e25a5f

Remove scripts, dependencies from package.json

ecd469f

		@@ -0,0 +1 @@
		- Switch to United Nations/Natural Earth geodata for building topojson used in geo plot

	test-stackgl-bundle:
	docker:
	- image: cimg/node:18.20.4
	working_directory: ~/plotly.js
	steps:
	- checkout
	- run:
	name: Set up build environment
	command: cd stackgl_modules && npm ci
	- run:
	name: Bundle bundle-stackgl/index.js
	command: cd stackgl_modules && cp index.js INDEX.js && npm run bundle-stackgl
	- run:
	name: Test stackgl_modules/index.js diff - If failed please remember this file in auto generated and you should not modify it directly until a dependeny change. To suggest changes please submit pull request to the relevant dependency.
	command: diff --unified --color stackgl_modules/INDEX.js stackgl_modules/index.js
	- store_artifacts:
	path: stackgl_modules/index.js
	destination: stackgl_modules/index.js

Uh oh!

feat: Switch geodata providers #7393

Are you sure you want to change the base?

feat: Switch geodata providers #7393

Uh oh!

Conversation

camdecoster commented Mar 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changes

Testing

Notes

TODO

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gvwilson commented Mar 25, 2025

Uh oh!

gvwilson commented Mar 25, 2025

Uh oh!

gvwilson commented Mar 25, 2025

Uh oh!

camdecoster commented Mar 25, 2025

Uh oh!

ndrezn commented Mar 25, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

archmoj Jul 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

archmoj commented Jul 7, 2025

Uh oh!

Uh oh!

camdecoster commented Mar 20, 2025 •

edited

Loading

archmoj Jul 7, 2025 •

edited

Loading