Upgrade Node 14 to Node 20 #1439

chris-bateman · 2023-11-19T23:48:21Z

Now upgraded to node 20 and webpack 5

Upgrade Node from 14 to 16 and associated packages and WebPack config.

The proposed change is because Node 14 is EOL and some packages are no longer supported.
https://endoflife.date/nodejs
Node 16 is still EOL but I wanted to introduce these changes gradually.

Node is now installed in the Dockerfile using the modern install method. The old script method is deprecated.

Replace ExtractTextPlugin with MiniCssExtractPlugin due to it being deprecated -

I upgraded WebPack slightly to ensure MiniCssExtractPlugin works.

Dockerfile compiled correctly on local system.
Tests completed successfully on local system.

ASDC Disclaimer - These changes will benefit the ASDC fork for supporting some custom frontend packages.

Upgrade Node from 14 to 16 and associated packages and WebPack config

chris-bateman · 2023-11-20T00:11:57Z

A few CSS issues are appearing, so will work on that.

pierotofy · 2023-11-20T01:43:46Z

If we're making the effort to upgrade node/webpack & friends, maybe this should be updated to the latest stable release of node, not 16, unless there's something that's being blocked with staying on 14 (are these the custom frontend packages?)

chris-bateman · 2023-11-20T22:31:14Z

The logic behind moving to 16 was to reduce the blast radius of such a large jump between 14 to 18. However, 16 is still EOL so best practice would be to move to 18. I will give it a try and see what the impact is. It should hopefully be easier* (famous last words) since I have done the move to 16 essentially.

Later versions of Uppy and webpack 5 aren't happy with Node 14 hence the upgrade.

Upgraded to node 18 and Webpack 5. Package cleanup for webpack 5

chris-bateman · 2023-11-20T23:41:18Z

@pierotofy in theory Node 18 and webpack 5.
Tests pass locally and in the GH action.
Basic build, login and check various pages and actions appear to work.

Please have a review and let me know if you have any questions or comments.

smathermather · 2023-11-21T00:30:30Z

@chris-bateman -- any reason why 21.04 instead of LTS 22.04? No comments otherwise and a happy to test. Thanks for this!

chris-bateman · 2023-11-21T00:33:35Z

@chris-bateman -- any reason why 21.04 instead of LTS 22.04? No comments otherwise and a happy to test. Thanks for this!

Didn't touch the docker file Ubuntu version as I didn't want to change too many things. I was going to consider changing that in another PR. But overall I do agree a bump there would be good.

smathermather · 2023-11-21T16:15:44Z

Ah, I forgot that was 21 already! Yeah, a bump if possible is always good, but not changing everything at once is useful.

Edit:
Now running as my daily. No issue yet, but no deeper testing. Anything other than "hey, this loads and I don't see issues!" do you want to see for testing? So far so good on my end.

chris-bateman · 2023-11-21T22:08:11Z

@smathermather quick note about a bump to the Ubuntu version. I just gave it a try and it's more than a 1 line fix. A few packages need to be adjusted. So yeah that will be a to do item for another time.

Let me know if anything else is needed for this to be merged and released.

smathermather · 2023-11-22T14:27:22Z

If there's no objection, I'll merge today.

pierotofy

This breaks the map view for me:

Probably due to:

    fallback: {
      "buffer": false
    }

package.json

Dockerfile

pierotofy · 2023-11-22T16:48:14Z

Now running as my daily.

Is the version number showing as 3.0.0 in the about section?

smathermather · 2023-11-22T17:30:08Z

Hmm. Nope. Looks like somehow I built wrong.

chris-bateman · 2023-11-22T19:37:55Z

This breaks the map view for me:

Probably due to:
    fallback: {
      "buffer": false
    }

I will take a look.

chris-bateman · 2023-11-22T23:33:17Z

This breaks the map view for me:

Probably due to:
    fallback: {
      "buffer": false
    }

Issue resolved. Just working on node 20 and seeing what happens there.

Added buffer as webpack 5 does not use it anymore. Version change in package.json

bump to node 20

chris-bateman · 2023-11-23T02:02:13Z

Latest run failed but just due to a network timeout connecting to Ubuntu. Happens every now and then.
Built locally just fine on my local system with Node 20. So just trigger a rebuild and it should be fine.

pierotofy · 2023-11-25T22:25:28Z

Thanks for the additional changes, but this is now breaking shapefile import in the map view. You should be able to drag & drop zipped shapefiles into a project (test files and link to a working instance of WebODM included).

test.zip

https://cloud1.webodm.net/public/task/329b1d24-826e-4cbf-a220-2e19be0349e1/map/

If you've upgraded webpack, test whether you've accidentally broken anything Javascript related. If you add a polyfill for Buffer, search for anything related to Buffer (in this case, shapefile import, perhaps other stuff) 🙏

smathermather · 2023-11-26T09:31:59Z

@pierotofy -- do we maintain a list of things to test (recognising this might have been caught without such a list). If not, I can at least go through pull requests and make a list of things that have broken as a starting place and we can build from there.

chris-bateman · 2023-11-26T09:58:30Z

Thanks for the additional changes, but this is now breaking shapefile import in the map view. You should be able to drag & drop zipped shapefiles into a project (test files and link to a working instance of WebODM included).

test.zip

https://cloud1.webodm.net/public/task/329b1d24-826e-4cbf-a220-2e19be0349e1/map/

If you've upgraded webpack, test whether you've accidentally broken anything Javascript related. If you add a polyfill for Buffer, search for anything related to Buffer (in this case, shapefile import, perhaps other stuff) 🙏

Thanks for testing. Comments noted and appreciated. As @smathermather mentions above, a test list would be appreciated. This could then be expanded into a test suite like Playwright in the future, for example.

I will review the impact of upgrading to WebPack 5 in more detail on a code level.

pierotofy · 2023-11-26T17:16:16Z

Webpack upgrade: touches every single Javascript output. So everything in the UI/frontend needs to be tested.
Buffer: any reference (code + libs in node_modules) should be identified and tested.

We have a minimal test suite for the frontend, but it's nowhere near exhaustive (see all .test.jsx files). Manual testing is still required.

chris-bateman · 2023-11-26T23:22:54Z

Thanks for the additional changes, but this is now breaking shapefile import in the map view. You should be able to drag & drop zipped shapefiles into a project (test files and link to a working instance of WebODM included).

test.zip

https://cloud1.webodm.net/public/task/329b1d24-826e-4cbf-a220-2e19be0349e1/map/

If you've upgraded webpack, test whether you've accidentally broken anything Javascript related. If you add a polyfill for Buffer, search for anything related to Buffer (in this case, shapefile import, perhaps other stuff) 🙏

This particular issue is now fixed. I will push the commit shortly.

Please feel free to test further. I will continue checking a few things in other sections of the frontend as well.

smathermather · 2023-12-05T22:51:01Z

Do we need more testing on this? Happy to run some tests if so.

chris-bateman · 2023-12-05T22:52:48Z

Do we need more testing on this? Happy to run some tests if so.

I am using it for the day to day ASDC workloads without issue but they are probably vastly different from day to day "vanilla" usage. Maybe release a beta to common power users?

smathermather · 2023-12-06T02:23:51Z

Is there something other than throwing the --dev flag that I should do to test it?

pierotofy · 2023-12-06T04:13:15Z

Yes.

docker build -t opendronemap/webodm_webapp .

Then ./webodm.sh start --dev

smathermather · 2023-12-08T13:27:15Z

Yes.

docker build -t opendronemap/webodm_webapp .

Then ./webodm.sh start --dev

Cool. In that case currently running this as a daily and have been for a few days. Though version still shows as 2.4.0, which is a bit confusing. Perhaps I'm not running it?

Sorry for the confusion: I never touch WebODM, always ODM or NodeODM for testing, so I'm a very old n00b.

chris-bateman · 2023-12-08T20:01:23Z

2.4.0 is right. I changed it to that from 3.

…

Sent from my Galaxy

-------- Original message -------- From: Stephen Mather ***@***.***> Date: 9/12/23 12:27 am (GMT+10:00) To: OpenDroneMap/WebODM ***@***.***> Cc: Chris Bateman ***@***.***>, Mention ***@***.***> Subject: Re: [OpenDroneMap/WebODM] Upgrade Node 14 to Node 20 (PR #1439) Yes. docker build -t opendronemap/webodm_webapp . Then ./webodm.sh start --dev Cool. In that case currently running this as a daily (for real this time). Though version still shows as 2.4.0, which is a bit confusing. Perhaps I'm not running it? Sorry for the confusion: I never touch WebODM, always ODM or NodeODM for testing, so I'm a very old n00b. — Reply to this email directly, view it on GitHub<#1439 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AC5EAF2KTH3WBHWJQVLMS7DYIMIT7AVCNFSM6AAAAAA7SCVKWGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNBXGE3DINZYGY>. You are receiving this because you were mentioned.Message ID: ***@***.***>

smathermather · 2023-12-11T16:43:42Z

Still running in production over here without issue, but I've asked @Saijin-Naib to kick the tires a bit too as I haven't had time to do any deeper testing yet.

Saijin-Naib · 2023-12-11T17:50:32Z

Oh, you want me to break stuff? That I can do 🤣

smathermather · 2023-12-21T14:58:11Z

Still running as my daily, so far without incident.

@Saijin-Naib -- any chance to kick the tires?

Saijin-Naib · 2023-12-21T19:24:00Z

Not yet!

smathermather · 2024-01-02T19:32:16Z

Still not seeing any issues running this on the daily. Do we have any remaining reservations about this pull request?

pierotofy · 2024-01-02T19:56:20Z

I think it's good.

Thank you @chris-bateman 🙏

chris-bateman · 2024-01-02T20:01:32Z

Happy new year everyone thanks for taking the time to test this.

Upgrade Node 14 to Node 16

3f42eaa

Upgrade Node from 14 to 16 and associated packages and WebPack config

Bump to node 18

6ea180d

Upgraded to node 18 and Webpack 5. Package cleanup for webpack 5

chris-bateman changed the title ~~Upgrade Node 14 to Node 16~~ Upgrade Node 14 to Node 18 Nov 20, 2023

pierotofy requested changes Nov 22, 2023

View reviewed changes

package.json Outdated Show resolved Hide resolved

pierotofy reviewed Nov 22, 2023

View reviewed changes

Dockerfile Outdated Show resolved Hide resolved

chris-bateman added 2 commits November 23, 2023 12:32

added buffer package and version change

dae49a7

Added buffer as webpack 5 does not use it anymore. Version change in package.json

bump to node 20

5ca0cf8

bump to node 20

chris-bateman changed the title ~~Upgrade Node 14 to Node 18~~ Upgrade Node 14 to Node 20 Nov 23, 2023

chris-bateman requested a review from pierotofy November 23, 2023 02:01

Buffer Polyfill for WP5

8ff81f7

pierotofy merged commit be8bd6e into OpenDroneMap:master Jan 2, 2024
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade Node 14 to Node 20 #1439

Upgrade Node 14 to Node 20 #1439

chris-bateman commented Nov 19, 2023 •

edited

Loading

chris-bateman commented Nov 20, 2023

pierotofy commented Nov 20, 2023

chris-bateman commented Nov 20, 2023

chris-bateman commented Nov 20, 2023

smathermather commented Nov 21, 2023

chris-bateman commented Nov 21, 2023

smathermather commented Nov 21, 2023 •

edited

Loading

chris-bateman commented Nov 21, 2023

smathermather commented Nov 22, 2023

pierotofy left a comment

pierotofy commented Nov 22, 2023

smathermather commented Nov 22, 2023

chris-bateman commented Nov 22, 2023

chris-bateman commented Nov 22, 2023

chris-bateman commented Nov 23, 2023

pierotofy commented Nov 25, 2023 •

edited

Loading

smathermather commented Nov 26, 2023

chris-bateman commented Nov 26, 2023

pierotofy commented Nov 26, 2023

chris-bateman commented Nov 26, 2023

smathermather commented Dec 5, 2023 •

edited

Loading

chris-bateman commented Dec 5, 2023

smathermather commented Dec 6, 2023

pierotofy commented Dec 6, 2023

smathermather commented Dec 8, 2023 •

edited

Loading

chris-bateman commented Dec 8, 2023 via email

smathermather commented Dec 11, 2023

Saijin-Naib commented Dec 11, 2023

smathermather commented Dec 21, 2023

Saijin-Naib commented Dec 21, 2023

smathermather commented Jan 2, 2024

pierotofy commented Jan 2, 2024

chris-bateman commented Jan 2, 2024

Upgrade Node 14 to Node 20 #1439

Upgrade Node 14 to Node 20 #1439

Conversation

chris-bateman commented Nov 19, 2023 • edited Loading

Now upgraded to node 20 and webpack 5

chris-bateman commented Nov 20, 2023

pierotofy commented Nov 20, 2023

chris-bateman commented Nov 20, 2023

chris-bateman commented Nov 20, 2023

smathermather commented Nov 21, 2023

chris-bateman commented Nov 21, 2023

smathermather commented Nov 21, 2023 • edited Loading

chris-bateman commented Nov 21, 2023

smathermather commented Nov 22, 2023

pierotofy left a comment

Choose a reason for hiding this comment

pierotofy commented Nov 22, 2023

smathermather commented Nov 22, 2023

chris-bateman commented Nov 22, 2023

chris-bateman commented Nov 22, 2023

chris-bateman commented Nov 23, 2023

pierotofy commented Nov 25, 2023 • edited Loading

smathermather commented Nov 26, 2023

chris-bateman commented Nov 26, 2023

pierotofy commented Nov 26, 2023

chris-bateman commented Nov 26, 2023

smathermather commented Dec 5, 2023 • edited Loading

chris-bateman commented Dec 5, 2023

smathermather commented Dec 6, 2023

pierotofy commented Dec 6, 2023

smathermather commented Dec 8, 2023 • edited Loading

chris-bateman commented Dec 8, 2023 via email

smathermather commented Dec 11, 2023

Saijin-Naib commented Dec 11, 2023

smathermather commented Dec 21, 2023

Saijin-Naib commented Dec 21, 2023

smathermather commented Jan 2, 2024

pierotofy commented Jan 2, 2024

chris-bateman commented Jan 2, 2024

chris-bateman commented Nov 19, 2023 •

edited

Loading

smathermather commented Nov 21, 2023 •

edited

Loading

pierotofy commented Nov 25, 2023 •

edited

Loading

smathermather commented Dec 5, 2023 •

edited

Loading

smathermather commented Dec 8, 2023 •

edited

Loading