Error while trying to paint a very large map

[neko259]

When trying to generate a map larter than 4096 pixels I get an error:
Warning: The width or height of the image to be created exceeds 4096 pixels! (Dimensions: 61872x61872)
I tried a workaround to scale it down, but the scale must be an integer more than 1, so that does not help either. There is also no setting to split the map to a set of tiles (that would be useful to run a server with OSM-based engine to load the map chunks dynamically).

[sfan5]

Creating an image of that size would take more than 10 GiB of RAM.
You need to limit the size when generating.

[neko259]

How can I do it without cropping the map and losing its part?

[HybridDog]

You'd generate a giant png image. Viewing it e.g. with an image viewer would not work because it'd need to allocate 4 bytes for each pixel.

There's a related issue l think: FLIF-hub/FLIF#12
If the FLIF image format supports that, it could be used for minetestmapper, couldn't it?

[paramat]

There is also no setting to split the map to a set of tiles

Generate multiple 4000x4000 maps and edit them together in an image editor.

[neko259]

I need to update the map on daily basis. Are you suggesting to edit the tiles manually each day instead of changing a couple lines of code? I worked with different open-source projects, but this is way out of line. 24 мар. 2018 г. 11:18 пользователь "Paramat" <[email protected]> написал: There is also no setting to split the map to a set of tiles Generate multiple 4000x4000 maps and edit them together in an image editor. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#48 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAvCBu14g9frxRslo4VpFlBJxmMMcRaSks5thg9kgaJpZM4QSI-e> .

[HybridDog]

For starters, you could add the possibility of setting the coordinates in node precision, leaftest requires this feature.

[paramat]

neko259 yes i agree, for frequent updating editing would be hard work.
Auto splitting an area into 4000x4000 tiles would be a good feature but you would still need to edit them together afterwards, so i can't see a way around that.

[HybridDog]

You can't simply edit them together. If you try to view such a big PNG image file the image viewer crashes because w * h * 4 Bytes need to be allocated where the image can be loaded into.

[ChimneySwift]

I'm surprised that there isn't an option to generate an entire map file in chunks, then scale each chunk down and stitch the final together to create an 8K or so image, but granted that would be no easy task.

I would have thought however, that if you try to generate an entire map it would tell you that you might run out of ram, and ask if you want to stop, not keep going, use up all your ram faster than you can close it and lock up your entire system.

[wziard]

I have created an (experimental for now) fork of minetestmapper with tiled output.

https://github.com/wziard/minetestmapper

there are some small problems at the edges with the height shading, but for the rest it works well.

[Poikilos]

Images larger than 8192x8192 will not load in browsers via JavaScript. If you want some kind of map that updates regularly, using small tiles and updating the tiles likely to change (such as the edges) may work, but locking large chunks of the database constantly will cause lag or "rubber banding" (players will place a block and it will go away because the server can't keep up)--maybe not with redis, but access should still be timed as opposed to constant. I have some work done on splitting up the map with mtanalyze (requires metadata to place the images, so uses a forked minetestmapper-python but I'm working on a redo using the C++ version now that Option for fixed image size is closed--even though my metadata idea was neither implemented nor addressed at that closed issue) but the corresponding PHP code there was written when I lacked skill at RESTful javascript.

If there were some way to know what parts had no data, then those tiles could be updated more frequently.
:edit: the metadata suggested at #49 is no longer needed, because now there is a noemptyimage option--map boundaries can be found that way (if image wasn't created, the region has no data). The other metadata is not needed since you know what region you used and now that is always the region of the output image, so you can write your own metadata (or organize and name tile images like 0/0/x0z0.jpg)

I look forward to trying wziard's fork and seeing how that can help as well.
Bump: #67 (implements tiled output, and addresses this issue (#48 ))

[cheapie]

This is a known bug in libgd: libgd/libgd#122

[GSunway]

Hi, im using Minetestmapper on windows10 and when I try to use it it gives me this error

Exit code:1
Exception: Failed to allocate 65536x65536 image

Could you help me to fix this?
Have a nice day. :)

[Poikilos]

@GSunway Even if you could make an image that big, you probably couldn't edit it unless it was a PSB (Photoshop Big) image or something. There is usually no purpose for creating an image that large (in any field, not just game development). At 32-bit (4 bytes per pixel), it would take up 16 GB of storage space uncompressed or of RAM when uncompressed to load it (65536x65536x4/1024/1024/1024) plus metadata.

• The image is larger than the maximum world size, so that doesn't make much sense unless you are using zoom. If not, maybe it is a bug unrelated to this issue. Please show the exact command you are using. A program or image viewer would be more efficient by doing the zoom itself (with nearest pixel scaling a.k.a. no interpolation) instead of having minetestmapper add more pixels (it would be a waste).
• What is the image for? Try limiting the size using the --geometry option. If you are trying to make some kind of map viewer or Leaflet, tiles of some reasonable size (probably "power of 2" sizes such as 1024x1024 but usually smaller) will make your program run more smoothly.

[cheapie]

@GSunway Even if you could make an image that big, you probably couldn't edit it unless it was a PSB (Photoshop Big) image or something. There is usually no purpose for creating an image that large (in any field, not just game development). At 32-bit (4 bytes per pixel), it would take up 16 GB of storage space uncompressed or of RAM when uncompressed to load it (65536x65536x4/1024/1024/1024) plus metadata.

16GB is hardly an unreasonable amount of RAM for a computer to have these days, and in some quick testing here GIMP seems to have no issues (aside from not being very fast) creating/editing/saving/loading a 65536x65536 PNG image.

The image is larger than the maximum world size

Not by very much, IIRC.

[Poikilos]

@cheapie,

• GIMP may be usable in a test case, but not in real editing scenarios apparently: https://www.reddit.com/r/GIMP/comments/3l91oi/gimp_becomes_unstable_handling_huge_images_any/
• For any kind of reasonable OpenGL speed, the image would have to go in video memory anyway, which is usually far less than 16GB. An RTX 3080 is ‎$699.99 and has 10GB.
• Some versions of OpenGL seem to simply show a black image if the image is roughly bigger than the screen--maybe due to cards with limited VRAM such as integrated ones.
• Notice that 2D games generally use tiles (or at most tile atlasses to reduce texture loading/unloading, but still reasonably-sized), Google Maps and other map sites use tiles, and Leaflet uses tiles--because doing so reduces latency and resource usage.
• Yeah, having 16GB is common, but, if you use 16GB of RAM on an image, forget about running an OS or any programs.
• Using such huge images is still incredibly wasteful. Even if you break the map apart, having tiles larger than 1024x1024 is going to be slow at some point: during loading, generating, scrolling, whatever--where the bottleneck occurs will depend on the hardware and software. Diagnosing exactly why the image is so huge is, let's say, a huge issue.