Unserialize Error

[tao]

After generating about ~3800 pages I receive this error:

   ErrorException

  unserialize(): Error at offset 8148 of 8178 bytes

  at vendor/spatie/fork/src/Task.php:115
    111▕         $output = $this->output;
    112▕
    113▕         if (str_starts_with($output, self::SERIALIZATION_TOKEN)) {
    114▕             $output = unserialize(
  ➜ 115▕                 substr($output, strlen(self::SERIALIZATION_TOKEN))
    116▕             );
    117▕         }
    118▕
    119▕         return $output;

      +20 vendor frames
  21  please:37

I got this with php please ssg:generate --workers=2 first and decided to try a classic php please ssg:generate but I received the same error too. After this latest update: if you don't specify the workers but still have spatie/fork code installed does it still call the fork code but runs it on 1 thread by default?

During the async call with two workers, it didn't crash immediately but a little while later this happened:

ErrorException

  socket_write(): unable to write to socket [32]: Broken pipe

  at vendor/spatie/fork/src/Connection.php:63
     59▕             }
     60▕
     61▕             $length = strlen($payload);
     62▕
  ➜  63▕             $amountOfBytesSent = socket_write($this->socket, $payload, $length);
     64▕
     65▕             if ($amountOfBytesSent === false || $amountOfBytesSent === $length) {
     66▕                 break;
     67▕             }

      +21 vendor frames
  22  please:37

I can try uninstall spatie/fork code and test the site generation again if that helps, and I'll see if I can try this on another machine with more ram and report back.

[tao]

I tried this with --workers 4 on my 10 year old quad-core Debian machine with 4 gigs of RAM. I was able to generate 4218 pages in ~20 mins without this error... so it's probably just my MacBook Air isn't suitable for such a large site.

I'll keep experimenting with it but the information may be useful if you want to add a warning or try / catch for this kind of error.

[globalexport]

Sadly, the same error can still happen. I am ran into it locally on a MacBook Pro after approx. 450 pages (4 workers).

But I am quite surprised about your performance (4218 pages in 20 min). What kind of structure do those pages have? Complex/simple, with/without images.

[globalexport]

I was not able to try-catch the error. It just broke the worker processes. Did you manage to do so?

[globalexport]

Maybe related: spatie/fork#22

[globalexport]

Too much payload to deserialize here:
https://github.com/statamic/ssg/blob/HEAD/src/Generator.php#L290

If you have only 2 workers, the closure will consist of half of the pages. If you increase the number of workers, the payload will decrease.

I suspect that it is a conceptual bug (wrong library).

Edit: On the other hand, the return value might not be supposed to carry other data. I was storing sitemap information here. The error disappeared after moving the sitemap logic out of the closure.

[jasonvarga]

Thanks for the research. That's sounding like it could be the problem.

We could potentially send lists of references to entries/terms/etc to the fork closures instead of the objects themselves, then re-fetch the items within the closures.

[simonhamp]

I'm bumping into this a lot... has anyone started anything to solve it?