Error creating folders when importing mbox files.

[chefren]

After creating a PST file from outlook, run readpst tool and obtained a set of folders and mbox files like below

• Inbox
  • Subfolder 1
    • mbox
    • .size
    • .type
  • mbox
  • .size
  • .type

The pst file came from outlook 2019
The mbox files were generated with: readpst -u -o ./ /media/input.pst

I added local files extension in thunderbird
Created an mbox local folder account
Try to import with all the mbox options re: subfolders or single file

The mbox files seem ok, but will get always an error like:

Error: Incorrect argument types for folders.create.

Error creating subfolder: Inbox

This is in Ubuntu 22.04 with latest Thunderbird (115.18.0 (64-bit)) and extension (14.1.4), a clean install.

I tried also in windows and get a more complex fault but still related to folder creation error: NS_ERROR_FILE_ACCESS_DENIED

nsresult: "0x80520015 (NS_ERROR_FILE_ACCESS_DENIED)"  location: "JS frame :: chrome://mboximport/content/mboximport/modules/ietngUtils.js :: rebuildSummary :: line 335"  data: no]

I am just using normal users and creating the local folders by the same user.

I also reproduced the error with a fresh export of an account within the same thunderbird install into a single mbox file

ie: Inbox file, import into local folder -> same error

Any help is appreciated.

[redarrow-rus]

My case is just simpler but the same result. I've got this error while trying to import a mbox file from another Thunderbird instance. Both instances are on Linux. The file permissions are correct.

Upd. Details from console:

IETNG: createSubfolder failed, retry for: Temp ietngUtils.js:367:19
IETNG: Recovery failed ietngUtils.js:376:19

Source code:

...
      // createSubfolder will fail under some circumstances when
      // doing large imports. Failures start around 250+ and become
      // persistent around 500+. The failures above 500 are likely
      // do to Windows file descriptor limits.
      // A rebuildSummary followed by a createSubfolder retry
      // recovers the operation in most circumstances.
      // Odd database behaviors have sometimes been observed
      // even if recovery succeeded

      try {
        let res = await window.WEXTcreateSubfolder(msgFolder, subFolderName);
      } catch (ex) {
        try {
          console.log(`IETNG: createSubfolder failed, retry for: ${subFolderName}`);
...

The file being imported is 2,7 Mb size.

[cleidigh]

@chefren
@redarrow-rus
Sorry this is not good and it's disconcerting considering all my successful testing. That said I think there's two separate issues.
@chefren
Your issue is that IETNG doesn't import the PST structure. It requires the sbd Subfolder structure that Thunderbird uses.. Several people have asked about supporting the PST structure and it is a consideration, but currently Thunderbird should already support this in its import capability. To use IETNG you would have to imprt the individual mbox files.

@redarrow-rus
Your issue is very troublesome considering you were just importing a single mbox with my new code. Can you roll back to v14.1.2 which has been out there for a while? I would like to see that it works. Then I will also see if I can reproduce on my Linux VM.
@cleidigh

[vineethribm]

I faced a similar issue while in version 115.6.1. Updating to latest 128.6.0esr version fixed the issue.

[redarrow-rus]

I made a rollback to 14.1.2 and confirm that a single mbox import is ok.

[cleidigh]

@redarrow-rus
Ok that's good for now...
I have to figure out some more debug. The changes I made were to use what is supposed to be a more reliable createSubfolder. There is one possible edge condition I can also look at. When I have a beta will you help with some testing?
Thanks
@cleidigh

[cleidigh]

@redarrow-rus
I have a beta 2 potential fix. Grab here :

• v14.1.5 - Maintenance Release - Status and Download #681

restart after installing
@cleidigh

[redarrow-rus]

Of course, I will test it in 7-8 hrs.

[redarrow-rus]

Unfortunately, the same error:

Error: Incorrect argument types for folders.create.

It cannot create subfolder.

Console:

IETNG: createSubfolder failed, retry for: Temp [ietngUtils.js:365:19](chrome://mboximport/content/mboximport/modules/ietngUtils.js)
IETNG: Recovery failed [ietngUtils.js:374:19](chrome://mboximport/content/mboximport/modules/ietngUtils.js)

[cleidigh]

@redarrow-rus
Thanks, we got a little more info.
I will add some more debug to see what argument is wrong.
Where are you importing the mbox? It's name is Temp?
@cleidigh

[redarrow-rus]

The original mbox file's name was Входящие (Inbox in Russian); after the first time error had occured I renamed it to Temp hoping it might help.

Both the file being imported and Thunderbird profile are on ext4 encrypted file system with all permissions for my user.

[cleidigh]

@redarrow-rus
In normal circumstances the Russian names should be of no issue.
I don't know if the encrypted drive has something to do with it, but if it's easy to test without it, might be helpful.
@cleidigh

[emefay]

I have this problem with 14.1.4 on Thunderbird versions 115.5.0 and 115.10.1 on Windows 10 & Windows 11. Trying to do a test export and import of a single email folder in mbox format. The Thunderbird folder name being exported was "Austrian citizenship" (with a space).

I get the same error messages on import to Local Folders as noted by @chefren. I reverted to ImportExportTools NG 14.1.2 as suggested by @redarrow-rus and exported file now imports ok.

Not sure if it's relevant, but the exported mbox file was exported to a Synology Drive folder on my local Windows PC (which is then copied to the Synology "cloud" NAS)? I don't think it's relevant as the actual exported file remains in a Windows folder on my PC.

[cleidigh]

@emefay
I am just now trying to get a beta with debug out. I do hundreds of imports and cannot reproduce, so hoping the debug will indicate. I will post here when I have it.
@cleidigh

[emefay]

Just tried to import the exported file using IET NG 14.1.5-b3 - same error as with 14.1.4

[cleidigh]

@emefay
I think I resolved this. Grab b4 here:

• v14.1.5 - Maintenance Release - Status and Download #681

@cleidigh

[cleidigh]

@emefay
I misspelled your id, see above

[cleidigh]

@chefren
@redarrow-rus
Try b4, believe I fixed this.

• v14.1.5 - Maintenance Release - Status and Download #681
  @cleidigh

[cleidigh]

@emefay
You need b4

[emefay]

@cleidigh So just tried b4 this morning... it appeared to import the folder, but then I couldn't access the folder thereafter...

Removed the add-on, re-installed 14.1.2 and checked that was ok.

Removed 14.1.2 and re-installed your b4 update. Tried the import again and this time it worked ok, though it does seem to have slowed down TB AND the import.

As an aside, I restart TB every time after I un-install the add-on, and every time after I re-install it.

[redarrow-rus]

I tried b4 and it was... strange. No alert, no import. But there is an error in console:

TypeError: MailServices is undefined

linking to ietngUtils.js:350:7.

[cleidigh]

@redarrow-rus
Do a restart of Thunderbird. I have seen this also.
@cleidigh

[cleidigh]

@emefay
There is an odd install issue where it says MailServices undefiled as
@redarrow-rus pointed out you would never see this except in the console. I have no clue as to this issue yet, but I suspect it's a cache issue.
@cleidigh

[redarrow-rus]

@redarrow-rus Do a restart of Thunderbird. I have seen this also. @cleidigh

Indeed... :) Now it's OK!

[cleidigh]

@redarrow-rus
@emefay
Great!
Hopefully I can figure out the Mail Service issue as it will throw everyone off.
@cleidigh

[alx-a]

@chefren @redarrow-rus Try b4, believe I fixed this.

* [v14.1.5 - Maintenance Release - Status and Download #681](https://github.com/thunderbird/import-export-tools-ng/issues/681)
  [@cleidigh](https://github.com/cleidigh)

Test: Identical error reported as fixed.

I got the Error at topic start when importing a 10GB Gmail mbox file. Release that got the error is current, as available via Thunderburd Addons install options on January 27 2024.

Using the release you have provided has fixed the error. Install as follows:

1. Download the release
2. Install via Menu (three stripes in menu bar) -> Addons and Themes -> Tools for Addons [Click Settings Gear Wheel ] -> Install Addon From File -> Select file from download location
3. Restart Thunderbird
   Import succeeded, mbox folder created, individual emails can be browsed throughout.

[cleidigh]

@alx-a
Thanks for the post. Sorry for the trouble. I'm still working on an odd caching issue related to this. I will post progress here so keep an eye out.
@cleidigh

[cleidigh]

@alx-a
I have so many parallel issues and betas. Can you tell me which beta got your 10GB import working?
@cleidigh

[alx-a]

import-export-tools-ng-14.1.5-b4-tb.xpi
I was under the impression that the comment I was quoting was linking to it.
Thanks for all the hard work! Consider your work a miracle, Google hick-upped twice while exporting these "simple" 10GB of an mbox!

[cleidigh]

@alx-a
Thanks for the support and kudos!
I've worked very hard on the mbox imports to handle large mboxes.
I did, however, mess up v14.1.4 and for a variety of reasons am having trouble getting it stable. My disability of programming by eyes isn't helping. So I appreciate everyone's patients.
@cleidigh

[cleidigh]

@chefren
@redarrow-rus
@emefay
I have rolled back the mboxImportExport module to that in v14.1.2 which other than some small oddities it was stable. Even b4 has crashed on me so t would not rely on that. I am thinking of releasing this if I can while I do research on why the changes I made in v14.1.4 caused issues.

Any testing help would be great !
Grab b8 and restart after installing

• v14.1.5 - Maintenance Release - Status and Download #681

@cleidigh

[cleidigh]

All
Anyone able to do some testing?
@cleidigh

[alx-a]

I might have missed this, is there a reference mbox file for testing? My 10 and 4.5 Gb takes a while to import, so a smaller size could be more practical.

I could test with a sample smaller file today if one is provided, otherwise at some point later with a re-import of my older stuff to a new profile within 5 days.

[cleidigh]

@alx-a
I'm particularly interested in the larger mbox imports as those were crashing Thunderbird. Maybe a couple test imports of your 4.5GBs?
I have a 7 year old i7 that takes about 42s per GB. I would be curious about your times and sytem if possible.
I really want to make sure I get v14.1.5 right.
I very much appreciate the help testing., but if not possible I understand. My goal is to release Sunday night.
@cleidigh

[alx-a]

I've got a half-assed sync between machines that somehow dropped the 4.5gb but I have the 10Gb file to test over the weekend. I'll try to time it, with second precision.

To confirm, this is the release? #681

[cleidigh]

@alx-a
Yes, b8 in

• v14.1.5 - Maintenance Release - Status and Download #681
  Is the latest.
  Thank you so much!
  Christopher

[redarrow-rus]

I confirm that b8 is ok :)

[alx-a]

TLDR: B8, B4 OK! 15 minutes per 10Gb import single 10.9Gb mbox Gmail.

For v14.1.5-b8:
Speed about the same, 868 seconds. Summary file building about another 60s. Appears good on first glance, will update if anything.

Test 1, Release: v14.1.5-b4

System load during test:
System running typical mixed work background tasks of browser and multimedia, including file download Import took most of CPU, with free memory, 30-40% available from 16GB. See htop image.

Future Retest needed? Can be retested with the import being the primary task, but at this time length, not sure how many users could abstain from doing other stuff during import.

Mbox contens: 60K messages, Message contents: about 5% at 10Mb size, image attachments or documents, about 10% messages contain images above 100Kb, almost no multimedia, 40% mass mailers and commercial, rest between 100 correspondents.

System Info Report, edited for brevity, done with LSI https://github.com/tdulcet/Linux-System-Information

Linux Distribution: Linux Mint 21.3
Linux Kernel: 5.15.0-119-generic
Computer Model: LENOVO ThinkPad P50
Processor (CPU): Intel(R) Core(TM) i7-6820HQ CPU @ 2.70GHz
Architecture: x86_64 (64-bit)
Total memory (RAM): (16GiB)
Total swap space: (42GiB)

Test for b8 in same thunderbird profile,
Deleted all the mail folders from the local .thunderbird storage before import. Load similar across, matching the htop image.

[cleidigh]

@redarrow-rus
Thank you for testing!
@cleidigh

[cleidigh]

@alx-a
Excellent!
Thank you so much for the very detailed report!!
Very much appreciated, especially over the weekend. It makes me quite confident to release.

I have an i7 believe 2.7GHz and a very fast ssd.
It takes 42s per GB so I am running about 2x your rate. I wonder if that's my ssd or my processor is running faster than I understand.

For now the biggest thing is stability. 10GB imports with solid Thunderbird behavior is excellent. I do have a couple of things to improve speed, but that will have to wait for another release. I'm not going to jeopardize v14.1.5.

Can I keep you on a test list for the future? I have trouble finding testers and my testing can only cover what I have to test.
Again, thank you.
Christopher

[alx-a]

I wrote the test report as a simple framework example that can be easily adapted as a guideline for future testers, so glad you like it.

Re testing - of course, I have several of those mboxes and will be using the product regulary. Glad to participate.

Separate issue for discussion of testing?
Shall I set up a separate issue for testing discussion, we need to discuss what should be considered, as well as possibly creating some sample mboxes for testing.

I can write a guide for future testers, I'll get back with some ideas as to how that can best be integrated, from past experiences. It does of course make a difference as to what the messages are, etc etc.

The machine I tested on is a weekend beater, with generic throw-away Kingston SSD. I'm sure that is the bottleneck, it is used for minor things and not kernel kompiling. The reason it is used is exactly because it is a beater from a better era - I can rip out the disk in 1 second the way I have it set up, and the motherboard and memory are accessible. There be swappable parts! After a reballing attempt on a failed Macbook, I am going to keep the older Thinkpads, screwdriver accessible, as my only choice. Hopefully for the next 5 years at least, then maybe Framework, though there is a charm of an encrypted beater laptop under 200 USD that you are not concerned to lose.

I suppose something 8GB to 16GB of ram but with a faster SSD and processor under 5 years old on a mid-priced laptop could be seen as a target user machine?

[cleidigh]

@alx-a
The idea of a test template is interesting. The only thing is I wonder how many people would actually follow it. Unfortunately most people are not as thorough as you are. My biggest issue is getting test help in general.
@cleidigh

[cleidigh]

All
v14.1.5 was released this morning.
@cleidigh

[chefren]

Hi , phew what a ride. I was not able to reply to this as my family was sick and was caring for them. But it seems that this last version is getting the files in, whatever the issue/s was/were - could only try windows but it works, so thank you all! I will close this now