On 4/26/2020 9:46 PM, Paul wrote:Not really sure, but I think TB does compression on its files. If you
Have you tried to "defragment" the drive ?
No, considering it's an SSD. But as you pointed out later, the optimize option is available for both of my SSD's, but optimize recognizes them
as SSD's, so the only optimization available is trimming, no defragging.
Normally, the "optimize" dialog will not offer defragmentation
as an option in Windows 10. It's supposed to offer "TRIM" as
the option for an SSD.
However, there is a "Copy On Write" or COW issue with SSDs.
Under the right circumstances, there will be a slowdown.
Yes, likely this is exactly that circumstance. Do you know what the
symptoms of that circumstance are?
Now, consider what you're doing. Your backup software uses VSS
to make a shadow copy. It's possible some "COW activity" is happening
during the backup.
Yes, VSS is used by the software, which is Macrium Reflect 6 BTW.
Reflect's logs show that it creates the VSS shadows immediately before beginning the backup.
This backup runs after midnight, and there is little activity while any
of the backups run. All of the backups run after midnight and they
finish relatively quickly, except this one.
The Optimize dialog knows about this, and the Optimize dialog
has some sort of metric it uses to decide what to do. While
most of the time, it will only offer TRIM, I bet in your
case, it's "going to have a COW" and defragment your drive.
This should not be as thorough as a regular defragment,
and the design of what's done, should have something to do
with whatever the root cause of "having a COW" is.
VSS is used on all of the backup jobs. None of the others exhibit this behaviour. In fact, I've experienced this issue for nearly a decade now.
The problem started on Windows XP, continued on into Windows 7, and continues to plague me in Windows 10. This particular folder has also
been migrated around from HDD to SSD, to a 2nd SSD, etc. So it's not a problem that is specific to HDD's or SSD's, or to any particular version
of Windows.
I'll tell you what this folder is. It's actually my Thunderbird News
folder (exactly what I'm using to ask this question here), which exists under the my User folder structure. The problem was discovered when I started doing daily backups of my User folder and discovered that the
User folder was taking forever. After investigating it some, I figured
out that the problem was this particular substructure under News. Once I excluded the News folder, backups finished 6 times faster! So I moved
the backups of the News folder to their own job, and let the rest of the User folder get backed up separately. Before, you ask, I only backup the News folder once a week, but it's still a pain in the ass watching it
take so long even once a week.
Some other background. When this particular backup is happening, it's
not the drives that are showing as busy, it's the CPU cores! 4 out of
the 8 cores on my FX-8300 are fluctuating between 50% to 100% busy,
while the other 4 are not that busy.
Yousuf Khan
Not really sure, but I think TB does compression on its files. If you don't allow it, that might be the cause.
I seem to remember at some time in the past, you offered
advice on putting an exception for an AV program,
so it does not scan that particular directory
(something in Thunderbird).
If your CPU cores are railed, I'd be tracing down the
PID of the offender.
One way to do it on a Pro SKU of OS, is
tasklist /svc # should not work on Home
Yousuf Khan wrote:
I have a folder on one of my SSD drives that takes 8 to 10 hours to back
up. It is only about 1.4 GB, but it is allocated 2.4 GB of space
altogether, and there are 580,000 files here. Indicates that per file
it's using up a little bit over half of a cluster on average. File
system is NTFS.
Meanwhile, this same drive can backup the remainder of the drive in
under 2 hours, and the remainder of the drive is 390 GB! Is NTFS this
inefficient for small files like this?
Yousuf Khan
Have you tried to "defragment" the drive ?
VSS is used on all of the backup jobs. None of the others exhibit this behaviour. In fact, I've experienced this issue for nearly a decade now.
The problem started on Windows XP, continued on into Windows 7, and continues to plague me in Windows 10. This particular folder has also
been migrated around from HDD to SSD, to a 2nd SSD, etc. So it's not a problem that is specific to HDD's or SSD's, or to any particular version
of Windows.
I'll tell you what this folder is. It's actually my Thunderbird News
folder (exactly what I'm using to ask this question here), which exists under the my User folder structure. The problem was discovered when I started doing daily backups of my User folder and discovered that the
User folder was taking forever. After investigating it some, I figured
out that the problem was this particular substructure under News. Once I excluded the News folder, backups finished 6 times faster! So I moved
the backups of the News folder to their own job, and let the rest of the User folder get backed up separately. Before, you ask, I only backup the News folder once a week, but it's still a pain in the ass watching it
take so long even once a week.
Some other background. When this particular backup is happening, it's
not the drives that are showing as busy, it's the CPU cores! 4 out of
the 8 cores on my FX-8300 are fluctuating between 50% to 100% busy,
whole the other 4 are not that busy.
Yousuf Khan--- Synchronet 3.18a-Linux NewsLink 1.112
I have a folder on one of my SSD drives that takes 8 to 10 hours to
back up. It is only about 1.4 GB, but it is allocated 2.4 GB of space altogether, and there are 580,000 files here. Indicates that per file
it's using up a little bit over half of a cluster on average. File
system is NTFS.
Meanwhile, this same drive can backup the remainder of the drive in
under 2 hours, and the remainder of the drive is 390 GB! Is NTFS this inefficient for small files like this?
On 4/26/2020 9:32 PM, VanguardLH wrote:
Using WHAT backup software? Doing a file-based or image-based backup?
Macrium, file-based.
Is it a direct access to the folder, or are you using a redirection,
like a junction (reparse point)? Does that folder itself have any
redirections which could run the backup program into a loop if it
doesn't specifically ignore those?
No, none of that. Straightforward unredirected.
On 4/27/2020 3:57 AM, Paul wrote:
I seem to remember at some time in the past, you offered
advice on putting an exception for an AV program,
so it does not scan that particular directory
(something in Thunderbird).
If your CPU cores are railed, I'd be tracing down the
PID of the offender.
One way to do it on a Pro SKU of OS, is
tasklist /svc # should not work on Home
Not even necessary, I can tell you right now which process is
responsible, it's the Macrium Reflect binary. Also the System process
which I assume the Reflect binary also makes heavy use of during this time.
I'll tell you what this folder is. It's actually my Thunderbird News
folder (exactly what I'm using to ask this question here), which exists under the my User folder structure. The problem was discovered when I started doing daily backups of my User folder and discovered that the
User folder was taking forever. After investigating it some, I figured
out that the problem was this particular substructure under News. Once I excluded the News folder, backups finished 6 times faster! So I moved
the backups of the News folder to their own job, and let the rest of the User folder get backed up separately. Before, you ask, I only backup the News folder once a week, but it's still a pain in the ass watching it
take so long even once a week.
Some other background. When this particular backup is happening, it's
not the drives that are showing as busy, it's the CPU cores! 4 out of
the 8 cores on my FX-8300 are fluctuating between 50% to 100% busy,
while the other 4 are not that busy.
Yousuf Khan
What did you use to check if there were junctions defined within the
folder? For example, you could use Nirsoft's NTFSLinksView tool to scan
for junctions to list them. You can specify the start folder from where
to search, like the folder with the 500K+ files, or search from the root folder of a drive (junctions cannot point to other drives). Alas, if
you pick the problematic folder, a scan will only show any junctions in
that folder, not those that point at that folder. You might want to
scan from the root folder, and then check if that folder is under a
junction. Windows has been using junctions for a long time, especially
when Microsoft decides to change the name of the special folder, like changing "Documents and Settings", the old name, and "Documents", that
both point to C:\Users. Could be your problematic folder is under a junction, like Documents.
If there are 580,000 files in the News folder, then you've probably configured your Thunderbird News account(s) to use one file for each
article instead of one file for each newsgroup.
If so, it's probably best to bite the bullet and convert to one file
per newsgroup. That probably needs an export and (re-)import and
probably will be time-consuming, but at least then you'll solve the
actual problem.
FYI, my setup - not Thunderbird - has nearly a million articles, but
only some 600 files.
As a test, disable your anti-virus software and run your TB data-only
backup job.
As another test, make sure to*exit* Thunderbird (check there are no instances of TB in Task Manager's Processes tab), and check if the
backup job is just as slow.
VSS will encounter problems with databases that are not VSS aware. Microsoft's SQL Server is VSS aware, but others are not. The
recommendation in backup programs, even those using VSS, for database programs that are not VSS aware is to schedule their shutdown before the backup, schedule the backup while the database program is down, and
restart the database program after the backup finishes. While this can
be done using Task Scheduler using event triggers (provided the database program issues an event on shutdown), it's a pain to figure out the script-like code you have to use to define for the trigger of the
scheduled event. There are schedulers that are more flexible that can
make their events dependent: task 3 runs only after task 2 ran and
returned good status which runs only after task 1 completed and returned
good status.
I have a folder on one of my SSD drives that takes 8 to 10 hours to backHi Yousuf,
up. It is only about 1.4 GB, but it is allocated 2.4 GB of space
altogether, and there are 580,000 files here. Indicates that per file
it's using up a little bit over half of a cluster on average. File
system is NTFS.
Meanwhile, this same drive can backup the remainder of the drive in
under 2 hours, and the remainder of the drive is 390 GB! Is NTFS this inefficient for small files like this?
Yousuf Khan
Since the folder is on an SSD, fragmentation shouldn't make any difference.And you will reduce your wear life doing a defragment
On 4/27/2020 12:04 PM, Frank Slootweg wrote:
If there are 580,000 files in the News folder, then you've probably
configured your Thunderbird News account(s) to use one file for each
article instead of one file for each newsgroup.
If so, it's probably best to bite the bullet and convert to one file
per newsgroup. That probably needs an export and (re-)import and
probably will be time-consuming, but at least then you'll solve the
actual problem.
FYI, my setup - not Thunderbird - has nearly a million articles, but
only some 600 files.
Yes, that is exactly the problem, I was getting at. Does Thunderbird
have a new news file format available? My assumption was that
Thunderbird only does 1 file/message? What's the option to convert?
Yousuf Khan
On 2020-04-27 07:30, Ken Blake wrote:
Since the folder is on an SSD, fragmentation shouldn't make any difference. >>
And you will reduce your wear life doing a defragment
On 4/27/2020 2:29 PM, VanguardLH wrote:
As a test, disable your anti-virus software and run your TB data-only
backup job.
Yes, that's been done years ago too. This folder has been a major
headache for years now. And at one time, I found that the AV software spending tons of time scanning this folder too, so I put an exclusion in
it for this folder. The AV doesn't ever scan in this folder anymore.
It's not related to VSS, I've already given you the most likely cause of
the problem: there are over half million files, and each file is inefficiently taking up little over half of the NTFS cluster, rather
than spreading a lesser number of files over many clusters. The real question is how can we make NTFS more efficient at handling all of these little files? NTFS is great at handling big files, but tiny little files
no so much.
On 4/27/2020 12:04 PM, Frank Slootweg wrote:
If there are 580,000 files in the News folder, then you've probably configured your Thunderbird News account(s) to use one file for each article instead of one file for each newsgroup.
If so, it's probably best to bite the bullet and convert to one file
per newsgroup. That probably needs an export and (re-)import and
probably will be time-consuming, but at least then you'll solve the
actual problem.
FYI, my setup - not Thunderbird - has nearly a million articles, but only some 600 files.
Yes, that is exactly the problem, I was getting at. Does Thunderbird
have a new news file format available? My assumption was that
Thunderbird only does 1 file/message? What's the option to convert?
Yousuf Khan <bbbl67@spammenot.yahoo.com> wrote:
On 4/27/2020 12:04 PM, Frank Slootweg wrote:
If there are 580,000 files in the News folder, then you've probablyYes, that is exactly the problem, I was getting at. Does Thunderbird
configured your Thunderbird News account(s) to use one file for each
article instead of one file for each newsgroup.
If so, it's probably best to bite the bullet and convert to one file
per newsgroup. That probably needs an export and (re-)import and
probably will be time-consuming, but at least then you'll solve the
actual problem.
FYI, my setup - not Thunderbird - has nearly a million articles, but
only some 600 files.
have a new news file format available? My assumption was that
Thunderbird only does 1 file/message? What's the option to convert?
It's not a new News file format, it's a different format.
You set the format in the News account: Tools -> Account settings ->
<your news account> in the left pane -> 'Server Settings' page ->
Message Storage -> Message Store Type:. This field *should* be set to
'File per folder (mbox)'. Yours is probably set to 'File per message (maildir)'.
For your account - i.e. an *existing* account - you probably cannot
change this setting, i.e. you can only set it when you create the
account. Hence my comment about exporting and (re-)importing. If you
cannot change the setting, you will have to export all the articles from
your current account and then re-import all articles into a new account
with 'Message Store Type: File per folder (mbox)'.
The basic Thunderbird program has no export facility and only very
limited import functionality.
For import of e-mail (from Windows Mail), I have used the Thunderbird ImportExportTools [1] Extension, but I have not used it for News and not
for export.
ImportExportTools can export on a per-folder basis, so you could try
to export just one folder/newsgroup and then import it into a new
account to see if it works for News. Exporting is a copy-type operation,
i.e. the source remains untouched, and if you import to a *new* account,
the old account remains untouched. IOW, it's a totally safe operation.
If ImportExportTools can not solve your problem, you'll probably have
to search the Thunderbird support site(s)/forum(s) or/and post there.
[1] <https://addons.thunderbird.net/en-GB/thunderbird/addon/importexporttools/?src=userprofile>
As for the availability of the MailboxStore option in the
Server settings, the claim is that you must use this
immediately when the installation of Thunderbird is
brand new.
In my experiments yesterday, I tried to "clean out"
my profile, and tried not to leave any .msf files, then
set the prefs.js with the maildirstore preference, and
that *still* wasn't enough to make it work. I'm going
to have to nuke the damn thing and start from scratch,
to see if I can get it to work.
OK, show me a chunk of nfi.exe output, just
for files in the magical folder. Just enough
to capture the essence of what's going on.
Yousuf Khan <bbbl67@spammenot.yahoo.com> wrote:
Yes, that is exactly the problem, I was getting at. Does Thunderbird
have a new news file format available? My assumption was that
Thunderbird only does 1 file/message? What's the option to convert?
It's not a new News file format, it's a different format.
You set the format in the News account: Tools -> Account settings ->
<your news account> in the left pane -> 'Server Settings' page ->
Message Storage -> Message Store Type:. This field *should* be set to
'File per folder (mbox)'. Yours is probably set to 'File per message (maildir)'.
For your account - i.e. an *existing* account - you probably cannot
change this setting, i.e. you can only set it when you create the
account. Hence my comment about exporting and (re-)importing. If you
cannot change the setting, you will have to export all the articles from
your current account and then re-import all articles into a new account
with 'Message Store Type: File per folder (mbox)'.
If ImportExportTools can not solve your problem, you'll probably have
to search the Thunderbird support site(s)/forum(s) or/and post there.
[1] <https://addons.thunderbird.net/en-GB/thunderbird/addon/importexporttools/?src=userprofile>
The one I was looking at the other day, said that it didn't
handle stuff in the News folder specifically.
As for the availability of the MailboxStore option in the
Server settings, the claim is that you must use this
immediately when the installation of Thunderbird is
brand new. In my experiments yesterday, I tried to "clean out"
my profile, and tried not to leave any .msf files, then
set the prefs.js with the maildirstore preference, and
that *still* wasn't enough to make it work. I'm going
to have to nuke the damn thing and start from scratch,
to see if I can get it to work.
One other weirdness from yesterdays experiment, is after
I was finished with my failed experiment, I took the ZIP
file holding my unbroken profile, and started to restore
it to my SSD drive. I was greeted by write rates of arounf
2MB/sec on my SSD. It took forever to restore the fleet
of .msf (file per box) style files. And when I opened
Task Manager, MsMpEng was railed on one core, scanning
everything being written into the profile area. I've done
plenty of other stuff on the computer, where it doesn't
do that with quite the same level of venom. (If I unpack
an .ova on a scratch drive, it does that at several hundred
megabytes per second. As if MsMpEng didn't care.)
Paul
I think that's not correct. The*installation* doesn't have to be
brand new, the*account* in Thunderbird must be new, i.e. just created.
I added a new New account and could set 'Message Store Type:' to
either 'File per folder (mbox)' or 'File per message (maildir)'.
Hi Yousuf,
When I see things like this, it is usually a failing
drive, especially when the index on teh offending
directory never finishes.
This will show up like a soar thumb if yo run your
drive through gsmartcontrol: check the error logs and
run the self tests
On 4/28/2020 3:14 PM, Paul wrote:
The one I was looking at the other day, said that it didn't
handle stuff in the News folder specifically.
As for the availability of the MailboxStore option in the
Server settings, the claim is that you must use this
immediately when the installation of Thunderbird is
brand new. In my experiments yesterday, I tried to "clean out"
my profile, and tried not to leave any .msf files, then
set the prefs.js with the maildirstore preference, and
that *still* wasn't enough to make it work. I'm going
to have to nuke the damn thing and start from scratch,
to see if I can get it to work.
One other weirdness from yesterdays experiment, is after
I was finished with my failed experiment, I took the ZIP
file holding my unbroken profile, and started to restore
it to my SSD drive. I was greeted by write rates of arounf
2MB/sec on my SSD. It took forever to restore the fleet
of .msf (file per box) style files. And when I opened
Task Manager, MsMpEng was railed on one core, scanning
everything being written into the profile area. I've done
plenty of other stuff on the computer, where it doesn't
do that with quite the same level of venom. (If I unpack
an .ova on a scratch drive, it does that at several hundred
megabytes per second. As if MsMpEng didn't care.)
Paul
Oh, it's a good thing I kept reading the replies, as it looks like you already tried what I was about to try. So it kept using the same file
format as before, even after nuking it and starting from scratch?
Yousuf Khan wrote:
Oh, it's a good thing I kept reading the replies, as it looks like you
already tried what I was about to try. So it kept using the same file
format as before, even after nuking it and starting from scratch?
I would refrain from working in this direction.
Sure, if you have backed up the various folders for TBird
before trying it (like I did when testing), then great.
Just don't do it, without having something to restore from.
It's pretty weird for a function to be existing in TBird
and presumably to be absorbing test time from release to
release, and then be hobbling the usage of it with
inept controls.
On 5/1/2020 3:56 AM, Paul wrote:
Yousuf Khan wrote:
Oh, it's a good thing I kept reading the replies, as it looks like
you already tried what I was about to try. So it kept using the same
file format as before, even after nuking it and starting from scratch?
I would refrain from working in this direction.
Sure, if you have backed up the various folders for TBird
before trying it (like I did when testing), then great.
Just don't do it, without having something to restore from.
It's pretty weird for a function to be existing in TBird
and presumably to be absorbing test time from release to
release, and then be hobbling the usage of it with
inept controls.
I just took a chance, and deleted all of the old newsgroup folders, that contained all of the old-style messages. Left all of the rest of the
files in that news server's base folder untouched. Then I started Thunderbird up again. It re-downloaded the messages, and it only
downloaded from where I last left off. It's now filling the data files
known as *.msf (e.g. alt.comp.os.windows-10.msf) rather than filling the folders! Interestingly, these *.msf files used to exist in this News
folder before, but they were just trivial 1K or 2K files, with nothing substantial inside them. They are now substantial files now, ranging
from 44 KB to 41 MB. So it looks like having those old folders there all
of this time was preventing Thunderbird from using the new style *.msf files, even though it had long ago created them!
Yousuf Khan
I just took a chance, and deleted all of the old newsgroup folders, that contained all of the old-style messages. Left all of the rest of the
files in that news server's base folder untouched. Then I started Thunderbird up again. It re-downloaded the messages, and it only
downloaded from where I last left off. It's now filling the data files
known as *.msf (e.g. alt.comp.os.windows-10.msf) rather than filling the folders! Interestingly, these *.msf files used to exist in this News
folder before, but they were just trivial 1K or 2K files, with nothing substantial inside them. They are now substantial files now, ranging
from 44 KB to 41 MB. So it looks like having those old folders there all
of this time was preventing Thunderbird from using the new style *.msf files, even though it had long ago created them!
<boggle!>
If you apparently did not mind to delete all the old articles, then
why did you keep 580,000 old articles in the first place!?
If you kept your original setup with the 500000 files,
you might try updating to 60.9.1 or so, and trying
to flip the control using that version. It seemed to
unsubscribe me from the one group I'd selected, but
it seems to have worked. I haven't had time to do much
other testing yet.
On 5/1/2020 10:55 AM, Frank Slootweg wrote:
<boggle!>
If you apparently did not mind to delete all the old articles, then
why did you keep 580,000 old articles in the first place!?
Simple, because I had no idea what the purpose of any of these files in
this folder were for in any detail, what was important, and where
exactly data resided, so I just backed up everything. That way I
wouldn't have to recreate everything from scratch, and go through hours
of debugging. I've had situations were just 1 important file goes
missing which screws up the entire configuration, and trying to find
that one missing file among half million is a needle in a haystack.
So now after the deletion, I'm down from half million to only about 600 files. And I did a test backup, and the backup went from over 8 hours,
down to only 2.5 minutes! My feeling is that perhaps a lot of those half-million files were just left over from decades of junk that
Thunderbird did not clear, even though it said it was clearing them.
I can - sort of - understand that, but because these 580,000 were
giving you so much hardship, I would have expected you to look at a
few of them, see that they were just News articles and take it from
there, i.e. set/lower the News retention settings in Thunderbird.
On 4/27/2020 6:15 PM, T wrote:means nothing. test it
Hi Yousuf,
When I see things like this, it is usually a failing
drive, especially when the index on teh offending
directory never finishes.
This will show up like a soar thumb if yo run your
drive through gsmartcontrol: check the error logs and
run the self tests
Brand new drive, less than a month old, hasn't had a chance to get old yet.
Yousuf Khan
On 5/1/2020 1:52 PM, Frank Slootweg wrote:
I can - sort of - understand that, but because these 580,000 were
giving you so much hardship, I would have expected you to look at a
few of them, see that they were just News articles and take it from
there, i.e. set/lower the News retention settings in Thunderbird.
No, I knew those were the message files, considering that there were so
many of them, what else could they have been? But often there are other files interspersed among them, that can often go overlooked because it's overwhelmed by the mass of all of the main files. Just let the backup software handle backing all of it up.
Yes, but "Just let the backup software handle backing all of it up."
was the*problem* which made you start this thread! You can't have it
both ways, it either was a problem/annoyance/<whatever>, or it wasn't!
As to "But often there are other files interspersed among them, that
can often go overlooked because it's overwhelmed by the mass of all of
the main files.", I didn't say to bluntly clobber all the*files*/
*folders*, but to set/lower the News retention settings in Thunderbird.
I.e. let*Thunderbird* do it*safely*, instead of you doing it
(possibly) unsafely.
You*do* know how to set/lower the News retention settings in
Thunderbird, don't you!? (You snipped my other comments about that, so I don't know if you've set/lowered them now.)
I have a folder on one of my SSD drives that takes 8 to 10 hours to back
up. It is only about 1.4 GB, but it is allocated 2.4 GB of space
altogether, and there are 580,000 files here. Indicates that per file
it's using up a little bit over half of a cluster on average. File
system is NTFS.
Meanwhile, this same drive can backup the remainder of the drive in
under 2 hours, and the remainder of the drive is 390 GB! Is NTFS this inefficient for small files like this?
Yousuf Khan
On 4/26/2020 9:24 PM, Yousuf Khan wrote:
I have a folder on one of my SSD drives that takes 8 to 10 hours to
back up. It is only about 1.4 GB, but it is allocated 2.4 GB of space
altogether, and there are 580,000 files here. Indicates that per file
it's using up a little bit over half of a cluster on average. File
system is NTFS.
Meanwhile, this same drive can backup the remainder of the drive in
under 2 hours, and the remainder of the drive is 390 GB! Is NTFS this
inefficient for small files like this?
Yousuf Khan
Okay, so after fixing the problem with my News folder, I kept
researching what these millions of little files were, that were clogging
up my News folder. The files had an extension of WDSEML. Later I found
out that these same files are also there in Email folders, hundreds of thousands of them too.
Initially, I thought that these must be the bodies of the messages that Thunderbird uses to store emails and newsgroup messages. But after a bit
of research, I found out that Thunderbird itself has no use for these
files. Thunderbird does generate them, but it doesn't use them itself. Instead it is generated only for the benefit of Windows' Search and
Indexing application. Windows Search uses it to be able to let you
search messages through the Windows Search box. So once Thunderbird generates these files for Windows Search, it no longer has any use for
them anymore, as it stores its own internal data in a different set of files. In fact, these WDSEML files are saved copies of individual
messages out of Thunderbird's own database. So Thunderbird maintains it
own database, but it never cleans up these copies ever in its life.
WDSEML means "Windows Desktop Search Email", in fact. I also think this
is only a specific problem with Thunderbird under Windows, it probably
isn't an issue in Thunderbird under other OS'es like Linux.
You can easily delete all of these messages, but of course Thunderbird
will regenerate them again as they come in. So what you have to do is
tell Thunderbird not to generate these files for Windows anymore. You go into Thunderbird's options menu and turn it off (Tools → Options, then select Advanced → General → System Integration → Allow Windows search to
search messages).
https://fileinfo.com/extension/wdseml
You can also delete them more easily by searching for and deleting just
the folders in which they reside, rather than the individual files.
These folders have an extension called *.MOZMSGS.
Yousuf Khan
Okay, so after fixing the problem with my News folder, I kept
researching what these millions of little files were, that were
clogging up my News folder. The files had an extension of WDSEML.
Later I found out that these same files are also there in Email
folders, hundreds of thousands of them too.
Initially, I thought that these must be the bodies of the messages
that Thunderbird uses to store emails and newsgroup messages. But
after a bit of research, I found out that Thunderbird itself has no
use for these files. Thunderbird does generate them, but it doesn't
use them itself. Instead it is generated only for the benefit of
Windows' Search and Indexing application. Windows Search uses it to
be able to let you search messages through the Windows Search box. So
once Thunderbird generates these files for Windows Search, it no
longer has any use for them anymore, as it stores its own internal
data in a different set of files. In fact, these WDSEML files are
saved copies of individual messages out of Thunderbird's own
database. So Thunderbird maintains it own database, but it never
cleans up these copies ever in its life. WDSEML means "Windows
Desktop Search Email", in fact. I also think this is only a specific
problem with Thunderbird under Windows, it probably isn't an issue in Thunderbird under other OS'es like Linux.
You can easily delete all of these messages, but of course
Thunderbird will regenerate them again as they come in. So what you
have to do is tell Thunderbird not to generate these files for
Windows anymore. You go into Thunderbird's options menu and turn it
off (Tools → Options, then select Advanced → General → System Integration → Allow Windows search to search messages).
https://fileinfo.com/extension/wdseml
You can also delete them more easily by searching for and deleting
just the folders in which they reside, rather than the individual
files. These folders have an extension called *.MOZMSGS.
Interesting find. I don't remember looking at this option when I
previously trialed Thunderbird. Is this option enabled by default? If
so, a very bad choice my Mozilla.
If I had not known about this option (and I was still using
Thunderbird), and after finding the superfluous and unwanted wdseml
files (since I do*not* want Windows search looking into my e-mails to confuse those hits with those of files where I want to find by name or content), I probably would've added them to the Include option in
CCleaner which I sometimes run manually but is also a daily scheduled
event in Task Scheduler to run before the daily backup. I have other programs that leave shit behind that I want purged, so I go into
CCleaner's options, Include section, and define a template of what to
include in CCleaner's cleanup. Some programs, for example, will save
files for a 'resume' function, like a downloader, to continue the
operation when I next load the program. Nope, if I killed/exited the
program then I do not want it wasting time when I next load the program.
I don't even let my web browsers resume a prior session, and configure
them to purge all local data upon their exit.
To me, having Windows Search dig around inside everything is for those
boobs that are slobs. They haven't a clue how to organize their data,
or are too lazy to do it. They pile thousands of e-mails into the Inbox folder instead of organize the old e-mails into separate pending or
archive folders, and God forbid they delete old e-mails. They'll pile thousands of image files into a single folder instead of use folders to organize them. Foldering is an organizational feature that some users
just seem incapable or unwilling to use. As disorganized is their data
is probably the same for how disorderly is their home.
I'm a bit surprized Mozilla, in developing a cross-platform product,
whould give a gnat's fart about kowtowing to Microsoft's search feature
in Windows. Hell, Mozilla doesn't even use the global certificate store
in Windows (use certmgr.msc to see) within Firefox, and instead uses a private cert store inside of Firefox (and why some programs have to do a double cert install: once into the Windows global cert store and again
into Firefox's private cert store). If users are going to search their
old e-mails, why would they not do that from inside of Thunderbird?
They're searching on e-mails, not on some pic they stored from their
camera or a copy of their tax form. Overreaching got even worse in
Windows 10 with Cortana (which I disabled). All this forensic-like
searching to cater to data slobs.
In the business that would be called a "lazy implementation".
All they would have to do, is write a "search provider" and Windows
could use that to pump the files in an OLE fashion. It could have
been done by making no temporary files at all (flow from MORK file
or MBOX or whatever, right into the Windows.edb, in terms of writes).
I guess there's some benefit to federated search that includes
your email, but to my way of thinking this would only clutter up
a search result later.
You might also discover the Windows.edb file is bloated
beyond recognition, because of that file set. It might
range around 1GB for a vanilla install, but after that
Thunderbird thing got indexed, it would likely double
at the very least.
You can rebuild the Windows.edb index file, using
the Indexing Options control panel in Windows 10.
I would give that a whirl after the TB folder has
had all the cruft removed. It'll take about three
hours to index the regular C: files (but this assumes
you've customized the searched folders to include
most of C: , versus the very shallow folder set used
by default).
Even finding Windows.edb is hard :-) The File Explorer
search won't allow you to find it. You'll need Agent
Ransack or Everything.exe to find that file, just so
you can see the current size, and decide whether it
needs a rebuild or not.
Yousuf Khan <bbbl67@spammenot.yahoo.com> wrote:[...]
You can easily delete all of these messages, but of course
Thunderbird will regenerate them again as they come in. So what you
have to do is tell Thunderbird not to generate these files for
Windows anymore. You go into Thunderbird's options menu and turn it
off (Tools ? Options, then select Advanced ? General ? System
Integration ? Allow Windows search to search messages).
https://fileinfo.com/extension/wdseml
You can also delete them more easily by searching for and deleting
just the folders in which they reside, rather than the individual
files. These folders have an extension called *.MOZMSGS.
Interesting find. I don't remember looking at this option when I
previously trialed Thunderbird. Is this option enabled by default? If
so, a very bad choice my Mozilla.
VanguardLH <V@nguard.lh> wrote:
Yousuf Khan <bbbl67@spammenot.yahoo.com> wrote:[...]
[About .wdseml (Windows Desktop Search Email) files/messages:]
You can easily delete all of these messages, but of course
Thunderbird will regenerate them again as they come in. So what you
have to do is tell Thunderbird not to generate these files for
Windows anymore. You go into Thunderbird's options menu and turn it
off (Tools ? Options, then select Advanced ? General ? System
Integration ? Allow Windows search to search messages).
https://fileinfo.com/extension/wdseml
You can also delete them more easily by searching for and deleting
just the folders in which they reside, rather than the individual
files. These folders have an extension called *.MOZMSGS.
Interesting find. I don't remember looking at this option when I
previously trialed Thunderbird. Is this option enabled by default? If
so, a very bad choice my Mozilla.
No, the option is off by default.
No, the option is off by default.
https://bugzilla.mozilla.org/show_bug.cgi?id=552769
I see someone requested all those .wdseml files (under the .mozmsgs
folders) get deleted if the "Allow Windows Search" option gets disabled. Opened on 10 YEARS AGO! Status is still New. Geezus.
Then at:
https://bugzilla.mozilla.org/show_bug.cgi?id=553048
users try to disable the option but it immediately reenables itself.
Rude! Provide the option but do not honor the user's choice. Yousuf
needs to check if the option: (1) remains disabled across multiple
restarts of Thunderbird; and, (2) if the option remains enabled if the .wdseml files that he deleted are not replace with newly generated
.wdseml files.
https://www.zdnet.com/article/mozilla-we-will-keep-thunderbird-after-all-so-long-as-its-not-a-burden-to-firefox/
"pull the plug with six months' notice if the Thunderbird project does
not make "meaningful progress in short order" in creating technical
infrastructure that's independent of Mozilla Corporation's."
That article is dated back in 2017. So, what magical evolution in development resources has occurred for Thunderbird in the meantime?
"Mozilla stopped throwing resources at the project in 2012"
Somewhat explains why a vast number of big tickets have never been
addressed, but there are tickets still listed as New dating back to
2004. From https://en.wikipedia.org/wiki/Mozilla_Thunderbird#History,
lots of wavering on what to do with this lead balloon.
The more I have dig into Thunderbird and its bugs whether reported or
not, the more I get the feeling that the "developers" are CSCI undergraduates, and over the years the turn over of volunteers resulted
in no old farts left that are intimate with the entire product.
On 12/05/2020 6:25 AM, Frank Slootweg wrote:
No, the option is off by default.
It's off in one section of the options, but it's on in another section.
The more I have dig into Thunderbird and its bugs whether reported or
not, the more I get the feeling that the "developers" are CSCI undergraduates, and over the years the turn over of volunteers resulted
in no old farts left that are intimate with the entire product.
When
Mozilla declared it was considering dumping Thunderbird onto other open-source organizations (like how OpenOffice got dumped at the Apache Software Foundation) was when I decided to terminate my trial of
Thunderbird. Well, that and my exasperation with Thunderbird that
pushed me to also dump it after a 6-month trial.
VanguardLH wrote:
The more I have dig into Thunderbird and its bugs whether reported or
not, the more I get the feeling that the "developers" are CSCI
undergraduates, and over the years the turn over of volunteers resulted
in no old farts left that are intimate with the entire product.
I expect the team is smaller than the one in the photo here.
https://blog.thunderbird.net/2014/11/thunderbird-reorganizes-at-2014-toronto-summit/
Nope, it's off there [1] as well.
I've been using Thunderbird for 10 years and there are no .wdseml
files, no .mozmsgs folders and the 'Allow Windows Search to search
messages' option is unticked in both [2] places.
[1] Where 'there' is as you mentioned:
Tools -> Options -> 'Advanced' major tab -> 'General' sub-tab -> System Integration -> Always check to see if Thunderbird is the default mail
client on startup -> Check Now.... -> In the 'System Integration' popup,
the 'Allow Windows Search to search messages' option is*not* ticked.
[2] It's of course [3] one option, which you can set or unset in these
two places.
[3] OTOH, considering this 'Mozilla' crap, 'of course' is a rather
tricky notion!:-(
Why did it take you *that* long [to abandon the trial of Tbird]! :-)
Out of interest: What are you using now (for e-mail)?
Sysop: | DaiTengu |
---|---|
Location: | Appleton, WI |
Users: | 975 |
Nodes: | 10 (0 / 10) |
Uptime: | 39:02:08 |
Calls: | 12,776 |
Calls today: | 2 |
Files: | 186,574 |
Messages: | 3,179,218 |