Unconfigured Ad Widget

**SkyHawk** · 11-11-2017, 9:11 PM

A file of any size consumes at least one cluster. Any excess consumes the next whole cluster. So a 1k file will consume a whole 8k cluster if 8k is the cluster size you have selected. A 10k file would consume 16k. Extrapolate that by hundreds of thousands or millions of small files and you see what a massive waste of space that would be.

The average amount of space that is lost in this manner can be calculated by using the equation (cluster size)/2 * (number of files).

Right click any Windows folder and view the properties. You will see two sizes given. The 'size' and the 'size on disk'. The difference between those numbers is the space wasted due to under utilized clusters. View the properties for a whole disk and you can see how it adds up.

Generally speaking, cluster size should be selected to match as close as possible, the average file size for a given environment, or a cluster multiple that is close. So if you store a bunch of 16k files, then 8k is good, just as a basic example. For high performance environments where space efficiency is of less concern, and manipulating large files is the job at hand (like video editing or CAD), then large clusters are OK.

**Robotron2k84** · 11-12-2017, 12:17 AM

Originally posted by the86d

Am I overlooking something else?

Bigly. Most OSs aggressively cache writes and use a minimum block size of 32K on recent hardware. This is why drives are now multiples of 32K cluster sizes for HDDs (4K sectors).

SSD controllers also do batch writes at larger scales which is why if you really care about your data you want a battery (or capacitor) backed controller. Modern file systems use journaling to provide fault tolerance, or dedicated temp partitions in larger arrays. SSDs, however, are notoriously bad in powerfail conditions where the sudden loss of power can brick the drive. You get what you pay for.

What you need to also remember is that most SSDs have ~20% extra capacity for wear leveling that is transparent to the software. Trim operations and batch deletes occur on large blocks of several megabytes as unused clusters are tagged but not cleared on the drive until the delete queue is large enough for a single operation. At only that time is the space returned to the free pool for reuse, minimizing overwrites.

Tl;dr - Cluster size means very little to modern hardware and SSDs

**the86d** · 11-12-2017, 4:50 AM

Originally posted by boo2112

What is the the physical sector size on the SSD (not the file system)?

512 bytes, on every consumer SSD I have ever checked.
(Kingspec, Sandisk, Samsung, Intel, and Transcend.)

Sandisk 256GB SSD:

Code:

C:\Users>fsutil fsinfo ntfsinfo c:
NTFS Volume Serial Number :       0x8a7479bf7479af17
Version :                         3.1
Number Sectors :                  0x000000001bef00f3
Total Clusters :                  0x00000000037de01e
Free Clusters  :                  0x00000000021c4910
Total Reserved :                  0x0000000000000810
Bytes Per Sector  :               512
[B][SIZE="4"]Bytes Per Physical Sector :       512[/SIZE][/B]
Bytes Per Cluster :               4096
Bytes Per FileRecord Segment    : 1024
Clusters Per FileRecord Segment : 0
Mft Valid Data Length :           0x0000000013700000
Mft Start Lcn  :                  0x00000000000c0000
Mft2 Start Lcn :                  0x0000000000000010
Mft Zone Start :                  0x00000000000d3700
Mft Zone End   :                  0x00000000000d7dc0
RM Identifier:        9334BDE2-5C4A-11E4-9A72-8634FF2AB897

Originally posted by Robotron2k84

Bigly. Most OSs aggressively cache writes and use a minimum block size of 32K on recent hardware...

This drive was purchased, 11/xy/16, I believe, and I don't use RAID controllers at home, just on-board... generally
(no JBOD, or RAID currently in use in my home).
Samsung 512GB SSD:

Code:

C:\Users>fsutil fsinfo ntfsinfo d:
NTFS Volume Serial Number :       0x5064f53564f51e82
Version :                         3.1
Number Sectors :                  0x000000003a3847ff
Total Clusters :                  0x000000001d1c23ff
Free Clusters  :                  0x000000000746164a
Total Reserved :                  0x000000000008a280
Bytes Per Sector  :               512
[B][SIZE="4"]Bytes Per Physical Sector :       512[/SIZE][/B]
Bytes Per Cluster :               1024
Bytes Per FileRecord Segment    : 1024
Clusters Per FileRecord Segment : 1
Mft Valid Data Length :           0x0000000005140000
Mft Start Lcn  :                  0x0000000000300000
Mft2 Start Lcn :                  0x0000000000000008
Mft Zone Start :                  0x000000001193c340
Mft Zone End   :                  0x000000001196e360
RM Identifier:        FA92F92D-B636-11E6-98F6-002185C55491

Apparently even Crucial, via "smartctl --all /dev/sda" on my dedicated Slackware box:

Code:

=== START OF INFORMATION SECTION ===
Model Family:     Crucial/Micron RealSSD m4/C400/P400
Device Model:     M4-CT256M4SSD2
Serial Number:    [OMITTED]
LU WWN Device Id: 5 00a075 1031d0fcc
Firmware Version: 070H
User Capacity:    256,060,514,304 bytes [256 GB]
[B][SIZE="4"]Sector Size:      512 bytes logical/physical[/SIZE][/B]

**Robotron2k84** · 11-12-2017, 11:43 AM

SSDs are not circular and sector size is a meaningless concept for flash storage. SSD controllers emulate both 512B and 4K sector sizes for the benefit of the OS only. A fair number of SSDs let you reconfigure them as 512 or 4K. As stated in my previous post the cluster size is not a determining factor due to the controller performing batch writes in the megabyte range. Everything is cached, resident to the controller until it is flushed in one large write, or the queue stalls and flush time has arrived.

My reference to 32K block sizes is the the minimum unit of transfer IN THE DRIVER. Attempts to divide that result in slower performance as the OS has to perform more operations to fill the bus.

Best performance is 4K "sectors" and 16-32K clusters at this point. This is for the benefit of the OS drivers, cache and bus, which reduce their logical operation with a higher metric. SSD controllers are pretty smart and optimize the read/write patterns transparently which obviates much of how things were thought of and done in the past.

This means that your changes to physical and logical sectoring and clustering are completely ignored by the controller and it will do the writes and reads as it sees fit. Fact is even if your file size is 1B on a 32K cluster, the SSD may only write the one byte, tagging the remainder of the cluster as usable but keeping it logically reserved. The battlefield over specs between vendors left streaming speed as its banner about 5 years ago and now they compete on the sophistication of the controller software.

Only if you have a billion+ small files would I even consider changing the cluster size as the file systems are 64-bit addressable and small file buffering meant more when you physically had to seek for them. And with SDD performance, no one cares about raw topology any more (Oracle). Believe it or not, but the OS vendors do make choices for their defaults that accommodate the largest use-cases, on modern hardware.

Bottom line: SSDs leveraged decades of experience with logical controllers in RAID/SAN storage to arrive at an optimized interface that does not care how it is spoken to. The flash engineers have seen to that.

- from a 25 year enterprise IT guy who spent more than his fair share of time supporting DBs on large arrays.

**the86d** · 11-13-2017, 5:26 AM

So, theoretically it MIGHT work, but I might be lied to by the OS and SMART, anyways.

I just shouldn't care, and just take the default 4k clusters instead of taking 2-3 minutes formatting ahead of time w/a smaller cluster for Windows boxes?

AND by the time they wear-level themselves into oblivion (10+ years), I will be buying a new one anyway.

Same with Linux one-trick-ponies too (one-drive), I assume.

Summing up, IF it works, I might only get 15 years off an SSD, instead of 10, and drives will be better then anyways, when replaced.

**Robotron2k84** · 11-13-2017, 8:52 AM

Consumer SSDs might get you a few years, 10 would be stretching it. MTBF for these devices in the field is <3 years.

The only SSDs worth your money are those that have write abort or full power protection, as well as overprovisioning. One good power failure is all it takes to brick a SSD without write abort, and you will start to rack up bad clusters without overprovisioning. Some enterprise SSDs have a rebuild program for arrays that if the FTL becomes junk can reformat the drive in situ for restore of data. When a consumer SSD's FTL fails it usually is thrown out. This is for models that may use a dedicated NVRAM chip for the FTL, without a DRAM intermediary.

The most damning thing about consumer SSDs is the wear leveling algorithm; stale data which just sits, occupies low cycle pages, but the algorithm forces that data to migrate to other pages periodically to make available low use cells for further writes. If a corrupting event happens during those operations, none of your data is safe, which generally wasn't true on spinning rust devices.

Enterprise drives are generally the only models that take these factors into account, and no M.2 drives meet all of these requirements. How much is your data and time worth?

The biggest issue I've faced with SSDs is that backups become more critical, and verification restores also need to be thought of as critical due to SSDs propensity for silent corruption. It's more time consuming to do this, but a trade off for the improved cost and performance.

**the86d** · 11-14-2017, 6:02 AM

Originally posted by Robotron2k84

Consumer SSDs might get you a few years, 10 would be stretching it. MTBF for these devices in the field is <3 years.

The only SSDs worth your money are those that have write abort or full power protection, as well as overprovisioning. One good power failure is all it takes to brick a SSD without write abort, and you will start to rack up bad clusters without overprovisioning. Some enterprise SSDs have a rebuild program for arrays that if the FTL becomes junk can reformat the drive in situ for restore of data. When a consumer SSD's FTL fails it usually is thrown out. This is for models that may use a dedicated NVRAM chip for the FTL, without a DRAM intermediary.

The most damning thing about consumer SSDs is the wear leveling algorithm; stale data which just sits, occupies low cycle pages, but the algorithm forces that data to migrate to other pages periodically to make available low use cells for further writes. If a corrupting event happens during those operations, none of your data is safe, which generally wasn't true on spinning rust devices.

Enterprise drives are generally the only models that take these factors into account, and no M.2 drives meet all of these requirements. How much is your data and time worth?

The biggest issue I've faced with SSDs is that backups become more critical, and verification restores also need to be thought of as critical due to SSDs propensity for silent corruption. It's more time consuming to do this, but a trade off for the improved cost and performance.

I have yet to see an SSD die, however this was my initial concern, and why I didn't get an SSD until I could gather data from others regarding reliability and longevity. We had some early-adopters/pushers at work, and they had issues with the Crucial (Micron?) M4 firmware bug, and the drive liked to disappear for like 2-3 firmware revs.

Can one carve data off a consumer SSD that dies during a write operation that failed as you mentioned, or does the controller whack-out in this case?

**Robotron2k84** · 11-14-2017, 11:07 AM

You might not have seen a drive die, so to speak, but the drives themselves are only rated for 1PB of writes on average, some lower. Generally they can all exceed this to some degree, but failure is a certainty as the drive approaches it's exhaustion of useful life. There was a chart a few years back comparing ultimate write longevity and errors over time. The high end drives did what you expect, they had very few errors and then fell off the cliff and locked up. A drive is supposed to stop functioning once it hits a certain number of hard errors, via SMART.

The Samsung's on the other hand continued allowing themselves to be written to past their useful life all the while spewing SMART errors and racking up bad blocks. Those are the ones I stay far away from.

Remember, too: the MTBF of a device in the field is not the same thing as the manufacturer's stated MTBF. MTBF for the vendor means that the customer only sees that if they replace the drives before their useful life is over, stay under the maximum writes per day, keeps the drive environmentally in spec and disable disk optimizations that may cause excessive wear. In the field, MTBF is how long before there is a support ticket saying my machine won't boot or I can't read my data.

Once a consumer SSD drive bricks itself then it's toast. You would have to send it out to a forensics company for recovery and hope for the best.

But, the data would be equally suspect on an enterprise disk, the only difference is that the disk can usually be resurrected (if still within service life). Drive failure in arrays is a common scenario, but powerfail and FTL errors are not because we use redundant power. But, when it happens, you can either swap the unit or attempt a rebuild. We usually attempt a single resurrection before tossing the drive as it becomes a liability. The array is raid 50 or 60, so there are enough copies to completely rebuild the data.

There are several vendors that sell you a rack mounted storage array device that does all this transparently and guarantees the data integrity over some useful number of years and writes. Spendy stuff: 40k for a few TB

But, the three most important words in data security are: backups, backups, backups.

Unconfigured Ad Widget

Cluster Size, SSDs are fast so why not extend life...?

Cluster Size, SSDs are fast so why not extend life...?

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Calguns.net Statistics

What's Going On