Calguns.net  

Home My iTrader Join the NRA Donate to CGSSA Sponsors CGN Google Search
CA Semiauto Ban(AW)ID Flowchart CA Handgun Ban ID Flowchart CA Shotgun Ban ID Flowchart
Go Back   Calguns.net > GENERAL DISCUSSION > Technology and Internet
Register FAQ Members List Calendar Mark Forums Read

Technology and Internet Emerging and current tech related issues. Internet, DRM, IP, and other technology related discussions.

Reply
 
Thread Tools Display Modes
  #1  
Old 11-11-2017, 8:41 PM
the86d's Avatar
the86d the86d is offline
Calguns Addict
 
Join Date: Jul 2011
Location: Pinko-occupied ObamaDerkaderkastan
Posts: 6,408
iTrader: 3 / 100%
Default Cluster Size, SSDs are fast so why not extend life...?

EDIT: This CLEARLY isn't for most iPhone owners... but feel-free to chime-in if you just happen to be knowledgeable in such subjects.

So I got another wild-hare again, and plugged in a MSATA adapter to my SATA2 supporting machine. I shrank a working Windows partition to allow for a 4GB partition at the end of the drive to tinker with, and this is not my OS drive, but a drive from another "tinker"-machine. I then created a partition with clusters of 512, 1k, 2k, 4k, and 8k. I proceeded to write ~2GB zero-filled file via Cygwin64 CLI "dd if=/dev/zero of=/cygdrive/g/2gb.out bs=1M count=2000".

It seems to me that there is virtually no difference in write speeds (there seems to be some variance, but doing this from a Windows box right now) between cluster sizes, and changes in small files (most common on my box) would require less writes to less sectors, increasing life by not having to write as many sectors (as flash has a limited number of writes), theoretically extending the life of an SSD, AND saving space, although I COULD drop a tree and find out how much theoretical, or dump a munch of files and find actual... but not willing to spend that much time on it right now...

Code:
512byte clusters, NTFS Zero-file:
Sat, Nov 11, 2017  8:20:28 PM
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB, 2.0 GiB) copied, 13.2421 s, 158 MB/s
Sat, Nov 11, 2017  8:20:41 PM

1K blocks, NTFS Zero-file:
Sat, Nov 11, 2017  8:22:09 PM
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB, 2.0 GiB) copied, 12.7188 s, 165 MB/s
Sat, Nov 11, 2017  8:22:22 PM

2k blocks, NTFS zero-file:
Sat, Nov 11, 2017  8:23:12 PM
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB, 2.0 GiB) copied, 13.2403 s, 158 MB/s
Sat, Nov 11, 2017  8:23:25 PM

4k blocks, ntfs zero-file:
Sat, Nov 11, 2017  8:24:02 PM
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB, 2.0 GiB) copied, 13.0993 s, 160 MB/s
Sat, Nov 11, 2017  8:24:15 PM

8K Blocks, NTFS zero-file:
Sat, Nov 11, 2017  8:25:09 PM
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB, 2.0 GiB) copied, 13.101 s, 160 MB/s
Sat, Nov 11, 2017  8:25:22 PM
Code:
Retest, 512 bytes blocks again, done after all the other tests up to 8k were completed:
Sat, Nov 11, 2017  8:26:13 PM
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB, 2.0 GiB) copied, 13.0248 s, 161 MB/s
Sat, Nov 11, 2017  8:26:26 PM
I did some tests a long time ago, and just wanted to try again with a MSATA SSD I had sitting next to my machine, plugged into a SATA adapter/board (passthrough?), on a SATA2 on my PC, just to see some data, and revisit.

I seem to recall similar results with Linux and different block sizes, and tested with different filesystems (ext2, ext3, ext4) on one of the 1st Intel SSDs I had in my hands and was plugging-away at.
Block sizes didn't seem to make much of a difference.

Thoughts?

Am I overlooking something else?
__________________
"That's what governments are for - get in a man's way." - Captain Malcolm 'Mal' Reynolds

Last edited by the86d; 11-12-2017 at 5:15 AM..
Reply With Quote
  #2  
Old 11-11-2017, 9:11 PM
SkyHawk's Avatar
SkyHawk SkyHawk is offline
☆☆ More Cowbell ☆☆
CGN Contributor
 
Join Date: Sep 2012
Location: Overhead Orbit
Posts: 9,951
iTrader: 122 / 100%
Default

A file of any size consumes at least one cluster. Any excess consumes the next whole cluster. So a 1k file will consume a whole 8k cluster if 8k is the cluster size you have selected. A 10k file would consume 16k. Extrapolate that by hundreds of thousands or millions of small files and you see what a massive waste of space that would be.

The average amount of space that is lost in this manner can be calculated by using the equation (cluster size)/2 * (number of files).

Right click any Windows folder and view the properties. You will see two sizes given. The 'size' and the 'size on disk'. The difference between those numbers is the space wasted due to under utilized clusters. View the properties for a whole disk and you can see how it adds up.

Generally speaking, cluster size should be selected to match as close as possible, the average file size for a given environment, or a cluster multiple that is close. So if you store a bunch of 16k files, then 8k is good, just as a basic example. For high performance environments where space efficiency is of less concern, and manipulating large files is the job at hand (like video editing or CAD), then large clusters are OK.
__________________
.



"Texans always move them!"
General Robert E. Lee, May 6, 1864 - Battle of the Wilderness

Last edited by SkyHawk; 11-11-2017 at 9:15 PM..
Reply With Quote
  #3  
Old 11-11-2017, 11:59 PM
boo2112 boo2112 is offline
Member
 
Join Date: Jan 2013
Posts: 204
iTrader: 0 / 0%
Default

Quote:
Originally Posted by the86d View Post
Am I overlooking something else?
What is the the physical sector size on the SSD (not the file system)?
Reply With Quote
  #4  
Old 11-12-2017, 12:17 AM
Robotron2k84 Robotron2k84 is offline
Member
 
Join Date: Sep 2017
Posts: 147
iTrader: 1 / 100%
Default

Quote:
Originally Posted by the86d View Post
Am I overlooking something else?
Bigly. Most OSs aggressively cache writes and use a minimum block size of 32K on recent hardware. This is why drives are now multiples of 32K cluster sizes for HDDs (4K sectors).

SSD controllers also do batch writes at larger scales which is why if you really care about your data you want a battery (or capacitor) backed controller. Modern file systems use journaling to provide fault tolerance, or dedicated temp partitions in larger arrays. SSDs, however, are notoriously bad in powerfail conditions where the sudden loss of power can brick the drive. You get what you pay for.

What you need to also remember is that most SSDs have ~20% extra capacity for wear leveling that is transparent to the software. Trim operations and batch deletes occur on large blocks of several megabytes as unused clusters are tagged but not cleared on the drive until the delete queue is large enough for a single operation. At only that time is the space returned to the free pool for reuse, minimizing overwrites.

Tl;dr - Cluster size means very little to modern hardware and SSDs

Last edited by Robotron2k84; 11-12-2017 at 1:13 AM..
Reply With Quote
  #5  
Old 11-12-2017, 4:50 AM
the86d's Avatar
the86d the86d is offline
Calguns Addict
 
Join Date: Jul 2011
Location: Pinko-occupied ObamaDerkaderkastan
Posts: 6,408
iTrader: 3 / 100%
Default

Quote:
Originally Posted by boo2112 View Post
What is the the physical sector size on the SSD (not the file system)?
512 bytes, on every consumer SSD I have ever checked.
(Kingspec, Sandisk, Samsung, Intel, and Transcend.)

Sandisk 256GB SSD:
Code:
C:\Users>fsutil fsinfo ntfsinfo c:
NTFS Volume Serial Number :       0x8a7479bf7479af17
Version :                         3.1
Number Sectors :                  0x000000001bef00f3
Total Clusters :                  0x00000000037de01e
Free Clusters  :                  0x00000000021c4910
Total Reserved :                  0x0000000000000810
Bytes Per Sector  :               512
Bytes Per Physical Sector :       512
Bytes Per Cluster :               4096
Bytes Per FileRecord Segment    : 1024
Clusters Per FileRecord Segment : 0
Mft Valid Data Length :           0x0000000013700000
Mft Start Lcn  :                  0x00000000000c0000
Mft2 Start Lcn :                  0x0000000000000010
Mft Zone Start :                  0x00000000000d3700
Mft Zone End   :                  0x00000000000d7dc0
RM Identifier:        9334BDE2-5C4A-11E4-9A72-8634FF2AB897
Quote:
Originally Posted by Robotron2k84 View Post
Bigly. Most OSs aggressively cache writes and use a minimum block size of 32K on recent hardware...
This drive was purchased, 11/xy/16, I believe, and I don't use RAID controllers at home, just on-board... generally
(no JBOD, or RAID currently in use in my home).
Samsung 512GB SSD:
Code:
C:\Users>fsutil fsinfo ntfsinfo d:
NTFS Volume Serial Number :       0x5064f53564f51e82
Version :                         3.1
Number Sectors :                  0x000000003a3847ff
Total Clusters :                  0x000000001d1c23ff
Free Clusters  :                  0x000000000746164a
Total Reserved :                  0x000000000008a280
Bytes Per Sector  :               512
Bytes Per Physical Sector :       512
Bytes Per Cluster :               1024
Bytes Per FileRecord Segment    : 1024
Clusters Per FileRecord Segment : 1
Mft Valid Data Length :           0x0000000005140000
Mft Start Lcn  :                  0x0000000000300000
Mft2 Start Lcn :                  0x0000000000000008
Mft Zone Start :                  0x000000001193c340
Mft Zone End   :                  0x000000001196e360
RM Identifier:        FA92F92D-B636-11E6-98F6-002185C55491
Apparently even Crucial, via "smartctl --all /dev/sda" on my dedicated Slackware box:
Code:
=== START OF INFORMATION SECTION ===
Model Family:     Crucial/Micron RealSSD m4/C400/P400
Device Model:     M4-CT256M4SSD2
Serial Number:    [OMITTED]
LU WWN Device Id: 5 00a075 1031d0fcc
Firmware Version: 070H
User Capacity:    256,060,514,304 bytes [256 GB]
Sector Size:      512 bytes logical/physical
__________________
"That's what governments are for - get in a man's way." - Captain Malcolm 'Mal' Reynolds

Last edited by the86d; 11-12-2017 at 5:43 AM..
Reply With Quote
  #6  
Old 11-12-2017, 10:14 AM
boo2112 boo2112 is offline
Member
 
Join Date: Jan 2013
Posts: 204
iTrader: 0 / 0%
Default

Quote:
Originally Posted by the86d View Post
512 bytes, on every consumer SSD I have ever checked.
Wouldn't that explain why the varying NTFS cluster sizes didn't seem to make much of a difference in your sequential write test (~160 MB/sec)? The SSD is writing in 512 bytes per sector regardless of the NTFS cluster size.
Reply With Quote
  #7  
Old 11-12-2017, 11:43 AM
Robotron2k84 Robotron2k84 is offline
Member
 
Join Date: Sep 2017
Posts: 147
iTrader: 1 / 100%
Default

SSDs are not circular and sector size is a meaningless concept for flash storage. SSD controllers emulate both 512B and 4K sector sizes for the benefit of the OS only. A fair number of SSDs let you reconfigure them as 512 or 4K. As stated in my previous post the cluster size is not a determining factor due to the controller performing batch writes in the megabyte range. Everything is cached, resident to the controller until it is flushed in one large write, or the queue stalls and flush time has arrived.

My reference to 32K block sizes is the the minimum unit of transfer IN THE DRIVER. Attempts to divide that result in slower performance as the OS has to perform more operations to fill the bus.

Best performance is 4K "sectors" and 16-32K clusters at this point. This is for the benefit of the OS drivers, cache and bus, which reduce their logical operation with a higher metric. SSD controllers are pretty smart and optimize the read/write patterns transparently which obviates much of how things were thought of and done in the past.

This means that your changes to physical and logical sectoring and clustering are completely ignored by the controller and it will do the writes and reads as it sees fit. Fact is even if your file size is 1B on a 32K cluster, the SSD may only write the one byte, tagging the remainder of the cluster as usable but keeping it logically reserved. The battlefield over specs between vendors left streaming speed as its banner about 5 years ago and now they compete on the sophistication of the controller software.

Only if you have a billion+ small files would I even consider changing the cluster size as the file systems are 64-bit addressable and small file buffering meant more when you physically had to seek for them. And with SDD performance, no one cares about raw topology any more (Oracle). Believe it or not, but the OS vendors do make choices for their defaults that accommodate the largest use-cases, on modern hardware.

Bottom line: SSDs leveraged decades of experience with logical controllers in RAID/SAN storage to arrive at an optimized interface that does not care how it is spoken to. The flash engineers have seen to that.

- from a 25 year enterprise IT guy who spent more than his fair share of time supporting DBs on large arrays.

Last edited by Robotron2k84; 11-12-2017 at 1:14 PM..
Reply With Quote
  #8  
Old 11-12-2017, 12:24 PM
boo2112 boo2112 is offline
Member
 
Join Date: Jan 2013
Posts: 204
iTrader: 0 / 0%
Default

Excellent post Robotron2k84.
Reply With Quote
  #9  
Old 11-13-2017, 5:26 AM
the86d's Avatar
the86d the86d is offline
Calguns Addict
 
Join Date: Jul 2011
Location: Pinko-occupied ObamaDerkaderkastan
Posts: 6,408
iTrader: 3 / 100%
Default

So, theoretically it MIGHT work, but I might be lied to by the OS and SMART, anyways.

I just shouldn't care, and just take the default 4k clusters instead of taking 2-3 minutes formatting ahead of time w/a smaller cluster for Windows boxes?
AND by the time they wear-level themselves into oblivion (10+ years), I will be buying a new one anyway.

Same with Linux one-trick-ponies too (one-drive), I assume.

Summing up, IF it works, I might only get 15 years off an SSD, instead of 10, and drives will be better then anyways, when replaced.
__________________
"That's what governments are for - get in a man's way." - Captain Malcolm 'Mal' Reynolds
Reply With Quote
  #10  
Old 11-13-2017, 8:52 AM
Robotron2k84 Robotron2k84 is offline
Member
 
Join Date: Sep 2017
Posts: 147
iTrader: 1 / 100%
Default

Consumer SSDs might get you a few years, 10 would be stretching it. MTBF for these devices in the field is <3 years.

The only SSDs worth your money are those that have write abort or full power protection, as well as overprovisioning. One good power failure is all it takes to brick a SSD without write abort, and you will start to rack up bad clusters without overprovisioning. Some enterprise SSDs have a rebuild program for arrays that if the FTL becomes junk can reformat the drive in situ for restore of data. When a consumer SSD's FTL fails it usually is thrown out. This is for models that may use a dedicated NVRAM chip for the FTL, without a DRAM intermediary.

The most damning thing about consumer SSDs is the wear leveling algorithm; stale data which just sits, occupies low cycle pages, but the algorithm forces that data to migrate to other pages periodically to make available low use cells for further writes. If a corrupting event happens during those operations, none of your data is safe, which generally wasn't true on spinning rust devices.

Enterprise drives are generally the only models that take these factors into account, and no M.2 drives meet all of these requirements. How much is your data and time worth?

The biggest issue I've faced with SSDs is that backups become more critical, and verification restores also need to be thought of as critical due to SSDs propensity for silent corruption. It's more time consuming to do this, but a trade off for the improved cost and performance.

Last edited by Robotron2k84; 11-13-2017 at 10:05 AM..
Reply With Quote
  #11  
Old 11-14-2017, 6:02 AM
the86d's Avatar
the86d the86d is offline
Calguns Addict
 
Join Date: Jul 2011
Location: Pinko-occupied ObamaDerkaderkastan
Posts: 6,408
iTrader: 3 / 100%
Default

Quote:
Originally Posted by Robotron2k84 View Post
Consumer SSDs might get you a few years, 10 would be stretching it. MTBF for these devices in the field is <3 years.

The only SSDs worth your money are those that have write abort or full power protection, as well as overprovisioning. One good power failure is all it takes to brick a SSD without write abort, and you will start to rack up bad clusters without overprovisioning. Some enterprise SSDs have a rebuild program for arrays that if the FTL becomes junk can reformat the drive in situ for restore of data. When a consumer SSD's FTL fails it usually is thrown out. This is for models that may use a dedicated NVRAM chip for the FTL, without a DRAM intermediary.

The most damning thing about consumer SSDs is the wear leveling algorithm; stale data which just sits, occupies low cycle pages, but the algorithm forces that data to migrate to other pages periodically to make available low use cells for further writes. If a corrupting event happens during those operations, none of your data is safe, which generally wasn't true on spinning rust devices.

Enterprise drives are generally the only models that take these factors into account, and no M.2 drives meet all of these requirements. How much is your data and time worth?

The biggest issue I've faced with SSDs is that backups become more critical, and verification restores also need to be thought of as critical due to SSDs propensity for silent corruption. It's more time consuming to do this, but a trade off for the improved cost and performance.
I have yet to see an SSD die, however this was my initial concern, and why I didn't get an SSD until I could gather data from others regarding reliability and longevity. We had some early-adopters/pushers at work, and they had issues with the Crucial (Micron?) M4 firmware bug, and the drive liked to disappear for like 2-3 firmware revs.

Can one carve data off a consumer SSD that dies during a write operation that failed as you mentioned, or does the controller whack-out in this case?
__________________
"That's what governments are for - get in a man's way." - Captain Malcolm 'Mal' Reynolds
Reply With Quote
  #12  
Old 11-14-2017, 11:07 AM
Robotron2k84 Robotron2k84 is offline
Member
 
Join Date: Sep 2017
Posts: 147
iTrader: 1 / 100%
Default

You might not have seen a drive die, so to speak, but the drives themselves are only rated for 1PB of writes on average, some lower. Generally they can all exceed this to some degree, but failure is a certainty as the drive approaches it's exhaustion of useful life. There was a chart a few years back comparing ultimate write longevity and errors over time. The high end drives did what you expect, they had very few errors and then fell off the cliff and locked up. A drive is supposed to stop functioning once it hits a certain number of hard errors, via SMART.

The Samsung's on the other hand continued allowing themselves to be written to past their useful life all the while spewing SMART errors and racking up bad blocks. Those are the ones I stay far away from.

Remember, too: the MTBF of a device in the field is not the same thing as the manufacturer's stated MTBF. MTBF for the vendor means that the customer only sees that if they replace the drives before their useful life is over, stay under the maximum writes per day, keeps the drive environmentally in spec and disable disk optimizations that may cause excessive wear. In the field, MTBF is how long before there is a support ticket saying my machine won't boot or I can't read my data.

Once a consumer SSD drive bricks itself then it's toast. You would have to send it out to a forensics company for recovery and hope for the best.

But, the data would be equally suspect on an enterprise disk, the only difference is that the disk can usually be resurrected (if still within service life). Drive failure in arrays is a common scenario, but powerfail and FTL errors are not because we use redundant power. But, when it happens, you can either swap the unit or attempt a rebuild. We usually attempt a single resurrection before tossing the drive as it becomes a liability. The array is raid 50 or 60, so there are enough copies to completely rebuild the data.

There are several vendors that sell you a rack mounted storage array device that does all this transparently and guarantees the data integrity over some useful number of years and writes. Spendy stuff: 40k for a few TB

But, the three most important words in data security are: backups, backups, backups.

Last edited by Robotron2k84; 11-14-2017 at 11:32 AM..
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump



All times are GMT -8. The time now is 7:37 AM.




Powered by vBulletin® Version 3.8.9
Copyright ©2000 - 2017, vBulletin Solutions, Inc.
Proudly hosted by GeoVario the Premier 2A host.
Calguns.net, the 'Calguns' name and all associated variants and logos are ® Trademark and © Copyright 2002-2018, Calguns.net an Incorporated Company All Rights Reserved.