A note that this setup runs a 671B model in Q4 quantization at 3-4 TPS, running a Q8 would need something beefier. To run a 671B model in the original Q8 at 6-8 TPS you’d need a dual socket EPYC server motherboard with 768GB of RAM.
A note that this setup runs a 671B model in Q4 quantization at 3-4 TPS, running a Q8 would need something beefier. To run a 671B model in the original Q8 at 6-8 TPS you’d need a dual socket EPYC server motherboard with 768GB of RAM.
Sure, but you really have to watch out what kind of hard drives you’re buying. There are a lot of SMR drives out there that are sold as regular drives and the only way to tell is to look through their data sheets. I find that “regular” HDDs (CMR/PMR) cost more now than SMR drives of similar capacity and spindle speed (probably because nobody wants them lol).
SMRs are meant for data storage aren’t they? Which is not to say they can’t write at all, they just don’t have as high speeds.
For AI models specifically the file just lives on the HDD, it gets loaded into Vram (and then CPU and RAM if you don’t have enough Vram for it) when you use it. For everything else then probably yeah, I don’t even know what kind of HDDs I have lol. Seems difficult to find 7200 rpm ones over 5400 but tbh with the prices of SSDs nowadays, I’m probably going to replace my last HDDS with SSDs. If you’re looking for 10tb or huge archival size then it’s probably still worth getting an HDD, but for a 1-2tb drive it makes more sense to go SSD I think.
Well if you’re downloading, copying or creating large models that are several hundred GBs you’re going to want a normal drive. SMRs have a small staging area and once that is full it has to start re-ordering the data on the platters. Once your drive is in the process of re-ordering your write speeds are going to make it look like a failing floppy disk. I had a large file copy operation (>1TB) to a RAID pool of SMRs take like 16 hours. And I also found out that my backup drive is SMR because it took several days to do a full backup from scratch, which caused me to look up its detailed specs.
It always starts out looking great but eventually the staging area will get full and then your CPU will spend most of its time twiddling its thumbs until a chunk of staging area becomes available again; Repeat until operation is done. The greatness of Shitty Magnetic Recording.
If you want to know what recording method your drive uses, you grab the model number from:
# smartctl -a /dev/YOURDRIVE | grep "Device Model"and look that up in a search engine. It should either lead you to a Data sheet or the manufacturers website where they list the specifications.
If durable and large SSDs were more affordable where I am I’d slowly replace all my spinning rust. But right now HDDs are overall still the better option for me, at least for mass storage.