undefined | Better HN

0 pointswalrus014y ago0 comments

> The problem here would be the "filesystem" (read: the distributed service): I don't have much experience with Ceph,

I think at that scale you would want a ceph expert on staff as a full time salaried position.

For an organization that has 10PB now and can project a growth path to 15, 20, 25PB in the future, you should talk with management about creating a vacant position for that role, and filling it.

> EDIT: btw, we let the dead drives "rot": replacing them would cost more, and the failure rate is not that bad, so they stay in the machine, and we disable them in fstabs, configs, etc.

I am a huge advocate of hosting stuff yourself on bare metal you own, but this is a ridiculous statement. Any drive in that class should come with a 3 or 5 year warranty. And the manual labor and hassle time to replace one (you have hundreds of thousands of dollars of storage and no ready to go cold spares on a shelf?!?!) is infinitesimal.

0 comments

pmlnr4y ago

OK, clarification: most of our fleet has a LOT of supermicro machines, where it's impossible to identify the drive unless by serial number. There's on UID light, the machines needs to go offline, plus some 10 screws needs to come out to open the chassis, 4 more by drives.

The amount of downtime this would generate for a single machine plus the operation cost doesn't worth the hassle unless the machine loses a significant chunk of drives.

barrkel4y ago

Chassis built for mass storage usually have lever caddies backed on to hotplug SAS backplanes.

pmlnr4y ago

One would think indeed, but not the early FatTwins.

closeparen4y ago

If the colo is far and there’s plenty of headroom, it might not justify much urgency.

cortesoft4y ago

That's what remote hands are for. Yes, you batch the replacements up, but this is exactly what remote hands are for.

Terretta4y ago

Costs less to leave them alone, and go once a year for a trash run. Cattle not pets, no trips to the vet. Don’t waste money diagnosing / fixing.

2 more replies

j / k navigate · click thread line to collapse

0 pointswalrus014y ago0 comments

> The problem here would be the "filesystem" (read: the distributed service): I don't have much experience with Ceph,

I think at that scale you would want a ceph expert on staff as a full time salaried position.

For an organization that has 10PB now and can project a growth path to 15, 20, 25PB in the future, you should talk with management about creating a vacant position for that role, and filling it.

> EDIT: btw, we let the dead drives "rot": replacing them would cost more, and the failure rate is not that bad, so they stay in the machine, and we disable them in fstabs, configs, etc.

0 comments

pmlnr4y ago

The amount of downtime this would generate for a single machine plus the operation cost doesn't worth the hassle unless the machine loses a significant chunk of drives.

barrkel4y ago

Chassis built for mass storage usually have lever caddies backed on to hotplug SAS backplanes.

pmlnr4y ago

One would think indeed, but not the early FatTwins.

closeparen4y ago

If the colo is far and there’s plenty of headroom, it might not justify much urgency.

cortesoft4y ago

That's what remote hands are for. Yes, you batch the replacements up, but this is exactly what remote hands are for.

Terretta4y ago

Costs less to leave them alone, and go once a year for a trash run. Cattle not pets, no trips to the vet. Don’t waste money diagnosing / fixing.

2 more replies

j / k navigate · click thread line to collapse