Hard Disk Storage...wha

[matsuki]

With all the HD resolution videos of me hockey games, music, and other goodies, I now have an array of external Hard Disks as monitor garnish Every time I buy another one, I keep telling myself there has to be another way...as I quietly ponder how many month it will be until I need the next one.

What are you guys using to store your massive caches of data (looking at you Coligny.....owner of the world's most comprehensive Yua Aida database and movie gallery) and other such assery? I can't be the only one supporting Buffalo's external drive division.

[cstaylor]

Thought about getting one of these?

http://h10010.www1.hp.com/wwpc/us/en/sm/WF05a/15351-15351-4237916-4237918-4237917-4248009.html

You'll need to purchase 4 2TB hard disks. If you boot the OS off of a USB memory key, you'll get the full 8 TB if you JBOD or RAID-0 them together.

I home-built my own storage system using a tall tower case and a spare set of MB/CPU/RAM. Since I'm the only user of the system, I went with Linux software RAID since the configuration is more portable if the motherboard dies.

As for software, OpenFiler works pretty well for what I need: http://www.openfiler.com/. But if you don't need iSCSI, you could install Ubuntu and run a SAMBA/Netatalk file server on there.

[6810]

With all the HD resolution videos of me hockey games, music, and other goodies, I now have an array of external Hard Disks as monitor garnish Every time I buy another one, I keep telling myself there has to be another way...as I quietly ponder how many month it will be until I need the next one.

What are you guys using to store your massive caches of data (looking at you Coligny.....owner of the world's most comprehensive Yua Aida database and movie gallery) and other such assery? I can't be the only one supporting Buffalo's external drive division.

Too easy.

Record. Watch. Delete.

Plenty more shit tomorrow. Plenty more shit next week. Plenty more shit ad infinitum.

[cstaylor]

Record. Watch. Delete.

Good point.

[Yokohammer]

Record. Watch. Delete.

Plenty more shit tomorrow. Plenty more shit next week. Plenty more shit ad infinitum.

This is signature-quality material.
Very nice, and so true.

[GomiGirl]

Thought about getting one of these?

http://h10010.www1.hp.com/wwpc/us/en/sm/WF05a/15351-15351-4237916-4237918-4237917-4248009.html

You'll need to purchase 4 2TB hard disks. If you boot the OS off of a USB memory key, you'll get the full 8 TB if you JBOD or RAID-0 them together

We are just putting something like to together for ourselves at home too. I think we are going down the Drobo road though.

http://www.drobo.com/products/professionals/drobo/

And buying 2 x 2TB disks. 4 disks seems a little much but we can expand if need be. We are just waiting for the costs of disks to come down again after the spike post Thai floods.

Basically I want some backups for my photos and music as these are what I care about the most.

Might do an off island/cloud back-up from time to time too just for that extra "just-in-case-there-is-another-earthquake-and-my-house-falls-down" type of redundancy.

[matsuki]

Hmmm, so if I want massive storage you guys are all suggesting servers with massive arrays of HDs?

Too easy.

Record. Watch. Delete.

Plenty more shit tomorrow. Plenty more shit next week. Plenty more shit ad infinitum.

Very very very true but besides hoarding, I seem to be accumulating data at a much faster rate than I can view/use/delete. For example, I am videoing my games with 2, soon to be 3 video cameras in 30fps HD...30GB, soon to be 45GB per game. As I described, it's getting a bit crazy.

[waruta]

Hmmm, so if I want massive storage you guys are all suggesting servers with massive arrays of HDs?



Very very very true but besides hoarding, I seem to be accumulating data at a much faster rate than I can view/use/delete. For example, I am videoing my games with 2, soon to be 3 video cameras in 30fps HD...30GB, soon to be 45GB per game. As I described, it's getting a bit crazy.

I looked into this last week or so, found quite a bit of pros/cons on both Drobo and ReadyNAS solutions. They are both pricey, and many reviews have remarked on the software that comes with it as well as the "vendor-approved" hard disk list for ReadyNAS, and in the end, for best performance, setup and usability, the Synology DS212j or DS212 looks to be a winner.

Just stuff it with 2 x 3TB drives and you could have a NAS accessible through the internet on a special encrypted home page as well as 6TB of storage (without RAID) for under 600 USD.

Plus it only weighs a kilo without the drives. Read/Write is about 110MB,54MB/s respectively.

Fancy Synology Performance Graphs here:

[tone]

very timely post

i think watch and delete is a good one but some of this stuff is worth hanging onto for a while

for your own content - I would consider using something like mpeg stream clip and batch encoding this stuff down to h.264 ~1300 k/s - if its self generated stuff, unless its gorgeous, you could probably get away with dropping to 720 also, as blown up on your tv or projector, that will still look similar and have less need for harsher compression.

what kind of games are you filming?

[Coligny]

What are you guys using to store your massive caches of data (looking at you Coligny.....owner of the world's most comprehensive Yua Aida database and movie gallery) and other such assery? I can't be the only one supporting Buffalo's external drive division.

Me... and my electricity bill are too hardcore for youz...

Currently 1 Intel I7 with 12 GB of ram and SSD for the system. with 2 internal sata arrays:

6 x 3 Tb soft raid 5
4 x 2 Tb soft raid 5
+an older USB 8 x 250 hardware raid 0
Total is around 20 Tb...

Expansion will need eSata port expander cards and outside boxes...

No LVM... because the second something goes strange everytime I used LVM it turned a minor problem into an Extinction Level Event.

The computer host 2 more Virtual machine:
1 Main DNS/DHCP/WINS/ intranet
1 Prototype CCTV server (using the USB array as video scratch disk)

Beside him their is the Older CCTV server (PMac G3 BW) still used by 1 older crappy cam but mainly used for network monitoring. Waiting for me to be bored enough to go back to Nagios and MRTG. (super fun but I tend to go nearly autistic when I focus too much on this, option not moving from under the covers on my bed with sandwiches and laptop as only lifestyle -oh... and dutch ovens-)

Still on the cardboard is another I7 waiting to become the next CCTV server. Still need a new SSD for the system and maybe some ram (not sure, can't remember...). And certainly a new battery bank for the UPS, I don't know if it shut down because of their age or because the management software on the Mac a complete fuckup...

Usually... any hiccups on these take the whole week to solve... not helped by the fact that a simple health check on the disk arrays take several days.

[cstaylor]

I looked into this last week or so, found quite a bit of pros/cons on both Drobo and ReadyNAS solutions. They are both pricey, and many reviews have remarked on the software that comes with it as well as the "vendor-approved" hard disk list for ReadyNAS, and in the end, for best performance, setup and usability, the Synology DS212j or DS212 looks to be a winner.

From past and painful experience, I can't recommend a proprietary RAID array system using commodity hardware. The benefit of the HP system is that it's an SFC for building the system yourself. For Mac users, any Linux running netatalk can act as a TimeCapsule: http://www.kremalicious.com/2008/06/ubuntu-as-mac-file-server-and-time-machine-volume/

Linux software RAID isn't difficult to set up, and it isn't dependent on underlying RAID hardware, so the disks can be moved to another system without any problems.

[Coligny]

Linux software RAID isn't difficult to set up, and it isn't dependent on underlying RAID hardware, so the disks can be moved to another system without any problems.

With webmin it's simple as clicky click... and the arrays even mount without any problem inside a virtual machine. The Host computer was running XP, with a VMWare workstation running Linux mounting directly the Array. Plug and play with no glitches.

[zero_oki]

http://www.dlink.com/products/?pid=509

DNS 323 NAS
1TB RAID 1 config

Simple, quiet, low power use, easy to manage, and works well for HTPC storage.

[plaid_knight]

I've accumulated a lot of stuff to watch as well on multiple hard disks.
Several people have made good suggestions (h.264 encoding) already. For the audio, you could encode that to aac (though I use ac3 because it works better with my amp).

If it's not absolutely gorgeous, you can always resize to 720p or 480p.

In regard to the storage problem, I just connected an old laptop to a lot of usb external hard disks connected through a hub. I set them up to be shared through my network and then connected the laptop to a network switch. I can access that throughout the house - and I don't have to check on it that often.

[Coligny]

FUCK BEANS... Just had a kernel panic while copying 13Gb over NFS...

That's it... UBUNTU is soooo out of my house...

Plaid Knight... I feel dirty to know where your avatar come from...

[matsuki]

very timely post

i think watch and delete is a good one but some of this stuff is worth hanging onto for a while

for your own content - I would consider using something like mpeg stream clip and batch encoding this stuff down to h.264 ~1300 k/s - if its self generated stuff, unless its gorgeous, you could probably get away with dropping to 720 also, as blown up on your tv or projector, that will still look similar and have less need for harsher compression.

what kind of games are you filming?

Mostly my hockey games but these lil GoPro cameras are addictive...and waterproof! Taking all kinds of videos with them. Good idea on shrinking the file size...but am I crazy for wanting to keep a copy of the raw data somewhere?

Last night I went in and deleted about 300GB of video...which should buy me another week or two. I just don't know if I need a server...as I don't need instant access to all this stuff at once. Just storage space. I guess I was hoping for something that doesn't yet exist...some sort of 100 terrabyte USB hard disk that I was just not finding. With the speed that HD space is increasing, it's just a waiting game I guess. In the mean time, maybe I should start burning all this shit on BluRay to free up space? It seems to be the cheapest solution (in initial cost and electricity) though I was really happy to rid myself of all those CDs, DVDs and other shit I had to sort through every time I had to find something or moved.

[cstaylor]

FUCK BEANS... Just had a kernel panic while copying 13Gb over NFS...

That's it... UBUNTU is soooo out of my house...

Plaid Knight... I feel dirty to know where your avatar come from...

Post the panic trace here or send it to me by PM... I'd be interested to see if one of the dm_* routines is in there.

[IparryU]

FUCK BEANS... Just had a kernel panic while copying 13Gb over NFS...

That's it... UBUNTU is soooo out of my house...

Plaid Knight... I feel dirty to know where your avatar come from...

i been using kubuntu for a while now and it is going ok for me thus far...

what distro are you goin to use now?

[GomiGirl]

Hmmm, so if I want massive storage you guys are all suggesting servers with massive arrays of HDs?

No - these are just a more efficient use of space than having daisy chains of Lacie or Buffalo external harddrives.

These are not servers, just storage arrays in a single box that has up to 4 HD's installed. Will take up less space on your desk than now.

[matsuki]

No - these are just a more efficient use of space than having daisy chains of Lacie or Buffalo external harddrives.

These are not servers, just storage arrays in a single box that has up to 4 HD's installed. Will take up less space on your desk than now.

Thanks for the clarification GG. That I would be interested in.

[waruta]

No - these are just a more efficient use of space than having daisy chains of Lacie or Buffalo external harddrives.

These are not servers, just storage arrays in a single box that has up to 4 HD's installed. Will take up less space on your desk than now.

Like this Chokonen:




But as everyone above liked to point out, they don't like using NAS (Network Attached Storage) units with proprietary software and such, but it does make setup simple if you don't know what you are doing with regards to Linux or RAID or whatnot. Think of this as the Plug 'n Play solution to storage. (God am I gonna catch some more heat from that last comment?) :flame:

[matsuki]

Thanks Waruta! I've worked with Linux before and my PC is setup with a small RAID array but yeah, that is more what I'm looking for right now. Don't have the time to relearn and set something up myself.

[Coligny]

Post the panic trace here or send it to me by PM... I'd be interested to see if one of the dm_* routines is in there.

Traditionnally, I NEVER investigate kernell panics. My approach is:

1st KP, check everything is seated properly, do a ram check.
2nd KP change the Linux distribution
3rd KP change the hardware.

So if you want panic trace... send me the URL of an How To and I'll see in a stress test if it KP's again then I'll send it to youz...

But I already got a new SSD and be getting ready for VMWare ESXi 5 level 1 hypervisor this time. (instead of Linux-KVM)

[Coligny]

So the system ssd totally crapped on me, replaced it, switched to centos and working on a reconfig of KVM...

[cstaylor]

So the system ssd totally crapped on me, replaced it, switched to centos and working on a reconfig of KVM...

Make sure to mount / as noatime... and don't you mean LVM?

[Coligny]

Make sure to mount / as noatime... and don't you mean LVM?

nope, KVM the virtual masheen thingamabob that work with libvirt stuff...

will check mah fstab though...

[cstaylor]

nope, KVM the virtual masheen thingamabob that work with libvirt stuff...

will check mah fstab though...

Ah, got it. Make sure to either disable swap or move it to your non-SSD disks. I also suggest mounting /var, /tmp, /home, and /root to your non-SSD disks. Not sure if Linux allows those directories to be symbolic links; if so, you can make a single partition on your non-SSD drives and mount it to say, /media/writetome or something, then have directories on there for /var, /tmp, /home, and /root.

[gaijinpunch]

Simple solution that is both safe (probably) and easy to transfer.

1: Get a 2nd HD for your machine. Preferably as big as you can find. Mine is 3TB.
2: CrashPlan+ account. Unlimited cloud storage for pretty cheap -- no need to have a RAID setup at home
3: When the disk fills up, get a bigger one. You can either buy a device to mount your old drive after putting in the new one or you could, in theory, resync from your cloud but that will take ages.

If you can fill up the the largest drives with your crap immediately, you can do the same premise but you'll need a 3rd drive, and the transfer(s) will become more cumbersome. If this is you, I'd recommend an external for storage of gomi... things you don't need regularly or quickly... like TV shows.

I had a 1TB raid setup on a 2nd computer that was just getting in the way. I finally retired the thing and went cloud. Of course, I've never lost my data and had to get it back yet, so the joke may be on me.

The problem w/ NAS devices I've found is the performance generally sucks for one reason or another. I use Linux mainly, w/ some Mac on the side (and very, very little Windows). Going through Wifi alone is painfully slow. Not all support NFS (at least was the case a while back) and CIFS is complete shit. I'm happy with my setup now, but I do have to go through the BT bucket every month or so and clean house.

/rant

[cstaylor]

The problem w/ NAS devices I've found is the performance generally sucks for one reason or another. I use Linux mainly, w/ some Mac on the side (and very, very little Windows).

OpenFiler is nice, and the performance is good. I wouldn't use CIFS though; pop netatalk on there and go AFP for your Mac clients (including TimeMachine support).

Wifi pre-n was very bad; as long as you're not loading down the wireless network, 802.11n gives good performance.

If you can swing it, and your NICs and switch support it, go wired ethernet with jumbo frames enabled.

[gkanai]

If you can swing it, and your NICs and switch support it, go wired ethernet with jumbo frames enabled.

How does one enable jumbo frames?