FTP server with compression for large text files
I need to set up an FTP server to host fairly large text files (.CSV). At the FTP client level these need to be uncompressed but I'd like to reduce server disk requirements by using compression. I guess I could either do this at a file system (FUSE) level or transparently within an FTP server, assuming there is an FTP server that can do this. Anyone got any experience with this or suggestions? -- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0844 251 1450 Registered in England (0456 0902) @ 13 Clarke Rd, Milton Keynes, MK1 1LG
Hi Mark, I'd do it at the file system level. I would ZFS for compression and deduplication. Cheers, Ewan On 13 May 2014 11:05, Mark Rogers <mark@quarella.co.uk> wrote:
I need to set up an FTP server to host fairly large text files (.CSV).
At the FTP client level these need to be uncompressed but I'd like to reduce server disk requirements by using compression. I guess I could either do this at a file system (FUSE) level or transparently within an FTP server, assuming there is an FTP server that can do this.
Anyone got any experience with this or suggestions?
--
Mark Rogers // More Solutions Ltd (Peterborough Office) // 0844 251 1450 Registered in England (0456 0902) @ 13 Clarke Rd, Milton Keynes, MK1 1LG
_______________________________________________ main@lists.alug.org.uk http://www.alug.org.uk/ http://lists.alug.org.uk/mailman/listinfo/main Unsubscribe? See message headers or the web site above!
On 14 May 2014 15:20, Ewan Slater <ewan.slater@googlemail.com> wrote:
I'd do it at the file system level. I would ZFS for compression and deduplication.
The final solution is going to end up on a hosted virtual server somewhere; I've not found any yet that offer ZFS, which is why I was looking at something like FUSE to run something on top of the default filesystem (generally ext4). I have had a quick play with avfs on my Ubuntu desktop and that looks useful - effectively allowing me to just gzip (or bzip2) the files and have them appear uncompressed to the FTP server, although it takes a bit of messing around (the transparently uncompressed version of ~/foo.gz becomes ~/.avfs/home/mark/foo.gz#, but I can create a symlink in the FTP root to make it appear there as "foo"). fusecompress isn't in the repos so I need to work a bit harder on that one. Mark -- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0844 251 1450 Registered in England (0456 0902) @ 13 Clarke Rd, Milton Keynes, MK1 1LG
But if that's the case you're going to need to tweak the VM anyway, so why not use ZFS? I've certainly used ZFS on AWS VMs. Sent from my iSheep
On 14 May 2014, at 17:04, Mark Rogers <mark@quarella.co.uk> wrote:
On 14 May 2014 15:20, Ewan Slater <ewan.slater@googlemail.com> wrote:
I'd do it at the file system level. I would ZFS for compression and deduplication.
The final solution is going to end up on a hosted virtual server somewhere; I've not found any yet that offer ZFS, which is why I was looking at something like FUSE to run something on top of the default filesystem (generally ext4).
I have had a quick play with avfs on my Ubuntu desktop and that looks useful - effectively allowing me to just gzip (or bzip2) the files and have them appear uncompressed to the FTP server, although it takes a bit of messing around (the transparently uncompressed version of ~/foo.gz becomes ~/.avfs/home/mark/foo.gz#, but I can create a symlink in the FTP root to make it appear there as "foo").
fusecompress isn't in the repos so I need to work a bit harder on that one.
Mark -- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0844 251 1450 Registered in England (0456 0902) @ 13 Clarke Rd, Milton Keynes, MK1 1LG
_______________________________________________ main@lists.alug.org.uk http://www.alug.org.uk/ http://lists.alug.org.uk/mailman/listinfo/main Unsubscribe? See message headers or the web site above!
On 14/05/14 17:21, Ewan Slater wrote:
But if that's the case you're going to need to tweak the VM anyway, so why not use ZFS?
I've certainly used ZFS on AWS VMs.
Is Fusecompress any use? https://code.google.com/p/fusecompress/wiki/Usage Steve
On 14 May 2014 22:44, <steve-ALUG@hst.me.uk> wrote:
Is Fusecompress any use?
It's looking the most likely, I just haven't tried it yet (it's not in the repos so I'm reluctant to just try it on my desktop, which means setting up a VM to play with, and I haven't had chance yet). If anyone has any experience of fusecompresss I'd love to hear about it! -- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0844 251 1450 Registered in England (0456 0902) @ 13 Clarke Rd, Milton Keynes, MK1 1LG
On 14 May 2014 17:21, Ewan Slater <ewan.slater@googlemail.com> wrote:
But if that's the case you're going to need to tweak the VM anyway, so why not use ZFS?
I've certainly used ZFS on AWS VMs.
The sort of VM I'm talking about is in a different league I'm afraid: the sort of league where £5/mo gets you 40GB storage and everything is provisioned using one of their standard images, all of which are are ext4. -- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0844 251 1450 Registered in England (0456 0902) @ 13 Clarke Rd, Milton Keynes, MK1 1LG
Who is the hosting partner out of morbid interest? On 15 May 2014 11:34, Mark Rogers <mark@quarella.co.uk> wrote:
On 14 May 2014 17:21, Ewan Slater <ewan.slater@googlemail.com> wrote:
But if that's the case you're going to need to tweak the VM anyway, so why not use ZFS?
I've certainly used ZFS on AWS VMs.
The sort of VM I'm talking about is in a different league I'm afraid: the sort of league where £5/mo gets you 40GB storage and everything is provisioned using one of their standard images, all of which are are ext4.
-- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0844 251 1450 Registered in England (0456 0902) @ 13 Clarke Rd, Milton Keynes, MK1 1LG
_______________________________________________ main@lists.alug.org.uk http://www.alug.org.uk/ http://lists.alug.org.uk/mailman/listinfo/main Unsubscribe? See message headers or the web site above!
On 15 May 2014 11:58, Ewan Slater <ewan.slater@googlemail.com> wrote:
Who is the hosting partner out of morbid interest?
We usually use Memset (mostly through habit!). The application here is pretty trivial though; we just need to store a few years' worth of data for a few devices that generate it in CSV format, and need a limited amount of scripting capability on the server (primarily for mirroring to clients servers) which means that a simple cloud storage package isn't sufficient. -- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0844 251 1450 Registered in England (0456 0902) @ 13 Clarke Rd, Milton Keynes, MK1 1LG
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Dropbox or a NoSQL database in the cloud? On 15 May 2014 13:52:17 BST, Mark Rogers <mark@quarella.co.uk> wrote:
On 15 May 2014 11:58, Ewan Slater <ewan.slater@googlemail.com> wrote:
Who is the hosting partner out of morbid interest?
We usually use Memset (mostly through habit!).
The application here is pretty trivial though; we just need to store a few years' worth of data for a few devices that generate it in CSV format, and need a limited amount of scripting capability on the server (primarily for mirroring to clients servers) which means that a simple cloud storage package isn't sufficient.
-- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0844 251 1450 Registered in England (0456 0902) @ 13 Clarke Rd, Milton Keynes, MK1 1LG
_______________________________________________ main@lists.alug.org.uk http://www.alug.org.uk/ http://lists.alug.org.uk/mailman/listinfo/main Unsubscribe? See message headers or the web site above!
- -- Sent from my Android device with K-9 Mail. Please excuse my brevity. -----BEGIN PGP SIGNATURE----- Version: APG v1.1.1 iQFNBAEBCgA3BQJTdLpNMBxFd2FuIFNoZXBoZXJkc29uIFNsYXRlciA8ZXdhbi5z bGF0ZXJAZ21haWwuY29tPgAKCRAN8Sm/pEjHno5dB/98+bqrOsWlqPyNUg3QyCme uWNOhKf0Yahdh7vwWc6gsUokNjQU6LG8oqFLG16YY86BmdNxVdaiwc3rCuzFjCce eQYqx8zGA4ZGVYc2lo5jca40dWTMXNcvY7JTewLIuwqx0pkmHqh0GQUY8ghk+edc 43wGonb6KUdrU5PyGkqrPsgwUSkjfjdgw6/rTacMheNROeb5S+BNgYn7wWk7Gn/U LwNkdLGx3nxH9MVfO76cnXaOAv17+JpWkrSq36M5mn8NsM26cze6RSHfuv3ahpa6 CR5SUXh93TNg6OP2PUWE2h6kv+LQf3qihPNxB4VvkXZFYj5X4h1c50ULWKHWWcdi =T0cx -----END PGP SIGNATURE-----
On 15 May 2014 13:59, Ewan Slater <ewan.slater@googlemail.com> wrote:
Dropbox or a NoSQL database in the cloud?
We currently use some pretty cheap cloud storage which is fine, but we now have a requirement to run some scripts on the data so we need to move away from pure storage. In truth though, no matter where the data is stored I have an instinctive dislike for storing large text files in an uncompressed format. Maybe this goes back to using Stacker under DOS all those many years ago, but the fact is that CPUs are faster than disks and generally it's quicker to retrieve a compressed file and uncompress it than it is to retrieve an uncompressed file, so as long as it can be done transparently that would be my preference. One of the few features I miss from Windows is the NTFS compression option, which I regularly use in areas where it makes sense (eg log directories). In contrast Linux makes it incredibly easy to work with compressed files (eg .gz/.bz2), but not so much when the compression needs to be "invisible". Mark -- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0844 251 1450 Registered in England (0456 0902) @ 13 Clarke Rd, Milton Keynes, MK1 1LG
participants (3)
-
Ewan Slater -
Mark Rogers -
steve-ALUG＠hst.me.uk