Backing up files to external HDD

No

>tfw always wondered why many small files take so much longer
is it just more reallocation?

something efficiency differences between sectors in controller something something

Actually, yes. Well if you have a pcie slot, there are expansion cards for that.

>
>>tfw always wondered why many small files take so much longer
>is it just more reallocation?

Assume 4k blocks, for every file on average, that 4k will be 1/2 empty. Still has to copy entire block. 100,000 4k files balloon to 390MB. and 1/2 of that is wasted. See the advantages of COMPRESSION?

7zip is worth its weight in GOLD.

>not using infra red + time machine

Tell me Cred Forums my external hard drive 1TB is disgracefully fragmented and it was taking pass after pass so I just stopped it, is it better for me to just reformat the drive and start from scratch? I can put everything that I have on it on another drive to put it back on after.

>not using time machine to start copying a month ago

im using usb 3 on win8.1
maybe itd be faster if i cloned the whole disk?

It's slow because you're copying a bunch of small fils.
What you should do:
1. Create a new ZIP/RAR/7Z file on your external drive
2. Put all your files in it
4. ????
5. PROFIT!

Why are you even here? You know so little about tech it's just painful. Leave now.
It's slow because it's an EXTERNAL hard drive. That means it's connected via USB. If he were using an internal hard drive with SATAIII he could get a bandwidth throughput of up to 600 MB/s. The moral of the story is don't store things on external hard drives. Use internal hard drives.

>yfw you find out USB 3 interferes with bluetooth and you have to go back to USB 2

It's pretty simple, segmented load will always be much slower than a sequential load for HDDs, no matter the interface.
He can either go out and buy a faster interface or make his load sequential to get a sizable boost for free.

Fuck off my Cred Forums you consumerist whore.

What's your profound opinion on eS-ATA?

It takes less time if you stop staring at the progress bar friend.

I remember doing a clean install and 500gb took about 45minutes on an external with usb2.0
Just what are you trying to back up user?

>2016
>not just taking the hdd out and using pic related

>you consumerist whore
Internal hard drives have been around longer than external hard drives. The people who buy external hard drives are the normie consumerist whores who don't understand that USB is going to be much slower than SATA. SATAIII has been around for a long time, and you can buy a SATAIII capable 1TB drive for under $70. External harddrives add EXTRA MARKUP to that price, because they know that normies (i..e consumerist whores, much like yourself) either don't have desktops or don't like to open up their desktop to make sure they safely store all their files on backup internal drives.
It's fine. Just don't expect the same speeds out of USB. That's what this thread is all about. I suppose I should have said "most external drives" instead of just saying all of them are like that. The person I was replying to was trying to give the wrong solution to a problem.

It'll have to write information about each file to the file system and also check that there are no conflicts. And allocate space one file at a time.

If you can reformat and place your files back then that might be a better choice, yes.
Assuming you use Windows, you can opt to use a greater block size during your format.
After backing up your files, this is how you do that in Windows:

>Open diskpart in Windows, or type "diskpart" in a command line;
>In the command line type "list disk";
>Find the disk you want to format, say "Disk 3";
>Type in "select disk 3";
>Type in "clean";
>Type in "convert gpt";
>Type in "create partition primary";
>Type in "format fs=ntfs unit=64K quick".

In order to prevent fragmentation when downloading torrents, most torrent clients have an option to "Pre-allocate disk space for all files", in Qbittorrent for example under Options > Downloads. Setting this option makes torrents starting take a few seconds longer, but prevents stalling and fragmentation.

>SATAIII capable 1TB drive for under $70
This was also a mistake, I overestimated. You can get them for like half that, but I tend to stick with WDs which are a bit more expensive than seagate or whatever.

You should always do a full format (i.e. remove the "quick" from the last command) whenever you get a new hard drive. Unfortunately, if it's connected via USB and it's large, it will take a long time. However, doing a full format will save you from potentially losing your files and your mind if there are bad sectors on the drive; which is certainly possible if your hard drive gets tossed around during shipment.

Good point. I suppose a quick format is fine, in this case, since the drive in question has been used for a long enough time to get thoroughly fragmented.

Bullshit usb 2.0 is not transfering with 40 if you are lucky its about 20 to maximum 30

Read the OP faggot, he asked for a better way.
Better ways are both optimizing the data and optimizing the interfaces/hardware.
Saying my solution was straight up wrong is a flat out lie.

Of course the best solution would be to get a thunderbolt SSD and call it a day.

Jesus fuck kids these days...

>not using btrfs send
fliping macfag
-posted from mpb

how much does 7zip weigh?

What do you mean exactly, that is simply reduces the filesize and thus the amount of empty space copied, or that compressed files don't behave as you described when copied?

Learning how memory actually behaves is interesting but I am quite the retard.

You don't need an SSD. You need SATA.
Even external hard drives, when taken out of their enclosure, are normally just bare (internal) drives.
If he were to take his external out of its enclosure, he could transfer files quickly to it.
However, if there is an encryption component to it (usually they do this to an encryption-capable HDD with the attachment to the SATA interface which also acts as a SATA-to-USB converter), and you used it (which some drives may enable by default, even if you don't set a password), then you probably wont be able to access those files without USB (you'd also have to reformat the drive just to see it).

To the poster I am replying to: You are ignorant. So very ignorant. I'm not telling him to throw extra money at the problem. And I'm not giving him stupid advice that won't help. Maybe you'll reduce the copy time from 3 days to 2 days and 18 hours...but then the files are compressed. Every time he needs to access those files he'll have to wait longer to access them. It's best to spend the time writing to disk rather than more time reading from a disk every single time you need to do something.

I use Clonezilla, (and when it for some reason doesn't work right, ddrescue,) for my two external HDD backups of my internal HDD. I like the idea that I can just easily plug in what's basically my entire system into anyone's halfway-decent computer if I want to. Am I still a consumerist whore?

Storage is divided into blocks/clusters. The default size of a block is 4096 bytes. So your storage device, is filled with tiny blocks each the size of 4096 bytes. When you write a file to your storage device, it fills in these small blocks. So a file the size of 4096 bytes fits neatly into one block. But if a file is the size of 8192 bytes, then it fits neatly into two blocks.
However, if your file is 3072 byte large, and you write it to the storage device, then it's 75% the size of the block, but that same block cannot be used for another file.
If you have 100 text files of 20 bytes. Then instead of taking up (100*20) 2000 bytes of disk space; not even half a block, it takes up 100 blocks, each 4096 bytes large to store the files. So instead of 2000 bytes of disk space used, you have (100*4096) 409600 bytes used.

That poster explains how you can remedy this by compressing your files, with for example 7zip. You take all the files, pack them together into one file, and thus don't have hundreds of half-empty blocks.

It can be useful for long term storage.

3TB of pure selected jav

I like how you think he's capable of dismantling his external HDD and plugging it internally without fucking everything up. I did what you recommended myself once, but I'm no stranger to poking around my PC.

Filesystem does not work per-bit but rather per-block. When you write, the whole block must be written, even if you only need couple bits from it.

The file-size reduction isn't the important bit, it's the fact that every file will, at it's end, probably bleed into a new block, and when you copy the file that whole block will have to be copied (size depends on how you formatted the drive). If you have many small files you'll have more of these mostly empty blocks to copy taking up more time.

100 small files = 100 of these mostly empty blocks.
1 big file = 1 mostly empty block.

Putting small files into an archive will make them all into one file as far as the drive is concerned, cutting down on copying mostly empty blocks.

Also you remove the overhead of allocating the file 100 times in MFT (or your filesystem equivalent) instead of just one allocation of a really big file.

Here as an example. 46 text files taking up double the size on disk.

>Putting small files into an archive will make them all into one file as far as the drive is concerned, cutting down on copying mostly empty blocks.
>Also you remove the overhead of allocating the file 100 times in MFT (or your filesystem equivalent) instead of just one allocation of a really big file.
This sums it up quite well. Also, provided your file system is not too fragmented, one file is stored in mostly adjacent blocks, which means that your disk doesn't have to seek back and forth all the time.

next time use raid

>using raid as backup
shiggydiggy

I see, that makes a lot of sense. I appreciate the replies Cred Forumsents.

>not using SSDs for storing all of your files
Do you want your files to degrade due to rotational velocidensity?

Faggots he's backing up data

dude just no

I have a pci express 1x to usb 3.0 and I get 100mb transfer speeds, worth the 20 bucks

2MB is pretty much the perfect file size for batching up multiple small files but still keeping them somewhat seperate.