Why Small File Transfers Are Slow (and How to Speed Them Up)

A 100 GB folder can take much longer to copy than a 100 GB file. Every item in the folder needs a path lookup, metadata handling, an open and close operation, and a destination entry. A transfer may also coordinate those operations over the network.

The right improvement depends on where that repeated work accumulates. This guide explains how to separate filesystem overhead from network waiting, then choose between packaging, session reuse, fewer scans, and controlled concurrency.

Why file count changes transfer time

For planning, divide the elapsed time into these components:

Total time ≈ payload time + file-list work + per-file overhead + storage work + verification

Payload time is the part most bandwidth calculators show. If 100 GB crosses a sustained 1 Gbps path, the byte movement has a predictable lower bound. But a directory is not only bytes. For each file, a transfer tool may need to:

1. Read its path, size, permissions, and timestamps.
2. Open the source file.
3. Announce or negotiate the next transfer.
4. Open or schedule a data operation.
5. Create the destination file.
6. Send the payload.
7. Flush, close, and record completion.
8. Optionally hash or verify the result.

Some implementations overlap these steps. Others perform much of the sequence serially. Either way, increasing file count increases work that does not appear in the headline file size.

Microsoft documents the same effect for SMB: small-file copies spend proportionally more time creating files and processing protocol and filesystem operations than moving payload data. Microsoft: Slow SMB file transfer

Traditional FTP also separates the control connection from data connections used for file transfers, so a directory can repeatedly pay setup and coordination costs as it advances through the file list. RFC 959: File Transfer Protocol

Rsync pipelines file transfers and modern versions begin recursive transfers before the full tree is scanned, but it must still enumerate entries and apply metadata. Its official documentation is a useful reminder that protocol efficiency does not eliminate filesystem work. See the rsync man page and the practical guide for diagnosing slow rsync over a WAN.

Diagnose the bottleneck before choosing a fix

A simple latency example

Imagine moving 100 GB between endpoints with a 100 ms round-trip time:

• Workload A: one 100 GB file
• Workload B: 100,000 files averaging 1 MB

The payload total is the same. The number of file operations is not.

If a workflow serialized even one additional network round trip per file, the simplified waiting-time budget would be:

100,000 files × 0.1 seconds = 10,000 seconds

That is 2 hours and 46 minutes of waiting before counting the payload, disk work, encryption, or verification. Real tools may pipeline requests, reuse sessions, or open multiple streams, so this is not a prediction for every protocol. It shows why per-file latency is the variable that matters.

At 2 ms on a local network, the same simplified penalty is about three minutes. At 100 ms across a continent, it is measured in hours.

What a real small-file copy showed

In September 2026 we copied 65,536 files of 4,096 bytes each, 268.4 MB in total, between VMs in Germany and the US about 141 ms apart. OpenSSH’s recursive SFTP client handles files one after another, and it did not finish within the 15-minute limit in any of five runs. Rsync and a tar stream through SSH keep data moving across file boundaries, and both completed at about 10 MB/s. The small-file transfer benchmark reports every run, a separate TCP and QUIC test, and the limits of the setup. The method and recorded results are also published on the benchmark research page.

The four bottlenecks behind small-file transfers

1. Per-file network round trips

Chatty protocols wait for remote acknowledgement or coordination between steps. Distance turns a small wait into a repeated tax.

2. Filesystem metadata and open/close operations

The source must enumerate the directory and read metadata. The destination must create entries, update metadata, and flush changes. Fast networking cannot eliminate storage IOPS limits.

3. Serial hashing, encryption, and scheduling

If a tool scans the entire tree, then hashes everything, then encrypts, then starts sending, the network sits idle between stages. The slowest serial stage controls the wall-clock result.

4. Tiny payloads that never fill the path

A small file may finish before a single stream has enough data in flight to use a high-bandwidth, high-latency route efficiently. A nominal 1 Gbps connection is not useful if the application repeatedly pauses.

Why creating a ZIP file often helps

Archiving a directory before transfer converts many file operations into one continuous stream. That is why tar, ZIP, and similar advice appears so often in system administration discussions.

It works, but it moves the workflow cost somewhere else:

• The sender may need temporary disk space for the archive.
• The recipient may need the same space again to extract it.
• Archiving and extraction add wall-clock time.
• A changed file may require rebuilding or resending a large archive.
• Resume and verification happen at archive granularity unless the tool adds its own index.

For an occasional folder, making an archive can be the simplest answer. For repeated production workflows, the transfer system should handle the packing problem itself.

How WarpSend’s parallel packing helps

WarpSend treats a directory as a continuous transfer workload instead of a queue of isolated uploads.

Its parallel packing pipeline overlaps:

• directory scanning;
• content hashing;
• packing and scheduling;
• encryption;
• network transmission; and
• destination writes.

Bounded concurrency keeps useful work in flight without loading the entire directory into memory. The transfer uses a persistent encrypted pipeline, so the long-distance route does not go idle while one stage finishes every file in sequence.

Compare the current releases using the same files, route, storage, and security settings. Record transfer-phase throughput separately from the time until the recipient has a verified copy. The WarpSend vs FTP benchmark method explains how to run that comparison; an FTP result does not establish performance against SFTP or Aspera.

Read the full WarpSend vs FTP benchmark and methodology.

When parallel packing will—and will not—help

Parallel packing is a strong fit when:

• a directory contains hundreds or thousands of files;
• endpoints are separated by high round-trip latency;
• a workflow repeatedly sends media assets, project trees, datasets, or NAS folders;
• encryption, resume, and verification must remain part of the transfer; or
• manually creating and extracting archives is operationally expensive.

It matters less when:

• the workload is already one large sequential file;
• both systems are on the same low-latency LAN;
• the storage devices are slower than the network; or
• a tuned transfer already sustains the practical limit of the path.

On a clean route with one large file, tuned FTP can be extremely fast. File pattern changes the winner.

How to benchmark your own folder

Use three datasets with approximately the same total size:

1. One large file.
2. A medium batch, such as 16 files.
3. A small-file batch, such as 1,000 or 100,000 files.

For every run, record:

• file count and average file size;
• source and destination storage;
• bandwidth, RTT, and packet loss;
• encryption and verification settings;
• wall-clock completion time;
• transfer-phase throughput; and
• a cryptographic verification result.

Repeat each workload and report the median, not only the fastest run. Test the actual route and hardware you plan to use.

For planning the payload portion, use our file transfer time calculator. For the distance side of the problem, see high-speed file transfer over long-distance networks.

Start a WarpSend transfer with a representative folder, then compare the wall-clock result, verification, and recovery behavior with your current workflow.

Frequently asked questions

Why is transferring many small files slower than one large file?

Every small file adds directory enumeration, metadata reads, open and close operations, destination creation, scheduling, and often network coordination. One large file amortizes that fixed work over a continuous payload, while thousands of small files repeat it.

Should I create an archive before transferring thousands of files?

An archive can turn many file operations into one stream and is practical for a one-off job. Count archive creation, temporary disk, extraction, restart granularity, and verification in the wall-clock result. Recurring workflows may benefit from automatic packing that preserves file boundaries.

Does adding more parallel transfers always help small files?

No. Bounded concurrency can overlap network and storage work, but too many workers contend for disk IOPS, CPU, memory, file handles, and network queues. Increase concurrency in controlled steps and stop when median completion time no longer improves.

How should I benchmark a small-file transfer?

Compare one large file, a medium batch, and the real small-file folder with similar total bytes. Record file count, storage, RTT, loss, CPU, encryption, verification, time to first payload, wall-clock completion, and repeat-run medians.