Small-file transfer experiment: data and method

Tool results: 65,536 small files

Selected existing-tool observations from run-20260922-153900, separate from the later Rust factor experiment. Payload: 65,536 regular files, 4,096 bytes each; 268,435,456 useful bytes. All speeds use decimal MB/s. The first three rounds preceded the original run interruption; rounds four and five were collected in the recorded continuation window. The SSH admission failure occurred in a different workload (64 KiB / rsync-8); none of these selected cells had that failure. Completed copies passed file-set, length and SHA-256 verification. All ten SFTP timeouts below are retained.

*Every SFTP attempt was stopped at 900 seconds. The completion-rate bound is 268.435456 / 900 MB/s, rounded upward; it is not a measured partial-transfer speed. Raw copied the entire 2,147,483,648-byte image and mounted it before stopping the timer. Its median image-byte rate is 13.64 MB/s; the useful-file rate is 1.71 MB/s. Snapshot means Btrfs full send/receive, including reconstruction. Incremental snapshots and changed-block replication were not benchmarked. These whole-tool measurements do not isolate a single factor or share a measurement window and implementation with the later Rust tests. The tool timer includes shell/process setup and completion; the Rust timer excludes identity preparation and final protocol draining. Sync and hash verification are separate in both.

Rust factor study: all observations

Run: run-20260923-155000. All 80 observations passed exact file-set, length and SHA-256 verification. Every payload contains 268,435,456 useful bytes. Speed is decimal MB/s. Five values are retained in round order. files means pipelined file streams (up to 4,096 outstanding), not a per-file stop-and-wait baseline. Both layouts stream across file boundaries. File-ready timing excludes fixture preparation, shutdown draining, sync and hash; all are outside the primary rate.

Matched changes within each repetition

Ratios compare corresponding randomized rounds, not simultaneous transfers. Public-route variability remains.

Fixed test method

The following method was recorded before the corrected formal run. Download the standalone Rust source and dependency lockfile.

This follow-up answers which changes help on the DE → US route: bundling, congestion control, transport implementation, and then independent parallel connections. It does not assume that BBR, QUIC, or bundling must win. It does not reuse the older Go benchmark as a causal baseline.

Build: Rust 1.95.0 on Linux, cargo build --release --locked in this directory. Use the same resulting executable on both hosts. No WarpSend workspace patches are present in this standalone Cargo workspace.

Matrix fixed before collecting formal results

• Core: 65,536 files × 4,096 bytes; files/bundle × TCP/QUIC × CUBIC/BBR, one connection: eight conditions.
• Large-file reference: one file of 268,435,456 bytes; TCP/QUIC × CUBIC/BBR, one connection: four conditions.
• Parallel follow-up: bundle + BBR, TCP/QUIC, eight independent connections, for both the small-file dataset and eight equal large files with the same total bytes: four conditions. The eight-large-file reference prevents attributing all parallel gains to small-file handling.
• Five repetitions of all 16 conditions = 80 planned observations. Fixed seed randomizes order within every repetition. A separate 10,000-small-file pilot checks both correctness and stream recycling beyond the 8,192-stream credit limit.
• Existing snapshots, raw-image copies, SFTP, rsync and Go results remain historical tool observations. They are not mixed into this factor matrix.

What the switches actually change

Both paths use exactly the same Rust file-record encoder/decoder, reader/writer threads, 64 KiB read blocks, bounded 16-message I/O queues, and 256 KiB stream buffers. Each record is path length (u16), content length (u64), path, then exact file bytes. No compression, deduplication, precooked archive, product code or privileged product transport patches are included.

• files: one application stream per file, a reused authenticated connection, up to 4,096 files in flight. Stream halves close and recycle concurrently; no stop-and-wait application-level per-file acknowledgment.
• bundle: identical records in one continuous stream per connection; the buffer spans file boundaries. The receiver creates files during transfer. This is a deliberately minimal streaming bundle, not the GNU tar format.
• tcp: Linux TCP, libp2p TLS 1.3 identity authentication, yamux streams. TCP_CONGESTION is set and read back on each accepted/connected benchmark socket. Loading tcp_bbr makes it available but does not change the host’s default controller.
• quic: upstream Quinn 0.11.9 / quinn-proto 0.11.13 with libp2p TLS identities; native QUIC streams. The controller is selected explicitly in TransportConfig.
• This is a minimal data channel using libp2p identity/TLS components, not a full libp2p Swarm with discovery or protocol negotiation. TCP and QUIC differ in their kernel/userspace implementation and multiplexing as well as wire protocol.
• Linux BBR and Quinn BBR are distinct implementations; results concern these pinned implementations, not every algorithm called BBR. The complete dependency graph is recorded in Cargo.lock. The exact same release binary runs at both endpoints.
• Quinn send, connection receive, and stream receive limits are each 16,777,216 bytes. Both multiplexers allow 8,192 simultaneous streams. Yamux uses its upstream auto-growing stream receive windows. Its 8,192 × 262,144-byte reserved stream credit is supplemented with 16,515,072 bytes of growth headroom, allowing a bulk stream to reach the same 16,777,216-byte window instead of capping it at its initial credit; Linux TCP retains its normal buffer autotuning. These are not claimed to be identical flow-control implementations. Large-file controls reveal practical stack ceilings.

Fixture and timing

Both hosts use the existing 4-vCPU Linux VMs. All source datasets are copied from the original verified Btrfs loop images and mounted read-only. Source bytes are preread before each cell. Every receiver cell gets a newly formatted 2.15 GB Btrfs loop image and an empty destination. Source manifests are fixed before transfer.

The binary’s ready_s includes connection setup, the transfer request, on-demand file reads, stream creation, transfer, and file creation/writes/close. Connection-level completion acknowledgments are required for success but separately timed. Identity/certificate preparation, server setup, source preread and fixture creation are outside it. Shutdown draining is also outside. Receiver process duration is recorded separately. Sync and full file-set/length/ SHA-256 verification are measured afterward. An unverified copy cannot count as a successful speed observation. Scope is regular-file paths and bytes, not ACLs, xattrs, owners, timestamps, crash durability, incremental backup or resumes.

Speed is useful file bytes / ready_s / 1,000,000, in decimal MB/s. All payloads have 268,435,456 useful bytes (268.4 MB), except the separately labeled pilot. Receiver files must be usable, not merely received into a memory sink.

Integrity and interruptions

One persistent SSH connection is the control channel; SSH carries no timed data. Only the US peer can access the temporary DE data port 32943. Listener/firewall and mount cleanup belong to this run. Existing endpoint daemons remain untouched.

Both ends sample host CPU, I/O, UDP errors, TCP statistics and benchmark sockets every two seconds. ICMP probes run throughout. No qlog or packet-level logging runs during formal measurements. Natural network variation remains part of a public-route study; shuffled repetitions do not guarantee invariant conditions.

Retain every result, including slow results and 900-second censored timeouts. Stop on a transfer/correctness error. Investigate independent logs, preserve the original failure, and retry only its affected condition after correcting a concrete failure. Do not restart the complete matrix or discard results because of rankings.

Planned interpretation

Report matched packing and controller ratios within each transport, and transport ratios under matched application settings. Show every repetition and medians; do not add overlapping improvements as though factors were independent. The BBR eight-connection extension answers whether parallelism still helps that configuration, not whether parallelism is unimportant for every controller. Compare packed small files with the corresponding large-file reference. A high fraction of that reference supports “special filesystem replication was not needed to approach this stack’s large-file rate”; it does not prove block/snapshot techniques are generally irrelevant. DE → US results do not establish performance on all routes.

Environment and limitations

Both existing VMs have four vCPUs and approximately 8 GB RAM, Linux 6.8.0-138-generic. Sources are preread read-only Btrfs loop volumes; each destination is a fresh empty Btrfs loop volume. Existing endpoint services remained running. These are shared VMs, not dedicated hardware. The same release executable and fixed dependency graph ran on both endpoints: Linux TCP/TLS/yamux versus upstream Quinn 0.11.9 / quinn-proto 0.11.13, using libp2p TLS identities. This is not a full libp2p Swarm or WarpSend build. Linux and Quinn congestion controllers are separate implementations.

The TCP controller is selected and read back per socket; the system default remained CUBIC. Loading the tcp_bbr module made BBR available. No global sysctl or qdisc was changed. TCP retained kernel buffer autotuning. Quinn’s connection, stream and send flow-control limits were each 16,777,216 bytes. Yamux’s auto-growing stream credit had enough reserved growth headroom for a bulk stream to reach that size.

Transport flow-control windows are not UDP kernel socket buffers. Both hosts reported net.core.rmem_max = net.core.wmem_max = 212992. Recorded DE benchmark UDP sockets showed rb212992,tb212992. The receiver’s unconnected UDP socket used an ephemeral port, so the monitoring filter for port 32943 did not capture its socket memory counters. We do not claim an observed receiver socket-buffer size.

The recorded network and host audit found:

• 2,160 ICMP replies, no missing sequence between the first and last reply; median 141 ms, 95th percentile 143 ms, maximum 151 ms. ICMP success does not prove that TCP/QUIC traffic had no loss or that the public route was invariant.
• US host UDP receive-buffer errors increased by 94,351; DE host UDP send-buffer errors increased by 6,633. These are host-wide counters, not per-socket attribution. They largely coincide with BBR QUIC transfers. They do not establish why CUBIC was slow or prove a protocol-wide limitation.
• US aggregate CPU steal averaged 0.34%, with a two-second sample peak of 25.61%. The first and fourth eight-large-file QUIC/BBR transfers overlap bracketing intervals averaging 6.46% and 7.73% steal. Their speeds, 69.81 and 66.57 MB/s, are retained. A correlation with slower samples is not an isolated causal diagnosis.

The controller comparisons describe these fixed stacks and defaults on this route. They do not establish intrinsic TCP/QUIC superiority, universally optimal settings, or that larger socket buffers would or would not help. No post-result tuning or removal of slow but successful observations was performed. Natural public-route and shared-VM variability remains visible in every figure.

The corrected run contains 80 verified copies, measured from 2026-09-23 15:47:42 to 16:23:23 UTC. There were no transfer errors, timeouts, interrupted observations or selective replacements in this run. Setup validation and the separate correctness pilot are excluded from the performance figures.

The earlier whole-tool study is a separate batch. For its small-file observations, rounds one through three preceded a control-channel interruption; rounds four and five were collected in the recorded continuation window. The failed transfer was in another workload. The article retains all ten SFTP timeouts and every completed small-file observation.

The downloads below are the 80 original Rust result records and the 40 selected small-file tool rows. The tool CSV preserves recorded timing and status fields, with its speed column converted to decimal MB/s. The source archive contains the standalone Rust sender, pinned Cargo dependencies, and the frozen method document; it contains no WarpSend product code or identity keys.

Downloads

• 80 original Rust observations (JSONL)
• 40 small-file tool observations (CSV, MB/s)
• Standalone Rust source, Cargo.lock and method (ZIP)
• SHA-256 checksums (JSON)