pcurl is a high-performance command-line tool written in C for accelerating large file downloads over HTTP/HTTPS. It improves download speed by splitting files into byte ranges (Range: bytes=...) and downloading them in parallel using a pool of POSIX threads (pthreads). Unlike some traditional multi-threaded downloaders that write separate temporary files and merge them afterwards, pcurl writes directly into a pre-allocated file using ftruncate() and pwrite(), avoiding extra disk I/O and data copying. This design reduces disk overhead and improves overall throughput for large transfers.
pcurl/
├── .github/
│ └── workflows/
│ └── build.yml # CI/CD automated build pipeline
├── src/
│ └── pcurl.c # Core C source code
├── tests/
│ ├── benchmark.sh # Benchmark test comparing performance against alternative tools
│ └── test_pcurl.sh # Automated integration tests
├── Makefile # Build and installation automation rules
├── README.md # Project documentation
└── LICENSE # License terms
- A C compiler (
gccorclang) - libcurl development libraries (
libcurl4-openssl-devon Debian/Ubuntu-based systems) - POSIX threads support (
pthread)
To build the binary locally, clone the repository and run:
makeTo install the executable system-wide (defaults to /usr/local/bin):
sudo make installTo clean up build artifacts:
make cleanUsage example: Download Apache Hadoop using twice as many chunks as there are threads available on the system.
CHUNKS=$(( $(nproc) * 2 ))
pcurl "https://archive.apache.org/dist/hadoop/core/hadoop-2.9.2/hadoop-2.9.2.tar.gz" "hadoop-2.9.2.tar.gz" $CHUNKSpcurl has been compared against single-thread traditional curl and the ultra fast download utility aria2c. You can perform the same comparison on your machine by running the script ./tests/benchmark.sh. Make sure to have both curl and aria2c installed before running the benchmark. This benchmark was run in a machine with the following hardware and software specifications:
| Feature | Description |
|---|---|
| CPU Model | Intel Core i9-12900K |
| Cores/Threads | 16/24 (8 P-cores + 8 E-cores) |
| Memory | 32 GiB (DDR5 5200 MHz) |
| Storage | 1 TB (SSD NVMe) |
| Network | 1 Gbps |
| OS/Kernel | Ubuntu 24.04.4 / 6.8.0 |
The following results were obtained:
| Configuration | Time (s) |
|---|---|
| curl (single thread) | 1055.947s |
| aria2c (1 connections, falloc) | 74.757s |
| aria2c (1 connections, prealloc) | 70.844s |
| aria2c (1 connections, trunc) | 65.312s |
| aria2c (2 connections, falloc) | 51.320s |
| aria2c (2 connections, prealloc) | 45.143s |
| aria2c (2 connections, trunc) | 40.680s |
| aria2c (4 connections, falloc) | 31.736s |
| aria2c (4 connections, prealloc) | 30.378s |
| aria2c (4 connections, trunc) | 34.624s |
| aria2c (8 connections, falloc) | 23.153s |
| aria2c (8 connections, prealloc) | 21.314s |
| aria2c (8 connections, trunc) | 23.532s |
| aria2c (16 connections, falloc) | 23.393s |
| aria2c (16 connections, prealloc) | 19.959s |
| aria2c (16 connections, trunc) | 21.291s |
| pcurl (16 threads) | 15.708s |
| pcurl (24 threads) | 16.159s |
| pcurl (48 threads) | 12.818s |
| pcurl (96 threads) | 12.821s |
| pcurl (192 threads) | 13.709s |
| pcurl (384 threads) | 8.886s |
It can be seen that pcurl outperforms aria2c performance, even when the same number of threads/connections is used (x1.27 speedup over the best aria2c configuration). Moreover, the best pcurl configuration (384 threads) achieves a speedup of x2.25 compared to the best aria2c configuration.