Encrypt uploads in the browser and send them in chunks

A 6 GB upload kept a customer waiting long after its progress bar
reached 100%. The server wrote every upload three times: PHP's
temporary file, Livewire's copy of it ("Processing files...") and the
encrypted file ("Create Share Link"), each a full rewrite of a slow
disk. The unencrypted copy also stayed behind in livewire-tmp.

Now the uploader's browser encrypts each file in 16 MB chunks with
WebCrypto and PUTs them one at a time; the server checks each chunk in
memory and writes it once, already encrypted. Creating the share only
wraps its key and saves the options. A 200 MB upload through the
Docker image took 2.8 s, and its download matched byte for byte.

- SEALCHK2: a 19-byte header (chunk size, 7-byte nonce prefix), then
  ciphertext and tag per chunk. Each nonce holds the chunk index and a
  last-chunk flag (the STREAM construction), so cut or reordered files
  fail to decrypt. SEALCHK1 and the single-block format still read.
- Envelope encryption: one random key per share. With a password it is
  wrapped with Argon2id (sodium, libsodium's interactive limits) in
  shares.wrapped_key, which names its parameters. Password shares from
  before keep their PBKDF2-derived key.
- The upload page registers each selection with FileUploader into a
  pending share of its own, lists the files with their progress, retries
  a failed chunk after 1-16 s, then offers Retry; Remove and Cancel
  abort. UploadChunkController only accepts chunks from the session that
  started the share: a repeat is acknowledged, a skip gets 409 with the
  count stored. Chunks go out as Blobs, which Chromium sends about eight
  times faster than ArrayBuffers.
- Uploads need a secure context: over plain HTTP the page says HTTPS is
  needed and takes no files. The Docker image gains AUTO_HTTPS, which
  serves Let's Encrypt on 443 for SERVER_NAME and redirects 80; without
  it the container stays on HTTP 80 behind a proxy. docker/Caddyfile was
  never loaded and is gone; docker/healthcheck.sh covers both modes.
- "Download all" streams the ZIP with maennchen/zipstream-php (STORE,
  ZIP64) instead of decrypting whole files into memory and writing the
  archive unencrypted to /tmp.
- Pending shares count towards the quota, stay out of the admin
  dashboard and 404 everywhere else. shares:cleanup deletes uploads idle
  for 4 hours and Livewire temporary files older than that.
- PHP's upload limits no longer cap the admin's max file size and
  default to 64M; LIVEWIRE_MAX_UPLOAD_TIME is gone and
  UPLOAD_CHUNK_SIZE_MB is new.
- Tests cover the format, key wrapping, registration limits, the chunk
  endpoint's answers, completing a share, the streamed ZIP, cleanup,
  and in Chromium a real chunked upload and the HTTPS warning; the
  selected-files overflow test runs again. README, website, CHANGELOG
  and .ai/rules follow.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Andreas Reinhold / reini
2026-09-16 20:49:17 +02:00
co-authored by Claude Opus 5
parent 504971ad7f
commit 40e35bab0e
53 changed files with 2280 additions and 946 deletions
+290 -184
View File
@@ -4,11 +4,30 @@ namespace App\Services;
use Generator;
use RuntimeException;
use Symfony\Component\HttpFoundation\HeaderUtils;
use Symfony\Component\HttpFoundation\StreamedResponse;
/**
* The encrypted file formats and the keys behind them.
*
* New files are `SEALCHK2`, written chunk by chunk as the uploader's browser sends them:
*
* [8 bytes: "SEALCHK2" magic]
* [4 bytes: chunk size S, uint32 big-endian]
* [7 bytes: random nonce prefix]
* Per chunk i: [ciphertext (S bytes, fewer on the last chunk)][16 bytes: GCM tag]
*
* Chunk i's nonce is the prefix, i as uint32 big-endian and a byte that is 1 on the last chunk
* and 0 on every other (the STREAM construction), so dropping, reordering or appending chunks
* fails authentication. The browser encrypts with the same layout (resources/js/share-uploader.js).
*
* `SEALCHK1` (a tag before each chunk, the index XORed into a 12-byte nonce, no last-chunk flag)
* and the single-block legacy format are still read for shares created before.
*/
class FileEncryptionService
{
public const HEADER_LENGTH = 19;
public const TAG_LENGTH = 16;
private const CIPHER = 'aes-256-gcm';
private const PBKDF2_ITERATIONS = 100000;
@@ -17,14 +36,17 @@ class FileEncryptionService
private const NONCE_LENGTH = 12;
private const TAG_LENGTH = 16;
private const NONCE_PREFIX_LENGTH = 7;
private const MAGIC_HEADER = 'SEALCHK1';
private const MAGIC = 'SEALCHK2';
private const DEFAULT_CHUNK_SIZE = 4 * 1024 * 1024; // 4 MB
private const LEGACY_CHUNKED_MAGIC = 'SEALCHK1';
private const WRAPPED_KEY_ALGORITHM = 'argon2id';
/**
* Derive an encryption key from a password and salt using PBKDF2-SHA256.
* Derive a key from a password and salt using PBKDF2-SHA256, as shares created before
* envelope encryption were keyed.
*/
public function deriveKey(string $password, string $salt): string
{
@@ -48,17 +70,147 @@ class FileEncryptionService
}
/**
* Encrypt a file using chunked AES-256-GCM.
* Wrap a share's data key with a key derived from its password (Argon2id).
*
* Output format:
* [8 bytes: "SEALCHK1" magic]
* [4 bytes: chunk size, uint32 big-endian]
* [12 bytes: base nonce]
* Per chunk:
* [16 bytes: GCM auth tag]
* [N bytes: ciphertext (up to chunk_size)]
* The result names its algorithm and parameters, so they can be raised later without breaking
* shares wrapped before: `argon2id$<opslimit>$<memlimit>$<salt>$<nonce>$<box>`, in hex.
*/
public function encryptFile(string $sourcePath, string $destPath, string $key): void
public function wrapKey(string $dataKeyHex, string $password): string
{
$salt = random_bytes(SODIUM_CRYPTO_PWHASH_SALTBYTES);
$opslimit = SODIUM_CRYPTO_PWHASH_OPSLIMIT_INTERACTIVE;
$memlimit = SODIUM_CRYPTO_PWHASH_MEMLIMIT_INTERACTIVE;
$wrappingKey = $this->deriveWrappingKey($password, $salt, $opslimit, $memlimit);
$nonce = random_bytes(SODIUM_CRYPTO_SECRETBOX_NONCEBYTES);
$box = sodium_crypto_secretbox(hex2bin($dataKeyHex), $nonce, $wrappingKey);
sodium_memzero($wrappingKey);
return implode('$', [self::WRAPPED_KEY_ALGORITHM, $opslimit, $memlimit, bin2hex($salt), bin2hex($nonce), bin2hex($box)]);
}
/**
* Unwrap a share's data key with its password; returns the key as hex.
*/
public function unwrapKey(string $wrappedKey, string $password): string
{
$parts = explode('$', $wrappedKey);
if (count($parts) !== 6 || $parts[0] !== self::WRAPPED_KEY_ALGORITHM) {
throw new RuntimeException('Unsupported wrapped key');
}
[, $opslimit, $memlimit, $salt, $nonce, $box] = $parts;
$wrappingKey = $this->deriveWrappingKey($password, hex2bin($salt), (int) $opslimit, (int) $memlimit);
$dataKey = sodium_crypto_secretbox_open(hex2bin($box), hex2bin($nonce), $wrappingKey);
sodium_memzero($wrappingKey);
if ($dataKey === false) {
throw new RuntimeException('Unwrapping failed - wrong password or corrupted key');
}
return bin2hex($dataKey);
}
/**
* The header a new encrypted file starts with, with a fresh random nonce prefix.
*/
public function createHeader(int $chunkSize): string
{
return self::MAGIC.pack('N', $chunkSize).random_bytes(self::NONCE_PREFIX_LENGTH);
}
/**
* Read a `SEALCHK2` header.
*
* @return array{chunkSize: int, noncePrefix: string}
*/
public function parseHeader(string $header): array
{
if (strlen($header) !== self::HEADER_LENGTH || ! str_starts_with($header, self::MAGIC)) {
throw new RuntimeException('Invalid encrypted file header');
}
return [
'chunkSize' => unpack('N', substr($header, 8, 4))[1],
'noncePrefix' => substr($header, 12, self::NONCE_PREFIX_LENGTH),
];
}
/**
* How many chunks a file of this size is sent in; an empty file is one empty chunk.
*/
public function chunkCount(int $size, int $chunkSize): int
{
return max(1, intdiv($size + $chunkSize - 1, $chunkSize));
}
/**
* Where chunk `$index` starts in the encrypted file.
*/
public function chunkOffset(int $index, int $chunkSize): int
{
return self::HEADER_LENGTH + $index * ($chunkSize + self::TAG_LENGTH);
}
/**
* Encrypt one chunk: its ciphertext followed by its tag, as WebCrypto returns it.
*/
public function encryptChunk(string $plaintext, string $key, string $noncePrefix, int $index, bool $isLast): string
{
$tag = '';
$ciphertext = openssl_encrypt(
$plaintext,
self::CIPHER,
$this->normalizeToBinaryKey($key),
OPENSSL_RAW_DATA,
$this->chunkNonce($noncePrefix, $index, $isLast),
$tag,
'',
self::TAG_LENGTH,
);
if ($ciphertext === false) {
throw new RuntimeException('Encryption failed at chunk '.$index);
}
return $ciphertext.$tag;
}
/**
* Decrypt one chunk, which fails unless its index and last-chunk flag are the ones it was
* encrypted with.
*/
public function decryptChunk(string $chunk, string $key, string $noncePrefix, int $index, bool $isLast): string
{
if (strlen($chunk) < self::TAG_LENGTH) {
throw new RuntimeException('Invalid encrypted file: truncated chunk '.$index);
}
$plaintext = openssl_decrypt(
substr($chunk, 0, -self::TAG_LENGTH),
self::CIPHER,
$this->normalizeToBinaryKey($key),
OPENSSL_RAW_DATA,
$this->chunkNonce($noncePrefix, $index, $isLast),
substr($chunk, -self::TAG_LENGTH),
);
if ($plaintext === false) {
throw new RuntimeException('Decryption failed - wrong key or corrupted data');
}
return $plaintext;
}
/**
* Encrypt a file on the server in the `SEALCHK2` format.
*/
public function encryptFile(string $sourcePath, string $destPath, string $key, int $chunkSize): void
{
$source = fopen($sourcePath, 'rb');
@@ -75,45 +227,16 @@ class FileEncryptionService
}
try {
$binaryKey = $this->normalizeToBinaryKey($key);
$baseNonce = random_bytes(self::NONCE_LENGTH);
$chunkSize = self::DEFAULT_CHUNK_SIZE;
$header = $this->createHeader($chunkSize);
$noncePrefix = $this->parseHeader($header)['noncePrefix'];
$chunkCount = $this->chunkCount((int) filesize($sourcePath), $chunkSize);
// Write header
fwrite($dest, self::MAGIC_HEADER);
fwrite($dest, pack('N', $chunkSize));
fwrite($dest, $baseNonce);
fwrite($dest, $header);
$chunkIndex = 0;
for ($index = 0; $index < $chunkCount; $index++) {
$plaintext = (string) fread($source, $chunkSize);
while (! feof($source)) {
$plaintext = fread($source, $chunkSize);
if ($plaintext === false || $plaintext === '') {
break;
}
$nonce = $this->deriveChunkNonce($baseNonce, $chunkIndex);
$tag = '';
$ciphertext = openssl_encrypt(
$plaintext,
self::CIPHER,
$binaryKey,
OPENSSL_RAW_DATA,
$nonce,
$tag,
'',
self::TAG_LENGTH,
);
if ($ciphertext === false) {
throw new RuntimeException('Encryption failed at chunk '.$chunkIndex);
}
fwrite($dest, $tag);
fwrite($dest, $ciphertext);
$chunkIndex++;
fwrite($dest, $this->encryptChunk($plaintext, $key, $noncePrefix, $index, $index === $chunkCount - 1));
}
} catch (RuntimeException $e) {
fclose($source);
@@ -127,74 +250,33 @@ class FileEncryptionService
fclose($dest);
}
/**
* Decrypt a file and return the plaintext content.
*/
public function decryptFile(string $encryptedPath, string $key): string
{
if ($this->isChunkedFormat($encryptedPath)) {
$parts = [];
foreach ($this->decryptChunks($encryptedPath, $key) as $chunk) {
$parts[] = $chunk;
}
return implode('', $parts);
}
return $this->decryptLegacy($encryptedPath, $key);
}
/**
* Decrypt a file and stream the response.
*/
public function decryptFileStream(string $encryptedPath, string $key, string $filename, string $mimeType, ?int $fileSize = null): StreamedResponse
{
$headers = [
'Content-Type' => $mimeType ?: 'application/octet-stream',
'Content-Disposition' => HeaderUtils::makeDisposition('attachment', $filename, 'download'),
];
if ($fileSize !== null) {
$headers['Content-Length'] = $fileSize;
}
if ($this->isChunkedFormat($encryptedPath)) {
return new StreamedResponse(function () use ($encryptedPath, $key): void {
foreach ($this->decryptChunks($encryptedPath, $key) as $chunk) {
echo $chunk;
flush();
}
}, 200, $headers);
}
$content = $this->decryptLegacy($encryptedPath, $key);
if (! isset($headers['Content-Length'])) {
$headers['Content-Length'] = strlen($content);
}
return new StreamedResponse(function () use ($content): void {
echo $content;
}, 200, $headers);
}
/**
* Stream decrypted file content directly to output (echo).
* Use this when you need to add post-streaming logic inside a StreamedResponse callback.
*/
public function streamDecryptedFile(string $encryptedPath, string $key): void
{
if ($this->isChunkedFormat($encryptedPath)) {
foreach ($this->decryptChunks($encryptedPath, $key) as $chunk) {
echo $chunk;
flush();
}
return;
foreach ($this->decryptedChunks($encryptedPath, $key) as $chunk) {
echo $chunk;
flush();
}
}
echo $this->decryptLegacy($encryptedPath, $key);
/**
* The decrypted content of a file in any of the three formats, chunk by chunk.
*
* @return Generator<int, string>
*/
public function decryptedChunks(string $encryptedPath, string $key): Generator
{
$magic = (string) file_get_contents($encryptedPath, false, null, 0, 8);
if ($magic === self::MAGIC) {
yield from $this->decryptChunks($encryptedPath, $key);
} elseif ($magic === self::LEGACY_CHUNKED_MAGIC) {
yield from $this->decryptLegacyChunks($encryptedPath, $key);
} else {
yield $this->decryptLegacy($encryptedPath, $key);
}
}
/**
@@ -205,73 +287,29 @@ class FileEncryptionService
return strlen($key) === 64 ? hex2bin($key) : $key;
}
/**
* Derive a unique nonce for a chunk by XORing the chunk index into the last 4 bytes.
*/
private function deriveChunkNonce(string $baseNonce, int $chunkIndex): string
private function deriveWrappingKey(string $password, string $salt, int $opslimit, int $memlimit): string
{
$nonce = $baseNonce;
$indexBytes = pack('N', $chunkIndex);
for ($i = 0; $i < 4; $i++) {
$nonce[self::NONCE_LENGTH - 4 + $i] = $nonce[self::NONCE_LENGTH - 4 + $i] ^ $indexBytes[$i];
}
return $nonce;
}
/**
* Check if a file uses the chunked encryption format.
*/
private function isChunkedFormat(string $path): bool
{
$handle = fopen($path, 'rb');
if ($handle === false) {
return false;
}
$magic = fread($handle, 8);
fclose($handle);
return $magic === self::MAGIC_HEADER;
}
/**
* Decrypt a legacy single-block encrypted file.
* Format: [12-byte nonce][16-byte auth tag][ciphertext]
*/
private function decryptLegacy(string $encryptedPath, string $key): string
{
$data = file_get_contents($encryptedPath);
if ($data === false) {
throw new RuntimeException("Cannot read encrypted file: {$encryptedPath}");
}
$binaryKey = $this->normalizeToBinaryKey($key);
$nonce = substr($data, 0, self::NONCE_LENGTH);
$tag = substr($data, self::NONCE_LENGTH, self::TAG_LENGTH);
$ciphertext = substr($data, self::NONCE_LENGTH + self::TAG_LENGTH);
$plaintext = openssl_decrypt(
$ciphertext,
self::CIPHER,
$binaryKey,
OPENSSL_RAW_DATA,
$nonce,
$tag,
return sodium_crypto_pwhash(
SODIUM_CRYPTO_SECRETBOX_KEYBYTES,
$password,
$salt,
$opslimit,
$memlimit,
SODIUM_CRYPTO_PWHASH_ALG_ARGON2ID13,
);
if ($plaintext === false) {
throw new RuntimeException('Decryption failed - wrong key or corrupted data');
}
return $plaintext;
}
/**
* Generator that yields decrypted plaintext chunks from a chunked encrypted file.
* A `SEALCHK2` chunk's nonce: the file's prefix, the chunk index and the last-chunk flag.
*/
private function chunkNonce(string $noncePrefix, int $index, bool $isLast): string
{
return $noncePrefix.pack('N', $index).($isLast ? "\x01" : "\x00");
}
/**
* Decrypt a `SEALCHK2` file; the chunk count comes from the file's length, so a file cut short
* at a chunk boundary fails on its new last chunk.
*
* @return Generator<int, string>
*/
@@ -284,17 +322,44 @@ class FileEncryptionService
}
try {
// Read header
$magic = fread($handle, 8);
['chunkSize' => $chunkSize, 'noncePrefix' => $noncePrefix] = $this->parseHeader((string) fread($handle, self::HEADER_LENGTH));
if ($magic !== self::MAGIC_HEADER) {
throw new RuntimeException('Invalid chunked file format');
$storedChunkSize = $chunkSize + self::TAG_LENGTH;
$payloadLength = (int) filesize($encryptedPath) - self::HEADER_LENGTH;
$chunkCount = intdiv($payloadLength + $storedChunkSize - 1, $storedChunkSize);
if ($chunkCount === 0) {
throw new RuntimeException('Invalid encrypted file: no chunks');
}
$chunkSizeData = fread($handle, 4);
$chunkSize = unpack('N', $chunkSizeData)[1];
for ($index = 0; $index < $chunkCount; $index++) {
$chunk = (string) fread($handle, $storedChunkSize);
$baseNonce = fread($handle, self::NONCE_LENGTH);
yield $this->decryptChunk($chunk, $key, $noncePrefix, $index, $index === $chunkCount - 1);
}
} finally {
fclose($handle);
}
}
/**
* Decrypt a `SEALCHK1` file.
*
* @return Generator<int, string>
*/
private function decryptLegacyChunks(string $encryptedPath, string $key): Generator
{
$handle = fopen($encryptedPath, 'rb');
if ($handle === false) {
throw new RuntimeException("Cannot read encrypted file: {$encryptedPath}");
}
try {
fread($handle, 8);
$chunkSize = unpack('N', (string) fread($handle, 4))[1];
$baseNonce = (string) fread($handle, self::NONCE_LENGTH);
if (strlen($baseNonce) !== self::NONCE_LENGTH) {
throw new RuntimeException('Invalid chunked file: truncated header');
@@ -320,14 +385,12 @@ class FileEncryptionService
throw new RuntimeException('Invalid chunked file: missing ciphertext at chunk '.$chunkIndex);
}
$nonce = $this->deriveChunkNonce($baseNonce, $chunkIndex);
$plaintext = openssl_decrypt(
$ciphertext,
self::CIPHER,
$binaryKey,
OPENSSL_RAW_DATA,
$nonce,
$this->legacyChunkNonce($baseNonce, $chunkIndex),
$tag,
);
@@ -342,4 +405,47 @@ class FileEncryptionService
fclose($handle);
}
}
/**
* A `SEALCHK1` chunk's nonce: the chunk index XORed into the last 4 bytes of the base nonce.
*/
private function legacyChunkNonce(string $baseNonce, int $chunkIndex): string
{
$nonce = $baseNonce;
$indexBytes = pack('N', $chunkIndex);
for ($i = 0; $i < 4; $i++) {
$nonce[self::NONCE_LENGTH - 4 + $i] = $nonce[self::NONCE_LENGTH - 4 + $i] ^ $indexBytes[$i];
}
return $nonce;
}
/**
* Decrypt a legacy single-block encrypted file.
* Format: [12-byte nonce][16-byte auth tag][ciphertext]
*/
private function decryptLegacy(string $encryptedPath, string $key): string
{
$data = file_get_contents($encryptedPath);
if ($data === false) {
throw new RuntimeException("Cannot read encrypted file: {$encryptedPath}");
}
$plaintext = openssl_decrypt(
substr($data, self::NONCE_LENGTH + self::TAG_LENGTH),
self::CIPHER,
$this->normalizeToBinaryKey($key),
OPENSSL_RAW_DATA,
substr($data, 0, self::NONCE_LENGTH),
substr($data, self::NONCE_LENGTH, self::TAG_LENGTH),
);
if ($plaintext === false) {
throw new RuntimeException('Decryption failed - wrong key or corrupted data');
}
return $plaintext;
}
}