Andreas Reinhold / reiniandClaude Opus 5
40e35bab0e
A 6 GB upload kept a customer waiting long after its progress bar
reached 100%. The server wrote every upload three times: PHP's
temporary file, Livewire's copy of it ("Processing files...") and the
encrypted file ("Create Share Link"), each a full rewrite of a slow
disk. The unencrypted copy also stayed behind in livewire-tmp.
Now the uploader's browser encrypts each file in 16 MB chunks with
WebCrypto and PUTs them one at a time; the server checks each chunk in
memory and writes it once, already encrypted. Creating the share only
wraps its key and saves the options. A 200 MB upload through the
Docker image took 2.8 s, and its download matched byte for byte.
- SEALCHK2: a 19-byte header (chunk size, 7-byte nonce prefix), then
ciphertext and tag per chunk. Each nonce holds the chunk index and a
last-chunk flag (the STREAM construction), so cut or reordered files
fail to decrypt. SEALCHK1 and the single-block format still read.
- Envelope encryption: one random key per share. With a password it is
wrapped with Argon2id (sodium, libsodium's interactive limits) in
shares.wrapped_key, which names its parameters. Password shares from
before keep their PBKDF2-derived key.
- The upload page registers each selection with FileUploader into a
pending share of its own, lists the files with their progress, retries
a failed chunk after 1-16 s, then offers Retry; Remove and Cancel
abort. UploadChunkController only accepts chunks from the session that
started the share: a repeat is acknowledged, a skip gets 409 with the
count stored. Chunks go out as Blobs, which Chromium sends about eight
times faster than ArrayBuffers.
- Uploads need a secure context: over plain HTTP the page says HTTPS is
needed and takes no files. The Docker image gains AUTO_HTTPS, which
serves Let's Encrypt on 443 for SERVER_NAME and redirects 80; without
it the container stays on HTTP 80 behind a proxy. docker/Caddyfile was
never loaded and is gone; docker/healthcheck.sh covers both modes.
- "Download all" streams the ZIP with maennchen/zipstream-php (STORE,
ZIP64) instead of decrypting whole files into memory and writing the
archive unencrypted to /tmp.
- Pending shares count towards the quota, stay out of the admin
dashboard and 404 everywhere else. shares:cleanup deletes uploads idle
for 4 hours and Livewire temporary files older than that.
- PHP's upload limits no longer cap the admin's max file size and
default to 64M; LIVEWIRE_MAX_UPLOAD_TIME is gone and
UPLOAD_CHUNK_SIZE_MB is new.
- Tests cover the format, key wrapping, registration limits, the chunk
endpoint's answers, completing a share, the streamed ZIP, cleanup,
and in Chromium a real chunked upload and the HTTPS warning; the
selected-files overflow test runs again. README, website, CHANGELOG
and .ai/rules follow.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
111 lines
4.2 KiB
PHP
111 lines
4.2 KiB
PHP
<?php
|
|
|
|
use App\Models\Setting;
|
|
use App\Models\ShareFile;
|
|
use App\Services\ShareService;
|
|
use Illuminate\Support\Facades\Storage;
|
|
|
|
/**
|
|
* PUT a chunk's bytes as the uploader's page does, with the given pending shares in the session.
|
|
*
|
|
* @param array<int, string> $pendingShares
|
|
*/
|
|
function putChunk(mixed $test, ShareFile $file, int $index, string $chunk, array $pendingShares): mixed
|
|
{
|
|
return $test->withSession(['pending_shares' => $pendingShares])->call(
|
|
'PUT',
|
|
route('upload.chunk', ['shareFile' => $file, 'index' => $index]),
|
|
server: ['CONTENT_TYPE' => 'application/octet-stream', 'HTTP_ACCEPT' => 'application/json'],
|
|
content: $chunk,
|
|
);
|
|
}
|
|
|
|
test('a chunk for a pending share this session started is stored', function () {
|
|
Storage::fake('shares');
|
|
config(['uploads.chunk_size' => 4]);
|
|
$file = app(ShareService::class)->registerFile(null, 'notes.txt', 6, null);
|
|
|
|
$response = putChunk($this, $file, 0, encryptedChunk($file, 'abcd', 0, false), [$file->share->token]);
|
|
|
|
$response->assertOk()->assertExactJson(['uploaded_chunks' => 1]);
|
|
expect($file->refresh()->uploaded_chunks)->toBe(1);
|
|
});
|
|
|
|
test('a chunk for a pending share another session started returns 404', function () {
|
|
Storage::fake('shares');
|
|
$file = app(ShareService::class)->registerFile(null, 'notes.txt', 4, null);
|
|
|
|
$response = putChunk($this, $file, 0, encryptedChunk($file, 'abcd', 0, true), ['someOtherToken12']);
|
|
|
|
$response->assertNotFound();
|
|
expect($file->refresh()->uploaded_chunks)->toBe(0);
|
|
});
|
|
|
|
test('a chunk for a share that is already complete returns 404', function () {
|
|
Storage::fake('shares');
|
|
$file = app(ShareService::class)->registerFile(null, 'notes.txt', 4, null);
|
|
$file->share->update(['completed_at' => now()]);
|
|
|
|
$response = putChunk($this, $file, 0, encryptedChunk($file, 'abcd', 0, true), [$file->share->token]);
|
|
|
|
$response->assertNotFound();
|
|
});
|
|
|
|
test('a chunk for a removed file returns 404', function () {
|
|
Storage::fake('shares');
|
|
$service = app(ShareService::class);
|
|
$file = $service->registerFile(null, 'notes.txt', 4, null);
|
|
$chunk = encryptedChunk($file, 'abcd', 0, true);
|
|
$token = $file->share->token;
|
|
$service->removeFile($file);
|
|
|
|
$response = putChunk($this, $file, 0, $chunk, [$token]);
|
|
|
|
$response->assertNotFound();
|
|
});
|
|
|
|
test('a chunk that skips ahead returns 409 with the number of chunks stored', function () {
|
|
Storage::fake('shares');
|
|
config(['uploads.chunk_size' => 4]);
|
|
$file = app(ShareService::class)->registerFile(null, 'notes.txt', 6, null);
|
|
|
|
$response = putChunk($this, $file, 1, encryptedChunk($file, 'ef', 1, true), [$file->share->token]);
|
|
|
|
$response->assertConflict()->assertExactJson(['uploaded_chunks' => 0]);
|
|
expect($file->refresh()->uploaded_chunks)->toBe(0);
|
|
});
|
|
|
|
test('a chunk sent again after it was stored is acknowledged without storing it twice', function () {
|
|
Storage::fake('shares');
|
|
config(['uploads.chunk_size' => 4]);
|
|
$file = app(ShareService::class)->registerFile(null, 'notes.txt', 6, null);
|
|
$chunk = encryptedChunk($file, 'abcd', 0, false);
|
|
putChunk($this, $file, 0, $chunk, [$file->share->token]);
|
|
|
|
$response = putChunk($this, $file->refresh(), 0, $chunk, [$file->share->token]);
|
|
|
|
$response->assertOk()->assertExactJson(['uploaded_chunks' => 1]);
|
|
expect($file->refresh()->uploaded_chunks)->toBe(1);
|
|
});
|
|
|
|
test('an invalid chunk returns 422 and is not stored', function () {
|
|
Storage::fake('shares');
|
|
$file = app(ShareService::class)->registerFile(null, 'notes.txt', 4, null);
|
|
|
|
$response = putChunk($this, $file, 0, str_repeat("\0", 20), [$file->share->token]);
|
|
|
|
$response->assertUnprocessable();
|
|
expect($file->refresh()->uploaded_chunks)->toBe(0);
|
|
});
|
|
|
|
test('a chunk is refused until the system password was entered', function () {
|
|
Storage::fake('shares');
|
|
Setting::set('system_password', bcrypt('system-secret'));
|
|
$file = app(ShareService::class)->registerFile(null, 'notes.txt', 4, null);
|
|
|
|
$response = putChunk($this, $file, 0, encryptedChunk($file, 'abcd', 0, true), [$file->share->token]);
|
|
|
|
$response->assertRedirect(route('system-password'));
|
|
expect($file->refresh()->uploaded_chunks)->toBe(0);
|
|
});
|