Encrypt uploads in the browser and send them in chunks

A 6 GB upload kept a customer waiting long after its progress bar
reached 100%. The server wrote every upload three times: PHP's
temporary file, Livewire's copy of it ("Processing files...") and the
encrypted file ("Create Share Link"), each a full rewrite of a slow
disk. The unencrypted copy also stayed behind in livewire-tmp.

Now the uploader's browser encrypts each file in 16 MB chunks with
WebCrypto and PUTs them one at a time; the server checks each chunk in
memory and writes it once, already encrypted. Creating the share only
wraps its key and saves the options. A 200 MB upload through the
Docker image took 2.8 s, and its download matched byte for byte.

- SEALCHK2: a 19-byte header (chunk size, 7-byte nonce prefix), then
  ciphertext and tag per chunk. Each nonce holds the chunk index and a
  last-chunk flag (the STREAM construction), so cut or reordered files
  fail to decrypt. SEALCHK1 and the single-block format still read.
- Envelope encryption: one random key per share. With a password it is
  wrapped with Argon2id (sodium, libsodium's interactive limits) in
  shares.wrapped_key, which names its parameters. Password shares from
  before keep their PBKDF2-derived key.
- The upload page registers each selection with FileUploader into a
  pending share of its own, lists the files with their progress, retries
  a failed chunk after 1-16 s, then offers Retry; Remove and Cancel
  abort. UploadChunkController only accepts chunks from the session that
  started the share: a repeat is acknowledged, a skip gets 409 with the
  count stored. Chunks go out as Blobs, which Chromium sends about eight
  times faster than ArrayBuffers.
- Uploads need a secure context: over plain HTTP the page says HTTPS is
  needed and takes no files. The Docker image gains AUTO_HTTPS, which
  serves Let's Encrypt on 443 for SERVER_NAME and redirects 80; without
  it the container stays on HTTP 80 behind a proxy. docker/Caddyfile was
  never loaded and is gone; docker/healthcheck.sh covers both modes.
- "Download all" streams the ZIP with maennchen/zipstream-php (STORE,
  ZIP64) instead of decrypting whole files into memory and writing the
  archive unencrypted to /tmp.
- Pending shares count towards the quota, stay out of the admin
  dashboard and 404 everywhere else. shares:cleanup deletes uploads idle
  for 4 hours and Livewire temporary files older than that.
- PHP's upload limits no longer cap the admin's max file size and
  default to 64M; LIVEWIRE_MAX_UPLOAD_TIME is gone and
  UPLOAD_CHUNK_SIZE_MB is new.
- Tests cover the format, key wrapping, registration limits, the chunk
  endpoint's answers, completing a share, the streamed ZIP, cleanup,
  and in Chromium a real chunked upload and the HTTPS warning; the
  selected-files overflow test runs again. README, website, CHANGELOG
  and .ai/rules follow.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Andreas Reinhold / reini
2026-09-16 20:49:17 +02:00
co-authored by Claude Opus 5
parent 504971ad7f
commit 40e35bab0e
53 changed files with 2280 additions and 946 deletions
+2 -8
View File
@@ -122,8 +122,8 @@
/*
* resources/views/livewire/file-uploader.blade.php: the drop zone's dashed outline and its
* primary tint while dragging. `data-dragging` is Alpine's, not the package's, since no
* component tracks a native drag over an arbitrary drop target; disabled while an upload runs
* blocks pointer events and dims to M3's disabled-content opacity, as a code dims elsewhere while
* component tracks a native drag over an arbitrary drop target; disabled where uploads cannot run
* (no secure context) blocks pointer events and dims to M3's disabled-content opacity, as a code dims elsewhere while
* busy (.settings-recovery-code--loading).
*/
.upload-drop-zone {
@@ -203,12 +203,6 @@
color: var(--md-sys-color-on-primary-container);
}
/* resources/views/livewire/file-uploader.blade.php: the "Processing files..." indicator at 1.x's smaller size, beside its label. */
.upload-processing-indicator {
inline-size: 2rem;
block-size: 2rem;
}
/* resources/views/livewire/file-uploader.blade.php: the selected-files list scrolls on its own past 1.x's cap instead of pushing the options and the submit button down the page. */
.upload-file-list {
max-block-size: 18rem;
+1
View File
@@ -1,3 +1,4 @@
// Livewire Material. Alpine is bundled and started by Livewire 4: never import it here as well.
import '../../vendor/nonameweb/livewire-material/resources/js/material.js'
import './share-created.js'
import './share-uploader.js'
+317
View File
@@ -0,0 +1,317 @@
/**
* `shareUploader`: the upload page's queue. Files are registered with the Livewire component in one
* batch per selection, then sent one at a time: each chunk is sliced from the file, encrypted here
* with AES-256-GCM and PUT on its own, so the server writes it once, already encrypted.
*
* The encrypted format is App\Services\FileEncryptionService's SEALCHK2: chunk i's nonce is the
* file's 7-byte prefix, i as a big-endian uint32 and a byte that is 1 on the last chunk; WebCrypto
* appends the 16-byte tag to the ciphertext, which is how the server stores it.
*
* A failed request is retried after 1, 2, 4, 8 and 16 seconds; after that the file waits for its
* Retry button, which picks up from the chunk the server last confirmed. The server answers 409
* with its own count when a chunk skips ahead, and acknowledges a chunk it already has.
*/
const RETRY_DELAYS = [1000, 2000, 4000, 8000, 16000]
document.addEventListener('alpine:init', () => {
window.Alpine.data('shareUploader', ({ csrfToken, messages }) => ({
secure: window.isSecureContext && Boolean(window.crypto?.subtle),
dragging: false,
busy: false,
/** Files waiting to be sent, in order: { id, file, target, nextIndex }. */
queue: [],
/** Every file this page registered, by id: { state: 'queued'|'uploading'|'uploaded'|'failed', sent, size }. */
uploads: {},
/** The files that failed, by id, kept for their Retry button. */
failed: {},
/** The request on its way, so Cancel and Remove can abort it. */
request: null,
get progress() {
const unfinished = Object.values(this.uploads).filter((upload) => upload.state !== 'failed')
const size = unfinished.reduce((total, upload) => total + upload.size, 0)
return size === 0 ? 0 : (unfinished.reduce((total, upload) => total + upload.sent, 0) / size) * 100
},
choose(event) {
this.add([...event.target.files].map((file) => ({ file, path: null })))
event.target.value = ''
},
handleDrop(event) {
this.dragging = false
if (! this.secure) {
return
}
const items = event.dataTransfer.items
const files = []
for (let i = 0; i < items.length; i++) {
const entry = items[i].webkitGetAsEntry?.()
if (entry) {
this.traverseEntry(entry, '', files)
} else if (items[i].kind === 'file') {
files.push({ file: items[i].getAsFile(), path: null })
}
}
// Directory entries are read asynchronously.
setTimeout(() => this.add(files), 500)
},
traverseEntry(entry, path, files) {
if (entry.isFile) {
entry.file((file) => files.push({ file, path: path ? `${path}/${file.name}` : null }))
} else if (entry.isDirectory) {
entry.createReader().readEntries((entries) => {
entries.forEach((child) => this.traverseEntry(child, path ? `${path}/${entry.name}` : entry.name, files))
})
}
},
async add(selection) {
if (! this.secure || selection.length === 0) {
return
}
const targets = await this.$wire.registerFiles(selection.map(({ file, path }) => ({ name: file.name, size: file.size, path })))
;(targets ?? []).forEach((target, position) => {
if (! target) {
return
}
this.uploads[target.id] = { state: 'queued', sent: 0, size: selection[position].file.size }
this.queue.push({ id: target.id, file: selection[position].file, target, nextIndex: 0 })
})
this.run()
},
async run() {
if (this.busy) {
return
}
this.busy = true
while (this.queue.length > 0) {
const item = this.queue[0]
const uploaded = await this.upload(item)
if (this.queue[0] === item) {
this.queue.shift()
}
if (uploaded) {
await this.$wire.$refresh()
}
}
this.busy = false
},
/**
* Send one file's remaining chunks; true once the server has them all.
*/
async upload(item) {
const { id, file, target } = item
const upload = this.uploads[id]
if (! upload) {
return false
}
upload.state = 'uploading'
try {
const key = await crypto.subtle.importKey('raw', bytesFromHex(target.key), 'AES-GCM', false, ['encrypt'])
const noncePrefix = bytesFromHex(target.noncePrefix)
while (item.nextIndex < target.chunkCount) {
const index = item.nextIndex
const start = index * target.chunkSize
const plaintext = await file.slice(start, start + target.chunkSize).arrayBuffer()
const ciphertext = await crypto.subtle.encrypt(
{ name: 'AES-GCM', iv: chunkNonce(noncePrefix, index, index === target.chunkCount - 1), tagLength: 128 },
key,
plaintext,
)
item.nextIndex = await this.send(upload, `${target.url}/${index}`, ciphertext, start, plaintext.byteLength)
upload.sent = Math.min(item.nextIndex * target.chunkSize, upload.size)
}
} catch (error) {
if (error?.name === 'AbortError') {
return false
}
upload.state = 'failed'
this.failed[id] = item
if (error?.status === 419) {
window.materialToast(messages.sessionExpired, { type: 'error' })
}
return false
}
upload.state = 'uploaded'
return true
},
/**
* PUT one encrypted chunk, retrying transient failures; resolves with the number of chunks
* the server holds for the file.
*/
async send(upload, url, body, offset, plaintextLength) {
for (let attempt = 0; ; attempt++) {
const response = await this.put(url, body, (loaded) => {
upload.sent = Math.min(offset + (loaded / body.byteLength) * plaintextLength, upload.size)
})
if ((response.status === 200 || response.status === 409) && Number.isInteger(response.uploadedChunks)) {
return response.uploadedChunks
}
if (response.status === 404 || response.status === 419 || attempt === RETRY_DELAYS.length) {
throw Object.assign(new Error('Upload failed'), { status: response.status })
}
await new Promise((resolve) => setTimeout(resolve, RETRY_DELAYS[attempt]))
}
},
put(url, body, onProgress) {
return new Promise((resolve, reject) => {
const xhr = new XMLHttpRequest()
xhr.open('PUT', url)
xhr.setRequestHeader('Content-Type', 'application/octet-stream')
xhr.setRequestHeader('Accept', 'application/json')
xhr.setRequestHeader('X-CSRF-TOKEN', csrfToken)
xhr.upload.onprogress = (event) => onProgress(event.loaded)
xhr.onload = () => {
this.request = null
resolve({ status: xhr.status, uploadedChunks: parseUploadedChunks(xhr.responseText) })
}
xhr.onerror = () => {
this.request = null
resolve({ status: 0 })
}
xhr.onabort = () => {
this.request = null
reject(new DOMException('Upload cancelled', 'AbortError'))
}
this.request = xhr
// Chromium sends a Blob body about eight times faster than the same ArrayBuffer.
xhr.send(new Blob([body]))
})
},
retry(id) {
const item = this.failed[id]
if (! item) {
return
}
delete this.failed[id]
this.uploads[id].state = 'queued'
this.queue.push(item)
this.run()
},
remove(id) {
this.forget([id])
this.$wire.removeFiles([id])
},
/**
* Stop everything still to send and take those files out of the share.
*/
cancel() {
const unfinished = Object.entries(this.uploads)
.filter(([, upload]) => upload.state !== 'uploaded')
.map(([id]) => Number(id))
this.forget(unfinished)
this.$wire.removeFiles(unfinished)
},
forget(ids) {
const current = this.queue[0]
this.queue = this.queue.filter((item) => ! ids.includes(item.id))
ids.forEach((id) => {
delete this.uploads[id]
delete this.failed[id]
})
if (current && ids.includes(current.id)) {
this.request?.abort()
}
},
statusOf(id, uploaded) {
const upload = this.uploads[id]
if (uploaded || upload?.state === 'uploaded') {
return messages.uploaded
}
if (upload?.state === 'uploading') {
return `${Math.round((upload.sent / Math.max(upload.size, 1)) * 100)}%`
}
return upload?.state === 'failed' ? messages.failed : messages.queued
},
warnBeforeLeaving(event) {
if (this.busy) {
event.preventDefault()
event.returnValue = ''
}
},
}))
})
function bytesFromHex(hex) {
return Uint8Array.from(hex.match(/.{2}/g), (pair) => parseInt(pair, 16))
}
/**
* Chunk `index`'s 12-byte nonce: the file's prefix, the index as a big-endian uint32 and the
* last-chunk flag.
*/
function chunkNonce(prefix, index, isLast) {
const nonce = new Uint8Array(12)
nonce.set(prefix, 0)
new DataView(nonce.buffer).setUint32(7, index)
nonce[11] = isLast ? 1 : 0
return nonce
}
function parseUploadedChunks(responseText) {
try {
return JSON.parse(responseText).uploaded_chunks
} catch {
return undefined
}
}
@@ -179,9 +179,7 @@
:label="__('Max file size (MB)')"
type="number"
min="1"
:max="$phpMaxUploadMb"
suffix="MB"
:hint="__('PHP limit: :max MB (upload_max_filesize / post_max_size)', ['max' => $phpMaxUploadMb])"
/>
<x-input full wire:model="maxFilesPerShare" :label="__('Max files per share')" type="number" min="1" />
@@ -12,85 +12,33 @@
</x-stack>
</x-stack>
@if ($isStorageFull)
{{-- Files this page already uploaded count towards the quota: they can still become a share. --}}
@if ($isStorageFull && $pendingFiles->isEmpty())
<x-alert color="warning" :title="__('Storage is full. Uploads are temporarily disabled.')" />
@else
<x-form
wire:submit="createShare"
x-data="{
uploading: false,
progress: 0,
dragging: false,
handleDrop(e) {
this.dragging = false;
const items = e.dataTransfer.items;
const files = [];
for (let i = 0; i < items.length; i++) {
const entry = items[i].webkitGetAsEntry?.();
if (entry) {
this.traverseEntry(entry, '', files);
} else if (items[i].kind === 'file') {
files.push({ file: items[i].getAsFile(), path: null });
}
}
setTimeout(() => {
if (! files.length) {
return;
}
const dt = new DataTransfer();
const paths = [];
files.forEach(f => {
dt.items.add(f.file);
paths.push(f.path);
});
$wire.relativePaths = [...($wire.relativePaths ?? []), ...paths];
this.uploading = true;
this.progress = 0;
$wire.uploadMultiple(
'files',
dt.files,
() => this.progress = 100,
() => this.resetUpload(),
(event) => this.progress = event.detail.progress,
() => this.resetUpload(),
);
}, 500);
},
resetUpload() {
this.uploading = false;
this.progress = 0;
},
traverseEntry(entry, path, files) {
if (entry.isFile) {
entry.file(file => {
files.push({ file, path: path ? path + '/' + file.name : null });
});
} else if (entry.isDirectory) {
const reader = entry.createReader();
reader.readEntries(entries => {
entries.forEach(e => this.traverseEntry(e, path ? path + '/' + entry.name : entry.name, files));
});
}
}
}"
x-init="$wire.$on('files-processed', () => resetUpload())"
x-on:livewire-upload-start="uploading = true; progress = 0"
x-on:livewire-upload-finish="progress = 100"
x-on:livewire-upload-cancel="resetUpload()"
x-on:livewire-upload-error="resetUpload()"
x-on:livewire-upload-progress="progress = $event.detail.progress"
x-data="shareUploader({
csrfToken: {{ \Illuminate\Support\Js::from(csrf_token()) }},
messages: {{ \Illuminate\Support\Js::from([
'queued' => __('Waiting'),
'uploaded' => __('Uploaded'),
'failed' => __('Upload failed'),
'sessionExpired' => __('Your session expired. Reload the page to upload again.'),
]) }},
})"
x-on:beforeunload.window="warnBeforeLeaving($event)"
>
{{-- WebCrypto, which encrypts the files in the browser, only exists on HTTPS (or localhost). --}}
<div x-show="! secure" x-cloak data-test="insecure-context">
<x-alert color="warning" :title="__('Uploads need a secure connection (HTTPS).')" :description="__('Ask the administrator to serve this site over HTTPS.')" />
</div>
{{-- Drop zone: the shape behind the icon turns into a burst while files are over it. --}}
<div
class="upload-drop-zone"
x-bind:data-dragging="dragging ? 'true' : 'false'"
x-bind:aria-disabled="uploading ? 'true' : 'false'"
x-bind:aria-disabled="secure ? 'false' : 'true'"
x-on:dragover.prevent="dragging = true"
x-on:dragleave.prevent="dragging = false"
x-on:drop.prevent="handleDrop($event)"
@@ -108,25 +56,21 @@
<p class="md-type-body-md md-ink-variant md-text-center">{{ __('or click to browse') }}</p>
</x-stack>
{{-- The button is the tab stop and opens the browser's own picker; the input only carries the upload. --}}
<x-button :label="__('Browse Files')" icon="folder_open" variant="outlined" x-on:click="$refs.picker.click()" x-bind:disabled="uploading" />
<input type="file" wire:model="files" multiple hidden x-ref="picker" x-bind:disabled="uploading" />
{{-- The button is the tab stop and opens the browser's own picker; the input only carries the selection. --}}
<x-button :label="__('Browse Files')" icon="folder_open" variant="outlined" x-on:click="$refs.picker.click()" x-bind:disabled="! secure" />
<input type="file" multiple hidden x-ref="picker" x-on:change="choose($event)" x-bind:disabled="! secure" data-test="file-input" />
</x-stack>
</div>
{{-- Upload progress --}}
<div x-show="uploading" x-cloak data-test="upload-progress">
<x-stack x-show="progress < 100" gap="space100">
{{-- Upload progress, over every file still to send --}}
<div x-show="busy" x-cloak data-test="upload-progress">
<x-stack gap="space100">
<x-row justify="between">
<span class="md-type-label-lg">{{ __('Uploading...') }} <span x-text="Math.round(progress)"></span>%</span>
<x-button :label="__('Cancel')" size="xs" x-on:click="$wire.cancelUpload('files')" />
<x-button :label="__('Cancel')" size="xs" x-on:click="cancel()" />
</x-row>
<x-progress bind="progress" wavy :label="__('Uploading')" />
</x-stack>
<x-row x-show="progress >= 100" gap="space100">
<x-loading class="upload-processing-indicator" :label="false" />
<span class="md-type-label-lg">{{ __('Processing files...') }}</span>
</x-row>
</div>
@error('files')
@@ -134,21 +78,28 @@
@enderror
{{-- Selected files --}}
@if (count($files))
@if ($pendingFiles->isNotEmpty())
<x-stack gap="space100">
<h2 class="md-type-title-lg">{{ __('Selected Files') }} ({{ count($files) }})</h2>
<h2 class="md-type-title-lg">{{ __('Selected Files') }} ({{ $pendingFiles->count() }})</h2>
<div class="upload-file-list">
<x-list segmented :label="__('Selected Files')">
@foreach ($files as $index => $file)
@foreach ($pendingFiles as $file)
<x-list-item
:title="$relativePaths[$index] ?? $file->getClientOriginalName()"
:title="$file->relative_path ?? $file->original_name"
icon="description"
wire:key="selected-file-{{ $index }}"
wire:key="selected-file-{{ $file->id }}"
data-test="selected-file"
>
<x-slot:description><span class="md-tabular">{{ Number::fileSize($file->getSize()) }}</span></x-slot:description>
<x-slot:description>
<span class="md-tabular">{{ Number::fileSize($file->file_size) }}</span>
· <span class="md-tabular" x-text="statusOf({{ $file->id }}, {{ $file->completed_at ? 'true' : 'false' }})" data-test="file-status"></span>
</x-slot:description>
<x-slot:end>
<x-button icon="close" :aria-label="__('Remove')" wire:click="removeFile({{ $index }})" />
<span x-show="uploads[{{ $file->id }}]?.state === 'failed'" x-cloak>
<x-button icon="refresh" :aria-label="__('Retry')" x-on:click="retry({{ $file->id }})" />
</span>
<x-button icon="close" :aria-label="__('Remove')" x-on:click="remove({{ $file->id }})" />
</x-slot:end>
</x-list-item>
@endforeach
@@ -217,7 +168,7 @@
size="md"
icon="link"
spinner="createShare"
x-bind:disabled="uploading || {{ count($files) === 0 ? 'true' : 'false' }}"
x-bind:disabled="busy || {{ $allFilesUploaded ? 'false' : 'true' }}"
data-test="create-share"
/>
</x-slot:actions>