Skip to content

feat: Add stall detection to recover from frozen uploads #775

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 11 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -208,6 +208,34 @@ Following example will trigger up to three retries, each after 1s, 3s and 5s res
retryDelays: [1000, 3000, 5000]
```

#### stallDetection

_Default value:_ `{ enabled: false, stallTimeout: 30000, checkInterval: 5000 }`

An object controlling the stall detection feature, which can automatically detect when an upload has stopped making progress and trigger a retry. This is useful for recovering from frozen uploads caused by network issues that don't trigger explicit errors.

The stall detection options are:
- `enabled`: Boolean indicating whether stall detection is active (default: `false`)
- `stallTimeout`: Time in milliseconds without progress before considering the upload stalled (default: `30000`)
- `checkInterval`: How often in milliseconds to check for stalls (default: `5000`)

**Note:** Stall detection only works with HTTP stacks that support progress events. Currently, this includes:
- `XHRHttpStack` (browser default) - Supported
- `NodeHttpStack` (Node.js default) - Supported
- `FetchHttpStack` - Not supported

When a stall is detected, the upload will be automatically retried according to your `retryDelays` configuration. If `retryDelays` is `null`, the stall will trigger an error instead.

Example configuration:

```js
stallDetection: {
enabled: true,
stallTimeout: 15000, // 15 seconds without progress
checkInterval: 2000 // Check every 2 seconds
}
```

#### storeFingerprintForResuming

_Default value:_ `true`
Expand Down Expand Up @@ -326,6 +354,7 @@ An object used as the HTTP stack for making network requests. This is an abstrac
interface HttpStack {
createRequest(method: string, url: string): HttpRequest;
getName(): string;
supportsProgressEvents(): boolean;
}

interface HttpRequest {
Expand Down Expand Up @@ -367,6 +396,14 @@ interface HttpResponse {

```

The `supportsProgressEvents()` method should return `true` if the HTTP stack implementation supports progress events during upload, or `false` otherwise. This is used by tus-js-client to determine whether features like stall detection can be enabled. The built-in HTTP stacks have the following support:

- `XHRHttpStack` (browser default): Returns `true` - XMLHttpRequest supports progress events
- `NodeHttpStack` (Node.js default): Returns `true` - Node.js HTTP module supports progress events
- `FetchHttpStack`: Returns `false` - Fetch API does not support upload progress events

If you're implementing a custom HTTP stack, you should return `true` only if your implementation can reliably call the progress handler set via `setProgressHandler` during the upload process.

#### urlStorage

_Default value:_ Environment-specific implementation
Expand Down
93 changes: 93 additions & 0 deletions lib/StallDetector.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
import { log } from './logger.js'
import type { StallDetectionOptions } from './options.js'

export class StallDetector {
private options: StallDetectionOptions
private onStallDetected: (reason: string) => void

private intervalId: ReturnType<typeof setInterval> | null = null
private lastProgressTime = 0
private lastProgressValue = 0
private isActive = false

constructor(options: StallDetectionOptions, onStallDetected: (reason: string) => void) {
this.options = options
this.onStallDetected = onStallDetected
}

/**
* Start monitoring for stalls
*/
start() {
if (this.intervalId) {
return // Already started
}

this.lastProgressTime = Date.now()
this.lastProgressValue = 0
this.isActive = true

log(
`tus: starting stall detection with checkInterval: ${this.options.checkInterval}ms, stallTimeout: ${this.options.stallTimeout}ms`,
)

// Setup periodic check
this.intervalId = setInterval(() => {
if (!this.isActive) {
return
}

const now = Date.now()
if (this._isProgressStalled(now)) {
this._handleStall('no progress')
}
}, this.options.checkInterval)
}

/**
* Stop monitoring for stalls
*/
stop(): void {
this.isActive = false
if (this.intervalId) {
clearInterval(this.intervalId)
this.intervalId = null
}
}

/**
* Update progress information
* @param progressValue The current progress value (bytes uploaded)
*/
updateProgress(progressValue: number): void {
// Only update progress time if the value has actually changed
if (progressValue !== this.lastProgressValue) {
this.lastProgressTime = Date.now()
this.lastProgressValue = progressValue
}
}

/**
* Check if upload has stalled based on progress events
*/
private _isProgressStalled(now: number): boolean {
const timeSinceProgress = now - this.lastProgressTime
const stallTimeout = this.options.stallTimeout
const isStalled = timeSinceProgress > stallTimeout

if (isStalled) {
log(`tus: no progress for ${timeSinceProgress}ms (limit: ${stallTimeout}ms)`)
}

return isStalled
}

/**
* Handle a detected stall
*/
private _handleStall(reason: string): void {
log(`tus: upload stalled: ${reason}`)
this.stop()
this.onStallDetected(reason)
}
}
5 changes: 5 additions & 0 deletions lib/browser/FetchHttpStack.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,11 @@ export class FetchHttpStack implements HttpStack {
getName() {
return 'FetchHttpStack'
}

supportsProgressEvents(): boolean {
// The Fetch API does not support progress events for uploads
return false
}
}

class FetchRequest implements HttpRequest {
Expand Down
5 changes: 5 additions & 0 deletions lib/browser/XHRHttpStack.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,11 @@ export class XHRHttpStack implements HttpStack {
getName() {
return 'XHRHttpStack'
}

supportsProgressEvents(): boolean {
// XMLHttpRequest supports progress events via the upload.onprogress event
return true
}
}

class XHRRequest implements HttpRequest {
Expand Down
26 changes: 24 additions & 2 deletions lib/browser/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,12 +19,32 @@ const defaultOptions = {

class Upload extends BaseUpload {
constructor(file: UploadInput, options: Partial<UploadOptions> = {}) {
const allOpts = { ...defaultOptions, ...options }
const allOpts = {
...defaultOptions,
...options,
// Deep merge stallDetection options if provided
...(options.stallDetection && {
stallDetection: {
...defaultOptions.stallDetection,
...options.stallDetection,
},
}),
}
super(file, allOpts)
}

static terminate(url: string, options: Partial<UploadOptions> = {}) {
const allOpts = { ...defaultOptions, ...options }
const allOpts = {
...defaultOptions,
...options,
// Deep merge stallDetection options if provided
...(options.stallDetection && {
stallDetection: {
...defaultOptions.stallDetection,
...options.stallDetection,
},
}),
}
return terminate(url, allOpts)
}
}
Expand All @@ -38,4 +58,6 @@ const isSupported =
// Note: The exported interface must be the same as in lib/node/index.ts.
// Any changes should be reflected in both files.
export { Upload, defaultOptions, isSupported, canStoreURLs, enableDebugLog, DetailedError }
export { XHRHttpStack } from './XHRHttpStack.js'
export { FetchHttpStack } from './FetchHttpStack.js'
export type * from '../options.js'
5 changes: 5 additions & 0 deletions lib/node/NodeHttpStack.ts
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,11 @@ export class NodeHttpStack implements HttpStack {
getName() {
return 'NodeHttpStack'
}

supportsProgressEvents(): boolean {
// Node.js HTTP stack supports progress tracking through streams
return true
}
}

class Request implements HttpRequest {
Expand Down
25 changes: 23 additions & 2 deletions lib/node/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,12 +19,32 @@ const defaultOptions = {

class Upload extends BaseUpload {
constructor(file: UploadInput, options: Partial<UploadOptions> = {}) {
const allOpts = { ...defaultOptions, ...options }
const allOpts = {
...defaultOptions,
...options,
// Deep merge stallDetection options if provided
...(options.stallDetection && {
stallDetection: {
...defaultOptions.stallDetection,
...options.stallDetection,
},
}),
}
super(file, allOpts)
}

static terminate(url: string, options: Partial<UploadOptions> = {}) {
const allOpts = { ...defaultOptions, ...options }
const allOpts = {
...defaultOptions,
...options,
// Deep merge stallDetection options if provided
...(options.stallDetection && {
stallDetection: {
...defaultOptions.stallDetection,
...options.stallDetection,
},
}),
}
return terminate(url, allOpts)
}
}
Expand All @@ -36,4 +56,5 @@ const isSupported = true
// Note: The exported interface must be the same as in lib/browser/index.ts.
// Any changes should be reflected in both files.
export { Upload, defaultOptions, isSupported, canStoreURLs, enableDebugLog, DetailedError }
export { NodeHttpStack } from './NodeHttpStack.js'
export type * from '../options.js'
15 changes: 15 additions & 0 deletions lib/options.ts
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,15 @@ export type UploadInput =
// available in React Native
| ReactNativeFile

/**
* Options for configuring stall detection behavior
*/
export interface StallDetectionOptions {
enabled: boolean
stallTimeout: number // Time in ms before considering progress stalled
checkInterval: number // How often to check for stalls
}

export interface UploadOptions {
endpoint?: string

Expand Down Expand Up @@ -84,6 +93,8 @@ export interface UploadOptions {
httpStack: HttpStack

protocol: typeof PROTOCOL_TUS_V1 | typeof PROTOCOL_IETF_DRAFT_03 | typeof PROTOCOL_IETF_DRAFT_05

stallDetection?: StallDetectionOptions
}

export interface OnSuccessPayload {
Expand Down Expand Up @@ -141,6 +152,10 @@ export type SliceResult =
export interface HttpStack {
createRequest(method: string, url: string): HttpRequest
getName(): string

// Indicates whether this HTTP stack implementation
// supports progress events during upload.
supportsProgressEvents: () => boolean
}

export type HttpProgressHandler = (bytesSent: number) => void
Expand Down
Loading
Loading