xet upload: parrallelize xorb/shard creation

They can be parallelized, either in current execution context (to offset network delays) or in separate workers (to offset both network costs and chuning/hashing costs)

Current unparallelized code:

huggingface.js/packages/hub/src/lib/commit.ts

Lines 326 to 362 in af1fd37

    
           for await (const event of uploadShards( 
        
           	(async function* () { 
        
           		for (const obj of json.objects) { 
        
           			const op = shaToOperation.get(obj.oid); 
        
           			if (!op || !obj.actions?.upload) { 
        
           				continue; 
        
           			} 
        
           			abortSignal?.throwIfAborted(); 
        
           			yield { content: op.content, path: op.path, sha256: obj.oid }; 
        
           		} 
        
           	})(), 
        
           	{ 
        
           		customFetch: params.fetch ?? fetch, 
        
           		accessToken, 
        
           		hubUrl: params.hubUrl ?? HUB_URL, 
        
           		repo: repoId, 
        
           		// todo: maybe leave empty if PR? 
        
           		rev: params.branch ?? "main", 
        
           	} 
        
           )) { 
        
           	if (event.event === "file") { 
        
           		yield { 
        
           			event: "fileProgress", 
        
           			path: event.path, 
        
           			progress: 1, 
        
           			state: "uploading", 
        
           		}; 
        
           	} else if (event.event === "fileProgress") { 
        
           		yield { 
        
           			event: "fileProgress", 
        
           			path: event.path, 
        
           			progress: event.progress, 
        
           			state: "uploading", 
        
           		}; 
        
           	} 
        
           }

Downside is that it will create more xorbs/shards, but xet team is working on shard consolidation cc @assafvayner

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

xet upload: parrallelize xorb/shard creation #1704

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	for await (const event of uploadShards(
	(async function* () {
	for (const obj of json.objects) {
	const op = shaToOperation.get(obj.oid);
	if (!op \|\| !obj.actions?.upload) {
	continue;
	}
	abortSignal?.throwIfAborted();

	yield { content: op.content, path: op.path, sha256: obj.oid };
	}
	})(),
	{
	customFetch: params.fetch ?? fetch,
	accessToken,
	hubUrl: params.hubUrl ?? HUB_URL,
	repo: repoId,
	// todo: maybe leave empty if PR?
	rev: params.branch ?? "main",
	}
	)) {
	if (event.event === "file") {
	yield {
	event: "fileProgress",
	path: event.path,
	progress: 1,
	state: "uploading",
	};
	} else if (event.event === "fileProgress") {
	yield {
	event: "fileProgress",
	path: event.path,
	progress: event.progress,
	state: "uploading",
	};
	}
	}

Search code, repositories, users, issues, pull requests...

xet upload: parrallelize xorb/shard creation #1704

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions