Add batched inference #771

New issue

Copy link

New issue

Copy link

Open

#951

Open

Add batched inference#771

#951

Copy link

Labels

enhancementNew feature or requestNew feature or requesthigh-priority

Description

abetlen

opened on Sep 30, 2023

Issue body actions

• Use llama_decode instead of deprecated llama_eval in Llama class
• Implement batched inference support for generate and create_completion methods in Llama class
• Add support for streaming / infinite completion

👍React with 👍50giangluu352001, harry-pham-wise, JackKCWong, bb-worm, ChristianWeyer and 45 more🚀React with 🚀21sengiv, ArtyomZemlyak, hamishc, bioshazard, gerred and 16 more👀React with 👀17esmeetu, robertritz, zhengzhanpeng, hamishc, ngupta10 and 12 more

Metadata

Metadata

Assignees

No one assigned

Labels

enhancementNew feature or requestNew feature or requesthigh-priority

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions