Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

CLI Arg to Dump Formatted Prompt Into NDJSON #5

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

tc-wolf
Copy link
Owner

@tc-wolf tc-wolf commented Apr 18, 2025

  • Update Makefile to set build type properly
  • Set formatted prompt path
    • From CLI in settings
    • In Llama in __init__
    • Retrieve in __getstate__
  • Write out to file (append per-request) if set on Llama object in llama_chat_format
  • Formatting
  • Minor changes for building to make smaller

tc-wolf added 6 commits April 17, 2025 16:46
- Add in ModelSettings as optional string
  - Will be added automatically to CLI args
- Set in kwargs for llama (also add to `__getstate__`) and in __init__
- Save prompt in `llama_chat_format` `chat_completion_handler`
- Set as env var rather than in CMAKE_ARGS as -DCMAKE_BUILD_TYPE - was
  seeing message in logs.
  - Can now see that being set properly.
- Needs Python 3.9 due to numpy, other deps anyway
  - Docker builder image already needs 3.9
- Add filelock as dep
  - Will use to ensure that the prompt file is not clobbered if multiple
    requests.
- Must end with `.ndjson` suffix
- Add tests
Ensure that if have multiple concurrent requests don't clobber file.

This is probably overkill since concurrent requests will actually
interrupt one another w/ default server settings.
Don't require installing all dev packages, just pyinstaller when doing `deploy.pyinstaller.mac`.

Also don't use temp var for lockfile, never referred to.
@tc-wolf tc-wolf merged commit f4fe27f into bumped_llama_cpp_with_disk_cache Apr 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant
Morty Proxy This is a proxified and sanitized view of the page, visit original site.