Important

Introduces worker-based token counting using workerpool, updates BaseProvider to utilize it, and adds tests for the tiktoken function.

• Behavior:
  • Introduces worker-based token counting using workerpool in countTokens.ts.
  • Updates BaseProvider in base-provider.ts to use countTokens with workers.
  • Adds tiktoken function in tiktoken.ts for token counting.
• Testing:
  • Adds tests for tiktoken in tiktoken.test.ts to verify token counting for text and image content.
• Dependencies:
  • Adds workerpool as a dependency in package.json.
• Configuration:
  • Updates esbuild.js to include worker configuration for countTokens.ts.
  • Modifies knip.json to ignore src/workers/**.

^{This description was created by for 8224d4c. You can customize this summary. It will automatically update as commits are pushed.}