Because of my use case, I uses LLMs with very high concurrency (~500), which means I need to spin up 500 inference request at same time on my local client, and make the end to end time unstable. For example in one of my tests, the total end to end latency for 500 concurrent inference request is around 4.19 seconds, but the longest request only takes 3.16s. Is it possible to handle this kind of batching request on tensorzero (just a grouped together normal inferences) to reduce overhead in creating http connection?

Also, if possible, providing method to avoid duplicate arguments across batched requests will be really helpful in reducing network overhead.
For example, my request could contain two arguments, context, search_result, and my (batched) inference requests could look like this:

[
    {
        "messages": [
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "arguments": {
                            "context": "User is asking about recent news.",
                            "search_result": "{news 1}"
                        }
                    }
                ]
            }
        ]
    },
    {
        "messages": [
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "arguments": {
                            "context": "User is asking about recent news.",
                            "search_result": "{news 2}"
                        }
                    }
                ]
            }
        ]
    }
]

The "User is asking about recent news." could be grouped to avoid repeating the same long context multiple times, which makes the request payload too large.