missing newline at EOF

So for this a user is expected to bring a list of Sample objects, which hold the input, prediction, and ground truth? Are we going to provide a way to build this list of Samples from given files or lists of each category, or is this moreso just for use with self-built scripts that import the Sample object and build?

Updated it such that dataset now is either a pathlib.Path object or a list of samples, and we read what we need to accordingly

We had discussed this earlier, I think you're going to want to remove this comment.

@alimaredia I'll update this comment because I believe you are confusing it's meaning with our earlier conversation. This comment relates to the seed not being supported by every model serving framework.

@RobotSail shouldn't we hit a mock client here too.

Instead of using gpt-3.5?

@abhi1092 We do test it with a mock-client, GPT-3.5 was just the first model that came to mind as a fill-in.

Updated this to use fake values to eliminate the possibility of it actually calling out to the real openai API.

@RobotSail maybe this method and also the class can instead take the client as input?

@abhi1092 That's a good idea. Although the way I've done it here is how it happens in MT-Bench as well. Would it make sense to make a follow-up issue for this?

See: https://github.com/RobotSail/eval/blob/ad12276878dfa691599e37aa5e18d6c1bd7f4afd/src/instructlab/eval/mt_bench_answers.py#L120 for reference

@abhi1092 Actually I will just make this change in this PR since it's small.

Updated the PR to include this.

Here too maybe we can call mock gpt-4o with pre-defined evaluation result. This way we only test the evaluation given the student model response, reference and judge model output

@abhi1092 It is getting mocked, see: https://github.com/instructlab/eval/pull/193/files#diff-07ba3868aa4f9cd9324d6bfebe02ad4b4a3928285c0f01d29ef3d12d3f3b09b4R94

I think there's an implicit requirement here that the dataset referred to by the path is well-formed (shaped like list[Sample]). Could consider doing a quick check to make sure the required columns are present in the df and failing here if they aren't.

Sure, I don't see a reason not to.

How is the API key supposed to be passed into the judge model? If we're assuming the environment variable is set for that we need When I run the script you the short script you sent me yesterday I hit this as an issue.

There were previous iterations of this PR that had a base_url and a key in the ModelConfig that I think are needed since the student and the judge need at least a base_url if not both.

I'm realizing now, maybe your intention is that the users OpenAI key for the judge model is already set before calling RagasEvaluator.run(). Could we have a check for that or make it more clear somehow?

I think since we want to lock it down to OpenAI for now just to have a consistent evaluation base, we can just expose the API key. We can make a follow-up where this is something configurable.

This comment is outdated, we've since updated this evaluation to also accept the judge model name as well.

What's the use of this function? Since it's only being used in one place and we can't configure what's being passed into rubrics.

The point is that since we select the metrics carefully, having them isolated allows us to treat them with more care. Also - when you add to this list it becomes very messy, so it's just slightly more readable to have it in a separate method.