Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings
This repository was archived by the owner on Apr 11, 2023. It is now read-only.

Commit 9d83463

Browse filesBrowse files
committed
more readme updates
1 parent 7e3994f commit 9d83463
Copy full SHA for 9d83463

File tree

2 files changed

+9
-9
lines changed
Filter options

2 files changed

+9
-9
lines changed

‎BENCHMARK.md

Copy file name to clipboardExpand all lines: BENCHMARK.md
+8-8Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,13 @@
11
## Submitting runs to the benchmark
22

3-
The Weights & Biases (W&B) benchmark tracks and compares models trained on the CodeSearchNet dataset by the global machine learning research community. Anyone is welcome to submit their results for review.
3+
The [Weights & Biases (W&B)](https://www.wandb.com) [benchmark](https://app.wandb.ai/github/codesearchnet/benchmark) tracks and compares models trained on the CodeSearchNet dataset by the global machine learning research community. Anyone is welcome to submit their results for review.
44

55
## Submission process
66

77
### Requirements
88

99
There are a few requirements for submitting a model to the benchmark.
10-
- You must a have a run logged to [W&B](https://app.wandb.ai)
10+
- You must a have a run logged to [W&B](https://app.wandb.ai).
1111
- Your run must have attached inference results in a file named `model_predictions.csv`. You can view all the files attached to a given run in the browser by clicking the "Files" icon from that run's main page.
1212
- The schema outlined in the submission format section below must be strictly followed.
1313

@@ -20,11 +20,11 @@ A valid submission to the CodeSeachNet Challenge requires a file named **model_p
2020
* `identifier`: this is an optional field that can help you track your data
2121
* `url`: the unique GitHub URL to the returned results, e.g. "https://github.com/JamesClonk/vultr/blob/fed59ad207c9bda0a5dfe4d18de53ccbb3d80c91/cmd/commands.go#L12-L190" . This information is available as a field in the data to be scored.
2222

23-
For further background and instructions on the submission process, see the root README.
23+
For further background and instructions on the submission process, see [the root README](README.md).
2424

2525
The row order corresponds to the result ranking in the search task. For example, if in row 5 there is an entry for the Python query "read properties file", and in row 60 another result for the Python query "read properties file", then the URL in row 5 is considered to be ranked higher than the URL in row 60 for that query and language.
2626

27-
The script we used to create the baseline submission is [src/predict.py](src/predict.py). You are not required to use this script to produce your submission file -- we only provide it for reference.
27+
The script we used to create the baseline submission is [src/predict.py](src/predict.py). You are not required to use this script to produce your submission file; we only provide it for reference.
2828

2929
Here is an example:
3030

@@ -49,12 +49,12 @@ You can submit your results to the benchmark as follows:
4949
2. Generate your own file of model predictions following the format above and name it \`model_predictions.csv\`.
5050
3. Upload a run to wandb with this \`model_predictions.csv\` file attached.
5151

52-
Our example script [src/predict.py](src/predict.py) takes care of steps 2 and 3 for a model whose training run has been logged to W&B, given the corresponding W&B run id, which you can find on the /overview page in the browser or by clicking the 'info' icon on a given run.
52+
Our example script [src/predict.py](src/predict.py) takes care of steps 2 and 3 for a model training run that has already been logged to W&B, given the corresponding W&B run id, which you can find on the /overview page in the browser or by clicking the 'info' icon on a given run.
5353

5454
Here is a short example script that will create a run in W&B and perform the upload (step 3) for a local file of predictions:
5555
```python
5656
import wandb
57-
wandb.init(project="codesearchnet", resume="must")
57+
wandb.init(project="codesearchnet")
5858
wandb.save('model_predictions.csv')
5959
```
6060

@@ -70,11 +70,11 @@ or by selecting a run from the runs table:
7070

7171
### Result evaluation
7272

73-
Once you upload your \`model_predictions.csv\` file, W&B will compute the normalized cumulative gain (NCG) of your model's predictions against the human-annotated relevance scores. Further details on the evaluation process and metrics are in the root README. For transparency, we include the script used to evaluate submissions: [src/relevanceeval.py](src/relevanceeval.py)
73+
Once you upload your \`model_predictions.csv\` file, W&B will compute the normalized discounted cumulative gain (NDCG) of your model's predictions against the human-annotated relevance scores. Further details on the evaluation process and metrics are in the [the root README](README.md). For transparency, we include the script used to evaluate submissions: [src/relevanceeval.py](src/relevanceeval.py)
7474

7575

7676
### Training the baseline model (optional)
7777

78-
Replicating our results for the CodeSearchNet baseline is optional, as we encourage the community to create their own models and methods for ranking search results. To replicate our baseline submission, you can start with the instructions in the [CodeSearchNet GitHub repository](https://github.com/ml-msr-github/CodeSearchNet). This baseline model uses [src/predict.py](src/predict.py) to generate the submission file.
78+
Replicating our results for the CodeSearchNet baseline is optional, as we encourage the community to create their own models and methods for ranking search results. To replicate our baseline submission, you can start with the instructions in the [CodeSearchNet GitHub repository](https://github.com/github/CodeSearchNet). This baseline model uses [src/predict.py](src/predict.py) to generate the submission file.
7979

8080
Your run will be logged to W&B, within a project that will be automatically linked to this benchmark.

‎README.md

Copy file name to clipboardExpand all lines: README.md
+1-1Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -85,7 +85,7 @@ More context regarding the motivation for this problem is in this [technical rep
8585

8686
## Evaluation
8787

88-
The metric we use for evaluation is [Normalized Discounted Cumalitive Gain](https://en.wikipedia.org/wiki/Discounted_cumulative_gain#Normalized_DCG). Please reference [this paper][paper] for further details regarding model evaluation.
88+
The metric we use for evaluation is [Normalized Discounted Cumulative Gain](https://en.wikipedia.org/wiki/Discounted_cumulative_gain#Normalized_DCG). Please reference [this paper][paper] for further details regarding model evaluation.
8989

9090
### Annotations
9191

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.