Next for Coding Challenges: submit your own ideas

Challenge success criteria has so far been an absolute mess - The fourth overly open-ended challenge serves as an extreme example of this. Awards at the end of the challenge feel pulled out of a hat to retroactively fit specific submissions that staff like. Voting on challenges is also entirely counterproductive, especially with no downvotes - It's entirely subjective, earlier submissions get voted to the top faster, and crowd appeal joke answers are encouraged (e.g. the challenge 4 submission that was a reposted xkcd panel).

Challenges need clear goals and success criteria to enable a range of different motivations to problem solve - Want to top the leaderboard? Here are the exact objectives you'll be judged against. Want to do better for yourself to improve your skills? Those same objectives let you look at your submission and say "hey, I did it" without having to care about whether others did "better". Sense of pride and accomplishment and all that.

These criteria should, in my opinion, also be relatively consistent between challenges at least initially to build an identity for the feature. If I want to challenge myself with problem sets and brush up on my algorithms, I go to Leetcode. If I want to golf I go to Code Golf SE. If I want to build something unique in a short time I go to a hackathon. The question SO needs to answer is "What would I go to SO Challenges for?" This is not something your users can do for you - This is something that should be decided before even running an experiment.

In their current state, challenges feel like a transparent attempt to create some kind of engagement to juice numbers on a dashboard somewhere and if SO is committed to keeping them around as a feature they need to have a clear purpose and motivate participation for reasons other than "maybe I'll get called most imaginative in a couple weeks".

I was expecting more realistic and in-depth analysis of Challenges and a bit clearer plan for moving forward, but let me offer my two cents anyway.

Let me say this upfront: The concept of Code Challenges itself has merit and there were some genuinely creative submissions. While I did not participate myself (besides moderating and some comments), it seems that folks enjoyed the experiment overall.

On the participation numbers:

In the first phase of the Coding Challenges experiment we posted 4 challenges for users to participate in, and we got a great response from the community overall. We were excited to see diverse and creative solutions to the prompts, as well as participation from new users.

...

Throughout the experiment thousands of users viewed challenges, and 426 unique users interacted with them. Specifically, 171 submitted an entry, 34 commented on an entry, and 298 voted. Excluding challenge 4, which was an outlier, each challenge got an average of 65 entries (not counting ones that were deleted for any reason).

Well, it wasn't all positive. You received an incredible amount of AI-generated content. So I would not interpret the number of submissions so positively. For the first 3 challenges, we were discussing the appropriate amount of moderation, and during the fourth one (when we at least could settle that blatant AIGC should not be allowed, and CMs released the kraken -- me) the number of legitimate (which was/is a low bar, as evident from some of the remaining submissions on Challenge #4) dropped significantly.

Excluding challenge 4, which was an outlier, each challenge got an average of 65 entries (not counting ones that were deleted for any reason).

Again, the reason was mostly blatant AI use. There were a handful of posts that were gibberish or spam, but out of 58 submissions (if my count is right), around 40 were AI-generated submissions.

On Challenge #4 specifically:

The fourth challenge was not received as well as the first three, in part because the prompt was too open-ended to be fun – lesson learned, challenges need to have constraints for people to get creative within!

I do agree that Challenge #4 had serious issues, but the raw number of submissions did not see a significant drop, so this statement is not entirely accurate.

On the platform choice:

When we first announced the Challenges experiment, we decided to temporarily take over the Discussions space, as an efficient vehicle for a quick test of a new idea.

Efficient in what sense? It was a fine space to perform the experiment. But it is far from anything remotely suitable. Several issues I can think of just now:

• No pagination and with all the long submissions (which is inherently in the nature of Challenges) it is impossible to navigate the page

• Moderation was extremely hard since all the deleted posts are cluttering the page

• Editing submissions (yours or someone else's) is extremely challenging since the page keeps scrolling down with every letter that you type

• The way we are announcing the results is also sub-optimal (an image added to the original post is not very effective)

• The system for voting is not fair. Showing the score can (and will) bias opinions. The submissions should be in random order (like election pages).

On the sandbox idea:

As the next step for this feature, we want to move towards allowing users to submit their own challenges for others to attempt. So before we roll out the next set of challenges we are going to start collecting challenge prompts and ideas in a Challenges Sandbox!

This is a good idea. But again, the UI is not suitable to handle collaboration and back and forth discussions (ironically enough as it was designed for Discussions).

Please pay the development debt before moving forward.

About Challenge #4

Excluding challenge 4, which was an outlier, each challenge got an average of 65 entries [...]. The fourth challenge was not received as well as the first three, in part because the prompt was too open-ended to be fun – lesson learned, challenges need to have constraints for people to get creative within!

I find it a little too easy to exclude the worse received challenge out of the numbers.

Aside from that, I'd say the reason why it was not well received is as much because it was too open-ended as because it was centered on AI, which is heavily disliked on SO. Sounds like you're ignoring the elephant in the room, here.

Also "too open-ended to be fun" sounds like an excuse to make it seem like you had the greatest of intentions, only wanted to make a fun challenge, and couldn't predict that such a negative reception would happen. It was "too open-ended", period. No "to be fun" needed here. Beyond enough feedback was provided under the four Meta posts to indicate that this would happen.

About previously given feedback

The four Meta posts made about Coding Challenges have received a ton of answers providing very valuable feedback, most of which has been ignored so far.

I would have expected this new post to tell us which parts of that feedback will you consider implementing and which will you not and why. Instead, you're calling for feedback a fifth time. I'm really, REALLY tired of repeating myself as I already have FOUR times. Are you going to keep ignoring existing feedback and calling for more until people give up?

About the Challenge Sandbox

This should have been a Meta post, not a Coding Challenge entry. As it is posted as a Coding Challenge, it's impossible to see people's challenge proposals unless we post one ourselves or wait for 10 days.

This will inevitably lead to duplicate and/or similar entries being posted instead of people upvoting each other. Besides, in this context, downvoting should be possible too, as it would make sense to dislike a coding challenge idea and not wanting it on the site as much as it'd make sense for the opposite.

Award criteria

A already pointed out by Marsroverr, awards were essentially given as Staff subjectively decided they would, except for the most upvoted award.

You need to give us criteria for the awards that are as objective as possible, without relying too much on upvotes.

Upvotes are also heavily biased due to the lack of downvotes, and as an extreme demonstration of this, the winner of Challenge #4 won for posting a joke, not even meeting the challenge's prompt.

Also, the "New contributor" award is extremely unfair. Awards should be given based on the submission, not who posted it. I get you want to encourage new users to contribute to the platform and blah blah blah, but if that means compromising objectivity in a contest, then getting those new contributors might chase off your old ones. Can you imagine how much of a scandal would it be if a new competitor in a sports competition got extra points because they're new?

About AI

I'd like you to read (or read again if you already have) the "Please don't encourage AI usage" and "AI usage instructions are, again, UNCLEAR" paragraphs of my answer under the Challenge #4 post, please, as it seems you still haven't gotten the message, judging by the fact that nothing was done about it for any of the four challenges even though it was said to you numerous times during the experiment.

In short: there's a reason why AI is so heavily disliked on SO, and allowing it in a Coding Contest is ruining the fun. There is no point in a contest if you are allowed to cheat, even in part. Challenges are meant to challenge people's skills, not the AI's.

Feedback:
I understand that you value the number and quality of submitted solutions, but don't forget: Not everyone who attends the challenge submits his/her solution or finishes in time.
It is a valid approach to start the challenge even after it is already closed.

Unlike the rest of the stackexchange sites, the goal is not to find the one best solution within a short time.
I find the ranking upon the number of votes not very useful in this context, also the listing of all submitted solutions can become overwhelming.
On the other hand side, it could still be useful to submit solutions even after the official end of the challenge. I see it more important to have fun during the challenge and maybe learn something new, no matter how well you perform.

All in all, I think the general display pattern from the stackexchange sites does not really fit for these challenges, but it is difficult to tell how to make it better.
Maybe an entire group per challenge could be useful, e.g. one topic for the main description and separate topics to submit the solutions in different categories, e.g. one per programming language if there are too many answers.

Not sure if this Idea should go into this topic or into the Challenges Sandbox.
I put it here as it is more a layout idea for challenges instead of an actual challenge.

I once participated in an interesting online coding challenge series which played around a bigger space traveling scenario where the coder had to fulfill certain tasks to proceed.
Unfortunately I forgot the name of the series and I am unable to find it anymore.

The attended challenge:
The tasks were in the form of a description and a function signature with predefined input parameters and an expected return value.
For example "Here is a list of available planets and positions. Find a step-by-step path to navigate to planet No. N".
The code was then loaded into a test framework which invoked the function with various tests to verify the functionality.

The framework accepted many different programming languages, so the programmer could choose almost any preferred language.
The single challenges were to implement the function in a very limited amount of time (e.g. 90 minutes).

My personal impression of the challenge:
The challenge itself sounded very promising, in theory it allows to build up a larger system which can be assembled step by step and I didn't had the burden to write tests.
I found it very nice to have a bit of a story-driven approach, but I don't see this as a must. What I disliked about the challenges was the school-like situation where you had to write something in a very short amount of time and then everything is taken away from you. From my experience this is not very realistic and more of a burden than a benefit.

My thoughts about how to make it better and for stackoverflow:

• I don't see a need to make such a complex online framework which has to be somehow robust against abuse. It should be sufficient to provide sets of tests which can be downloaded by the attendees for multiple programming languages.
• For those unlike me, who want such a tight timing challenge, it is easily possible to give a recommended time limit for each challenge.
• If someone does not manage to fulfil one single challange the attendee can get inspiration from other published solutions or simply take over an existing solution and proceed to the next challenge of the series, maybe retry the failed challenge some time later.

I hope there are others out there who love to assemble something bigger over time and get it to run instead of writing just small independent pieces of code.
I am aware that such a setup does not match all types of challenges, e.g. this won't really work for user interfaces, but I am sure this setup can fit for multiple challenges.