Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

Commit 21ee91e

Browse filesBrowse files
authored
Add survey results (rust-gamedev#1520)
1 parent c67106b commit 21ee91e
Copy full SHA for 21ee91e

16 files changed

+214
-0
lines changed

‎content/blog/survey-02/ai.png

Copy file name to clipboard
55.2 KB
Loading
76.4 KB
Loading

‎content/blog/survey-02/comment.png

Copy file name to clipboard
590 KB
Loading
47.5 KB
Loading
51.4 KB
Loading
51.4 KB
Loading

‎content/blog/survey-02/ease.png

Copy file name to clipboard
41.9 KB
Loading
55.6 KB
Loading

‎content/blog/survey-02/excitement.png

Copy file name to clipboard
44.9 KB
Loading
Loading

‎content/blog/survey-02/frequency.png

Copy file name to clipboard
51.4 KB
Loading

‎content/blog/survey-02/improve.png

Copy file name to clipboard
746 KB
Loading

‎content/blog/survey-02/index.md

Copy file name to clipboard
+214Lines changed: 214 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,214 @@
1+
+++
2+
title = "Newsletter Survey Results"
3+
date = 2024-06-03
4+
transparent = true
5+
draft = false
6+
+++
7+
8+
Since we are [rebooting the newsletter](https://gamedev.rs/blog/newsletter-changes/), we wanted to know more about our readers.
9+
52 of you filled out the survey last month. Thank you very much!
10+
11+
The biggest takeaways are:
12+
- People are generally excited about the newsletter
13+
- The current frequency of the newsletter is good
14+
- Readers do not want anything in the newsletter generated by AI
15+
- Contributing to the newsletter could be easier. If you've got ideas on how to make this happen, please [let us know](https://github.com/rust-gamedev/rust-gamedev.github.io/issues/1519)!
16+
17+
We will now go through the results in the same order as the questions were asked. The full analysis and data is open-sourced on [GitHub](https://github.com/janhohenheim/rust-gamedev-statistics/tree/main/jan-hohenheim-2024).
18+
19+
## Excitement
20+
21+
![Excitement barplot](excitement.png)
22+
23+
On average, readers are excited about the newsletter. The mean excitement level is 3.6 out of 5, the median is 4.
24+
Our 95% confidence interval is [3.32, 3.91] using a standard error of 0.15 (sd = 1.05, n = 52).
25+
26+
These are fairly nice results. Anecdotally, we got a lot of messages about issues with the newsletter and how to improve it,
27+
so we are happy to see that the excitement is still high. Still, the data shows that we have room for improvement.
28+
29+
## Content Quantity
30+
31+
![Content quantity barplot](content_quantity.png)
32+
33+
34+
When asked about how to change the amount of content per newsletter, the majority of readers (58%) voted to leave the amount as-is or don't care.
35+
On the other hand, this means nearly half of the readers would change something about the content quantity.
36+
17% voted for "less content; keep only the most important news" and 25% for "more content; add sections for minor news".
37+
These two options are luckily not mutually exclusive.
38+
One option we could implement is to have a new section for "minor news" where we don't go into detail,
39+
and a section for "miscellaneous links" where we only list some links without any commentary.
40+
41+
## Newsletter Frequency
42+
43+
![Newsletter frequency barplot](frequency.png)
44+
45+
73% of readers are either happy with the current frequency or don't care. A minority of 21% would like the newsletter to become quarterly.
46+
Arguments we've heard for this are that a lower frequency would allow editors to improve the quality that goes into each newsletter.
47+
Counterarguments include that a lower frequency would make the newsletter less timely.
48+
Things like calls for playtesters or job offers would be less useful if they were only sent out every three months.
49+
50+
## AI
51+
52+
![AI barplot](ai.png)
53+
54+
This question was a catalyst for a lot of discussion on [Discord].
55+
56+
If we interpret the answers as a scale of 1-5, where 1 is "not okay at all" and 5 is "I love it", the mean answer was 2.25, the median 2.
57+
Notably, the mode is tied at 1 and 2. The 95% confidence interval is [1.89, 2.61] using a standard error of 0.18 (sd = 1.30, n = 52).
58+
People are generally against using an LLM to generate summaries. 79.2% of readers would prefer not to use AI.
59+
60+
Viewed from another angle: while a majority of readers (65%) are at least okay with AI-generated summaries,
61+
a significant minority (35%) are not okay at all with this proposal.
62+
These include very active members of the community and
63+
contributors who have announced that they would no longer want their content to be included in the newsletter if AI was used.
64+
65+
The reasons people gave for not wanting AI-generated summaries were varied.
66+
Among these were:
67+
- Solidarity with the large number of creatives who recently lost their jobs due to AI-generated content,
68+
inside and outside the game development industry.
69+
- Concerns about the quality of AI-generated summaries.
70+
- Skepticism about AI-generated summaries saving time if they still need to be edited by hand.
71+
72+
## Tone
73+
74+
![Tone barplot](tone.png)
75+
76+
A significant majority of readers (86.5%) are happy with the current tone of the newsletter, with a minority of 11.5% wanting a less formal tone.
77+
While votes for the latter did not reach a majority, the written feedback we got included quite a few requests for more "personality" in the newsletter.
78+
79+
## Contributions
80+
81+
![Contributions barplot](contributions.png)
82+
83+
The majority of readers (61.5%) have not yet contributed to the newsletter and 26.9% have contributed 2-5 times.
84+
Only 3.8% contributed exactly once, while the rest (7.7%) are heavy contributors, helping us out more than five times.
85+
86+
While it might seem weird that more people contributed 2-5 times than exactly once,
87+
keep in mind that the former is the sum of people who contributed twice, thrice, four times, and five times.
88+
The reason we binned these together is that we are interested in the following categories:
89+
- Pure readers
90+
- People who contributed once and then stopped
91+
- People who contributed a few times
92+
- People who are regular contributors
93+
94+
We are happy to see that people who contributed once seem to continue contributing in the future.
95+
96+
## Ease
97+
98+
![Ease barplot](ease.png)
99+
100+
The mean ease of contributing is 3.0, and the median is 3. The 95% confidence interval is [2.5, 3.6] using a standard error of 0.26 (sd = 1.26, n = 23).
101+
102+
Readers generally feel neutral about the ease of contributing to the newsletter.
103+
104+
We can do better here, but we are not sure yet how.
105+
We'd love to hear your ideas on [GitHub](https://github.com/rust-gamedev/rust-gamedev.github.io/issues/1519) or on [Discord] (ping @janhohenheim).
106+
107+
## Keeping up with the newsletter
108+
109+
![Keeping up barplot](informed.png)
110+
111+
This was a multiple-choice question. The most popular source of information about the newsletter is RSS (27.5%).
112+
If we add the choices for the official [Rust GameDev Discord server][Discord] (21.7%) and other Discord servers (11.6%),
113+
Discord in general becomes the leading source of information (33.3%), taking up nearly exactly a third of all votes.
114+
115+
We can see the shift from X / Twitter to Mastodon reported by many OSS communities in our readers as well.
116+
Lemmy is not looking popular as an alternative to Reddit yet, with no reader reporting it as a source of information.
117+
118+
The "Email" option in the survey is meant for people who have set up some kind of email alerts manually.
119+
120+
Per written feedback, a lot of people want to see proper email subscriptions implemented. While this was a goal for this month,
121+
we have not managed to implement it yet.
122+
We will try to [get this done](https://github.com/rust-gamedev/rust-gamedev.github.io/issues/24) for the next newsletter.
123+
124+
## What is going well
125+
126+
![What is going well wordcloud](like.png)
127+
128+
This was a free-text question. The above is a word cloud of the answers with some obvious words like "game" or "newsletter" removed.
129+
Note that the inclusion of the word "AI" is misleading, as it was only mentioned in answers that read similar to
130+
"I like that we don't use AI, please don't use LLMs".
131+
132+
Going through the feedback by hand, common things readers enjoy about the newsletter are:
133+
- A good mix of content
134+
- Very open to contributions
135+
- Small-scale games are featured, not just success stories or technical articles
136+
137+
## What needs to be improved
138+
139+
![What needs to be improved wordcloud](improve.png)
140+
141+
Another free-text question. The feedback here is fairly diverse. The most common complaints we already mentioned in previous sections are:
142+
- Add an email subscription
143+
- Improve the ease of contributing
144+
145+
Additionally, many people feel like the "Games" section reads more like an advertisement than an article aimed at other game developers.
146+
147+
Among the more unique suggestions were:
148+
- Conduct interviews
149+
- Have a stronger sense of personality in the writing
150+
- Make the newsletter more consistent in timing and quantity
151+
- Have more editors to not overburden the current ones
152+
- Have some more clarity of purpose
153+
154+
155+
## Comments
156+
157+
![Comments wordcloud](comment.png)
158+
159+
This last free-text question was meant for any additional comments readers might have.
160+
The word cloud above is dominated by one sentence: "Thank you for your work". Thank you very very much for your kind words!
161+
We are working on this newsletter in our free time because we love the community and Rust game development, so reading this means a lot to us.
162+
163+
## Correlations
164+
165+
We were interested in how the responses to some questions correlated with how much people had already contributed to the newsletter.
166+
Long story short: it seems like there is no significant correlation between how much people contributed and how they answered the other questions.
167+
168+
Let's look at the correlations in turn now.
169+
Note that all the following plots are jittered to make the data more readable.
170+
171+
172+
### Excitement By Contributions
173+
174+
![Excitement correlation](excitement_contributions.png)
175+
176+
We hypothesized that people who contributed more to the newsletter would be more excited about it.
177+
We found no evidence for this (the p-value of a Jonckheere-Terpstra test for increasing trend is 0.986).
178+
Based on the plot above, we then hypothesized that the opposite might be true, namely that frequent contributors are less excited about the newsletter.
179+
This actually might be the case (p-value is 0.021), but do not take this as a strong result.
180+
It is a posthoc hypothesis and the resulting p-value is not very low considering the number of tests we run in this analysis.
181+
For these reasons, we do not consider this result to be significant.
182+
183+
### Feelings About AI By Contributions
184+
185+
![AI correlation](ai_contributions.png)
186+
187+
We hypothesized that there would be a correlation between how much people contributed to the newsletter and how they felt about AI-generated summaries.
188+
We found no evidence for this (the p-value of a Jonckheere-Terpstra test for a two-sided alternative is 0.57).
189+
190+
### Ease of Contributing By Contributions
191+
192+
![Ease correlation](ease_contributions.png)
193+
194+
We hypothesized that there would be a correlation between how much people contributed to the newsletter and how easily they found it to contribute.
195+
196+
We found no evidence for this (the p-value of a Jonckheere-Terpstra test for a two-sided alternative is 0.25).
197+
198+
## Conclusion
199+
200+
All in all, we are happy with the results of the survey.
201+
It seems like our readers are generally happy with the newsletter, and have good ideas on how to improve it.
202+
We will discuss how to implement these ideas in the future and keep you updated on our progress.
203+
204+
If you are interested in helping us out, we are always looking for new editors and contributors. Just leave us a message on [Discord] or [GitHub].
205+
206+
Again, thank you very much for your feedback. Rebooting the newsletter was a big ordeal for us,
207+
and we are happy to see such an active interest in the community. We hope that we can continue to provide you with a newsletter you enjoy.
208+
209+
Until next time!
210+
211+
~ The Rust GameDev Newsletter Team, and Jan Hohenheim in particular
212+
213+
[Discord]: https://discord.gg/yNtPTb2
214+
[GitHub]: https://github.com/rust-gamedev/rust-gamedev.github.io

‎content/blog/survey-02/informed.png

Copy file name to clipboard
92 KB
Loading

‎content/blog/survey-02/like.png

Copy file name to clipboard
497 KB
Loading

‎content/blog/survey-02/tone.png

Copy file name to clipboard
43.1 KB
Loading

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.