I also participated in this project, and I will submit my two cents about the project and what can be interesting for the future.
Collaborative editing
I had fun, but it was also time consuming, specially teaching each OP to format a post (instructions in chat, wait for result, repeat). My temporary solution after a while was to only concentrate to get all information in the post (and leave out formatting), then once it was posted I went to edit it (format code, etc.). It would have been nice if we had collaborative editing.
Keep the OP in chat
Keeping the OP in chat (have them respond) was a challenge as you pointed out in your post. Some thought needs to go into this to not waste both the mentor's and OP's time. Personally I had a simple strategy asking first simple questions like "How are you"?, to see if I could get a response and also to establish a minimum relationship avoiding that the OP gave up (darn all this stuff I need to improve) on me in middle of the mentor session. However I think that some hard interface should be put in place that encourage the mentee to stay in chat.
Avoiding off-topic or blunt duplicates
In your post, you don't speak about how many posts entered mentor chat, but were actually never posted. My personal experience was that probably all questions in my domain (Java) were never posted. The reason often was that I could easily find a good duplicate, indicate a typo in yet another scanner problem, etc. In all these cases the OP seemed happy to not post (compared to all the mess in comments on heavily down-voted posts).
As I see it, this was a major positive result by the mentor chat (both for community users and OPs, and. Jon hasand others have more time to answer a good question). I am aware that this can also be dangerous; chat becomes a help-center to teach how to search the web, but currently I prefer to indicate how to search, instead of involving multiple community users to review and execute moderation actions on athe post.
We also avoid a fair amount of resource requests, super user, etc. questions even if it was harder to convince the OP and keep them in a good mood.
Opt-in to mentor only in certain tags
This is related to the previous point, but also naturally it's simpler (and for some, more fun) to mentor when you have domain knowledge. The final result of the mentor session will normally be better, and I think it will be easier to get mentors to participate if they can do it on tags that they like; they feel more secure and often more motivated.
Conclusion
Probably it's no surprise that a new user with some mentoring can post (or not post off-topic) a question better than without. The challenge is to find mentors (of the 63 volunteers probably only a handful was really active), that are willing to invest time and have fun.
I had fun, thank you Kristina.