@Florian: You're all clear now!

The user story at T100508#2014479 is just one specific case (and I don't believe it is the most common case). It's a good starting point, but we need more user stories and more UI ideas.

@bd808: How hard would it be to build some kind of API proxy on Labs for the Yandex search API (so that we can access it via Tool Labs through a single IP)?

@Niharika: Needs 1 additional change. See comment at pull request.

@Niharika: Thats a good idea. Let's set the User Agent string as a const in the checkIfDead class and use that with the curl requests. I don't believe there are any legal or ethical problems with spoofing user agent strings. Internet Explorer has been spoofing Netscape Navigator since 1998 :)

@Niharika: Please build this on top of https://gerrit.wikimedia.org/r/#/c/263575/ rather than as a new branch.

@Florian: I ended up abandoning the previous changes, but the refactoring still needs to be merged at https://gerrit.wikimedia.org/r/#/c/283584.

@Earwig: I'll be turning off the Bing Search API on Monday.

Looks like we can't use the Bing API due to their restrictive Terms of Use. I'm currently in talks with them about getting an exemption, but who knows how long that will take (especially since lawyers will be involved).

@Niharika: Do you have any examples of the pages behind paywalls?

@Niharika: See my comments at https://en.wikipedia.org/wiki/Wikipedia:Bots/Requests_for_approval/Cyberbot_II_5a#DB_Results.

Nice work guys! That's a big improvement.

merged.

Reopening, since this will be useful for the new copy and paste bot interface as well.

Pull request at https://github.com/MusikAnimal/pageviews/pull/65.

The cookies issue should be fixed now, per https://github.com/MusikAnimal/pageviews/pull/65 (at least as soon as that's merged and deployed).

In reply to your open questions, it seems that the best way that we could remove redundancy would be to create a centralized API (similar to what plagiabot already offers), but that offered more semantic data (rather than HTML), and was integrated into both Turnitin and a general search engine such as Bing.

Putting this in Blocked column since I need some help navigating the cache invalidation code for modules (which is rather complicated).

Dan requested that the EasyPrivacy list whitelist the API on April 2nd. No response yet.

Is that what XTools does? refresh the page with use_lang=en-US? Seems kind of janky :)

Fixed. Just waiting for @MusikAnimal to review and deploy.

@Volans: Thanks for the info. I'll check up on things again after the 18th.

@Volans: I asked two questions that haven't been answered. Perhaps you could help answer them?

@jcrespo: ping.

Currently waiting on feedback from @Legoktm per comments in gerrit (https://gerrit.wikimedia.org/r/#/c/279924/).

Question: are https://gerrit.wikimedia.org/r/#/c/260681/ and https://gerrit.wikimedia.org/r/#/c/281113/ competing or complimentary solutions?

@Nemo_bis: Unless you're actually going to work on this bug (or are managing people that are), please do not set the priority.