The Wayback Machine - https://web.archive.org/web/20160322130858/https://secure.phabricator.com/T1562

Build "Facts", an ETL pipeline and charting application
Open, NormalPublic

DESCRIPTION

Build a real ETL pipeline to support various sorts of stat collection and aggregation. See D3022, D3062.

DETAILS

Blocks
T4171: Building reporting and data systems
T6041: Metrics for Maniphest
Restricted Maniphest Task
T1135: Build ETL-based statistics for Differential
Differential Revisions
Restricted Differential Revision
Restricted Differential Revision
Restricted Differential Revision
Restricted Differential Revision
Restricted Differential Revision
Restricted Differential Revision
Restricted Differential Revision
Restricted Differential Revision
Restricted Differential Revision
epriestley triaged this task as "Normal" priority.Jul 26 2012, 1:20 PM
epriestley claimed this task.
epriestley added subscribers: epriestley, vrana, btrahan.
epriestley added a project: Facts.
epriestley edited this Maniphest Task.Jul 26 2012, 2:20 PM
epriestley edited this Maniphest Task.Jul 26 2012, 2:28 PM
epriestley edited this Maniphest Task.Jul 26 2012, 3:52 PM
epriestley edited this Maniphest Task.Jul 26 2012, 6:19 PM
mbeck added a subscriber: mbeck.Jul 27 2012, 11:35 AM
epriestley edited this Maniphest Task.Jul 27 2012, 7:28 PM
epriestley edited this Maniphest Task.Jul 27 2012, 8:05 PM
epriestley edited this Maniphest Task.Jul 27 2012, 8:17 PM
epriestley edited this Maniphest Task.Jul 30 2012, 1:09 PM
epriestley edited this Maniphest Task.Jul 30 2012, 2:43 PM
epriestley edited this Maniphest Task.Aug 9 2012, 4:50 PM
epriestley edited this Maniphest Task.Jul 30 2013, 7:12 PM
qgil added a subscriber: qgil.Apr 25 2014, 11:12 PM
svemir added a subscriber: svemir.Oct 3 2014, 12:06 AM
mbeck removed a subscriber: mbeck.Oct 3 2014, 12:20 AM
Comment Actions

What is missing in order to consider this task done?

Wikimedia is just a few weeks away from its Phabricator prime-time. We know we will not have proper metrics on Day 1, but we will need them at some point. We might help building missing blocks if the pending tasks are well identified.

Comment Actions

Only a small amount of work has been completed here. We don't have implementable plans for the remaining work yet.

This is mostly infrastructure work for T4171, and both are very low priorities for the upstream. Realistically, you shouldn't expect much support from us on this front for a long time.

qgil added a comment.Oct 20 2014, 2:53 PM
Comment Actions

Just to confirm (and I'm not trying to be cynical!), you are saying that you are not planning to work on this any time soon, and you don't encourage others to try either, right? If this is the case, perhaps the best option for us will be to try to find a shorter term solution poking the Conduit API and creating the metrics outside, as we are doing at http://korma.wmflabs.org

I'd rather invest in the Metrics-in-Phabricator path, but I understand that this might be just too complex for an external contributor today.

Comment Actions

That's right. Even if someone somehow built exactly what we plan to build (which would be effectively impossible since we aren't totally sure yet ourselves and it definitely doesn't exist as a written implementation plan anywhere), we wouldn't have the time to review, integrate, and support it right now, so it would just sit in limbo until we got around to prioritizing these things.

Roughly, the only viable way forward in the upstream is:

  • Some time far in the future, this will become a priority for us.
  • We'll implement a basic system (ETL, charting) and get it to a place where we're satisfied with it from an architecture and scalability point of view. Contributors probably can't help very much with this because we won't have a detailed plan yet and will sort of be exploring approches.
  • Once the basics are solid and we're convinced we've built a working solution to the hard parts of the problem, we will be able to accept contributions for new fact extractors, chart types, charting features, etc. But basically feature expansions of a product which will be fairly well-defined by that point.

As a downstream, some options available to you are:

  • Wait a long time for upstream prioritization.
  • Build outside of Phabricator, using Conduit (possibly not a great fit) or raw MySQL.
  • Build in Phabricator, but don't expect any upstream support (review, integration, guidance) for a long time.
qgil moved this task to Important on the Wikimedia workboard.Dec 4 2014, 12:39 PM
AMonk changed the visibility from "All Users" to "Public (No Login Required)".Apr 11 2015, 8:15 PM
epriestley changed the title from "Build an ETL pipeline" to "Build "Facts", an ETL pipeline and charting application".Apr 26 2015, 3:12 AM
eadler added a subscriber: eadler.Apr 30 2015, 4:14 AM
joshma added a subscriber: joshma.Jul 22 2015, 1:39 AM
kwk added a subscriber: kwk.Aug 5 2015, 7:30 AM
ox added a subscriber: ox.Nov 6 2015, 7:54 PM
eadler added a project: Restricted Project.Jan 9 2016, 12:46 AM
eadler moved this task to Restricted Project Column on the Restricted Project workboard.
eadler moved this task to Restricted Project Column on the Restricted Project workboard.Feb 18 2016, 6:29 PM

Add Comment

Morty Proxy This is a proxified and sanitized view of the page, visit original site.