Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings
Discussion options

Hi Folks 👋

I've been experimenting with using Flat Data to import data into Neo4j.

To make this workflow easier I published an Action called Flat Graph that allows for defining an Action step that includes the database connection credentials and a database query (using Cypher, the query language for Neo4j) to run that handles importing the data fetched by the Flat action in the previous step. For example, here's how you would use it to import data from the Lobsters JSON feed:

name: Flat Graph for Neo4j

on:
  push:
    paths:
      - .github/workflows/flat.yml
  workflow_dispatch:
  schedule:
    - cron: '*/60 * * * *'

jobs:
  scheduled:
    runs-on: ubuntu-latest
    steps:
      - name: Check out repo
        uses: actions/checkout@v2
      - name: Setup deno
        uses: denoland/setup-deno@main
        with:
          deno-version: v1.x
      - name: Fetch newest 
        uses: githubocto/flat@v2
        with:
          http_url: https://lobste.rs/newest.json
          downloaded_filename: newest.json
      - name: Neo4j import
        uses: johnymontana/flat-graph@v1.1
        with:
          neo4j-user: ${{secrets.NEO4J_USER}}
          neo4j-password: ${{secrets.NEO4J_PASSWORD}}
          neo4j-uri: ${{secrets.NEO4J_URI}}
          filename: 'newest.json'
          cypher-query: >
              UNWIND $value AS article
              MERGE (u:User {username: article.submitter_user.username})
              MERGE (a:Article {id: article.short_id})
              SET a.url           = article.url,
                  a.id_url        = article.short_id_url,
                  a.created_at    = article.created_at,
                  a.title         = article.title,
                  a.score         = article.score,
                  a.flags         = article.flags,
                  a.comment_count = article.comment_count,
                  a.description   = article.description,
                  a.comments_url  = article.comments_url
              MERGE (u)-[:SUBMITTED]->(a)
              WITH article, a
              UNWIND article.tags AS tag
              MERGE (t:Tag {name: tag})
              MERGE (a)-[:HAS_TAG]->(t)        

To see it in action I wrote a tutorial / blog post showing how to use Flat Data + Neo4j.

Would love to know what folks think of this workflow or any suggestions on how to improve it.

Cheers,
Will

You must be logged in to vote

Replies: 1 comment · 1 reply

Comment options

@johnymontana that's lovely! It's cool to see flat composed into workflows this way.

You mention in the writeup that you could have done the writing to neo4j in a postprocessing step, but you ended up writing the flat-graph action instead. There's nothing at all wrong with that! But I'm curious what pain with the postprocessing led you to write the action instead.

You must be logged in to vote
1 reply
@johnymontana
Comment options

Well I started with the postprocessing script, but it was a lot of boilerplate (create a database driver instance, read the downloaded JSON file, pass it as a query parameter, etc) so I thought it would be a nicer abstraction to be able to just write the database query in the same yaml file when defining the action workflow. So the reason for the separate action was the pain of copy/paste boilerplate setup code in the postprocessing script and also some issues with using the Neo4j driver in Deno (but that's a Neo4j problem).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
2 participants
Morty Proxy This is a proxified and sanitized view of the page, visit original site.