docs: update documentation to account for parquet source by Meghajit · Pull Request #190 · raystack/dagger

Meghajit · Aug 12, 2022

Closes #189

prakharmathur82 · Aug 17, 2022

docs/docs/concepts/architecture.md

this path is broken

@prakharmathur82 But the image is getting rendered in the page. Do you mean you are not able to open the image via the link ?

I meant I can't see here : https://github.com/Meghajit/dagger/blob/feat/dstrm%231009-update-dagger-documentation/docs/docs/concepts/architecture.md

Yeah, images don't get rendered on the markdown page for some reason. It was happening earlier also. You can check the master branch: https://github.com/odpf/dagger/blob/main/docs/docs/concepts/architecture.md

Updated the relative paths
Fixed via commit 2de8360

prakharmathur82 · Aug 17, 2022

docs/docs/concepts/lifecycle.md

Image is getting rendered though

Updated the relative paths
Fixed via commit 2de8360

prakharmathur82 · Aug 17, 2022

docs/docs/intro.md

Image is getting rendered though

Updated the relative paths
Fixed via commit 2de8360

docs/static/img/lifecycle/dagger_lifecycle.drawio

prakharmathur82 · Aug 17, 2022

docs/static/img/system_design/dagger_system_design.drawio

why is source connected to proto handler?
why call it proto handler?
again check for pre processor workflow
same for name of proto handler towards sink

why is source connected to proto handler?

Hmm. I guess the source should only be connected to the deserializer. The deserializer and proto handler work together to parse raw data ( parquet records, kafka records) into Row. Will fix this

why call it proto handler?

Yes, I should have called it Type Handler. It was a mistake during copy paste from the earlier diagram. I will edit the name.

again check for pre processor workflow

Will do this.

same for name of proto handler towards sink

Will change it to Type Handler

Updated the diagrams
Fixed via commit 7242b10

docs/docs/concepts/architecture.md

[raystack#189]

prakharmathur82 · Aug 22, 2022

README.md

how about real-time streaming and historical data?

@prakharmathur82 We are able to process both real time + historical batched data ( parquet files) as a stream. Hence, put the streaming keyword after. Does the below look ok ?

Dagger or Data Aggregator is an easy-to-use, configuration over code, cloud-native framework built on top of Apache Flink for stateful processing of streaming data, both real time and historical.

We can just say "stateful processing of data"

we can say stateful processing of data

Cool 👍
will do the change

Updated via commit ab4a272

README.md

prakharmathur82 · Aug 22, 2022

docs/docs/concepts/architecture.md

same as above

Commented above for the same

Fixed via commit f7bb8b0

prakharmathur82 · Aug 22, 2022

docs/docs/guides/create_dagger.md

link is broken

Fixed via commit 5accebb

docs/docs/guides/choose_source.md

prakharmathur82 · Aug 22, 2022

docs/docs/guides/overview.md

link is broken

Seems like the overview page links are broken even in master branch.

Fixed via commit 49eca41

[raystack#189]

ravisuhag · Aug 23, 2022

@Meghajit can you cross check if docs build is passing by running yarn build in /docs folder.

[raystack#189]

Meghajit · Aug 23, 2022

@Meghajit can you cross check if docs build is passing by running yarn build in /docs folder.

@ravisuhag Yes, it passed

docs: create new documentation images

f712d56

[raystack#189]

Meghajit self-assigned this Aug 12, 2022

Meghajit changed the title ~~docs: create new documentation images~~ docs: update documentation to account for parquet source Aug 12, 2022

Meghajit marked this pull request as ready for review August 16, 2022 11:19

Meghajit requested a review from prakharmathur82 August 16, 2022 11:19

Meghajit added the documentation Improvements or additions to documentation label Aug 16, 2022

prakharmathur82 requested changes Aug 17, 2022

View reviewed changes

Meghajit added 5 commits August 19, 2022 16:15

docs: update design diagrams

7242b10

[raystack#189]

docs: fixed the relative paths

2de8360

[raystack#189]

docs: updated the descriptions to account for parquet

eb9cc0b

[raystack#189]

docs: rename proto handler to type handler

b21852b

[raystack#189]

docs: create a page for source

4fe1e05

[raystack#189]

prakharmathur82 requested changes Aug 22, 2022

View reviewed changes

Meghajit added 2 commits August 23, 2022 10:40

docs: fix broken links in the Overview page

49eca41

[raystack#189]

docs: fix broken links in the create dagger page

5accebb

[raystack#189]

Meghajit added 4 commits August 23, 2022 11:08

docs: give proper nested headings in configuration page

c471073

[raystack#189]

docs: link configuration in choose_source page

1b40318

[raystack#189]

docs: update description of dagger

ab4a272

[raystack#189]

docs: update description in architecture page

f7bb8b0

[raystack#189]

prakharmathur82 approved these changes Aug 23, 2022

View reviewed changes

prakharmathur82 merged commit 6133f7f into raystack:main Aug 23, 2022

Search code, repositories, users, issues, pull requests...

Conversation

Meghajit commented Aug 12, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Meghajit Aug 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Meghajit Aug 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Meghajit Aug 23, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Meghajit Aug 23, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ravisuhag commented Aug 23, 2022

Uh oh!

Meghajit commented Aug 23, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Meghajit Aug 18, 2022 •

edited

Loading

Meghajit Aug 18, 2022 •

edited

Loading

Meghajit Aug 23, 2022 •

edited

Loading

Meghajit Aug 23, 2022 •

edited

Loading