Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

feat:add support python udf#172

Merged
prakharmathur82 merged 8 commits intomainraystack/dagger:mainfrom
feat/add-support-python-udfraystack/dagger:feat/add-support-python-udfCopy head branch name to clipboard
Jul 12, 2022
Merged

feat:add support python udf#172
prakharmathur82 merged 8 commits intomainraystack/dagger:mainfrom
feat/add-support-python-udfraystack/dagger:feat/add-support-python-udfCopy head branch name to clipboard

Conversation

@jesrypandawa
Copy link
Contributor

@jesrypandawa jesrypandawa commented Jul 6, 2022

Currently, dagger uses flink version 1.14.3 which is already supported for Python udf. But, dagger still does not implement this feature.

This PR adds the functionality of dagger to also support adding Python udf.

  • Configuration to enable udfs.
  • code changes in dagger to register udfs from GCS.
  • Code abstraction in dagger.
  • Add CI to publish compressed python files into release assets
  • Create a dagger-py-functions directory at odpf/dagger root
  • zip the entire directory including data, requirements.txt, and publish to release assets
  • Actual python udf(in scope: One sample UDF)
  • Documentation on configuration, how to add and use python udf
  • Passed for the test

MayurGubrele and others added 6 commits April 21, 2022 13:17
* chore: publish python zip on release
* refactor: update requirements.txt
* refactor: remove unused publish python package
* docs: add python udf docs

* docs: fix docs config

* docs: update python docs

* docs: add python environment setup
@prakharmathur82 prakharmathur82 merged commit faea8c6 into main Jul 12, 2022
@prakharmathur82 prakharmathur82 deleted the feat/add-support-python-udf branch July 12, 2022 05:45
Meghajit added a commit that referenced this pull request Jul 15, 2022
* feat: dynamic header capability for http post-processor. (#134)

Co-authored-by: mayankrai09 <92782366+mayankrai09@users.noreply.github.com>

* feat: upgrade stencil client version (#141)

* chore: bump up core dependencies version for stencil upgrade (#151)

* feat: add dependencies in build.gradle for parquet source and python udfs (#167)

* doc: rfc for python udf (#129)

* doc: rfc for python udf

* docs: update rfc

Co-authored-by: jesrypandawa <jesry.p.pandawa@gmail.com>

* feat:add support python udf (#172)

* fix: fix docs for udf contribution  (#175)

Co-authored-by: Prakhar Mathur <prakharmathur82@gmail.com>
Co-authored-by: mayankrai09 <92782366+mayankrai09@users.noreply.github.com>
Co-authored-by: Hari krishna <harikrishnakanchi@gmail.com>
Co-authored-by: gauravsinghania <gauravs@sahajsoft.com>
Co-authored-by: jesrypandawa <jesry.p.pandawa@gmail.com>
Co-authored-by: jesrypandawa <78069094+jesrypandawa@users.noreply.github.com>
Meghajit added a commit that referenced this pull request Jul 18, 2022
* feat: dynamic header capability for http post-processor. (#134)

Co-authored-by: mayankrai09 <92782366+mayankrai09@users.noreply.github.com>

* feat: upgrade stencil client version (#141)

* chore: bump up core dependencies version for stencil upgrade (#151)

* feat: add dependencies in build.gradle for parquet source and python udfs (#167)

* doc: rfc for python udf (#129)

* doc: rfc for python udf

* docs: update rfc

Co-authored-by: jesrypandawa <jesry.p.pandawa@gmail.com>

* feat:add support python udf (#172)

* fix: fix docs for udf contribution  (#175)

Co-authored-by: Prakhar Mathur <prakharmathur82@gmail.com>
Co-authored-by: mayankrai09 <92782366+mayankrai09@users.noreply.github.com>
Co-authored-by: Hari krishna <harikrishnakanchi@gmail.com>
Co-authored-by: gauravsinghania <gauravs@sahajsoft.com>
Co-authored-by: jesrypandawa <jesry.p.pandawa@gmail.com>
Co-authored-by: jesrypandawa <78069094+jesrypandawa@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: enhance dagger functions to support python UDF feat: enhance dagger core to be able to add python UDF

3 participants

Comments

Close sidebar
Morty Proxy This is a proxified and sanitized view of the page, visit original site.