Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

Conversation

@syspulse
Copy link

Refactored a little bit docker builds to support:

  1. Spark version (tested on 2.4 and 3.0)
  2. Scala version (tested on 2.11 and 2.12)
  3. Hadoop dependencies (for Spark3)
    To avoid default parameters mistakes SPARK_VERSION and SCALA_VERSION are mandatory now (example shell scripts included)

@jeremyrsmith
Copy link
Contributor

Hi and welcome @syspulse! Appreciate this contribution, but I have to admit some noob-ism when it comes to docker. Can you describe a little bit what the repercussions of your changes are when it comes to the build process, the install process, etc?

@syspulse
Copy link
Author

syspulse commented Oct 9, 2020

I think the repercussion is the docker build: https://github.com/polynote/polynote/blob/master/.github/workflows/dist.yml
At the moment you only export SCALA_VERSION to build image. Because of possible permutations, I suggest to have SPARK_VERSION and Hadoop versions dependencies. When Spark migrates to Scala 2.13 or Dotty this should be easier to add support.
This is how dist.yml run would look like:

echo "${DOCKER_PASSWORD}" | docker login -u ${DOCKER_USERNAME} --password-stdin
export POLYNOTE_VERSION=${GITHUB_REF#refs/tags/}
export SCALA_VERSION="2.11"
docker build -t polynote-base:$POLYNOTE_VERSION-$SCALA_VERSION --build-arg POLYNOTE_VERSION=$POLYNOTE_VERSION --build-arg SCALA_VERSION=$SCALA_VERSION docker/base
docker tag polynote-base:${POLYNOTE_VERSION}-${SCALA_VERSION} polynote/polynote:${POLYNOTE_VERSION}-${SCALA_VERSION}
docker push polynote/polynote:${POLYNOTE_VERSION}-${SCALA_VERSION}

export SPARK_VERSION="2.4.7"
docker build -t polynote:$POLYNOTE_VERSION-scala$SCALA_VERSION-spark$SPARK_VERSION --build-arg SPARK_VERSION=$SPARK_VERSION --build-arg SCALA_VERSION=$SCALA_VERSION --build-arg POLYNOTE_VERSION=$POLYNOTE_VERSION --build-arg HADOOP_VERSION=2.7.7 --build-arg HADOOP_VERSION_MAJOR=2.7 docker/spark
docker tag polynote:$POLYNOTE_VERSION-scala$SCALA_VERSION-spark$SPARK_VERSION polynote/polynote:${POLYNOTE_VERSION}-scala${SCALA_VERSION}-spark$SPARK_VERSION
docker push polynote/polynote:${POLYNOTE_VERSION}-scala${SCALA_VERSION}-spark$SPARK_VERSION
docker tag polynote:$POLYNOTE_VERSION-scala$SCALA_VERSION-spark$SPARK_VERSION polynote/polynote:latest
docker push polynote/polynote:latest

export SCALA_VERSION="2.12"
docker build -t polynote-base:$POLYNOTE_VERSION-$SCALA_VERSION --build-arg POLYNOTE_VERSION=$POLYNOTE_VERSION --build-arg SCALA_VERSION=$SCALA_VERSION docker/base
docker tag polynote-base:${POLYNOTE_VERSION}-${SCALA_VERSION} polynote/polynote:${POLYNOTE_VERSION}-${SCALA_VERSION}
docker push polynote/polynote:${POLYNOTE_VERSION}-${SCALA_VERSION}

export SPARK_VERSION="2.4.7"
docker build -t polynote:$POLYNOTE_VERSION-scala$SCALA_VERSION-spark$SPARK_VERSION --build-arg SPARK_VERSION=$SPARK_VERSION --build-arg SCALA_VERSION=$SCALA_VERSION --build-arg POLYNOTE_VERSION=$POLYNOTE_VERSION --build-arg HADOOP_VERSION=2.7.7 --build-arg HADOOP_VERSION_MAJOR=2.7 docker/spark
docker tag polynote:$POLYNOTE_VERSION-scala$SCALA_VERSION-spark$SPARK_VERSION polynote/polynote:${POLYNOTE_VERSION}-scala${SCALA_VERSION}-spark$SPARK_VERSION
docker push polynote/polynote:${POLYNOTE_VERSION}-scala${SCALA_VERSION}-spark$SPARK_VERSION

export SPARK_VERSION="3.0.1"
docker build -t polynote:$POLYNOTE_VERSION-scala$SCALA_VERSION-spark$SPARK_VERSION --build-arg SPARK_VERSION=$SPARK_VERSION --build-arg SCALA_VERSION=$SCALA_VERSION --build-arg POLYNOTE_VERSION=$POLYNOTE_VERSION --build-arg HADOOP_VERSION=3.2.1 --build-arg HADOOP_VERSION_MAJOR=3.2 docker/spark
docker tag polynote:$POLYNOTE_VERSION-scala$SCALA_VERSION-spark$SPARK_VERSION polynote/polynote:${POLYNOTE_VERSION}-scala${SCALA_VERSION}-spark$SPARK_VERSION
docker push polynote/polynote:${POLYNOTE_VERSION}-scala${SCALA_VERSION}-spark$SPARK_VERSION

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

Morty Proxy This is a proxified and sanitized view of the page, visit original site.