- references
- What's Cooking in ZIO Test by Adam Fraser
- Using Aspects To Transform Your Code With ZIO Environment
- https://zio.dev/reference/observability/logging
- https://zio.dev/reference/test/aspects/
- https://zio.dev/reference/test/property-testing/
- https://www.zionomicon.com
- https://github.com/adamgfraser/0-to-100-with-zio-test
- https://docs.spring.io/spring-framework/docs/4.3.15.RELEASE/spring-framework-reference/html/aop.html
- https://dotty.epfl.ch/docs/reference/new-types/polymorphic-function-types.html
- zio/zio#4601
- https://zio.dev/reference/test/property-testing/built-in-generators/
- Kacper Korban - Scala 3, but I have trust issues
- goals of this workshop
- introduction to
- functional programming aspects
- property based testing
- understanding how to use test aspects in practice
- creating data generators
- introduction to
- workshops
- task1: implement generator of
Accounts- then derive it using
zio-test/magnolia - solution:
AccountGenerators
- then derive it using
- task2: derive generator of
Contributors- then switch it to generate
Contributorfrom filesrc/test/resources/contributors.txt - solution:
ContributorGenerators
- then switch it to generate
- task3: implement and plug aspect to set specific seed (
TestSeed.seed) before each test- solution:
MainSpec
- solution:
- task4: experiment with intentionally failing some tests to see a seed
- try to reproduce problem by setting correct seed in TestSeed
- task1: implement generator of
- lib: caliban
- example
val api = graphQL(???) @@ maxDepth(50) @@ timeout(3 seconds) @@ printSlowQueries(500 millis) @@ apolloTracing @@ apolloCaching - supports aspects (called wrappers) that allow modifying: query parsing, validation and execution
- example
- introduction
- in any domain there are cross-cutting concerns that are shared among different parts of our main program logic
- often these concerns are tangled with each part of our main program logic and scattered across different parts
- we want to increase the modularity of our programs by separating these concerns from our main program logic
- cross-cutting concerns are typically related to how we do something rather than what we are doing
- what level of authorization should this transfer require?
- how should this transfer be logged?
- how should this transfer be recorded to our database
- example: testing
- main program logic: tests
- concerns
- how many times should we run a test?
- what environments should we run the test on?
- what sample size should we use for property based tests?
- what degree of parallelism?
- what timeout to use?
- example: graphql
- main program logic: queries
- concerns
- what is the maximum depth of nested queries we should support
- what is the maximum number of fields we should support
- what timeout should we use?
- how should we handle slow queries?
- what kind of tracing and caching should we use?
- traditional approach: metaprogramming
- example: AspectJ
@Aspect public class BeforeExample { @Before("execution(* com.xyz.myapp.dao.*.*(..))") public void doAccessCheck() { // ... } } - relies on implementation details such as class and method names that may change
- no longer able to statically type check if code is dynamically generated
- example: AspectJ
- functional approach: polymorphic functions
- aspects are polymorphic functions
- polymorphic function: scala3
// A polymorphic method: def foo[A](xs: List[A]): List[A] = xs.reverse // A polymorphic function value: val bar: [A] => List[A] => List[A] // ^^^^^^^^^^^^^^^^^^^^^^^^^ // a polymorphic function type = [A] => (xs: List[A]) => foo[A](xs) - example: zio
trait Aspect[-R, +E] { def apply[R1 <: R, E1 >: E, A](zio: ZIO[R1, E1, A]): ZIO[R1, E1, A] }- potentially constraining the environment or widening the error type
- transforms the how but not the what
- composable
implicit final class AspectSyntax[-R, +E, +A)(private val zio: ZIO[R, E, A]) { def @@[R1 <: R, E1 >: E](aspect: Aspect[R1, E1]): ZIO[R1, E1, A] = aspect(zio) }
- example
test("concurrency test") { ??? } timeout(60.seconds) - seamlessly control how tests are executed
- example
- without aspects
test("foreachPar preserves ordering") { val zio = ZIO.foreach(1 to 100) { _ => ZIO.foreachPar(1 to 100)(ZIO.succeed(_)).map(_ == (1 to 100)) }.map(_.forall(identity)) assert(zio)(isTrue) } } - with aspects
test("foreachPar preserves ordering") { assert(ZIO.foreachPar(1 to 100)(ZIO.succeed(_)))(equalTo(1 to 100)) } } @@ nonFlaky
- without aspects
- example
- common test aspects
- diagnose - do a localized fiber dump if a test times out
- nonFlaky - run a test repeatedly to make sure it is stable
- timed - time a test to identify slow tests
- timeout - time out a test after specified duration
- tag - tag a test for reporting
- example: "this test is about database"
- composable
- test @@ nonFlaky @@ timeout(60.seconds)
- apply to tests, suites or entire specs
- order matters
- repeat(10) @@ timeout(60s)
- timeout(60s) @@ repeat(10)
- implementing aspects
- when we need access to test itself
new TestAspect.PerTest.AtLeastR[TestEnvironment] { override def perTest[R >: Nothing <: TestEnvironment, E >: Nothing <: Any] (test: ZIO[R, TestFailure[E], TestSuccess])(implicit trace: Trace): ZIO[R, TestFailure[E], TestSuccess] = for { result <- test // here comes the logic and we have handle to test itself } yield result } - when we want to do something independent of test itself
- TestAspect.before(zio.Console.printLine("before each"))
- TestAspect.beforeAll(zio.Console.printLine("before all"))
- etc
- when we need access to test itself
- example: ZIO test
test("encode and decode is an identity") { // check operator, one or more generators, assertion check(genEvents) { event => assert(decode(encode(event)))(equalTo(event)) } } - is an approach where the framework generates test cases
- strategies
- hard to prove, easy to verify
- example:
sorting
- example:
- there and back again
- example:
reverse(reverse(list)) == list
- example:
- different paths same destination
- example: inverting binary tree
invertTree(Node(Leaf, 0, t)) == Node(invertTree(t), 0, Leaf)
- example: inverting binary tree
- hard to prove, easy to verify
- advantage: allows to quickly test a large number of test cases
- potentially: reveal not obvious counterexamples
- typically generate ~ 100-200 test cases
- Int ~2 billion values
- complex data types => number of possibilities increases exponentially
- complement property based testing with traditional tests (for particular degenerate cases)
- common mistake: generator is not general enough
- example: generating user input using ASCII
- what about: 普通话 ?
- example: generating user input using ASCII
- generator represents a distribution of potential values
- each time we run a property based test we sample values from that distribution
final case class Gen[-R, +A]( sample: ZStream[R, Nothing, Sample[R, A]] ) final case class Sample[-R, +A]( value: A, shrinks: ZStream[R, Nothing, Sample[R, A]] )
- each time we run a property based test we sample values from that distribution
- create generators
- construct generators for each field
- combine with operators
- example: flatMap, map, oneOf, zip
- recommended: flexible, explicit, composable
- example
val genAccountStatus = Gen.fromIterable(AccountStatus.values) val genNonEmptyString = Gen.stringBounded(1, 10)(Gen.char).map(NonEmptyString.unsafeFrom) val genAccount2: Gen[Any, Account] = (Gen.uuid <*> // symbolic alias for zip and zipWith; generate values in parallel genAccountStatus <*> Gen.string1(Gen.char) ).map { case (uuid, status, str) => Account(AccountId(uuid), status, NonEmptyString.unsafeFrom(str)) } - problem: sealed non-enum traits
- we don't have access to all values
- you need to explicitly enumerate values
- every new case class should be added to generator
- example
sealed trait TransactionParameters case class BitcoinTransactionParameters(...) extends TransactionParameters case class EthereumTransactionParameters(...) extends TransactionParameters case class XrpTransactionParameters(...) extends TransactionParameters val genTransactionParameters: Gen[Any, TransactionParameters] = Gen.oneOf(genBitcoinTxParams, genEthereumTxParams, genXrpTxParams) - easy to forget
- solution: auto-deriving generator
- example
- we don't have access to all values
- auto-deriving generator
- mutual correspondence gen <-> derive
- gen -> derive: DeriveGen.instance(genA)
val genA: Gen[Any, A] = ... val deriveGenA: DeriveGen[A] = DeriveGen.instance(genA) - derive -> gen:
val deriveGenA: DeriveGen[A] = ... val genA: Gen[Any, A] = deriveGenA.derive
- gen -> derive: DeriveGen.instance(genA)
- usually used for sealed non-enum hierarchies
- example
sealed trait TransactionParameters case class BitcoinTransactionParameters(...) extends TransactionParameters case class EthereumTransactionParameters(...) extends TransactionParameters case class XrpTransactionParameters(...) extends TransactionParameters val deriveTransactionParameters: Gen[Any, TransactionParameters] = DeriveGen[TransactionParameters]
- example
- deriving is macro-based
- it is sometimes hard to know which generators will be used
- especially in case of multi-files imports
- we cannot just use
show implicitsIJ option
- it is sometimes hard to know which generators will be used
- it is hard to maintain specific constraints in multi-file imports
- usually we require objects that are correct/valid for our tests
- correct/valid = not complete random
- only one implicit for each type allowed
- usually we require objects that are correct/valid for our tests
- not composable
- solution: unpack the
DeriveGeninstance to get aGen(composable)
- solution: unpack the
- example
- from case classes
val genAccount: Gen[Any, Account] = DeriveGen[Account] // implicit for each field - same file implicits
val genActiveAccountStatus: Gen[Any, AccountStatus] = Gen.fromIterable(AccountStatus.activeStatuses) implicit val deriveActiveAccountStatus: DeriveGen[AccountStatus] = DeriveGen.instance(genActiveAccountStatus) val genActiveAccount: Gen[Any, Account] = DeriveGen[Account] - multi-file implicits
object AccountStatusGenerators { val genActiveAccountStatus: Gen[Any, AccountStatus] = Gen.fromIterable(AccountStatus.activeStatuses) implicit val deriveActiveAccountStatus = DeriveGen.instance(genActiveAccountStatus) }import app.AccountStatusGenerators._ object AccountGenerators { val genActiveAccount: Gen[Any, Account] = DeriveGen[Account] }
- from case classes
- lib: https://zio.dev/api/zio/test/magnolia/index.html
- mutual correspondence gen <-> derive
- don't use filter - transform instead
- filtering = "throw away" data that doesn’t satisfy our predicate
- example
val evens: Gen[Random, Int] = ints.map(n => if (n % 2 == 0) n else n + 1) // transformation
- shrinking
- counterexample will typically not be the "simplest"
- test framework tries to shrink failures to ones that
- are "simpler" (in some sense)
- example: smaller integers, smaller collections
- and still violate the property
- are "simpler" (in some sense)
- test framework tries to shrink failures to ones that
- ZIO Test uses "integrated shrinking"
- every generator already knows how to shrink itself
- all operators keep this property
- example: generator of even integers can’t shrink to 1
- every generator already knows how to shrink itself
- under the hood
- Sample contains a "tree" of possible "shrinkings" for the value
- root: original value
- invariants
- any given level: value earlier in the stream, must be "smaller" than later values
- all children must be "smaller" than their parents
- machinery
- generate the first Sample in the shrink stream
- test whether its value is also a counterexample to the property being tested
- counterexample => recurse on that sample
- not => repeat with the next Sample in shrink stream
- example: shrinking logic for int
- first tries to shrink to zero
- then to half the distance between counterexample and zero
- then to half that distance, and so on
- Sample contains a "tree" of possible "shrinkings" for the value
- counterexample will typically not be the "simplest"
- TestRandom service
- provides a testable implementation of the Random service
- serves as a purely functional random number generator
- implementation takes care of passing the updated seed
- we can set the seed and generate a value based on that seed
- default seed
/** * An arbitrary initial seed for the `TestRandom`. */ val DefaultData: Data = Data(1071905196, 1911589680) - we could set/get seed using: TestRandom.getSeed / TestRandom.setSeed
- default seed