This repository contains a set of network function test cases and the framework to build more. It also generates reports (claim.json) on the result of a test run. The tests and framework are intended to test the interaction of Cloud-Native Network Functions (CNFs) with OpenShift Container Platform.

The suite is provided here in part so that CNF Developers can use the suite to test their CNFs readiness for certification. Please see "CNF Developers" below for more information.

A container that can be used to run the tests is available from quay.io

To pull the latest container and run the tests you use the following command. There are several required arguments:

• -t gives the local directory that contains tnf config files set up for the test.
• -o gives the local directory that the test results will be available in once the container exits.
• Finally, list the specs to be run must be specified, space-separated.

Optional arguments are:

• -i gives a name to a custom TNF container image. Supports local images, as well as images from external registries.
• -k gives a path to one or more kubeconfig files to be used by the container to authenticate with the cluster. Paths must be separated by a colon.
• -n gives the network mode of the container. Defaults to bridge. See the docker run --network parameter reference for more information on how to configure network settings.

If -k is not specified, autodiscovery is performed. The autodiscovery first looks for paths in the $KUBECONFIG environment variable on the host system, and if the variable is not set or is empty, the default configuration stored in $HOME/.kube/config is checked.

./run-container.sh -k ~/.kube/config -t ~/tnf/config -o ~/tnf/output diagnostic generic

Note: Tests must be specified after all other arguments!

You can build an image locally by using the command below. Use the value of TNF_VERSION to set a branch, a tag, or a hash of a commit that will be installed into the image.

docker build -t test-network-function:v1.0.5 --build-arg TNF_VERSION=v1.0.5 .

To build an image that installs TNF from an unofficial source (e.g. a fork of the TNF repository), use the TNF_SRC_URL build argument to override the URL to a source repository.

docker build -t test-network-function:v1.0.5 \
  --build-arg TNF_VERSION=v1.0.5 \
  --build-arg TNF_SRC_URL=https://github.com/test-network-function/test-network-function .

To make run-container.sh use the newly built image, specify the custom TNF image using the -i parameter.

./run-container.sh -i test-network-function:v1.0.5 -t ~/tnf/config -o ~/tnf/output diagnostic generic

At a minimum, the following dependencies must be installed prior to running make install-tools.

Other binary dependencies required to run tests can be installed using the following command:

make install-tools

Finally the source dependencies can be installed with

make update-deps

Note: You must also make sure that $GOBIN (default $GOPATH/bin) is on your $PATH.

Note: Efforts to containerize this offering are considered a work in progress.

There are two categories for CNF tests; 'General' and 'CNF-specific'.

The 'General' tests are designed to test any commodity CNF running on OpenShift, and include specifications such as 'Default' network connectivity.

'CNF-specific' tests are designed to test some unique aspects of the CNF under test are behaving correctly. This could include specifications such as issuing a GET request to a web server, or passing traffic through an IPSEC tunnel.

when test on CNFs that run on k8s only envirnment, execute shell command below before compile tool and run test shell script.

export TNF_MINIKUBE_ONLY=true

The general-purpose category covers most tests. It consists of multiple suites that can be run in any combination as is appropriate for the CNF(s) under test:

Further information about the current offering for each test spec is included below.

The diagnostic test spec issues commands to poll environment information which can be appended to the claim file. This information is necessary to ensure a properly spec'd environment is used, and allows the claim to be reproduced. As of today, the diagnostic test suite just polls Node information for all Nodes in the cluster. Future iterations may consider running lshw or similar types of diagnostic tests.

The generic test spec tests:

1. Default network connectivity between containers.
2. That CNF container images are RHEL based.

To test Default network connectivity, a test partner pod is installed on the network. The test partner pod is used as a known point when testing network connectivity of test target pods, pinging to and from each pod under test. The test asserts that the test partner pod receives the correct number of replies, and vice versa.

In the future, other networking protocols aside from ICMPv4 should be tested.

Similar to the generic test spec, the multus test spec is utilized for CNFs that utilize multiple network interfaces. As of today, the multus test suite just tests that a test partner pod can successfully ping the secondary interface of the CNF containers. Since SR-IOV is often utilized, and the secondary interface of a CNF cannot be accessed in user space, the test is unidirectional.

Currently, the operator test spec is limited to three test cases called OPERATOR_STATUS. OPERATOR_STATUS checks that the CSV corresponding to the CNF Operator is properly installed and operator Subscription is available. It checks whether operator is using privileged permissions by inspecting clusterPermissions specified in CSV.

In the future, tests surrounding Operational Lifecycle Management will be added.

The container test spec has the following test cases:

In the future, we are considering additional tests to ensure aspects such as un-alteration of the container image.

Currently, all available tests are part of the "CNF Certification Test Suite" test suite, which serves as the entrypoint to run all test specs. CNF Certification 1.0 is not containerized, and involves pulling, building, then running the tests.

By default, test-network-function emits results to test-network-function/cnf-certification-tests_junit.xml.

The included default configuration is for running generic and multus suites on the trivial example at cnf-certification-test-partner. To configure for your own environment, please see config.md.

In order to pull the code, issue the following command:

mkdir ~/workspace
cd ~/workspace
git clone git@github.com:test-network-function/test-network-function.git
cd test-network-function

In order to build the test executable, first make sure you have satisfied the dependencies.

make build-cnf-tests

Gotcha: The make build* commands run unit tests where appropriate. They do NOT test the CNF.

Once the executable is built, a CNF can be tested by specifying which suites to run using the run-cnf-suites.sh helper script. Any combintation of the suites listed above can be run, e.g.

./run-cnf-suites.sh diagnostic
./run-cnf-suites.sh diagnostic generic
./run-cnf-suites.sh diagnostic generic multus
./run-cnf-suites.sh diagnostic operator
./run-cnf-suites.sh diagnostic generic multus container operator

By default the claim file will be output into the same location as the test executable. The -o argument for run-cnf-suites.sh can be used to provide a new location that the output files will be saved to. For more detailed control over the outputs, see the output of test-network-function.test --help.

cd test-network-function && ./test-network-function.test --help

Gotcha: The generic test suite requires that the CNF has both ping and ip binaries installed. Please add them manually if the CNF under test does not include these. Automated installation of missing dependencies is targetted for a future version.

Detailed configuration of the individual specs is explained in config.md.

We are currently transitioning to using resource labels to automate as much configuration as possible. Automatic configuration will only happen if the environment variable TNF_ENABLE_CONFIG_AUTODISCOVER is set:

TNF_ENABLE_CONFIG_AUTODISCOVER=true

Pods can be labelled at creation by including the label in their definition, or at any time using the oc label command. Annotations are almost identical:

oc label pod test -n tnf test-network-function.com/generic=target
oc annotate pod test -n tnf test-network-function.com/skip_connectivity_tests=true

NOTE Currently when this environment variable is set the generic section of the config file will be ENTIRELY replaced with the autodiscovered configuration, to avoid potentially non-obvious errors.

Autodiscovery is currently only supported in the generic spec, where the following applies:

• The test partner pod is identified by the label test-network-function.com/generic=orchestrator. This must identify a single pod with a single container. This is equivalent to having an entry listed under partnerContainers in the config file.
• Each pod under is identified by the label test-network-function.com/generic=target. There must be at least one such pod. This is equivalent to having a pod listed under containersUnderTest in the config file.
• If an FS Diff Master Pod is present it should be identified with, test-network-function.com/generic=fs_diff_master. This is equivalent to listing the pod under fsDiffMasterContainer in the config file.
• If a pod is not suitable for network connectivity tests because it lacks binaries (e.g. ping), it should be given the label test-network-function.com/skip_connectivity_tests to exclude it from those tests. The label value is not important, only its presence. Equivalent to excludeContainersFromConnectivityTests in the config file.

The autodiscovery mechanism will attempt to identify the default network device and all the IP addresses of the pods it needs, though that information can be explicitly set using annotations if needed. The following rules apply:

• The annotation test-network-function.com/multusips is the highest priority, and must contain a JSON-encoded list of IP addresses to be tested for the pod. This must be explicitly set.
• If the above is not present, the k8s.v1.cni.cncf.io/networks-status annotation is checked and all IPs from it are used. This annotation is automatically managed in OpenShift but may not be present in K8s.
• If neither of the above is present, then only known IPs associated with the pod are used (the pod .status.ips field).

• The annotation test-network-function.com/defaultnetworkinterface is the highest priority, and must contain a JSON-encoded string of the primary network interface for the pod. This must be explicitly set if needed. Examples can be seen in cnf-certification-test-partner
• If the above is not present, the k8s.v1.cni.cncf.io/networks-status annotation is checked and the "interface" from the first entry found with "default"=true is used. This annotation is automatically managed in OpenShift but may not be present in K8s.

The test suite generates a "claim" file, which describes the system(s) under test, the tests that were run, and the outcome of all of the tests. This claim file is the proof of the test run that is evaluated by Red Hat when "certified" status is being considered. For more information about the contents of the claim file please see the schema. You can read more about the purpose of the claim file and CNF Certification in the Guide.

e.g. Adding a cnf platform test results to your existing claim file.

You can use the claim cli tool to append other related test suite results to your existing claim.json file. The output of the tool will be an updated claim file.

go run cmd/tools/cmd/main.go claim-add --claimfile=claim.json --reportdir=/home/$USER/reports

Args:
--claimfile is an existing claim.json file --repordir :path to test results that you want to include.

The tests result files from the given report dir will be appended under the result section of the claim file using file name as the key/value pair. The tool will ignore the test result, if the key name is already present under result section of the claim file.

 "results": {
 "cnf-certification-tests_junit": {
 "testsuite": {
 "-errors": "0",
 "-failures": "2",
 "-name": "CNF Certification Test Suite",
 "-tests": "14",

When run the CNF test suite will output a report to the terminal that is primarily useful for Developers to evaluate and address problems. This output is similar to many testing tools.

Here's an example of a Test pass. It shows the Test running a command to extract the contents of /etc/redhat-release and using a regular expression to match allowed strings. It also prints out the string that matched.:

------------------------------
generic when test(test) is checked for Red Hat version 
  Should report a proper Red Hat version
  /Users/$USER/cnf-cert/test-network-function/test-network-function/generic/suite.go:149
2020/12/15 15:27:49 Sent: "if [ -e /etc/redhat-release ]; then cat /etc/redhat-release; else echo \"Unknown Base Image\"; fi\n"
2020/12/15 15:27:49 Match for RE: "(?m)Red Hat Enterprise Linux Server release (\\d+\\.\\d+) \\(\\w+\\)" found: ["Red Hat Enterprise Linux Server release 7.9 (Maipo)" "7.9"] Buffer: "Red Hat Enterprise Linux Server release 7.9 (Maipo)\n"
•

The following is the output from a Test failure. In this case, the test is checking that a CSV (ClusterServiceVersion) is installed correctly, but does not find it (the operator was not present on the cluster under test):

------------------------------
operator Runs test on operators when under test is: my-etcd/etcdoperator.v0.9.4  
  tests for: CSV_INSTALLED
  /Users/$USER/cnf-cert/test-network-function/test-network-function/operator/suite.go:122
2020/12/15 15:28:19 Sent: "oc get csv etcdoperator.v0.9.4 -n my-etcd -o json | jq -r '.status.phase'\n"

• Failure [10.002 seconds]
operator
/Users/$USER/cnf-cert/test-network-function/test-network-function/operator/suite.go:58
  Runs test on operators
  /Users/$USER/cnf-cert/test-network-function/test-network-function/operator/suite.go:71
    when under test is: my-etcd/etcdoperator.v0.9.4 
    /Users/$USER/cnf-cert/test-network-function/test-network-function/operator/suite.go:121
      tests for: CSV_INSTALLED [It]
      /Users/$USER/cnf-cert/test-network-function/test-network-function/operator/suite.go:122

      Expected
          <int>: 0
      to equal
          <int>: 1

The following is the output from a Test failure. In this case, the test is checking that a Subscription is installed correctly, but does not find it (the operator was not present on the cluster under test):

------------------------------
operator Runs test on operators when under test is: my-etcd/etcd
  tests for: SUBSCRIPTION_INSTALLED
  /Users/$USER/cnf-cert/test-network-function/test-network-function/operator/suite.go:129
2021/04/09 12:37:10 Sent: "oc get subscription etcd -n my-etcd -ojson | jq -r '.spec.name'\n"

• Failure [10.000 seconds]
operator
/Users/$USER/cnf-cert/test-network-function/test-network-function/operator/suite.go:55
  Runs test on operators
  /Users/$USER/cnf-cert/test-network-function/test-network-function/operator/suite.go:68
    when under test is: default/etcdoperator.v0.9.4 
    /Users/$USER/cnf-cert/test-network-function/test-network-function/operator/suite.go:128
      tests for: SUBSCRIPTION_INSTALLED [It]
      /Users/$USER/cnf-cert/test-network-function/test-network-function/operator/suite.go:129

      Expected
          <int>: 0
      to equal
          <int>: 1

The following is the output from a Test failure. In this case, the test is checking clusterPermissions for specific CSV, but does not find it (the operator was not present on the cluster under test):

------------------------------
operator Runs test on operators when under test is: my-etcd/etcdoperator.v0.9.4  
  tests for: CSV_SCC
  /Users/$USER/cnf-cert/test-network-function/test-network-function/operator/suite.go:129
2021/04/20 14:47:52 Sent: "oc get csv etcdoperator.v0.9.4 -n my-etcd -o json | jq -r 'if .spec.install.spec.clusterPermissions == null then null else . end | if . == null then \"EMPTY\" else .spec.install.spec.clusterPermissions[].rules[].resourceNames end'\n"

• Failure [10.001 seconds]
operator
/Users/$USER/cnf-cert/test-network-function/test-network-function/operator/suite.go:55
  Runs test on operators
  /Users/$USER/cnf-cert/test-network-function/test-network-function/operator/suite.go:68
    when under test is: my-etcd/etcdoperator.v0.9.4 
    /Users/$USER/cnf-cert/test-network-function/test-network-function/operator/suite.go:128
      tests for: CSV_SCC [It]
      /Users/$USER/cnf-cert/test-network-function/test-network-function/operator/suite.go:129

      Expected
          <int>: 0
      to equal
          <int>: 1

Developers of CNFs, particularly those targeting CNF Certification with Red Hat on OpenShift, can use this suite to test the interaction of their CNF with OpenShift. If you are interested in CNF Certification please contact Red Hat.

Refer to the rest of the documentation in this file to see how to install and run the tests as well as how to interpret the results.

You will need an OpenShift 4.4 installation running your CNF, and at least one other machine available to host the test suite. The cnf-certification-test-partner repository has a very simple example of this you can model your setup on.

When dealing with large output, you may occasionally overrun the default buffer size. The manifestation of this issue is a json.SyntaxError, and may look similar to the following:

    Expected
        <*json.SyntaxError | 0xc0002bc020>: {
            msg: "unexpected end of JSON input",
            Offset: 660,
        }
    to be nil

In such cases, you will need to set the TNF_DEFAULT_BUFFER_SIZE to a sufficient size (in bytes) to handle the expected output.

For example:

TNF_DEFAULT_BUFFER_SIZE=32768 ./run-cnf-suites.sh diagnostic generic

In some cases, containers do not provide ping or ip binary utilities. Since these binaries are required for the connectivity tests, we must exclude such containers from the connectivity test suite. In order to exclude these containers, please issue add the following to test-network-function/generic_test_configuration.yaml:

excludeContainersFromConnectivityTests:
  - namespace: <namespace>
    podName: <podName>
    containerName: <containerName>

Note: Future work may involve installing missing binary dependencies.