-
-
Notifications
You must be signed in to change notification settings - Fork 1
/
testing.qmd
176 lines (133 loc) · 8.1 KB
/
testing.qmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
---
title: "Testing The Workbench"
---
::: {.callout-warning}
### This section is still under construction!
We are still assembling the documentation for this part of the site. If you
would like to contribute, please feel free to open an issue.
:::
## Introduction
> The first stage of your testing journey is to become convinced that testing
> has enough benefits to justify the work. For some of us, this is easy to
> accept. Others must learn the hard way.
>
> --- Wickham and Bryan, [Testing
> Basics](https://r-pkgs.org/testing-basics.html), **R Packages** second edition
> If you use software that lacks automated tests, you are the tests.
>
> --- Jenny Bryan [source tweet (2018-09-22 01:13 UTC)](https://twitter.com/jennybryan/status/1043307291909316609)
Every single package that runs code in the lesson infrastructure is tested
before it ever reaches any lesson. This is important because we want to give the
lesson authors and maintainers as much freedom as they need to write a lesson
while maintaining predictability and integrity. We also want to give our
community confidence that this system works.
Whenever a new feature or bug fix is added to The Workbench, it is imperative
that a test is associated and verified before it gets sent into production.
Tests can be run locally and via continuous integration. This page introduces
some of the testing strategies used in The Workbench and the caveats that come
with these strategies.
## Unit Testing {#sec-unit-tests}
The tests under `test/testthat/` are run in alphabetical order using the
{testthat} package (see https://r-pkgs.org/testing-basics.html) via
`devtools::test()` or `devtools::check()`.
### Conditionally Skipped Tests
The tests often need special conditions in order to run and sometimes those
conditions are not possible. One of the most common conditions to skip is if
the testing happens on CRAN. They are very hawkish about how long test suites
can run and it's often difficult to detect the state of a CRAN machine, so it's
better to skip long-running tests or those with complex environmental
dependencies on CRAN (which does not yet apply to {sandpaper}).
## Continous Integration {#sec-ci}
All the unit tests are run in continuous integration for every push and pull
request that occurs. They also run every week. This provisions the current
releases of the R package dependencies along with development versions of
critical dependencies such as {renv}.
In continous integration, we run on with the following conditions to make sure
it works not only on GitHub, but also on local user machines:
- test coverage (no package structure) with released versions on Ubuntu Linux
(though reporting is stalled)
- For each platform (Ubuntu Linux, macOS, and Windows)
- `R CMD check`, which checks the structure of the package and documentation
- all run on these versions of R: current, devel, and two previous R versions
Because of occasional provisioning failures on macOS and Windows, we require
only that Ubuntu Linux latest version passes check for merging pull requests.
## Lesson Integration Testing {#sec-integration}
Unit tests are great for testing all functionality using known and stable
inputs, it is important to test using known inputs that are in a constant state
of flux: live lessons. This is where [The Workbench Integration
Test](https://github.com/carpentries/workbench-integration-test) comes in. **It
will run a weekly test on a defined set of lessons using the current development
versions of varnish and sandpaper.** These tests are useful for three purposes:
1. ongoing integrity for the workbench lessons lessons that use different
features of The Workbench such as R code execution
2. real-world effects of new sandpaper and varnish versions (including those in
pull requests)
3. inspecting changes in HTML and markdown output
These lessons that we use are
[Instructor Training](https://github.com/carpentries/instructor-training)
: Lesson with the most content, contributors, activity, and used features. This
particular lesson is a bit of a stress test for the infrastructure.
[R for SocialScientists](https://github.com/datacarpentry/r-socialsci)
: This is one of the first R-based lessons to be transitioned and it uses the
tidyverse as a dependency.
[Workbench Documentation](https://github.com/carpentries/sandpaper-docs)
: The workbench documentation. If this doesn't work, nothing will.
[Raster and Vector Geospatial Data with R](https://github.com/datacarpentry/r-raster-vector-geospatial)
: This lesson uses R packages that rely on a geospatial software stack, which
can be complex. Failures here likely mean that there are problems with
provisioning external C libraries.
[BioConductor RNAseq](https://github.com/carpentries-incubator/bioc-rnaseq)
: Lesson using BioConductor R packages by people at BioConductor. If this does
not work, then there likely is a provisioning problem between BioConductor and
{renv}.
### Testing sandpaper and varnish pull requests
To test a pull request version, you can head over the [the main
workflow](https://github.com/carpentries/workbench-integration-test/actions/workflows/main.yml)
and use the button that says "Run Workflow". When you want to test a varnish or
sandpaper pull request, you can use the `[REPO]#[PR]` syntax (e.g.
`carpentries/sandpaper#429` to run sandpaper pull request 429) in the entry
fields for varnish and sandpaper version. If you don't have a pull request to
work from, you can use the `[REPO]@[REF]` syntax (e.g.
`carpentries/sandpaper@test-this-thing` to run sandpaper test-this-thing
branch).
### Inspecting changes
The output for [all the tests are stored in
branches](https://github.com/carpentries/workbench-integration-test/branches/active)
that are named respective for their test repositories. For example,
`datacarpentry/r-socialsci/markdown` and `datacarpentry/r-socialsci/site`
contain the markdown and HTML outputs for the R for Social Scientists lesson. By
inspecting the diffs from the commits, you can see how the output has changed
from run to run, which is useful if you are confirming that a feature will be
automatically deployed.
## Ad-hoc Testing {#sec-ad-hoc}
There are times when you cannot automate your way through testing and you just
have to suck it up and get your (virtual) hands dirty. You find yourself in this
kind of situation if you are testing out GitHub workflows, GitHub actions, or if
you are implementing a new feature and you need to see that it works reliably
and safely. It's this point in time when you use ad-hoc testing on a brand new
lesson repository that you give yourself permission to mess around in and will
delete when finished.
### GitHub Workflows
A GitHub Workflow is a YAML document that lives inside the `.github/workflows`
folder of a repository. This sets up the environment needed to build a lesson.
When you debug these, ask yourself if you really want to update the GitHub
Actions instead. Because these are copied to each lesson, they need to be
updated in each lesson (which is accomplished through the automated pull request
workflow). If you've determined that the workflow needs to be modified, you can
modify them inside your test lesson until you get the desired results. Once that
is done, copy them over to `sandpaper/inst/workflows` and add a NEWS item that's
under the heading `## CONTINUOUS INTEGRATION` describing your change.
### GitHub Actions
A GitHub Action is a single step in a GitHub Workflow and can be written in
nearly any language.
The GitHub actions that are in [carpentries/actions](https://github.com/carpentries/actions)
are written in BASH, node JavaScript, and R and cobbled together with YAML. When
developing a new feature, work on a branch and then, in your shiny new test
lesson, replace the `@main` in the GitHub Workflows to your branch name. This
way, you can know immediately if the fix or feature worked without having to
interrupt someone's flow.
### New config items or settings
If you implement a new config item (e.g. a `lang:` tag), a temporary lesson is a
great way to test it. To do so, you can use the `sandpaper: ` or `varnish: `
keys in your lesson config to specify the version of sandpaper or varnish you
want to test.