flight-rules.qmd

---
title: "Examples and Flight Rules"
subtitle: "aka: what to do when something's not working"
toc-depth: 4
---

<!---

##### Issue Report Template

Replace {pkg} with the first letter of the package the issue occurs in and
{num} with the issue number

###### {pkg}{num} Situation {.unnumbered}

 - Issue: [carpentries/{pkg}#{num}](https://github.com/carpentries/{pkg}/issues/num)
 - Resolver: zkamvar
 - Type: Local
 - OS: Rocky Linux 8.8
 - Package: {pkg}

###### {pkg}{num} Diagnosis {.unnumbered}

###### {pkg}{num} Solution {.unnumbered}

###### {pkg}{num} Alternative Solution {.unnumbered}

###### {pkg}{num} Narrative {.unnumbered}

--->

This part of the developers manual is a living document and serves as a window
into the mind of a Workbench Maintainer as they address issues and new feature
requests.

The following sections of this appendix will not distinguish between bugs and
features, because often the process for fixing bugs is not sufficiently
different than that of adding a new feature (though there are exceptions, which
I will highlight below). Rather, a more logical division is to distinguish
between if the issue is within The Workbench Packages, In an Upstream
dependency, with GitHub Actions, and finally, issues that are more structural
in nature and require more extensive user experience (UX) testing.

## Within The Workbench R Packages {#sec-within}

Bugs or features in this category are entirely within our control and are
theoretically the easiest/most quick implementations. Items in this category
can be split into either single package items which can be fixed with a single
pull request or cross-package items, which require coordination of pull requests
to achieve.

### Single Package {#sec-single}

In this section, we outline issues that are addressed within a single package.
Note that this does not indicate that these issues are straightforward to 
address.

#### Single-Function Issues

If you are here, you have determined that the bug or feature that you are
working on will affect a single function or data pathway. These issues are often
the most straightforward to address. Below, I've documented narratives for these
issues. 

##### Markdown file for 404 page created with read-only permissions {.unnumbered}


###### s479 Situation {.unnumbered}

 - Issue: [carpentries/sandpaper#479](https://github.com/carpentries/sandpaper/issues/479)
 - Resolver: zkamvar
 - Type: Local
 - OS: Rocky Linux 8.8
 - Package: sandpaper

The user is attempting to build a lesson, but they are unable to because an
error appears during the "Create 404 page" step:

```r
── Creating 404 page ───────────────────────────────────────────────────────────────────────────────────────────────────
Error in file(file, ifelse(append, "a", "w")) : 
  cannot open the connection
In addition: Warning messages:
[snip]
7: In file(file, ifelse(append, "a", "w")) :
  cannot open file '/tmp/RtmpmaiZ5B/file73cdc48b90540.md': Permission denied
```

###### s479 Diagnosis {.unnumbered}


This permissions issue was a problem with copying a read-only file without
adjusting the subsequent permissions. It stemmed from
[`build_404()`](https://github.com/carpentries/sandpaper/blob/cde890790d509c5c92c539d0adf69a6672354094/R/build_404.R#L12-L13)
calling
[`render_html()`](https://github.com/carpentries/sandpaper/blob/055824fbf643bd7e363b56228deffc98632aaeba/R/render_html.R#L48-L56)
with the presence of a `links.md` document at the top of the lesson. Packages in
the user's R library are installed by the systems administrator and the user
does not have permissions to write or append any file in that folder. When 
`build_404()` runs, it passes a template markdown file to `render_html()`. If
there is an option called `sandpaper.links` set to a filepath, then this
function will copy the input file to a temporary file and append the links to
that temporary file before rendering it to HTML. Because the input file was
read-only, copying the file to the temporary directory retained its permissions
and we were unable to append the links.

###### s479 Solution {.unnumbered}

Pull Request: [carpentries/sandpaper#482](https://github.com/carpentries/sandpaper/pull/482)

The solution was to add a single line adding user write permissions before the
links file was appended: `fs::file_chmod(tmpin, "u+w")`.

###### s479 Alternative Solution {.unnumbered}

This additionally could have been avoided by temporarily unsetting the
`sandpaper.links` option for the `build_404()` function before it calls
`render_html()`. This would prevent it entering the loop where it appends the
links, making the process _slightly_ faster. The only downside is that we would
need to do that for all of the other templated pages (though I believe that
might be the only one). 

###### s479 Narrative {.unnumbered}

 - Issue: [carpentries/sandpaper#479](https://github.com/carpentries/sandpaper/issues/479)
  - First impression: this is running on a version of Linux that Zhian does not
    know anything about. My thoughts are that it will be a difficult fix.
  - I suspect the bug might be due to the {fs} package and ask for the user if
    they could test out the following code snippet:
    
    ```r
    tmp <- fs::file_temp(ext = ".md")
    cat("test\n", file = tmp, append = TRUE)
    readLines(tmp)
    ```
  - I looked again did not recognise the code snippet where the issue seemed to be (`file(file, ifelse(append, 'a', 'w'))`) - so this seems to be a problem with a non-Workbench package.
  - initially went looking for this code in the `fs` package, but GitHub search showed that this code doesn't appear there either.
  - next guess is the `cat` function, which is used to add link references to
    the end of files. I searched the R code base and [found the snippet in the
    `cat()`function](https://github.com/wch/r-source/blob/cd31f24ae0046784465e0fc938d53a2cd8a7d3a4/src/library/base/R/cat.R#L28)
  - that seemed to be where that code snippet was coming from, but the problem
    really originated a few lines above the call to cat: when the template for
    the 404 page (which is saved where the package was installed) is
    copied---`file_copy()` copies a file and _all of its permissions_---so the
    copy is read-only for non-admin users.
  - I opened a PR to test for the error, then applied a fix to prevent it from
    being thrown again.
  - Asked the reporter to install the patch on their system and report back on
    whether it worked.
  - They reported back that it did---and pointed out a typo!
  - After merging PR, create a new release (see process in [The Release Workflow](releases.html))

#### Multi-Function Issues


#### Test Failures With No User Impact


### Aross Packages {#sec-across}

Some features require manipulation across packages. Most often, this will mean
a feature that is needed between {sandpaper} (user interface) and {varnish}
(templates). To a rarer extent would you need features bewteen {sandpaper} and
{pegboard}. You would almost never need a feature between {pegboard} and
{varnish}. 

#### Order of Operations

Features across packages require a bit of extra care because you are dealing
with two moving targets. Remember that it's not impossible to change features
spanning packages in a backwards-compatible manner. If you are worried that
something will break, I will remind you that not only do we have unit tests, 
but we also have [The Workbench integration
tests](https://github.com/carpentries/workbench-integration-test) that allows us
to test development versions of The Workbench across a variety of lessons. On
top of that, you have [the ability to set the versions of The
Workbench](https://carpentries.github.io/sandpaper/reference/set_config.html#custom-engines)
used to build any given lesson. With these three tools, you should be able to
address any issue that arises from a multi-package feature/issue.

This section talks about the order of operations in testing and release, which
can be boiled down to these general rules:

 1. **Always test {sandpaper} against the development version** of the other
    package
 2. **Release the dependent package first.** Unless you have [made breaking
    changes](https://carpentries.github.io/varnish/news/index.html#varnish-100-2023-12-13),
    {varnish} and {pegboard} should be released _before_ {sandpaper}.

With that, here are scattered details of the order of operations for developing
features for two packages.

Make sure to use the `Remotes` field in the {sandpaper} DESCRIPTION file to
ensure that you are testing against the correct version of whatever other
package you are using. That is, if you are testing `feature-a` in {sandpaper}
and there is a companion `feature-a` branch in {varnish}, you want to specify
`Remotes: carpentries/varnish@feature-a` in the `DESCRIPTION` file for
{sandpaper}. This will not only allow the unit tests to work against the correct
version of the dependent package, but it will also allow the integration tests
and user tests to work.

In general, when you are creating a feature across two packages you will want to
ensure the following:

 - [ ] there is an issue open in the workbench repository
 - [ ] the branch names for each feature are identical (or nearly so)
 - [ ] the pull request for {sandpaper} sets the `Remotes`
 - [ ] the dependent package is released _before_ sandpaper.

#### Examples

##### Adding DOI badge for peer reviewed lessons {.unnumbered}

Replace {pkg} with the first letter of the package the issue occurs in and
{num} with the issue number

###### s535v97 Situation {.unnumbered}

 - Issue: [carpentries/sandpaper#535](https://github.com/carpentries/sandpaper/pull/525), [carpentries/varnish#97](https://github.com/carpentries/varnish/pull/97)
 - Resolver: tobyhodges, zkamvar
 - Type: Local
 - OS: NA
 - Package: {sandpaper}, {varnish}

###### s535v97 Diagnosis {.unnumbered}

In [carpentries/workbench#67](https://github.com/carpentries/workbench/issues/67) and [carpentries/varnish#40](https://github.com/carpentries/varnish/issues/40), [Toby Hodges](https://github.com/tobyhodges) had pointed out that while we had badges for lessons in pre-alpha, alpha, and beta phases of development, we did not yet have a badge for a published lesson with a DOI. 

In the former infrastructure, we had indicated these statuses by adding a banner
to the top of the lesson. This banner would take up extra room at the top of the
lesson, so we changed it to a badge since we had room next to the logo. In this
case, Toby was asking if we could add a `doi` key to the `config.yaml` that
would allow the badge to be displayed on the website.


###### s535v97 Solution {.unnumbered}

Toby had opened two pull requests on 2023-10-30:

 - [carpentries/sandpaper#535](https://github.com/carpentries/sandpaper/pull/535)
   where he modified the `inst/templates/pkgdown-yaml-template.txt` and
   `create_pkgdown_yaml()` to include a `doi` element.
 - [carpentries/varnish#97](https://github.com/carpentries/varnish/pull/97)
   where he added HTML to the header that would insert a stylized DOI badge

On 2023-11-10, Zhian Kamvar [added a catch for full URL of a DOI vs the raw DOI
itself](https://github.com/carpentries/sandpaper/pull/535/commits/a8579c78349522cb6fbd484c02d16325d1d34bcc) in {sandpaper} and determined that the contribution to {varnish} needed a couple of extra tweaks before being merged.

{varnish} was merged at 11:43AM and {sandpaper} was merged at 11:47AM

###### s535v97 Alternative Solution {.unnumbered}

One of the tools that Toby had used was passing data through
`create_pkgdown_yaml()`. It could have been done without that function and just
passed through `initialize_metadata()` or something similar. For details on how
data flows from {sandpaper} to {varnish}, read [Data Flow from Source to
Website](https://carpentries.github.io/sandpaper/articles/data-flow.html) in the
{sandpaper} documentation.

It's important to understand _why_ Zhian used `create_pkgdown_yaml()` in the
first place: to get {pkgdown} to recognise the `site/` folder as a package so he
could hijack {pkgdown}'s machinery to provision a website. When he first
started, he did not realize that he could pass any number of data elements
directly to the {varnish} templates (as is done through the global variable
caches), so everything was written to `site/_pkgdown.yaml`. This was cumbersome
because it involves the IO for the pkgdown template file, the config file, and
the resulting pkgdown yaml. It was much easier to load the config file via
`initialize_metadata()` and use that to carry the configuration items. 


## Upstream R Packages {#sec-upstream}

### renv {#sec-renv}

#### Background

The {renv} package is a key player for allowing {sandpaper} to provision and
maintain the R packages required to build R-based lessons. The motivation and
strategy for how it works can be found in the [Building Lessons With A Package
Cache](https://carpentries.github.io/sandpaper/articles/building-with-renv.html)
article in the {sandpaper} docuementation.

It's worth diving into [carpentries/sandpaper#21](https://github.com/carpentries/sandpaper/issues/21)
to see the discussion and thoughts around the origin of the design for using
this feature. It was implemented during a three week period between 2021-08-24
and 2021-09-16, as detailed in the pull request [carpentries/sandpaper#158](https://github.com/carpentries/sandpaper/pull/158).

#### In practices

We have to consider {renv} in practice from the standpoint of both local
computers and on GitHub, which can behave very differently and require different
tools to address their tasks.

There are three tools and packages that use {renv}:

1. {sandpaper} is designed to provide a way to manage dependencies in a lesson
2. [{vise}](https://carpentries.github.io/vise) was originally intended as a
   project to split out {sandpaper} code that used {renv} to simplify the
   testing. At the moment, it provides utilities for automatically provisioning
   C libraries on Ubunutu Linux and running the equivalent of
   `sandpaper::update_cache()` in a GitHub Actions context. 
3. [carpentries/actions](https://github.com/carpentries/actions) these contain
   R code within YAML files 😱 that will call {renv} and {vise}. 

#### Debugging Tips

##### Setting up a reproducible environment

Browse the [{renv} issues opened by \@zkamvar](https://github.com/rstudio/renv/issues?q=is%3Aissue+author%3Azkamvar+is%3Aclosed+sort%3Acreated-asc). In nearly all of these issues, I provide a reproducible example. They generally follow the pattern of:

1. create a temporary file
2. make it a directory and move there
3. add any files that are needed _before_ the renv project is set up (if
   specific to problem)
4. set up a {renv} project with `renv::init()`
5. demonstrate problem


```r
tmp <- tempfile()
dir.create(tmp)
setwd(tmp)
writeLines("library(R6)", "test.R")
renv::init(profile = "test")
# demonstrate problem here
```

##### Choosing a minimal example

Presenting a failing CI run with a Workbench lesson _is_ reproducible, but it's
not minimal. Presenting the same with a smaller lesson is still not minimal.
Often times, the issue involves detecting and installing a new package. In this
case, you want to choose a package that has few to no dependencies and _is not
listed as a dependency for knitr_. One example that I use often is the 
[{cowsay}](https://cran.r-project.org/package=cowsay) package. It has a total of
three dependencies and is not depended on by anything. If I need a quick
package with zero dependencies, I will reach for [{R6}](https://r6.r-lib.org)
or [{aweek}](https://cran.r-project.org/package=aweek). Both of these packages
are under 100 Kilobytes, are pure R code, do not have any dependencies, and do
not require compillation.

##### When you just can't reproduce it locally

There have been times when {renv} seems to fail _only_ on GitHub Actions. This
was the case for {renv} version 0.17.0, as I reported in
[rstudio/renv#1161](https://github.com/rstudio/renv/issues/1161).

The important thing in these situations is to stay calm and try to narrow down
as much as possible the exact conditions that will create the problem. Once you
have those, you can open an issue on the {renv} issue tracker. If you are at
this point, it's important to not expect this to be resolved quickly, because it
is likely out of your control. The best you can do is to try the debugging
techniques that Kevin provides and report back on the issue thread.

Once the issue is resolved: **Thank Kevin for his help.** This is a very
important point. Maintainers often only hear from their users if somehting is
going wrong, so it's important to let them know that they are appreciated.

## GitHub Actions {#sec-actions}

There is a category of issues that fail explicitly with GitHub actions only. As
with the other issues, it is important not to panic when you encounter these,
but instead, take stock of what happens when these happen. In general, here are
the steps you should take to diagnosing problems:

0. Prepare your inbox to filter messages from GitHub. A good way is to check
   that the subject line matches "Run failed 01 Build and Deploy" or "Run failed
   03 Update Cache".
1. Find and read the error message from the log. You can find this by looking
   for the red X and expanding that section. It will normally scroll down to the
   error. 
2. Read the _context_ of the error message. Most errors happen because of
   problems upstream, so it's important to see what was happening when that
   error happened
3. Restart the action. GitHub has a button near the top right of the action. If
   it's ephemeral (e.g. networking error) then it will run fine on the next run.
4. Test the build locally
5. Test it on the `carpentries/sandpaper-docs` repository
6. Test it on a brand new repository created via the templates 
   (<https://bit.ly/new-lesson-md> or <https://bit.ly/new-lesson-rmd>)

Below I will attempt to outline common problem areas and their potential
solutions. Note: this is not an exhaustive list, but you have to remember that
in these situations, you are effectively debugging someone else's computer, so
patience is absolutely required because this is more difficult than plucking a
single grain of rice with metal chopsticks.

### Networking Failures

These are often the easiest problems to solve. You will encounter an error that
says somthing like "network timed out" and it fails when either checking out the
repository or installing pandoc. In these situations, you should check
<https://status.github.com> to see if this is a problem with GitHub, and then
you should restart the build. Most of the time, the build will work on the
second try. 

### Upstream System Dependency Issues

#### SSL error {#sec-bioc-intro109 .unnumbered}

##### bioc-intro109 Situation {.unnumbered}

 - Issue: <https://github.com/carpentries-incubator/bioc-intro/issues/109>
 - Resolver: NA
 - Type: Remote
 - OS: Ubuntu 22.04
 - Package: NA

On 2023-07-20, \@lgatto, who maintains the carpentries-incubator/bioc-intro
repository reported an error in their builds during the "Setup Package Cache" of
the `sandpaper-main.yaml` workflow:

```
Register Repositories
Using github PAT from envvar GITHUB_PAT
Error: Error: Failed to install 'vise' from GitHub:
  SSL peer certificate or SSH remote key was not OK: [api.github.com] SSL: no alternative certificate subject name matches target host name 'api.github.com'
Execution halted
Error: Process completed with exit code 1.
```

This error was causing the [carpentries/vise](https://carpentries.github.io/vise)
package to not be installed and the {renv} package cache could not be
provisioned for lessons. Because it involved the {renv} package cache, this was
a problem that was limited to R-based lessons.

##### bioc-intro109 Diagnosis {.unnumbered}

The cause of the problem was [identified by Gabor Csardi](https://fosstodon.org/@gaborcsardi/110741646942210567) as a bug specific to the development version of `curl`
on Ubuntu: <https://bugs.launchpad.net/ubuntu/+source/curl/+bug/2028170>.

The key error message here was `SSL: no alternative certificate subject name
matches target host name 'api.github.com'`. The `curl` program was checking the
validity of the SSL certificate for `github.com`, but it was ignoring the rules
for `*.github.com`, so when it found `api.github.com`, it saw that as an invalid
site that should not be trusted.

Even though the default version of curl on GitHub's runners is _not_ the dev
version, when we provision the workbench in the "Setup Lesson Engine" step, we
query the system dependencies from the R-universe:

```bash
$ curl https://carpentries.r-universe.dev/stats/sysdeps 2> /dev/null \
  | jq -r '.headers[0] | select(. != null)'
```

```
libcurl4-openssl-dev
libfontconfig-dev
libfreetype-dev
libfribidi-dev
libharfbuzz-dev
libicu-dev
libgit2-dev
libjpeg-turbo8-dev
libpng-dev
libtiff-dev
libxml2-dev
libxslt1-dev
libssl-dev
```

This installed the development version of curl into our system and introduced
the bug during the "Setup Package Cache" step. 

##### bioc-intro109 Solution {.unnumbered}

The solution was to wait for a fix.

##### bioc-intro109 Alternative Solution {.unnumbered}

An alternative solution that we tested was to switch the download method for R
packages to "wget", but this was a non-starter because this caused the
installation times for R packages _and_ the workbench to rise because they would
need to be compiled (posit PPM did not serve the binary packages over wget).

##### bioc-intro109 Narrative {.unnumbered}

 - I get the notification of the issue and \@lgatto suspects it's because of
   {vise}.
 - I install {vise} using `remotes::install_github("carpentries/vise")` to check
   that it's installable and I am successful.
 - I rerun the action and get the same error. I run the action on the
   [carpentries/sandpaper-docs](https://github.com/carpentries/sandpaper-docs/actions/runs/5600450591/job/15170692838)
   repository, but it also fails similarly. This confirms that it's a broader
   issue. 
 - I thought maybe also it was a problem in the {remotes} package since it
   hasn't been updated on CRAN since 2021. 
 - I found <https://github.com/r-lib/remotes/issues/762> which points to an
   issue in `utils::download.file()` as the source of the problems with curl.
 - I notice that {vise} is still recorded as "zkamvar/vise" in the github
   actions files. I change it in [carpentries/actions@2de7e3fef](https://github.com/carpentries/actions/commit/2de7e3fef36e242cb7b6dae348bfdb274b2791fd)
 - This [does not fix the issue](https://github.com/carpentries/sandpaper-docs/actions/runs/5600450591/job/15171251862) and I post to mastodon: <https://fosstodon.org/@zkamvar/110741428815173845>
 - I update the installation for vise to use "wget" as a fallback (see this
   [comparison of commits](https://github.com/carpentries/actions/compare/e9405d9b8f2f99a2a7fd16a31194fd8563af95c9..3fb4f689d3fed641e2644e11ce3b312edc991c70)
 - I create a new branch in carpentries/actions to set
   `options(download.file.method = "wget")` in setup-sandpaper ([view the
   diff](https://github.com/carpentries/actions/compare/2023-07-ssl-errors)).
 - I create https://github.com/zkamvar/2023-07-19-test-actions, which [fails
   initially](https://github.com/zkamvar/2023-07-19-test-actions/actions/runs/5601090073/job/15172762540)
   and replace all `@main` declarations in `.github/workflows/` with
   `@2023-07-ssl-errors` to test the fix that I created. The [next run also
   fails](https://github.com/zkamvar/2023-07-19-test-actions/actions/runs/5601160026/job/15173081859)
   and it takes > 17 minutes to set up the workbench (which normally takes ~1
   minute or less). I understand that the "wget" method will not work.
 - [Gabor Csardi identifies the issue as coming from the dev version of curl](https://fosstodon.org/@gaborcsardi/110741646942210567)
 - I check the version of curl in [the ubuntu runner image](https://github.com/actions/runner-images/blob/ubuntu22/20230710.1/images/linux/Ubuntu2204-Readme.md), which I found by expanding "Setup Job" and then "Runner Image" in the GitHub action run and finding the URL above, but it is not devel.
 - I create <https://github.com/zkamvar/test-github-actions-ssl> to as a MWE of
   the error. 
 - I [create a workflow that runs the command from the ubuntu bug report](https://github.com/zkamvar/test-github-actions-ssl/commit/7b9bd2269c5c81ef7fe0658aaca2ef976cdeab8e) and find that it works.
 - I [update the workflow](https://github.com/zkamvar/test-github-actions-ssl/commit/ee637acccf45cbf41f5b60e963b72a07a28bcca7) so that it demonstrates the working example and then demonstrates the failing example. [It successfully fails](https://github.com/zkamvar/test-github-actions-ssl/actions/runs/5602298664/job/15176571806)
 - I wait and check the bug has been fixed for ubuntu and I rerun the workflow.
   [It succeeds](https://github.com/zkamvar/test-github-actions-ssl/actions/runs/5602298664/job/15185938773) and I re-run the bioc-intro and sandpaper-docs builds to find that they work.

### Workflow Mis-Configuration

Sometimes the workflow files themselves are mis-configured. In these cases, the
fix will involve updating the files in sandpaper's `inst/workflows/` directory.
If the `pr-comment.yaml` or `update-workflows.yaml` files are not affected, then
a pull request will be created automatically to all lessons within a week after
you submit the patch, but if these files are affected, then you will have to
manually submit the patch. 

#### files with spaces in names cause `pr-comment.yaml` workflow to fail {#s399 .unnumbered}

##### s399 Situation {.unnumbered}

 - Issue: [carpentries/sandpaper#399](https://github.com/carpentries/sandpaper/issues/399)
 - Resolver: zkamvar
 - Type: Remote
 - OS: Ubuntu 22.04
 - Package: sandpaper

##### s399 Diagnosis {.unnumbered}

##### s399 Solution {.unnumbered}

##### s399 Alternative Solution {.unnumbered}

##### s399 Narrative {.unnumbered}


### Permissions Changes

#### carpentries-bot

### Actions

## Structural Features {#sec-structural}