bug 1562412 - iscript: split notarization into 3 tasks

[escapewindow]

Originally #6. We've shifted gears yet again; opening a new issue.

Apple can occasionally take hours to get back to us about notarization status. In the most recent incident, they didn't even acknowledge the UUIDs existed for hours. We need to be able to handle this case during a chemspill. The solution in #6, of multiple monolithic notarization tasks per worker, doesn't account for multi-hour turnarounds.

We should split notarization into 3 tasks. The first and third are straightforward; the second requires a special implementation.

Task number one should download the unsigned dmg, sign it, create the pkg and sign it, and submit both for notarization. It should upload the signed artifacts, as well as a manifest of notarization UUID(s) to poll.

Task number three should download the signed .app and .pkg, and staple notarization to them and upload them for complete mar creation and beetmoving.

The second task should be a semi-dummy task. It should block the third task from running (dependency). We need a service to notice these tasks, download the notarization manifests from the signing tasks, and poll the UUIDs. When all of the UUIDs are good, we should resolve the task, unblocking the third task. However, we don't want a one-to-one mapping between these tasks and the worker or service; rather, we likely want this service to be able to handle hundreds of tasks and UUIDs.

Because the polling requires XCode and the notarization creds, this will likely run from mac hardware, possibly as a second daemon on the notarization boxes. We may need to look at whether polling creates a lockfile, and whether we need to have a semaphore to limit concurrent polling.

Because of the nature of this solution, we'll need:

• iscript patches:
  • to allow for mac_notarize to stop at stapling (and to create a UUID manifest)
  • to allow for mac_staple to download signed artifacts, staple them, and upload
• in-tree patches to create the three tasks
• special code (a special worker implementation?) to deal with the second task
  • we may need shared semaphore logic from the previous patchsets to share notary accounts between the iscript scriptworkers and this worker. Alternately, we could create a second set of accounts that the polling service can use.

[escapewindow]

• Notarization poller wip: bug 1562412 - Notarization poller worker #121
• iscript notarize_{1,3}_behavior wip: add notarize_{1,3}_behaviors #122