Use tar and ssh to publish package metadata files. (#1049)

* Use sftp via shell to publish package metadata files.

The publish-over-ssh plugin adds an extra configuration step that's hard
to manage securely as it does not use the Jenkins ssh-agent and
credential system, nor does it support openssh formatted or ed22519
keys, unlike the Jenkins private key credential.

Although it requires a bit more finagling ourselves, I think this is an
equally valid way to publish this content which only requires one copy
of the ssh private key in the ssh-agent and not an out-of-band system.

I chose sftp over rsync in order to batch the transfers of these smaller
files without doing wildly long rsync commands.

* Avoid use of pushd/popd which aren't available on agents.

* Use find instead of globs to aggregate metadata files.

Using the sftp batch as written had two regressions in behavior:

1. globs that did not have matches resulted in errors

2. file paths were not preserved by sftp so every matching file was put
   directly into the `noetic` directory rather than preserving its path
   relative to noetic.

1 is trivial to fix by prefixing the sftp command with `-` but 2 is
  substantially more challenging, after attempting a few sftp and rsync
  changes, the problem came down to the fact that not every glob will
  always have matching files and when the glob fails to expand, the glob
  is passed to the command literally. Which meant that an rsync or scp
  command would get a glob (that is known not to have any valid
  expansion) and fail because the file with the glob characters in it
  does not exist.

So instead we change modes entirely and use `find` to match regular
expressions that are equivalent to the glob expressions. It seemed to me
that the most convenient way to transmit the found files, preserving
relative paths, was with tar. Which is why the files are tar'd and
piped to ssh to be untar'd at the correct path.

ros_buildfarm/templates/doc/doc_job.xml.em

@@ -250,26 +250,25 @@ else:
         'fi',
     ]),
 ))@
-@(SNIPPET(
-    'builder_publish-over-ssh',
-    config_name='docs',
-    remote_directory=rosdistro_name,
-    source_files=[
-        'generated_documentation/api/**/manifest.yaml',
-        'generated_documentation/api/**/stamp',
-        'generated_documentation/changelogs/**/*.html',
-        'generated_documentation/symbols/*.tag',
-
-        'generated_documentation/deps/*',
-        'generated_documentation/hashes/*',
-        'generated_documentation/locations/*',
-        'generated_documentation/metapackage_deps/*',
-    ],
-    remove_prefix='generated_documentation',
-))@
 @(SNIPPET(
     'builder_shell',
     script='\n'.join([
+        'echo "# BEGIN SECTION: Publish package metadata"',
+        'cd "$WORKSPACE/generated_documentation"',
+        "ssh -T %s@%s 'test -d %s  || mkdir %s'" % \
+          (upload_user, upload_host, os.path.join(upload_root, rosdistro_name), os.path.join(upload_root, rosdistro_name)),
+        'tar -cz $(find -regextype posix-egrep' +
+        " -regex '\./api(/.+)?/manifest\.yaml'" +
+        " -o -regex '\./api(/.+)?/stamp'" +
+        " -o -regex '\./changelogs(/.+)?/[^/]+\.html'" +
+        " -o -regex '\./symbols/[^/]+\.tag'" +
+        " -o -regex '\./deps/[^/]+'" +
+        " -o -regex '\./hashes/[^/]+'" +
+        " -o -regex '\./locations/[^/]+'" +
+        " -o -regex '\./metapackage_deps/[^/]+')" +
+        ' | ssh -T %s@%s tar -C %s -xz' % \
+          (upload_user, upload_host, os.path.join(upload_root, rosdistro_name)),
+        'echo "# END SECTION"',
         'if [ -d "$WORKSPACE/generated_documentation/api_rosdoc" ]; then',
         '  echo "# BEGIN SECTION: rsync API documentation to server"',
         '  cd $WORKSPACE/generated_documentation/api_rosdoc',