Skip to content

Commit

Permalink
feat: migrate serverless Ghost functions to DigitalOcean (#366)
Browse files Browse the repository at this point in the history 
* feat: update file structure and deployment scripts, get the add / update function for Ghost basically working

* fix: get updates to work for certain events in the addOrUpdateRecord Ghost function

* feat: get deletePosts Ghost function working when posts are unpublished or deleted

* chore: remove test file

* chore: Update README, remove Serverless Framework instructions, files, folders, and ignores

* fix: wording in README

* chore: fix linting issue
  • Loading branch information
@scissorsneedfoodtoo
scissorsneedfoodtoo authored Nov 5, 2024
1 parent 2efd90a commit 54f3644
Show file tree
Hide file tree
Showing 26 changed files with 2,566 additions and 19,194 deletions.
3 changes: 0 additions & 3 deletions .gitignore
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,6 @@
node_modules
jspm_packages

# serverless directories
.serverless

# optional eslint cache
.eslintcache

Expand Down
3 changes: 0 additions & 3 deletions .npmignore
View file

This file was deleted.

147 changes: 41 additions & 106 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,33 +1,45 @@
# Search Indexer

## DigitalOcean Functions
This is a set of serverless functions triggered by Hashnode or Ghost webhooks to index posts from our publications into Algolia indicies for our search bar. They're built and configured for the [DigitalOcean Functions platform](https://cloud.digitalocean.com/functions) and are deployed using the [doctl](https://docs.digitalocean.com/reference/doctl/) CLI to the fCC Team's DigitalOcean account.

Note: Currently only the English publication on Hashnode is using DigitalOcean Functions. The other publications are using AWS Lambda deployed via the Serverless Framework.
Here are the current Hashnode events / webhooks and their corresponding functions:

This is a pair of functions that are triggered by Hashnode webhooks to index posts from our English publication into Algolia for our search bar. They're built with DigitalOcean Functions and are deployed using the [doctl](https://docs.digitalocean.com/reference/doctl/) CLI to the fCC Team's DigitalOcean account.
| Hashnode Event | Function |
| ---------------------- | --------------------------------- |
| Post published | .../hashnode/add-or-update-record |
| Post deleted | .../hashnode/delete-record |
| Published post updated | .../hashnode/add-or-update-record |

Here are the current Hashnode events / webhooks, and their endpoints:
And here are the current Ghost events / webhooks and their endpoints:

| Hashnode Event | Endpoint |
| ---------------------- | ------------------------ |
| Post published | .../add-or-update-record |
| Post deleted | .../delete-record |
| Published post updated | .../add-or-update-record |
| Ghost Event | Function |
| ---------------------- | ------------------------------ |
| Post published | .../ghost/add-or-update-record |
| Published post updated | .../ghost/add-or-update-record |
| Post unpublished | .../ghost/delete-record |
| Post deleted | .../ghost/delete-record |

And here are the currently configured Algolia indices:
Finally, here are the currently configured Algolia indices for each publication:

| Publication URL | Algolia index |
| ---------------------------------- | ------------- |
| https://www.freecodecamp.org/news/ | news |
| Publication URL | CMS | Algolia index |
| --------------------------------------------- | -------- | ------------- |
| https://www.freecodecamp.org/news/ | Hashnode | news |
| https://chinese.freecodecamp.org/news/ | Ghost | news-zh |
| https://www.freecodecamp.org/espanol/news/ | Ghost | news-es |
| https://www.freecodecamp.org/italian/news/ | Ghost | news-it |
| https://www.freecodecamp.org/japanese/news/ | Ghost | news-ja |
| https://www.freecodecamp.org/korean/news/ | Ghost | news-ko |
| https://www.freecodecamp.org/portuguese/news/ | Ghost | news-pt-br |
| https://www.freecodecamp.org/ukrainian/news/ | Ghost | news-uk |

**Prerequisites**:
## Prerequisites

- [doctl](https://docs.digitalocean.com/reference/doctl/) installed
- An account on DigitalOcean with member-level access to the fCC Team
- Access to the Technical Accounts vault on 1Password
- An [Algolia](https://www.algolia.com/) account access to the pre-configured indices above (see table)

## How to prepare your local machine
## How to Prepare Your Local Machine

1. Clone this repo and run `npm ci` from the root directory to install the necessary packages.
1. Go to 1Password, search for the shared `[.env.*] [dev] [prd] Search Indexer` note.
Expand All @@ -36,102 +48,25 @@ And here are the currently configured Algolia indices:
1. Go to 1Password, search for the shared `[PAT] [Digital Ocean] fcc-dev-doctl-token (serverless indexer)` note, copy the PAT, and paste it into the terminal when prompted.
1. Run `doctl auth switch --context fcc-dev` to switch to the `fcc-dev` context.

## General notes about DigitalOcean Functions
## How to Develop or Update Functions

1. Currently, since we're supporting both AWS Lambda and DigitalOcean Functions, all DO Function related code is in the `do-functions` directory. Later on, we can move the DO Functions to the root directory and update the deployment scripts accordingly.
1. Functions are all within the `packages` directory, and each function has its own directory with an `index.js` and `.include` file. The `.include` file is used to include the necessary dependencies for the function.
1. The `lib` directory contains shared code that is used by all functions. This includes the `utils/helpers.js` file, which contains the Algolia client and other shared functions, the `package.json` file, and `node_modules` directory, which is created during the build process and should be copied to each function directory using the `.include` file.
1. New packages, which are groups of functions, and individual functions, should be added to the `project.yml` file. Each package and function is mapped to directories within the `packages` directory. The `project.yml` file is used by the `doctl` CLI to deploy the functions to DigitalOcean.
1. Functions are all within the `packages/<platform>/<function-name>` directory, and each function has its own directory with an `index.js` and `.include` file. The `.include` file is used to include the necessary dependencies for the function.
1. Each function is housed within a `packages/<platform>/<function-name>` directory, where `<platform>` is either `hashnode` or `ghost`, and `<function-name>` is the name of the function. The logic for each function is in the `index.js` file, and the `.include` file contains the dependencies that should be included in the function's `__deployer__.zip` file during the build process.
1. The `lib` directory contains shared code that is used by all functions. This includes the `utils/helpers.js` file, which contains the Algolia client and other shared functions, the shared `package.json` for all DO functions, and `node_modules` directory, which is created during the build process and should be copied to each function directory using the `.include` file.
1. New packages (groups of functions) and new individual functions should be added to the `project.yml` file. Each package and function is mapped to directories within the `packages` directory. The `project.yml` file is used by the `doctl` CLI to deploy the functions to DigitalOcean.

## How to develop and test updates
## How to Test Updates

1. For testing updates to code or NPm packages, use a personal Hashnode publication with webhooks pointing to the functions in the `dev-search-indexer` namespace, which is pre-configured on DigitalOcean. This way you can test changes without affecting the production Algolia index.
1. To deploy changes to the `dev-search-indexer` namespace, run `npm run do-deploy-dev` from the root directory.
1. Once you're satisfied with the changes, deploy the changes to the production namespace with `npm run do-deploy-prd` from the root directory.

## Serverless Framework

These are a group of Lambda functions that are designed to be triggered by webhooks on our self-hosted [Ghost](https://ghost.org/) instances. These functions are created using the [Serverless](https://www.serverless.com/) framework and are used to index articles from various news sources into a universal search bar.

Here are the current Ghost events / webhooks, and their endpoints:

| Ghost Event | Endpoint |
| ---------------------- | ------------------------- |
| Post published | .../_stage_/add-index |
| Post unpublished | .../_stage_/delete-index |
| Post deleted | .../_stage_/deleted-index |
| Published post updated | .../_stage_/update-index |

The Lambda function on each endpoint receives the blog post from the Ghost webhook and updates the correct [Algolia](https://www.algolia.com/) index based on the origin of the Ghost webhook.

Here are the currently configured origins and Algolia indices:

| Ghost Origin | Algolia index |
| --------------------------------------------- | ------------- |
| https://chinese.freecodecamp.org/news/ | news-zh |
| https://www.freecodecamp.org/espanol/news/ | news-es |
| https://www.freecodecamp.org/italian/news/ | news-it |
| https://www.freecodecamp.org/japanese/news/ | news-ja |
| https://www.freecodecamp.org/korean/news/ | news-ko |
| https://www.freecodecamp.org/portuguese/news/ | news-pt-br |
| https://www.freecodecamp.org/ukrainian/news/ | news-uk |

**Prerequisites**:

- AWS CLI and an AWS account with access to Secrets Manager
- An [Algolia](https://www.algolia.com/) account access to the pre-configured indices above (see table)

## How to prepare your local machine
1. For testing updates to code or new versions of npm packages, use a personal Hashnode publication and the Spanish Dockerized Ghost instance from the [news repo](https://github.com/freecodecamp/news/). Use the tables above to configure the webhooks for each platform to point to the functions in the `dev-search-indexer` namespace, which is pre-configured on DigitalOcean.

1. Create an Algolia account with access to the freeCodeCamp News indices
1. Go to LastPass, search for the shared `[keys] [AWS] [Lambda] - News Indexer` note, and add the `[news-indexer-lambda]` credentials to `~/.aws/credentials`
1. Clone this repo and install the necessary packages with `npm ci`
1. Copy `sample.env.yml` to a new file named `.env.yml`
1. In `.env.yml` set `AWS_PROFILE` to `'news-indexer-lambda'`
1. Start developing
## How to Deploy Changes

## How to develop and test locally

1. Make your changes and run `npm start`
1. Open your local Ghost Integrations dashboard at http://localhost:2368/ghost/#/settings/integrations
1. Click "Add custom integration" to create a new custom integration
1. Check the console output and add the Lambda function URLs (ex: https://localhost:3000/dev/add-index) to the correct Ghost events. Check the Ghost events and endpoints table near the top of this document for more information
1. Publish, update, and delete articles on Ghost locally. You can view your changes on Algolia on the `dev` application in the `news-dev` index
1. If you make changes while the server is running, kill the server with Ctrl / Cmd + C. Then restart the server with `npm start` to test your changes

## How to deploy changes

1. After a PR with changes is accepted and merged, deploy changes with `npm run deploy-dev` or `npm run deploy`

## How to change Algolia API keys on AWS

### Generate a new key on Algolia

1. Sign into Algolia with the [email protected] account
1. Select the `dev` or `Production` application from the dropdown at the top of the page
1. Click "API Keys", then click the "All API Keys" tab near the top of the screen
1. Click the "New API Key" button, and generate a new key with the following ACLs (action control lists): `search`, `addObject`, `deleteObject`, `browse`, `listIndexes`, `deleteIndex`
1. Click the "Create" button
1. Copy the new Algolia API key

### Update the Algolia key entry in AWS Secrets Manager

1. Create an [IAM](https://aws.amazon.com/iam/) account under the freeCodeCamp AWS team with access to [Secrets Manager](https://aws.amazon.com/secrets-manager/)
1. Sign in and go to https://us-east-2.console.aws.amazon.com/secretsmanager/
1. Click "serverless-indexer"
1. Click "Retrieve secret value" and click the "Edit" button
1. Paste the new Algolia API key in as the value of `ALGOLIA_ADMIN_KEY` in either the `dev` or `prod` object
1. Click the "Save" button

## How to remove a service

**WARNING**: Removing a `dev` or `prod` service is only necessary in a few cases, such as migrating to a new region.

By removing a service, the base URL for the Lambda functions will be lost. This means that all the webhooks in each Ghost instance will need to be updated once the service is redeployed.
1. To deploy changes to the `dev-search-indexer` namespace, run `npm run do-deploy-dev` from the root directory.
1. Once you're satisfied with the changes, deploy the changes to the production namespace with `npm run do-deploy-prd` from the root directory.

To deploy updates, it's not necessary to remove a service first. Just follow the deployment instructions [here](#how-to-deploy-changes).
## How to Destroy Functions or Namespaces

Use the following commands with caution:
**WARNING**: Destroying `dev` or `prd` functions is only necessary in a few cases, such as renaming an existing function. While destroying a function should still preserve the base URL for the function, it may cause the indicies for one or more publications to get out of sync. Destroying a namespace will remove all functions within that namespace, and change the base URL, requiring all webhooks all publications to be updated.

1. Install Serverless globally with `npm install -g serverless`
1. Run `serverless remove --stage [dev / prod] --region us-east-2`
1. To remove a function, manually remove the function from the `project.yml` file and remove the function's directory from the `packages` directory. Next, go to the [DigitalOcean Functions dashboard](https://cloud.digitalocean.com/functions), select either the `dev-search-indexer` or `prd-search-indexer` namespace, and destroy the function from the dashboard.
1. To remove a namespace, go to the [DigitalOcean Functions dashboard](https://cloud.digitalocean.com/functions) and destroy either the `dev-search-indexer` or `prd-search-indexer` namespace. This will remove all functions within the namespace and change the base URL for the functions. This should only need to be done when migrating to a new region or if the namespace is no longer needed.
115 changes: 0 additions & 115 deletions handler.js
View file

This file was deleted.

80 changes: 0 additions & 80 deletions helper-functions.js
View file

This file was deleted.

0 do-functions/lib/package-lock.json → lib/package-lock.json
View file
File renamed without changes.
Loading

0 comments on commit 54f3644

Please sign in to comment.