Set up dev environment

On Mac OS

1. Install Brew

Run:

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Full instructions on https://brew.sh/

2. Install dependencies

Python 3, textract dependencies, pip et virtualenv:

brew install python3, pipenv
pip install -U pip

3. Clone the project

git clone [email protected]:wellcometrust/wsf-web-scraper.git
cd wsf-web-scraper

4. Creating a virtualenv with our dependencies

pipenv install
pipenv shell

brew install libpoppler-cpp-dev, poppler-utils

On Linux (Ubuntu/Debian)

1. Install dependencies

Python 3, textract dependencies, pip et virtualenv:

sudo apt-get install python3 python3-dev gcc python-pip
pip install -U pip

2. Clone the project

git clone [email protected]:wellcometrust/wsf-web-scraper.git
cd wsf-web-scraper

3. Creating a virtualenv with our dependencies

pipenv install
pipenv shell

sudo apt-get install libpoppler-gcc-dev, poppler-utils

More details about Scrapy installation can be found here:

• Scrapy