My tool-kit for tiny long-run batch jobs

  • I only want to pay for what I use (scale-to-zero)
  • I don’t have a lot of time available for learning or building (no steep learning curves without substantial time-savings)
  • I don’t have time for maintenance activities (no patching servers, automated scale-up)
  • I’m not a good UI designer or front end engineer (design systems are great)

TL;DR

  • AWS Batch with AWS Elastic Container Service (ECS) and AWS Elastic Container Registry (ECR)
  • Docker with Node.js on AWS EC2
  • AWS S3
  • AWS EventBridge (formerly the “events” part of CloudWatch)
  • Github Actions

The business problem

  • Goodreads doesn’t have an API (its owner, Amazon, shut it down 👿)
  • The library catalogues don’t have public APIs
Screenshot from wcib.apps.cronin.nz

The basic process

Three problems to solve

  • I only want to pay for what I use (scale-to-zero)
  • I don’t have a lot of time available for learning or building (no steep learning curves without substantial time-savings)
  • I don’t have time for maintenance activities (no patching servers, automated scale-up)

The execution environment

The container

FROM node:latest
WORKDIR /usr/src/app
COPY lib lib
COPY populatefirestore.js .
COPY package.json .
COPY firebaseconfig.json .
RUN npm install
ENTRYPOINT [ "node", "populatefirestore.js", "-e"]

The repo

The job execution

Deployment

name: Production build for populatefirestore in Docker container on ECRon:
push:
branches: [ master ]
jobs:
deploy:
name: Deploy
runs-on: ubuntu-latest
steps:
- name: Checkout Repo
uses: actions/checkout@master
- name: Write Firebase credentials from secret
env:
FIREBASE_CREDENTIALS: ${{secrets.FIREBASE_CREDENTIALS}}
run: 'echo "$FIREBASE_CREDENTIALS" > firebaseconfig.json'
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ secrets.AWS_REGION }}
- name: Login to Amazon ECR
id: login-ecr
uses: aws-actions/amazon-ecr-login@v1
- name: Build, tag, and push image to Amazon ECR
env:
ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
ECR_REPOSITORY: <myrepoid>
IMAGE_TAG: latest
run: |
docker build -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG .
docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG

Summary

--

--

--

Technology leader for Xero in Auckland, New Zealand, former start-up founder, father of two, maker of t-shirts and small software products

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

A better NerdTree setup

Changing logging configurations at runtime on Wildfly Server

Behind the scenes: Navigating to a URL on your browser

A Major Milestone!

How to Optimize Your Apache Spark Application with Partitions

Understanding Processwire Templates, Fields and Pages

Inside Extreme Networks SLX architecture

A step-by-step guide to backing up your Roam Research database

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Gareth Cronin

Gareth Cronin

Technology leader for Xero in Auckland, New Zealand, former start-up founder, father of two, maker of t-shirts and small software products

More from Medium

Reducing Initial Server Response Time with CloudFront

How to attach IAM policies to users on AWS as temporary?

AWS Data Exchange for APIs, How to Protect Your APIs, APIs Aren’t Just for Developers

AWS Amplify Pitfalls and Solutions