My tool-kit for tiny short-run batch jobs

  • I only want to pay for what I use (scale-to-zero)
  • I don’t have a lot of time available for learning or building (no steep learning curves without substantial time-savings)
  • I don’t have time for maintenance activities (no patching servers, automated scale-up)
  • I’m not a good UI designer or front end engineer (design systems are great)

TL;DR

  • Serverless Framework
  • AWS Lambda and Node.js
  • AWS DynamoDB
  • AWS EventBridge (formerly the “events” part of CloudWatch)
  • AWS Simple Email Service with Route 53
  • Github Actions

The business problem

The basic process

  • looks in a DynamoDB table to find the least recently visited site
  • checks the site for changes and emails me if there are any (or if the scrape fails)
  • updates the last visit time in the DynamoDB table

Three problems to solve

  • I only want to pay for what I use (scale-to-zero)
  • I don’t have a lot of time available for learning or building (no steep learning curves without substantial time-savings)
  • I don’t have time for maintenance activities (no patching servers, automated scale-up)

The handler

The database

Serverless Framework

service: watch-recycling-sitesframeworkVersion: ">=1.1.0 <2.0.0"plugins:
- serverless-dynamodb-local
- serverless-offline
custom:
dynamodb:
stages:
- dev
start:
port: 8000
inMemory: true
seed: true
migrate: true
migration:
dir: offlinemigrations
seed:
domain:
sources:
- table: ${self:service}-${opt:stage, self:provider.stage}-v1
sources: [./offlineseeding/sites.json]
provider:
name: aws
runtime: nodejs10.x
stage: dev
region: ap-southeast-2
environment:
DYNAMODB_TABLE: ${self:service}-${opt:stage, self:provider.stage}-v1
iamRoleStatements:
- Effect: Allow
Action:
- dynamodb:Query
- dynamodb:Scan
- dynamodb:GetItem
- dynamodb:PutItem
- dynamodb:UpdateItem
- dynamodb:DeleteItem
- ses:SendEmail
Resource:
- "arn:aws:dynamodb:${opt:region, self:provider.region}:*:table/${self:provider.environment.DYNAMODB_TABLE}"
- "arn:aws:ses:*:*:identity/*"
functions:
diffsite:
handler: handlers/diffsite.handler
timeout: 300
events:
- schedule: rate(4 hours)
resources:
Resources:
TodosDynamoDbTable:
Type: 'AWS::DynamoDB::Table'
DeletionPolicy: Retain
Properties:
AttributeDefinitions:
-
AttributeName: id
AttributeType: S
KeySchema:
-
AttributeName: id
KeyType: HASH
BillingMode: PAY_PER_REQUEST
TableName: ${self:provider.environment.DYNAMODB_TABLE}

Mail out

Deployment

name: Deploy serverlesson:
push:
branches:
- master
jobs:
deploy:
name: deploy
runs-on: ubuntu-latest
strategy:
matrix:
node-version: [14.x]
steps:
- uses: actions/checkout@v2
- name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v1
with:
node-version: ${{ matrix.node-version }}
- run: npm ci
- name: serverless deploy
uses: serverless/github-action@master
with:
args: deploy
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

Summary

--

--

--

Technology leader for Xero in Auckland, New Zealand, former start-up founder, father of two, maker of t-shirts and small software products

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

How to Fix the WordPress Update or Published Failed Error

19coders

Funny Linux Commands

Using MTurk with Zapier

System Design: Low Latency Localization Service

News — WordPress 5.6 Release Candidate — WordPress.org

19coders

The Good, The Bad & The Ugly

8 Ways Web Servers Can Tell Your Web Crawler Apart From Humans

Agile Software Development

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Gareth Cronin

Gareth Cronin

Technology leader for Xero in Auckland, New Zealand, former start-up founder, father of two, maker of t-shirts and small software products

More from Medium

The openness of SaaS solutions

The Growing Importance of Physical Addresses in an Online World

Address Verification Services

AWS re:Invent introduces, CloudWatch RUM

How does Treblle scale on AWS without breaking the bank?