Home Speakers Sydney 2024 Auckland 2024 🎟️ Tickets

Emily Melhuish

Lead Engineer at Multitudes

Twitter LinkedIn

Emily loves to be involved in a range of things from blowing up circuits to riding horses to rockets. She's an Electrical Engineer by trade but has found herself working in the realms of software and data science.

With a passion for both hardware and software, she hopes to combine the two to make technology smarter and more sustainable. Currently, she is leading engineering at Multitudes, a startup providing engineering metrics that aren't creepy to unlock happier and higher-performing teams.

Building a real-time-ish data pipeline with AWS Lambda, SQS and DynamoDB

It can be easy to get trapped in expensive streaming products when looking at building a real-time data pipeline. As a startup, you don't have big pockets to foot a big bill, nor the time to learn a complicated new platform. Can we achieve a close to real-time pipeline using cheap serverless products? Emily thinks yes!

She'll dive into how she built a data pipeline to process GitHub pull requests with AWS Lambda and SQS that is fast enough for customers plus it is cheap! We will look into the technical details for how to set up a Lambda webhook, an SQS reader and a Lambda that then processes data from SQS and loads it into DynamoDB. This pipeline keeps up with events coming in every microsecond and data updates almost immediately PLUS does not cost an arm and a leg.

When building a data pipeline, you must think about how/when/why the data is used so you can build a platform that is fit for purpose and your budget. Sometimes the simplest solution is best.

Technologies we will cover: AWS Lambda, AWS SQS, AWS Dynamodb, Terraform, Typescript

Octopus Unicorn Moth Owl Dino Anteater Robofax
Organisers Speakers Sydney 2024 Auckland 2024 Code of Conduct Privacy Policy
anz@serverlessdays.io @SLSDaysANZ