Serverless streaming patterns

Serverless applications are almost always event-driven. Event-driven apps are not new, and we built many of them using more traditional non-serverless architectures.

However, the biggest game-changer with serverless architecture is the pricing model where we pay only for used capacity. Serverless functions are cheap when people use them correctly. Instead of waiting for async tasks to finish in our serverless functions, we want to move these tasks to some background processes that will trigger one or multiple functions when they need data processing.

In AWS, we can use the following services to stream and process data in the background, without keeping our users waiting:

1. Amazon Simple Queue Service (SQS) is a fully managed message queuing service.
2. Amazon Simple Notification Service is a highly available, durable, secure, fully managed pub/sub messaging service that enables us to decouple microservices, distributed systems, and serverless applications.
3. Amazon EventBridge is a serverless event bus that makes it easy to connect applications using data from our applications, integrated SaaS apps, and AWS services.
4. Amazon Kinesis makes it easy to collect, process, and analyze real-time, streaming data so we can get timely insights and react quickly to new information.

Streaming services, message buses, and queues are excellent for background processing. But as serverless functions charge us per execution duration, these services can help us speed up processing large data sets faster by chunking data and using multiple tasks to handle it. In serverless, the cost for one function that runs for 1 minute is the same as the cost for 60 functions that run for 1 second, or 600 functions that run for 100 ms.

Streaming services, message buses, and queues are also helpful when we are building hybrid apps. As already mentioned, serverless functions scale fast, but most of our databases are not able to follow all the traffic peaks. For RDS databases on AWS infrastructure, we can use Amazon RDS proxy, a fully managed, highly available database proxy that makes our apps more scalable, more resilient to database failures, and more secure. But what if our database is hosted with another cloud vendor or on-premise?

Task

Our client loves our receipt app, but they have an on-premise database. They have a similar app that helps customers submit their expenses. The existing app works fine, but they have a high traffic peak near the end of January when their customers fill their taxes. They would love to add our receipt processing app that will allow their customers to upload a large number of receipt photos, process them fast, and store them in their on-premise database, but they are afraid that the high load will crash the database.

The following diagram represents our app connected to their existing system.

// ADD DIAGRAM

You have two following tasks:

1. Create an architecture diagram that will extend our app to receive hundreds of receipt images in the single request, and process them fast.
2. Create an architecture diagram that will connect our extended app to clients on-premise infrastructure without crashing their database.

IMPORTANT NOTE: This exercise does not require any coding. Creating a diagram and explaining the architecture is enough. You can try to create a code for this solution as a bonus exercise at the end of the workshop or home.

Hints

...