Orchestration using Step functions

Background tasks are great for workload optimization, but it's hard to build and test complex workflows without some advanced orchestration. We can use Lambda for the orchestration, but the current timeout for Lambda functions is 15 minutes, which makes workflows more complicated and expensive. Also, Lambda functions are not good at storing states and not built to be state machines.

NOTE: Lambda functions are not stateless. Each function can store up to 500MB of data in the /tmp folder. If we run a function multiple times, there's a chance that we'll get the same micro-VM (similar to containers), and we'll be able to access the data we stored in the temporary folder. We can use temporary data for optimization (i.e., cache connection or data), but we can't rely on it to store important data.

Our client wants us to check if images are in the right format before processing, and also save a copy of the image customer uploaded in the archive. However, photos from new phones can be large, and we need to resize the picture to be 1024x1024 pixels or smaller to store it in their archive. We can do that using multiple Lambda functions triggered sequentially. But is there a better way?

If we want orchestration in AWS, we can use AWS Step Functions. Step Functions makes it easier to orchestrate multiple AWS services to accomplish tasks. It allows us to create steps in a process where the output of one step becomes the input for another step, all using a visual workflow editor. It also provides automatic retry handling, triggering and tracking for each workflow step, and ensuring actions executes in the correct order.

Task

Our client wants us to add an orchestration that will check if the image is in the correct format, and if it is, use Amazon Textract to get the text, and create a smaller image for archive in parallel.

Your task is to create an architecture diagram and Step Functions flow that will add the following workflow:

• Get image EXIF data
• Check if the image is in the correct format
• If the image is in the correct format, run Amazon Transcribe, and resize and archive the picture, in parallel
• Store the data to the DynamoDB table

IMPORTANT NOTE: This exercise does not require any coding. Creating a diagram and explaining the architecture is enough. You can try to create a code for this solution as a bonus exercise at the end of the workshop or home.

Hints

...