Skip to content

ESM-v2: Add SNS as a failure destination for Kinesis and DynamoDB #4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

lizard-boy
Copy link

Motivation

This PR enables and tests SNS as a failure destination DLQ in Localstack ESM v2 for Kinesis and DynamoDBStreams events, where AWS event source mapping supports adding either an SQS queue or SNS topic as a failure destination.

See snippet AWS::Lambda::EventSourceMapping OnFailure docs:

Changes

  • Allows for an SNS topic ARN to be selected as an OnFailure.Destination for StreamPoller type pollers (Kinesis & DynamoDBStreams)

Testing

To ensure the correct payload was being sent to the SNS DLQ, the tests create an SQS object and subscribe it to the SNS topic, with the resulting object being snapshotted and used for parity testing:

  • DynamoDB Streams:
tests/aws/services/lambda_/event_source_mapping/test_lambda_integration_dynamodbstreams.py::TestDynamoDBEventSourceMapping::test_dynamodb_event_source_mapping_with_sns_on_failure_destination_confi
  • Kinesis:
tests/aws/services/lambda_/event_source_mapping/test_lambda_integration_kinesis.py::TestKinesisSource::test_kinesis_event_source_mapping_with_sns_on_failure_destination_config

Copy link

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR Summary

This PR adds support for using SNS as a failure destination (DLQ) for Kinesis and DynamoDB Streams event source mappings in Lambda functions, aligning LocalStack's functionality with AWS.

  • Modified StreamPoller class to handle SNS topics as failure destinations
  • Added 'sns:*' to allowed actions in s3_lambda_permission policy for Lambda testing
  • Implemented new test cases for DynamoDB and Kinesis event source mappings with SNS failure destinations
  • Updated test snapshots and validation files to include new SNS-related test scenarios
  • Enhanced error handling and verification processes in the new test cases

8 file(s) reviewed, 1 comment(s)
Edit PR Review Bot Settings

Comment on lines +446 to +447
sleep = 15 if is_aws_cloud() else 5
messages = retry(verify_failure_received, retries=15, sleep=sleep, sleep_before=5)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

style: Consider using a constant for sleep time instead of a conditional

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants