Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AWS IAM authentication #178

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

AWS IAM authentication #178

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
5d31f1f
Added dependency on AWS IAM jars and updated documentation
andsel Aug 19, 2024
6f1b45c
Forcibly added all transient dependencies of aws-msk-iam-auth library
andsel Aug 23, 2024
bef696e
Bumped 11.5.2 version and updated Changelog to expose the settings to…
andsel Aug 23, 2024
4c37ca9
Updated AWS IAM MKS library to the latest
andsel Sep 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
## 11.5.2
- Expose the SASL client callback class setting to the Logstash configuration [#177](https://github.com/logstash-plugins/logstash-integration-kafka/pull/177)
- Bundles all libraries to use AWS IAM authentication as SASL client [#178](https://github.com/logstash-plugins/logstash-integration-kafka/pull/178)

## 11.5.1
- Specify that only headers with UTF-8 encoded values are supported in extended decoration [#174](https://github.com/logstash-plugins/logstash-integration-kafka/pull/174)

Expand Down
26 changes: 26 additions & 0 deletions build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,32 @@ dependencies {
implementation 'com.github.luben:zstd-jni:1.5.5-4'
implementation 'org.lz4:lz4-java:1.8.0'
implementation 'org.xerial.snappy:snappy-java:1.1.10.5'
implementation 'software.amazon.msk:aws-msk-iam-auth:1.1.4'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use more modern versions of these libraries? These are somewhat out of date

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested with commit 4c37ca9 and was able to connect.
So @robbavey please could you review it again 🙏

// transitive deps used by software.amazon.msk:aws-msk-iam-auth
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😳

This seems like... a lot. Are we concerned that we are going to have issues with dependency clashes with other plugins and Logstash core?

cc @jsvd

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In 4c37ca9 you can see the difference switching from 1.1.4 to 2.2.0. While there is a consistent part of AWS libraries (part of groups software.amazon.awssdk and com.amazonaws) with this new version it adds:

  • commons-codec:commons-codec
  • commons-logging:commons-logging
  • io.netty:netty-*
  • org.reactivestreams:reactive-streams
  • org.apache.httpcomponents httpclient and httpcore

While Netty http and http2 codecs are present it was added also a dependency on Apache's httpclient so seems redundant.
The risky one is Netty, because used in other Logstash plugins.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From Logstash core we have:

  • httpcore-4.4.16.jar vs this 4.4.13 (no apparent conflict)
  • httpclient-4.5.14.jar vs this 4.5.13 (no apparent conflict)
  • commons-codec-1.17.0.jar vs this 1.15 API should be compatible
  • commons-logging-1.3.1.jar vs this 1.2 API should be compatible
  • Netty in Beats/TCP/HTTP inputs are at 4.1.109 vs this 4.1.108 so not so far and shouldn't break

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pluggable security classes root in 2 classes:

  • software.amazon.msk.auth.iam.IAMLoginModule
  • software.amazon.msk.auth.iam.IAMClientCallbackHandler

The IAMClientCallbackHandler explicitly instantiates MSKCredentialProvider at: https://github.com/aws/aws-msk-iam-auth/blob/v2.2.0/src/main/java/software/amazon/msk/auth/iam/IAMClientCallbackHandler.java#L54

The MSKCredentialProvider has direct dependency on 3 main AWSSDK packages:

  • software.amazon.awssdk.auth provided by software.amazon.awssdk:auth
  • software.amazon.awssdk.core provided by software.amazon.awssdk:sdk-core
  • software.amazon.awssdk.services.sts provide by software.amazon.awssdk:sts

Looking at the source of aws-msk-iam-auth the artifact software.amazon.awssdk:sdk-core requires the following dependency tree:

+--- software.amazon.awssdk:sdk-core:2.26.8
    +--- software.amazon.awssdk:annotations:2.26.8
    +--- software.amazon.awssdk:http-client-spi:2.26.8
    |    +--- software.amazon.awssdk:annotations:2.26.8
    |    +--- software.amazon.awssdk:utils:2.26.8 (*)
    |    +--- software.amazon.awssdk:metrics-spi:2.26.8
    |    |    +--- software.amazon.awssdk:annotations:2.26.8
    |    |    \--- software.amazon.awssdk:utils:2.26.8 (*)
    |    \--- org.reactivestreams:reactive-streams:1.0.4
    +--- software.amazon.awssdk:metrics-spi:2.26.8 (*)
    +--- software.amazon.awssdk:endpoints-spi:2.26.8
    |    \--- software.amazon.awssdk:annotations:2.26.8
    +--- software.amazon.awssdk:http-auth-spi:2.26.8
    |    +--- software.amazon.awssdk:annotations:2.26.8
    |    +--- software.amazon.awssdk:utils:2.26.8 (*)
    |    +--- software.amazon.awssdk:http-client-spi:2.26.8 (*)
    |    +--- org.reactivestreams:reactive-streams:1.0.4
    |    \--- software.amazon.awssdk:identity-spi:2.26.8
    |         +--- software.amazon.awssdk:annotations:2.26.8
    |         \--- software.amazon.awssdk:utils:2.26.8 (*)
    +--- software.amazon.awssdk:http-auth-aws:2.26.8
    |    +--- software.amazon.awssdk:annotations:2.26.8
    |    +--- software.amazon.awssdk:utils:2.26.8 (*)
    |    +--- software.amazon.awssdk:identity-spi:2.26.8 (*)
    |    +--- software.amazon.awssdk:http-client-spi:2.26.8 (*)
    |    +--- software.amazon.awssdk:http-auth-spi:2.26.8 (*)
    |    +--- software.amazon.awssdk:checksums-spi:2.26.8
    |    |    \--- software.amazon.awssdk:annotations:2.26.8
    |    \--- software.amazon.awssdk:checksums:2.26.8
    |         +--- software.amazon.awssdk:annotations:2.26.8
    |         \--- software.amazon.awssdk:checksums-spi:2.26.8 (*)
    +--- software.amazon.awssdk:checksums-spi:2.26.8 (*)
    +--- software.amazon.awssdk:checksums:2.26.8 (*)
    +--- software.amazon.awssdk:identity-spi:2.26.8 (*)
    +--- software.amazon.awssdk:utils:2.26.8 (*)
    +--- software.amazon.awssdk:profiles:2.26.8
    |    +--- software.amazon.awssdk:utils:2.26.8 (*)
    |    \--- software.amazon.awssdk:annotations:2.26.8
    +--- software.amazon.awssdk:retries-spi:2.26.8
    |    +--- software.amazon.awssdk:annotations:2.26.8
    |    \--- software.amazon.awssdk:utils:2.26.8 (*)
    +--- software.amazon.awssdk:retries:2.26.8
    |    +--- software.amazon.awssdk:retries-spi:2.26.8 (*)
    |    +--- software.amazon.awssdk:annotations:2.26.8
    |    \--- software.amazon.awssdk:utils:2.26.8 (*)
    \--- org.reactivestreams:reactive-streams:1.0.4

software.amazon.awssdk:sts has direct dependency on software.amazon.awssdk:apache-client and software.amazon.awssdk:netty-nio-client which kicks in all the netty related dependencies.

Despite the class MSKCredentialProvider doesn't directly refer to classes contained in software.amazon.awssdk:sso and software.amazon.awssdk:ssooidc, removing those doesn't provide any gain in terms of numerosity of dependencies.

So my conclusion is that we can't strip any jar.

implementation 'com.amazonaws:aws-java-sdk-core:1.12.290'
implementation 'com.amazonaws:aws-java-sdk-sts:1.12.290'
implementation 'software.amazon.awssdk:annotations:2.17.267'
implementation 'software.amazon.awssdk:apache-client:2.17.267'
implementation 'software.amazon.awssdk:auth:2.17.267'
implementation 'software.amazon.awssdk:aws-core:2.17.267'
implementation 'software.amazon.awssdk:aws-json-protocol:2.17.267'
implementation 'software.amazon.awssdk:aws-query-protocol:2.17.267'
implementation 'software.amazon.awssdk:http-client-spi:2.17.267'
implementation 'software.amazon.awssdk:json-utils:2.17.267'
implementation 'software.amazon.awssdk:metrics-spi:2.17.267'
implementation 'software.amazon.awssdk:netty-nio-client:2.17.267'
implementation 'software.amazon.awssdk:profiles:2.17.267'
implementation 'software.amazon.awssdk:protocol-core:2.17.267'
implementation 'software.amazon.awssdk:regions:2.17.267'
implementation 'software.amazon.awssdk:sdk-core:2.17.267'
implementation 'software.amazon.awssdk:sso:2.17.267'
implementation 'software.amazon.awssdk:sts:2.17.267'
implementation 'software.amazon.awssdk:third-party-jackson-core:2.17.267'
implementation 'software.amazon.awssdk:utils:2.17.267'
implementation 'software.amazon.msk:aws-msk-iam-auth:1.1.4'



}
task generateGemJarRequiresFile {
doLast {
Expand Down
14 changes: 14 additions & 0 deletions docs/input-kafka.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,19 @@ For more information see https://kafka.apache.org/{kafka_client_doc}/documentati

Kafka consumer configuration: https://kafka.apache.org/{kafka_client_doc}/documentation.html#consumerconfigs

==== AWS MSK IAM authentication
If you use AWS MSK, the AWS MSK IAM access control enables you to handle both authentication and authorization for your MSK cluster with AWS IAM.
For more information on this AWS MSK feature see the https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html[AWS documentation].

To use this Kafka input with AWS MSK IAM authentication, set the following configuration:
```
security_protocol => "SASL_SSL"
sasl_mechanism => "AWS_MSK_IAM"
sasl_jaas_config => "software.amazon.msk.auth.iam.IAMLoginModule required;"
sasl_client_callback_handler_class => "software.amazon.msk.auth.iam.IAMClientCallbackHandler"
```
For more IAM authentication configurations, see the https://github.com/aws/aws-msk-iam-auth[AWS MSK IAM authentication library documentation].

==== Metadata fields

The following metadata from Kafka broker are added under the `[@metadata]` field:
Expand Down Expand Up @@ -598,6 +611,7 @@ This can be defined either in Kafka's JAAS config or in Kafka's config.

http://kafka.apache.org/documentation.html#security_sasl[SASL mechanism] used for client connections.
This may be any mechanism for which a security provider is available.
For AWS MSK IAM authentication use `AWS_MSK_IAM`.
GSSAPI is the default mechanism.

[id="plugins-{type}s-{plugin}-schema_registry_key"]
Expand Down
14 changes: 14 additions & 0 deletions docs/output-kafka.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,19 @@ https://kafka.apache.org/{kafka_client_doc}/documentation.html#producerconfigs

NOTE: This plugin does not support using a proxy when communicating to the Kafka broker.

==== AWS MSK IAM authentication
If you use AWS MSK, the AWS MSK IAM access control enables you to handle both authentication and authorization for your MSK cluster with AWS IAM.
For more information on this AWS MSK feature see the https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html[AWS documentation].

To use this Kafka input with AWS MSK IAM authentication, set the following configuration:
```
security_protocol => "SASL_SSL"
sasl_mechanism => "AWS_MSK_IAM"
sasl_jaas_config => "software.amazon.msk.auth.iam.IAMLoginModule required;"
sasl_client_callback_handler_class => "software.amazon.msk.auth.iam.IAMClientCallbackHandler"
```
For more IAM authentication configurations, see the https://github.com/aws/aws-msk-iam-auth[AWS MSK IAM authentication library documentation].

[id="plugins-{type}s-{plugin}-options"]
==== Kafka Output Configuration Options

Expand Down Expand Up @@ -434,6 +447,7 @@ This can be defined either in Kafka's JAAS config or in Kafka's config.

http://kafka.apache.org/documentation.html#security_sasl[SASL mechanism] used for client connections.
This may be any mechanism for which a security provider is available.
For AWS MSK IAM authentication use `AWS_MSK_IAM`.
GSSAPI is the default mechanism.

[id="plugins-{type}s-{plugin}-security_protocol"]
Expand Down
2 changes: 1 addition & 1 deletion logstash-integration-kafka.gemspec
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
Gem::Specification.new do |s|
s.name = 'logstash-integration-kafka'
s.version = '11.5.1'
s.version = '11.5.2'
s.licenses = ['Apache-2.0']
s.summary = "Integration with Kafka - input and output plugins"
s.description = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline "+
Expand Down