feat: Add Azure EventHubs module #1183

WakaToa · 2024-05-22T12:04:27Z

What does this PR do?

Microsoft released an emulator for EventHubs yesterday, see Azure/azure-service-bus#223 (comment)

This PR included a module for the emulator.

Info

You need to provide an Azurite (emulator) instance/endpoint to start the EventHub emulator. This can be done using the EventHubsConfiguration.

You need to specify the following configuration file when starting the emulator.

{
  "UserConfig": {
    "NamespaceConfig": [
      {
        "Type": "EventHub",
        "Name": "emulatorNs1",
        "Entities": [
          {
            "Name": "eh1",
            "PartitionCount": "2",
            "ConsumerGroups": [
              {
                "Name": "cg1"
              }
            ]
          }
        ]
      }
    ], 
    "LoggingConfig": {
      "Type": "File"
    }
  }
}

It will be dynamically mapped and i have built a floating builder so that the user can set the configuration as easily as possible.
The namespace "emulatorNs1" is mandatory and it is the only one currently supported by the emulator.

var configurationBuilder = ConfigurationBuilder
    .Create()
    .WithEventHub(EventHubName, "2", new[] { EventHubConsumerGroupName });

var builder = new EventHubsBuilder()
    .WithNetwork(_network)
    .WithConfigurationBuilder(configurationBuilder)
    .WithAzuriteBlobEndpoint(AzuriteNetworkAlias)
    .WithAzuriteTableEndpoint(AzuriteNetworkAlias);

_eventHubsContainer = builder.Build();

await _eventHubsContainer.StartAsync();

netlify · 2024-05-22T12:04:42Z

✅ Deploy Preview for testcontainers-dotnet ready!

Name	Link
🔨 Latest commit	`b264e42`
🔍 Latest deploy log	https://app.netlify.com/sites/testcontainers-dotnet/deploys/664de1efb02bc800082444d3
😎 Deploy Preview	https://deploy-preview-1183--testcontainers-dotnet.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

eddumelendez · 2024-05-31T10:37:18Z

tests/Testcontainers.EventHubs.Tests/EventHubsContainerTest.cs

+    private readonly AzuriteContainer _azuriteContainer;
+
+    private EventHubsContainer _eventHubsContainer;
+
+    private readonly INetwork _network = new NetworkBuilder().WithName(NetworkName).Build();
+
+    private const string NetworkName = "eh-network";
+    private const string AzuriteNetworkAlias = "azurite";
+
+    private const string EventHubName = "testeventhub";
+    private const string EventHubConsumerGroupName = "testconsumergroup";
+
+    private EventHubsContainerTest()
+    {
+        _azuriteContainer = new AzuriteBuilder()
+            .WithNetwork(_network)
+            .WithNetworkAliases(AzuriteNetworkAlias)
+            .Build();
+    }


This is an idea still but due to EventHubs emulator will not work without Azurite then we can provide two ways to run EventHub Container implementation.

Embed Azurite, so, users don't worry about it.

Allow external Azurite in case users need a specific configuration

/cc @kiview WDYT?

Does a pattern exist already ?

recently, service bus was added with the same pattern. See https://github.com/testcontainers/testcontainers-dotnet/blob/develop/src/Testcontainers.ServiceBus/ServiceBusBuilder.cs#L114

I think it makes sense to align with the pattern Service Bus is using and then follow up with something that allows us to use an existing Azurite instance. Please also see this issue: #1323. This is something we need to figure out and support in general.

eddumelendez · 2024-06-25T21:25:40Z

Also, please add documentation for the module.

Saglodha · 2025-01-06T06:55:28Z

Hi @WakaToa / @eddumelendez / @kiview ,

I am from Azure Event Hubs Product team and was looking at this PR. I do realize that there were some plausible solutions offered above to get this integration setup.

Could you please let us know if we could extend any help from our side to integrate Event Hubs emulator with testcontainers framework? We'll be happy to provide any product related assistance here.

HofmeisterAn

Thanks for the PR. There are a few things we need to address before we can merge the PR.

HofmeisterAn · 2025-01-08T05:39:15Z

src/Testcontainers.EventHubs/EventHubsBuilder.cs

+    {
+        Validate();
+
+        var waitStrategy = Wait.ForUnixContainer().UntilMessageIsLogged("Emulator Service is Successfully Up!");


The wait strategy should be moved to Init().

HofmeisterAn · 2025-01-08T05:44:37Z

src/Testcontainers.EventHubs/Configuration/ConfigurationBuilder.cs

@@ -0,0 +1,53 @@
+namespace Testcontainers.EventHubs.Configuration


Please change all to file scoped namespaces.

HofmeisterAn · 2025-01-08T05:52:30Z

src/Testcontainers.EventHubs/EventHubsBuilder.cs

+    public EventHubsBuilder WithConfigurationBuilder(ConfigurationBuilder configurationBuilder)
+    {
+        var configBytes = Encoding.UTF8.GetBytes(configurationBuilder.Build());
+
+        return Merge(DockerResourceConfiguration, new EventHubsConfiguration(configurationBuilder: configurationBuilder))
+            .WithResourceMapping(configBytes, "Eventhubs_Emulator/ConfigFiles/Config.json");
+    }


Does this builder cover all Event Hubs configuration? Are we responsible for it? Is there a default configuration? The EventHubsBuilder will fail if the user does not call WithConfigurationBuilder(ConfigurationBuilder) explicitly. Until now, no other module requires an additional explicit call to a builder method (expect license agreements). _ = new FooBuilder.Build() always returns a working configuration. This is something I favor.

HofmeisterAn · 2025-01-08T05:54:39Z

src/Testcontainers.EventHubs/EventHubsBuilder.cs

+    {
+        return base.Init()
+            .WithImage(EventHubsImage)
+            .WithEnvironment("ACCEPT_EULA", "Y")


Please do not hide license agreements. Right now, we use the following pattern.

HofmeisterAn · 2025-01-08T05:57:48Z

src/Testcontainers.EventHubs/EventHubsConfiguration.cs

+    /// <summary>
+    /// Gets the configuration builder
+    /// </summary>
+    public ConfigurationBuilder ConfigurationBuilder { get; }


It looks like we do not use the ConfigurationBuilder sometime later. In that case, we do not need to store it in the EventHubsConfiguration.

HofmeisterAn · 2025-01-08T06:06:06Z

src/Testcontainers.EventHubs/EventHubsBuilder.cs

@@ -0,0 +1,127 @@
+using DotNet.Testcontainers;


Please use the global usings file.

HofmeisterAn · 2025-01-08T06:08:42Z

tests/Testcontainers.EventHubs.Tests/EventHubsContainerTest.cs

+
+    private EventHubsContainer _eventHubsContainer;
+
+    private readonly INetwork _network = new NetworkBuilder().WithName(NetworkName).Build();


Please do not use static network names. This may clash with already existing resources and may prevent tests from running in parallel. See our best practices.

HofmeisterAn · 2025-01-08T06:11:05Z

tests/Testcontainers.EventHubs.Tests/EventHubsContainerTest.cs

+    private readonly AzuriteContainer _azuriteContainer;
+
+    private EventHubsContainer _eventHubsContainer;
+
+    private readonly INetwork _network = new NetworkBuilder().WithName(NetworkName).Build();
+
+    private const string NetworkName = "eh-network";
+    private const string AzuriteNetworkAlias = "azurite";
+
+    private const string EventHubName = "testeventhub";
+    private const string EventHubConsumerGroupName = "testconsumergroup";
+
+    private EventHubsContainerTest()
+    {
+        _azuriteContainer = new AzuriteBuilder()
+            .WithNetwork(_network)
+            .WithNetworkAliases(AzuriteNetworkAlias)
+            .Build();
+    }


I think it makes sense to align with the pattern Service Bus is using and then follow up with something that allows us to use an existing Azurite instance. Please also see this issue: #1323. This is something we need to figure out and support in general.

HofmeisterAn · 2025-01-08T06:11:42Z

tests/Testcontainers.EventHubs.Tests/Testcontainers.EventHubs.Tests.csproj

@@ -0,0 +1,22 @@
+<Project Sdk="Microsoft.NET.Sdk">
+    <PropertyGroup>
+        <TargetFrameworks>net8.0</TargetFrameworks>


Suggested change

<TargetFrameworks>net8.0</TargetFrameworks>

<TargetFrameworks>net9.0</TargetFrameworks>

HofmeisterAn · 2025-01-08T06:12:36Z

tests/Testcontainers.EventHubs.Tests/Testcontainers.EventHubs.Tests.csproj

+    <ItemGroup>
+      <PackageVersion Update="Azure.Messaging.EventHubs" Version="5.11.3" />
+    </ItemGroup>


Suggested change

<ItemGroup>

<PackageVersion Update="Azure.Messaging.EventHubs" Version="5.11.3" />

</ItemGroup>

Already included above.

WakaToa added 2 commits May 22, 2024 13:34

Support Azure EventHubs

8fe0070

Adjust EventHubsConfiguration

3d5c226

WakaToa added 2 commits May 22, 2024 14:14

Add CPM for Azure.Messaging.EventHubs

ebccc8c

better ordering of package version Azure.Messaging.EventHubs

b264e42

HofmeisterAn added enhancement New feature or request module An official Testcontainers module labels May 27, 2024

eddumelendez reviewed May 31, 2024

View reviewed changes

HofmeisterAn requested changes Jan 8, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add Azure EventHubs module #1183

feat: Add Azure EventHubs module #1183

WakaToa commented May 22, 2024

netlify bot commented May 22, 2024 •

edited

Loading

eddumelendez May 31, 2024

scrocquesel-ml150 Jul 31, 2024

eddumelendez Jan 7, 2025

HofmeisterAn Jan 8, 2025

eddumelendez commented Jun 25, 2024

Saglodha commented Jan 6, 2025

HofmeisterAn left a comment

HofmeisterAn Jan 8, 2025

HofmeisterAn Jan 8, 2025

HofmeisterAn Jan 8, 2025

HofmeisterAn Jan 8, 2025

HofmeisterAn Jan 8, 2025

HofmeisterAn Jan 8, 2025

HofmeisterAn Jan 8, 2025

HofmeisterAn Jan 8, 2025

HofmeisterAn Jan 8, 2025

HofmeisterAn Jan 8, 2025

		@@ -0,0 +1,53 @@
		namespace Testcontainers.EventHubs.Configuration


		private EventHubsContainer _eventHubsContainer;

		private readonly INetwork _network = new NetworkBuilder().WithName(NetworkName).Build();

	<TargetFrameworks>net8.0</TargetFrameworks>
	<TargetFrameworks>net9.0</TargetFrameworks>

	<ItemGroup>
	<PackageVersion Update="Azure.Messaging.EventHubs" Version="5.11.3" />
	</ItemGroup>

feat: Add Azure EventHubs module #1183

Are you sure you want to change the base?

feat: Add Azure EventHubs module #1183

Conversation

WakaToa commented May 22, 2024

What does this PR do?

Info

netlify bot commented May 22, 2024 • edited Loading

✅ Deploy Preview for testcontainers-dotnet ready!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eddumelendez commented Jun 25, 2024

Saglodha commented Jan 6, 2025

HofmeisterAn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

netlify bot commented May 22, 2024 •

edited

Loading