Crash during 3.13 -> 4.0 upgrade when there is an enabled feature flag of a disabled plugin #12963

gomoripeti · 2024-12-17T00:43:19Z

gomoripeti
Dec 17, 2024

Describe the bug

When a plugin is enabled its feature flags are discovered however when the same plugin is disabled its feature flags remain discovered. It is even possible to enable such feature flag leading to various problems, crashes, and nodes refusing to start.

We have originally faced this issue with the MQTT plugin and the rabbit_mqtt_qos0_queue feature flag. On a new 3.13.7 cluster mqtt plugin was enabled then disabled. Then all feature flags were enabled enabling rabbit_mqtt_qos0_queue. Finally during a rolling upgrade to 4.0.4 the bellow crash was seen preventing the first node to start:

boot crash

2024-12-16 12:39:58.423260+00:00 [error] <0.267.0> BOOT FAILED
2024-12-16 12:39:58.423260+00:00 [error] <0.267.0> ===========
2024-12-16 12:39:58.423260+00:00 [error] <0.267.0> Exception during startup:
2024-12-16 12:39:58.423260+00:00 [error] <0.267.0>
2024-12-16 12:39:58.423260+00:00 [error] <0.267.0> error:{badmatch,#{feature_flags => #{classic_mirrored_queue_version => #{name => classic_mirrored_queue_version,desc => "Support setting version for classic mirrored queues",stability => required,provided_by => rabbit},classic_queue_type_delivery_support => #{name => classic_queue_type_delivery_support,desc => "Bug fix for classic queue deliveries using mixed versions",stability => required,doc_url => "https://github.com/rabbitmq/rabbitmq-server/issues/5931",depends_on => [stream_queue],provided_by => rabbit},detailed_queues_endpoint => #{name => detailed_queues_endpoint,desc => "Add a detailed queues HTTP API endpoint. Reduce number of metrics in the default endpoint.",stability => stable,depends_on => [feature_flags_v2],provided_by => rabbitmq_management},direct_exchange_routing_v2 => #{name => direct_exchange_routing_v2,desc => "v2 direct exchange routing implementation",stability => required,depends_on => [feature_flags_v2,implicit_default_bindings],provided_by => rabbit},drop_unroutable_metric => #{name => drop_unroutable_metric,desc => "Count unroutable publishes to be dropped in stats",stability => required,provided_by => rabbitmq_management_agent},empty_basic_get_metric => #{name => empty_basic_get_metric,desc => "Count AMQP `basic.get` on empty queues in stats",stability => required,provided_by => rabbitmq_management_agent},feature_flags_v2 => #{name => feature_flags_v2,desc => "Feature flags subsystem V2",stability => required,provided_by => rabbit},implicit_default_bindings => #{name => implicit_default_bindings,desc => "Default bindings are now implicit, instead of being stored in the database",stability => required,provided_by => rabbit},listener_records_in_ets => #{name => listener_records_in_ets,desc => "Store listener records in ETS instead of Mnesia",stability => required,depends_on => [feature_flags_v2],provided_by => rabbit},maintenance_mode_status => #{name => maintenance_mode_status,desc => "Maintenance mode status",stability => required,provided_by => rabbit},message_containers => #{name => message_containers,desc => "Message containers.",stability => stable,depends_on => [feature_flags_v2],provided_by => rabbit},message_containers_deaths_v2 => #{name => message_containers_deaths_v2,desc => "Bug fix for dead letter cycle detection",stability => stable,doc_url => "https://github.com/rabbitmq/rabbitmq-server/issues/11159",depends_on => [message_containers],provided_by => rabbit},quorum_queue => #{name => quorum_queue,desc => "Support queues of type `quorum`",stability => required,doc_url => "https://www.rabbitmq.com/quorum-queues.html",provided_by => rabbit},quorum_queue_non_voters => #{name => quorum_queue_non_voters,desc => "Allows new quorum queue members to be added as non voters initially.",stability => stable,depends_on => [quorum_queue],provided_by => rabbit},restart_streams => #{name => restart_streams,desc => "Support for restarting streams with optional preferred next leader argument.Used to implement stream leader rebalancing",stability => stable,depends_on => [stream_queue],provided_by => rabbit},stream_filtering => #{name => stream_filtering,desc => "Support for stream filtering.",stability => stable,depends_on => [stream_queue],provided_by => rabbit},stream_queue => #{name => stream_queue,desc => "Support queues of type `stream`",stability => required,doc_url => "https://www.rabbitmq.com/stream.html",depends_on => [quorum_queue],provided_by => rabbit},stream_sac_coordinator_unblock_group => #{name => stream_sac_coordinator_unblock_group,desc => "Bug fix to unblock a group of consumers in a super stream partition",stability => stable,doc_url => "https://github.com/rabbitmq/rabbitmq-server/issues/7743",depends_on => [stream_single_active_consumer],provided_by => rabbit},stream_single_active_consumer => #{name => stream_single_active_consumer,desc => "Single active consumer for streams",stability => required,doc_url => "https://www.rabbitmq.com/stream.html",depends_on => [stream_queue],provided_by => rabbit},stream_update_config_command => #{name => stream_update_config_command,desc => "A new internal command that is used to update streams as part of a policy.",stability => stable,depends_on => [stream_queue],provided_by => rabbit},tracking_records_in_ets => #{name => tracking_records_in_ets,desc => "Store tracking records in ETS instead of Mnesia",stability => required,depends_on => [feature_flags_v2],provided_by => rabbit},user_limits => #{name => user_limits,desc => "Configure connection and channel limits for a user",stability => required,provided_by => rabbit},virtual_host_metadata => #{name => virtual_host_metadata,desc => "Virtual host metadata (description, tags, etc)",stability => required,provided_by => rabbit},transient_nonexcl_queues => #{messages => #{when_permitted => "Feature `transient_nonexcl_queues` is deprecated.\nBy default, this feature can still be used for now.\nIts use will not be permitted by default in a future minor RabbitMQ version and the feature will be removed from a future major RabbitMQ version; actual versions to be determined.\nTo continue using this feature when it is not permitted by default, set the following parameter in your configuration:\n    \"deprecated_features.permit.transient_nonexcl_queues = true\"\nTo test RabbitMQ as if the feature was removed, set this in your configuration:\n    \"deprecated_features.permit.transient_nonexcl_queues = false\"",when_denied => "Feature `transient_nonexcl_queues` is deprecated.\nIts use is not permitted per the configuration (overriding the default, which is permitted):\n    \"deprecated_features.permit.transient_nonexcl_queues = false\"\nIts use will not be permitted by default in a future minor RabbitMQ version and the feature will be removed from a future major RabbitMQ version; actual versions to be determined.\nTo continue using this feature when it is not permitted by default, set the following parameter in your configuration:\n    \"deprecated_features.permit.transient_nonexcl_queues = true\""},name => transient_nonexcl_queues,deprecation_phase => permitted_by_default,doc_url => "https://blog.rabbitmq.com/posts/2021/08/4.0-deprecation-announcements/#removal-of-transient-non-exclusive-queues",provided_by => rabbit},global_qos => #{messages => #{when_permitted => "Feature `global_qos` is deprecated.\nBy default, this feature can still be used for now.\nIts use will not be permitted by default in a future minor RabbitMQ version and the feature will be removed from a future major RabbitMQ version; actual versions to be determined.\nTo continue using this feature when it is not permitted by default, set the following parameter in your configuration:\n    \"deprecated_features.permit.global_qos = true\"\nTo test RabbitMQ as if the feature was removed, set this in your configuration:\n    \"deprecated_features.permit.global_qos = false\"",when_denied => "Feature `global_qos` is deprecated.\nIts use is not permitted per the configuration (overriding the default, which is permitted):\n    \"deprecated_features.permit.global_qos = false\"\nIts use will not be permitted by default in a future minor RabbitMQ version and the feature will be removed from a future major RabbitMQ version; actual versions to be determined.\nTo continue using this feature when it is not permitted by default, set the following parameter in your configuration:\n    \"deprecated_features.permit.global_qos = true\""},name => global_qos,deprecation_phase => permitted_by_default,doc_url => "https://blog.rabbitmq.com/posts/2021/08/4.0-deprecation-announcements/#removal-of-global-qos",provided_by => rabbit},khepri_db => #{name => khepri_db,desc => "Use the new Khepri Raft-based metadata store",stability => experimental,doc_url => [],depends_on => [feature_flags_v2,direct_exchange_routing_v2,maintenance_mode_status,user_limits,virtual_host_metadata,tracking_records_in_ets,listener_records_in_ets,classic_queue_mirroring,ram_node_type],provided_by => rabbit},classic_queue_mirroring => #{messages => #{when_permitted => "Classic mirrored queues are deprecated.\nBy default, they can still be used for now.\nTheir use will not be permitted by default in the next minorRabbitMQ version (if any) and they will be removed from RabbitMQ 4.0.0.\nTo continue using classic mirrored queues when they are not permitted by default, set the following parameter in your configuration:\n    \"deprecated_features.permit.classic_queue_mirroring = true\"\nTo test RabbitMQ as if they were removed, set this in your configuration:\n    \"deprecated_features.permit.classic_queue_mirroring = false\"",when_denied => "Classic mirrored queues are deprecated.\nTheir use is not permitted per the configuration (overriding the default, which is permitted):\n    \"deprecated_features.permit.classic_queue_mirroring = false\"\nTheir use will not be permitted by default in the next minor RabbitMQ version (if any) and they will be removed from RabbitMQ 4.0.0.\nTo continue using classic mirrored queues when they are not permitted by default, set the following parameter in your configuration:\n    \"deprecated_features.permit.classic_queue_mirroring = true\""},name => classic_queue_mirroring,callbacks => #{enable => {rabbit_deprecated_features,enable_underlying_feature_flag_cb},is_feature_used => {rabbit_mirror_queue_misc,are_cmqs_used}},deprecation_phase => permitted_by_default,doc_url => "https://blog.rabbitmq.com/posts/2021/08/4.0-deprecation-announcements/#removal-of-classic-queue-mirroring",provided_by => rabbit},ram_node_type => #{messages => #{when_permitted => "Feature `ram_node_type` is deprecated.\nBy default, this feature can still be used for now.\nIts use will not be permitted by default in a future minor RabbitMQ version and the feature will be removed from a future major RabbitMQ version; actual versions to be determined.\nTo continue using this feature when it is not permitted by default, set the following parameter in your configuration:\n    \"deprecated_features.permit.ram_node_type = true\"\nTo test RabbitMQ as if the feature was removed, set this in your configuration:\n    \"deprecated_features.permit.ram_node_type = false\"",when_denied => "Feature `ram_node_type` is deprecated.\nIts use is not permitted per the configuration (overriding the default, which is permitted):\n    \"deprecated_features.permit.ram_node_type = false\"\nIts use will not be permitted by default in a future minor RabbitMQ version and the feature will be removed from a future major RabbitMQ version; actual versions to be determined.\nTo continue using this feature when it is not permitted by default, set the following parameter in your configuration:\n    \"deprecated_features.permit.ram_node_type = true\""},name => ram_node_type,deprecation_phase => permitted_by_default,doc_url => "https://blog.rabbitmq.com/posts/2021/08/4.0-deprecation-announcements/#removal-of-ram-nodes",provided_by => rabbit},management_metrics_collection => #{messages => #{when_permitted => "Feature `management_metrics_collection` is deprecated.\nBy default, this feature can still be used for now.\nIts use will not be permitted by default in a future minor RabbitMQ version and the feature will be removed from a future major RabbitMQ version; actual versions to be determined.\nTo continue using this feature when it is not permitted by default, set the following parameter in your configuration:\n    \"deprecated_features.permit.management_metrics_collection = true\"\nTo test RabbitMQ as if the feature was removed, set this in your configuration:\n    \"deprecated_features.permit.management_metrics_collection = false\"",when_denied => "Feature `management_metrics_collection` is deprecated.\nIts use is not permitted per the configuration (overriding the default, which is permitted):\n    \"deprecated_features.permit.management_metrics_collection = false\"\nIts use will not be permitted by default in a future minor RabbitMQ version and the feature will be removed from a future major RabbitMQ version; actual versions to be determined.\nTo continue using this feature when it is not permitted by default, set the following parameter in your configuration:\n    \"deprecated_features.permit.management_metrics_collection = true\""},name => management_metrics_collection,deprecation_phase => permitted_by_default,doc_url => "https://blog.rabbitmq.com/posts/2021/08/4.0-deprecation-announcements/#disable-metrics-delivery-via-the-management-api--ui",provided_by => rabbitmq_management_agent}},states_per_node => #{'rabbit@healthy-cobalt-greyhound-03' => #{rabbit_mqtt_qos0_queue => true,classic_mirrored_queue_version => true,classic_queue_type_delivery_support => true,detailed_queues_endpoint => true,direct_exchange_routing_v2 => true,drop_unroutable_metric => true,empty_basic_get_metric => true,feature_flags_v2 => true,implicit_default_bindings => true,listener_records_in_ets => true,maintenance_mode_status => true,message_containers => true,message_containers_deaths_v2 => true,quorum_queue => true,quorum_queue_non_voters => true,restart_streams => true,stream_filtering => true,stream_queue => true,stream_sac_coordinator_unblock_group => true,stream_single_active_consumer => true,stream_update_config_command => true,tracking_records_in_ets => true,user_limits => true,virtual_host_metadata => true,transient_nonexcl_queues => false,global_qos => false,khepri_db => false,classic_queue_mirroring => false,ram_node_type => false,management_metrics_collection => false,delete_ra_cluster_mqtt_node => false,mqtt_v5 => false},'rabbit@healthy-cobalt-greyhound-02' => #{rabbit_mqtt_qos0_queue => true,classic_mirrored_queue_version => true,classic_queue_type_delivery_support => true,detailed_queues_endpoint => true,direct_exchange_routing_v2 => true,drop_unroutable_metric => true,empty_basic_get_metric => true,feature_flags_v2 => true,implicit_default_bindings => true,listener_records_in_ets => true,maintenance_mode_status => true,message_containers => true,message_containers_deaths_v2 => true,quorum_queue => true,quorum_queue_non_voters => true,restart_streams => true,stream_filtering => true,stream_queue => true,stream_sac_coordinator_unblock_group => true,stream_single_active_consumer => true,stream_update_config_command => true,tracking_records_in_ets => true,user_limits => true,virtual_host_metadata => true,transient_nonexcl_queues => false,global_qos => false,khepri_db => false,classic_queue_mirroring => false,ram_node_type => false,management_metrics_collection => false,delete_ra_cluster_mqtt_node => false,mqtt_v5 => false}},applications_per_node => #{'rabbit@healthy-cobalt-greyhound-03' => [rabbit,rabbit_common,rabbitmq_consistent_hash_exchange,rabbitmq_federation,rabbitmq_federation_management,rabbitmq_management,rabbitmq_management_agent,rabbitmq_prelaunch,rabbitmq_prometheus,rabbitmq_shovel,rabbitmq_shovel_management,rabbitmq_web_dispatch],'rabbit@healthy-cobalt-greyhound-02' => [rabbit,rabbit_common,rabbitmq_consistent_hash_exchange,rabbitmq_federation,rabbitmq_federation_management,rabbitmq_management,rabbitmq_management_agent,rabbitmq_prelaunch,rabbitmq_prometheus,rabbitmq_shovel,rabbitmq_shovel_management,rabbitmq_web_dispatch]}}}
2024-12-16 12:39:58.423260+00:00 [error] <0.267.0>
2024-12-16 12:39:58.423260+00:00 [error] <0.267.0>     rabbit_ff_controller:-check_one_way_compatibility/2-fun-0-/3, line 514
2024-12-16 12:39:58.423260+00:00 [error] <0.267.0>     lists:all_1/2, line 1520
2024-12-16 12:39:58.423260+00:00 [error] <0.267.0>     rabbit_ff_controller:are_compatible/2, line 496
2024-12-16 12:39:58.423260+00:00 [error] <0.267.0>     rabbit_ff_controller:check_node_compatibility_task1/4, line 437
2024-12-16 12:39:58.423260+00:00 [error] <0.267.0>     rabbit_db_cluster:check_compatibility/1, line 376
2024-12-16 12:39:58.423260+00:00 [error] <0.267.0>     rabbit_mnesia:check_cluster_consistency/2, line 687
2024-12-16 12:39:58.423260+00:00 [error] <0.267.0>     lists:foldl/3, line 1594
2024-12-16 12:39:58.423260+00:00 [error] <0.267.0>     rabbit_mnesia:check_cluster_consistency/0, line 648

But similar issues can be reproduced on new 4.0.4 cluster with the rabbitmq_management plugin and detailed_queues_endpoint feature flag. See repro steps.

It is possible that feature flag handling changed between 3.13.7 and 4.0 so that this crash that happened during upgrade couldn't happen for later upgrades (eg 4.0.x -> 4.1.x)

Reproduction steps

Start local 3-node 4.0.x cluster from git repo

% make start-cluster TEST_TMPDIR=/tmp/rabbitmq RABBITMQ_ENABLED_PLUGINS=

Plugins are not enabled and the detailed_queues_endpoint feature flag is not available

% for i in 1 2 3; do sbin/rabbitmq-plugins list -q rabbitmq_management --node rabbit-$i; done
[  ] rabbitmq_management            4.0.3+112.g78602ec
[  ] rabbitmq_management_agent      4.0.3+112.g78602ec
[  ] rabbitmq_management            4.0.3+112.g78602ec
[  ] rabbitmq_management_agent      4.0.3+112.g78602ec
[  ] rabbitmq_management            4.0.3+112.g78602ec
[  ] rabbitmq_management_agent      4.0.3+112.g78602ec

% for i in 1 2 3; do sbin/rabbitmqctl list_feature_flags --node rabbit-$i | grep detailed ; done
<empty response>

Enable management plugin

% sbin/rabbitmq-plugins enable rabbitmq_management --node rabbit-1
Enabling plugins on node rabbit-1@Peters-MBP:
rabbitmq_management
The following plugins have been configured:
  rabbitmq_management
  rabbitmq_management_agent
  rabbitmq_web_dispatch
Applying plugin configuration to rabbit-1@Peters-MBP...
The following plugins have been enabled:
  rabbitmq_management
  rabbitmq_management_agent
  rabbitmq_web_dispatch

started 3 plugins.

The feature flag is listed on node-1 as disabled. This is already unexpected. I would expect it to be "unavailable" as it is not available on all nodes

% for i in 1 2 3; do echo rabbit-$i; sbin/rabbitmqctl list_feature_flags --node rabbit-$i | grep detailed ; done
rabbit-1
detailed_queues_endpoint	disabled
rabbit-2
rabbit-3

Disable the management plugin

% sbin/rabbitmq-plugins disable rabbitmq_management --node rabbit-1
Disabling plugins on node rabbit-1@Peters-MBP:
rabbitmq_management
All plugins have been disabled.
Applying plugin configuration to rabbit-1@Peters-MBP...
The following plugins have been disabled:
  rabbitmq_web_dispatch
  rabbitmq_management_agent
  rabbitmq_management

stopped 3 plugins.

Feature flag still listed on node-1

% for i in 1 2 3; do echo rabbit-$i; sbin/rabbitmqctl list_feature_flags --node rabbit-$i | grep detailed ; done
rabbit-1
detailed_queues_endpoint	disabled
rabbit-2
rabbit-3

Trying to enable the plugin on node-2 leads to a crash and node-2 goes down

% sbin/rabbitmqctl enable_feature_flag detailed_queues_endpoint --node rabbit-2
Enabling feature flag "detailed_queues_endpoint" ...
Error:
{{{:badmap, :undefined}, [{:rabbit_ff_controller, :run_callback, 5, []}, {:rabbit_ff_controller, :do_enable, 3, [file: 'rabbit_ff_controller.erl', line: 1043]}, {:rabbit_ff_controller, :update_feature_state_and_enable, 2, [file: 'rabbit_ff_controller.erl', line: 980]}, {:rabbit_ff_controller, :enable_with_registry_locked, 2, [file: 'rabbit_ff_controller.erl', line: 851]}, {:rabbit_ff_controller, :lock_registry_and_enable, 2, [file: 'rabbit_ff_controller.erl', line: 819]}, {:rabbit_ff_controller, :enable_many, 2, [file: 'rabbit_ff_controller.erl', line: 776]}, {:rabbit_ff_controller, :updating_feature_flag_states, 3, [file: 'rabbit_ff_controller.erl', line: 276]}, {:gen_statem, :loop_state_callback, 11, [file: 'gen_statem.erl', line: 1395]}]}, {:gen_statem, :call, [:rabbit_ff_controller, {:enable, [:detailed_queues_endpoint]}, :infinity]}}

Expected behavior

if the plugin is only loaded on part of the cluster nodes: list it as unavailable
it is not possible to enable a feature flag of a disabled plugin

Additional context

No response

Answered by gomoripeti

Jan 17, 2025

I did experiment a bit more with this, and I came to the same conclusion that on 4.0.x and above the same problem cannot happen (even with contrived scenarios like enabling a plugin/feature flag during a netsplit) It is specific to the difference of feature flag inventory representation between 3.13 and 4.0 in the very specific case of a plugin being enabled and then disabled.

For the 3.13.x -> 4.0.x rolling upgrade case there is a workaround: enabling the MQTT plugin and its feature flags before the upgrade and then the plugin can be re-disabled after the upgrade.

Therefore I consider this issue resolved. Thank you again for the help with it.

View full answer

gomoripeti · 2024-12-17T01:01:13Z

gomoripeti
Dec 17, 2024
Author

One difference between 3.13.7 and mqtt plugin and 4.0.x and management plugin is that on 3.13.7 with a disabled mqtt plugin I got the following error printout

sudo rabbitmqctl enable_feature_flag all 
 Enabling all feature flags ... 
 Error: 
 This feature flag is not supported by node rabbit@node-01

And out of the 3 feature flags defined by the mqtt plugin (rabbit_mqtt_qos0_queue, delete_ra_cluster_mqtt_node and mqtt_v5) only the first one got enabled.

0 replies

michaelklishin · 2024-12-17T05:05:34Z

michaelklishin
Dec 17, 2024
Maintainer

Such exotic scenarios where not all nodes have all the plugins enabled, or they are being disabled and re-enabled during an upgrade… are they a really good use of our time?

I highly doubt any feature flag- or upgrades-related integration suites enable plugins "mid-upgrade".

Feature flags are considered and documented as cluster-wide state. This means that the enabled set of plugins must be stable during an upgrade, or else the state is mutated by something the feature flag controller has no knowledge of, or control over.

0 replies

michaelklishin · 2024-12-17T05:13:19Z

michaelklishin
Dec 17, 2024
Maintainer

@gomoripeti I have a fundamental problem with our small team's resources being spent on highly exotic scenarios like this.

Disabling and re-enabling plugins during a version upgrade is not a good idea. During a routine restart where the list of plugin changes feature flags should play no role if the version is stable.

In a hypothetical scenario where one of the enabled plugin automatically needs to enable feature flags, it will not be able to do that until all nodes are compatible.

We can spend a few person-weeks on this or we can spend it on improvements such as Ra 2.16 which deliver double digit efficiency improvements on workflows that affect most QQ users.

0 replies

michaelklishin · 2024-12-17T05:20:22Z

michaelklishin
Dec 17, 2024
Maintainer

if the plugin is only loaded on part of the cluster nodes: list it as unavailable

Enabled plugins is a node's state. Feature flags is a cluster-wide state.

It would likely raise a lot of eyebrows if a plugin was listed as "disabled" or unavailable due to feature flag state.

it is not possible to enable a feature flag of a disabled plugin

This would imply that when a plugin is disabled, so should be its feature flags, and a feature flag cannot be disabled once enabled, or at least it won't be possible while other nodes have the plugin and the FF enabled.

Instead of trying to change RabbitMQ, reduce the number of changes are allowed during a version upgrade in your system. A version upgrade is not the time when plugins should be enabled or disabled, but they can be enabled or disabled once all nodes are on the same version, in which case you won't run into any surprises around feature flags.

0 replies

Zerpet · 2024-12-17T09:57:11Z

Zerpet
Dec 17, 2024
Maintainer

I echo Michael's thoughts. I think this scenario is quite unusual, perhaps even artificial. That being said, I think we should print a friendly error, as we did in 3.13.x.

0 replies

dumbbell · 2024-12-17T10:39:44Z

dumbbell
Dec 17, 2024
Maintainer

This scenario is already supported. This is clearly a bug: the feature flag controller isn’t supposed to run a callback on a node that doesn’t know the feature flag. The list of nodes is already filtered, so there is an issue with the filtering or the input of that filtering.

The feature flag is listed on node-1 as disabled. This is already unexpected. I would expect it to be "unavailable" as it is not available on all nodes

This is expected because a node can have a plugin enabled while other nodes in the cluster don’t have it enabled or don’t have it at all. When that plugin is enabled elsewhere, it is supposed to pick up the feature state from the rest of the cluster. The sync is the same as when a node joins a cluster or is restarted.

1 reply

dumbbell Dec 17, 2024
Maintainer

I submitted a patch in #12966. Could you please give it a try?

Unrelated to this, but in main (future 4.1.x), I tried to improve the user experience around feature flags. If you think something still needs improvements, please share!

gomoripeti · 2024-12-17T11:38:01Z

gomoripeti
Dec 17, 2024
Author

Sorry for the misleading and bad repro steps using 4.0.x and the management plugin. I tried to come up with an example on a recent version but I failed.

I'd like to clarify that there is no change happening during the upgrade. The real world issue what we faced is the following (that I also described and attached the logs for)

On a new 3.13.7 cluster mqtt plugin was enabled then disabled 2 months ago
Then (a few days ago) a 3.13.7 -> 4.0.4 upgrade was initiated, we enable all feature flags before starting that process.
First problem: enabling all feature flags returned an error but out of the 3 feature flags defined by the mqtt plugin (rabbit_mqtt_qos0_queue, delete_ra_cluster_mqtt_node and mqtt_v5) the first one still got enabled.
```
sudo rabbitmqctl enable_feature_flag all 
 Enabling all feature flags ... 
 Error: 
 This feature flag is not supported by node rabbit@node-01 
```
Second problem: during a rolling upgrade to 4.0.4 a big crash was seen preventing the first node to start (crash included in the original post)

...while typing the core team already provided a potential patch :) thank you very much, I am giving it a try

20 replies

michaelklishin Dec 19, 2024
Maintainer

rabbit_ff_controller can be updated in v4.0.x to be more defensive, e.g. by using a case with more heads rather than an if with just two https://github.com/rabbitmq/rabbitmq-server/blob/main/deps/rabbit/src/rabbit_ff_controller.erl#L514. If that alone makes the upgrade scenario proceed, I'd say we should do that.

michaelklishin Dec 19, 2024
Maintainer

@gomoripeti would you like to verify the above hypothesis, assuming that you have a way to reproduce this behavior reliably?

gomoripeti Dec 20, 2024
Author

I am happy to verify things or even experiment with some changes - however unfortunately holiday season starts, so I can only do it after.

(just noting that line 514 on v4.0.4 is a different peace of code https://github.com/rabbitmq/rabbitmq-server/blob/v4.0.4/deps/rabbit/src/rabbit_ff_controller.erl#L514. dumbbell already provided a rewrite of this part in #12963 (reply in thread) but as far as I understand it changes behaviour of the FF subsystem and such changes are usually not back ported to previous minor versions)

let me get back to this topic in the new year. until then happy holidays!

gomoripeti Jan 17, 2025
Author

I did experiment a bit more with this, and I came to the same conclusion that on 4.0.x and above the same problem cannot happen (even with contrived scenarios like enabling a plugin/feature flag during a netsplit) It is specific to the difference of feature flag inventory representation between 3.13 and 4.0 in the very specific case of a plugin being enabled and then disabled.

For the 3.13.x -> 4.0.x rolling upgrade case there is a workaround: enabling the MQTT plugin and its feature flags before the upgrade and then the plugin can be re-disabled after the upgrade.

Therefore I consider this issue resolved. Thank you again for the help with it.

Answer selected by michaelklishin

michaelklishin Jan 17, 2025
Maintainer

Thank you for reporting back.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Crash during 3.13 -> 4.0 upgrade when there is an enabled feature flag of a disabled plugin #12963

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 7 comments 21 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Crash during 3.13 -> 4.0 upgrade when there is an enabled feature flag of a disabled plugin #12963

gomoripeti Dec 17, 2024

Describe the bug

Reproduction steps

Expected behavior

Additional context

Replies: 7 comments · 21 replies

gomoripeti Dec 17, 2024 Author

michaelklishin Dec 17, 2024 Maintainer

michaelklishin Dec 17, 2024 Maintainer

michaelklishin Dec 17, 2024 Maintainer

Zerpet Dec 17, 2024 Maintainer

dumbbell Dec 17, 2024 Maintainer

dumbbell Dec 17, 2024 Maintainer

gomoripeti Dec 17, 2024 Author

michaelklishin Dec 19, 2024 Maintainer

michaelklishin Dec 19, 2024 Maintainer

gomoripeti Dec 20, 2024 Author

gomoripeti Jan 17, 2025 Author

michaelklishin Jan 17, 2025 Maintainer

gomoripeti
Dec 17, 2024

Replies: 7 comments 21 replies

gomoripeti
Dec 17, 2024
Author

michaelklishin
Dec 17, 2024
Maintainer

michaelklishin
Dec 17, 2024
Maintainer

michaelklishin
Dec 17, 2024
Maintainer

Zerpet
Dec 17, 2024
Maintainer

dumbbell
Dec 17, 2024
Maintainer

dumbbell Dec 17, 2024
Maintainer

gomoripeti
Dec 17, 2024
Author

michaelklishin Dec 19, 2024
Maintainer

michaelklishin Dec 19, 2024
Maintainer

gomoripeti Dec 20, 2024
Author

gomoripeti Jan 17, 2025
Author

michaelklishin Jan 17, 2025
Maintainer