Sync dependencies to Redis

[yhabteab]

SSIA 🤪! Just kidding! This PR synchronises all the required points from Icinga/icingadb#347 (comment) to Redis. However, I'm not going to explain every implementation detail of the PR here, but it can be roughly understood as follows.

• 0af4679 and fe8169a syncs the host/service.affected_children and state.affects_children attributes to Redis as described in Track effect of an object on dependent children #10158.
• 19862bd Adds the no_user_modify flag to the redundancy_group attribute in the dependency.ti file to prevent any runtime alteration of its value, as there is now too much logic and functionality depending on this value and changing it at runtime would have a disastrous end.
• 9f4655b Introduces a new auxiliary class DependencyGroup to easily group and manage identical dependencies of any checkable in one place. Yes, this is also used to group non-redundant dependencies, but such a group is entirely unique for each checkable and is never referenced by other checkables.
• 6638f36 Introduces a global shared registry for the dependency groups. In addition, here is also where all the dependency deduplication magic happens.
• f478e5b Introduces an auxiliary method for determining the state of any DependencyGroup at any given time as described in Let redundancy groups not just fail #10190.
• 49e6ae2 and 2ea7e59 Dumps all dependency-related information to Redis at Icinga 2 startup/reload.
• 1fa8840 Same as the above two commits but only for dependencies runtime state updates.
• e75d884 Processes runtime removed/created dependency objects.
• ae269a8 Removes the obsolete failedDependency parameter of the Checkable::IsReachable() method. It is obsolete because none of the callers make use of it, it just adds unnecessary complexity to the method for no reason.
• 90f20b8 Changes the implementation of the Checkable::IsReachable() method and utilises the DependencyGroup::GetState() method introduced above.
• 6e7157e In order to handle runtime removed/created dependencies without additional overhead, the activation_priority of the Dependency object is set to -10 (just like for downtime objects). This way, the dependency objects will always get activated before their child/parent Checkables.

fixes #10158
fixes #10190
fixes #10227
fixes #10014
fixes #10143

[julianbrost]

I'm somewhat confused by the DependencyGroup class as it doesn't really map to the mental model I had from our discussions on that topic.

So in my understanding, a DependencyGroup would represent a set a set of checkables that are used as parents in dependency config objects combined with the attributes from that dependency object that affect how the availability of that dependency is determined, i.e. ignore_soft_states, period, and states. For dependencies without a redundancy group, that information is all that's needed to determine if all dependency objects that share the parent and these attribute mark all their children as unreachable. With a redundancy group, you have to look at all the parents from the redundancy group with the three aforementioned additional attributes. So that would be how to determine what creates a DependencyGroup object for redundancy groups.

For dependencies without a redundancy group, this grouping provides no value in itself, the dependency objects can be considered individually. There are two reasons why we might instantiate such trivial groups explicitly nonetheless: for one, it may allow simpler code by being able to treat both cases consistently, but more importantly, there was a request from Johannes that if two children depend on the same parent in such a way that the state of these dependencies is always the same (i.e. the three aforementioned attributes are identical), then the different graph edges should refer to the same shared state. These groups may be used for this deduplication as well.

Consider the following example (P = parent checkable, RG = redundancy group as represented in the generated graph, C = child checkable):

graph BT;
p1((P1));
p2((P2));
p3((P3));
c1((C1));
c2((C2));
c3((C3));
c4((C4));
c5((C5));
rg1(RG1);
c1-->rg1;
c2-->rg1;
c3-->rg1;
rg1-->p1;
rg1-->p2;
c4-->p3;
c5-->p3;

Loading

Here I'd expect the creation of the following two DependencyGroups (... refers to the three magic attributes attached to the parent in the corresponding dependency objects):

• {(P1, ...), (P2, ...)}: This basically represents RG1
• {(P3, ...)}: This is a if there was an imaginary second redundancy with only one parent, P3.

[julianbrost]

What's the incompatible change that makes this necessary?

[yhabteab]

The state.affects_children column is not nullable, Icinga DB would crash otherwise.

[yhabteab]

Even the new Redis keys(streamed via the existing runtime:state pipeline) crashed Icinga DB without the expected Redis version.

2025-01-20T09:07:53.941Z	FATAL	icingadb	no object type for redis key icinga:dependency:edge:state found
github.com/icinga/icingadb/pkg/icingadb.(*RuntimeUpdates).Sync.(*RuntimeUpdates).xRead.func9
	/go/src/github.com/Icinga/icingadb/pkg/icingadb/runtime_updates.go:268
golang.org/x/sync/errgroup.(*Group).Go.func1
	/go/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:78
runtime.goexit
	/usr/local/go/src/runtime/asm_arm64.s:1223
exit status 1

[julianbrost]

It should be possible to create an index on GetCompositeKey() directly, see CheckableNextCheckExtractor. That should make defining struct Hash and struct Equal unnecessary.

[yhabteab]

It should be possible to create an index on GetCompositeKey() directly, see CheckableNextCheckExtractor.

Seriously, what does CheckableNextCheckExtractor do with key extraction from functions? You could literally replace that functor with a simple boost::multi_index::member<CheckableScheduleInfo, double, &CheckableScheduleInfo::NextCheck> and it would still work. If I wanted to create the index in this way, I would have done it as in the second index below that one.

That should make defining struct Hash and struct Equal unnecessary.

No, it wouldn't! Have you actually tested it or is this just a this should be done differently thing? This container stores DependencyGroup* objects, so if it wants to perform an equality check, it has to pass a DependencyGroup* object as one of the arguments to std::equal_to<String>, which will never work.

[julianbrost]

Aren't your custom equal and hash function creating an index on GetCompositeKey()? I'm wondering why this can't be done directly. CheckableNextCheckExtractor is just an "I know we do something similar somewhere else already" example, though there even seems to be something specifically for this in Boost directly: https://www.boost.org/doc/libs/1_87_0/libs/multi_index/doc/reference/key_extraction.html#mem_fun_synopsis

[yhabteab]

though there even seems to be something specifically for this in Boost directly: https://www.boost.org/doc/libs/1_87_0/libs/multi_index/doc/reference/key_extraction.html#mem_fun_synopsis

Yes, there's and I'm already making use of it here!

icinga2/lib/icinga/dependency.hpp

Lines 255 to 257 in a1175d1

	boost::multi_index::hashed_non_unique<
	boost::multi_index::const_mem_fun<DependencyGroup, const String&, &DependencyGroup::GetName>,
	std::hash<String>

What I'm trying to say is that if I do this as the above, I'll only be able to eliminate the Hash functor (at the cost of manually calling the method every time you want to perform some op with the identity), but not both of them, because the above index is a non-unique one and doesn't need a predicate, while the first one is a unique one and I need provide the predicate as the default implementation std::equal_to<> doesn't apply to the value type stored in the container.

[julianbrost]

That class needs more documentation/explanation what a group and its members are/represent.

[yhabteab]

Done!

[julianbrost]

I don't immediately see why there is such a difference in which values are used here. Conceptually, a dependency outside of a redundancy group should behave the same as if it was alone in its very own redundancy group, the time period should either be relevant for both types or for neither.

Also, is the String type only used to have references to different object types in the same member of that tuple? So why can't it be Checkable* instead of the name for example? If there's really a difference needed between redundancy groups and individual dependencies, std::variant could also be an option.

[yhabteab]

See the updated documentation!

[julianbrost]

What exactly is the inconsistency and how will it (not) show in practice?

[yhabteab]

See the updated docs!

[julianbrost]

What would happen if C1 had two more dependencies to new checkables P3 and P4 without any redundancy group? If I understand your explanation correctly, that would create a single new DependencyGroup that contains both new C1->P3 and C1->P4 dependencies. If so, I find it surprising that dependencies inside a redundancy are grouped by parent but those outside of any redundancy group are grouped by child.

[yhabteab]

If I understand your explanation correctly, that would create a single new DependencyGroup that contains both new C1->P3 and C1->P4 dependencies.

Exactly 👍!

If so, I find it surprising that dependencies inside a redundancy are grouped by parent but those outside of any redundancy group are grouped by child.

Please read the documentation for the CompositeKeyType and MemberValueType type aliases to understand why!