Fuzzing redisraft crash on updating log upon append entries

[ds-testing-user]

We are a group of researchers testing distributed protocol implementations. While testing redisraft, we encountered the following crash (with a 3 node cluster).

logs for node: 2

----- Stdout -----
1363727:C 14 Oct 2023 14:22:38.687 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
1363727:C 14 Oct 2023 14:22:38.687 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1363727:C 14 Oct 2023 14:22:38.687 * Redis version=255.255.255, bits=64, commit=e04ebdb8, modified=1, pid=1363727, just started
1363727:C 14 Oct 2023 14:22:38.687 * Configuration loaded
1363727:M 14 Oct 2023 14:22:38.687 * monotonic clock: POSIX clock_gettime
1363727:M 14 Oct 2023 14:22:38.688 * Running mode=standalone, port=5042.
1363727:M 14 Oct 2023 14:22:38.692 * <raft> RedisRaft version 255.255.255 [c259f3d2]
1363727:M 14 Oct 2023 14:22:38.696 * <raft> Initializing test network
1363727:M 14 Oct 2023 14:22:38.696 * <raft> Created test network client
1363727:M 14 Oct 2023 14:22:38.718 * <raft> Raft module loaded, state is 'loading'
1363727:M 14 Oct 2023 14:22:38.718 * Module 'raft' loaded from /Fuzzing/redisraft-fuzzing/redisraft.so
1363727:M 14 Oct 2023 14:22:38.719 * Server initialized
1363727:M 14 Oct 2023 14:22:38.719 . The AOF directory appendonlydir doesn't exist
1363727:M 14 Oct 2023 14:22:38.719 * Ready to accept connections tcp
1363727:M 14 Oct 2023 14:22:38.719 . 0 clients connected (0 replicas), 1746688 bytes in use

----- Stderr -----




logs for node: 3

----- Stdout -----
1363047:C 14 Oct 2023 14:22:36.644 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
1363047:C 14 Oct 2023 14:22:36.644 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1363047:C 14 Oct 2023 14:22:36.644 * Redis version=255.255.255, bits=64, commit=e04ebdb8, modified=1, pid=1363047, just started
1363047:C 14 Oct 2023 14:22:36.644 * Configuration loaded
1363047:M 14 Oct 2023 14:22:36.644 * monotonic clock: POSIX clock_gettime
1363047:M 14 Oct 2023 14:22:36.645 * Running mode=standalone, port=5043.
1363047:M 14 Oct 2023 14:22:36.648 * <raft> RedisRaft version 255.255.255 [c259f3d2]
1363047:M 14 Oct 2023 14:22:36.653 * <raft> Initializing test network
1363047:M 14 Oct 2023 14:22:36.654 * <raft> Created test network client
1363047:M 14 Oct 2023 14:22:36.678 * <raft> Raft module loaded, state is 'loading'
1363047:M 14 Oct 2023 14:22:36.678 * Module 'raft' loaded from /Fuzzing/redisraft-fuzzing/redisraft.so
1363047:M 14 Oct 2023 14:22:36.679 * Server initialized
1363047:M 14 Oct 2023 14:22:36.679 . The AOF directory appendonlydir doesn't exist
1363047:M 14 Oct 2023 14:22:36.679 * Ready to accept connections tcp
1363047:M 14 Oct 2023 14:22:36.679 . 0 clients connected (0 replicas), 1746848 bytes in use
1363047:M 14 Oct 2023 14:22:36.754 - Accepted 127.0.0.1:35662
1363047:M 14 Oct 2023 14:22:36.754 # Missing implement of connection type tls
1363047:M 14 Oct 2023 14:22:36.778 * <raft> Loading: Redis loading complete, snapshot NOT LOADED
1363047:M 14 Oct 2023 14:22:36.779 * <raft> Cluster Membership: term:0 index:0 nodes: id=3,voting=0,active=1,addr=-
1363047:M 14 Oct 2023 14:22:36.779 * <raft> Test network starting message polling thread
1363047:M 14 Oct 2023 14:22:36.780 * <raft> Cluster Membership: term:0 index:4 nodes: id=3,voting=0,active=1,addr=- id=1,voting=1,active=1,addr=localhost:5041
1363047:M 14 Oct 2023 14:22:36.780 * <raft> Cluster Membership: term:0 index:4 nodes: id=3,voting=0,active=1,addr=- id=1,voting=1,active=1,addr=localhost:5041 id=2,voting=0,active=1,addr=localhost:5042
1363047:M 14 Oct 2023 14:22:36.780 * <raft> Raft state after loading log: log_count=4, first_idx=1, current_idx=4, last_applied_idx=0
1363047:M 14 Oct 2023 14:22:36.780 . <raft> Raft term=1, vote=-1
1363047:M 14 Oct 2023 14:22:36.960 - Accepted 127.0.0.1:35666


=== REDIS BUG REPORT START: Cut & paste starting from here ===
1363047:M 14 Oct 2023 14:22:37.325 # Redis 255.255.255 crashed by signal: 6, si_code: -6
1363047:M 14 Oct 2023 14:22:37.325 # Crashed running the instruction at: 0x7f1c44f1ece1

------ STACK TRACE ------
EIP:
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x141)[0x7f1c44f1ece1]

Backtrace:
/lib/x86_64-linux-gnu/libpthread.so.0(+0x13140)[0x7f1c450cd140]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x141)[0x7f1c44f1ece1]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x123)[0x7f1c44f08537]
/lib/x86_64-linux-gnu/libc.so.6(+0x2240f)[0x7f1c44f0840f]
/lib/x86_64-linux-gnu/libc.so.6(+0x31662)[0x7f1c44f17662]
/Fuzzing/redisraft-fuzzing/redisraft.so(+0x3c7ac)[0x7f1c44e947ac]
/Fuzzing/redisraft-fuzzing/redisraft.so(raft_delete_entry_from_idx+0xa6)[0x7f1c44e959f6]
/Fuzzing/redisraft-fuzzing/redisraft.so(raft_recv_appendentries+0x2d7)[0x7f1c44e97137]
/Fuzzing/redisraft-fuzzing/redisraft.so(handleTestNetworkMessage+0x170)[0x7f1c44e8d500]
/Fuzzing/redisraft-fuzzing/redisraft.so(poll_redis_test_messages+0x89)[0x7f1c44e8d659]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7)[0x7f1c450c1ea7]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f1c44fe1a2f]

------ REGISTERS ------
1363047:M 14 Oct 2023 14:22:37.325 # 
RAX:0000000000000000 RBX:00007f1c38ffe700
RCX:00007f1c44f1ece1 RDX:0000000000000000
RDI:0000000000000002 RSI:00007f1c38ffd010
RBP:00007f1c4507f688 RSP:00007f1c38ffd010
R8 :0000000000000000 R9 :00007f1c38ffd010
R10:0000000000000008 R11:0000000000000246
R12:00007f1c44ec06e0 R13:0000000000000149
R14:00007f1c44ec0938 R15:000000000000000b
RIP:00007f1c44f1ece1 EFL:0000000000000246
CSGSFS:002b000000000033
1363047:M 14 Oct 2023 14:22:37.325 # (00007f1c38ffd01f) -> 0000000000000000
1363047:M 14 Oct 2023 14:22:37.325 # (00007f1c38ffd01e) -> 0000000000000000
1363047:M 14 Oct 2023 14:22:37.325 # (00007f1c38ffd01d) -> 0000000000000000
1363047:M 14 Oct 2023 14:22:37.325 # (00007f1c38ffd01c) -> 0000000000000000
1363047:M 14 Oct 2023 14:22:37.325 # (00007f1c38ffd01b) -> 0000000000000000
1363047:M 14 Oct 2023 14:22:37.325 # (00007f1c38ffd01a) -> 00007f1c14004c1c
1363047:M 14 Oct 2023 14:22:37.325 # (00007f1c38ffd019) -> 00007f1c14004af0
1363047:M 14 Oct 2023 14:22:37.325 # (00007f1c38ffd018) -> 00007f1c14004c1c
1363047:M 14 Oct 2023 14:22:37.325 # (00007f1c38ffd017) -> 00007f1c14004b9b
1363047:M 14 Oct 2023 14:22:37.325 # (00007f1c38ffd016) -> 00007f1c14004af0
1363047:M 14 Oct 2023 14:22:37.325 # (00007f1c38ffd015) -> 00007f1c14004af0
1363047:M 14 Oct 2023 14:22:37.325 # (00007f1c38ffd014) -> 00007f1c14004b55
1363047:M 14 Oct 2023 14:22:37.325 # (00007f1c38ffd013) -> 00007f1c14004af0
1363047:M 14 Oct 2023 14:22:37.325 # (00007f1c38ffd012) -> 00007f1cfbad8000
1363047:M 14 Oct 2023 14:22:37.325 # (00007f1c38ffd011) -> 00007f1c44f5f2ac
1363047:M 14 Oct 2023 14:22:37.325 # (00007f1c38ffd010) -> 0000000000000000

------ INFO OUTPUT ------
# Server
redis_version:255.255.255
redis_git_sha1:e04ebdb8
redis_git_dirty:1
redis_build_id:1f78dcc7b4fbdd25
redis_mode:standalone
os:Linux 5.15.111.1.amd64-smp x86_64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:10.2.1
process_id:1363047
process_supervised:no
run_id:ed023cedbded3a5ed87c91f0c3de62138bee0301
tcp_port:5043
server_time_usec:1697286157283397
uptime_in_seconds:1
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:2787341
executable:/Fuzzing/redis/src/redis-server
config_file:
io_threads_active:0
listener0:name=tcp,bind=0.0.0.0,port=5043

# Clients
connected_clients:2
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:20480
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
total_blocking_keys:0
total_blocking_keys_on_nokey:0

# Memory
used_memory:1828920
used_memory_human:1.74M
used_memory_rss:15843328
used_memory_rss_human:15.11M
used_memory_peak:1840312
used_memory_peak_human:1.76M
used_memory_peak_perc:99.38%
used_memory_overhead:1791688
used_memory_startup:1746704
used_memory_dataset:37232
used_memory_dataset_perc:45.29%
allocator_allocated:2446456
allocator_active:3112960
allocator_resident:13533184
total_system_memory:270386737152
total_system_memory_human:251.82G
used_memory_lua:31744
used_memory_vm_eval:31744
used_memory_lua_human:31.00K
used_memory_scripts_eval:0
number_of_cached_scripts:0
number_of_functions:0
number_of_libraries:0
used_memory_vm_functions:32768
used_memory_vm_total:64512
used_memory_vm_total_human:63.00K
used_memory_functions:184
used_memory_scripts:184
used_memory_scripts_human:184B
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.27
allocator_frag_bytes:666504
allocator_rss_ratio:4.35
allocator_rss_bytes:10420224
rss_overhead_ratio:1.17
rss_overhead_bytes:2310144
mem_fragmentation_ratio:8.68
mem_fragmentation_bytes:14017992
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_total_replication_buffers:0
mem_clients_slaves:0
mem_clients_normal:44800
mem_cluster_links:0
mem_aof_buffer:0
mem_allocator:jemalloc-5.3.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0

# Persistence
loading:0
async_loading:0
current_cow_peak:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1697286156
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_saves:0
rdb_last_cow_size:0
rdb_last_load_keys_expired:0
rdb_last_load_keys_loaded:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_rewrites:0
aof_rewrites_consecutive_failures:0
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

# Stats
total_connections_received:2
total_commands_processed:8
instantaneous_ops_per_sec:11
total_net_input_bytes:66
total_net_output_bytes:10
total_net_repl_input_bytes:0
total_net_repl_output_bytes:0
instantaneous_input_kbps:0.04
instantaneous_output_kbps:0.01
instantaneous_input_repl_kbps:0.00
instantaneous_output_repl_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:0
evicted_keys:0
evicted_clients:0
total_eviction_exceeded_time:0
current_eviction_exceeded_time:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
pubsubshard_channels:0
latest_fork_usec:0
total_forks:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
total_active_defrag_time:0
current_active_defrag_time:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:0
dump_payload_sanitizations:0
total_reads_processed:2
total_writes_processed:2
io_threaded_reads_processed:0
io_threaded_writes_processed:0
reply_buffer_shrinks:2
reply_buffer_expands:0
acl_access_denied_auth:0
acl_access_denied_cmd:0
acl_access_denied_key:0
acl_access_denied_channel:0

# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:3356d50b2b1de77cd9947aea90e4dc3e50e1dd78
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:0.022660
used_cpu_user:0.022660
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
used_cpu_sys_main_thread:0.010781
used_cpu_user_main_thread:0.016171

# Modules
module:name=raft,ver=1,api=1,filters=1,usedby=[],using=[],options=[]

# Commandstats
cmdstat_config|get:calls=3,usec=5,usec_per_call=1.67,rejected_calls=0,failed_calls=0
cmdstat_config|set:calls=1,usec=6,usec_per_call=6.00,rejected_calls=0,failed_calls=0
cmdstat_auth:calls=2,usec=2,usec_per_call=1.00,rejected_calls=0,failed_calls=0
cmdstat_command:calls=2,usec=1471,usec_per_call=735.50,rejected_calls=0,failed_calls=0

# Errorstats

# Latencystats
latency_percentiles_usec_config|get:p50=2.007,p99=2.007,p99.9=2.007
latency_percentiles_usec_config|set:p50=6.015,p99=6.015,p99.9=6.015
latency_percentiles_usec_auth:p50=1.003,p99=1.003,p99.9=1.003
latency_percentiles_usec_command:p50=696.319,p99=778.239,p99.9=778.239

# Cluster
cluster_enabled:0

# Keyspace

------ CLIENT LIST OUTPUT ------
id=8 addr=127.0.0.1:35662 laddr=127.0.0.1:5043 fd=11 name= age=1 idle=1 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=22400 events=r cmd=auth user=default redir=-1 resp=2 lib-name= lib-ver=
id=9 addr=127.0.0.1:35666 laddr=127.0.0.1:5043 fd=14 name= age=1 idle=1 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=22400 events=r cmd=auth user=default redir=-1 resp=2 lib-name= lib-ver=

------ MODULES INFO OUTPUT ------
# raft_version
raft_version:255.255.255
raft_git_sha1:c259f3d2

# raft_general
raft_dbid:431be1a10a047abb4287dc3b070372d9
raft_node_id:3
raft_state:up
raft_role:follower
raft_is_voting:no
raft_voted_for:-1
raft_leader_id:1
raft_current_term:2
raft_num_nodes:3
raft_num_voting_nodes:1
raft_node1:id=1,state=connected,voting=yes,addr=localhost,port=5041,last_conn_secs=0,conn_errors=1,conn_oks=1
raft_node2:id=2,state=connected,voting=no,addr=localhost,port=5042,last_conn_secs=0,conn_errors=0,conn_oks=1

# raft_log
raft_log_entries:3
raft_current_index:3
raft_commit_index:0
raft_last_applied_index:0
raft_file_size:845
raft_cache_memory_size:0
raft_cache_entries:0
raft_client_attached_entries:0
raft_fsync_count:0
raft_fsync_max_microseconds:0
raft_fsync_avg_microseconds:0

# raft_snapshot
raft_snapshot_filename:redis3.rdb
raft_snapshot_last_idx:0
raft_snapshot_last_term:0
raft_snapshot_size:0
raft_snapshot_time_secs:-1
raft_snapshots_created:0
raft_snapshots_received:0
raft_snapshot_in_progress:no
raft_snapshot_in_progress_last_idx:-1
raft_snapshot_in_progress_last_term:-1

# raft_clients
raft_proxy_reqs:0
raft_proxy_failed_reqs:0
raft_proxy_failed_responses:0
raft_proxy_outstanding_reqs:0

# raft_stats
raft_appendreq_received:0
raft_appendreq_with_entry_received:0
raft_snapshotreq_received:0
raft_exec_throttled:0
raft_num_sessions:0

------ CONFIG DEBUG OUTPUT ------
lazyfree-lazy-user-flush no
lazyfree-lazy-user-del no
io-threads-do-reads no
lazyfree-lazy-eviction no
slave-read-only yes
repl-diskless-load disabled
sanitize-dump-payload no
io-threads 1
client-query-buffer-limit 1gb
repl-diskless-sync yes
proto-max-bulk-len 512mb
lazyfree-lazy-expire no
activedefrag no
list-compress-depth 0
lazyfree-lazy-server-del no
replica-read-only yes

------ FAST MEMORY TEST ------
1363047:M 14 Oct 2023 14:22:37.325 # main thread terminated
1363047:M 14 Oct 2023 14:22:37.325 # Bio worker thread #0 terminated
1363047:M 14 Oct 2023 14:22:37.325 # Bio worker thread #1 terminated
1363047:M 14 Oct 2023 14:22:37.325 # Bio worker thread #2 terminated

Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

------ DUMPING CODE AROUND EIP ------
Symbol: gsignal (base: 0x7f1c44f1eba0)
Module: /lib/x86_64-linux-gnu/libc.so.6 (base 0x7f1c44ee6000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=0x7f1c44f1eba0 -D -b binary -m i386:x86-64 /tmp/dump.bin
------
1363047:M 14 Oct 2023 14:22:37.326 # dump of function (hexdump of 449 bytes):
4881ec180100004189f841ba0800000031ff64488b042528000000488984240801000031c04989e1488db4248000000048b8ffffff7ffeffffff48898424800000004c89cab80e00000048c7842488000000ffffffff48c7842490000000ffffffff48c7842498000000ffffffff48c78424a0000000ffffffff48c78424a8000000ffffffff48c78424b0000000ffffffff48c78424b8000000ffffffff48c78424c0000000ffffffff48c78424c8000000ffffffff48c78424d0000000ffffffff48c78424d8000000ffffffff48c78424e0000000ffffffff48c78424e8000000ffffffff48c78424f0000000ffffffff48c78424f8000000ffffffff0f05b92700000089c80f054889c7b8ba0000000f0589c64489c2b8ea0000000f05483d00f0ffff77394189c041ba0800000031d24c89cebf02000000b80e0000000f05488b84240801000064482b04252800000075204489c04881c418010000c390488b1569511900f7d841b8ffffffff648902ebb6e8f7270d000f1f800000000085ff780cf7dfe9550300000f1f440000488b053951190064c70016000000b8ffffffffc3662e0f1f84000000000066909066662e0f1f8400000000000f1f400048
Function at 0x7f1c44ff1510 is __stack_chk_fail

=== REDIS BUG REPORT END. Make sure to include from START to END. ===

       Please report the crash by opening an issue on github:

           http://github.com/redis/redis/issues

  If a Redis module was involved, please open in the module's repo instead.

  Suspect RAM error? Use redis-server --test-memory to verify it.

  Some other issues could be detected by redis-server --check-system

----- Stderr -----



logs for node: 1

----- Stdout -----
1363647:C 14 Oct 2023 14:22:38.555 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
1363647:C 14 Oct 2023 14:22:38.555 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1363647:C 14 Oct 2023 14:22:38.555 * Redis version=255.255.255, bits=64, commit=e04ebdb8, modified=1, pid=1363647, just started
1363647:C 14 Oct 2023 14:22:38.555 * Configuration loaded
1363647:M 14 Oct 2023 14:22:38.555 * monotonic clock: POSIX clock_gettime
1363647:M 14 Oct 2023 14:22:38.556 * Running mode=standalone, port=5041.
1363647:M 14 Oct 2023 14:22:38.560 * <raft> RedisRaft version 255.255.255 [c259f3d2]
1363647:M 14 Oct 2023 14:22:38.563 * <raft> Initializing test network
1363647:M 14 Oct 2023 14:22:38.564 * <raft> Created test network client
1363647:M 14 Oct 2023 14:22:38.608 * <raft> Raft module loaded, state is 'loading'
1363647:M 14 Oct 2023 14:22:38.608 * Module 'raft' loaded from /Fuzzing/redisraft-fuzzing/redisraft.so
1363647:M 14 Oct 2023 14:22:38.609 * Server initialized
1363647:M 14 Oct 2023 14:22:38.609 . The AOF directory appendonlydir doesn't exist
1363647:M 14 Oct 2023 14:22:38.609 * Ready to accept connections tcp
1363647:M 14 Oct 2023 14:22:38.609 . 0 clients connected (0 replicas), 1746688 bytes in use
1363647:M 14 Oct 2023 14:22:38.709 * <raft> Loading: Redis loading complete, snapshot NOT LOADED
1363647:M 14 Oct 2023 14:22:38.709 * <raft> Cluster Membership: term:0 index:0 nodes: id=1,voting=0,active=1,addr=-
1363647:M 14 Oct 2023 14:22:38.710 * <raft> Test network starting message polling thread
1363647:M 14 Oct 2023 14:22:38.711 * <raft> Cluster Membership: term:0 index:7 nodes: id=1,voting=1,active=1,addr=- id=2,voting=0,active=1,addr=localhost:5042
1363647:M 14 Oct 2023 14:22:38.711 * <raft> Cluster Membership: term:0 index:7 nodes: id=1,voting=1,active=1,addr=- id=2,voting=0,active=1,addr=localhost:5042 id=3,voting=0,active=1,addr=localhost:5043
1363647:M 14 Oct 2023 14:22:38.711 * <raft> Raft state after loading log: log_count=7, first_idx=1, current_idx=7, last_applied_idx=0
1363647:M 14 Oct 2023 14:22:38.711 . <raft> Raft term=2, vote=-1

----- Stderr -----