Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fuzzing redisraft crash invoking callRaftPeriodic #646

Open
ds-testing-user opened this issue Oct 19, 2023 · 0 comments
Open

Fuzzing redisraft crash invoking callRaftPeriodic #646

ds-testing-user opened this issue Oct 19, 2023 · 0 comments

Comments

@ds-testing-user
Copy link

We are a group of researchers testing distributed protocol implementations. While testing redisraft, we encountered the following crash (with a 3 node cluster). The crash occurs when invoking the redisraft periodic function.

logs for node: 3

----- Stdout -----
1872413:C 18 Oct 2023 06:23:07.271 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
1872413:C 18 Oct 2023 06:23:07.271 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1872413:C 18 Oct 2023 06:23:07.271 * Redis version=255.255.255, bits=64, commit=e04ebdb8, modified=1, pid=1872413, just started
1872413:C 18 Oct 2023 06:23:07.271 * Configuration loaded
1872413:M 18 Oct 2023 06:23:07.272 * monotonic clock: POSIX clock_gettime
1872413:M 18 Oct 2023 06:23:07.272 * Running mode=standalone, port=5063.
1872413:M 18 Oct 2023 06:23:07.277 * <raft> RedisRaft version 255.255.255 [c259f3d2]
1872413:M 18 Oct 2023 06:23:07.281 * <raft> Initializing test network
1872413:M 18 Oct 2023 06:23:07.282 * <raft> Created test network client
1872413:M 18 Oct 2023 06:23:07.287 * <raft> Raft module loaded, state is 'loading'
1872413:M 18 Oct 2023 06:23:07.287 * Module 'raft' loaded from /Fuzzing/redisraft-fuzzing/redisraft.so
1872413:M 18 Oct 2023 06:23:07.288 * Server initialized
1872413:M 18 Oct 2023 06:23:07.288 . The AOF directory appendonlydir doesn't exist
1872413:M 18 Oct 2023 06:23:07.288 * Ready to accept connections tcp
1872413:M 18 Oct 2023 06:23:07.288 . 0 clients connected (0 replicas), 1746624 bytes in use
1872413:M 18 Oct 2023 06:23:07.291 - Accepted 127.0.0.1:46024
1872413:M 18 Oct 2023 06:23:07.291 # Missing implement of connection type tls
1872413:M 18 Oct 2023 06:23:07.388 * <raft> Loading: Redis loading complete, snapshot NOT LOADED
1872413:M 18 Oct 2023 06:23:07.388 * <raft> Cluster Membership: term:0 index:0 nodes: id=3,voting=0,active=1,addr=-
1872413:M 18 Oct 2023 06:23:07.389 * <raft> Test network starting message polling thread


=== REDIS BUG REPORT START: Cut & paste starting from here ===
1872413:M 18 Oct 2023 06:23:07.389 # Redis 255.255.255 crashed by signal: 6, si_code: -6
1872413:M 18 Oct 2023 06:23:07.389 # Crashed running the instruction at: 0x7f2b4b73ece1

------ STACK TRACE ------
EIP:
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x141)[0x7f2b4b73ece1]

Backtrace:
/lib/x86_64-linux-gnu/libpthread.so.0(+0x13140)[0x7f2b4b8ed140]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x141)[0x7f2b4b73ece1]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x123)[0x7f2b4b728537]
/lib/x86_64-linux-gnu/libc.so.6(+0x2240f)[0x7f2b4b72840f]
/lib/x86_64-linux-gnu/libc.so.6(+0x31662)[0x7f2b4b737662]
/Fuzzing/redisraft-fuzzing/redisraft.so(+0x3cc2a)[0x7f2b4b6b4c2a]
/Fuzzing/redisraft-fuzzing/redisraft.so(callRaftPeriodic+0x477)[0x7f2b4b6a51b7]
/Fuzzing/redis/src/redis-server 0.0.0.0:5063(moduleTimerHandler+0x21c)[0x55fb7aa2598c]
/Fuzzing/redis/src/redis-server 0.0.0.0:5063(+0x81705)[0x55fb7a92f705]
/Fuzzing/redis/src/redis-server 0.0.0.0:5063(aeMain+0x1c8)[0x55fb7a939cb8]
/Fuzzing/redis/src/redis-server 0.0.0.0:5063(main+0x3cd)[0x55fb7a92ee2d]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea)[0x7f2b4b729d0a]
/Fuzzing/redis/src/redis-server 0.0.0.0:5063(_start+0x2a)[0x55fb7a92f5aa]

------ REGISTERS ------
1872413:M 18 Oct 2023 06:23:07.389 # 
RAX:0000000000000000 RBX:00007f2b4b705680
RCX:00007f2b4b73ece1 RDX:0000000000000000
RDI:0000000000000002 RSI:00007ffc08cb0080
RBP:00007f2b4b89f688 RSP:00007ffc08cb0080
R8 :0000000000000000 R9 :00007ffc08cb0080
R10:0000000000000008 R11:0000000000000246
R12:00007f2b4b6e06e0 R13:00000000000006a6
R14:00007f2b4b6e101c R15:0000000000000000
RIP:00007f2b4b73ece1 EFL:0000000000000246
CSGSFS:002b000000000033
1872413:M 18 Oct 2023 06:23:07.389 # (00007ffc08cb008f) -> 0000000000000000
1872413:M 18 Oct 2023 06:23:07.389 # (00007ffc08cb008e) -> 0000000000000000
1872413:M 18 Oct 2023 06:23:07.389 # (00007ffc08cb008d) -> 0000000000000000
1872413:M 18 Oct 2023 06:23:07.389 # (00007ffc08cb008c) -> 0000000000000000
1872413:M 18 Oct 2023 06:23:07.389 # (00007ffc08cb008b) -> 0000000000000000
1872413:M 18 Oct 2023 06:23:07.389 # (00007ffc08cb008a) -> 000055fb7af12adc
1872413:M 18 Oct 2023 06:23:07.389 # (00007ffc08cb0089) -> 000055fb7af129b0
1872413:M 18 Oct 2023 06:23:07.389 # (00007ffc08cb0088) -> 000055fb7af12adc
1872413:M 18 Oct 2023 06:23:07.389 # (00007ffc08cb0087) -> 000055fb7af12a55
1872413:M 18 Oct 2023 06:23:07.389 # (00007ffc08cb0086) -> 000055fb7af129b0
1872413:M 18 Oct 2023 06:23:07.389 # (00007ffc08cb0085) -> 000055fb7af129b0
1872413:M 18 Oct 2023 06:23:07.389 # (00007ffc08cb0084) -> 000055fb7af12a15
1872413:M 18 Oct 2023 06:23:07.389 # (00007ffc08cb0083) -> 000055fb7af129b0
1872413:M 18 Oct 2023 06:23:07.389 # (00007ffc08cb0082) -> 00007f2bfbad8000
1872413:M 18 Oct 2023 06:23:07.389 # (00007ffc08cb0081) -> 00007f2b4b77f2ac
1872413:M 18 Oct 2023 06:23:07.389 # (00007ffc08cb0080) -> 0000000000000000

------ INFO OUTPUT ------
# Server
redis_version:255.255.255
redis_git_sha1:e04ebdb8
redis_git_dirty:1
redis_build_id:1f78dcc7b4fbdd25
redis_mode:standalone
os:Linux 5.15.111.1.amd64-smp x86_64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:10.2.1
process_id:1872413
process_supervised:no
run_id:297c54aaff9cf4ddde56364dde26550ff55a1eb7
tcp_port:5063
server_time_usec:1697602987388114
uptime_in_seconds:0
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:3104171
executable:/Fuzzing/redis/src/redis-server
config_file:
io_threads_active:0
listener0:name=tcp,bind=0.0.0.0,port=5063

# Clients
connected_clients:1
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:0
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
total_blocking_keys:0
total_blocking_keys_on_nokey:0

# Memory
used_memory:1816840
used_memory_human:1.73M
used_memory_rss:15421440
used_memory_rss_human:14.71M
used_memory_peak:1816840
used_memory_peak_human:1.73M
used_memory_peak_perc:100.39%
used_memory_overhead:1746664
used_memory_startup:1746480
used_memory_dataset:70176
used_memory_dataset_perc:99.74%
allocator_allocated:2051792
allocator_active:2580480
allocator_resident:10817536
total_system_memory:270386737152
total_system_memory_human:251.82G
used_memory_lua:31744
used_memory_vm_eval:31744
used_memory_lua_human:31.00K
used_memory_scripts_eval:0
number_of_cached_scripts:0
number_of_functions:0
number_of_libraries:0
used_memory_vm_functions:32768
used_memory_vm_total:64512
used_memory_vm_total_human:63.00K
used_memory_functions:184
used_memory_scripts:184
used_memory_scripts_human:184B
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.26
allocator_frag_bytes:528688
allocator_rss_ratio:4.19
allocator_rss_bytes:8237056
rss_overhead_ratio:1.43
rss_overhead_bytes:4603904
mem_fragmentation_ratio:8.83
mem_fragmentation_bytes:13674816
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_total_replication_buffers:0
mem_clients_slaves:0
mem_clients_normal:0
mem_cluster_links:0
mem_aof_buffer:0
mem_allocator:jemalloc-5.3.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0

# Persistence
loading:0
async_loading:0
current_cow_peak:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1697602987
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_saves:0
rdb_last_cow_size:0
rdb_last_load_keys_expired:0
rdb_last_load_keys_loaded:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_rewrites:0
aof_rewrites_consecutive_failures:0
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

# Stats
total_connections_received:1
total_commands_processed:7
instantaneous_ops_per_sec:23
total_net_input_bytes:33
total_net_output_bytes:5
total_net_repl_input_bytes:0
total_net_repl_output_bytes:0
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
instantaneous_input_repl_kbps:0.00
instantaneous_output_repl_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:0
evicted_keys:0
evicted_clients:0
total_eviction_exceeded_time:0
current_eviction_exceeded_time:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
pubsubshard_channels:0
latest_fork_usec:0
total_forks:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
total_active_defrag_time:0
current_active_defrag_time:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:0
dump_payload_sanitizations:0
total_reads_processed:1
total_writes_processed:1
io_threaded_reads_processed:0
io_threaded_writes_processed:0
reply_buffer_shrinks:0
reply_buffer_expands:0
acl_access_denied_auth:0
acl_access_denied_cmd:0
acl_access_denied_key:0
acl_access_denied_channel:0

# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:f0f1bca43ca33dba06c09969859929d9c522f174
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:0.004488
used_cpu_user:0.011520
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
used_cpu_sys_main_thread:0.003735
used_cpu_user_main_thread:0.011205

# Modules
module:name=raft,ver=1,api=1,filters=1,usedby=[],using=[],options=[]

# Commandstats
cmdstat_auth:calls=1,usec=1,usec_per_call=1.00,rejected_calls=0,failed_calls=0
cmdstat_command:calls=2,usec=1590,usec_per_call=795.00,rejected_calls=0,failed_calls=0
cmdstat_config|get:calls=3,usec=4,usec_per_call=1.33,rejected_calls=0,failed_calls=0
cmdstat_config|set:calls=1,usec=6,usec_per_call=6.00,rejected_calls=0,failed_calls=0

# Errorstats

# Latencystats
latency_percentiles_usec_auth:p50=1.003,p99=1.003,p99.9=1.003
latency_percentiles_usec_command:p50=741.375,p99=851.967,p99.9=851.967
latency_percentiles_usec_config|get:p50=1.003,p99=2.007,p99.9=2.007
latency_percentiles_usec_config|set:p50=6.015,p99=6.015,p99.9=6.015

# Cluster
cluster_enabled:0

# Keyspace

------ CLIENT LIST OUTPUT ------
id=8 addr=127.0.0.1:46024 laddr=127.0.0.1:5063 fd=11 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=auth user=default redir=-1 resp=2 lib-name= lib-ver=

------ MODULES INFO OUTPUT ------
# raft_version
raft_version:255.255.255
raft_git_sha1:c259f3d2

# raft_general
raft_dbid:eabb81d268c2c42c1a747ecf88625ca7
raft_node_id:3
raft_state:up
raft_role:follower
raft_is_voting:no
raft_voted_for:-1
raft_leader_id:-1
raft_current_term:0
raft_num_nodes:1
raft_num_voting_nodes:0

# raft_log
raft_log_entries:0
raft_current_index:0
raft_commit_index:0
raft_last_applied_index:0
raft_file_size:101
raft_cache_memory_size:0
raft_cache_entries:0
raft_client_attached_entries:0
raft_fsync_count:0
raft_fsync_max_microseconds:0
raft_fsync_avg_microseconds:0

# raft_snapshot
raft_snapshot_filename:redis3.rdb
raft_snapshot_last_idx:0
raft_snapshot_last_term:0
raft_snapshot_size:0
raft_snapshot_time_secs:-1
raft_snapshots_created:0
raft_snapshots_received:0
raft_snapshot_in_progress:no
raft_snapshot_in_progress_last_idx:-1
raft_snapshot_in_progress_last_term:-1

# raft_clients
raft_proxy_reqs:0
raft_proxy_failed_reqs:0
raft_proxy_failed_responses:0
raft_proxy_outstanding_reqs:0

# raft_stats
raft_appendreq_received:0
raft_appendreq_with_entry_received:0
raft_snapshotreq_received:0
raft_exec_throttled:0
raft_num_sessions:0

------ CONFIG DEBUG OUTPUT ------
proto-max-bulk-len 512mb
lazyfree-lazy-user-del no
lazyfree-lazy-eviction no
repl-diskless-sync yes
lazyfree-lazy-expire no
io-threads-do-reads no
lazyfree-lazy-server-del no
repl-diskless-load disabled
lazyfree-lazy-user-flush no
activedefrag no
client-query-buffer-limit 1gb
io-threads 1
sanitize-dump-payload no
replica-read-only yes
slave-read-only yes
list-compress-depth 0

------ FAST MEMORY TEST ------
1872413:M 18 Oct 2023 06:23:07.390 # Bio worker thread #0 terminated
1872413:M 18 Oct 2023 06:23:07.390 # Bio worker thread #1 terminated
1872413:M 18 Oct 2023 06:23:07.390 # Bio worker thread #2 terminated
*** Preparing to test memory region 55fb7ac05000 (2273280 bytes)
*** Preparing to test memory region 55fb7aecb000 (405504 bytes)
*** Preparing to test memory region 7f2b30000000 (135168 bytes)
*** Preparing to test memory region 7f2b38000000 (135168 bytes)
*** Preparing to test memory region 7f2b3e200000 (8388608 bytes)
*** Preparing to test memory region 7f2b3ea00000 (4194304 bytes)
*** Preparing to test memory region 7f2b3efff000 (8388608 bytes)
*** Preparing to test memory region 7f2b3f800000 (8388608 bytes)
*** Preparing to test memory region 7f2b40000000 (6291456 bytes)
*** Preparing to test memory region 7f2b407fa000 (8388608 bytes)
*** Preparing to test memory region 7f2b40ffb000 (8388608 bytes)
*** Preparing to test memory region 7f2b417fc000 (8388608 bytes)
*** Preparing to test memory region 7f2b41ffd000 (8388608 bytes)
*** Preparing to test memory region 7f2b427fe000 (8388608 bytes)
*** Preparing to test memory region 7f2b42fff000 (8388608 bytes)
*** Preparing to test memory region 7f2b43800000 (8388608 bytes)
*** Preparing to test memory region 7f2b44000000 (135168 bytes)
*** Preparing to test memory region 7f2b480f7000 (8388608 bytes)
*** Preparing to test memory region 7f2b488f8000 (8388608 bytes)
*** Preparing to test memory region 7f2b490f9000 (8388608 bytes)
*** Preparing to test memory region 7f2b498f9000 (2621440 bytes)
*** Preparing to test memory region 7f2b49c0a000 (8192 bytes)
*** Preparing to test memory region 7f2b49c4a000 (4096 bytes)
*** Preparing to test memory region 7f2b4a278000 (8192 bytes)
*** Preparing to test memory region 7f2b4a4cb000 (8192 bytes)
*** Preparing to test memory region 7f2b4a623000 (4096 bytes)
*** Preparing to test memory region 7f2b4ae00000 (8388608 bytes)
*** Preparing to test memory region 7f2b4b6fb000 (45056 bytes)
*** Preparing to test memory region 7f2b4b8d6000 (16384 bytes)
*** Preparing to test memory region 7f2b4b8f8000 (16384 bytes)
*** Preparing to test memory region 7f2b4ba46000 (8192 bytes)
*** Preparing to test memory region 7f2b4ba49000 (8192 bytes)
*** Preparing to test memory region 7f2b4ba67000 (4096 bytes)
*** Preparing to test memory region 7f2b4baa8000 (4096 bytes)
*** Preparing to test memory region 7f2b4baab000 (4096 bytes)
.O.O.O.1872413:M 18 Oct 2023 06:23:07.398 # Redis 255.255.255 crashed by signal: 11, si_code: 128
1872413:M 18 Oct 2023 06:23:07.398 # Accessing address: (nil)
1872413:M 18 Oct 2023 06:23:07.398 # Crashed running the instruction at: 0x7f2b4b78be48

------ STACK TRACE ------
OEIP:
./lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x118)[0x7f2b4b78be48]

Backtrace:
/lib/x86_64-linux-gnu/libpthread.so.0(+0x13140)[0x7f2b4b8ed140]
/lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x118)[0x7f2b4b78be48]
/Tools/json-c/lib/libjson-c.so.5(+0x5385)[0x7f2b4ba6d385]
/Tools/json-c/lib/libjson-c.so.5(json_object_new_object+0x1e)[0x7f2b4ba6da56]
/Fuzzing/redisraft-fuzzing/redisraft.so(serializeAEResp+0xf)[0x7f2b4b6acbaf]
/Fuzzing/redisraft-fuzzing/redisraft.so(testNetworkSendAppendEntriesResponse+0x22)[0x7f2b4b6ad1d2]
/Fuzzing/redisraft-fuzzing/redisraft.so(handleTestNetworkMessage+0x185)[0x7f2b4b6ad515]
/Fuzzing/redisraft-fuzzing/redisraft.so(poll_redis_test_messages+0x89)[0x7f2b4b6ad659]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7)[0x7f2b4b8e1ea7]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f2b4b801a2f]

------ REGISTERS ------
1872413:M 18 Oct 2023 06:23:07.398 # 
RAX:00007f2b38001290 RBX:0000000000000002
RCX:0000000000000012 RDX:000000000000aaa9
RDI:0000000000000030 RSI:0000000000000030
RBP:0000000000000030 RSP:00007f2b3f7fd2e0
R8 :aaaaaaaaaaaaaaaa R9 :0000000000000001
R10:0000000000000000 R11:0000000000000000
R12:fffffffffffff510 R13:00007f2b4b6fcf00
R14:0000000000000000 R15:00007f2b49995800
RIP:00007f2b4b78be48 EFL:0000000000010206
CSGSFS:002b000000000033
1872413:M 18 Oct 2023 06:23:07.398 # (00007f2b3f7fd2ef) -> 00007f2b4b6acbaf
1872413:M 18 Oct 2023 06:23:07.398 # (00007f2b3f7fd2ee) -> 0000000000000001
1872413:M 18 Oct 2023 06:23:07.398 # (00007f2b3f7fd2ed) -> 0000000000000008
1872413:M 18 Oct 2023 06:23:07.398 # (00007f2b3f7fd2ec) -> 0000000000000027
1872413:M 18 Oct 2023 06:23:07.398 # (00007f2b3f7fd2eb) -> 00007f2b4ba6da56
1872413:M 18 Oct 2023 06:23:07.398 # (00007f2b3f7fd2ea) -> 00007f2b3f7fd350
1872413:M 18 Oct 2023 06:23:07.398 # (00007f2b3f7fd2e9) -> 00007f2b4b6b7221
1872413:M 18 Oct 2023 06:23:07.399 # (00007f2b3f7fd2e8) -> 00007f2b49995800
1872413:M 18 Oct 2023 06:23:07.399 # (00007f2b3f7fd2e7) -> 0000000400000001
1872413:M 18 Oct 2023 06:23:07.399 # (00007f2b3f7fd2e6) -> 0000000000000030
1872413:M 18 Oct 2023 06:23:07.399 # (00007f2b3f7fd2e5) -> 00007f2b4ba6d73d
1872413:M 18 Oct 2023 06:23:07.399 # (00007f2b3f7fd2e4) -> 00007f2b49995800
1872413:M 18 Oct 2023 06:23:07.399 # (00007f2b3f7fd2e3) -> 00007f2b4ba6d385
1872413:M 18 Oct 2023 06:23:07.399 # (00007f2b3f7fd2e2) -> 00007f2b3f7fd388
1872413:M 18 Oct 2023 06:23:07.399 # (00007f2b3f7fd2e1) -> 00007f2b3f7fd330
1872413:M 18 Oct 2023 06:23:07.399 # (00007f2b3f7fd2e0) -> 00007f2b3f7fd3c0

------ INFO OUTPUT ------
# Server
redis_version:255.255.255
redis_git_sha1:e04ebdb8
redis_git_dirty:1
redis_build_id:1f78dcc7b4fbdd25
redis_mode:standalone
os:Linux 5.15.111.1.amd64-smp x86_64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:10.2.1
process_id:1872413
process_supervised:no
run_id:297c54aaff9cf4ddde56364dde26550ff55a1eb7
tcp_port:5063
server_time_usec:1697602987388114
uptime_in_seconds:0
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:3104171
executable:/Fuzzing/redis/src/redis-server
config_file:
io_threads_active:0
listener0:name=tcp,bind=0.0.0.0,port=5063

# Clients
connected_clients:1
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:0
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
total_blocking_keys:0
total_blocking_keys_on_nokey:0

# Memory
used_memory:1816880
used_memory_human:1.73M
used_memory_rss:15421440
used_memory_rss_human:14.71M
used_memory_peak:1816880
used_memory_peak_human:1.73M
used_memory_peak_perc:100.00%
used_memory_overhead:1746664
used_memory_startup:1746480
used_memory_dataset:70216
used_memory_dataset_perc:99.74%
allocator_allocated:2051792
allocator_active:2580480
allocator_resident:10817536
total_system_memory:270386737152
total_system_memory_human:251.82G
used_memory_lua:31744
used_memory_vm_eval:31744
used_memory_lua_human:31.00K
used_memory_scripts_eval:0
number_of_cached_scripts:0
number_of_functions:0
number_of_libraries:0
used_memory_vm_functions:32768
used_memory_vm_total:64512
used_memory_vm_total_human:63.00K
used_memory_functions:184
used_memory_scripts:184
used_memory_scripts_human:184B
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.26
allocator_frag_bytes:528688
allocator_rss_ratio:4.19
allocator_rss_bytes:8237056
rss_overhead_ratio:1.43
rss_overhead_bytes:4603904
mem_fragmentation_ratio:8.83
mem_fragmentation_bytes:13674816
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_total_replication_buffers:0
mem_clients_slaves:0
mem_clients_normal:0
mem_cluster_links:0
mem_aof_buffer:0
mem_allocator:jemalloc-5.3.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0

# Persistence
loading:0
async_loading:0
current_cow_peak:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1697602987
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_saves:0
rdb_last_cow_size:0
rdb_last_load_keys_expired:0
rdb_last_load_keys_loaded:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_rewrites:0
aof_rewrites_consecutive_failures:0
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

# Stats
total_connections_received:1
total_commands_processed:7
instantaneous_ops_per_sec:23
total_net_input_bytes:33
total_net_output_bytes:5
total_net_repl_input_bytes:0
total_net_repl_output_bytes:0
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
instantaneous_input_repl_kbps:0.00
instantaneous_output_repl_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:0
evicted_keys:0
evicted_clients:0
total_eviction_exceeded_time:0
current_eviction_exceeded_time:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
pubsubshard_channels:0
latest_fork_usec:0
total_forks:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
total_active_defrag_time:0
current_active_defrag_time:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:0
dump_payload_sanitizations:0
total_reads_processed:1
total_writes_processed:1
io_threaded_reads_processed:0
io_threaded_writes_processed:0
reply_buffer_shrinks:0
reply_buffer_expands:0
acl_access_denied_auth:0
acl_access_denied_cmd:0
acl_access_denied_key:0
acl_access_denied_channel:0

# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:f0f1bca43ca33dba06c09969859929d9c522f174
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:0.004488
used_cpu_user:0.019753
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
used_cpu_sys_main_thread:0.000000
used_cpu_user_main_thread:0.000725

# Modules
module:name=raft,ver=1,api=1,filters=1,usedby=[],using=[],options=[]

# Commandstats
cmdstat_auth:calls=1,usec=1,usec_per_call=1.00,rejected_calls=0,failed_calls=0
cmdstat_command:calls=2,usec=1590,usec_per_call=795.00,rejected_calls=0,failed_calls=0
cmdstat_config|get:calls=3,usec=4,usec_per_call=1.33,rejected_calls=0,failed_calls=0
cmdstat_config|set:calls=1,usec=6,usec_per_call=6.00,rejected_calls=0,failed_calls=0

# Errorstats

# Latencystats
latency_percentiles_usec_auth:p50=1.003,p99=1.003,p99.9=1.003
latency_percentiles_usec_command:p50=741.375,p99=851.967,p99.9=851.967
latency_percentiles_usec_config|get:p50=1.003,p99=2.007,p99.9=2.007
latency_percentiles_usec_config|set:p50=6.015,p99=6.015,p99.9=6.015

# Cluster
cluster_enabled:0

# Keyspace

------ CLIENT LIST OUTPUT ------
id=8 addr=127.0.0.1:46024 laddr=127.0.0.1:5063 fd=11 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=auth user=default redir=-1 resp=2 lib-name= lib-ver=

------ MODULES INFO OUTPUT ------
# raft_version
raft_version:255.255.255
raft_git_sha1:c259f3d2

# raft_general
raft_dbid:eabb81d268c2c42c1a747ecf88625ca7
raft_node_id:3
raft_state:up
raft_role:follower
raft_is_voting:no
raft_voted_for:-1
raft_leader_id:1
raft_current_term:1
raft_num_nodes:1
raft_num_voting_nodes:0

# raft_log
raft_log_entries:0
raft_current_index:0
raft_commit_index:0
raft_last_applied_index:0
raft_file_size:101
raft_cache_memory_size:0
raft_cache_entries:0
raft_client_attached_entries:0
raft_fsync_count:0
raft_fsync_max_microseconds:0
raft_fsync_avg_microseconds:0

# raft_snapshot
raft_snapshot_filename:redis3.rdb
raft_snapshot_last_idx:0
raft_snapshot_last_term:0
raft_snapshot_size:0
raft_snapshot_time_secs:-1
raft_snapshots_created:0
raft_snapshots_received:0
raft_snapshot_in_progress:no
raft_snapshot_in_progress_last_idx:-1
raft_snapshot_in_progress_last_term:-1

# raft_clients
raft_proxy_reqs:0
raft_proxy_failed_reqs:0
raft_proxy_failed_responses:0
raft_proxy_outstanding_reqs:0

# raft_stats
raft_appendreq_received:0
raft_appendreq_with_entry_received:0
raft_snapshotreq_received:0
raft_exec_throttled:0
raft_num_sessions:0

------ CONFIG DEBUG OUTPUT ------
proto-max-bulk-len 512mb
lazyfree-lazy-user-del no
lazyfree-lazy-eviction no
repl-diskless-sync yes
lazyfree-lazy-expire no
io-threads-do-reads no
lazyfree-lazy-server-del no
repl-diskless-load disabled
lazyfree-lazy-user-flush no
activedefrag no
client-query-buffer-limit 1gb
io-threads 1
sanitize-dump-payload no
replica-read-only yes
slave-read-only yes
list-compress-depth 0

------ FAST MEMORY TEST ------
O.O.O.O.O.E.E.E.E.E.E.E.E.E.E.E.E.E.E.E.E.E.E.E.E.E.E.E.E.E.E
!!! MEMORY ERROR DETECTED! Check your memory ASAP !!!

------ DUMPING CODE AROUND EIP ------
Symbol: gsignal (base: 0x7f2b4b73eba0)
Module: /lib/x86_64-linux-gnu/libc.so.6 (base 0x7f2b4b706000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=0x7f2b4b73eba0 -D -b binary -m i386:x86-64 /tmp/dump.bin
------
1872413:M 18 Oct 2023 06:23:07.766 # dump of function (hexdump of 449 bytes):
4881ec180100004189f841ba0800000031ff64488b042528000000488984240801000031c04989e1488db4248000000048b8ffffff7ffeffffff48898424800000004c89cab80e00000048c7842488000000ffffffff48c7842490000000ffffffff48c7842498000000ffffffff48c78424a0000000ffffffff48c78424a8000000ffffffff48c78424b0000000ffffffff48c78424b8000000ffffffff48c78424c0000000ffffffff48c78424c8000000ffffffff48c78424d0000000ffffffff48c78424d8000000ffffffff48c78424e0000000ffffffff48c78424e8000000ffffffff48c78424f0000000ffffffff48c78424f8000000ffffffff0f05b92700000089c80f054889c7b8ba0000000f0589c64489c2b8ea0000000f05483d00f0ffff77394189c041ba0800000031d24c89cebf02000000b80e0000000f05488b84240801000064482b04252800000075204489c04881c418010000c390488b1569511900f7d841b8ffffffff648902ebb6e8f7270d000f1f800000000085ff780cf7dfe9550300000f1f440000488b053951190064c70016000000b8ffffffffc3662e0f1f84000000000066909066662e0f1f8400000000000f1f400048
Function at 0x7f2b4b811510 is __stack_chk_fail

=== REDIS BUG REPORT END. Make sure to include from START to END. ===

       Please report the crash by opening an issue on github:

           http://github.com/redis/redis/issues

  If a Redis module was involved, please open in the module's repo instead.

  Suspect RAM error? Use redis-server --test-memory to verify it.

  Some other issues could be detected by redis-server --check-system

----- Stderr -----
redis-server: /Fuzzing/redisraft-fuzzing/deps/raft/src/raft_server.c:1702: raft_get_first_entry_idx: Assertion `0 < raft_get_current_idx(me)' failed.




logs for node: 1

----- Stdout -----
1873036:C 18 Oct 2023 06:23:07.945 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
1873036:C 18 Oct 2023 06:23:07.945 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1873036:C 18 Oct 2023 06:23:07.945 * Redis version=255.255.255, bits=64, commit=e04ebdb8, modified=1, pid=1873036, just started
1873036:C 18 Oct 2023 06:23:07.945 * Configuration loaded
1873036:M 18 Oct 2023 06:23:07.946 * monotonic clock: POSIX clock_gettime
1873036:M 18 Oct 2023 06:23:07.946 * Running mode=standalone, port=5061.
1873036:M 18 Oct 2023 06:23:07.950 * <raft> RedisRaft version 255.255.255 [c259f3d2]
1873036:M 18 Oct 2023 06:23:07.955 * <raft> Initializing test network
1873036:M 18 Oct 2023 06:23:07.955 * <raft> Created test network client
1873036:M 18 Oct 2023 06:23:07.957 * <raft> Raft module loaded, state is 'loading'
1873036:M 18 Oct 2023 06:23:07.957 * Module 'raft' loaded from /Fuzzing/redisraft-fuzzing/redisraft.so
1873036:M 18 Oct 2023 06:23:07.958 * Server initialized
1873036:M 18 Oct 2023 06:23:07.958 . The AOF directory appendonlydir doesn't exist
1873036:M 18 Oct 2023 06:23:07.958 * Ready to accept connections tcp
1873036:M 18 Oct 2023 06:23:07.958 . 0 clients connected (0 replicas), 1746720 bytes in use

----- Stderr -----




logs for node: 2

----- Stdout -----
1870041:C 18 Oct 2023 06:23:04.663 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
1870041:C 18 Oct 2023 06:23:04.663 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1870041:C 18 Oct 2023 06:23:04.663 * Redis version=255.255.255, bits=64, commit=e04ebdb8, modified=1, pid=1870041, just started
1870041:C 18 Oct 2023 06:23:04.663 * Configuration loaded
1870041:M 18 Oct 2023 06:23:04.663 * monotonic clock: POSIX clock_gettime
1870041:M 18 Oct 2023 06:23:04.664 * Running mode=standalone, port=5062.
1870041:M 18 Oct 2023 06:23:04.669 * <raft> RedisRaft version 255.255.255 [c259f3d2]
1870041:M 18 Oct 2023 06:23:04.674 * <raft> Initializing test network
1870041:M 18 Oct 2023 06:23:04.675 * <raft> Created test network client
1870041:M 18 Oct 2023 06:23:04.676 * <raft> Raft module loaded, state is 'uninitialized'
1870041:M 18 Oct 2023 06:23:04.676 * Module 'raft' loaded from /Fuzzing/redisraft-fuzzing/redisraft.so
1870041:M 18 Oct 2023 06:23:04.676 * Server initialized
1870041:M 18 Oct 2023 06:23:04.677 . The AOF directory appendonlydir doesn't exist
1870041:M 18 Oct 2023 06:23:04.678 * Ready to accept connections tcp
1870041:M 18 Oct 2023 06:23:04.678 . 0 clients connected (0 replicas), 1726192 bytes in use
1870041:M 18 Oct 2023 06:23:04.689 - Accepted 127.0.0.1:44060
1870041:M 18 Oct 2023 06:23:04.689 # Missing implement of connection type tls
1870041:M 18 Oct 2023 06:23:04.776 - <raft> join cluster, connecting to localhost:5061
1870041:M 18 Oct 2023 06:23:04.778 * <raft> Joined Raft cluster, node id: 2, dbid: eabb81d268c2c42c1a747ecf88625ca7
1870041:M 18 Oct 2023 06:23:04.784 * <raft> Cluster Membership: term:0 index:0 nodes: id=2,voting=0,active=1,addr=-
1870041:M 18 Oct 2023 06:23:04.785 * <raft> Test network starting message polling thread
1870041:M 18 Oct 2023 06:23:04.848 - Accepted 127.0.0.1:44066
1870041:M 18 Oct 2023 06:23:05.189 * <raft> Cluster Membership: term:1 index:2 nodes: id=2,voting=0,active=1,addr=- id=1,voting=1,active=1,addr=localhost:5061
1870041:M 18 Oct 2023 06:23:05.190 * <raft> Cluster Membership: term:1 index:4 nodes: id=2,voting=0,active=1,addr=- id=1,voting=1,active=1,addr=localhost:5061 id=3,voting=0,active=1,addr=localhost:5063
1870041:M 18 Oct 2023 06:23:05.580 * <raft> State change: Election starting, node is now a pre-candidate, term 1
1870041:M 18 Oct 2023 06:23:05.580 * <raft> Cluster Membership: term:1 index:5 nodes: id=2,voting=0,active=1,addr=- id=1,voting=1,active=1,addr=localhost:5061 id=3,voting=0,active=1,addr=localhost:5063
1870041:M 18 Oct 2023 06:23:05.611 - Client closed connection id=9 addr=127.0.0.1:44066 laddr=127.0.0.1:5062 fd=10 name= age=1 idle=1 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=22400 events=r cmd=auth user=default redir=-1 resp=2 lib-name= lib-ver=
1870041:M 18 Oct 2023 06:23:05.830 * <raft> State change: Node is now a follower, term 1
1870041:M 18 Oct 2023 06:23:05.830 * <raft> Cluster Membership: term:1 index:5 nodes: id=2,voting=0,active=1,addr=- id=1,voting=1,active=1,addr=localhost:5061 id=3,voting=0,active=1,addr=localhost:5063
1870041:M 18 Oct 2023 06:23:05.887 - Accepted 127.0.0.1:44068
1870041:M 18 Oct 2023 06:23:05.953 - Client closed connection id=10 addr=127.0.0.1:44068 laddr=127.0.0.1:5062 fd=13 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=0 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=auth user=default redir=-1 resp=2 lib-name= lib-ver=
1870041:M 18 Oct 2023 06:23:06.468 - Accepted 127.0.0.1:44072
1870041:M 18 Oct 2023 06:23:06.484 * <raft> State change: Election starting, node is now a pre-candidate, term 1
1870041:M 18 Oct 2023 06:23:06.484 * <raft> Cluster Membership: term:1 index:6 nodes: id=2,voting=1,active=1,addr=- id=1,voting=1,active=1,addr=localhost:5061 id=3,voting=0,active=1,addr=localhost:5063
1870041:M 18 Oct 2023 06:23:06.718 * <raft> State change: Node is now a follower, term 1
1870041:M 18 Oct 2023 06:23:06.718 * <raft> Cluster Membership: term:1 index:6 nodes: id=2,voting=1,active=1,addr=- id=1,voting=1,active=1,addr=localhost:5061 id=3,voting=0,active=1,addr=localhost:5063
1870041:M 18 Oct 2023 06:23:06.873 - Client closed connection id=11 addr=127.0.0.1:44072 laddr=127.0.0.1:5062 fd=10 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=22400 events=r cmd=auth user=default redir=-1 resp=2 lib-name= lib-ver=
1870041:M 18 Oct 2023 06:23:07.089 * <raft> State change: Election starting, node is now a pre-candidate, term 1
1870041:M 18 Oct 2023 06:23:07.089 * <raft> Cluster Membership: term:1 index:7 nodes: id=2,voting=1,active=1,addr=- id=1,voting=1,active=1,addr=localhost:5061 id=3,voting=0,active=1,addr=localhost:5063
1870041:M 18 Oct 2023 06:23:07.491 * <raft> State change: Election starting, node is now a pre-candidate, term 1
1870041:M 18 Oct 2023 06:23:07.491 * <raft> Cluster Membership: term:1 index:7 nodes: id=2,voting=1,active=1,addr=- id=1,voting=1,active=1,addr=localhost:5061 id=3,voting=0,active=1,addr=localhost:5063
1870041:M 18 Oct 2023 06:23:07.847 - Accepted 127.0.0.1:44078
1870041:M 18 Oct 2023 06:23:07.850 - Client closed connection id=12 addr=127.0.0.1:44078 laddr=127.0.0.1:5062 fd=10 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=auth user=default redir=-1 resp=2 lib-name= lib-ver=
1870041:M 18 Oct 2023 06:23:07.893 * <raft> State change: Election starting, node is now a pre-candidate, term 1
1870041:M 18 Oct 2023 06:23:07.893 * <raft> Cluster Membership: term:1 index:7 nodes: id=2,voting=1,active=1,addr=- id=1,voting=1,active=1,addr=localhost:5061 id=3,voting=0,active=1,addr=localhost:5063

----- Stderr -----

We have been able to reproduce the crash on multiple machines.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ds-testing-user