[Bug]: [gnmi] test_gnoi_killprocess.py fails to run on kvm testbed

[yutongzhang-microsoft]

Issue Description

test_gnoi_killprocess.py fails to run on kvm testbed because of below reason catched by log analyzer

https://elastictest.org/scheduler/testplan/676bcc4d98ec838ee83d592b?testcase=gnmi%2Ftest_gnoi_killprocess.py&type=console

Results you see

E               Failed: Processes "['analyze_logs--<MultiAsicSonicHost vlab-01>']" failed with exit code "1"
E               Exception:
E               match: 2
E               expected_match: 0
E               expected_missing_match: 0
E               
E               Match Messages:
E               2024 Dec 25 09:49:18.306689 vlab-01 ERR sonic-db-cli: :- guard: RedisReply catches system_error: command: *9\r\n$4\r\nHSET\r\n$25\r\nDEVICE_METADATA|localhost\r\n$21\r\nchassis_serial_number\r\n$6\r\nFailed\r\n$2\r\nto\r\n$4\r\nread\r\n$6\r\nsystem\r\n$6\r\nEEPROM\r\n$4\r\ninfo\r\n, reason: ERR wrong number of arguments for 'hset' command: Input/output error
E               
E               2024 Dec 25 09:49:18.308198 vlab-01 INFO snmp.sh[69266]: RedisReply catches system_error: command: *9\r\n$4\r\nHSET\r\n$25\r\nDEVICE_METADATA|localhost\r\n$21\r\nchassis_serial_number\r\n$6\r\nFailed\r\n$2\r\nto\r\n$4\r\nread\r\n$6\r\nsystem\r\n$6\r\nEEPROM\r\n$4\r\ninfo\r\n, reason: ERR wrong number of arguments for 'hset' command: Input/output error: Input/output error
E               
E               Traceback:
E               Traceback (most recent call last):
E                 File "/var/src/sonic-mgmt/tests/common/helpers/parallel.py", line 35, in run
E                   Process.run(self)
E                 File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
E                   self._target(*self._args, **self._kwargs)
E                 File "/var/src/sonic-mgmt/tests/common/helpers/parallel.py", line 245, in wrapper
E                   target(*args, **kwargs)
E                 File "/var/src/sonic-mgmt/tests/common/plugins/loganalyzer/__init__.py", line 45, in analyze_logs
E                   dut_analyzer.analyze(markers[node.hostname], fail_test, store_la_logs=store_la_logs)
E                 File "/var/src/sonic-mgmt/tests/common/plugins/loganalyzer/loganalyzer.py", line 409, in analyze
E                   self._verify_log(analyzer_summary)
E                 File "/var/src/sonic-mgmt/tests/common/plugins/loganalyzer/loganalyzer.py", line 140, in _verify_log
E                   raise LogAnalyzerError(result_str)
E               tests.common.plugins.loganalyzer.loganalyzer.LogAnalyzerError: match: 2
E               expected_match: 0
E               expected_missing_match: 0
E               
E               Match Messages:
E               2024 Dec 25 09:49:18.306689 vlab-01 ERR sonic-db-cli: :- guard: RedisReply catches system_error: command: *9\r\n$4\r\nHSET\r\n$25\r\nDEVICE_METADATA|localhost\r\n$21\r\nchassis_serial_number\r\n$6\r\nFailed\r\n$2\r\nto\r\n$4\r\nread\r\n$6\r\nsystem\r\n$6\r\nEEPROM\r\n$4\r\ninfo\r\n, reason: ERR wrong number of arguments for 'hset' command: Input/output error
E               
E               2024 Dec 25 09:49:18.308198 vlab-01 INFO snmp.sh[69266]: RedisReply catches system_error: command: *9\r\n$4\r\nHSET\r\n$25\r\nDEVICE_METADATA|localhost\r\n$21\r\nchassis_serial_number\r\n$6\r\nFailed\r\n$2\r\nto\r\n$4\r\nread\r\n$6\r\nsystem\r\n$6\r\nEEPROM\r\n$4\r\ninfo\r\n, reason: ERR wrong number of arguments for 'hset' command: Input/output error: Input/output error

Results you expected to see

Please confirm if this script can run on kvm or not.

Is it platform specific

generic

Relevant log output

No response

Output of show version

No response

Attach files (if any)

No response

[hdwhdw]

My kvm run fails but with a slightly different message suggesting dbus doesn't support gnmi.

ERROR gnmi/test_gnoi_killprocess.py::test_gnoi_killprocess_then_restart[gnmi-False-Dbus does not support gnmi service management] - Failed: Failed to start gnmi server
ERROR gnmi/test_gnoi_killprocess.py::test_gnoi_killprocess_then_restart[nonexistent-False-Dbus does not support nonexistent service management] - Failed: Failed to start gnmi server
ERROR gnmi/test_gnoi_killprocess.py::test_gnoi_killprocess_then_restart[-False-Dbus stop_service called with no service specified] - Failed: Failed to start gnmi server

Looks like the test isn't supported on kvm. This is a real test gap and I should fix it.

[hdwhdw]

The (set of) tests has multiple problems:

• Before it tries to kill a process, it will assert its existence in https://github.com/sonic-net/sonic-mgmt/blob/master/tests/gnmi/test_gnoi_killprocess.py#L32, which in some image lacking say telemetry container, can throw exceptions.
• killing some process such as snmp, pmon and rsyslog can cause errors like

2025 Jan  2 12:48:10.699687 vlab-01 ERR swss#orchagent: :- isAutoNegEnabled: Failed to get port AutoNeg status for port pid:1000000000021

Not sure the cause but we can disable these tests for now.

[hdwhdw]

https://github.com/sonic-net/sonic-swss/blob/eae729e22864ba31a412efcaa575fc0d61a2b25c/orchagent/portsorch.cpp#L3186

[hdwhdw]

Across multiple runs, killing and restarting pmon and snmp consistently show the same error, rsyslogd is flaky (with the same error).

2025 Jan  6 15:22:37.489359 vlab-01 ERR swss#orchagent: :- isAutoNegEnabled: Failed to get port AutoNeg status for port pid:1000000000018