remove comand sets from device2yaml.rb and document ATOMS

- Preparing merge into master (some polishing is still needed)

CHANGELOG.md

@@ -13,6 +13,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 ### Changed
 - sonicos: accept policy message. Fixes #3339 (@Steve-M-C, @robertcheramy)
 - input/ssh: change input.debug to dump all characters and include sent commands. (@robertcheramy)
+- model unit tests: the tests are automated and simpler to use (@ytti, @robertcheramy)
+- device2yaml.rb: moved to extras, commands can be specified from the command line or from a file (no cmdsets provided anymore) (@robertcheramy)
 
 ### Fixed
 - tplink: send 'enable' before the enable password. Fixes #3271 (@robertcheramy)

docs/Creating-Models.md

@@ -146,7 +146,7 @@ developments could break it, and facilitates debugging issues without having
 access to a physical network device for the model.
 
 In order to simulate the device in the unit test, you need a
-[YAML simulation file](/examples/device-simulation/), have a look at the
+[YAML simulation file](/docs/DeviceSimulation.md), have a look at the
 link for an explanation on how to create one.
 
 Creating the unit test itself is explained in

docs/DeviceSimulation.md

@@ -36,19 +36,28 @@ editors automatically remove trailing spaces, so we code them with \x20.
 Although a YAML file could be written by hand, this is quite a tedious task to
 catch all the extra codes and code them into YAML. This can be
 automated with the ruby script
-[device2yaml.rb](/extras/device2yaml.rb).
+[extras/device2yaml.rb](/extras/device2yaml.rb).
 
 `device2yaml.rb` needs ruby and the gem
 [net-ssh](https://rubygems.org/gems/net-ssh/) to run. On debian, you can install
 them with `sudo apt install ruby-net-ssh`
 
-Run `device2yaml.rb`, the online help tells you the options.
+Run `extras/device2yaml.rb`, the online help tells you the options.
 ```
 oxidized$ extra/device2yaml.rb
 Missing a host to connect to...
 
-Usage: device2yaml.rb [user@]host [options]
-    -c, --cmdset set                 Mandatory: specify the commands to be run
+Usages:
+- device2yaml.rb [user@]host -i file [options]
+- device2yaml.rb [user@]host -c "command1
+  command2
+  command3" [options]
+
+-i and -c are mutualy exclusive, one must be specified
+
+[options]:
+    -c, --commands "command list"    specify the commands to be run
+    -i, --input file                 Specify an input file for commands to be run
     -o, --output file                Specify an output YAML-file
     -t, --timeout value              Specify the idle timeout beween commands (default: 5 seconds)
     -e, --exec-mode                  Run ssh in exec mode (without tty)
@@ -59,16 +68,15 @@ Usage: device2yaml.rb [user@]host [options]
 password will be prompted interactively by the script. If you do not specify a
 user, it will use the user executing the script.
 - The commands that will be run on the device must be defined
-in `deviceyaml.rb`. If you specify an unknown commandset, you will be provided
-with a list of implemented ones. If you implement a new command set, it is
-important that you enter exactly the commands used by the oxidized model,
-and no abbreviation like `sh run`. Do not forget to insert the `post_login`
-commands at the beginning if the model has some and also the `pre_logout`
-commands at the end.
+in `deviceyaml.rb`. You can give the commands online with `-c` or read them
+from a file (one line per command) with `-i`. The commands should match exactly
+the one of the model (no abbreviations) and include the command of the
+`post_login` and `pre_logout` sections. When using -c and editing the command
+line, `CTRL-V CTRL-J` is very useful to add a new line.
 - `device2yaml.rb` waits an idle timeout after the last received data before
 sending the next command. The default is 5 seconds. If your device makes a
 longer pause than 5 seconds before or within a command, you will see that the
-output of the command is shortened or slips into the next command in the yaml
+output of the command is shortened or slips into the next command in the YAML
 file. You will have to change the idle timeout to a greater value to address
 this.
 - When run without the output argument, `device2yaml.rb` will only print the ssh
@@ -77,16 +85,21 @@ to store the collected data in a YAML file.
 - If your oxidized model uses ssh exec mode (look for `exec true` in the model),
 you will have to use the option `-e` to run device2yaml in ssh exec mode.
 
-Note that `extra/device2yaml.rb` takes some time to run because of the idle
+Note that `device2yaml.rb` takes some time to run because of the idle
 timeout of (default) 5 seconds between each command. You can press the "Escape"
 key if you know there is no more data to come for the current command (when you
 see the prompt for the next command), and the script will stop waiting and
 directly process the next command.
 
-Here are two examples of how to run the script:
+Running the script against an ios device would look like:
 ```shell
-extra/device2yaml.rb OX-SW123.sample.domain -c aoscx -o spec/model/data/aoscx:R8N85A-C6000-48G-CL4_PL.10.08.1010:simumation.yaml
-extra/device2yaml.rb admin@r7 -c routeros -e -o spec/model/data/routeros:CHR_7.10.1:simulation.yaml
+extra/device2yaml.rb oxidized@r61 -c "terminal length 0
+terminal width 0
+show version
+show vtp status
+show inventory
+show running-config
+exit" -o spec/model/data/ios:C8200L_16.12.1:simulation.yaml
 ```
 
 ### Publishing the YAML simulation file to oxidized
@@ -98,20 +111,16 @@ You should pay attention to removing or replacing anything you don't want to
 share with the rest of the world, for example:
 
 - Passwords
-- IP Adresses
+- IP Addresses
 - Serial numbers
 
 You can also shorten the configuration if you want - we don't need 48 times the
-same config for each interface, but it doesn't hurt either.
+same configuration for each interface, but it doesn't hurt either.
 
 Take your time, this is an important task: after you have
 uploaded your file on github, it may be impossible to remove it. You can use
 search/replace to make consistent and faster changes (change the hostname).
 
-You can leave the section `oxidized_output` unchanged, it is only used for
-[model unit tests](/spec/model). You will find an explanation of how to produce
-the `oxidized_output`-section in the README.md there.
-
 The YAML simulation files are stored under
 [/spec/model/data/](/spec/model/data/), with the naming convention
 `model:description:simulation.yaml`, in which `model` is the lowercase name

docs/Issues.md

@@ -49,7 +49,7 @@ contributing code via a pull request (PR) or hiring a developer.
 
 ## Sumbit a YAML Simulation File
 To help developers troubleshoot device-specific issues, you may be asked to submit a
-[YAML simulation file](https://github.com/ytti/oxidized/blob/master/examples/device-simulation/README.md#creating-a-yaml-file-with-device2yamlrb) for your device.
+[YAML simulation file](/docs/DeviceSimulation.md#creating-a-yaml-file-with-device2yamlrb) for your device.
 
 Here's a brief overview how to do it, you can find more details in the link
 above.
@@ -63,22 +63,28 @@ sudo apt install git ruby-net-ssh
 ```
 git clone [email protected]:<your github user>/oxidized.git
 ```
-- run the device2yaml.rb script (you’ll be provided with the command set and
-  output filename to use)
+- run the extras/device2yaml.rb script (you’ll be provided with the command to
+run) from the repository root:
+
 ```
-cd oxidized/examples/device-simulation
-# Replace user and devicename to appropriate values
-./device2yaml.rb user@devicename -c cmdsets/ios -o yaml/asr900_26.8.1b.yaml
+extra/device2yaml.rb oxidized@r61 -c "terminal length 0
+terminal width 0
+show version
+show vtp status
+show inventory
+show running-config
+exit" -o spec/model/data/ios:C8200L_16.12.1:simulation.yaml
 ```
+
 - The script waits 5 seconds between commands, and outputs the response of the
   device. You can press "ESC" if you see the prompt and want to pass to next
   command without waiting for the timeout.
-- The result will be stored in `oxidized/examples/device-simulation/yaml/`.
+- The result will be stored in `spec/model/data/`.
 - Replace any sensitive information with placeholder values in the output file.
 - Commit & push the file to github
 ```
-git add yaml/asr900_26.8.1b.yaml
-git commit -m "Device simulation for ASR900"
+git add spec/model/data/ios:C8200L_16.12.1:simulation.yaml
+git commit -m "Device simulation for C8200L"
 git push
 ```
 - Create a pull request (PR) in GitHub, referencing the issue number (e.g.,

docs/ModelUnitTests.md

@@ -0,0 +1,172 @@
+# Model unit tests
+Oxidized includes automated unit tests for its model, wich require very few
+efforts to use. There are three different unit tests for models:
+- [device simulation](ModelUnitTests.md#device-simulation)
+- [device prompt](ModelUnitTests.md#device-prompt)
+- [secrets](ModelUnitTests.md#secrets)
+
+You only need to provide test files under [/spec/model/data](/spec/model/data)
+and the tests will be run automatically with `rake test`.
+See [CONTRIBUTING.md](/CONTRIBUTING.md) for explanations on how to run
+`rake test` in a development environment. In the following examples,
+we use [Bundler](https://bundler.io/) and prefix all commands with
+`bundle exec `.
+
+## Device Simulation
+### YAML simulation file
+You need a [YAML simulation file](/docs/DeviceSimulation.md) for your
+device, see the link on how to produce it.
+
+The YAML simulation files are stored under
+[/spec/model/data/](/spec/model/data/), with the naming convention
+`model:description:simulation.yaml`, in which `model` is the lowercase name
+of the oxidized model and `description` is the name of the test case and is
+generally formatted as #hardware_#software or
+#model_#hardware_#software_#description.
+
+Using a correct name for the file is important in order to be included in
+automatic model unit tests.
+
+### Expected output
+You need a second file wich contains the expected output, wich has the same
+name as the YAML simulation file but ends with `:output.txt`
+instead of `:simulation.yaml`
+
+You can automatically produce an output file based on the current model for
+all YAML simulation files missing an `:output.txt`:
+```shell
+bundle exec ruby spec/model/atoms_generate.rb
+```
+
+In the following example,
+`spec/model/data/aoscx:R8N85A-C6000-48G-CL4_PL.10.08.1010:output.txt` (the
+second file in the list) was missing:
+
+```shell
+oxidized$ bundle exec ruby spec/model/atoms_generate.rb
+Run options: --seed 57811
+
+# Running:
+
+Generating output file for aoscx:R0X25A-6410_FL.10.10.1100:simulation... SKIP, output already exists
+Generating output file for aoscx:R8N85A-C6000-48G-CL4_PL.10.08.1010:simulation... OK
+Generating output file for arubainstant:IAP515_8.10.0.6_VWLC:simulation... SKIP, output already exists
+Generating output file for asa:5512_9.12-4-67_single-context:simulation... SKIP, output already exists
+Generating output file for garderos:R7709_003_006_068:simulation... SKIP, output already exists
+Generating output file for ios:C8200L_16.12.1:simulation... FAIL, no simulation file
+Generating output file for ios:C9200L-24P-4G_17.09.04a:simulation... SKIP, output already exists
+Generating output file for ios:C9800-L-F-K9_17.06.05:simulation... SKIP, output already exists
+Generating output file for ios:asr920_16.8.1b:simulation... SKIP, output already exists
+Generating output file for junos:srx300_22.4:simulation... SKIP, output already exists
+Generating output file for opnsense:nano_23.7:simulation... SKIP, output already exists
+Generating output file for pfsense:CE_2.7.2:simulation... SKIP, output already exists
+Generating output file for routeros:CHR_7.10.1:simulation... SKIP, output already exists
+Generating output file for routeros:CHR_7.16:simulation... SKIP, output already exists
+Generating output file for routeros:L009UiGS_7.15.2:simulation... SKIP, output already exists
+.
+
+Finished in 0.904792s, 1.1052 runs/s, 0.0000 assertions/s.
+
+1 runs, 0 assertions, 0 failures, 0 errors, 0 skips
+Coverage report generated for RSpec to /home/xxx/oxidized/coverage/coverage.xml. 651 / 1122 LOC (58.02%) covered
+Coverage report generated for RSpec to /home/xxx/oxidized/coverage.
+Line Coverage: 58.02% (651 / 1122)
+```
+
+### Running the tests
+You can modify the `:output.txt` file to match your expectations and modify the
+model accordingly. run `bundle exec rake` to run the tests.
+
+Here is an example when the output of the VTP command is missing in the
+expected output:
+
+```
+oxidized$ bundle exec rake test
+/usr/bin/ruby3.1 -I"lib:spec" /home/xxx/oxidized/vendor/bundle/ruby/3.1.0/gems/rake-13.2.1/lib/rake/rake_test_loader.rb "spec/cli_spec.rb" "spec/config_spec.rb" "spec/hook/githubrepo_spec.rb" "spec/input/ssh_spec.rb" "spec/manager_spec.rb" "spec/model/apc_aos_spec.rb" "spec/model/model_atoms_spec.rb" "spec/model/model_helper_spec.rb" "spec/node_spec.rb" "spec/nodes_spec.rb" "spec/output/file_spec.rb" "spec/output/git_spec.rb" "spec/refinements_spec.rb" "spec/source/csv_spec.rb" "spec/source/http_spec.rb" "spec/source/jsonfile_spec.rb" "spec/source/sql_spec.rb"
+Run options: --seed 31447
+
+# Running:
+
+...............................................................SS..................F.............................SS..
+
+Finished in 7.963602s, 14.6918 runs/s, 48.7217 assertions/s.
+
+  1) Failure:
+ATOMS tests#test_0006_ios:C9200L-24P-4G_17.09.04a:output has expected output [spec/model/model_atoms_spec.rb:12]:
+--- expected
++++ actual
+@@ -9,6 +9,21 @@
+ ! CPU: ARM64
+ ! Memory: nvram 2048K
+ !
++! VTP: VTP Version capable             : 1 to 3
++! VTP: VTP version running             : 1
++! VTP: VTP Domain Name                 : Oxidized
++! VTP: VTP Pruning Mode                : Disabled (Operationally Disabled)
++! VTP: VTP Traps Generation            : Disabled
++! VTP: Device ID                       : 40f0.7800.0000
++! VTP: Feature VLAN:
++! VTP: --------------
++! VTP: VTP Operating Mode                : Transparent
++! VTP: Maximum VLANs supported locally   : 1005
++! VTP: Number of existing VLANs          : 10
++! VTP: Configuration Revision            : 0
++! VTP: MD5 digest                        : 0x35 0x00 0x00 0x00 0x00 0x00 0x7F 0xB4
++! VTP:                                     0x07 0x00 0x00 0x00 0x00 0x00 0x09 0x6D
++!
+ ! NAME: \"c92xxL Stack\", DESCR: \"c92xxL Stack\"
+ ! PID: C9200L-24P-4G     , VID: V01  , SN: JAE24FFFFFF
+ !
+
+
+117 runs, 388 assertions, 1 failures, 0 errors, 4 skips
+
+You have skipped tests. Run with --verbose for details.
+Coverage report generated for RSpec to /home/xxx/oxidized/coverage/coverage.xml. 2167 / 3131 LOC (69.21%) covered
+Coverage report generated for RSpec to /home/xxx/oxidized/coverage.
+Line Coverage: 69.21% (2167 / 3131)
+rake aborted!
+Command failed with status (1): [ruby -I"lib:spec" /home/xxx/oxidized/vendor/bundle/ruby/3.1.0/gems/rake-13.2.1/lib/rake/rake_test_loader.rb "spec/cli_spec.rb" "spec/config_spec.rb" "spec/hook/githubrepo_spec.rb" "spec/input/ssh_spec.rb" "spec/manager_spec.rb" "spec/model/apc_aos_spec.rb" "spec/model/model_atoms_spec.rb" "spec/model/model_helper_spec.rb" "spec/node_spec.rb" "spec/nodes_spec.rb" "spec/output/file_spec.rb" "spec/output/git_spec.rb" "spec/refinements_spec.rb" "spec/source/csv_spec.rb" "spec/source/http_spec.rb" "spec/source/jsonfile_spec.rb" "spec/source/sql_spec.rb" ]
+/home/xxx/oxidized/vendor/bundle/ruby/3.1.0/gems/rake-13.2.1/exe/rake:27:in `<top (required)>'
+Tasks: TOP => test
+(See full trace by running task with --trace)
+```
+
+
+## Device prompt
+You can specify device prompts to test in a YAML file named
+`spec/model/data/<model>:generic:prompt.yaml`.
+
+The YAML file has three sections containing a list of prompts to test:
+- pass: these prompts will pass the prompt regexp
+- pass_with_expect: these prompts will pass the prompt regexp after having been cleaned by the expect commands.
+- fail: these prompts wil fail the prompt regexp
+
+Here is an example:
+```yaml
+pass:
+  - "LAB-R1234_Garderos# "
+pass_with_expect:
+  - "\e[4m\rLAB-R1234_Garderos#\e[m "
+fail:
+  - "\e[4m\rLAB-R1234_Garderos#\e[m "
+  - "#LAB#"
+```
+
+## Secrets
+You can test if the model effectively remove secrets from your YAML simulation
+file with a YAML file named like the YAML simulation, but with the suffix
+`:secret.yaml`.
+
+The YAML file has three sections containing a list of strings to test:
+- fail: the test will fail if the output contains this strings
+- pass: the test will pass only if the output contain these strings
+
+```yaml
+fail:
+  - 'AAAAAAAAAABBBBBBBBBBCCCCCCCCCC'
+pass:
+  - 'snmp-server host 10.10.42.12 version 2c <secret removed> inform'
+  - 'hash-mgmt-user oxidized password hash <secret removed>'
+  - 'hash-mgmt-user rocks password hash <secret removed> usertype read-only'
+```