Merge pull request #28 from Nancy-Chauhan/add-yosys-scripts

Add Yosys synthesys metrics parser to the LibreCores CI Docker image

librecores-ci-openrisc/Dockerfile

@@ -19,7 +19,7 @@ RUN tar xC /tmp/tools -f  or1k-elf-9.0.0-20181112.tar.xz
 ENV PATH="/tmp/tools/or1k-elf/bin:${PATH}"
 
 # Download and compile or1k-tests
-RUN git clone https://github.com/openrisc/or1k-tests.git 
+RUN git clone https://github.com/openrisc/or1k-tests.git
 WORKDIR /tmp/src/tools/or1k-tests/native
 RUN git checkout ${OR1K_TESTS_VERSION}
-RUN make -j8                                          
+RUN make -j8

librecores-ci-openrisc/README.md

@@ -3,9 +3,9 @@ LibreCores CI image for OpenRISC development
 
 [![Docker Pulls](https://img.shields.io/docker/pulls/librecores/librecores-ci-openrisc.svg)](https://hub.docker.com/r/librecores/librecores-ci-openrisc/)
 
-Librecores CI image for OpenRISC development is based on the standard [Librecores-CI](https://github.com/librecores/docker-images/tree/master/librecores-ci) docker image and it largely targets the [FuseSoC](https://github.com/olofk/fusesoc) use cases. This image mainly focuses on [OpenRISC](https://github.com/openrisc) project that creates a free and open processor for embedded systems. 
- 
- ### Quick Start 
+Librecores CI image for OpenRISC development is based on the standard [Librecores-CI](https://github.com/librecores/docker-images/tree/master/librecores-ci) docker image and it largely targets the [FuseSoC](https://github.com/olofk/fusesoc) use cases. This image mainly focuses on [OpenRISC](https://github.com/openrisc) project that creates a free and open processor for embedded systems.
+
+ ### Quick Start
 
 Currently librecores-ci-openrisc is successfully implemented in [mor1kx](https://github.com/openrisc/mor1kx) and [or1k-marocchino](https://github.com/Nancy-Chauhan/or1k_marocchino) projects.
 

librecores-ci/Dockerfile

@@ -85,6 +85,9 @@ RUN make config-gcc
 RUN make
 RUN make install
 
+# test-scipts:extract-yosys-stats.py outputs 'Printing Statistics' for monitoring resource usages by running yosys synthesis.
+COPY test-scripts /test-scripts 
+
 # pytest, nose and tappy
 RUN pip install pytest==${PYTEST_VERSION}
 RUN pip install tap.py==${TAPPY_VERSION}

librecores-ci/README.md

@@ -14,6 +14,7 @@ This is a base image which includes common EDA tools.
 * [cocotb](https://github.com/potentialventures/cocotb)
 * [pytest](https://docs.pytest.org/en/latest/)
 * [tap.py](https://pypi.org/project/tap.py/)
+* Yosys Parser 
 
 ### Quick start 
 
@@ -25,3 +26,49 @@ The [or1k Continuous Integration (CI) suite](https://github.com/openrisc/or1k-te
 Librecores-CI docker container in [Travis-CI](https://github.com/openrisc/mor1kx/blob/master/.travis.yml). 
 Parallel execution of tests runs in Librecores-CI docker environment. As a reference one can follow up the [blog](http://nancychauhan.in/stories/2019/06/08/gsoc-week1_2/) to get an idea of how librecores docker images can be 
 integrated to existing continuous integration suite. 
+
+#### Yosys Parser
+Yosys is a framework for Verilog RTL synthesis. For monitoring resource usages of any hardware project, this parser script (extract-yosys-stats.py) can help to visualise results better. This take input from huge yosys.log file and outputs 'Printing Statistics' like :
+```
+wire bits,16199
+public wires,1058
+public wire bits,11151
+memories,0
+memory bits,0
+processes,0
+cells,7310
+SB_CARRY,426
+SB_DFF,166
+SB_DFFE,885
+SB_DFFESR,569
+SB_DFFESS,13
+SB_DFFSR,56
+SB_DFFSS,2
+SB_LUT4,5182
+SB_RAM40_4K,11
+```
+
+#### Quick Start 
+
+To use Yosys Parser, you can see its implementation in <a href="https://github.com/openrisc/mor1kx">openrisc project<a>.
+
+One can quickly get started with Fusesoc, a package manager and a set of build tools for HDL (Hardware Description Language) code.
+Fusesoc provides the <a href = "https://github.com/olofk/edalize">icestorm backend</a> ( edalize ) which uses yosys to handle synthesis, arachne-pnr for place & route and icepack for creating the bitstream.
+1) Do changes in core description file to support icestorm for your project. One can follow <a href="https://fusesoc.readthedocs.io/en/rtd/fusesoc.html"> quick tutorial <a>for writing a core description file. Example:
+```
+ synth:
+ default_tool : icestorm
+ filesets : [X, Y]
+ tools:
+ icestorm:
+ pnr: none
+ toplevel : {XYZ}
+ ```
+
+2) Once modification in core file is done, you are set to run yosys synthesis in librecores/librecores-ci docker environment
+( make sure to run the command in librecores/librecores-ci docker environment ), For example, for mor1kx following commands are run to obtain printing statistics while running yosys synthesis :
+```
+fusesoc library add mor1kx /src
+fusesoc run --target=synth mor1kx
+test-scripts/extract-yosys-stats.py < build/mor1kx_*/synth-icestorm/yosys.log
+```

librecores-ci/test-scripts/extract-yosys-stats.py

@@ -0,0 +1,76 @@
+#!/usr/bin/env python3
+import sys
+import re
+import csv
+
+parser_state = 0
+matches = []
+cells = []
+
+# Printing statistics are extracted from yosys.log
+for line in sys.stdin:
+    if parser_state == 0:
+        if re.match(r"(?:\d+\.)* Printing statistics\.", line):
+            parser_state = 1
+    elif parser_state == 1:
+        match_result = re.match(r"\s{3}Number of ([\w\s]+):\s+(\d+)", line)
+        if match_result:
+            metric = match_result.group(1)
+            value = match_result.group(2)
+            matches.append((metric, value))
+
+            if metric == "cells":
+                parser_state = 2
+    elif parser_state == 2:
+        match_result = re.match(r"\s{5}(\w+)\s+(\d+)", line)
+        if match_result:
+            cell_type = match_result.group(1)
+            count = match_result.group(2)
+
+            cells.append((cell_type, count))
+
+print(matches)
+print(cells)
+
+# Outputs the stats in CSV format
+with open('result.csv', 'w') as csvFile:
+    writer = csv.writer(csvFile)
+    writer.writerows(matches)
+    writer.writerows(cells)
+
+with open('result.csv', newline='') as f:
+    r = csv.reader(f)
+    data = [line for line in r]
+with open('result.csv', 'w', newline='')as f:
+    w = csv.writer(f)
+    w.writerow(['label', 'elapsed'])
+    w.writerows(data)
+
+# Code for getting JMeter compatible CSV file for Performance Plugin
+# TODO: result.csv with Key Value has to be made compatible with Jenkins Perforamce plugin
+in_file = 'result.csv'
+out_file = 'report.csv'
+with open(in_file, 'r') as in_f, open(out_file, 'w', newline='') as out_f:
+    rdr = csv.DictReader(in_f)
+    fieldnames = ['timeStamp', 'responseCode', 'responseMessage',
+                  'threadName', 'dataType', 'success', 'bytes']
+    fieldnames.extend(rdr.fieldnames)
+    wrtr = csv.DictWriter(out_f, fieldnames=fieldnames)
+    wrtr.writeheader()
+    for row_id, row in enumerate(rdr, start=1):
+        row['timeStamp'] = '2019-08-14 14:15:25.321'.format(row_id)
+        row['responseCode'] = '200'.format(row_id)
+        row['responseMessage'] = 'OK'.format(row_id)
+        row['threadName'] = 'Thread Group 1-1'.format(row_id)
+        row['dataType'] = 'text'.format(row_id)
+        row['success'] = 'true'.format(row_id)
+        row['bytes'] = '3478'.format(row_id)
+        wrtr.writerow(row)
+
+with open('report.csv', 'r') as infile, open('output.csv', 'a') as outfile:
+    fieldnames = ['timeStamp', 'elapsed', 'label', 'responseCode',
+                  'responseMessage', 'threadName', 'dataType', 'success', 'bytes']
+    writer = csv.DictWriter(outfile, fieldnames=fieldnames)
+    writer.writeheader()
+    for row in csv.DictReader(infile):
+        writer.writerow(row)