https://open-source-silicon.dev logo
Join Slack
Powered by
Hello all, I am running the MPW precheck for my de...
# shuttle-precheck
p
Hello all, I am running the MPW precheck for my design and it seems to hang in a weird state when doing the magic DRC. The process consumes as much as 180GB of RAM and runs for an extremely long time. There is no output into any of the logs, and it will run until the process times out (over 8 hours the longest I let it run). Has anyone run into this and resolved it? My design uses SRAM macros from OpenRAM, which I saw mentioned in this case but without a clear resolution. My repo is here, if it helps.
Copy code
023-10-28 01:06:02 - [INFO] - {{STEP UPDATE}} Executing Check 2 of 12: Makefile
2023-10-28 01:06:02 - [INFO] - {{MAKEFILE CHECK PASSED}} Makefile valid.
2023-10-28 01:06:02 - [INFO] - {{STEP UPDATE}} Executing Check 3 of 12: Consistency
2023-10-28 01:19:00 - [INFO] - PORTS CHECK PASSED: Netlist user_project_wrapper ports match the golden wrapper ports
2023-10-28 01:19:00 - [INFO] - COMPLEXITY CHECK PASSED: Netlist user_project_wrapper contains at least 1 instances (309658 instances). 
2023-10-28 01:19:00 - [INFO] - MODELING CHECK PASSED: Netlist user_project_wrapper is structural.
2023-10-28 01:19:00 - [INFO] - LAYOUT CHECK PASSED: The GDS layout for user_project_wrapper matches the provided structural netlist.
2023-10-28 01:19:01 - [INFO] - POWER CONNECTIONS CHECK PASSED: All instances in user_project_wrapper are connected to power
2023-10-28 01:19:01 - [INFO] - PORT TYPES CHECK PASSED: Netlist user_project_wrapper port types match the golden wrapper port types.
2023-10-28 01:19:01 - [INFO] - {{NETLIST CONSISTENCY CHECK PASSED}} user_project_wrapper netlist passed all consistency checks.
2023-10-28 01:19:01 - [INFO] - {{CONSISTENCY CHECK PASSED}} The user netlist and the top netlist are valid.
2023-10-28 01:19:01 - [INFO] - {{STEP UPDATE}} Executing Check 4 of 12: XOR
2023-10-28 01:19:52 - [INFO] - {{XOR CHECK UPDATE}} Total XOR differences: 0, for more details view /home/prherrma/runners/r0/_work/tapeout-ci-2311/tapeout-ci-2311/mpw_precheck_result/outputs/user_project_wrapper.xor.gds
2023-10-28 01:19:52 - [INFO] - {{XOR CHECK PASSED}} The GDS file has no XOR violations.
2023-10-28 01:19:52 - [INFO] - {{STEP UPDATE}} Executing Check 5 of 12: Magic DRC
m
@Peter Herrmann There should be something that abstracts the sram for DRC, I think.
Looks like it might only work for the srams in 
$PDK_ROOT/$PDK/libs.ref/sky130_sram_macros
. Are you using custom sram configurations?
✅ 1
mpw_precheck/checks/drc_checks/magic/magic_gds_drc_check.py
p
Im using the sky130_sram_2kbyte_1rw1r_32x512_8 from here, I didn't realize they were packaged with the pdk as well.
To use the ones with the pdk, do I just need to adjust the following values to point to the pdk?
Copy code
"VERILOG_INCLUDE_DIRS": ["dir::../../openlane/macros"],
	"VERILOG_FILES_BLACKBOX": ["dir::../../openlane/macros/sky130_sram_2kbyte_1rw1r_32x512_8.v"],
    "EXTRA_LEFS": ["dir::../../openlane/macros/sky130_sram_2kbyte_1rw1r_32x512_8.lef"],
	"EXTRA_LIBS": ["dir::../../openlane/macros/sky130_sram_2kbyte_1rw1r_32x512_8_TT_1p8V_25C.lib"],
	"EXTRA_GDS_FILES": ["dir::../../openlane/macros/sky130_sram_2kbyte_1rw1r_32x512_8.gds"],
m
If you’re running openlane in docker, the pdk is mounted at /openlane/pdks so you can probably use
Copy code
/openlane/pdks/sky130A/libs.ref/sky130_sram_macros
to access the data from the pdk directly without having to copy everything to your directory. (or sky130B if that’s what you’re using.) Looks like you might also be able to use this whether or not you’re using docker.
Copy code
pdk_dir::libs.ref/sky130_sram_macros
p
The problem I ran into with this macro originally was the use of timing constructs in the verilog blackbox. The linter throws an error (which can be suppressed in the flow):
Copy code
Warning: : Timing constructs found in the RTL. Please remove them or wrap them around an ifdef. It heavily unrecommended to rely on timing constructs for synthesis.
Warning: : 5 errors found by linter
Even if I were to suppress linter errors, STA fails with the following:
Copy code
Error while reading /home/prherrma/runners/r0/_work/pdk/sky130A/libs.ref/sky130_sram_macros/verilog/sky130_sram_2kbyte_1rw1r_32x512_8.v:
Make sure that this a gate-level netlist not an RTL file
You can add the following comment '/// sta-blackbox' in the file to skip it and blackbox the modules inside if needed.
I could easily get around this by modifying the blackbox file (adding 
/// sta-blackbox
), but I don't think I can get away with modifying the PDK version. Is it necessary to use the PDK version of the verilog blackbox to get the SRAM abstraction for Magic DRC or are the .def, .lef, and .gds sufficent? I may also just be looking at this the completely wrong way.
m
I think you want to add this file to 
VERILOG_FILES_BLACKBOX
in your 
config.json
file.
p
"This file" meaning the PDK version of the verilog blackbox? Unfortunately, doing that does not by itself tell the STA tools to blackbox them (explained here), there still needs to be the 
/// sta-blackbox
comment in the file. Kind of a catch-22 between STA failing because this flag isn't in the PDK verilog file, and the DRC hanging because I'm not using the PDK verilog file, and I don't see how to get around it.
Basically, if I use the following in 
configs.json
:
Copy code
"VERILOG_FILES_BLACKBOX": "pdk_dir::libs.ref/sky130_sram_macros/verilog/sky130_sram_2kbyte_1rw1r_32x512_8.v",
    "EXTRA_LEFS": "pdk_dir::libs.ref/sky130_sram_macros/lef/sky130_sram_2kbyte_1rw1r_32x512_8.lef",
	"EXTRA_LIBS": "pdk_dir::libs.ref/sky130_sram_macros/lib/sky130_sram_2kbyte_1rw1r_32x512_8_TT_1p8V_25C.lib",
	"EXTRA_GDS_FILES": "pdk_dir::libs.ref/sky130_sram_macros/gds/sky130_sram_2kbyte_1rw1r_32x512_8.gds",
Then STA fails with:
Copy code
Make sure that this a gate-level netlist not an RTL file
You can add the following comment '/// sta-blackbox' in the file to skip it and blackbox the modules inside if needed.
If I add 
/// sta-blackbox
as the error message suggests, then the flow completes, but it hangs forever during Magic DRC check for the MPW precheck.
m
Can you share your 
precheck/<tag>/logs/magic_drc_check.log
file?
p
Sure. This is the magic_drc_check.log from this commit
magic_drc_check.log
m
From 
mpw_precheck/checks/drc_checks/magic/magic_drc_check.tcl
Copy code
if { $HAS_SRAM } {
    gds noduplicates yes
    puts "Detected an SRAM module"
    foreach x $SRAM_MODULES {
        puts "Pre-loading a maglef of the SRAM block: ${x}"
        load $PDK_PATH/libs.ref/sky130_sram_macros/maglef/${x}.mag
    }
} 
gds read $GDS_UT_PATH
So there’s an 
Detected an SRAM module
message in your log, but no 
Pre-loading a maglef…
messages.
This is a precheck problem. I’ll log an issue. Thanks for reporting!
p
Oh dang, ok!
@Mitch Bailey I just ran my design using the sram-drc-fix version of mpw-precheck, and it seems to be doing the same thing still. This commit uses the fix version (confirmed by reading 
mpw_precheck/checks/drc_checks/magic/magic_drc_check.tcl
in the runner's local code). After 2 hours the Magic DRC check is using 150 GB of RAM, and I don't see "Pre-loading a maglef of the SRAM block:" in the log.
magic_drc_check.log
Here is the version of the script mentioned in the log (
/home/prherrma/runners/r1/_work/tapeout-ci-2311/tapeout-ci-2311/mpw_precheck/checks/drc_checks/magic/magic_drc_check.tcl
)
magic_drc_check.tcl
m
@Peter Herrmann Thanks for reporting! I thought I checked the pr changes against my test setup, but I’ll look again. Could you make this change and let me know the output?
Copy code
puts "Detected an SRAM module $SRAM_MODULES"
👍 1
p
@Mitch Bailey With 
puts "Detected an SRAM module: $SRAM_MODULES"
, I just get the following:
Copy code
Loading "/home/prherrma/runners/r0/_work/tapeout-ci-2311/tapeout-ci-2311/mpw_precheck/checks/drc_checks/magic/magic_drc_check.tcl" from command line.
Detected an SRAM module: 
Warning: Calma reading is not undoable!  I hope that's OK.
So it looks like 
HAS_SRAM
is set but somehow 
SRAM_MODULES
is empty.
m
yep. Can you check your 
PDK_ROOT
, 
PDK
and 
PDKPATH
variables?
Here are some debugging changes I made to 
checks/drc_checks/magic/magic_gds_drc_check.py
Copy code
+++ b/checks/drc_checks/magic/magic_gds_drc_check.py
@@ -60,11 +60,14 @@ def magic_gds_drc_check(gds_ut_path, design_name, pdk_path, output_directory):
     sram_maglef_files_generator = Path(pdk_path / 'libs.ref/sky130_sram_macros/maglef').glob('*.mag')
     for installed_sram in sram_maglef_files_generator:
         installed_sram_modules_names.append(installed_sram.stem)
+    print(f"DEBUG: generator {sram_maglef_files_generator}")
+    print(f"DEBUG: installed sram {installed_sram_modules_names}")
     sram_modules_in_gds = []
     for sram in installed_sram_modules_names:
         if check_if_binary_has(sram, gds_ut_path):
             sram_modules_in_gds.append(sram)  # only the name of the module
 
+    print(f"DEBUG: detected sram {sram_modules_in_gds}")
     magicrc_file_path = parent_directory.parent.parent / 'tech-files' / 'sky130A.magicrc'
     magic_drc_tcl_path = parent_directory / 'magic_drc_check.tcl'
     design_magic_drc_mag_file_path = outputs_directory / f"{design_name}.magic.drc.mag"
I suspect it might be a missing 
PDKPATH
variable, but if it’s set, can you add the debugging statements above and share the results?
p
Here are my variables from these previous runs:
Copy code
PDK_ROOT: /home/prherrma/runners/r1/_work/pdk
    PDK: sky130A
    PDKPATH: /home/prherrma/runners/r1/_work/pdk/sky130A
Doing a run with those print debugs now
👍 1
m
I remember someone mentioning that there was a version of the pdk that did not have the sram libraries. Do you have anything in 
$PDK_ROOT/$PDK/libs.ref/sky130_sram_macros
?
p
Yeah, they’re all there:
Copy code
root@build_server:/home/prherrma/runners/r1/_work/pdk/sky130A# tree -f | grep "sky130_sram_2kbyte_1rw1r_32x512_8"
│       │   ├── ./libs.ref/sky130_sram_macros/gds/sky130_sram_2kbyte_1rw1r_32x512_8.gds
│       │   ├── ./libs.ref/sky130_sram_macros/lef/sky130_sram_2kbyte_1rw1r_32x512_8.lef
│       │   ├── ./libs.ref/sky130_sram_macros/lib/sky130_sram_2kbyte_1rw1r_32x512_8_TT_1p8V_25C.lib
│       │   ├── ./libs.ref/sky130_sram_macros/spice/sky130_sram_2kbyte_1rw1r_32x512_8.spice
│           ├── ./libs.ref/sky130_sram_macros/verilog/sky130_sram_2kbyte_1rw1r_32x512_8.v
Here's the output with the above debug:
Copy code
DEBUG: generator <generator object Path.glob at 0x7f22db920040>
DEBUG: installed sram []
DEBUG: detected sram []
m
I get
Copy code
DEBUG: generator <generator object Path.glob at 0x7fd37ab11930>
DEBUG: installed sram ['sky130_sram_2kbyte_1rw1r_32x512_8', 'sky130_sram_1kbyte_1rw1r_8x1024_8', 'sky130_sram_1kbyte_1rw1r_32x256_8', 'sram_1rw1r_32_256_8_sky130']
DEBUG: detected sram ['sky130_sram_2kbyte_1rw1r_32x512_8']
So this is where the problem is. Now we just need to figure out why it’s not finding the maglef directory. Does
Copy code
ls $PDK_ROOT/$PDK/libs.ref/sky130_sram_macros/maglef/*.mag
returns this?
Copy code
<PDK_ROOT>/sky130A/libs.ref/sky130_sram_macros/maglef/sky130_sram_1kbyte_1rw1r_32x256_8.mag
<PDK_ROOT>/sky130A/libs.ref/sky130_sram_macros/maglef/sky130_sram_1kbyte_1rw1r_8x1024_8.mag
<PDK_ROOT>/sky130A/libs.ref/sky130_sram_macros/maglef/sky130_sram_2kbyte_1rw1r_32x512_8.mag
<PDK_ROOT>/sky130A/libs.ref/sky130_sram_macros/maglef/sram_1rw1r_32_256_8_sky130.mag
Looks like you’re missing the 
maglef
directory.
p
yup
Copy code
# ls
gds  lef  lib  spice  verilog
m
What version of the PDK are you using?
Copy code
volare ls
Or are you not using volare?
p
I am following the flow in the CI template in the caravel_user_project, so the pdk is set up with 
volare-0.13.0
, as best I can tell.
Copy code
pdk: ["sky130A"]
    steps:
      - uses: actions/checkout@v2

      - name: Set up QEMU
        uses: docker/setup-qemu-action@v1

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1

      - name: Export Environment Variables
        run: |
          export WORK_DIR=${GITHUB_WORKSPACE}/../..
          echo "PDK_ROOT=${WORK_DIR}/pdk" >> $GITHUB_ENV
          echo "PDK=${{ matrix.pdk }}" >> $GITHUB_ENV
          echo "PDKPATH=${WORK_DIR}/pdk/${{ matrix.pdk }}" >> $GITHUB_ENV
          echo "OPENLANE_ROOT=${WORK_DIR}/caravel_user_project/openlane" >> $GITHUB_ENV

      - name: Install Caravel
        run: |
          make install

      - name: Setup Openlane
        run: |
          make openlane

      - name: Build PDK
        run: |
          make pdk-with-volare
Without the 
sky130A
symlink, the paths are:
Copy code
./pdk/volare/sky130/versions/78b7bc32ddb4b6f14f76883c2e2dc5b5de9d1cbc/sky130A/libs.ref/sky130_sram_macros/gds/sky130_sram_2kbyte_1rw1r_32x512_8.gds
./pdk/volare/sky130/versions/78b7bc32ddb4b6f14f76883c2e2dc5b5de9d1cbc/sky130A/libs.ref/sky130_sram_macros/lef/sky130_sram_2kbyte_1rw1r_32x512_8.lef
./pdk/volare/sky130/versions/78b7bc32ddb4b6f14f76883c2e2dc5b5de9d1cbc/sky130A/libs.ref/sky130_sram_macros/lib/sky130_sram_2kbyte_1rw1r_32x512_8_TT_1p8V_25C.lib
./pdk/volare/sky130/versions/78b7bc32ddb4b6f14f76883c2e2dc5b5de9d1cbc/sky130A/libs.ref/sky130_sram_macros/spice/sky130_sram_2kbyte_1rw1r_32x512_8.spice
./pdk/volare/sky130/versions/78b7bc32ddb4b6f14f76883c2e2dc5b5de9d1cbc/sky130A/libs.ref/sky130_sram_macros/verilog/sky130_sram_2kbyte_1rw1r_32x512_8.v
Just looked and the flow is installing an old version of the PDK:
Copy code
ifeq ($(PDK),sky130A)
	SKYWATER_COMMIT=f70d8ca46961ff92719d8870a18a076370b85f6c
	export OPEN_PDKS_COMMIT?=78b7bc32ddb4b6f14f76883c2e2dc5b5de9d1cbc
	export OPENLANE_TAG?=2023.07.19
	MPW_TAG ?= mpw-9e
I'll update this to 
OPEN_PDKS_COMMIT?=559a117b163cef2f920f33f30f6f690aa0b47e4c
and report back
👍 1
Copy code
Detected an SRAM module: sky130_sram_2kbyte_1rw1r_32x512_8
Pre-loading a maglef of the SRAM block: sky130_sram_2kbyte_1rw1r_32x512_8
Cell sky130_sram_2kbyte_1rw1r_32x512_8 read from path /home/prherrma/runners/r3/_work/pdk/sky130A/libs.ref/sky130_sram_macros/maglef
DRC check completed!
👍 1
Thanks so much for the help!
m
Thank you for reporting the problem.
3 Views
Open in Slack
PreviousNext
Powered by