What I'm mostly curious about is why that wasn't caught by the timing analysis ?

That is of course what we are delving into right now. The main issue was that Open RCX, the OpenROAD tool for extracting wire parasitics, was not available at the time that MPW-one was made. Magic extracts pretty detailed parasitics, but only in SPICE. I did figure out that I could do pretty accurate timing simulation with IRSIM, but it took me a long time to create the setup for sky130, and there were so many unknowns down that path that I would not be able to trust the results without correlating them against some other tools.

Interesting, I wouldn't have expected parasitics to cause hold violations.

The hold violations are all caused by clock skew. So the parasitics just make the clock skew worse.

I see. Thanks for the details !

out of curiosity was there a specific bug in the clock tree generation? or just general skew issues

It seems that the clock tree generator did not understand the concept of a balanced clock tree.

ouch!

Seems like a pretty fundamental misunderstanding! Sheesh...

huh ... I mean, even without parasitics, I would have expected just the gate delay of an unbalanced tree to show up a pretty high skew during timing analysis. Or do you mean balanced in the sense of placement ? (TBH I don't remember if the clock tree is done after placement here to try and make it "spatially" good)

I had many issues with MPW two and trying to get a reasonable clock tree. And I was still skeptical

The openlane team is now working on adding methodology to the clock tree generation to use results of post-layout extracted timing to make adjustments to the timing of clocks at the leaf nodes via engineering change orders. That should be available for MPW-three.

@Matthew Guthaus could you comment further on what a reasonable clock tree is, and why you were skeptical?

@Matt Venn it was mostly dealing with multiple different clocks. For example, if you want to create your own clock in the user space. If you don't get it recognized by OpenLane then Yosys will buffer it using its high fanout buffering algorithm which is not intended for clocks.

but when you saw the buffered clock tree - it looked ok? I'm just wondering if there was something we could have done to have caught this earlier

I didn't look at it, just the timing reports

all my projects passed timing, but I don't think that actually meant much with the missing values

What I find weird is that the parasitics made so much of a difference. Even without accounting for exact wire delays, there are default values for net delays and the clock buffers also have delays in the library and those should have already highlighted a fairly high skew in the clock.

Yeah, mine actually had ideal clock as well. 0.00 30.00 30.00 clock io_in[17] (rise edge) 0.00 30.00 clock network delay (ideal) (this was using a GPIO as the clock)

I'll check my timing reports and then re-run with the mpw3 tools, then we should at least get some info about whether that issue is fixed in mpw-3 tagged tools

Why is that? I spent time with some TCL code to do this

the only way to setup the GPIO is to use caravel's picorv32 to load some firmware to setup the IOs. But that doesn't work

Oh, I'm not even commenting on that.

RE STA timing on extracted CTS of the design I'm just checking

I think that is reporting violating paths which aren't found

ok ok.

yep

would be good if someone else could confirm.

@Matt Venn Maybe try adding

set_clock_skew -propagated

to the

sta.tcl

script

I'm also interested to hear back from @Tim Edwards about how he actually tested the design and saw hold violations. It would be good to repeat that test

Err,

set_propagated_clock wb_clk_i

maybe.

image.png

Heh, I wasn't far 😅

I just added to my script here and see what happens

Got it to generate for mpw-1 with propagated clocks.

great!

Added

set_propagated_clock [all_clocks]

to the SDC file.

would you mind adding that to the gdoc?

But if you modify

scripts/base.sdc

and re-run that should work.

what is the sdc file?

It's the file describing the clock constaints.

I only set the clock constraints in the config.tcl

the

base.sdc

file is in openlane itself.

image.png

openlane doesn't stop on timing error AFAIR

@Matt Venn: I am still reading through this long thread, but to answer one of your questions: We detected and measured hold violations in two different ways. The first was using IRSIM, which needs a Tcl script setup to get working which I put in the caravel repository under the irsim/ directory; and the second was using OpenROAD's newer tool Open RCX to get parasitic extraction for the mgmt_core module, and then plugging those values back into gate-level simulation. But iverilog doesn't do delay annotation, so we used another verilog simulator called "cvc" that does, and then I had to write a script change to open_pdks that will put the "specify" section back into the verilog for all of the standard cells.

Cool, lots to learn here. Have you seen we can also see violations by adding the extra line to the openlane scripts? But I don't really know what I'm doing here.

I am not familiar with the SDC script commands for setting up the timing constraints, but you need to both propagate the clock, and to tell OpenSTA to output all violations (or 1000, or whatever; the default seems to be 1, which isn't very helpful).

@Tim Edwards yeah, something else I noticed is that if you read min/max timing liberty file, it seems to use those in the same path. so the source clock delay is made with one corner and destination clock delay with another corner which sounds a bit pessimistic to me.

@tnt : Not sure about that. I know that generally, setup violations will be calculated at slow/low/hot while hold violations are calculated at fast/high/cold. If two different corners are used for the same calculation, though, I'd agree that's needlessly pessimistic.

@tnt I was able to analyze my design using propagated clocks, but interestingly there was a problem because inputs/outputs seem to always use "ideal" clocks even when you propagate clocks.

Startpoint: io_in[15] (input port clocked by io_in[17])
Endpoint: _3004_ (rising edge-triggered flip-flop clocked by io_in[17])
Path Group: io_in[17]
Path Type: min

Fanout     Cap    Slew   Delay    Time   Description
-----------------------------------------------------------------------------
                  0.00    0.00    0.00   clock io_in[17] (rise edge)
                          0.00    0.00   clock network delay (ideal)
                          0.00    0.00 v input external delay
                  0.00    0.00    0.00 v io_in[15] (in)
     2    0.00                           io_in[15] (net)
                  0.00    0.00    0.00 v hold564/A (sky130_fd_sc_hd__dlygate4sd3_1)
                  0.04    0.35    0.36 v hold564/X (sky130_fd_sc_hd__dlygate4sd3_1)
     2    0.00                           net564 (net)
                  0.04    0.00    0.36 v hold4/A (sky130_fd_sc_hd__dlygate4sd3_1)
                  0.04    0.37    0.72 v hold4/X (sky130_fd_sc_hd__dlygate4sd3_1)
     2    0.00                           net4 (net)
                  0.04    0.00    0.72 v hold563/A (sky130_fd_sc_hd__dlygate4sd3_1)
                  1.40    1.35    2.07 v hold563/X (sky130_fd_sc_hd__dlygate4sd3_1)
     2    0.44                           net563 (net)
                  1.42    0.12    2.20 v _1317_/B (sky130_fd_sc_hd__and2b_2)
                  0.03    0.51    2.71 v _1317_/X (sky130_fd_sc_hd__and2b_2)
     2    0.00                           _0825_ (net)
                  0.03    0.00    2.71 v hold567/A (sky130_fd_sc_hd__dlygate4sd3_1)
                  0.04    0.36    3.07 v hold567/X (sky130_fd_sc_hd__dlygate4sd3_1)
     2    0.00                           net567 (net)
                  0.04    0.00    3.07 v hold6/A (sky130_fd_sc_hd__dlygate4sd3_1)
                  0.83    0.90    3.97 v hold6/X (sky130_fd_sc_hd__dlygate4sd3_1)
     4    0.26                           net6 (net)
                  0.83    0.00    3.97 v hold566/A (sky130_fd_sc_hd__dlygate4sd3_1)
                  1.00    1.12    5.09 v hold566/X (sky130_fd_sc_hd__dlygate4sd3_1)
     6    0.32                           net566 (net)
                  1.30    0.43    5.53 v _1702_/A (sky130_fd_sc_hd__buf_1)
                  0.04    0.30    5.82 v _1702_/X (sky130_fd_sc_hd__buf_1)
     2    0.00                           _0946_ (net)
                  0.04    0.00    5.82 v hold19/A (sky130_fd_sc_hd__dlygate4sd3_1)
                  0.65    0.74    6.56 v hold19/X (sky130_fd_sc_hd__dlygate4sd3_1)
    10    0.20                           net19 (net)
                  0.73    0.17    6.73 v _1703_/A (sky130_fd_sc_hd__buf_1)
                  0.45    0.53    7.26 v _1703_/X (sky130_fd_sc_hd__buf_1)
    10    0.11                           _0947_ (net)
                  0.45    0.01    7.27 v _1710_/A (sky130_fd_sc_hd__buf_1)
                  0.08    0.21    7.48 v _1710_/X (sky130_fd_sc_hd__buf_1)
    10    0.02                           _0949_ (net)
                  0.08    0.00    7.49 v _1712_/B (sky130_fd_sc_hd__and2_2)
                  0.03    0.14    7.62 v _1712_/X (sky130_fd_sc_hd__and2_2)
     2    0.00                           _0546_ (net)
                  0.03    0.00    7.62 v _3004_/D (sky130_fd_sc_hd__dfxtp_2)
                                  7.62   data arrival time

                  0.00    0.00    0.00   clock io_in[17] (rise edge)
                         19.86   19.86   clock network delay (propagated)
                          0.00   19.86   clock reconvergence pessimism
                                 19.86 ^ _3004_/CLK (sky130_fd_sc_hd__dfxtp_2)
                         -0.01   19.84   library hold time
                                 19.84   data required time
-----------------------------------------------------------------------------
                                 19.84   data required time
                                 -7.62   data arrival time
-----------------------------------------------------------------------------
                                -12.22   slack (VIOLATED)

@Matthew Guthaus: I assume it uses an ideal clock because it does not have information about the drive or load at the pin. You can specify default assumptions for such pins, although it would be better to have a liberty file definition for the GPIO. I assume that the pin input comes from the gpio_control_block cell, so you could get pretty reasonable assumptions about the slew of the signal, at least.

@Tim Edwards I understand what your are saying but I think it's a little more general of a problem. The clock gpio in has a big clock network delay compared to the data gpio in. Since the data input is "clocked" by a virtual clock signal without actual implementation (since it is external to my project and off chip). I'm just not sure how to model it in STA. Maybe I should just remove that input delay so it only analyzes flop to flop timing..

@Matthew Guthaus: All of the GPIO pads share the same circuitry, so the only significant difference between timing of signals on external pins would be due to the finite drive strength of the gate. In this case, that gate would be the

sky130_fd_sc_hd__buf_2

connected to

user_gpio_in

in the cell

gpio_control_block

. I don't know offhand how to set up the STA configuration to tell it to assume the drive strength of a

buf_2

cell on all inputs, though.

This isn't an issue with the GPIO models. I'm modeling user_project_wrapper in isolation. The issue is that the external clock is ideal but the internal clock is not. This results in a lot of clock skew.

When I wrote vesta for qflow, I just had it report input-pin-to-flop and flop-to-output-pin timing values and let the end-user decide whether or not they're acceptable. There's no way to automatically establish timing between externally-applied signals automatically, but I would assume that for OpenSTA there's some way to do that in the SDC file. Otherwise, yes, you either have to tell it to ignore input timing paths or get very specific and detailed about which paths are false timing paths.