https://open-source-silicon.dev logo
Join Slack
Powered by
hello , I was trying to use Caravel template to st...
# general
m
hello , I was trying to use Caravel template to start using OpenLane but, when i try to run the example project I got this, Does anyone know how to solve this issue?
j
Hi! It looks like some permission issue on a configu file... Can try to run 
ls -l /mnt/.../config_in.tcl
? With 
/mnt/.../config_in.tcl
replaced by the file name you have.
m
first I would like to say thanks for you response , I run the command and I got this
m
Did you install or execute any part of openlane as 
root
?
m
no I didn't I just clone it from github and follow the tutorial video from efabless
j
It looks like all files were created with the current user, all fine here... Now what do you have inside 
23_06_07_16_10
, could you also run 
ls -l 23_06_07_16_10
?
It was the 
config_in.tcl
in it that seemed to be unreadable by python...
m
All folders here are empty it doesn't contain the config_in.tcl file
j
Oh right, it was trying to open the directory to write in it (
open(output, "w")
), not read it...
Therefore, could further try 
touch 23_06_07_16_10/config_in.tcl
? To try to create the file?
I see this is being created in 
/mnt/...../
It could be a docker or any other kindof virtual environment, in which case these disks might be mounted read-only.
m
I create the config_in.tcl in 23_06_07_16_10 folder but each time i tried to (make user_proj_example) command it create a new folder which will be a empty one
so it can't find the config_in.tcl file
j
Well, why not try the same thing python does then... I propose to first remove he file you just created: 
rm 23_06_07_16_10/config_in.tcl
Then try to use the same method as python did, as seen on the screenshot: 
python3 -c "with open('23_06_07_16_10/config_in.tcl', 'w') as f: pass"
Even if it is an empty directory, it would be able to create a new file to write to (
'w'
argument to 
open()
)
m
I run it but it can't create the file
oh sorry
i gave him wrong path it create it
j
no worries
It also does not show the same error message when it is a non-existing path, so it seems like non-existing file/directory is not the problem here
m
when i run (make user_proj_example) it create a new folder and it will be empty one , so can I give it the path to the folder that contain the config_in.tcl which i just create instead of creating a new one ?
j
You could try that, but it seems that the bug is elsewhere: it is not expected to require the user to create an empty file so that the script work
Though, useful for debugging...
Did the python command to create the file work?
Also, the result of 
mount
would show all the state of the filesystem, giving more insight of how these /mnt/... storage are configured
m
how can I run it ?
j
the command 
mount
would give you a listing of what is currently mounted everywhere, and that is something I could inspect and hopefully see some "read-only" indication in here (often 
ro
)
Copy code
$ mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,noexec,relatime,size=10240k,nr_inodes=1001506,mode=755,inode64)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,inode64)
/dev/dm-0 on / type ext4 (rw,relatime)
tmpfs on /run type tmpfs (rw,nosuid,nodev,size=1607264k,nr_inodes=819200,mode=755,inode64)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
/dev/nvme0n1p1 on /boot type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=utf8,shortname=mixed,errors=remount-ro)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,relatime,inode64)
Here for instance, I have several disks mounted at various locations and none is read-only (all setup as read/write: 
...(rw,...
)
m
I got this $ mount sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) udev on /dev type devtmpfs (rw,nosuid,relatime,size=5994112k,nr_inodes=1498528,mode=755,inode64) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=1212576k,mode=755,inode64) /dev/nvme0n1p3 on / type ext4 (rw,noatime,errors=remount-ro) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64) tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64) cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5 ,direct,pipe_ino=22658) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime) debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime) tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime) fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime) configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime) ramfs on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700) /var/lib/snapd/snaps/bare_5.snap on /snap/bare/5 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hid e) /var/lib/snapd/snaps/core18_2745.snap on /snap/core18/2745 type squashfs (ro,nodev,relatime,errors=continue ,x-gdu.hide) /var/lib/snapd/snaps/gnome-3-28-1804_194.snap on /snap/gnome-3-28-1804/194 type squashfs (ro,nodev,relatime ,errors=continue,x-gdu.hide) /var/lib/snapd/snaps/gnome-3-28-1804_198.snap on /snap/gnome-3-28-1804/198 type squashfs (ro,nodev,relatime ,errors=continue,x-gdu.hide) /var/lib/snapd/snaps/gtk-common-themes_1534.snap on /snap/gtk-common-themes/1534 type squashfs (ro,nodev,re latime,errors=continue,x-gdu.hide) /var/lib/snapd/snaps/gtk-common-themes_1535.snap on /snap/gtk-common-themes/1535 type squashfs (ro,nodev,re latime,errors=continue,x-gdu.hide) /var/lib/snapd/snaps/magnus_388.snap on /snap/magnus/388 type squashfs (ro,nodev,relatime,errors=continue,x -gdu.hide) /var/lib/snapd/snaps/magnus_393.snap on /snap/magnus/393 type squashfs (ro,nodev,relatime,errors=continue,x -gdu.hide) /var/lib/snapd/snaps/snapd_19122.snap on /snap/snapd/19122 type squashfs (ro,nodev,relatime,errors=continue ,x-gdu.hide) /dev/nvme0n1p1 on /boot/efi type vfat (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=iso8859-1,s hortname=mixed,errors=remount-ro) /dev/nvme0n1p4 on /home type ext4 (rw,noatime,errors=remount-ro) /dev/nvme0n1p2 on /recovery type ext4 (rw,noatime,errors=remount-ro) /dev/sda2 on /mnt/Free type ext4 (rw,nosuid,nodev,relatime,x-gvfs-show) /dev/sda5 on /mnt/0e6e44af-e5d7-4180-b808-68fe5c5a2775 type ext4 (rw,nosuid,nodev,relatime,x-gvfs-show) /dev/sda6 on /mnt/0075c426-3aad-46cd-9b6c-a7ac07c801c7 type ext4 (rw,nosuid,nodev,relatime,x-gvfs-show) tmpfs on /run/qemu type tmpfs (rw,nosuid,nodev,relatime,mode=755,inode64) binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime) tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=1212572k,nr_inodes=303143,mode=700,uid=10 00,gid=1000,inode64) gvfsd-fuse on /run/user/1000/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000 ) portal on /run/user/1000/doc type fuse.portal (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000) /var/lib/snapd/snaps/snapd_19361.snap on /snap/snapd/19361 type squashfs (ro,nodev,relatime,errors=continue ,x-gdu.hide) /var/lib/snapd/snaps/core18_2751.snap on /snap/core18/2751 type squashfs (ro,nodev,relatime,errors=continue ,x-gdu.hide)
j
Thanks!
m
and this is the result for mnt $mount | grep mnt /dev/sda2 on /*mnt*/Free type ext4 (rw,nosuid,nodev,relatime,x-gvfs-show) /dev/sda5 on /*mnt*/0e6e44af-e5d7-4180-b808-68fe5c5a2775 type ext4 (rw,nosuid,nodev,relatime,x-gvfs-show) /dev/sda6 on /*mnt*/0075c426-3aad-46cd-9b6c-a7ac07c801c7 type ext4 (rw,nosuid,nodev,relatime,x-gvfs-show)
j
Yes I was looking at this as well... It looks all mounted as read/write 
rw
... Not sure what manages these volumes, likey a pop-os thing but it does not appear to be a problem here..
m
i would like to notice that i tried to change the path to the "config_in.tcl" file but it gave the same error (File "/openlane/scripts/config/tcl.py", line 426, in from_json with open("/mnt/0075c426-3aad-46cd-9b6c-a7ac07c801c7/Trainning/assignment/Dr.yacoup_Assignment/4bit_add er_assignment/4bit_adder_efabless/openlane/user_proj_example/runs/23_06_07_16_10/config_in.tcl", "w") as f: PermissionError: [Errno 13] Permission denied: '/mnt/0075c426-3aad-46cd-9b6c-a7ac07c801c7/Trainning/assignm ent/Dr.yacoup_Assignment/4bit_adder_assignment/4bit_adder_efabless/openlane/user_proj_example/runs/23_06_07 _16_10/config_in.tcl' )
j
Could you link the instructions you did follow? And the OpenLane repo you did clone eventually as well... There are several forks of that project
It seems to use docker, maybe it spins a docker which cannot access to this path, then complains, then turns docker off... I am not too familiar with openlane yet
Where do you specify the custom config file? it seems like it is using the exact same path here by the look of the error mesage.
Something you could try for a config file path: 
/tmp/config_in.tcl
, most systems, dockerized or not, have a writable 
/tmp
...
or even 
/dev/null
as a way to check. This will not work, but this should show a different error message.
And then we'd know it did at least applied our config file path indication
docker is great for wrapping up tools in a box with a nice handle, but box itsef adds a lot of difficulty for debugging as you can see...
a bit easier to install, a bit harder to run... that is the compromise 🙂
✅ 1
m
I tried the tmp/config_in.tcl and i got this
j
Just to make sure, in case that was implicity... did you try 
/tmp/config_in.tcl
or 
tmp/config_in.tcl
?
Interesting here though: we do have a different message,but TCL is still trying open the file at the generated path from before
It seems to happen roughly at the same step as before:

https://open-source-silicon.slack.com/files/U05BS2K8YJV/F05C3BUTAN4/image.png▾

But I would bet that after python did successfully create the file at the wrong place (
/tmp/config_in.tcl
we did set for testing), then tcl kept the original config location, so complains about it... Though at least, likely the python could create the TCL confg file!
So there seems to be something very weird about that path...
m
just a random Q what dose the output parameter here do ?
this is the tcl config file and you can see where I change the path of /tmp/config_in.tcl
j
OpenLane uses tools which are configured in a TCL language: a really simple/basic language For some reason the OpenLane authors wanted to write python rather than TCL scripts as part of OpenLane (I like TCL too though 😛 but using python allows to have relatively well maintained packages for about everything like fetching and decoding a .json file from a server etc.), and to do some glue between the python environment and the TCL environment, they write a TCL configuration file from python, with the data coming from JSON.
the output file contains the path to the .tcl file that needs to be written by that function.
Indeed, it looks like everything is run in Docker, so there might be different rules than the one we have here for the filesystem...
In order to inspect that all once again, could you insert that same "mount" command in the python script?
You could add this line just before 
with open...
you already modified: 
import os; os.system(f"mount; ls -l {output}")
m
new error !
[ERROR]: overlay on / type overlay (rw,relatime,lowerdir=/home/mahdesouqi/.local/share/docker/overlay2/l/LF MJXEENALMF6J4EC34MKKHKDP/home/mahdesouqi/.local/share/docker/overlay2/l/63FPJU2T5UMFUVOOKA6LTABFKG/home/m ahdesouqi/.local/share/docker/overlay2/l/IKAVQ5GEHMBEMTDDGEW24ZHXWR:/home/mahdesouqi/.local/share/docker/ov erlay2/l/CWBSFHNRRWAN5OMWSGDCFWPOEU:/home/mahdesouqi/.local/share/docker/overlay2/l/EJBAEHVS452ZCB3ZJBZWAAT DJM/home/mahdesouqi/.local/share/docker/overlay2/l/IRVMFRHEJSK53YGYLRMI4T56C2/home/mahdesouqi/.local/shar e/docker/overlay2/l/EIHCHVKMGFP27BVISYZMWPVBY5:/home/mahdesouqi/.local/share/docker/overlay2/l/BOPRBAOCTK6X UOWL4K3FNK5YRQ/home/mahdesouqi/.local/share/docker/overlay2/l/V54OOVZFQKGOBHOYVXXKA2FXOW/home/mahdesouqi/ .local/share/docker/overlay2/l/UJNPGT2LQTJD4BOC6YTPVDWQTT:/home/mahdesouqi/.local/share/docker/overlay2/l/5 B32N33W2G7VWBDBGLLY3PJA7C/home/mahdesouqi/.local/share/docker/overlay2/l/ZRQGLEAUDTIZBJAAPCJ2M23K5A/home/ mahdesouqi/.local/share/docker/overlay2/l/FSZ66UBU6K2ENLYYBGD26P66AC:/home/mahdesouqi/.local/share/docker/o verlay2/l/DNGBEAVKHLHCNB3XNJCVGPMGOR:/home/mahdesouqi/.local/share/docker/overlay2/l/I6EB6CFSB7EHKV2XU3AMQG FEAC/home/mahdesouqi/.local/share/docker/overlay2/l/R2RZK2RR62HBBM57B7QJ7ZQKNL/home/mahdesouqi/.local/sha re/docker/overlay2/l/CHY233TJM73VTYD364XZ5H6IBS:/home/mahdesouqi/.local/share/docker/overlay2/l/QXKWUYJNCGL SJBGOZI2PSG5GTD/home/mahdesouqi/.local/share/docker/overlay2/l/V2DRHDREB5LOBPHUMRRQ7GQ5RH/home/mahdesouqi /.local/share/docker/overlay2/l/7ELMBAE3QFG3XISJUVOKSPA4R7:/home/mahdesouqi/.local/share/docker/overlay2/l/ 64CYDRL5QHZLNED4VDKCDRXOWN/home/mahdesouqi/.local/share/docker/overlay2/l/XL7OCGBHBCUMEHTLR5BR45FKDD/home /mahdesouqi/.local/share/docker/overlay2/l/VVVXTNMNKNRTAC7YVARKSLM4MC:/home/mahdesouqi/.local/share/docker/ overlay2/l/A3NI2H3TPMMXZ447FO4REDJ46U:/home/mahdesouqi/.local/share/docker/overlay2/l/FTYZPVGZLFM22ZJ6F5XDA MUYEB/home/mahdesouqi/.local/share/docker/overlay2/l/5THLEXJYSDJKH5RZ4FFFRAYIT2/home/mahdesouqi/.local/sh are/docker/overlay2/l/3N6ZTBLBTYZ7YOZ47LOSELBDEI:/home/mahdesouqi/.local/share/docker/overlay2/l/3QBZRP5RIA LMGXOE673FP2Y6PX/home/mahdesouqi/.local/share/docker/overlay2/l/RICCNM7VN77XZHMAOTEZNNSPSH/home/mahdesouq i/.local/share/docker/overlay2/l/37PZENJJ43PD36A6ETNVJBNFCL:/home/mahdesouqi/.local/share/docker/overlay2/l /BSSUNRZR2IM7NQHAAMHEZULJRZ,upperdir=/home/mahdesouqi/.local/share/docker/overlay2/c4ceb230ed701fdd6ea6abde c35d77230a2429392a4c72e0f821381cfddc06ed/diff,workdir=/home/mahdesouqi/.local/share/docker/overlay2/c4ceb23 0ed701fdd6ea6abdec35d77230a2429392a4c72e0f821381cfddc06ed/work,userxattr) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev type tmpfs (rw,nosuid,size=65536k,mode=755,uid=1000,gid=1000,inode64) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=100004,mode=620,ptmxmode=666) sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime) cgroup on /sys/fs/cgroup type cgroup2 (ro,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot) mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime) shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k,uid=1000,gid=1000,inode64) /dev/sda6 on /pdk type ext4 (rw,nosuid,nodev,relatime) /dev/sda6 on /openlane type ext4 (rw,nosuid,nodev,relatime) /dev/nvme0n1p4 on /etc/resolv.conf type ext4 (rw,noatime,errors=remount-ro) /dev/nvme0n1p4 on /etc/hostname type ext4 (rw,noatime,errors=remount-ro) /dev/nvme0n1p4 on /etc/hosts type ext4 (rw,noatime,errors=remount-ro) /dev/sda6 on /mnt/0075c426-3aad-46cd-9b6c-a7ac07c801c7/Trainning/assignment/Dr.yacoup_Assignment/4bit_adder _assignment/4bit_adder_efabless type ext4 (rw,nosuid,nodev,relatime) /dev/sda6 on /mnt/0075c426-3aad-46cd-9b6c-a7ac07c801c7/Trainning/assignment/Dr.yacoup_Assignment/4bit_adder _assignment/4bit_adder_efabless/dependencies/pdk type ext4 (rw,nosuid,nodev,relatime) udev on /dev/null type devtmpfs (rw,nosuid,relatime,size=5994112k,nr_inodes=1498528,mode=755,inode64) udev on /dev/random type devtmpfs (rw,nosuid,relatime,size=5994112k,nr_inodes=1498528,mode=755,inode64) udev on /dev/full type devtmpfs (rw,nosuid,relatime,size=5994112k,nr_inodes=1498528,mode=755,inode64) udev on /dev/tty type devtmpfs (rw,nosuid,relatime,size=5994112k,nr_inodes=1498528,mode=755,inode64) udev on /dev/zero type devtmpfs (rw,nosuid,relatime,size=5994112k,nr_inodes=1498528,mode=755,inode64) udev on /dev/urandom type devtmpfs (rw,nosuid,relatime,size=5994112k,nr_inodes=1498528,mode=755,inode64) devpts on /dev/console type devpts (rw,nosuid,noexec,relatime,gid=100004,mode=620,ptmxmode=666) proc on /proc/bus type proc (ro,nosuid,nodev,noexec,relatime) proc on /proc/fs type proc (ro,nosuid,nodev,noexec,relatime) proc on /proc/irq type proc (ro,nosuid,nodev,noexec,relatime) proc on /proc/sys type proc (ro,nosuid,nodev,noexec,relatime) proc on /proc/sysrq-trigger type proc (ro,nosuid,nodev,noexec,relatime) tmpfs on /proc/asound type tmpfs (ro,relatime,uid=1000,gid=1000,inode64) tmpfs on /proc/acpi type tmpfs (ro,relatime,uid=1000,gid=1000,inode64) udev on /proc/kcore type devtmpfs (rw,nosuid,relatime,size=5994112k,nr_inodes=1498528,mode=755,inode64) udev on /proc/keys type devtmpfs (rw,nosuid,relatime,size=5994112k,nr_inodes=1498528,mode=755,inode64) udev on /proc/timer_list type devtmpfs (rw,nosuid,relatime,size=5994112k,nr_inodes=1498528,mode=755,inode64 ) tmpfs on /proc/scsi type tmpfs (ro,relatime,uid=1000,gid=1000,inode64) tmpfs on /sys/firmware type tmpfs (ro,relatime,uid=1000,gid=1000,inode64) ls: cannot access /mnt/0075c426-3aad-46cd-9b6c-a7ac07c801c7/Trainning/assignment/Dr.yacoup_Assignment/4bit_ adder_assignment/4bit_adder_efabless/openlane/user_proj_example/runs/23_06_07_18_33/config_in.tcl: No such file or directory
j
Oh right, "no such file or directory" because we'd want to list the information of the directory, not the file itself..., my bad.
Though, the large error message here is our output! This is how the python script sees its surrounding directories itself.
We have a veeery long line that was not there before: 
overlay on / type overlay (rw,relatime,lowerdir=/home/mahdesouqi/.local/share/docker/overlay2/l/LFMJXEENALMF6J4EC34MKKHKDP:/home/mahdesouqi/.local/share/docker/overlay2/l/63FPJU2T5UMFUVOOKA6LTABFKG:/home/mahdesouqi/.local/share/docker/overlay2/l/IKAVQ5GEHMBEMTDDGEW24ZHXWR:/home/mahdesouqi/.local/share/docker/overlay2/l/CWBSFHNRRWAN5OMWSGDCFWPOEU:/home/mahdesouqi/.local/share/docker/overlay2/l/EJBAEHVS452ZCB3ZJBZWAATDJM:/home/mahdesouqi/.local/share/docker/overlay2/l/IRVMFRHEJSK53YGYLRMI4T56C2:/home/mahdesouqi/.local/share/docker/overlay2/l/EIHCHVKMGFP27BVISYZMWPVBY5:/home/mahdesouqi/.local/share/docker/overlay2/l/BOPRBAOCTK6XUOWL4K3FNK5YRQ:/home/mahdesouqi/.local/share/docker/overlay2/l/V54OOVZFQKGOBHOYVXXKA2FXOW:/home/mahdesouqi/.local/share/docker/overlay2/l/UJNPGT2LQTJD4BOC6YTPVDWQTT:/home/mahdesouqi/.local/share/docker/overlay2/l/5B32N33W2G7VWBDBGLLY3PJA7C:/home/mahdesouqi/.local/share/docker/overlay2/l/ZRQGLEAUDTIZBJAAPCJ2M23K5A:/home/mahdesouqi/.local/share/docker/overlay2/l/FSZ66UBU6K2ENLYYBGD26P66AC:/home/mahdesouqi/.local/share/docker/overlay2/l/DNGBEAVKHLHCNB3XNJCVGPMGOR:/home/mahdesouqi/.local/share/docker/overlay2/l/I6EB6CFSB7EHKV2XU3AMQGFEAC:/home/mahdesouqi/.local/share/docker/overlay2/l/R2RZK2RR62HBBM57B7QJ7ZQKNL:/home/mahdesouqi/.local/share/docker/overlay2/l/CHY233TJM73VTYD364XZ5H6IBS:/home/mahdesouqi/.local/share/docker/overlay2/l/QXKWUYJNCGLSJBGOZI2PSG5GTD:/home/mahdesouqi/.local/share/docker/overlay2/l/V2DRHDREB5LOBPHUMRRQ7GQ5RH:/home/mahdesouqi/.local/share/docker/overlay2/l/7ELMBAE3QFG3XISJUVOKSPA4R7:/home/mahdesouqi/.local/share/docker/overlay2/l/64CYDRL5QHZLNED4VDKCDRXOWN:/home/mahdesouqi/.local/share/docker/overlay2/l/XL7OCGBHBCUMEHTLR5BR45FKDD:/home/mahdesouqi/.local/share/docker/overlay2/l/VVVXTNMNKNRTAC7YVARKSLM4MC:/home/mahdesouqi/.local/share/docker/overlay2/l/A3NI2H3TPMMXZ447FO4REDJ46U:/home/mahdesouqi/.local/share/docker/overlay2/l/FTYZPVGZLFM22ZJ6F5XDAMUYEB:/home/mahdesouqi/.local/share/docker/overlay2/l/5THLEXJYSDJKH5RZ4FFFRAYIT2:/home/mahdesouqi/.local/share/docker/overlay2/l/3N6ZTBLBTYZ7YOZ47LOSELBDEI:/home/mahdesouqi/.local/share/docker/overlay2/l/3QBZRP5RIALMGXOE673FP2Y6PX:/home/mahdesouqi/.local/share/docker/overlay2/l/RICCNM7VN77XZHMAOTEZNNSPSH:/home/mahdesouqi/.local/share/docker/overlay2/l/37PZENJJ43PD36A6ETNVJBNFCL:/home/mahdesouqi/.local/share/docker/overlay2/l/BSSUNRZR2IM7NQHAAMHEZULJRZ,upperdir=/home/mahdesouqi/.local/share/docker/overlay2/c4ceb230ed701fdd6ea6abdec35d77230a2429392a4c72e0f821381cfddc06ed/diff,workdir=/home/mahdesouqi/.local/share/docker/overlay2/c4ceb230ed701fdd6ea6abdec35d77230a2429392a4c72e0f821381cfddc06ed/work,userxattr)
This is how docker configures the storage
Unfolded, that line becomes:
Copy code
lowerdir

/home/mahdesouqi/.local/share/docker/overlay2/l/LFMJXEENALMF6J4EC34MKKHKDP

/home/mahdesouqi/.local/share/docker/overlay2/l/63FPJU2T5UMFUVOOKA6LTABFKG

/home/mahdesouqi/.local/share/docker/overlay2/l/IKAVQ5GEHMBEMTDDGEW24ZHXWR

/home/mahdesouqi/.local/share/docker/overlay2/l/CWBSFHNRRWAN5OMWSGDCFWPOEU

/home/mahdesouqi/.local/share/docker/overlay2/l/EJBAEHVS452ZCB3ZJBZWAATDJM

/home/mahdesouqi/.local/share/docker/overlay2/l/IRVMFRHEJSK53YGYLRMI4T56C2

/home/mahdesouqi/.local/share/docker/overlay2/l/EIHCHVKMGFP27BVISYZMWPVBY5

/home/mahdesouqi/.local/share/docker/overlay2/l/BOPRBAOCTK6XUOWL4K3FNK5YRQ

/home/mahdesouqi/.local/share/docker/overlay2/l/V54OOVZFQKGOBHOYVXXKA2FXOW

/home/mahdesouqi/.local/share/docker/overlay2/l/UJNPGT2LQTJD4BOC6YTPVDWQTT

/home/mahdesouqi/.local/share/docker/overlay2/l/5B32N33W2G7VWBDBGLLY3PJA7C

/home/mahdesouqi/.local/share/docker/overlay2/l/ZRQGLEAUDTIZBJAAPCJ2M23K5A

/home/mahdesouqi/.local/share/docker/overlay2/l/FSZ66UBU6K2ENLYYBGD26P66AC

/home/mahdesouqi/.local/share/docker/overlay2/l/DNGBEAVKHLHCNB3XNJCVGPMGOR

/home/mahdesouqi/.local/share/docker/overlay2/l/I6EB6CFSB7EHKV2XU3AMQGFEAC

/home/mahdesouqi/.local/share/docker/overlay2/l/R2RZK2RR62HBBM57B7QJ7ZQKNL

/home/mahdesouqi/.local/share/docker/overlay2/l/CHY233TJM73VTYD364XZ5H6IBS

/home/mahdesouqi/.local/share/docker/overlay2/l/QXKWUYJNCGLSJBGOZI2PSG5GTD

/home/mahdesouqi/.local/share/docker/overlay2/l/V2DRHDREB5LOBPHUMRRQ7GQ5RH

/home/mahdesouqi/.local/share/docker/overlay2/l/7ELMBAE3QFG3XISJUVOKSPA4R7

/home/mahdesouqi/.local/share/docker/overlay2/l/64CYDRL5QHZLNED4VDKCDRXOWN

/home/mahdesouqi/.local/share/docker/overlay2/l/XL7OCGBHBCUMEHTLR5BR45FKDD

/home/mahdesouqi/.local/share/docker/overlay2/l/VVVXTNMNKNRTAC7YVARKSLM4MC

/home/mahdesouqi/.local/share/docker/overlay2/l/A3NI2H3TPMMXZ447FO4REDJ46U

/home/mahdesouqi/.local/share/docker/overlay2/l/FTYZPVGZLFM22ZJ6F5XDAMUYEB

/home/mahdesouqi/.local/share/docker/overlay2/l/5THLEXJYSDJKH5RZ4FFFRAYIT2

/home/mahdesouqi/.local/share/docker/overlay2/l/3N6ZTBLBTYZ7YOZ47LOSELBDEI

/home/mahdesouqi/.local/share/docker/overlay2/l/3QBZRP5RIALMGXOE673FP2Y6PX

/home/mahdesouqi/.local/share/docker/overlay2/l/RICCNM7VN77XZHMAOTEZNNSPSH

/home/mahdesouqi/.local/share/docker/overlay2/l/37PZENJJ43PD36A6ETNVJBNFCL

/home/mahdesouqi/.local/share/docker/overlay2/l/BSSUNRZR2IM7NQHAAMHEZULJRZ


upperdir

/home/mahdesouqi/.local/share/docker/overlay2/c4ceb230ed701fdd6ea6abdec35d77230a2429392a4c72e0f821381cfddc06ed/diff


workdir

/home/mahdesouqi/.local/share/docker/overlay2/c4ceb230ed701fdd6ea6abdec35d77230a2429392a4c72e0f821381cfddc06ed/work
These are all the system disk used by docker, we can skip that... Likely not what we are after.
The disk with the config file is: 
/dev/sda6 on /*mnt*/0075c426-3aad-46cd-9b6c-a7ac07c801c7 type ext4 (rw,nosuid,nodev,relatime,x-gvfs-show)
It is also there on your docker environment...
Copy code
/dev/sda6 on /pdk type ext4 (rw,nosuid,nodev,relatime)
/dev/sda6 on /openlane type ext4 (rw,nosuid,nodev,relatime)
/dev/sda6 on /mnt/0075c426-3aad-46cd-9b6c-a7ac07c801c7/Trainning/assignment/Dr.yacoup_Assignment/4bit_adder_assignment/4bit_adder_efabless type ext4 (rw,nosuid,nodev,relatime)
/dev/sda6 on /mnt/0075c426-3aad-46cd-9b6c-a7ac07c801c7/Trainning/assignment/Dr.yacoup_Assignment/4bit_adder_assignment/4bit_adder_efabless/dependencies/pdk type ext4 (rw,nosuid,nodev,relatime)
And on that list, we do have some read-write storage for that config file path: 
/mnt/0075c426-3aad-46cd-9b6c-a7ac07c801c7/Trainning/assignment/Dr.yacoup_Assignment/4bit_adder_assignment/4bit_adder_efabless
Maybe the path name was too long? Could you try to mount that disk at another location? I'll try to build-up some example command you can use for that...
Tip: you can resize that side panel! Just discovered that. It helps with all the large text pastes
m
thanks for the tip
you ask me where I got the project from this is the link to to caravel repo that I clone it from https://github.com/efabless/caravel_user_project and this is the tutorial video

https://www.youtube.com/watch?v=MNuoYz_MM-c▾

j
Thanks! I'll use that to learn myself then 🙂
m
last Q is the problem in my device or in the caravel template ?
because it's work fine in the video
j
I would say the problem is from the "setup": combination of tools and system and software version, etc.
So nothing to do with the content of the design (like what maybe @Mariam Rakka for which I am a bit clueless ^_^') and rather a problem with the "device".
I mean nothing wrong with the CPU or HDD or anything like that...
And it is possible that software works one day, and stop the other, because the maintainer updated some package and it broke the scripts from OpenLane, or a server went down, or anything else happening on other people's machine ^_^
Do you have any way to eject the disk you used for storing the PDK?
Then we will try to attach it under a different, shorter name, just to check if related to our error here...
As the path is kind of long, maybe docker is not happy about paths this long.
m
not really i run this on my own laptop
I'm still a fresh graduate and I don't know a lot of these things
j
Yes, and to be honest, the people who knows a lot about things like UNIX command line, TCL scripting, C programming, Docker and ASICs are likely very few, and possibly, half of them are on this server trying to build tools that work for everyone...
I think you did install these large OpenLane data on a separate disk... Just to check if that is the problem, you could try to install these large data in your home directory instead.
m
I will try this and thanks a lot for your help i really appreciate
j
For instance, running this again:

https://www.youtube.com/watch?v=MNuoYz_MM-c▾

But from somewhere in the home directory instead...
(or any directory under your home)
Hope that you get this sorted out!
so that you can play with ASICs : )
Somehow, the idea that everyone get to install a full ASIC toolchain on its own machine is insane! Kudos to the OpenLane team and everyone teaching it.
m
I just move the folder to the home and try to run it and i got a different result but still not working \
m
From here, make sure you have 
OPENLANE_ROOT
, 
PDK_ROOT
, and 
PDK
set before running 
make setup
.
m
yeah I've just add it but still the same result
m
Try 
echo $OPENLANE_ROOT
m
$ echo $OPENLANE_ROOT /home/mahdesouqi/forTest/4bit_adder_assignment/4bit_adder_efabless/openlane/dependencies/openlane_src
m
Run 
make user_proj_example
from 
~/forTest?4bit_adder_assignment/4bit_adder_efabless
m
yes I tried this you can see the result here

https://open-source-silicon.slack.com/files/U05BS2K8YJV/F05BCCX9SUV/image.png▾

m
Ouch. You don’t want your 
OPENLANE_ROOT
to be under your user project.
m
and that means ??
j
I understand that as having one directory with the openlane sources, and one directory with the project sources, rather than having the openlane environment part of the project.
m
At least you don’t want it to be under 
openlane
. The instructions say,
Copy code
cd <project_name> # project_name is the name of your repo

mkdir dependencies

export OPENLANE_ROOT=$(pwd)/dependencies/openlane_src # you need to export this whenever you start a new shell

export PDK_ROOT=$(pwd)/dependencies/pdks # you need to export this whenever you start a new shell

# export the PDK variant depending on your shuttle, if you don't know leave it to the default

# for sky130 MPW shuttles....
export PDK=sky130A
If 
~/forTest/4bit_adder_assignment/4bit_adder_efabless
is if your cloned 
caravel_user_project
, then you should have 
~/forTest/4bit_adder_assignment/4bit_adder_efabless/dependencies
j
So if I got it:
Copy code
.../4bit_adder_efabless/openlane/dependencies/openlane_src
    ^^^^^^^^^^^^^^^^^^^          ^^^^^^^^^^^^
    this is the project          this should be outside the openlane directory, directly under the project directory
👍 1
m
OPENLANE_ROOT=~/forTest/4bit_adder_assignment/4bit_adder_efabless/dependencies/openlane_src
PDK_ROOT=/forTest/4bit_adder_assignment/4bit_adder_efabless/dependencies/pdks
j
<josuah> takes note for himself as well...
m
yes there is one already here $*/forTest/4bit_adder_assignment/4bit_adder_efabless*$ ls | grep dependencies dependencies
should i run the make command in the openlane_src ?
m
Nope. Everything runs in your 
caravel_user_project
cloned directory. You may need to rerun 
make setup
after fixing 
OPENLANE_ROOT
and 
PDK_ROOT
.
m
ok i will try this
👍 1
should i clone it again or just fix the 
OPENLANE_ROOT
and 
PDK_ROOT
and do 
make setup
?
m
If you haven’t changed much, I suggest erasing everything and starting over.
👍 1
m
I've clone it again and do the 
make setup
but still have the same error 😞
and i make sure that all of the env var are there
do you have any idea what goes wrong ?
j
This leaves off any possibility of problem that goes along with /mnt setup drives, and tell us more about the kind of problem that this might be.
But we still have to fix that bug.
m
yeah I will try on it later after make it work
j
I do not see the exact same line number for 
with open(output, "w") as f:
and indeed, the introduction video shows a different branch (version) for the repository. Which branch did you clone? You can check with 
git status
m
no it's the same line #473
I didn't follow the video i follow the repo instruction
j
Yes ok
m
I tried to copy and past it still the same error but it's now at line #489 😞
j
Do you mean to copy-paste that file? I was only remarking that ti was a different version than what you have on the repo.
m
even so it didn't make any difference
also , even if I'm working on old version of it it should work
j
It seems like it is also possible to run a virtual machine. Even if you are running pop-os which is ubuntu based, you cal also try in a Ubuntu Server virtual machine just like in CI automation script https://github.com/efabless/caravel_user_project/blob/main/.github/workflows/user_project_ci.yml#L12
That way, you would move away everything that could be a problem from your host, and offer a way to accurately reproduce the problem.
For short: rather than trying to solve this difficult bug, start by finding another environment where things work well.
[edit: removed a few older messages not relevant anymore]
It is not a complete fix, but could let you get things going until this gets fixed.
m
Thanks for looking into this @Josuah Demangeon. @Mahde Souqi What do you get with
Copy code
id -u $USER
id -g $USER
j
I think that from the make output, we can guess it to be 
1000
for both:
From previous output:
m
That looks like it might be the root group. @Mahde Souqi what does this return? 
grep :1000: /etc/passwd /etc/group
j
on my local machine, I have 1000 for the first user created:
Copy code
$ id
uid=1000(josuah) gid=1000(josuah)...
Copy code
$ grep :1000: /etc/passwd /etc/group
/etc/passwd:josuah:x:1000:1000:josuah:/home/josuah:/bin/oksh
/etc/group:josuah:x:1000:
Maybe it is a relatively common thing to have for Linux distributions. System groups are typically below 1000.
Copy code
$ id root
uid=0(root) gid=0(root)
This 
id
command is convenient!
m
that the result of the id command
but what is the benefit from this ?
j
I think it was only checking if the user IDs are as expected. 
id
has no action itself
m
yes a different error 🙂
image.png
it cann't read the config.tcl file
but it's right there in the path
$ ls | grep config.tcl
config.tcl
is that error occur in the docker env or in my env ?
j
yes, though a blank Docker setup has en empty filesystem: it has no files at all, and the docker configuration has to set all the paths that should be available on the inside.
what do you have under dependencies/pdks?
This is the paths being mapped to /pdk
m
*dependencies/pdk*$ ls sky130A sky130B volare
j
And do you have that config.tcl under that sky130A directory as listed in the error message above?
m
yes
dependencies/pdk/sky130A/libs.te ch/openlane$ 
ls | grep config.tcl
config.tcl
j
So something might have gone wrong with the path binding...
As it does not even have the sky130A directory
m
who print this warning the docker ?
j
It comes from inside the Docker, but since it cannot even find /pdk/sky130A, then /pdk/sky130A/... would not be there either
could you copy-paste the full log? You can hit ````` in the chat to make a code block
in the full text output from 
make
there will be the command executed, along with which the docker invocation with the configuration visible
m
make user_proj_example
make -C openlane user_proj_example
make[1]: Entering directory '/home/mahdesouqi/forTest/4bit_adder_assignment/4bit_adder_efabless-main/openla
ne'
# user_proj_example
mkdir -p ./user_proj_example/runs/23_06_08_23_07
rm -rf ./user_proj_example/runs/user_proj_example
ln -s $(realpath ./user_proj_example/runs/23_06_08_23_07) ./user_proj_example/runs/user_proj_example
docker run -it -v $(realpath /home/mahdesouqi/forTest/4bit_adder_assignment/4bit_adder_efabless-main/..):$(
realpath /home/mahdesouqi/forTest/4bit_adder_assignment/4bit_adder_efabless-main/..) -v /home/mahdesouqi/fo
rTest/4bit_adder_assignment/4bit_adder_efabless-main/dependencies/pdks:/pdk -v /home/mahdesouqi/forTest/4bi
t_adder_assignment/4bit_adder_efabless-main/caravel:/home/mahdesouqi/forTest/4bit_adder_assignment/4bit_add
er_efabless-main/caravel -v /home/mahdesouqi/forTest/4bit_adder_assignment/4bit_adder_efabless-main/depende
ncies/openlane_src:/openlane -v /home/mahdesouqi/forTest/4bit_adder_assignment/4bit_adder_efabless-main/mgm
t_core_wrapper:/home/mahdesouqi/forTest/4bit_adder_assignment/4bit_adder_efabless-main/mgmt_core_wrapper -e
PDK_ROOT=/pdk -e PDK=sky130A -e MISMATCHES_OK=1 -e CARAVEL_ROOT=/home/mahdesouqi/forTest/4bit_adder_assign
ment/4bit_adder_efabless-main/caravel -e OPENLANE_RUN_TAG=23_06_08_23_07 -e MCW_ROOT=/home/mahdesouqi/forTe
st/4bit_adder_assignment/4bit_adder_efabless-main/mgmt_core_wrapper \
efabless/openlane:2023.02.23 sh -c "flow.tcl -design $(realpath ./user_proj_example) -save_path $(r
ealpath ..) -save -tag 23_06_08_23_07 -overwrite -ignore_mismatches"
OpenLane a35b64aa200c91e9eb7dde56db787d6b4c0ea12a
All rights reserved. (c) 2020-2022 Efabless Corporation and contributors.
Available under the Apache License, version 2.0. See the LICENSE file for more details.
[WARNING]: OpenLane may not function properly: Failed to compare PDKs.
/pdk/sky130A not found.
Traceback (most recent call last):
File "/openlane/dependencies/verify_versions.py", line 76, in verify_versions
raise Exception(f"{pdk_dir} not found.")
Exception: /pdk/sky130A not found.
[INFO]: Using configuration in '../home/mahdesouqi/forTest/4bit_adder_assignment/4bit_adder_efabless-main/o
penlane/user_proj_example/config.json'...
[INFO]: PDK Root: /pdk
[INFO]: Process Design Kit: sky130A
couldn't read file "/pdk/sky130A/libs.tech/openlane/config.tcl": no such file or directory
while executing
"source $pdk_config"
(procedure "prep" line 170)
invoked from within
"prep {*}$args"
(procedure "run_non_interactive_mode" line 12)
invoked from within
"run_non_interactive_mode {*}$argv"
invoked from within
"if { [info exists flags_map(-interactive)] || [info exists flags_map(-it)] } {
if { [info exists arg_values(-file)] } {
run_file [file nor..."
(file "/openlane/flow.tcl" line 408)
make[1]: *** [Makefile:75: user_proj_example] Error 1
make[1]: Leaving directory '/home/mahdesouqi/forTest/4bit_adder_assignment/4bit_adder_efabless-main/openlan
e'
make: *** [Makefile:107: user_proj_example] Error 2
Copy code
make user_proj_example 
make -C openlane user_proj_example
make[1]: Entering directory '/home/mahdesouqi/forTest/4bit_adder_assignment/4bit_adder_efabless-main/openlane'
# user_proj_example
mkdir -p ./user_proj_example/runs/23_06_08_23_07 
rm -rf ./user_proj_example/runs/user_proj_example
ln -s $(realpath ./user_proj_example/runs/23_06_08_23_07) ./user_proj_example/runs/user_proj_example
docker run -it -v $(realpath /home/mahdesouqi/forTest/4bit_adder_assignment/4bit_adder_efabless-main/..):$(realpath /home/mahdesouqi/forTest/4bit_adder_assignment/4bit_adder_efabless-main/..) -v /home/mahdesouqi/forTest/4bit_adder_assignment/4bit_adder_efabless-main/dependencies/pdks:/pdk -v /home/mahdesouqi/forTest/4bit_adder_assignment/4bit_adder_efabless-main/caravel:/home/mahdesouqi/forTest/4bit_adder_assignment/4bit_adder_efabless-main/caravel -v /home/mahdesouqi/forTest/4bit_adder_assignment/4bit_adder_efabless-main/dependencies/openlane_src:/openlane -v /home/mahdesouqi/forTest/4bit_adder_assignment/4bit_adder_efabless-main/mgmt_core_wrapper:/home/mahdesouqi/forTest/4bit_adder_assignment/4bit_adder_efabless-main/mgmt_core_wrapper -e PDK_ROOT=/pdk -e PDK=sky130A -e MISMATCHES_OK=1 -e CARAVEL_ROOT=/home/mahdesouqi/forTest/4bit_adder_assignment/4bit_adder_efabless-main/caravel -e OPENLANE_RUN_TAG=23_06_08_23_07 -e MCW_ROOT=/home/mahdesouqi/forTest/4bit_adder_assignment/4bit_adder_efabless-main/mgmt_core_wrapper  \
        efabless/openlane:2023.02.23 sh -c "flow.tcl -design $(realpath ./user_proj_example) -save_path $(realpath ..) -save -tag 23_06_08_23_07 -overwrite -ignore_mismatches"
OpenLane a35b64aa200c91e9eb7dde56db787d6b4c0ea12a
All rights reserved. (c) 2020-2022 Efabless Corporation and contributors.
Available under the Apache License, version 2.0. See the LICENSE file for more details.

[WARNING]: OpenLane may not function properly: Failed to compare PDKs.
/pdk/sky130A not found.
Traceback (most recent call last):
  File "/openlane/dependencies/verify_versions.py", line 76, in verify_versions
    raise Exception(f"{pdk_dir} not found.")
Exception: /pdk/sky130A not found.

[INFO]: Using configuration in '../home/mahdesouqi/forTest/4bit_adder_assignment/4bit_adder_efabless-main/openlane/user_proj_example/config.json'...
[INFO]: PDK Root: /pdk
[INFO]: Process Design Kit: sky130A
couldn't read file "/pdk/sky130A/libs.tech/openlane/config.tcl": no such file or directory
    while executing
"source $pdk_config"
    (procedure "prep" line 170)
    invoked from within
"prep {*}$args"
    (procedure "run_non_interactive_mode" line 12)
    invoked from within
"run_non_interactive_mode {*}$argv"
    invoked from within
"if { [info exists flags_map(-interactive)] || [info exists flags_map(-it)] } {
    if { [info exists arg_values(-file)] } {
        run_file [file nor..."
    (file "/openlane/flow.tcl" line 408)
make[1]: *** [Makefile:75: user_proj_example] Error 1
make[1]: Leaving directory '/home/mahdesouqi/forTest/4bit_adder_assignment/4bit_adder_efabless-main/openlane'
make: *** [Makefile:107: user_proj_example] Error 2
thanks for the trick ^_^
j
Could you try this? 
ls /home/mahdesouqi/forTest/4bit_adder_assignment/4bit_adder_efabless-main/dependencies/pdks
with the full path, just to make sure.
m
empty
j
Ah, here we go!
you would need the PDK installed here
That "dependencies/pdk/..." path you had was not the same as the one above it seems.
Copy code
dependencies/pdk/sky130A/libs.tech/openlane$
m
I'm sorry but I don't get what you said
j
You listed a pdk/sky130A earlier, it had the PDK installed in it, so it looked surprising that openlane could not find the config file. Here you did try to list the actual path openlane was using: 
/home/mahdesouqi/forTest/4bit_adder_assignment/4bit_adder_efabless-main/dependencies/pdks
and this directory is empty
so you would need to check in the install process that you are picking the right path for installing the dependencies including the PDK data.
m
the problem is that I can't check bec. it done automatically when apply the 
make setup
command
can I just copy the PDK to the path (/home/mahdesouqi/forTest/4bit_adder_assignment/4bit_adder_efabless-main/dependencies/pdks)
j
m
that's right
j
so maybe through the install process, there is a moment where the PDK is installed from the definition of those variable.
m
I think I can change the PDK_ROOT env var
it value is 
/home/mahdesouqi/forTest/4bit_adder_assignment/4bit_adder_efabless-main/dependencies/pdks
should i make it 
/home/mahdesouqi/forTest/4bit_adder_assignment/4bit_adder_efabless-main/dependencies/pdk
?
it's work
it's work
👍 1
guys I can't say thanks enough for you effort
j
\o/ great news!
that was a long road, but it has some light at the end.
m
yes it's and you're the source of light , really thanks sir
j
I'm standing on the shoulders of giants 🙂
❤️ 1
8 Views
Open in Slack
PreviousNext
Powered by