Need help debugging cse549 branch

[tajayi-hx]

Hello,

I am running into issues running into a crash with hammerbench memcpy app after updating to the cse549 branch from the spring2023 branch. The only change has been to pull new branches (tool versions, environment, etc are all the same).

Here is a snippet from the end of profile.log

[BSG_INFO][VCORE_PROFILER] t=494666172 x,y=22,12 printing stats.
[BSG_INFO][VCORE_PROFILER] t=494746092 x,y=19,12 printing stats.
[BSG_INFO][VCORE_PROFILER] t=494869302 x,y=22,13 printing stats.
[BSG_INFO][VCORE_PROFILER] t=495015822 x,y=22,11 printing stats.
[BSG_INFO][VCORE_PROFILER] t=495460710 x,y=23,12 printing stats.
[BSG_INFO][VCORE_PROFILER] t=495890946 x,y=28,11 printing stats.
[BSG_INFO][VCORE_PROFILER] t=496611558 x,y=23,13 printing stats.
[BSG_INFO][VCORE_PROFILER] t=496617552 x,y=26,13 printing stats.
[BSG_INFO][VCORE_PROFILER] t=497421414 x,y=25,11 printing stats.
[BSG_INFO][VCORE_PROFILER] t=497861640 x,y=25,12 printing stats.
[BSG_INFO][VCORE_PROFILER] t=497947554 x,y=24,11 printing stats.
[BSG_INFO][VCORE_PROFILER] t=498095406 x,y=20,13 printing stats.

An unexpected termination has occurred in /sim/tutu/hammertime/cse549/bsg_bladerunner/bsg_replicant/machines/pod_X1Y1_ruche_X16Y8_hbm_one_pseudo_channel/bigblade-vcs/profile/simvHostname sim02

During DPI-C function call Function="cosim_main" from /sim/tutu/hammertime/cse549/bsg_bladerunner/bsg_replicant/libraries/platforms/common/dpi/hardware/dpi_top.sv, 382

Command line: /sim/tutu/hammertime/cse549/bsg_bladerunner/bsg_replicant/machines/pod_X1Y1_ruche_X16Y8_hbm_one_pseudo_channel/bigblade-vcs/profile/simv +ntb_random_seed_automatic -reportstats "+c_args=main.riscv memcpy" +c_path=/sim/tutu/hammertime/cse549/bsg_bladerunner/bsg_replicant/examples/hb_hammerbench/apps/memcpy/tile-x_16__tile-y_8__buffer-size_524288__warm-cache_no/main.so
INFO:    .
--- Stack trace follows:

Dumping VCS Annotated Stack:
#0  0x00002aaab4f4060c in waitpid () from /lib64/libc.so.6
#1  0x00002aaab4ebdf62 in do_system () from /lib64/libc.so.6
#2  0x00002aaaac57c930 in SNPSle_10ee25eff68cd8461c9146fa1d0b35e87067f3c8015b313e639d2928478c79b3f673f99203bcf8be64600612100082236bffb2007f1e0ef9 () from /sim/tutu/tools/vcs/U-2023.03-SP2-8/linux64/lib/liberrorinf.so
#3  0x00002aaaac57e48c in SNPSle_10ee25eff68cd8461c9146fa1d0b35e87067f3c8015b313efba706aab251478fa49e66610e453774633a6c152e7ef778f93045171fb3645d () from /sim/tutu/tools/vcs/U-2023.03-SP2-8/linux64/lib/liberrorinf.so
#4  0x00002aaaac576533 in SNPSle_d35ca1ff70d465c2b9b1a72eee90a50630165806651fae96c1bbda5b5d02066c () from /sim/tutu/tools/vcs/U-2023.03-SP2-8/linux64/lib/liberrorinf.so
#5  0x00002aaaaee34cae in SNPSle_35e7b9a27865b52ac940acb351a35d73c891237c4c361ccf () from /sim/tutu/tools/vcs/U-2023.03-SP2-8/linux64/lib/libvcsnew.so
#6  0x00002aaaaec74b0a in SNPSle_067230e04427ed652dd772e98dbb69cfcc4898a4a5107270478b96f3846f24ed () from /sim/tutu/tools/vcs/U-2023.03-SP2-8/linux64/lib/libvcsnew.so
#7  0x00002aaaae8cf4e3 in SNPSle_82244d58c54c18c70d63edc9becab634 () from /sim/tutu/tools/vcs/U-2023.03-SP2-8/linux64/lib/libvcsnew.so
#9  0x00002aaab4fd207d in __memcpy_ssse3_back () from /lib64/libc.so.6
#10 0x00002aaaaaf1a981 in hb_mc_dma_read(hb_mc_manycore*, hb_mc_npa_t const*, void*, unsigned long) () from /sim/tutu/hammertime/cse549/bsg_bladerunner/bsg_replicant/libraries/platforms/bigblade-vcs/libbsg_manycore_runtime.so.1
#11 0x00002aaaaaefcdad in hb_mc_manycore_dma_read_no_cache_afl (mc=0x1055010, npa=0x2aab008e9b94, data=0x2aab00bb3ff0, sz=64) at /sim/tutu/hammertime/cse549/bsg_bladerunner/bsg_replicant/libraries/bsg_manycore.cpp:1504
#12 0x00002aaaaaf0d262 in hb_mc_manycore_eva_read_internal<int (*)(hb_mc_manycore*, hb_mc_npa_t const*, void*, unsigned long)> (mc=0x1055010, map=0x2aaaab139e80 <default_map>, tgt=0x1088348, eva=0x2aab00ce9cb0, data=0x2aab00ae9cb0, sz=1268928, read_function=0x2aaaaaefcd22 <hb_mc_manycore_dma_read_no_cache_afl(hb_mc_manycore_t*, hb_mc_npa_t const*, void*, size_t)>) at /sim/tutu/hammertime/cse549/bsg_bladerunner/bsg_replicant/libraries/bsg_manycore_eva.cpp:1403
#13 0x00002aaaaaf0ce82 in hb_mc_manycore_eva_read_dma (mc=0x1055010, map=0x2aaaab139e80 <default_map>, tgt=0x1088348, eva=0x2aab00ce9cb0, data=0x2aab00ae9cb0, sz=2097152) at /sim/tutu/hammertime/cse549/bsg_bladerunner/bsg_replicant/libraries/bsg_manycore_eva.cpp:1434
#14 0x00002aaaaaf0768c in hb_mc_device_pod_dma_to_host (device=0x2aab00ce9d20, pod_id=0, jobs=0x2aab00ce9cb0, count=1) at /sim/tutu/hammertime/cse549/bsg_bladerunner/bsg_replicant/libraries/bsg_manycore_cuda.cpp:2172
#15 0x00002aaaaaf07742 in hb_mc_device_dma_to_host (device=0x2aab00ce9d20, jobs=0x2aab00ce9cb0, count=1) at /sim/tutu/hammertime/cse549/bsg_bladerunner/bsg_replicant/libraries/bsg_manycore_cuda.cpp:2213
#16 0x00002aab00cef7c5 in kernel_memcpy (argc=3, argv=0x2aab00ce9e10) at /sim/tutu/hammertime/cse549/bsg_bladerunner/bsg_replicant/examples/hb_hammerbench/apps/memcpy/main.c:119
#17 0x00002aab00cef996 in vcs_main (argc=3, argv=0x2aab00ce9e10) at /sim/tutu/hammertime/cse549/bsg_bladerunner/bsg_replicant/examples/hb_hammerbench/apps/memcpy/main.c:138
#18 0x00002aaaaacd0f0d in cosim_main () from /sim/tutu/hammertime/cse549/bsg_bladerunner/bsg_replicant/libraries/platforms/bigblade-vcs/libbsg_manycore_regression.so.1
#19 0x000000000041fa1f in cosim_main ()
#20 0x00002aaaab88b303 in T_VCSgd_q6vHr_2_0 () from /sim/tutu/hammertime/cse549/bsg_bladerunner/bsg_replicant/machines/pod_X1Y1_ruche_X16Y8_hbm_one_pseudo_channel/bigblade-vcs/profile/simv.daidir/_prev_archive_1.so
#21 0x00002aaab0dada2f in SNPSle_8019c4c21dc45cf13790e71a253a4d8d () from /sim/tutu/tools/vcs/U-2023.03-SP2-8/linux64/lib/libvcsnew.so
#22 0x00002aaabe79e018 in ?? ()
#23 0x00002aaabfae2018 in ?? ()
#24 0x00002aaab66ab671 in ?? ()
#25 0x00002aaab672d2ac in ?? ()
#26 0x0000000000000000 in ?? ()

Process VmPeak: 1425792 kb, VmSize: 1425784 kb
System Free Memory: 52337884 kb, System Free Swap: 20970492 kb

Completed context dump phase data location*

Error-[FENCE-INVALID-READ-WRITE-LOW] Invalid Read Write
  Out of Bounds Read/Write detected at above stack trace in DPI domain at DPI 
  String location.
  Address 0x2aab00bb4020 is 4064 byte  before 4096 bytes allocated  from 
  0x2aab00bb6000 to 0x2aab00bb5000.
  Please check for -ve array indexing. 
  Please run valgrind with option --trace-children=yes.

Fatal: "/sim/tutu/hammertime/cse549/bsg_bladerunner/basejump_stl/bsg_test/bsg_nonsynth_dpi_gpio.sv", 64: replicant_tb_top.trace_control: at time 550636480 ps
BSG ERROR (replicant_tb_top.trace_control): final block executed before fini() was called
Fatal: "/sim/tutu/hammertime/cse549/bsg_bladerunner/basejump_stl/bsg_test/bsg_nonsynth_dpi_from_fifo.sv", 88: replicant_tb_top.mc_dpi.f2d_req_i: at time 550636480 ps
BSG ERROR (replicant_tb_top.mc_dpi.f2d_req_i): fini() was not called before $finish
Fatal: "/sim/tutu/hammertime/cse549/bsg_bladerunner/basejump_stl/bsg_test/bsg_nonsynth_dpi_from_fifo.sv", 88: replicant_tb_top.mc_dpi.f2d_rsp_i: at time 550636480 ps
BSG ERROR (replicant_tb_top.mc_dpi.f2d_rsp_i): fini() was not called before $finish
Fatal: "/sim/tutu/hammertime/cse549/bsg_bladerunner/basejump_stl/bsg_test/bsg_nonsynth_dpi_to_fifo.sv", 95: replicant_tb_top.mc_dpi.d2f_req_i: at time 550636480 ps
BSG ERROR (replicant_tb_top.mc_dpi.d2f_req_i): fini() was not called before $finish
Fatal: "/sim/tutu/hammertime/cse549/bsg_bladerunner/basejump_stl/bsg_test/bsg_nonsynth_dpi_to_fifo.sv", 95: replicant_tb_top.mc_dpi.d2f_rsp_i: at time 550636480 ps
BSG ERROR (replicant_tb_top.mc_dpi.d2f_rsp_i): fini() was not called before $finish
Fatal: "/sim/tutu/hammertime/cse549/bsg_bladerunner/basejump_stl/bsg_test/bsg_nonsynth_dpi_rom.sv", 66: replicant_tb_top.mc_dpi.rom: at time 550636480 ps
BSG ERROR (replicant_tb_top.mc_dpi.rom): final block executed before fini() was called
           V C S   S i m u l a t i o n   R e p o r t 
Time: 550636480 ps

Compilation Performance Summary
===============================
VCS started at        :  Fri Sep 13 10:27:47 2024
Elapsed time          :  41 sec
CPU Time              :  31.3 sec
Virtual memory size   :  455.3 MB
Resident set size     :  229.5 MB
Shared memory size    :  2.3 MB
Private memory size   :  227.2 MB
Major page faults     :  0
Machine name          :  sim02
===============================


Simulation Performance Summary
==============================
Simulation started at :  Fri Sep 13 10:28:28 2024
Elapsed Time          :  1870 sec
CPU Time              :  1814.9 sec
Virtual memory size   :  1392.4 MB
Resident set size     :  1181.6 MB
Shared memory size    :  2.8 MB
Private memory size   :  1178.8 MB
Major page faults     :  9
Machine name          :  sim02
==============================

Fri Sep 13 10:59:37 2024
make[1]: Leaving directory '/sim/tutu/hammertime/cse549/bsg_bladerunner/bsg_replicant/examples/hb_hammerbench/apps/memcpy/tile-x_16__tile-y_8__buffer-size_524288__warm-cache_no'

Any pointers would be appreciated. Also happy to provide more info for debugging