Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[inetstack] Enhancement: Remove circular dependencies #1464

Open
wants to merge 3 commits into
base: dev
Choose a base branch
from
Open

[inetstack] Enhancement: Remove circular dependencies #1464

wants to merge 3 commits into from
+673 −856

Conversation

iyzhang
Copy link
Contributor

@iyzhang iyzhang commented Nov 21, 2024

This PR removes circular dependencies between the sender, receiver and control block in established TCP sockets. We move all state into the control block structure, per the TCP recommendation, and to make it easier to move the state later. However, we pass the control block to static functions in the receiver and sender for receive and send path processing.

@iyzhang iyzhang self-assigned this Nov 21, 2024
@github-actions GitHub Actions
Copy link

libos = catpowder
commit id = 7cc5ed9

collapsed_stack num_calls cycles_per_call nanoseconds_per_call percent_time
bgc::inetstack::poll_recv;inetstack::layer4_endpoint::poll_once 18841023.90 699.38 301.63 85.64
bgc::inetstack::poll_recv 9420511.95 1639.86 707.23 53.83
bgc::inetstack::tcp::established::background 11053.30 1641.71 706.72 0.20
bgc::inetstack::tcp::established::background;tcp::established::background::acknowledger 11053.30 478.35 207.87 23.25
bgc::inetstack::tcp::established::background;tcp::established::background::retransmitter 11044.96 389.22 167.46 29.59
bgc::inetstack::tcp::established::background;tcp::established::background::sender 11003.02 916.30 389.21 24.17
ioc::network::libos::pop 9030.55 511.95 221.06 0.06
demikernel::sgafree 8772.60 105.07 45.92 0.02
ioc::network::libos::push;inetstack::push 8762.80 6108.75 2652.35 2.31
ioc::network::libos::push 8073.74 33495552.39 14966753.98 40.42
demikernel::push 8073.74 643.44 277.73 0.06
demikernel::sgaalloc 6556.29 171.14 73.84 0.03
demikernel::pop 6035.06 585.14 253.56 0.04
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::for 5278.36 2501.45 1078.86 1.35
ioc::network::libos::pushto 1104.00 4264.80 1902.02 4.27
ioc::network::libos::pushto;inetstack::push 1104.00 3855.64 1719.33 90.41
demikernel::pushto 1104.00 461.76 206.26 0.46
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::for;udp::receive 201.33 199.07 91.02 62.39
ioc::network::libos::close 47.48 6626.57 2850.69 0.11
ioc::network::libos::connect 41.86 40338.53 17467.12 0.51
bgc::inetstack::tcp::passiveopen::background 41.86 31718.20 13893.89 0.01
ioc::network::libos::accept 38.07 2026.57 885.05 0.00
demikernel::async_close 32.50 677.99 285.97 0.01
demikernel::accept 21.86 1214.71 530.46 0.00
demikernel::connect 20.93 4155.00 1795.31 0.02
demikernel::socket 10.30 1907.87 830.49 0.00
bgc::inetstack::arp::background 6.27 2102.74 909.12 0.00
bgc::inetstack::poll_recv;inetstack::poll;inetstack::layer4_endpoint::poll_once 2.00 6218.58 2685.93 83.88
bgc::inetstack::icmp::background 1.03 3200.22 1379.90 0.00
demikernel::new 1.00 92702843.55 40065043.19 19.62
demikernel::bind 1.00 28271.25 12175.90 0.00
bgc::inetstack::poll_recv;inetstack::poll 1.00 14763.77 6374.86 0.04
demikernel::listen 1.00 3584.43 1547.80 0.00
demikernel::close 0.00 nan nan 0.00

@github-actions GitHub Actions
Copy link

libos = catnip
commit id = 7cc5ed9

collapsed_stack num_calls cycles_per_call nanoseconds_per_call percent_time
bgc::inetstack::poll_recv;inetstack::layer4_endpoint::poll_once 29405758.00 337.16 145.62 75.58
bgc::inetstack::poll_recv;inetstack::layer4_endpoint::poll_once;catnip::runtime::receive 29405758.00 197.60 85.34 58.73
bgc::inetstack::poll_recv 14702879.00 897.48 388.06 35.51
bgc::inetstack::tcp::established::background 11094.96 1106.99 469.94 0.11
bgc::inetstack::tcp::established::background;tcp::established::background::acknowledger 11094.96 331.30 139.73 26.95
bgc::inetstack::tcp::established::background;tcp::established::background::retransmitter 11083.84 349.37 150.57 32.40
bgc::inetstack::tcp::established::background;tcp::established::background::sender 11041.68 298.92 121.40 14.47
ioc::network::libos::pop 9062.67 366.81 157.47 0.02
demikernel::sgafree 8777.36 85.84 36.96 0.01
ioc::network::libos::push;inetstack::push 8762.80 2041.68 908.54 0.69
ioc::network::libos::push;inetstack::push;catnip::runtime::transmit 8762.80 332.16 146.90 15.46
ioc::network::libos::push 8073.74 33413461.40 13539617.88 25.94
demikernel::push 8073.74 772.12 335.86 0.04
demikernel::sgaalloc 6532.33 158.40 67.45 0.01
demikernel::pop 6051.24 544.06 232.95 0.02
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::for 4585.29 1925.97 796.57 2.07
ioc::network::libos::pushto 1188.00 1311.38 590.51 0.05
ioc::network::libos::pushto;inetstack::push 1188.00 702.40 317.04 53.79
demikernel::pushto 1188.00 506.80 229.46 0.02
ioc::network::libos::pushto;inetstack::push;catnip::runtime::transmit 1188.00 253.08 113.88 35.93
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::for;udp::receive 544.00 359.94 163.45 77.10
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::for;catnip::runtime::transmit 60.14 613.71 267.00 3.40
ioc::network::libos::close 49.47 5930.76 2558.98 0.01
bgc::inetstack::tcp::passiveopen::background 41.86 36063.27 16252.73 0.01
ioc::network::libos::connect 41.86 30443.18 13171.62 0.04
ioc::network::libos::connect;catnip::runtime::transmit 41.86 6335.06 2734.09 22.31
ioc::network::libos::accept 38.21 2041.76 903.83 0.00
demikernel::async_close 32.50 585.72 235.87 0.00
demikernel::accept 21.86 1424.16 625.84 0.00
bgc::inetstack::tcp::passiveopen::background;catnip::runtime::transmit 20.93 17260.28 7864.07 17.63
demikernel::connect 20.93 2737.40 1187.94 0.00
bgc::inetstack::tcp::established::background;tcp::established::background::sender;catnip::runtime::transmit 20.93 579.55 250.98 11.57
bgc::inetstack::tcp::established::background;tcp::established::background::acknowledger;catnip::runtime::transmit 13.00 1403.65 562.72 18.09
demikernel::socket 10.30 1733.37 751.48 0.00
bgc::inetstack::arp::background 6.47 2311.82 998.26 0.00
bgc::inetstack::poll_recv;inetstack::poll;inetstack::layer4_endpoint::poll_once 2.00 20622.90 8901.54 95.37
bgc::inetstack::poll_recv;inetstack::poll;inetstack::layer4_endpoint::poll_once;catnip::runtime::receive 2.00 18479.80 7976.93 89.57
bgc::inetstack::tcp::established::background;tcp::established::background::retransmitter;catnip::runtime::transmit 1.75 817.75 351.31 0.50
bgc::inetstack::icmp::background 1.05 3244.87 1407.47 0.00
demikernel::new 1.00 1758666898.68 758985943.78 47.85
bgc::inetstack::poll_recv;inetstack::poll 1.00 43234.83 18661.07 0.18
demikernel::bind 1.00 11005.50 4742.23 0.00
demikernel::listen 1.00 3297.93 1407.95 0.00
demikernel::close 0.00 nan nan 0.00

@github-actions GitHub Actions
Copy link

libos = catnap
commit id = 7cc5ed9

collapsed_stack num_calls cycles_per_call nanoseconds_per_call percent_time
bgc::catnap::transport::epoll 13588311.38 2126.05 939.20 33.28
demikernel::sgafree 11569.98 71.39 30.71 0.05
demikernel::sgaalloc 11358.92 119.17 52.20 0.11
ioc::network::libos::push;catnap::linux::transport::push 9066.12 648.77 283.49 3.98
ioc::network::libos::push 8800.49 17032.11 7462.24 7.18
demikernel::push 8800.49 556.93 242.91 0.17
ioc::network::libos::pop;catnap::linux::transport::pop 8031.19 416.82 179.64 4.87
ioc::network::libos::pop 7822.52 6116812.80 2316087.24 25.27
demikernel::pop 7822.52 578.69 249.02 0.13
ioc::network::libos::pushto 1929.67 7784.82 3037.75 23.78
demikernel::pushto 1929.67 438.63 172.27 1.48
ioc::network::libos::pushto;catnap::linux::transport::push 1929.67 420.37 165.40 5.55
demikernel::async_close 32.50 922.75 377.84 0.02
ioc::network::libos::connect 24.95 4928274.30 2206781.13 23.71
ioc::network::libos::connect;catnap::linux::transport::connect 24.95 75652.63 33202.97 3.09
demikernel::connect 24.95 2042.36 905.04 0.01
ioc::network::libos::close 18.35 24191.97 10430.25 0.07
ioc::network::libos::close;catnap::linux::transport::close 18.35 22895.58 9871.45 94.70
ioc::network::libos::accept 15.42 913942841.35 390821600.32 44.92
demikernel::accept 15.42 1523.25 654.66 0.00
ioc::network::libos::accept;catnap::linux::transport::accept 15.39 7681.11 3278.16 0.00
demikernel::socket 9.50 157093.85 67349.64 0.28
demikernel::socket;catnap::linux::transport::socket 9.50 154838.61 66379.23 96.57
demikernel::new 1.00 265083.15 114378.55 0.12
demikernel::bind 1.00 29105.58 12415.40 0.01
demikernel::bind;catnap::linux::transport::bind 1.00 24596.95 10496.57 84.40
demikernel::listen 1.00 15496.97 6660.58 0.00
demikernel::listen;catnap::linux::transport::listen 1.00 14343.97 6165.90 92.51
demikernel::close 0.00 nan nan 0.00

@github-actions GitHub Actions
Copy link

libos = catnap
commit id = bf14926

collapsed_stack num_calls cycles_per_call nanoseconds_per_call percent_time
bgc::catnap::transport::epoll 13360750.51 2015.43 885.61 32.75
demikernel::sgafree 11555.42 76.99 32.79 0.05
demikernel::sgaalloc 11318.40 121.40 52.34 0.10
ioc::network::libos::push;catnap::linux::transport::push 9066.12 630.65 276.34 3.95
ioc::network::libos::push 8800.49 16693.33 7328.42 6.48
demikernel::push 8800.49 548.41 239.21 0.15
ioc::network::libos::pop;catnap::linux::transport::pop 8031.05 420.60 181.13 4.51
ioc::network::libos::pop 7822.39 6390593.37 2615608.17 25.77
demikernel::pop 7822.39 557.73 239.83 0.11
ioc::network::libos::pushto 1361.67 7341.66 3159.36 23.86
demikernel::pushto 1361.67 468.60 201.58 1.53
ioc::network::libos::pushto;catnap::linux::transport::push 1361.67 420.96 181.49 5.77
demikernel::async_close 32.50 870.34 346.02 0.01
ioc::network::libos::connect 24.95 4547729.36 1977278.40 25.54
ioc::network::libos::connect;catnap::linux::transport::connect 24.95 80135.07 34488.92 2.97
demikernel::connect 24.95 2045.10 880.72 0.01
ioc::network::libos::close 18.35 25840.91 11101.85 0.06
ioc::network::libos::close;catnap::linux::transport::close 18.35 24379.36 10472.63 94.80
demikernel::accept 15.42 1561.46 680.45 0.00
ioc::network::libos::accept 15.39 890231259.97 384216592.72 45.08
ioc::network::libos::accept;catnap::linux::transport::accept 15.39 4253.04 1862.31 0.00
demikernel::socket 9.50 156323.51 67279.73 0.26
demikernel::socket;catnap::linux::transport::socket 9.50 154047.89 66294.46 96.59
demikernel::new 1.00 262063.74 113229.24 0.08
demikernel::bind 1.00 29686.98 12872.12 0.00
demikernel::bind;catnap::linux::transport::bind 1.00 24550.92 10667.69 83.64
demikernel::listen 1.00 14893.24 6463.47 0.00
demikernel::listen;catnap::linux::transport::listen 1.00 13878.21 6023.54 93.06
demikernel::close 0.00 nan nan 0.00

@github-actions GitHub Actions
Copy link

libos = catpowder
commit id = bf14926

collapsed_stack num_calls cycles_per_call nanoseconds_per_call percent_time
bgc::inetstack::poll_recv;inetstack::layer4_endpoint::poll_once 19039665.93 699.62 308.42 85.90
bgc::inetstack::poll_recv 9519832.97 1633.08 719.74 53.47
bgc::inetstack::tcp::established::background 11053.52 1630.49 701.05 0.20
bgc::inetstack::tcp::established::background;tcp::established::background::acknowledger 11053.52 477.70 203.08 23.21
bgc::inetstack::tcp::established::background;tcp::established::background::retransmitter 11045.38 391.85 170.72 29.54
bgc::inetstack::tcp::established::background;tcp::established::background::sender 11003.45 882.31 375.14 23.88
ioc::network::libos::pop 9030.41 427.50 189.01 0.04
ioc::network::libos::push;inetstack::push 8762.80 6114.64 2722.97 2.34
demikernel::sgafree 8748.00 101.29 50.18 0.01
ioc::network::libos::push 8073.74 33474487.66 14635938.33 39.67
demikernel::push 8073.74 583.25 258.11 0.06
demikernel::sgaalloc 6502.58 140.78 62.59 0.02
demikernel::pop 6034.96 576.08 253.54 0.04
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::for 5277.75 2381.64 1024.55 1.25
ioc::network::libos::pushto 669.33 4463.45 1974.79 3.00
ioc::network::libos::pushto;inetstack::push 669.33 4040.98 1787.88 90.53
demikernel::pushto 669.33 502.73 222.41 0.33
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::for;udp::receive 190.00 194.11 100.46 61.46
ioc::network::libos::close 47.33 6421.92 2860.40 0.11
ioc::network::libos::connect 41.86 42063.85 18278.24 0.56
bgc::inetstack::tcp::passiveopen::background 41.86 32137.08 14457.13 0.01
ioc::network::libos::accept 38.04 2091.44 943.28 0.00
demikernel::async_close 32.50 780.95 331.63 0.01
demikernel::accept 21.86 1235.44 549.83 0.00
demikernel::connect 20.93 7346.55 3245.31 0.03
demikernel::socket 10.30 2344.32 1029.11 0.00
bgc::inetstack::arp::background 6.38 1972.20 869.41 0.00
bgc::inetstack::poll_recv;inetstack::poll;inetstack::layer4_endpoint::poll_once 2.00 6799.87 3019.46 84.55
bgc::inetstack::icmp::background 1.03 3098.90 1370.58 0.00
demikernel::new 1.00 94405915.40 41788798.25 20.52
demikernel::bind 1.00 26875.06 11821.40 0.00
bgc::inetstack::poll_recv;inetstack::poll 1.00 16021.80 7107.01 0.07
demikernel::listen 1.00 3853.86 1695.08 0.00
demikernel::close 0.00 nan nan 0.00

@github-actions GitHub Actions
Copy link

libos = catnip
commit id = bf14926

collapsed_stack num_calls cycles_per_call nanoseconds_per_call percent_time
bgc::inetstack::poll_recv;inetstack::layer4_endpoint::poll_once 29440153.83 339.23 151.09 76.07
bgc::inetstack::poll_recv;inetstack::layer4_endpoint::poll_once;catnip::runtime::receive 29440153.83 197.02 87.69 58.22
bgc::inetstack::poll_recv 14720076.92 897.25 399.81 35.73
bgc::inetstack::tcp::established::background 11133.88 1094.33 479.11 0.11
bgc::inetstack::tcp::established::background;tcp::established::background::acknowledger 11133.88 330.02 142.90 26.75
bgc::inetstack::tcp::established::background;tcp::established::background::retransmitter 11123.52 342.65 153.31 32.60
bgc::inetstack::tcp::established::background;tcp::established::background::sender 11081.36 293.18 119.58 14.59
ioc::network::libos::pop 9082.08 352.58 156.34 0.02
demikernel::sgafree 8775.87 79.57 35.76 0.01
ioc::network::libos::push;inetstack::push 8762.74 1982.39 934.85 0.66
ioc::network::libos::push;inetstack::push;catnip::runtime::transmit 8762.74 326.82 154.04 15.79
ioc::network::libos::push 8073.71 33393466.66 15601165.73 25.76
demikernel::push 8073.71 709.13 327.74 0.04
demikernel::sgaalloc 6530.71 148.35 67.30 0.01
demikernel::pop 6050.16 560.01 247.56 0.03
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::for 4621.80 1734.85 739.46 2.02
ioc::network::libos::pushto 1162.00 1330.06 598.65 0.05
ioc::network::libos::pushto;inetstack::push 1162.00 709.79 319.37 53.59
demikernel::pushto 1162.00 477.65 214.89 0.02
ioc::network::libos::pushto;inetstack::push;catnip::runtime::transmit 1162.00 261.82 117.77 36.84
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::for;udp::receive 482.33 247.92 111.90 70.18
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::for;catnip::runtime::transmit 99.93 648.44 293.23 3.62
ioc::network::libos::close 49.48 5667.35 2538.53 0.01
bgc::inetstack::tcp::passiveopen::background 41.86 35055.32 16185.50 0.01
ioc::network::libos::connect 41.86 29362.17 13365.13 0.04
ioc::network::libos::connect;catnip::runtime::transmit 41.86 6451.00 3019.72 22.72
ioc::network::libos::accept 37.89 2143.88 976.91 0.00
demikernel::async_close 32.50 569.21 225.89 0.00
demikernel::accept 21.86 1373.88 634.70 0.00
bgc::inetstack::tcp::passiveopen::background;catnip::runtime::transmit 20.93 17725.30 8317.08 17.86
demikernel::connect 20.93 2567.71 1192.47 0.00
bgc::inetstack::tcp::established::background;tcp::established::background::sender;catnip::runtime::transmit 20.93 631.15 286.72 12.17
demikernel::socket 10.30 1630.81 736.50 0.00
bgc::inetstack::arp::background 6.47 2277.01 1010.86 0.00
bgc::inetstack::tcp::established::background;tcp::established::background::acknowledger;catnip::runtime::transmit 4.33 2377.15 961.38 15.76
bgc::inetstack::poll_recv;inetstack::poll;inetstack::layer4_endpoint::poll_once 2.00 21473.80 9552.47 95.83
bgc::inetstack::poll_recv;inetstack::poll;inetstack::layer4_endpoint::poll_once;catnip::runtime::receive 2.00 19236.35 8560.06 89.35
bgc::inetstack::icmp::background 1.05 3033.83 1345.53 0.00
demikernel::new 1.00 1764360544.37 785260785.53 47.75
bgc::inetstack::poll_recv;inetstack::poll 1.00 44775.60 19918.29 0.20
demikernel::bind 1.00 10949.19 4853.15 0.00
demikernel::listen 1.00 3384.07 1494.63 0.00
bgc::inetstack::tcp::established::background;tcp::established::background::retransmitter;catnip::runtime::transmit 1.00 449.00 186.72 0.68
demikernel::close 0.00 nan nan 0.00

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@anandbonde anandbonde Awaiting requested review from anandbonde

At least 1 approving review is required to merge this pull request.

Assignees

@iyzhang iyzhang

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@iyzhang