Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........X.....X.....................X...............X.X..X] duration: 30.002452852s FAIL (2025-09-07 12:49:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:48:58.116001438 (23h20m34.155574548s) [upload=20575]: Starting upload Sep 7 12:48:58.116072374 (23h20m34.155645494s) [upload=20575]: Done starting upload Sep 7 12:48:58.116197363 (23h20m34.155770483s) [upload=20575]: Uploading segments... Sep 7 12:48:58.116277867 (23h20m34.155850977s) [upload=20575]: Got next segment. Inline: false Sep 7 12:48:58.116311772 (23h20m34.155884892s) [upload=20575]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:48:58.116316230 (23h20m34.15588935s) [upload=20575]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:48:58.380021338 (23h20m34.419594459s) [seg_pos={0 0}, upload=20575]: Begin upload segment... Sep 7 12:48:58.380063899 (23h20m34.419637009s) [seg_pos={0 0}, upload=20575]: Done begin upload segment. Sep 7 12:48:58.380109367 (23h20m34.419682487s) [noise=true , address=45.91.116.12:4225 , piece_id=JF4SUSZ2DJ5AEVWFJ7B4ULL35GLALXBXKFM3I6OKQ6NTNFOU6STA, seg_pos={0 0}, upload=20575]: Uploading piece... Sep 7 12:48:58.380478654 (23h20m34.420051774s) [noise=true , address=45.91.116.6:4973 , piece_id=UBD42NHDYXDP6PP6Y4YXGCVOZ4NJ7HW3QSCQAR4TS5LG72X4CI7A, seg_pos={0 0}, upload=20575]: Uploading piece... Sep 7 12:48:58.380748190 (23h20m34.42032131s) [noise=true , address=45.91.116.4:4940 , piece_id=ZSG7SYDNSDTDO76N3JA7ZOWQJ5PK6OUAV7OIGRB5W4FFJJRTQ5KQ, seg_pos={0 0}, upload=20575]: Uploading piece... Sep 7 12:48:58.380940768 (23h20m34.420513888s) [noise=true , address=77.90.31.13:4628 , piece_id=4FUP7ZW735JXXHV7U2RKH6SPZTBJIUWPAK62KGZRGLPRGAA52MVA, seg_pos={0 0}, upload=20575]: Uploading piece... Sep 7 12:48:58.381069183 (23h20m34.420642303s) [noise=true , address=45.91.116.7:4011 , piece_id=A37ZUMPN5KXME7BGRYRAZ6BKV3EWWHAPHX5RC2NLEVYDDWZAAOCA, seg_pos={0 0}, upload=20575]: Uploading piece... Sep 7 12:48:58.381251823 (23h20m34.420824943s) [noise=true , address=77.90.31.10:4518 , piece_id=OB74R5OOJCOCEDA74QCFGG362GDZVHT2ZFHHDEVS3BKFZM5XJ5RQ, seg_pos={0 0}, upload=20575]: Uploading piece... Sep 7 12:48:58.381336305 (23h20m34.420909425s) [noise=true , address=45.91.116.6:4974 , piece_id=K5IXFECP4GRWUPH55TT366GHXEC2BN57BAJ2ZVYT5R7TQKZU4H3A, seg_pos={0 0}, upload=20575]: Uploading piece... Sep 7 12:48:58.381574370 (23h20m34.42114748s) [noise=true , address=77.90.31.12:4592 , piece_id=7W72B4EHYFTGZPW6OPPEZ3HYIWRIOIR6JLXUOCUCDI6KNEERUPRQ, seg_pos={0 0}, upload=20575]: Uploading piece... Sep 7 12:48:58.381649795 (23h20m34.421222915s) [noise=true , address=45.91.116.7:4010 , piece_id=G3VOV5DGLGPKOIB3VCEYSVT7HMHO5B4VJDEZYCU3R26HAT6SCJZA, seg_pos={0 0}, upload=20575]: Uploading piece... Sep 7 12:48:58.381945550 (23h20m34.42151866s) [noise=true , address=45.91.116.12:4226 , piece_id=2Q5DQQ6SHHS6VJMYYNIHRDFHD475ZOBZUJDZAJUTT3TQ7KE2BBIA, seg_pos={0 0}, upload=20575]: Uploading piece... Sep 7 12:48:58.382069458 (23h20m34.421642568s) [noise=true , address=77.90.31.10:4519 , piece_id=QSIHMVF7CX5P46AD3B5OGMMHYCRX7J5LPGMCDG5YC3YP4AWCU4KQ, seg_pos={0 0}, upload=20575]: Uploading piece... Sep 7 12:48:58.382178987 (23h20m34.421752107s) [noise=true , address=77.90.31.19:4843 , piece_id=ZAH6WQ4H552X4S4AMVZE27U5YKB7AQ4A7HOALXIAVVYUFHPLNE2Q, seg_pos={0 0}, upload=20575]: Uploading piece... Sep 7 12:48:58.382760600 (23h20m34.42233377s) [noise=true , address=45.91.116.11:4117 , piece_id=IZ2VR5WLWH6UL6AXVLMNFWOEEGBLEKK4P5ULXCZCJG5NQMQVISLQ, seg_pos={0 0}, upload=20575]: Uploading piece... Sep 7 12:48:58.383351381 (23h20m34.422924521s) [noise=true , address=77.90.31.21:4878 , piece_id=DIB5WDIPG2CYKKYMNJCFILRVRJTFRSVPF2C3A7ZUN2DO7MIIKVGQ, seg_pos={0 0}, upload=20575]: Uploading piece... Sep 7 12:48:58.383850506 (23h20m34.423423626s) [noise=true , address=77.90.31.18:4805 , piece_id=ILGVNDJBEJ4CNEJP5ENDE2R5CXK7KXY6F6T2DZQFVYRWVSMCKPFA, seg_pos={0 0}, upload=20575]: Uploading piece... Sep 7 12:48:58.384283344 (23h20m34.423856464s) [noise=true , address=45.91.116.10:4082 , piece_id=HTX4NHS5W4J7NLYHE3T2ORJHIY2IZJP254FHDCPW7K7RI63IKGQQ, seg_pos={0 0}, upload=20575]: Uploading piece... Sep 7 12:49:00.185575988 (23h20m36.225149108s) [upload=20575]: Next returned nil segment Sep 7 12:49:00.185594243 (23h20m36.225167363s) [upload=20575]: Waiting for error group managing segments... Sep 7 12:49:00.511630022 (23h20m36.551203132s) [noise=true , address=45.91.116.11:4117 , piece_id=IZ2VR5WLWH6UL6AXVLMNFWOEEGBLEKK4P5ULXCZCJG5NQMQVISLQ, seg_pos={0 0}, upload=20575]: Done uploading piece. err: <nil> Sep 7 12:49:00.529870615 (23h20m36.569443735s) [noise=true , address=45.91.116.10:4082 , piece_id=HTX4NHS5W4J7NLYHE3T2ORJHIY2IZJP254FHDCPW7K7RI63IKGQQ, seg_pos={0 0}, upload=20575]: Done uploading piece. err: <nil> Sep 7 12:49:00.530677109 (23h20m36.570250229s) [noise=true , address=45.91.116.12:4225 , piece_id=JF4SUSZ2DJ5AEVWFJ7B4ULL35GLALXBXKFM3I6OKQ6NTNFOU6STA, seg_pos={0 0}, upload=20575]: Done uploading piece. err: <nil> Sep 7 12:49:00.533979208 (23h20m36.573552358s) [noise=true , address=45.91.116.6:4974 , piece_id=K5IXFECP4GRWUPH55TT366GHXEC2BN57BAJ2ZVYT5R7TQKZU4H3A, seg_pos={0 0}, upload=20575]: Done uploading piece. err: <nil> Sep 7 12:49:00.534428707 (23h20m36.574001857s) [noise=true , address=77.90.31.18:4805 , piece_id=ILGVNDJBEJ4CNEJP5ENDE2R5CXK7KXY6F6T2DZQFVYRWVSMCKPFA, seg_pos={0 0}, upload=20575]: Done uploading piece. err: <nil> Sep 7 12:49:00.537129455 (23h20m36.576702575s) [noise=true , address=77.90.31.12:4592 , piece_id=7W72B4EHYFTGZPW6OPPEZ3HYIWRIOIR6JLXUOCUCDI6KNEERUPRQ, seg_pos={0 0}, upload=20575]: Done uploading piece. err: <nil> Sep 7 12:49:00.549904380 (23h20m36.5894775s) [noise=true , address=77.90.31.10:4518 , piece_id=OB74R5OOJCOCEDA74QCFGG362GDZVHT2ZFHHDEVS3BKFZM5XJ5RQ, seg_pos={0 0}, upload=20575]: Done uploading piece. err: <nil> Sep 7 12:49:00.564118860 (23h20m36.60369197s) [noise=true , address=45.91.116.7:4010 , piece_id=G3VOV5DGLGPKOIB3VCEYSVT7HMHO5B4VJDEZYCU3R26HAT6SCJZA, seg_pos={0 0}, upload=20575]: Done uploading piece. err: <nil> Sep 7 12:49:00.576059709 (23h20m36.615632829s) [noise=true , address=45.91.116.12:4226 , piece_id=2Q5DQQ6SHHS6VJMYYNIHRDFHD475ZOBZUJDZAJUTT3TQ7KE2BBIA, seg_pos={0 0}, upload=20575]: Done uploading piece. err: <nil> Sep 7 12:49:00.591711369 (23h20m36.631284489s) [noise=true , address=77.90.31.13:4628 , piece_id=4FUP7ZW735JXXHV7U2RKH6SPZTBJIUWPAK62KGZRGLPRGAA52MVA, seg_pos={0 0}, upload=20575]: Done uploading piece. err: <nil> Sep 7 12:49:00.635137125 (23h20m36.674710245s) [noise=true , address=77.90.31.21:4878 , piece_id=DIB5WDIPG2CYKKYMNJCFILRVRJTFRSVPF2C3A7ZUN2DO7MIIKVGQ, seg_pos={0 0}, upload=20575]: Done uploading piece. err: <nil> Sep 7 12:49:00.666662991 (23h20m36.706236111s) [noise=true , address=45.91.116.6:4973 , piece_id=UBD42NHDYXDP6PP6Y4YXGCVOZ4NJ7HW3QSCQAR4TS5LG72X4CI7A, seg_pos={0 0}, upload=20575]: Done uploading piece. err: <nil> Sep 7 12:49:00.723770908 (23h20m36.763344028s) [noise=true , address=45.91.116.7:4011 , piece_id=A37ZUMPN5KXME7BGRYRAZ6BKV3EWWHAPHX5RC2NLEVYDDWZAAOCA, seg_pos={0 0}, upload=20575]: Done uploading piece. err: <nil> Sep 7 12:49:00.723808149 (23h20m36.763381269s) [seg_pos={0 0}, upload=20575]: Segment reached optimal threshold of 13 pieces. Sep 7 12:49:00.723978817 (23h20m36.763551937s) [noise=true , address=45.91.116.4:4940 , piece_id=ZSG7SYDNSDTDO76N3JA7ZOWQJ5PK6OUAV7OIGRB5W4FFJJRTQ5KQ, seg_pos={0 0}, upload=20575]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Sep 7 12:49:00.724102282 (23h20m36.763675402s) [noise=true , address=77.90.31.19:4843 , piece_id=ZAH6WQ4H552X4S4AMVZE27U5YKB7AQ4A7HOALXIAVVYUFHPLNE2Q, seg_pos={0 0}, upload=20575]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Sep 7 12:49:00.724228204 (23h20m36.763801324s) [noise=true , address=77.90.31.10:4519 , piece_id=QSIHMVF7CX5P46AD3B5OGMMHYCRX7J5LPGMCDG5YC3YP4AWCU4KQ, seg_pos={0 0}, upload=20575]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:44730->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:44730->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44730->77.90.31.10:4519: use of closed network connection Sep 7 12:49:00.724261607 (23h20m36.763834738s) [seg_pos={0 0}, upload=20575]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:49:00.724357832 (23h20m36.763930952s) [upload=20575]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:49:00.724365256 (23h20m36.763938376s) [upload=20575]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:49:01.053219518 (23h20m37.092792638s) [upload=20575]: Done uploading segments... Sep 7 12:49:01.053241079 (23h20m37.092814199s) [upload=20575]: Upload finished. err: <nil> Duration: 30.002477529s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........X.....X.....................X...............X.X..] duration: 11.709118281s FAIL (2025-09-07 12:48:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:46:39.535477608 (23h18m15.575050728s) [upload=20543]: Starting upload Sep 7 12:46:39.535533515 (23h18m15.575106635s) [upload=20543]: Done starting upload Sep 7 12:46:39.535626553 (23h18m15.575199673s) [upload=20543]: Uploading segments... Sep 7 12:46:39.535742064 (23h18m15.575315184s) [upload=20543]: Got next segment. Inline: false Sep 7 12:46:39.535793983 (23h18m15.575367103s) [upload=20543]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:46:39.535798933 (23h18m15.575372054s) [upload=20543]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:46:40.960034818 (23h18m16.999607938s) [seg_pos={0 0}, upload=20543]: Begin upload segment... Sep 7 12:46:40.960096547 (23h18m16.999669657s) [seg_pos={0 0}, upload=20543]: Done begin upload segment. Sep 7 12:46:40.960119721 (23h18m16.999693092s) [noise=true , address=45.91.116.8:4045 , piece_id=HSYSWUKVJHY4HKJ4RY2BAQM6LWQZJ2UGQV7T2NNQC5HQ7KSYO76Q, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.960180207 (23h18m16.999753327s) [noise=true , address=77.90.31.19:4842 , piece_id=IEMALQLCDW3G2Y2YJMGN6XRJC6XNSV7XCS7CBHYY4YLXCB6ZEMTA, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.960515027 (23h18m17.000088147s) [noise=true , address=45.91.116.6:4974 , piece_id=2URYYBMTZGKH4AIEZMHUY7ZNWANLPLDDOVXGL7ILCJAV7AXPMVNQ, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.960765336 (23h18m17.000338456s) [noise=true , address=45.91.116.10:4083 , piece_id=FSE2M2GRAXTKIQOQCOJQBLUUMI6BDBAUVAHWVIKFXDE77OFIXU2Q, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.960859487 (23h18m17.000432607s) [noise=true , address=77.90.31.14:4663 , piece_id=GFO4FE3N2WDXYRC7ZQUDRDB77LRKF46FH3YK7N2JP4DTCBJTG6QQ, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.961150082 (23h18m17.000723202s) [noise=true , address=45.91.116.13:4154 , piece_id=XBKAUHCSUURU2HPAMIYXSKBUYZH4N5PSCQYO5EC57GOILMKXUL2Q, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.961197273 (23h18m17.000770393s) [noise=true , address=77.90.31.16:4733 , piece_id=7IAWGTLS4YWN2SNLTYKOGENJUC3Q2VBUXB3OGPABAIHEGAXBYP3Q, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.961508638 (23h18m17.001081758s) [noise=true , address=45.91.116.4:4940 , piece_id=ST2J73V4Y3QIS2WDQ4V56WWWADBXINVGWCJKDEYC7LFNL2ITVYXA, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.961518277 (23h18m17.001091387s) [noise=true , address=45.91.116.11:4119 , piece_id=NHORDQHOLI6SYBBLYJNY3AOQVCOJQPUWL6OVDAO6QVAFSFL6F55Q, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.961616985 (23h18m17.001190105s) [noise=true , address=77.90.31.14:4662 , piece_id=LBZEVTII7WLL3AD24HZPZ3EWAQDN5WIBACISZIFDC62VDUA2QIOQ, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.962001351 (23h18m17.001574471s) [noise=true , address=45.91.116.12:4225 , piece_id=64VJADUTCOHT7AU2T32MJACH4SMGCNS7CG2ILETY52GDLBTPQVWA, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.962130749 (23h18m17.001703869s) [noise=true , address=77.90.31.13:4628 , piece_id=OWSFAQX247GPTJZYBLL6QXNQGQ2QVKS6ZOLGZNLK6R2DZSKS4ZFQ, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.962306986 (23h18m17.001880107s) [noise=true , address=45.91.116.5:4298 , piece_id=2QPXKNI62BLOJK6NESKZHMVXXALJJPFIBLUSCEBG7Z2GXM2KAGAA, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.962522518 (23h18m17.002095638s) [noise=true , address=77.90.31.21:4878 , piece_id=QQHXG4N7GVEUGH5LMT37LO5UHV3XS4QBJIM7655EQY4KBRLH7FHQ, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.962885073 (23h18m17.002458193s) [noise=true , address=45.91.116.7:4010 , piece_id=LPDCYX33M457KKCCGNLV2UC6WKVAPW4VO3TA3YD44NIUJSGRNMUQ, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.963116425 (23h18m17.002689545s) [noise=true , address=45.91.116.6:4975 , piece_id=JYLROY6GTC76WPQEAJCB246L4AC32R567AUI36I4BTPPAM3DNAIQ, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:42.819462085 (23h18m18.859035205s) [upload=20543]: Next returned nil segment Sep 7 12:46:42.819479668 (23h18m18.859052788s) [upload=20543]: Waiting for error group managing segments... Sep 7 12:46:43.110148001 (23h18m19.149721111s) [noise=true , address=77.90.31.19:4842 , piece_id=IEMALQLCDW3G2Y2YJMGN6XRJC6XNSV7XCS7CBHYY4YLXCB6ZEMTA, seg_pos={0 0}, upload=20543]: Done uploading piece. err: <nil> Sep 7 12:46:43.126737375 (23h18m19.166310495s) [noise=true , address=45.91.116.5:4298 , piece_id=2QPXKNI62BLOJK6NESKZHMVXXALJJPFIBLUSCEBG7Z2GXM2KAGAA, seg_pos={0 0}, upload=20543]: Done uploading piece. err: <nil> Sep 7 12:46:43.131654754 (23h18m19.171227874s) [noise=true , address=77.90.31.13:4628 , piece_id=OWSFAQX247GPTJZYBLL6QXNQGQ2QVKS6ZOLGZNLK6R2DZSKS4ZFQ, seg_pos={0 0}, upload=20543]: Done uploading piece. err: <nil> Sep 7 12:46:43.147300583 (23h18m19.186873704s) [noise=true , address=77.90.31.14:4663 , piece_id=GFO4FE3N2WDXYRC7ZQUDRDB77LRKF46FH3YK7N2JP4DTCBJTG6QQ, seg_pos={0 0}, upload=20543]: Done uploading piece. err: <nil> Sep 7 12:46:43.157906207 (23h18m19.197479328s) [noise=true , address=45.91.116.7:4010 , piece_id=LPDCYX33M457KKCCGNLV2UC6WKVAPW4VO3TA3YD44NIUJSGRNMUQ, seg_pos={0 0}, upload=20543]: Done uploading piece. err: <nil> Sep 7 12:46:43.158695538 (23h18m19.198268658s) [noise=true , address=45.91.116.8:4045 , piece_id=HSYSWUKVJHY4HKJ4RY2BAQM6LWQZJ2UGQV7T2NNQC5HQ7KSYO76Q, seg_pos={0 0}, upload=20543]: Done uploading piece. err: <nil> Sep 7 12:46:43.172467922 (23h18m19.212041042s) [noise=true , address=45.91.116.4:4940 , piece_id=ST2J73V4Y3QIS2WDQ4V56WWWADBXINVGWCJKDEYC7LFNL2ITVYXA, seg_pos={0 0}, upload=20543]: Done uploading piece. err: <nil> Sep 7 12:46:43.196334731 (23h18m19.235907851s) [noise=true , address=45.91.116.11:4119 , piece_id=NHORDQHOLI6SYBBLYJNY3AOQVCOJQPUWL6OVDAO6QVAFSFL6F55Q, seg_pos={0 0}, upload=20543]: Done uploading piece. err: <nil> Sep 7 12:46:43.226122300 (23h18m19.26569542s) [noise=true , address=45.91.116.12:4225 , piece_id=64VJADUTCOHT7AU2T32MJACH4SMGCNS7CG2ILETY52GDLBTPQVWA, seg_pos={0 0}, upload=20543]: Done uploading piece. err: <nil> Sep 7 12:46:43.236720330 (23h18m19.27629345s) [noise=true , address=45.91.116.6:4974 , piece_id=2URYYBMTZGKH4AIEZMHUY7ZNWANLPLDDOVXGL7ILCJAV7AXPMVNQ, seg_pos={0 0}, upload=20543]: Done uploading piece. err: <nil> Sep 7 12:46:43.272336823 (23h18m19.311909934s) [noise=true , address=77.90.31.14:4662 , piece_id=LBZEVTII7WLL3AD24HZPZ3EWAQDN5WIBACISZIFDC62VDUA2QIOQ, seg_pos={0 0}, upload=20543]: Done uploading piece. err: <nil> Sep 7 12:46:43.282439575 (23h18m19.322012695s) [noise=true , address=77.90.31.21:4878 , piece_id=QQHXG4N7GVEUGH5LMT37LO5UHV3XS4QBJIM7655EQY4KBRLH7FHQ, seg_pos={0 0}, upload=20543]: Done uploading piece. err: <nil> Sep 7 12:46:43.286514593 (23h18m19.326087713s) [noise=true , address=45.91.116.10:4083 , piece_id=FSE2M2GRAXTKIQOQCOJQBLUUMI6BDBAUVAHWVIKFXDE77OFIXU2Q, seg_pos={0 0}, upload=20543]: Done uploading piece. err: <nil> Sep 7 12:46:43.286538989 (23h18m19.326112109s) [seg_pos={0 0}, upload=20543]: Segment reached optimal threshold of 13 pieces. Sep 7 12:46:43.286654741 (23h18m19.326227861s) [noise=true , address=45.91.116.6:4975 , piece_id=JYLROY6GTC76WPQEAJCB246L4AC32R567AUI36I4BTPPAM3DNAIQ, seg_pos={0 0}, upload=20543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 12:46:43.286718042 (23h18m19.326291162s) [noise=true , address=45.91.116.13:4154 , piece_id=XBKAUHCSUURU2HPAMIYXSKBUYZH4N5PSCQYO5EC57GOILMKXUL2Q, seg_pos={0 0}, upload=20543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Sep 7 12:46:43.286811511 (23h18m19.326384631s) [noise=true , address=77.90.31.16:4733 , piece_id=7IAWGTLS4YWN2SNLTYKOGENJUC3Q2VBUXB3OGPABAIHEGAXBYP3Q, seg_pos={0 0}, upload=20543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: expected piece hash; context canceled; EOF Sep 7 12:46:43.286853201 (23h18m19.326426321s) [seg_pos={0 0}, upload=20543]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:46:43.286929978 (23h18m19.326503098s) [upload=20543]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:46:43.286934066 (23h18m19.326507186s) [upload=20543]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:46:43.615037982 (23h18m19.654611112s) [upload=20543]: Done uploading segments... Sep 7 12:46:43.615060595 (23h18m19.654633725s) [upload=20543]: Upload finished. err: <nil> Duration: 11.709143368s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X...........X.....X.....................X...............X.X.] duration: 20.675830728s FAIL (2025-09-07 12:47:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:46:39.535477608 (23h18m15.575050728s) [upload=20543]: Starting upload Sep 7 12:46:39.535533515 (23h18m15.575106635s) [upload=20543]: Done starting upload Sep 7 12:46:39.535626553 (23h18m15.575199673s) [upload=20543]: Uploading segments... Sep 7 12:46:39.535742064 (23h18m15.575315184s) [upload=20543]: Got next segment. Inline: false Sep 7 12:46:39.535793983 (23h18m15.575367103s) [upload=20543]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:46:39.535798933 (23h18m15.575372054s) [upload=20543]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:46:40.960034818 (23h18m16.999607938s) [seg_pos={0 0}, upload=20543]: Begin upload segment... Sep 7 12:46:40.960096547 (23h18m16.999669657s) [seg_pos={0 0}, upload=20543]: Done begin upload segment. Sep 7 12:46:40.960119721 (23h18m16.999693092s) [noise=true , address=45.91.116.8:4045 , piece_id=HSYSWUKVJHY4HKJ4RY2BAQM6LWQZJ2UGQV7T2NNQC5HQ7KSYO76Q, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.960180207 (23h18m16.999753327s) [noise=true , address=77.90.31.19:4842 , piece_id=IEMALQLCDW3G2Y2YJMGN6XRJC6XNSV7XCS7CBHYY4YLXCB6ZEMTA, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.960515027 (23h18m17.000088147s) [noise=true , address=45.91.116.6:4974 , piece_id=2URYYBMTZGKH4AIEZMHUY7ZNWANLPLDDOVXGL7ILCJAV7AXPMVNQ, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.960765336 (23h18m17.000338456s) [noise=true , address=45.91.116.10:4083 , piece_id=FSE2M2GRAXTKIQOQCOJQBLUUMI6BDBAUVAHWVIKFXDE77OFIXU2Q, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.960859487 (23h18m17.000432607s) [noise=true , address=77.90.31.14:4663 , piece_id=GFO4FE3N2WDXYRC7ZQUDRDB77LRKF46FH3YK7N2JP4DTCBJTG6QQ, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.961150082 (23h18m17.000723202s) [noise=true , address=45.91.116.13:4154 , piece_id=XBKAUHCSUURU2HPAMIYXSKBUYZH4N5PSCQYO5EC57GOILMKXUL2Q, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.961197273 (23h18m17.000770393s) [noise=true , address=77.90.31.16:4733 , piece_id=7IAWGTLS4YWN2SNLTYKOGENJUC3Q2VBUXB3OGPABAIHEGAXBYP3Q, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.961508638 (23h18m17.001081758s) [noise=true , address=45.91.116.4:4940 , piece_id=ST2J73V4Y3QIS2WDQ4V56WWWADBXINVGWCJKDEYC7LFNL2ITVYXA, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.961518277 (23h18m17.001091387s) [noise=true , address=45.91.116.11:4119 , piece_id=NHORDQHOLI6SYBBLYJNY3AOQVCOJQPUWL6OVDAO6QVAFSFL6F55Q, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.961616985 (23h18m17.001190105s) [noise=true , address=77.90.31.14:4662 , piece_id=LBZEVTII7WLL3AD24HZPZ3EWAQDN5WIBACISZIFDC62VDUA2QIOQ, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.962001351 (23h18m17.001574471s) [noise=true , address=45.91.116.12:4225 , piece_id=64VJADUTCOHT7AU2T32MJACH4SMGCNS7CG2ILETY52GDLBTPQVWA, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.962130749 (23h18m17.001703869s) [noise=true , address=77.90.31.13:4628 , piece_id=OWSFAQX247GPTJZYBLL6QXNQGQ2QVKS6ZOLGZNLK6R2DZSKS4ZFQ, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.962306986 (23h18m17.001880107s) [noise=true , address=45.91.116.5:4298 , piece_id=2QPXKNI62BLOJK6NESKZHMVXXALJJPFIBLUSCEBG7Z2GXM2KAGAA, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.962522518 (23h18m17.002095638s) [noise=true , address=77.90.31.21:4878 , piece_id=QQHXG4N7GVEUGH5LMT37LO5UHV3XS4QBJIM7655EQY4KBRLH7FHQ, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.962885073 (23h18m17.002458193s) [noise=true , address=45.91.116.7:4010 , piece_id=LPDCYX33M457KKCCGNLV2UC6WKVAPW4VO3TA3YD44NIUJSGRNMUQ, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.963116425 (23h18m17.002689545s) [noise=true , address=45.91.116.6:4975 , piece_id=JYLROY6GTC76WPQEAJCB246L4AC32R567AUI36I4BTPPAM3DNAIQ, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:42.819462085 (23h18m18.859035205s) [upload=20543]: Next returned nil segment Sep 7 12:46:42.819479668 (23h18m18.859052788s) [upload=20543]: Waiting for error group managing segments... Sep 7 12:46:43.110148001 (23h18m19.149721111s) [noise=true , address=77.90.31.19:4842 , piece_id=IEMALQLCDW3G2Y2YJMGN6XRJC6XNSV7XCS7CBHYY4YLXCB6ZEMTA, seg_pos={0 0}, upload=20543]: Done uploading piece. err: <nil> Sep 7 12:46:43.126737375 (23h18m19.166310495s) [noise=true , address=45.91.116.5:4298 , piece_id=2QPXKNI62BLOJK6NESKZHMVXXALJJPFIBLUSCEBG7Z2GXM2KAGAA, seg_pos={0 0}, upload=20543]: Done uploading piece. err: <nil> Sep 7 12:46:43.131654754 (23h18m19.171227874s) [noise=true , address=77.90.31.13:4628 , piece_id=OWSFAQX247GPTJZYBLL6QXNQGQ2QVKS6ZOLGZNLK6R2DZSKS4ZFQ, seg_pos={0 0}, upload=20543]: Done uploading piece. err: <nil> Sep 7 12:46:43.147300583 (23h18m19.186873704s) [noise=true , address=77.90.31.14:4663 , piece_id=GFO4FE3N2WDXYRC7ZQUDRDB77LRKF46FH3YK7N2JP4DTCBJTG6QQ, seg_pos={0 0}, upload=20543]: Done uploading piece. err: <nil> Sep 7 12:46:43.157906207 (23h18m19.197479328s) [noise=true , address=45.91.116.7:4010 , piece_id=LPDCYX33M457KKCCGNLV2UC6WKVAPW4VO3TA3YD44NIUJSGRNMUQ, seg_pos={0 0}, upload=20543]: Done uploading piece. err: <nil> Sep 7 12:46:43.158695538 (23h18m19.198268658s) [noise=true , address=45.91.116.8:4045 , piece_id=HSYSWUKVJHY4HKJ4RY2BAQM6LWQZJ2UGQV7T2NNQC5HQ7KSYO76Q, seg_pos={0 0}, upload=20543]: Done uploading piece. err: <nil> Sep 7 12:46:43.172467922 (23h18m19.212041042s) [noise=true , address=45.91.116.4:4940 , piece_id=ST2J73V4Y3QIS2WDQ4V56WWWADBXINVGWCJKDEYC7LFNL2ITVYXA, seg_pos={0 0}, upload=20543]: Done uploading piece. err: <nil> Sep 7 12:46:43.196334731 (23h18m19.235907851s) [noise=true , address=45.91.116.11:4119 , piece_id=NHORDQHOLI6SYBBLYJNY3AOQVCOJQPUWL6OVDAO6QVAFSFL6F55Q, seg_pos={0 0}, upload=20543]: Done uploading piece. err: <nil> Sep 7 12:46:43.226122300 (23h18m19.26569542s) [noise=true , address=45.91.116.12:4225 , piece_id=64VJADUTCOHT7AU2T32MJACH4SMGCNS7CG2ILETY52GDLBTPQVWA, seg_pos={0 0}, upload=20543]: Done uploading piece. err: <nil> Sep 7 12:46:43.236720330 (23h18m19.27629345s) [noise=true , address=45.91.116.6:4974 , piece_id=2URYYBMTZGKH4AIEZMHUY7ZNWANLPLDDOVXGL7ILCJAV7AXPMVNQ, seg_pos={0 0}, upload=20543]: Done uploading piece. err: <nil> Sep 7 12:46:43.272336823 (23h18m19.311909934s) [noise=true , address=77.90.31.14:4662 , piece_id=LBZEVTII7WLL3AD24HZPZ3EWAQDN5WIBACISZIFDC62VDUA2QIOQ, seg_pos={0 0}, upload=20543]: Done uploading piece. err: <nil> Sep 7 12:46:43.282439575 (23h18m19.322012695s) [noise=true , address=77.90.31.21:4878 , piece_id=QQHXG4N7GVEUGH5LMT37LO5UHV3XS4QBJIM7655EQY4KBRLH7FHQ, seg_pos={0 0}, upload=20543]: Done uploading piece. err: <nil> Sep 7 12:46:43.286514593 (23h18m19.326087713s) [noise=true , address=45.91.116.10:4083 , piece_id=FSE2M2GRAXTKIQOQCOJQBLUUMI6BDBAUVAHWVIKFXDE77OFIXU2Q, seg_pos={0 0}, upload=20543]: Done uploading piece. err: <nil> Sep 7 12:46:43.286538989 (23h18m19.326112109s) [seg_pos={0 0}, upload=20543]: Segment reached optimal threshold of 13 pieces. Sep 7 12:46:43.286654741 (23h18m19.326227861s) [noise=true , address=45.91.116.6:4975 , piece_id=JYLROY6GTC76WPQEAJCB246L4AC32R567AUI36I4BTPPAM3DNAIQ, seg_pos={0 0}, upload=20543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 12:46:43.286718042 (23h18m19.326291162s) [noise=true , address=45.91.116.13:4154 , piece_id=XBKAUHCSUURU2HPAMIYXSKBUYZH4N5PSCQYO5EC57GOILMKXUL2Q, seg_pos={0 0}, upload=20543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Sep 7 12:46:43.286811511 (23h18m19.326384631s) [noise=true , address=77.90.31.16:4733 , piece_id=7IAWGTLS4YWN2SNLTYKOGENJUC3Q2VBUXB3OGPABAIHEGAXBYP3Q, seg_pos={0 0}, upload=20543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: expected piece hash; context canceled; EOF Sep 7 12:46:43.286853201 (23h18m19.326426321s) [seg_pos={0 0}, upload=20543]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:46:43.286929978 (23h18m19.326503098s) [upload=20543]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:46:43.286934066 (23h18m19.326507186s) [upload=20543]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:46:43.615037982 (23h18m19.654611112s) [upload=20543]: Done uploading segments... Sep 7 12:46:43.615060595 (23h18m19.654633725s) [upload=20543]: Upload finished. err: <nil> Duration: 20.675854673s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X...........X.....X.....................X...............X.X] duration: 30.002026305s FAIL (2025-09-07 12:47:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:46:39.535477608 (23h18m15.575050728s) [upload=20543]: Starting upload Sep 7 12:46:39.535533515 (23h18m15.575106635s) [upload=20543]: Done starting upload Sep 7 12:46:39.535626553 (23h18m15.575199673s) [upload=20543]: Uploading segments... Sep 7 12:46:39.535742064 (23h18m15.575315184s) [upload=20543]: Got next segment. Inline: false Sep 7 12:46:39.535793983 (23h18m15.575367103s) [upload=20543]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:46:39.535798933 (23h18m15.575372054s) [upload=20543]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:46:40.960034818 (23h18m16.999607938s) [seg_pos={0 0}, upload=20543]: Begin upload segment... Sep 7 12:46:40.960096547 (23h18m16.999669657s) [seg_pos={0 0}, upload=20543]: Done begin upload segment. Sep 7 12:46:40.960119721 (23h18m16.999693092s) [noise=true , address=45.91.116.8:4045 , piece_id=HSYSWUKVJHY4HKJ4RY2BAQM6LWQZJ2UGQV7T2NNQC5HQ7KSYO76Q, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.960180207 (23h18m16.999753327s) [noise=true , address=77.90.31.19:4842 , piece_id=IEMALQLCDW3G2Y2YJMGN6XRJC6XNSV7XCS7CBHYY4YLXCB6ZEMTA, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.960515027 (23h18m17.000088147s) [noise=true , address=45.91.116.6:4974 , piece_id=2URYYBMTZGKH4AIEZMHUY7ZNWANLPLDDOVXGL7ILCJAV7AXPMVNQ, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.960765336 (23h18m17.000338456s) [noise=true , address=45.91.116.10:4083 , piece_id=FSE2M2GRAXTKIQOQCOJQBLUUMI6BDBAUVAHWVIKFXDE77OFIXU2Q, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.960859487 (23h18m17.000432607s) [noise=true , address=77.90.31.14:4663 , piece_id=GFO4FE3N2WDXYRC7ZQUDRDB77LRKF46FH3YK7N2JP4DTCBJTG6QQ, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.961150082 (23h18m17.000723202s) [noise=true , address=45.91.116.13:4154 , piece_id=XBKAUHCSUURU2HPAMIYXSKBUYZH4N5PSCQYO5EC57GOILMKXUL2Q, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.961197273 (23h18m17.000770393s) [noise=true , address=77.90.31.16:4733 , piece_id=7IAWGTLS4YWN2SNLTYKOGENJUC3Q2VBUXB3OGPABAIHEGAXBYP3Q, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.961508638 (23h18m17.001081758s) [noise=true , address=45.91.116.4:4940 , piece_id=ST2J73V4Y3QIS2WDQ4V56WWWADBXINVGWCJKDEYC7LFNL2ITVYXA, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.961518277 (23h18m17.001091387s) [noise=true , address=45.91.116.11:4119 , piece_id=NHORDQHOLI6SYBBLYJNY3AOQVCOJQPUWL6OVDAO6QVAFSFL6F55Q, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.961616985 (23h18m17.001190105s) [noise=true , address=77.90.31.14:4662 , piece_id=LBZEVTII7WLL3AD24HZPZ3EWAQDN5WIBACISZIFDC62VDUA2QIOQ, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.962001351 (23h18m17.001574471s) [noise=true , address=45.91.116.12:4225 , piece_id=64VJADUTCOHT7AU2T32MJACH4SMGCNS7CG2ILETY52GDLBTPQVWA, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.962130749 (23h18m17.001703869s) [noise=true , address=77.90.31.13:4628 , piece_id=OWSFAQX247GPTJZYBLL6QXNQGQ2QVKS6ZOLGZNLK6R2DZSKS4ZFQ, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.962306986 (23h18m17.001880107s) [noise=true , address=45.91.116.5:4298 , piece_id=2QPXKNI62BLOJK6NESKZHMVXXALJJPFIBLUSCEBG7Z2GXM2KAGAA, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.962522518 (23h18m17.002095638s) [noise=true , address=77.90.31.21:4878 , piece_id=QQHXG4N7GVEUGH5LMT37LO5UHV3XS4QBJIM7655EQY4KBRLH7FHQ, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.962885073 (23h18m17.002458193s) [noise=true , address=45.91.116.7:4010 , piece_id=LPDCYX33M457KKCCGNLV2UC6WKVAPW4VO3TA3YD44NIUJSGRNMUQ, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:40.963116425 (23h18m17.002689545s) [noise=true , address=45.91.116.6:4975 , piece_id=JYLROY6GTC76WPQEAJCB246L4AC32R567AUI36I4BTPPAM3DNAIQ, seg_pos={0 0}, upload=20543]: Uploading piece... Sep 7 12:46:42.819462085 (23h18m18.859035205s) [upload=20543]: Next returned nil segment Sep 7 12:46:42.819479668 (23h18m18.859052788s) [upload=20543]: Waiting for error group managing segments... Sep 7 12:46:43.110148001 (23h18m19.149721111s) [noise=true , address=77.90.31.19:4842 , piece_id=IEMALQLCDW3G2Y2YJMGN6XRJC6XNSV7XCS7CBHYY4YLXCB6ZEMTA, seg_pos={0 0}, upload=20543]: Done uploading piece. err: <nil> Sep 7 12:46:43.126737375 (23h18m19.166310495s) [noise=true , address=45.91.116.5:4298 , piece_id=2QPXKNI62BLOJK6NESKZHMVXXALJJPFIBLUSCEBG7Z2GXM2KAGAA, seg_pos={0 0}, upload=20543]: Done uploading piece. err: <nil> Sep 7 12:46:43.131654754 (23h18m19.171227874s) [noise=true , address=77.90.31.13:4628 , piece_id=OWSFAQX247GPTJZYBLL6QXNQGQ2QVKS6ZOLGZNLK6R2DZSKS4ZFQ, seg_pos={0 0}, upload=20543]: Done uploading piece. err: <nil> Sep 7 12:46:43.147300583 (23h18m19.186873704s) [noise=true , address=77.90.31.14:4663 , piece_id=GFO4FE3N2WDXYRC7ZQUDRDB77LRKF46FH3YK7N2JP4DTCBJTG6QQ, seg_pos={0 0}, upload=20543]: Done uploading piece. err: <nil> Sep 7 12:46:43.157906207 (23h18m19.197479328s) [noise=true , address=45.91.116.7:4010 , piece_id=LPDCYX33M457KKCCGNLV2UC6WKVAPW4VO3TA3YD44NIUJSGRNMUQ, seg_pos={0 0}, upload=20543]: Done uploading piece. err: <nil> Sep 7 12:46:43.158695538 (23h18m19.198268658s) [noise=true , address=45.91.116.8:4045 , piece_id=HSYSWUKVJHY4HKJ4RY2BAQM6LWQZJ2UGQV7T2NNQC5HQ7KSYO76Q, seg_pos={0 0}, upload=20543]: Done uploading piece. err: <nil> Sep 7 12:46:43.172467922 (23h18m19.212041042s) [noise=true , address=45.91.116.4:4940 , piece_id=ST2J73V4Y3QIS2WDQ4V56WWWADBXINVGWCJKDEYC7LFNL2ITVYXA, seg_pos={0 0}, upload=20543]: Done uploading piece. err: <nil> Sep 7 12:46:43.196334731 (23h18m19.235907851s) [noise=true , address=45.91.116.11:4119 , piece_id=NHORDQHOLI6SYBBLYJNY3AOQVCOJQPUWL6OVDAO6QVAFSFL6F55Q, seg_pos={0 0}, upload=20543]: Done uploading piece. err: <nil> Sep 7 12:46:43.226122300 (23h18m19.26569542s) [noise=true , address=45.91.116.12:4225 , piece_id=64VJADUTCOHT7AU2T32MJACH4SMGCNS7CG2ILETY52GDLBTPQVWA, seg_pos={0 0}, upload=20543]: Done uploading piece. err: <nil> Sep 7 12:46:43.236720330 (23h18m19.27629345s) [noise=true , address=45.91.116.6:4974 , piece_id=2URYYBMTZGKH4AIEZMHUY7ZNWANLPLDDOVXGL7ILCJAV7AXPMVNQ, seg_pos={0 0}, upload=20543]: Done uploading piece. err: <nil> Sep 7 12:46:43.272336823 (23h18m19.311909934s) [noise=true , address=77.90.31.14:4662 , piece_id=LBZEVTII7WLL3AD24HZPZ3EWAQDN5WIBACISZIFDC62VDUA2QIOQ, seg_pos={0 0}, upload=20543]: Done uploading piece. err: <nil> Sep 7 12:46:43.282439575 (23h18m19.322012695s) [noise=true , address=77.90.31.21:4878 , piece_id=QQHXG4N7GVEUGH5LMT37LO5UHV3XS4QBJIM7655EQY4KBRLH7FHQ, seg_pos={0 0}, upload=20543]: Done uploading piece. err: <nil> Sep 7 12:46:43.286514593 (23h18m19.326087713s) [noise=true , address=45.91.116.10:4083 , piece_id=FSE2M2GRAXTKIQOQCOJQBLUUMI6BDBAUVAHWVIKFXDE77OFIXU2Q, seg_pos={0 0}, upload=20543]: Done uploading piece. err: <nil> Sep 7 12:46:43.286538989 (23h18m19.326112109s) [seg_pos={0 0}, upload=20543]: Segment reached optimal threshold of 13 pieces. Sep 7 12:46:43.286654741 (23h18m19.326227861s) [noise=true , address=45.91.116.6:4975 , piece_id=JYLROY6GTC76WPQEAJCB246L4AC32R567AUI36I4BTPPAM3DNAIQ, seg_pos={0 0}, upload=20543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Sep 7 12:46:43.286718042 (23h18m19.326291162s) [noise=true , address=45.91.116.13:4154 , piece_id=XBKAUHCSUURU2HPAMIYXSKBUYZH4N5PSCQYO5EC57GOILMKXUL2Q, seg_pos={0 0}, upload=20543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Sep 7 12:46:43.286811511 (23h18m19.326384631s) [noise=true , address=77.90.31.16:4733 , piece_id=7IAWGTLS4YWN2SNLTYKOGENJUC3Q2VBUXB3OGPABAIHEGAXBYP3Q, seg_pos={0 0}, upload=20543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: expected piece hash; context canceled; EOF Sep 7 12:46:43.286853201 (23h18m19.326426321s) [seg_pos={0 0}, upload=20543]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:46:43.286929978 (23h18m19.326503098s) [upload=20543]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:46:43.286934066 (23h18m19.326507186s) [upload=20543]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:46:43.615037982 (23h18m19.654611112s) [upload=20543]: Done uploading segments... Sep 7 12:46:43.615060595 (23h18m19.654633725s) [upload=20543]: Upload finished. err: <nil> Duration: 30.00204925s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X...........X.....X.....................X...............X.] duration: 18.879266881s FAIL (2025-09-07 12:46:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:45:41.753226889 (23h17m17.792799999s) [upload=20529]: Starting upload Sep 7 12:45:41.753298637 (23h17m17.792871747s) [upload=20529]: Done starting upload Sep 7 12:45:41.753510883 (23h17m17.793084013s) [upload=20529]: Uploading segments... Sep 7 12:45:41.753658315 (23h17m17.793231435s) [upload=20529]: Got next segment. Inline: false Sep 7 12:45:41.753702911 (23h17m17.793276041s) [upload=20529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:45:41.753721045 (23h17m17.793294165s) [upload=20529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:45:42.016399167 (23h17m18.055972287s) [seg_pos={0 0}, upload=20529]: Begin upload segment... Sep 7 12:45:42.016445867 (23h17m18.056018987s) [seg_pos={0 0}, upload=20529]: Done begin upload segment. Sep 7 12:45:42.016463650 (23h17m18.05603677s) [noise=true , address=45.91.116.14:4191 , piece_id=AQWX2NIR2PPZVBKKPZSIV37OMFZOYBEISNOXNF6M3FOU4GJBQMIQ, seg_pos={0 0}, upload=20529]: Uploading piece... Sep 7 12:45:42.016474822 (23h17m18.056047942s) [noise=true , address=77.90.31.19:4842 , piece_id=YW2BM3NCS4MIDQBMARZAORNPA3ZUS3MLQCCOB5SDVMLNZHBU3ZGA, seg_pos={0 0}, upload=20529]: Uploading piece... Sep 7 12:45:42.016838588 (23h17m18.056411708s) [noise=true , address=45.91.116.11:4118 , piece_id=2YJH2GP3AXJOUNVJMJ37TEZ47FXDJ7DKGBKQYNFLHPP63JLRFKIA, seg_pos={0 0}, upload=20529]: Uploading piece... Sep 7 12:45:42.016853476 (23h17m18.056426596s) [noise=true , address=45.91.116.9:4261 , piece_id=MMGRQYHVO5KCGYPWZZQ4BHM3QOS7BR5O2DU77Y5VOPF4KYECWZGQ, seg_pos={0 0}, upload=20529]: Uploading piece... Sep 7 12:45:42.017159011 (23h17m18.056732131s) [noise=true , address=45.91.116.8:4045 , piece_id=BTSUTWYL5JNQPWZD5VGNPDNNBLB5NKFOA4CDJXMUVSGG7BU2ZVIA, seg_pos={0 0}, upload=20529]: Uploading piece... Sep 7 12:45:42.017467762 (23h17m18.057040882s) [noise=true , address=77.90.31.19:4841 , piece_id=RQUN4BXV3W3JBCAS4NE43CWFZZH6ZQPGCMDDFQ5HEMCKB6CZN4CA, seg_pos={0 0}, upload=20529]: Uploading piece... Sep 7 12:45:42.021988592 (23h17m18.061561712s) [noise=true , address=77.90.31.10:4518 , piece_id=BYM7QXV6HUWBCKGCM7J3OVXWPPOQFNXINBK3ADZYFKPN7FB6H3TA, seg_pos={0 0}, upload=20529]: Uploading piece... Sep 7 12:45:42.023349066 (23h17m18.062922186s) [noise=true , address=77.90.31.14:4662 , piece_id=VO56NH5GVZQ435IM6XF4CHNQ2X527U46LZEGY4MJFHSGXY425D6Q, seg_pos={0 0}, upload=20529]: Uploading piece... Sep 7 12:45:42.023644070 (23h17m18.063217211s) [noise=true , address=45.91.116.4:4938 , piece_id=Q5OUAH2X2HU3DKVL77AWQHHNGJIMC4HKEBV4YG7RSOIRUOASKDEA, seg_pos={0 0}, upload=20529]: Uploading piece... Sep 7 12:45:42.023983750 (23h17m18.06355687s) [noise=true , address=45.91.116.5:4299 , piece_id=6FL5GYAJXZR2G7YNVFA3EPPFY7NPYHT5IQNHAHBMYK7ARRZWHTKQ, seg_pos={0 0}, upload=20529]: Uploading piece... Sep 7 12:45:42.024275609 (23h17m18.063848729s) [noise=true , address=45.91.116.12:4226 , piece_id=3QDLZGCFTMRGJGWMRLTGWONGQV4EAOFFRKSB2MBTUJGVV6LXVLRQ, seg_pos={0 0}, upload=20529]: Uploading piece... Sep 7 12:45:42.024527320 (23h17m18.06410044s) [noise=true , address=77.90.31.15:4697 , piece_id=S2VUSY7MUIGUNC44YL76R6FGTGMIFM42I7M3H27YBKG3J6CZ6ISQ, seg_pos={0 0}, upload=20529]: Uploading piece... Sep 7 12:45:42.024776457 (23h17m18.064349577s) [noise=true , address=77.90.31.16:4734 , piece_id=MLDJZEE3DYDSEAUG2GV7264UNGX65UYL2QYDNHAX3FYAB446HLIQ, seg_pos={0 0}, upload=20529]: Uploading piece... Sep 7 12:45:42.025071942 (23h17m18.064645062s) [noise=true , address=45.91.116.10:4082 , piece_id=FTTAPOMHIIIEYUPRX3VJVSGKXGZJROQRAVGOEHMJRCHP7EHOOI2Q, seg_pos={0 0}, upload=20529]: Uploading piece... Sep 7 12:45:42.025360325 (23h17m18.064933445s) [noise=true , address=45.91.116.7:4011 , piece_id=ZI7P636ZRLMNT26EDOSEFC5MRK73BK4LYRGGVMVL3RQTMFDB2OFQ, seg_pos={0 0}, upload=20529]: Uploading piece... Sep 7 12:45:42.025736214 (23h17m18.065309334s) [noise=true , address=45.91.116.5:4298 , piece_id=C42LHBKS5IQ7H7NKVLWZETEIW323PF4MBQ2CUOZIMUNGZNMM7JIA, seg_pos={0 0}, upload=20529]: Uploading piece... Sep 7 12:45:43.861844659 (23h17m19.901417789s) [upload=20529]: Next returned nil segment Sep 7 12:45:43.861862323 (23h17m19.901435443s) [upload=20529]: Waiting for error group managing segments... Sep 7 12:45:44.183998409 (23h17m20.223571529s) [noise=true , address=45.91.116.10:4082 , piece_id=FTTAPOMHIIIEYUPRX3VJVSGKXGZJROQRAVGOEHMJRCHP7EHOOI2Q, seg_pos={0 0}, upload=20529]: Done uploading piece. err: <nil> Sep 7 12:45:44.184904823 (23h17m20.224477944s) [noise=true , address=77.90.31.10:4518 , piece_id=BYM7QXV6HUWBCKGCM7J3OVXWPPOQFNXINBK3ADZYFKPN7FB6H3TA, seg_pos={0 0}, upload=20529]: Done uploading piece. err: <nil> Sep 7 12:45:44.187827797 (23h17m20.227400917s) [noise=true , address=45.91.116.9:4261 , piece_id=MMGRQYHVO5KCGYPWZZQ4BHM3QOS7BR5O2DU77Y5VOPF4KYECWZGQ, seg_pos={0 0}, upload=20529]: Done uploading piece. err: <nil> Sep 7 12:45:44.193957667 (23h17m20.233530787s) [noise=true , address=45.91.116.11:4118 , piece_id=2YJH2GP3AXJOUNVJMJ37TEZ47FXDJ7DKGBKQYNFLHPP63JLRFKIA, seg_pos={0 0}, upload=20529]: Done uploading piece. err: <nil> Sep 7 12:45:44.194859643 (23h17m20.234432763s) [noise=true , address=45.91.116.8:4045 , piece_id=BTSUTWYL5JNQPWZD5VGNPDNNBLB5NKFOA4CDJXMUVSGG7BU2ZVIA, seg_pos={0 0}, upload=20529]: Done uploading piece. err: <nil> Sep 7 12:45:44.204934661 (23h17m20.244507781s) [noise=true , address=45.91.116.5:4298 , piece_id=C42LHBKS5IQ7H7NKVLWZETEIW323PF4MBQ2CUOZIMUNGZNMM7JIA, seg_pos={0 0}, upload=20529]: Done uploading piece. err: <nil> Sep 7 12:45:44.215082019 (23h17m20.254655189s) [noise=true , address=45.91.116.12:4226 , piece_id=3QDLZGCFTMRGJGWMRLTGWONGQV4EAOFFRKSB2MBTUJGVV6LXVLRQ, seg_pos={0 0}, upload=20529]: Done uploading piece. err: <nil> Sep 7 12:45:44.228488201 (23h17m20.268061322s) [noise=true , address=77.90.31.16:4734 , piece_id=MLDJZEE3DYDSEAUG2GV7264UNGX65UYL2QYDNHAX3FYAB446HLIQ, seg_pos={0 0}, upload=20529]: Done uploading piece. err: <nil> Sep 7 12:45:44.272543292 (23h17m20.312116412s) [noise=true , address=45.91.116.7:4011 , piece_id=ZI7P636ZRLMNT26EDOSEFC5MRK73BK4LYRGGVMVL3RQTMFDB2OFQ, seg_pos={0 0}, upload=20529]: Done uploading piece. err: <nil> Sep 7 12:45:44.275942046 (23h17m20.315515166s) [noise=true , address=45.91.116.14:4191 , piece_id=AQWX2NIR2PPZVBKKPZSIV37OMFZOYBEISNOXNF6M3FOU4GJBQMIQ, seg_pos={0 0}, upload=20529]: Done uploading piece. err: <nil> Sep 7 12:45:44.302383832 (23h17m20.341956973s) [noise=true , address=77.90.31.15:4697 , piece_id=S2VUSY7MUIGUNC44YL76R6FGTGMIFM42I7M3H27YBKG3J6CZ6ISQ, seg_pos={0 0}, upload=20529]: Done uploading piece. err: <nil> Sep 7 12:45:44.315241366 (23h17m20.354814496s) [noise=true , address=45.91.116.4:4938 , piece_id=Q5OUAH2X2HU3DKVL77AWQHHNGJIMC4HKEBV4YG7RSOIRUOASKDEA, seg_pos={0 0}, upload=20529]: Done uploading piece. err: <nil> Sep 7 12:45:44.353274323 (23h17m20.392847443s) [noise=true , address=77.90.31.19:4841 , piece_id=RQUN4BXV3W3JBCAS4NE43CWFZZH6ZQPGCMDDFQ5HEMCKB6CZN4CA, seg_pos={0 0}, upload=20529]: Done uploading piece. err: <nil> Sep 7 12:45:44.353302177 (23h17m20.392875297s) [seg_pos={0 0}, upload=20529]: Segment reached optimal threshold of 13 pieces. Sep 7 12:45:44.353437315 (23h17m20.393010425s) [noise=true , address=77.90.31.19:4842 , piece_id=YW2BM3NCS4MIDQBMARZAORNPA3ZUS3MLQCCOB5SDVMLNZHBU3ZGA, seg_pos={0 0}, upload=20529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 7 12:45:44.353507639 (23h17m20.393080759s) [noise=true , address=45.91.116.5:4299 , piece_id=6FL5GYAJXZR2G7YNVFA3EPPFY7NPYHT5IQNHAHBMYK7ARRZWHTKQ, seg_pos={0 0}, upload=20529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Sep 7 12:45:44.353661925 (23h17m20.393235045s) [noise=true , address=77.90.31.14:4662 , piece_id=VO56NH5GVZQ435IM6XF4CHNQ2X527U46LZEGY4MJFHSGXY425D6Q, seg_pos={0 0}, upload=20529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:45494->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:45494->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45494->77.90.31.14:4662: use of closed network connection Sep 7 12:45:44.353700859 (23h17m20.393273979s) [seg_pos={0 0}, upload=20529]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:45:44.353777546 (23h17m20.393350666s) [upload=20529]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:45:44.353783156 (23h17m20.393356276s) [upload=20529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:45:44.696843874 (23h17m20.736416994s) [upload=20529]: Done uploading segments... Sep 7 12:45:44.696860815 (23h17m20.736433936s) [upload=20529]: Upload finished. err: <nil> Duration: 18.87928738s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X...........X.....X.....................X...............X] duration: 30.003268261s FAIL (2025-09-07 12:46:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:45:41.753226889 (23h17m17.792799999s) [upload=20529]: Starting upload Sep 7 12:45:41.753298637 (23h17m17.792871747s) [upload=20529]: Done starting upload Sep 7 12:45:41.753510883 (23h17m17.793084013s) [upload=20529]: Uploading segments... Sep 7 12:45:41.753658315 (23h17m17.793231435s) [upload=20529]: Got next segment. Inline: false Sep 7 12:45:41.753702911 (23h17m17.793276041s) [upload=20529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:45:41.753721045 (23h17m17.793294165s) [upload=20529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:45:42.016399167 (23h17m18.055972287s) [seg_pos={0 0}, upload=20529]: Begin upload segment... Sep 7 12:45:42.016445867 (23h17m18.056018987s) [seg_pos={0 0}, upload=20529]: Done begin upload segment. Sep 7 12:45:42.016463650 (23h17m18.05603677s) [noise=true , address=45.91.116.14:4191 , piece_id=AQWX2NIR2PPZVBKKPZSIV37OMFZOYBEISNOXNF6M3FOU4GJBQMIQ, seg_pos={0 0}, upload=20529]: Uploading piece... Sep 7 12:45:42.016474822 (23h17m18.056047942s) [noise=true , address=77.90.31.19:4842 , piece_id=YW2BM3NCS4MIDQBMARZAORNPA3ZUS3MLQCCOB5SDVMLNZHBU3ZGA, seg_pos={0 0}, upload=20529]: Uploading piece... Sep 7 12:45:42.016838588 (23h17m18.056411708s) [noise=true , address=45.91.116.11:4118 , piece_id=2YJH2GP3AXJOUNVJMJ37TEZ47FXDJ7DKGBKQYNFLHPP63JLRFKIA, seg_pos={0 0}, upload=20529]: Uploading piece... Sep 7 12:45:42.016853476 (23h17m18.056426596s) [noise=true , address=45.91.116.9:4261 , piece_id=MMGRQYHVO5KCGYPWZZQ4BHM3QOS7BR5O2DU77Y5VOPF4KYECWZGQ, seg_pos={0 0}, upload=20529]: Uploading piece... Sep 7 12:45:42.017159011 (23h17m18.056732131s) [noise=true , address=45.91.116.8:4045 , piece_id=BTSUTWYL5JNQPWZD5VGNPDNNBLB5NKFOA4CDJXMUVSGG7BU2ZVIA, seg_pos={0 0}, upload=20529]: Uploading piece... Sep 7 12:45:42.017467762 (23h17m18.057040882s) [noise=true , address=77.90.31.19:4841 , piece_id=RQUN4BXV3W3JBCAS4NE43CWFZZH6ZQPGCMDDFQ5HEMCKB6CZN4CA, seg_pos={0 0}, upload=20529]: Uploading piece... Sep 7 12:45:42.021988592 (23h17m18.061561712s) [noise=true , address=77.90.31.10:4518 , piece_id=BYM7QXV6HUWBCKGCM7J3OVXWPPOQFNXINBK3ADZYFKPN7FB6H3TA, seg_pos={0 0}, upload=20529]: Uploading piece... Sep 7 12:45:42.023349066 (23h17m18.062922186s) [noise=true , address=77.90.31.14:4662 , piece_id=VO56NH5GVZQ435IM6XF4CHNQ2X527U46LZEGY4MJFHSGXY425D6Q, seg_pos={0 0}, upload=20529]: Uploading piece... Sep 7 12:45:42.023644070 (23h17m18.063217211s) [noise=true , address=45.91.116.4:4938 , piece_id=Q5OUAH2X2HU3DKVL77AWQHHNGJIMC4HKEBV4YG7RSOIRUOASKDEA, seg_pos={0 0}, upload=20529]: Uploading piece... Sep 7 12:45:42.023983750 (23h17m18.06355687s) [noise=true , address=45.91.116.5:4299 , piece_id=6FL5GYAJXZR2G7YNVFA3EPPFY7NPYHT5IQNHAHBMYK7ARRZWHTKQ, seg_pos={0 0}, upload=20529]: Uploading piece... Sep 7 12:45:42.024275609 (23h17m18.063848729s) [noise=true , address=45.91.116.12:4226 , piece_id=3QDLZGCFTMRGJGWMRLTGWONGQV4EAOFFRKSB2MBTUJGVV6LXVLRQ, seg_pos={0 0}, upload=20529]: Uploading piece... Sep 7 12:45:42.024527320 (23h17m18.06410044s) [noise=true , address=77.90.31.15:4697 , piece_id=S2VUSY7MUIGUNC44YL76R6FGTGMIFM42I7M3H27YBKG3J6CZ6ISQ, seg_pos={0 0}, upload=20529]: Uploading piece... Sep 7 12:45:42.024776457 (23h17m18.064349577s) [noise=true , address=77.90.31.16:4734 , piece_id=MLDJZEE3DYDSEAUG2GV7264UNGX65UYL2QYDNHAX3FYAB446HLIQ, seg_pos={0 0}, upload=20529]: Uploading piece... Sep 7 12:45:42.025071942 (23h17m18.064645062s) [noise=true , address=45.91.116.10:4082 , piece_id=FTTAPOMHIIIEYUPRX3VJVSGKXGZJROQRAVGOEHMJRCHP7EHOOI2Q, seg_pos={0 0}, upload=20529]: Uploading piece... Sep 7 12:45:42.025360325 (23h17m18.064933445s) [noise=true , address=45.91.116.7:4011 , piece_id=ZI7P636ZRLMNT26EDOSEFC5MRK73BK4LYRGGVMVL3RQTMFDB2OFQ, seg_pos={0 0}, upload=20529]: Uploading piece... Sep 7 12:45:42.025736214 (23h17m18.065309334s) [noise=true , address=45.91.116.5:4298 , piece_id=C42LHBKS5IQ7H7NKVLWZETEIW323PF4MBQ2CUOZIMUNGZNMM7JIA, seg_pos={0 0}, upload=20529]: Uploading piece... Sep 7 12:45:43.861844659 (23h17m19.901417789s) [upload=20529]: Next returned nil segment Sep 7 12:45:43.861862323 (23h17m19.901435443s) [upload=20529]: Waiting for error group managing segments... Sep 7 12:45:44.183998409 (23h17m20.223571529s) [noise=true , address=45.91.116.10:4082 , piece_id=FTTAPOMHIIIEYUPRX3VJVSGKXGZJROQRAVGOEHMJRCHP7EHOOI2Q, seg_pos={0 0}, upload=20529]: Done uploading piece. err: <nil> Sep 7 12:45:44.184904823 (23h17m20.224477944s) [noise=true , address=77.90.31.10:4518 , piece_id=BYM7QXV6HUWBCKGCM7J3OVXWPPOQFNXINBK3ADZYFKPN7FB6H3TA, seg_pos={0 0}, upload=20529]: Done uploading piece. err: <nil> Sep 7 12:45:44.187827797 (23h17m20.227400917s) [noise=true , address=45.91.116.9:4261 , piece_id=MMGRQYHVO5KCGYPWZZQ4BHM3QOS7BR5O2DU77Y5VOPF4KYECWZGQ, seg_pos={0 0}, upload=20529]: Done uploading piece. err: <nil> Sep 7 12:45:44.193957667 (23h17m20.233530787s) [noise=true , address=45.91.116.11:4118 , piece_id=2YJH2GP3AXJOUNVJMJ37TEZ47FXDJ7DKGBKQYNFLHPP63JLRFKIA, seg_pos={0 0}, upload=20529]: Done uploading piece. err: <nil> Sep 7 12:45:44.194859643 (23h17m20.234432763s) [noise=true , address=45.91.116.8:4045 , piece_id=BTSUTWYL5JNQPWZD5VGNPDNNBLB5NKFOA4CDJXMUVSGG7BU2ZVIA, seg_pos={0 0}, upload=20529]: Done uploading piece. err: <nil> Sep 7 12:45:44.204934661 (23h17m20.244507781s) [noise=true , address=45.91.116.5:4298 , piece_id=C42LHBKS5IQ7H7NKVLWZETEIW323PF4MBQ2CUOZIMUNGZNMM7JIA, seg_pos={0 0}, upload=20529]: Done uploading piece. err: <nil> Sep 7 12:45:44.215082019 (23h17m20.254655189s) [noise=true , address=45.91.116.12:4226 , piece_id=3QDLZGCFTMRGJGWMRLTGWONGQV4EAOFFRKSB2MBTUJGVV6LXVLRQ, seg_pos={0 0}, upload=20529]: Done uploading piece. err: <nil> Sep 7 12:45:44.228488201 (23h17m20.268061322s) [noise=true , address=77.90.31.16:4734 , piece_id=MLDJZEE3DYDSEAUG2GV7264UNGX65UYL2QYDNHAX3FYAB446HLIQ, seg_pos={0 0}, upload=20529]: Done uploading piece. err: <nil> Sep 7 12:45:44.272543292 (23h17m20.312116412s) [noise=true , address=45.91.116.7:4011 , piece_id=ZI7P636ZRLMNT26EDOSEFC5MRK73BK4LYRGGVMVL3RQTMFDB2OFQ, seg_pos={0 0}, upload=20529]: Done uploading piece. err: <nil> Sep 7 12:45:44.275942046 (23h17m20.315515166s) [noise=true , address=45.91.116.14:4191 , piece_id=AQWX2NIR2PPZVBKKPZSIV37OMFZOYBEISNOXNF6M3FOU4GJBQMIQ, seg_pos={0 0}, upload=20529]: Done uploading piece. err: <nil> Sep 7 12:45:44.302383832 (23h17m20.341956973s) [noise=true , address=77.90.31.15:4697 , piece_id=S2VUSY7MUIGUNC44YL76R6FGTGMIFM42I7M3H27YBKG3J6CZ6ISQ, seg_pos={0 0}, upload=20529]: Done uploading piece. err: <nil> Sep 7 12:45:44.315241366 (23h17m20.354814496s) [noise=true , address=45.91.116.4:4938 , piece_id=Q5OUAH2X2HU3DKVL77AWQHHNGJIMC4HKEBV4YG7RSOIRUOASKDEA, seg_pos={0 0}, upload=20529]: Done uploading piece. err: <nil> Sep 7 12:45:44.353274323 (23h17m20.392847443s) [noise=true , address=77.90.31.19:4841 , piece_id=RQUN4BXV3W3JBCAS4NE43CWFZZH6ZQPGCMDDFQ5HEMCKB6CZN4CA, seg_pos={0 0}, upload=20529]: Done uploading piece. err: <nil> Sep 7 12:45:44.353302177 (23h17m20.392875297s) [seg_pos={0 0}, upload=20529]: Segment reached optimal threshold of 13 pieces. Sep 7 12:45:44.353437315 (23h17m20.393010425s) [noise=true , address=77.90.31.19:4842 , piece_id=YW2BM3NCS4MIDQBMARZAORNPA3ZUS3MLQCCOB5SDVMLNZHBU3ZGA, seg_pos={0 0}, upload=20529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Sep 7 12:45:44.353507639 (23h17m20.393080759s) [noise=true , address=45.91.116.5:4299 , piece_id=6FL5GYAJXZR2G7YNVFA3EPPFY7NPYHT5IQNHAHBMYK7ARRZWHTKQ, seg_pos={0 0}, upload=20529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Sep 7 12:45:44.353661925 (23h17m20.393235045s) [noise=true , address=77.90.31.14:4662 , piece_id=VO56NH5GVZQ435IM6XF4CHNQ2X527U46LZEGY4MJFHSGXY425D6Q, seg_pos={0 0}, upload=20529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:45494->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:45494->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45494->77.90.31.14:4662: use of closed network connection Sep 7 12:45:44.353700859 (23h17m20.393273979s) [seg_pos={0 0}, upload=20529]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:45:44.353777546 (23h17m20.393350666s) [upload=20529]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:45:44.353783156 (23h17m20.393356276s) [upload=20529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:45:44.696843874 (23h17m20.736416994s) [upload=20529]: Done uploading segments... Sep 7 12:45:44.696860815 (23h17m20.736433936s) [upload=20529]: Upload finished. err: <nil> Duration: 30.003378674s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X...........X.....X.....................X...............] duration: 23.520972066s FAIL (2025-09-07 12:45:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:32:24.971479778 (23h4m1.011052898s) [upload=20341]: Starting upload Sep 7 12:32:24.971535354 (23h4m1.011108474s) [upload=20341]: Done starting upload Sep 7 12:32:24.971630667 (23h4m1.011203787s) [upload=20341]: Uploading segments... Sep 7 12:32:24.971695100 (23h4m1.01126822s) [upload=20341]: Got next segment. Inline: false Sep 7 12:32:24.971723614 (23h4m1.011296735s) [upload=20341]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:32:24.971728113 (23h4m1.011301233s) [upload=20341]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:32:25.259639680 (23h4m1.2992128s) [seg_pos={0 0}, upload=20341]: Begin upload segment... Sep 7 12:32:25.259682412 (23h4m1.299255532s) [seg_pos={0 0}, upload=20341]: Done begin upload segment. Sep 7 12:32:25.259731626 (23h4m1.299304756s) [noise=true , address=45.91.116.9:4263 , piece_id=VSYUTOSVLDLAOB74XZFESXWLL5EZN5Z25LYDH3WZQN5LMSDSGELQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.259743980 (23h4m1.2993171s) [noise=true , address=45.91.116.12:4226 , piece_id=HMUGW35FHAMUHRVKAIFMOR2PJHDFJILZZPWHRHN7252NZNTAGXKA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260150017 (23h4m1.299723137s) [noise=true , address=45.91.116.13:4155 , piece_id=MYOQX2PWNRR3DFBMJLIUQ45IZIMCL5Q3MOEZC72VPSG6MTDHJIRA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260179443 (23h4m1.299752563s) [noise=true , address=45.91.116.6:4974 , piece_id=H4W6QGMIHMT3FZ3BQ52ABQVFOLJ4TRJD3F3VBCLCE6VC6OGWSLSQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260415534 (23h4m1.299988654s) [noise=true , address=45.91.116.5:4299 , piece_id=2C24J3IYULB7NANTJ4EKWBIXVK2LU6ZH7TT4KVDFMBJ25LMYPHUQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260440222 (23h4m1.300013342s) [noise=true , address=77.90.31.16:4735 , piece_id=J36NH3XBSV2G4M263IK4KSKFQ37CU7FLUX3CTWXRXNKCMAVN5ULQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260672537 (23h4m1.300245677s) [noise=true , address=45.91.116.5:4298 , piece_id=35TNWDJSGJPFJLK7YHLPV7I7HOBUXVIBTF7MMN5KSDJXWUSTCCBA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260696543 (23h4m1.300269663s) [noise=true , address=45.91.116.11:4118 , piece_id=MZA3HUVQCAVINXEJ7MKPRWCBUEWGGQREVNPMXGAWLOPMDXJL52SQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260923076 (23h4m1.300496196s) [noise=true , address=45.91.116.8:4048 , piece_id=GLZM2GCPYEYOGFYZOVRUXASMAOZC6KEU5HJQDHFNNFD2GT2JR3IQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261018137 (23h4m1.300591257s) [noise=true , address=45.91.116.4:4937 , piece_id=XDDLQBKVLHHK3I4C6SCSRUG6CCL6CCRXC3XTOBKB2SUAV7MEF32Q, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261248188 (23h4m1.300821308s) [noise=true , address=77.90.31.18:4805 , piece_id=5HIXWZFYKCYCVJO7SCNCK4EGH5WLBD4IBOMLNPWLXNIFVKJO7V3Q, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261263337 (23h4m1.300836457s) [noise=true , address=45.91.116.6:4975 , piece_id=YNEVVYRYEBHPY33K3LSHPL7QQBLVN3A7MCTFDWC7KPC6MOYTXDMQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261474661 (23h4m1.301047781s) [noise=true , address=45.91.116.8:4045 , piece_id=A2WPFGBEILYT2GPJ2CIPXZJJ6DF7E25DPJBR7KGOXS2WNUQK5BGA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261495622 (23h4m1.301068732s) [noise=true , address=77.90.31.12:4590 , piece_id=H7EHZ76GBVNA7T5ECSBZ6EV6UPLMTU4XRQDILCILS6L3JQTCU3DQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261742434 (23h4m1.301315554s) [noise=true , address=77.90.31.21:4877 , piece_id=R3JXTAY54SDCPJTFCBOEQOZXSZINIAY624OSOFD3SMOIVXJHIWEQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261965330 (23h4m1.30153845s) [noise=true , address=45.91.116.4:4940 , piece_id=Z4A5UBCYQHLULLTWUTB66QGY5GXDRZKWA4XJHYBBJIDDTYAGY7CA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:27.083171850 (23h4m3.12274497s) [upload=20341]: Next returned nil segment Sep 7 12:32:27.083189514 (23h4m3.122762624s) [upload=20341]: Waiting for error group managing segments... Sep 7 12:32:27.397997027 (23h4m3.437570137s) [noise=true , address=77.90.31.21:4877 , piece_id=R3JXTAY54SDCPJTFCBOEQOZXSZINIAY624OSOFD3SMOIVXJHIWEQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.422075642 (23h4m3.461648762s) [noise=true , address=45.91.116.4:4940 , piece_id=Z4A5UBCYQHLULLTWUTB66QGY5GXDRZKWA4XJHYBBJIDDTYAGY7CA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.422196694 (23h4m3.461769804s) [noise=true , address=45.91.116.8:4048 , piece_id=GLZM2GCPYEYOGFYZOVRUXASMAOZC6KEU5HJQDHFNNFD2GT2JR3IQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.426177602 (23h4m3.465750722s) [noise=true , address=45.91.116.9:4263 , piece_id=VSYUTOSVLDLAOB74XZFESXWLL5EZN5Z25LYDH3WZQN5LMSDSGELQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.433234004 (23h4m3.472807124s) [noise=true , address=45.91.116.5:4298 , piece_id=35TNWDJSGJPFJLK7YHLPV7I7HOBUXVIBTF7MMN5KSDJXWUSTCCBA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.442931240 (23h4m3.48250436s) [noise=true , address=45.91.116.6:4975 , piece_id=YNEVVYRYEBHPY33K3LSHPL7QQBLVN3A7MCTFDWC7KPC6MOYTXDMQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.452017787 (23h4m3.491590907s) [noise=true , address=45.91.116.5:4299 , piece_id=2C24J3IYULB7NANTJ4EKWBIXVK2LU6ZH7TT4KVDFMBJ25LMYPHUQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.471965458 (23h4m3.511538568s) [noise=true , address=45.91.116.11:4118 , piece_id=MZA3HUVQCAVINXEJ7MKPRWCBUEWGGQREVNPMXGAWLOPMDXJL52SQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.523292133 (23h4m3.562865253s) [noise=true , address=45.91.116.13:4155 , piece_id=MYOQX2PWNRR3DFBMJLIUQ45IZIMCL5Q3MOEZC72VPSG6MTDHJIRA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.536854595 (23h4m3.576427715s) [noise=true , address=77.90.31.18:4805 , piece_id=5HIXWZFYKCYCVJO7SCNCK4EGH5WLBD4IBOMLNPWLXNIFVKJO7V3Q, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.563355926 (23h4m3.602929036s) [noise=true , address=45.91.116.12:4226 , piece_id=HMUGW35FHAMUHRVKAIFMOR2PJHDFJILZZPWHRHN7252NZNTAGXKA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.579270911 (23h4m3.618844031s) [noise=true , address=45.91.116.8:4045 , piece_id=A2WPFGBEILYT2GPJ2CIPXZJJ6DF7E25DPJBR7KGOXS2WNUQK5BGA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.596306278 (23h4m3.635879398s) [noise=true , address=77.90.31.12:4590 , piece_id=H7EHZ76GBVNA7T5ECSBZ6EV6UPLMTU4XRQDILCILS6L3JQTCU3DQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.596332468 (23h4m3.635905588s) [seg_pos={0 0}, upload=20341]: Segment reached optimal threshold of 13 pieces. Sep 7 12:32:27.596482004 (23h4m3.636055124s) [noise=true , address=77.90.31.16:4735 , piece_id=J36NH3XBSV2G4M263IK4KSKFQ37CU7FLUX3CTWXRXNKCMAVN5ULQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Sep 7 12:32:27.596621401 (23h4m3.636194511s) [noise=true , address=45.91.116.6:4974 , piece_id=H4W6QGMIHMT3FZ3BQ52ABQVFOLJ4TRJD3F3VBCLCE6VC6OGWSLSQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Sep 7 12:32:27.596667589 (23h4m3.636240709s) [noise=true , address=45.91.116.4:4937 , piece_id=XDDLQBKVLHHK3I4C6SCSRUG6CCL6CCRXC3XTOBKB2SUAV7MEF32Q, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection Sep 7 12:32:27.596683911 (23h4m3.636257031s) [seg_pos={0 0}, upload=20341]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:32:27.596751590 (23h4m3.63632471s) [upload=20341]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:32:27.596757722 (23h4m3.636330842s) [upload=20341]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:32:27.904840427 (23h4m3.944413547s) [upload=20341]: Done uploading segments... Sep 7 12:32:27.904858301 (23h4m3.944431421s) [upload=20341]: Upload finished. err: <nil> Duration: 23.52099524s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X...........X.....X.....................X..............] duration: 10.786594214s FAIL (2025-09-07 12:44:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:32:24.971479778 (23h4m1.011052898s) [upload=20341]: Starting upload Sep 7 12:32:24.971535354 (23h4m1.011108474s) [upload=20341]: Done starting upload Sep 7 12:32:24.971630667 (23h4m1.011203787s) [upload=20341]: Uploading segments... Sep 7 12:32:24.971695100 (23h4m1.01126822s) [upload=20341]: Got next segment. Inline: false Sep 7 12:32:24.971723614 (23h4m1.011296735s) [upload=20341]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:32:24.971728113 (23h4m1.011301233s) [upload=20341]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:32:25.259639680 (23h4m1.2992128s) [seg_pos={0 0}, upload=20341]: Begin upload segment... Sep 7 12:32:25.259682412 (23h4m1.299255532s) [seg_pos={0 0}, upload=20341]: Done begin upload segment. Sep 7 12:32:25.259731626 (23h4m1.299304756s) [noise=true , address=45.91.116.9:4263 , piece_id=VSYUTOSVLDLAOB74XZFESXWLL5EZN5Z25LYDH3WZQN5LMSDSGELQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.259743980 (23h4m1.2993171s) [noise=true , address=45.91.116.12:4226 , piece_id=HMUGW35FHAMUHRVKAIFMOR2PJHDFJILZZPWHRHN7252NZNTAGXKA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260150017 (23h4m1.299723137s) [noise=true , address=45.91.116.13:4155 , piece_id=MYOQX2PWNRR3DFBMJLIUQ45IZIMCL5Q3MOEZC72VPSG6MTDHJIRA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260179443 (23h4m1.299752563s) [noise=true , address=45.91.116.6:4974 , piece_id=H4W6QGMIHMT3FZ3BQ52ABQVFOLJ4TRJD3F3VBCLCE6VC6OGWSLSQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260415534 (23h4m1.299988654s) [noise=true , address=45.91.116.5:4299 , piece_id=2C24J3IYULB7NANTJ4EKWBIXVK2LU6ZH7TT4KVDFMBJ25LMYPHUQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260440222 (23h4m1.300013342s) [noise=true , address=77.90.31.16:4735 , piece_id=J36NH3XBSV2G4M263IK4KSKFQ37CU7FLUX3CTWXRXNKCMAVN5ULQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260672537 (23h4m1.300245677s) [noise=true , address=45.91.116.5:4298 , piece_id=35TNWDJSGJPFJLK7YHLPV7I7HOBUXVIBTF7MMN5KSDJXWUSTCCBA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260696543 (23h4m1.300269663s) [noise=true , address=45.91.116.11:4118 , piece_id=MZA3HUVQCAVINXEJ7MKPRWCBUEWGGQREVNPMXGAWLOPMDXJL52SQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260923076 (23h4m1.300496196s) [noise=true , address=45.91.116.8:4048 , piece_id=GLZM2GCPYEYOGFYZOVRUXASMAOZC6KEU5HJQDHFNNFD2GT2JR3IQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261018137 (23h4m1.300591257s) [noise=true , address=45.91.116.4:4937 , piece_id=XDDLQBKVLHHK3I4C6SCSRUG6CCL6CCRXC3XTOBKB2SUAV7MEF32Q, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261248188 (23h4m1.300821308s) [noise=true , address=77.90.31.18:4805 , piece_id=5HIXWZFYKCYCVJO7SCNCK4EGH5WLBD4IBOMLNPWLXNIFVKJO7V3Q, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261263337 (23h4m1.300836457s) [noise=true , address=45.91.116.6:4975 , piece_id=YNEVVYRYEBHPY33K3LSHPL7QQBLVN3A7MCTFDWC7KPC6MOYTXDMQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261474661 (23h4m1.301047781s) [noise=true , address=45.91.116.8:4045 , piece_id=A2WPFGBEILYT2GPJ2CIPXZJJ6DF7E25DPJBR7KGOXS2WNUQK5BGA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261495622 (23h4m1.301068732s) [noise=true , address=77.90.31.12:4590 , piece_id=H7EHZ76GBVNA7T5ECSBZ6EV6UPLMTU4XRQDILCILS6L3JQTCU3DQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261742434 (23h4m1.301315554s) [noise=true , address=77.90.31.21:4877 , piece_id=R3JXTAY54SDCPJTFCBOEQOZXSZINIAY624OSOFD3SMOIVXJHIWEQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261965330 (23h4m1.30153845s) [noise=true , address=45.91.116.4:4940 , piece_id=Z4A5UBCYQHLULLTWUTB66QGY5GXDRZKWA4XJHYBBJIDDTYAGY7CA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:27.083171850 (23h4m3.12274497s) [upload=20341]: Next returned nil segment Sep 7 12:32:27.083189514 (23h4m3.122762624s) [upload=20341]: Waiting for error group managing segments... Sep 7 12:32:27.397997027 (23h4m3.437570137s) [noise=true , address=77.90.31.21:4877 , piece_id=R3JXTAY54SDCPJTFCBOEQOZXSZINIAY624OSOFD3SMOIVXJHIWEQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.422075642 (23h4m3.461648762s) [noise=true , address=45.91.116.4:4940 , piece_id=Z4A5UBCYQHLULLTWUTB66QGY5GXDRZKWA4XJHYBBJIDDTYAGY7CA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.422196694 (23h4m3.461769804s) [noise=true , address=45.91.116.8:4048 , piece_id=GLZM2GCPYEYOGFYZOVRUXASMAOZC6KEU5HJQDHFNNFD2GT2JR3IQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.426177602 (23h4m3.465750722s) [noise=true , address=45.91.116.9:4263 , piece_id=VSYUTOSVLDLAOB74XZFESXWLL5EZN5Z25LYDH3WZQN5LMSDSGELQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.433234004 (23h4m3.472807124s) [noise=true , address=45.91.116.5:4298 , piece_id=35TNWDJSGJPFJLK7YHLPV7I7HOBUXVIBTF7MMN5KSDJXWUSTCCBA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.442931240 (23h4m3.48250436s) [noise=true , address=45.91.116.6:4975 , piece_id=YNEVVYRYEBHPY33K3LSHPL7QQBLVN3A7MCTFDWC7KPC6MOYTXDMQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.452017787 (23h4m3.491590907s) [noise=true , address=45.91.116.5:4299 , piece_id=2C24J3IYULB7NANTJ4EKWBIXVK2LU6ZH7TT4KVDFMBJ25LMYPHUQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.471965458 (23h4m3.511538568s) [noise=true , address=45.91.116.11:4118 , piece_id=MZA3HUVQCAVINXEJ7MKPRWCBUEWGGQREVNPMXGAWLOPMDXJL52SQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.523292133 (23h4m3.562865253s) [noise=true , address=45.91.116.13:4155 , piece_id=MYOQX2PWNRR3DFBMJLIUQ45IZIMCL5Q3MOEZC72VPSG6MTDHJIRA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.536854595 (23h4m3.576427715s) [noise=true , address=77.90.31.18:4805 , piece_id=5HIXWZFYKCYCVJO7SCNCK4EGH5WLBD4IBOMLNPWLXNIFVKJO7V3Q, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.563355926 (23h4m3.602929036s) [noise=true , address=45.91.116.12:4226 , piece_id=HMUGW35FHAMUHRVKAIFMOR2PJHDFJILZZPWHRHN7252NZNTAGXKA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.579270911 (23h4m3.618844031s) [noise=true , address=45.91.116.8:4045 , piece_id=A2WPFGBEILYT2GPJ2CIPXZJJ6DF7E25DPJBR7KGOXS2WNUQK5BGA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.596306278 (23h4m3.635879398s) [noise=true , address=77.90.31.12:4590 , piece_id=H7EHZ76GBVNA7T5ECSBZ6EV6UPLMTU4XRQDILCILS6L3JQTCU3DQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.596332468 (23h4m3.635905588s) [seg_pos={0 0}, upload=20341]: Segment reached optimal threshold of 13 pieces. Sep 7 12:32:27.596482004 (23h4m3.636055124s) [noise=true , address=77.90.31.16:4735 , piece_id=J36NH3XBSV2G4M263IK4KSKFQ37CU7FLUX3CTWXRXNKCMAVN5ULQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Sep 7 12:32:27.596621401 (23h4m3.636194511s) [noise=true , address=45.91.116.6:4974 , piece_id=H4W6QGMIHMT3FZ3BQ52ABQVFOLJ4TRJD3F3VBCLCE6VC6OGWSLSQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Sep 7 12:32:27.596667589 (23h4m3.636240709s) [noise=true , address=45.91.116.4:4937 , piece_id=XDDLQBKVLHHK3I4C6SCSRUG6CCL6CCRXC3XTOBKB2SUAV7MEF32Q, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection Sep 7 12:32:27.596683911 (23h4m3.636257031s) [seg_pos={0 0}, upload=20341]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:32:27.596751590 (23h4m3.63632471s) [upload=20341]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:32:27.596757722 (23h4m3.636330842s) [upload=20341]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:32:27.904840427 (23h4m3.944413547s) [upload=20341]: Done uploading segments... Sep 7 12:32:27.904858301 (23h4m3.944431421s) [upload=20341]: Upload finished. err: <nil> Duration: 10.786613781s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X...........X.....X.....................X.............] duration: 27.689488719s FAIL (2025-09-07 12:43:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:32:24.971479778 (23h4m1.011052898s) [upload=20341]: Starting upload Sep 7 12:32:24.971535354 (23h4m1.011108474s) [upload=20341]: Done starting upload Sep 7 12:32:24.971630667 (23h4m1.011203787s) [upload=20341]: Uploading segments... Sep 7 12:32:24.971695100 (23h4m1.01126822s) [upload=20341]: Got next segment. Inline: false Sep 7 12:32:24.971723614 (23h4m1.011296735s) [upload=20341]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:32:24.971728113 (23h4m1.011301233s) [upload=20341]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:32:25.259639680 (23h4m1.2992128s) [seg_pos={0 0}, upload=20341]: Begin upload segment... Sep 7 12:32:25.259682412 (23h4m1.299255532s) [seg_pos={0 0}, upload=20341]: Done begin upload segment. Sep 7 12:32:25.259731626 (23h4m1.299304756s) [noise=true , address=45.91.116.9:4263 , piece_id=VSYUTOSVLDLAOB74XZFESXWLL5EZN5Z25LYDH3WZQN5LMSDSGELQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.259743980 (23h4m1.2993171s) [noise=true , address=45.91.116.12:4226 , piece_id=HMUGW35FHAMUHRVKAIFMOR2PJHDFJILZZPWHRHN7252NZNTAGXKA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260150017 (23h4m1.299723137s) [noise=true , address=45.91.116.13:4155 , piece_id=MYOQX2PWNRR3DFBMJLIUQ45IZIMCL5Q3MOEZC72VPSG6MTDHJIRA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260179443 (23h4m1.299752563s) [noise=true , address=45.91.116.6:4974 , piece_id=H4W6QGMIHMT3FZ3BQ52ABQVFOLJ4TRJD3F3VBCLCE6VC6OGWSLSQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260415534 (23h4m1.299988654s) [noise=true , address=45.91.116.5:4299 , piece_id=2C24J3IYULB7NANTJ4EKWBIXVK2LU6ZH7TT4KVDFMBJ25LMYPHUQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260440222 (23h4m1.300013342s) [noise=true , address=77.90.31.16:4735 , piece_id=J36NH3XBSV2G4M263IK4KSKFQ37CU7FLUX3CTWXRXNKCMAVN5ULQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260672537 (23h4m1.300245677s) [noise=true , address=45.91.116.5:4298 , piece_id=35TNWDJSGJPFJLK7YHLPV7I7HOBUXVIBTF7MMN5KSDJXWUSTCCBA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260696543 (23h4m1.300269663s) [noise=true , address=45.91.116.11:4118 , piece_id=MZA3HUVQCAVINXEJ7MKPRWCBUEWGGQREVNPMXGAWLOPMDXJL52SQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260923076 (23h4m1.300496196s) [noise=true , address=45.91.116.8:4048 , piece_id=GLZM2GCPYEYOGFYZOVRUXASMAOZC6KEU5HJQDHFNNFD2GT2JR3IQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261018137 (23h4m1.300591257s) [noise=true , address=45.91.116.4:4937 , piece_id=XDDLQBKVLHHK3I4C6SCSRUG6CCL6CCRXC3XTOBKB2SUAV7MEF32Q, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261248188 (23h4m1.300821308s) [noise=true , address=77.90.31.18:4805 , piece_id=5HIXWZFYKCYCVJO7SCNCK4EGH5WLBD4IBOMLNPWLXNIFVKJO7V3Q, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261263337 (23h4m1.300836457s) [noise=true , address=45.91.116.6:4975 , piece_id=YNEVVYRYEBHPY33K3LSHPL7QQBLVN3A7MCTFDWC7KPC6MOYTXDMQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261474661 (23h4m1.301047781s) [noise=true , address=45.91.116.8:4045 , piece_id=A2WPFGBEILYT2GPJ2CIPXZJJ6DF7E25DPJBR7KGOXS2WNUQK5BGA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261495622 (23h4m1.301068732s) [noise=true , address=77.90.31.12:4590 , piece_id=H7EHZ76GBVNA7T5ECSBZ6EV6UPLMTU4XRQDILCILS6L3JQTCU3DQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261742434 (23h4m1.301315554s) [noise=true , address=77.90.31.21:4877 , piece_id=R3JXTAY54SDCPJTFCBOEQOZXSZINIAY624OSOFD3SMOIVXJHIWEQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261965330 (23h4m1.30153845s) [noise=true , address=45.91.116.4:4940 , piece_id=Z4A5UBCYQHLULLTWUTB66QGY5GXDRZKWA4XJHYBBJIDDTYAGY7CA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:27.083171850 (23h4m3.12274497s) [upload=20341]: Next returned nil segment Sep 7 12:32:27.083189514 (23h4m3.122762624s) [upload=20341]: Waiting for error group managing segments... Sep 7 12:32:27.397997027 (23h4m3.437570137s) [noise=true , address=77.90.31.21:4877 , piece_id=R3JXTAY54SDCPJTFCBOEQOZXSZINIAY624OSOFD3SMOIVXJHIWEQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.422075642 (23h4m3.461648762s) [noise=true , address=45.91.116.4:4940 , piece_id=Z4A5UBCYQHLULLTWUTB66QGY5GXDRZKWA4XJHYBBJIDDTYAGY7CA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.422196694 (23h4m3.461769804s) [noise=true , address=45.91.116.8:4048 , piece_id=GLZM2GCPYEYOGFYZOVRUXASMAOZC6KEU5HJQDHFNNFD2GT2JR3IQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.426177602 (23h4m3.465750722s) [noise=true , address=45.91.116.9:4263 , piece_id=VSYUTOSVLDLAOB74XZFESXWLL5EZN5Z25LYDH3WZQN5LMSDSGELQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.433234004 (23h4m3.472807124s) [noise=true , address=45.91.116.5:4298 , piece_id=35TNWDJSGJPFJLK7YHLPV7I7HOBUXVIBTF7MMN5KSDJXWUSTCCBA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.442931240 (23h4m3.48250436s) [noise=true , address=45.91.116.6:4975 , piece_id=YNEVVYRYEBHPY33K3LSHPL7QQBLVN3A7MCTFDWC7KPC6MOYTXDMQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.452017787 (23h4m3.491590907s) [noise=true , address=45.91.116.5:4299 , piece_id=2C24J3IYULB7NANTJ4EKWBIXVK2LU6ZH7TT4KVDFMBJ25LMYPHUQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.471965458 (23h4m3.511538568s) [noise=true , address=45.91.116.11:4118 , piece_id=MZA3HUVQCAVINXEJ7MKPRWCBUEWGGQREVNPMXGAWLOPMDXJL52SQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.523292133 (23h4m3.562865253s) [noise=true , address=45.91.116.13:4155 , piece_id=MYOQX2PWNRR3DFBMJLIUQ45IZIMCL5Q3MOEZC72VPSG6MTDHJIRA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.536854595 (23h4m3.576427715s) [noise=true , address=77.90.31.18:4805 , piece_id=5HIXWZFYKCYCVJO7SCNCK4EGH5WLBD4IBOMLNPWLXNIFVKJO7V3Q, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.563355926 (23h4m3.602929036s) [noise=true , address=45.91.116.12:4226 , piece_id=HMUGW35FHAMUHRVKAIFMOR2PJHDFJILZZPWHRHN7252NZNTAGXKA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.579270911 (23h4m3.618844031s) [noise=true , address=45.91.116.8:4045 , piece_id=A2WPFGBEILYT2GPJ2CIPXZJJ6DF7E25DPJBR7KGOXS2WNUQK5BGA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.596306278 (23h4m3.635879398s) [noise=true , address=77.90.31.12:4590 , piece_id=H7EHZ76GBVNA7T5ECSBZ6EV6UPLMTU4XRQDILCILS6L3JQTCU3DQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.596332468 (23h4m3.635905588s) [seg_pos={0 0}, upload=20341]: Segment reached optimal threshold of 13 pieces. Sep 7 12:32:27.596482004 (23h4m3.636055124s) [noise=true , address=77.90.31.16:4735 , piece_id=J36NH3XBSV2G4M263IK4KSKFQ37CU7FLUX3CTWXRXNKCMAVN5ULQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Sep 7 12:32:27.596621401 (23h4m3.636194511s) [noise=true , address=45.91.116.6:4974 , piece_id=H4W6QGMIHMT3FZ3BQ52ABQVFOLJ4TRJD3F3VBCLCE6VC6OGWSLSQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Sep 7 12:32:27.596667589 (23h4m3.636240709s) [noise=true , address=45.91.116.4:4937 , piece_id=XDDLQBKVLHHK3I4C6SCSRUG6CCL6CCRXC3XTOBKB2SUAV7MEF32Q, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection Sep 7 12:32:27.596683911 (23h4m3.636257031s) [seg_pos={0 0}, upload=20341]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:32:27.596751590 (23h4m3.63632471s) [upload=20341]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:32:27.596757722 (23h4m3.636330842s) [upload=20341]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:32:27.904840427 (23h4m3.944413547s) [upload=20341]: Done uploading segments... Sep 7 12:32:27.904858301 (23h4m3.944431421s) [upload=20341]: Upload finished. err: <nil> Duration: 27.689527663s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X...........X.....X.....................X............] duration: 13.690932786s FAIL (2025-09-07 12:42:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:32:24.971479778 (23h4m1.011052898s) [upload=20341]: Starting upload Sep 7 12:32:24.971535354 (23h4m1.011108474s) [upload=20341]: Done starting upload Sep 7 12:32:24.971630667 (23h4m1.011203787s) [upload=20341]: Uploading segments... Sep 7 12:32:24.971695100 (23h4m1.01126822s) [upload=20341]: Got next segment. Inline: false Sep 7 12:32:24.971723614 (23h4m1.011296735s) [upload=20341]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:32:24.971728113 (23h4m1.011301233s) [upload=20341]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:32:25.259639680 (23h4m1.2992128s) [seg_pos={0 0}, upload=20341]: Begin upload segment... Sep 7 12:32:25.259682412 (23h4m1.299255532s) [seg_pos={0 0}, upload=20341]: Done begin upload segment. Sep 7 12:32:25.259731626 (23h4m1.299304756s) [noise=true , address=45.91.116.9:4263 , piece_id=VSYUTOSVLDLAOB74XZFESXWLL5EZN5Z25LYDH3WZQN5LMSDSGELQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.259743980 (23h4m1.2993171s) [noise=true , address=45.91.116.12:4226 , piece_id=HMUGW35FHAMUHRVKAIFMOR2PJHDFJILZZPWHRHN7252NZNTAGXKA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260150017 (23h4m1.299723137s) [noise=true , address=45.91.116.13:4155 , piece_id=MYOQX2PWNRR3DFBMJLIUQ45IZIMCL5Q3MOEZC72VPSG6MTDHJIRA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260179443 (23h4m1.299752563s) [noise=true , address=45.91.116.6:4974 , piece_id=H4W6QGMIHMT3FZ3BQ52ABQVFOLJ4TRJD3F3VBCLCE6VC6OGWSLSQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260415534 (23h4m1.299988654s) [noise=true , address=45.91.116.5:4299 , piece_id=2C24J3IYULB7NANTJ4EKWBIXVK2LU6ZH7TT4KVDFMBJ25LMYPHUQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260440222 (23h4m1.300013342s) [noise=true , address=77.90.31.16:4735 , piece_id=J36NH3XBSV2G4M263IK4KSKFQ37CU7FLUX3CTWXRXNKCMAVN5ULQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260672537 (23h4m1.300245677s) [noise=true , address=45.91.116.5:4298 , piece_id=35TNWDJSGJPFJLK7YHLPV7I7HOBUXVIBTF7MMN5KSDJXWUSTCCBA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260696543 (23h4m1.300269663s) [noise=true , address=45.91.116.11:4118 , piece_id=MZA3HUVQCAVINXEJ7MKPRWCBUEWGGQREVNPMXGAWLOPMDXJL52SQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260923076 (23h4m1.300496196s) [noise=true , address=45.91.116.8:4048 , piece_id=GLZM2GCPYEYOGFYZOVRUXASMAOZC6KEU5HJQDHFNNFD2GT2JR3IQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261018137 (23h4m1.300591257s) [noise=true , address=45.91.116.4:4937 , piece_id=XDDLQBKVLHHK3I4C6SCSRUG6CCL6CCRXC3XTOBKB2SUAV7MEF32Q, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261248188 (23h4m1.300821308s) [noise=true , address=77.90.31.18:4805 , piece_id=5HIXWZFYKCYCVJO7SCNCK4EGH5WLBD4IBOMLNPWLXNIFVKJO7V3Q, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261263337 (23h4m1.300836457s) [noise=true , address=45.91.116.6:4975 , piece_id=YNEVVYRYEBHPY33K3LSHPL7QQBLVN3A7MCTFDWC7KPC6MOYTXDMQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261474661 (23h4m1.301047781s) [noise=true , address=45.91.116.8:4045 , piece_id=A2WPFGBEILYT2GPJ2CIPXZJJ6DF7E25DPJBR7KGOXS2WNUQK5BGA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261495622 (23h4m1.301068732s) [noise=true , address=77.90.31.12:4590 , piece_id=H7EHZ76GBVNA7T5ECSBZ6EV6UPLMTU4XRQDILCILS6L3JQTCU3DQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261742434 (23h4m1.301315554s) [noise=true , address=77.90.31.21:4877 , piece_id=R3JXTAY54SDCPJTFCBOEQOZXSZINIAY624OSOFD3SMOIVXJHIWEQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261965330 (23h4m1.30153845s) [noise=true , address=45.91.116.4:4940 , piece_id=Z4A5UBCYQHLULLTWUTB66QGY5GXDRZKWA4XJHYBBJIDDTYAGY7CA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:27.083171850 (23h4m3.12274497s) [upload=20341]: Next returned nil segment Sep 7 12:32:27.083189514 (23h4m3.122762624s) [upload=20341]: Waiting for error group managing segments... Sep 7 12:32:27.397997027 (23h4m3.437570137s) [noise=true , address=77.90.31.21:4877 , piece_id=R3JXTAY54SDCPJTFCBOEQOZXSZINIAY624OSOFD3SMOIVXJHIWEQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.422075642 (23h4m3.461648762s) [noise=true , address=45.91.116.4:4940 , piece_id=Z4A5UBCYQHLULLTWUTB66QGY5GXDRZKWA4XJHYBBJIDDTYAGY7CA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.422196694 (23h4m3.461769804s) [noise=true , address=45.91.116.8:4048 , piece_id=GLZM2GCPYEYOGFYZOVRUXASMAOZC6KEU5HJQDHFNNFD2GT2JR3IQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.426177602 (23h4m3.465750722s) [noise=true , address=45.91.116.9:4263 , piece_id=VSYUTOSVLDLAOB74XZFESXWLL5EZN5Z25LYDH3WZQN5LMSDSGELQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.433234004 (23h4m3.472807124s) [noise=true , address=45.91.116.5:4298 , piece_id=35TNWDJSGJPFJLK7YHLPV7I7HOBUXVIBTF7MMN5KSDJXWUSTCCBA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.442931240 (23h4m3.48250436s) [noise=true , address=45.91.116.6:4975 , piece_id=YNEVVYRYEBHPY33K3LSHPL7QQBLVN3A7MCTFDWC7KPC6MOYTXDMQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.452017787 (23h4m3.491590907s) [noise=true , address=45.91.116.5:4299 , piece_id=2C24J3IYULB7NANTJ4EKWBIXVK2LU6ZH7TT4KVDFMBJ25LMYPHUQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.471965458 (23h4m3.511538568s) [noise=true , address=45.91.116.11:4118 , piece_id=MZA3HUVQCAVINXEJ7MKPRWCBUEWGGQREVNPMXGAWLOPMDXJL52SQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.523292133 (23h4m3.562865253s) [noise=true , address=45.91.116.13:4155 , piece_id=MYOQX2PWNRR3DFBMJLIUQ45IZIMCL5Q3MOEZC72VPSG6MTDHJIRA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.536854595 (23h4m3.576427715s) [noise=true , address=77.90.31.18:4805 , piece_id=5HIXWZFYKCYCVJO7SCNCK4EGH5WLBD4IBOMLNPWLXNIFVKJO7V3Q, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.563355926 (23h4m3.602929036s) [noise=true , address=45.91.116.12:4226 , piece_id=HMUGW35FHAMUHRVKAIFMOR2PJHDFJILZZPWHRHN7252NZNTAGXKA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.579270911 (23h4m3.618844031s) [noise=true , address=45.91.116.8:4045 , piece_id=A2WPFGBEILYT2GPJ2CIPXZJJ6DF7E25DPJBR7KGOXS2WNUQK5BGA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.596306278 (23h4m3.635879398s) [noise=true , address=77.90.31.12:4590 , piece_id=H7EHZ76GBVNA7T5ECSBZ6EV6UPLMTU4XRQDILCILS6L3JQTCU3DQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.596332468 (23h4m3.635905588s) [seg_pos={0 0}, upload=20341]: Segment reached optimal threshold of 13 pieces. Sep 7 12:32:27.596482004 (23h4m3.636055124s) [noise=true , address=77.90.31.16:4735 , piece_id=J36NH3XBSV2G4M263IK4KSKFQ37CU7FLUX3CTWXRXNKCMAVN5ULQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Sep 7 12:32:27.596621401 (23h4m3.636194511s) [noise=true , address=45.91.116.6:4974 , piece_id=H4W6QGMIHMT3FZ3BQ52ABQVFOLJ4TRJD3F3VBCLCE6VC6OGWSLSQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Sep 7 12:32:27.596667589 (23h4m3.636240709s) [noise=true , address=45.91.116.4:4937 , piece_id=XDDLQBKVLHHK3I4C6SCSRUG6CCL6CCRXC3XTOBKB2SUAV7MEF32Q, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection Sep 7 12:32:27.596683911 (23h4m3.636257031s) [seg_pos={0 0}, upload=20341]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:32:27.596751590 (23h4m3.63632471s) [upload=20341]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:32:27.596757722 (23h4m3.636330842s) [upload=20341]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:32:27.904840427 (23h4m3.944413547s) [upload=20341]: Done uploading segments... Sep 7 12:32:27.904858301 (23h4m3.944431421s) [upload=20341]: Upload finished. err: <nil> Duration: 13.690959637s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X...........X.....X.....................X...........] duration: 7.154534452s FAIL (2025-09-07 12:41:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:32:24.971479778 (23h4m1.011052898s) [upload=20341]: Starting upload Sep 7 12:32:24.971535354 (23h4m1.011108474s) [upload=20341]: Done starting upload Sep 7 12:32:24.971630667 (23h4m1.011203787s) [upload=20341]: Uploading segments... Sep 7 12:32:24.971695100 (23h4m1.01126822s) [upload=20341]: Got next segment. Inline: false Sep 7 12:32:24.971723614 (23h4m1.011296735s) [upload=20341]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:32:24.971728113 (23h4m1.011301233s) [upload=20341]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:32:25.259639680 (23h4m1.2992128s) [seg_pos={0 0}, upload=20341]: Begin upload segment... Sep 7 12:32:25.259682412 (23h4m1.299255532s) [seg_pos={0 0}, upload=20341]: Done begin upload segment. Sep 7 12:32:25.259731626 (23h4m1.299304756s) [noise=true , address=45.91.116.9:4263 , piece_id=VSYUTOSVLDLAOB74XZFESXWLL5EZN5Z25LYDH3WZQN5LMSDSGELQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.259743980 (23h4m1.2993171s) [noise=true , address=45.91.116.12:4226 , piece_id=HMUGW35FHAMUHRVKAIFMOR2PJHDFJILZZPWHRHN7252NZNTAGXKA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260150017 (23h4m1.299723137s) [noise=true , address=45.91.116.13:4155 , piece_id=MYOQX2PWNRR3DFBMJLIUQ45IZIMCL5Q3MOEZC72VPSG6MTDHJIRA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260179443 (23h4m1.299752563s) [noise=true , address=45.91.116.6:4974 , piece_id=H4W6QGMIHMT3FZ3BQ52ABQVFOLJ4TRJD3F3VBCLCE6VC6OGWSLSQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260415534 (23h4m1.299988654s) [noise=true , address=45.91.116.5:4299 , piece_id=2C24J3IYULB7NANTJ4EKWBIXVK2LU6ZH7TT4KVDFMBJ25LMYPHUQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260440222 (23h4m1.300013342s) [noise=true , address=77.90.31.16:4735 , piece_id=J36NH3XBSV2G4M263IK4KSKFQ37CU7FLUX3CTWXRXNKCMAVN5ULQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260672537 (23h4m1.300245677s) [noise=true , address=45.91.116.5:4298 , piece_id=35TNWDJSGJPFJLK7YHLPV7I7HOBUXVIBTF7MMN5KSDJXWUSTCCBA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260696543 (23h4m1.300269663s) [noise=true , address=45.91.116.11:4118 , piece_id=MZA3HUVQCAVINXEJ7MKPRWCBUEWGGQREVNPMXGAWLOPMDXJL52SQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260923076 (23h4m1.300496196s) [noise=true , address=45.91.116.8:4048 , piece_id=GLZM2GCPYEYOGFYZOVRUXASMAOZC6KEU5HJQDHFNNFD2GT2JR3IQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261018137 (23h4m1.300591257s) [noise=true , address=45.91.116.4:4937 , piece_id=XDDLQBKVLHHK3I4C6SCSRUG6CCL6CCRXC3XTOBKB2SUAV7MEF32Q, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261248188 (23h4m1.300821308s) [noise=true , address=77.90.31.18:4805 , piece_id=5HIXWZFYKCYCVJO7SCNCK4EGH5WLBD4IBOMLNPWLXNIFVKJO7V3Q, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261263337 (23h4m1.300836457s) [noise=true , address=45.91.116.6:4975 , piece_id=YNEVVYRYEBHPY33K3LSHPL7QQBLVN3A7MCTFDWC7KPC6MOYTXDMQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261474661 (23h4m1.301047781s) [noise=true , address=45.91.116.8:4045 , piece_id=A2WPFGBEILYT2GPJ2CIPXZJJ6DF7E25DPJBR7KGOXS2WNUQK5BGA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261495622 (23h4m1.301068732s) [noise=true , address=77.90.31.12:4590 , piece_id=H7EHZ76GBVNA7T5ECSBZ6EV6UPLMTU4XRQDILCILS6L3JQTCU3DQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261742434 (23h4m1.301315554s) [noise=true , address=77.90.31.21:4877 , piece_id=R3JXTAY54SDCPJTFCBOEQOZXSZINIAY624OSOFD3SMOIVXJHIWEQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261965330 (23h4m1.30153845s) [noise=true , address=45.91.116.4:4940 , piece_id=Z4A5UBCYQHLULLTWUTB66QGY5GXDRZKWA4XJHYBBJIDDTYAGY7CA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:27.083171850 (23h4m3.12274497s) [upload=20341]: Next returned nil segment Sep 7 12:32:27.083189514 (23h4m3.122762624s) [upload=20341]: Waiting for error group managing segments... Sep 7 12:32:27.397997027 (23h4m3.437570137s) [noise=true , address=77.90.31.21:4877 , piece_id=R3JXTAY54SDCPJTFCBOEQOZXSZINIAY624OSOFD3SMOIVXJHIWEQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.422075642 (23h4m3.461648762s) [noise=true , address=45.91.116.4:4940 , piece_id=Z4A5UBCYQHLULLTWUTB66QGY5GXDRZKWA4XJHYBBJIDDTYAGY7CA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.422196694 (23h4m3.461769804s) [noise=true , address=45.91.116.8:4048 , piece_id=GLZM2GCPYEYOGFYZOVRUXASMAOZC6KEU5HJQDHFNNFD2GT2JR3IQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.426177602 (23h4m3.465750722s) [noise=true , address=45.91.116.9:4263 , piece_id=VSYUTOSVLDLAOB74XZFESXWLL5EZN5Z25LYDH3WZQN5LMSDSGELQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.433234004 (23h4m3.472807124s) [noise=true , address=45.91.116.5:4298 , piece_id=35TNWDJSGJPFJLK7YHLPV7I7HOBUXVIBTF7MMN5KSDJXWUSTCCBA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.442931240 (23h4m3.48250436s) [noise=true , address=45.91.116.6:4975 , piece_id=YNEVVYRYEBHPY33K3LSHPL7QQBLVN3A7MCTFDWC7KPC6MOYTXDMQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.452017787 (23h4m3.491590907s) [noise=true , address=45.91.116.5:4299 , piece_id=2C24J3IYULB7NANTJ4EKWBIXVK2LU6ZH7TT4KVDFMBJ25LMYPHUQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.471965458 (23h4m3.511538568s) [noise=true , address=45.91.116.11:4118 , piece_id=MZA3HUVQCAVINXEJ7MKPRWCBUEWGGQREVNPMXGAWLOPMDXJL52SQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.523292133 (23h4m3.562865253s) [noise=true , address=45.91.116.13:4155 , piece_id=MYOQX2PWNRR3DFBMJLIUQ45IZIMCL5Q3MOEZC72VPSG6MTDHJIRA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.536854595 (23h4m3.576427715s) [noise=true , address=77.90.31.18:4805 , piece_id=5HIXWZFYKCYCVJO7SCNCK4EGH5WLBD4IBOMLNPWLXNIFVKJO7V3Q, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.563355926 (23h4m3.602929036s) [noise=true , address=45.91.116.12:4226 , piece_id=HMUGW35FHAMUHRVKAIFMOR2PJHDFJILZZPWHRHN7252NZNTAGXKA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.579270911 (23h4m3.618844031s) [noise=true , address=45.91.116.8:4045 , piece_id=A2WPFGBEILYT2GPJ2CIPXZJJ6DF7E25DPJBR7KGOXS2WNUQK5BGA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.596306278 (23h4m3.635879398s) [noise=true , address=77.90.31.12:4590 , piece_id=H7EHZ76GBVNA7T5ECSBZ6EV6UPLMTU4XRQDILCILS6L3JQTCU3DQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.596332468 (23h4m3.635905588s) [seg_pos={0 0}, upload=20341]: Segment reached optimal threshold of 13 pieces. Sep 7 12:32:27.596482004 (23h4m3.636055124s) [noise=true , address=77.90.31.16:4735 , piece_id=J36NH3XBSV2G4M263IK4KSKFQ37CU7FLUX3CTWXRXNKCMAVN5ULQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Sep 7 12:32:27.596621401 (23h4m3.636194511s) [noise=true , address=45.91.116.6:4974 , piece_id=H4W6QGMIHMT3FZ3BQ52ABQVFOLJ4TRJD3F3VBCLCE6VC6OGWSLSQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Sep 7 12:32:27.596667589 (23h4m3.636240709s) [noise=true , address=45.91.116.4:4937 , piece_id=XDDLQBKVLHHK3I4C6SCSRUG6CCL6CCRXC3XTOBKB2SUAV7MEF32Q, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection Sep 7 12:32:27.596683911 (23h4m3.636257031s) [seg_pos={0 0}, upload=20341]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:32:27.596751590 (23h4m3.63632471s) [upload=20341]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:32:27.596757722 (23h4m3.636330842s) [upload=20341]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:32:27.904840427 (23h4m3.944413547s) [upload=20341]: Done uploading segments... Sep 7 12:32:27.904858301 (23h4m3.944431421s) [upload=20341]: Upload finished. err: <nil> Duration: 7.154563697s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X...........X.....X.....................X..........] duration: 6.421198204s FAIL (2025-09-07 12:40:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:32:24.971479778 (23h4m1.011052898s) [upload=20341]: Starting upload Sep 7 12:32:24.971535354 (23h4m1.011108474s) [upload=20341]: Done starting upload Sep 7 12:32:24.971630667 (23h4m1.011203787s) [upload=20341]: Uploading segments... Sep 7 12:32:24.971695100 (23h4m1.01126822s) [upload=20341]: Got next segment. Inline: false Sep 7 12:32:24.971723614 (23h4m1.011296735s) [upload=20341]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:32:24.971728113 (23h4m1.011301233s) [upload=20341]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:32:25.259639680 (23h4m1.2992128s) [seg_pos={0 0}, upload=20341]: Begin upload segment... Sep 7 12:32:25.259682412 (23h4m1.299255532s) [seg_pos={0 0}, upload=20341]: Done begin upload segment. Sep 7 12:32:25.259731626 (23h4m1.299304756s) [noise=true , address=45.91.116.9:4263 , piece_id=VSYUTOSVLDLAOB74XZFESXWLL5EZN5Z25LYDH3WZQN5LMSDSGELQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.259743980 (23h4m1.2993171s) [noise=true , address=45.91.116.12:4226 , piece_id=HMUGW35FHAMUHRVKAIFMOR2PJHDFJILZZPWHRHN7252NZNTAGXKA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260150017 (23h4m1.299723137s) [noise=true , address=45.91.116.13:4155 , piece_id=MYOQX2PWNRR3DFBMJLIUQ45IZIMCL5Q3MOEZC72VPSG6MTDHJIRA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260179443 (23h4m1.299752563s) [noise=true , address=45.91.116.6:4974 , piece_id=H4W6QGMIHMT3FZ3BQ52ABQVFOLJ4TRJD3F3VBCLCE6VC6OGWSLSQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260415534 (23h4m1.299988654s) [noise=true , address=45.91.116.5:4299 , piece_id=2C24J3IYULB7NANTJ4EKWBIXVK2LU6ZH7TT4KVDFMBJ25LMYPHUQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260440222 (23h4m1.300013342s) [noise=true , address=77.90.31.16:4735 , piece_id=J36NH3XBSV2G4M263IK4KSKFQ37CU7FLUX3CTWXRXNKCMAVN5ULQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260672537 (23h4m1.300245677s) [noise=true , address=45.91.116.5:4298 , piece_id=35TNWDJSGJPFJLK7YHLPV7I7HOBUXVIBTF7MMN5KSDJXWUSTCCBA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260696543 (23h4m1.300269663s) [noise=true , address=45.91.116.11:4118 , piece_id=MZA3HUVQCAVINXEJ7MKPRWCBUEWGGQREVNPMXGAWLOPMDXJL52SQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260923076 (23h4m1.300496196s) [noise=true , address=45.91.116.8:4048 , piece_id=GLZM2GCPYEYOGFYZOVRUXASMAOZC6KEU5HJQDHFNNFD2GT2JR3IQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261018137 (23h4m1.300591257s) [noise=true , address=45.91.116.4:4937 , piece_id=XDDLQBKVLHHK3I4C6SCSRUG6CCL6CCRXC3XTOBKB2SUAV7MEF32Q, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261248188 (23h4m1.300821308s) [noise=true , address=77.90.31.18:4805 , piece_id=5HIXWZFYKCYCVJO7SCNCK4EGH5WLBD4IBOMLNPWLXNIFVKJO7V3Q, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261263337 (23h4m1.300836457s) [noise=true , address=45.91.116.6:4975 , piece_id=YNEVVYRYEBHPY33K3LSHPL7QQBLVN3A7MCTFDWC7KPC6MOYTXDMQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261474661 (23h4m1.301047781s) [noise=true , address=45.91.116.8:4045 , piece_id=A2WPFGBEILYT2GPJ2CIPXZJJ6DF7E25DPJBR7KGOXS2WNUQK5BGA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261495622 (23h4m1.301068732s) [noise=true , address=77.90.31.12:4590 , piece_id=H7EHZ76GBVNA7T5ECSBZ6EV6UPLMTU4XRQDILCILS6L3JQTCU3DQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261742434 (23h4m1.301315554s) [noise=true , address=77.90.31.21:4877 , piece_id=R3JXTAY54SDCPJTFCBOEQOZXSZINIAY624OSOFD3SMOIVXJHIWEQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261965330 (23h4m1.30153845s) [noise=true , address=45.91.116.4:4940 , piece_id=Z4A5UBCYQHLULLTWUTB66QGY5GXDRZKWA4XJHYBBJIDDTYAGY7CA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:27.083171850 (23h4m3.12274497s) [upload=20341]: Next returned nil segment Sep 7 12:32:27.083189514 (23h4m3.122762624s) [upload=20341]: Waiting for error group managing segments... Sep 7 12:32:27.397997027 (23h4m3.437570137s) [noise=true , address=77.90.31.21:4877 , piece_id=R3JXTAY54SDCPJTFCBOEQOZXSZINIAY624OSOFD3SMOIVXJHIWEQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.422075642 (23h4m3.461648762s) [noise=true , address=45.91.116.4:4940 , piece_id=Z4A5UBCYQHLULLTWUTB66QGY5GXDRZKWA4XJHYBBJIDDTYAGY7CA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.422196694 (23h4m3.461769804s) [noise=true , address=45.91.116.8:4048 , piece_id=GLZM2GCPYEYOGFYZOVRUXASMAOZC6KEU5HJQDHFNNFD2GT2JR3IQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.426177602 (23h4m3.465750722s) [noise=true , address=45.91.116.9:4263 , piece_id=VSYUTOSVLDLAOB74XZFESXWLL5EZN5Z25LYDH3WZQN5LMSDSGELQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.433234004 (23h4m3.472807124s) [noise=true , address=45.91.116.5:4298 , piece_id=35TNWDJSGJPFJLK7YHLPV7I7HOBUXVIBTF7MMN5KSDJXWUSTCCBA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.442931240 (23h4m3.48250436s) [noise=true , address=45.91.116.6:4975 , piece_id=YNEVVYRYEBHPY33K3LSHPL7QQBLVN3A7MCTFDWC7KPC6MOYTXDMQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.452017787 (23h4m3.491590907s) [noise=true , address=45.91.116.5:4299 , piece_id=2C24J3IYULB7NANTJ4EKWBIXVK2LU6ZH7TT4KVDFMBJ25LMYPHUQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.471965458 (23h4m3.511538568s) [noise=true , address=45.91.116.11:4118 , piece_id=MZA3HUVQCAVINXEJ7MKPRWCBUEWGGQREVNPMXGAWLOPMDXJL52SQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.523292133 (23h4m3.562865253s) [noise=true , address=45.91.116.13:4155 , piece_id=MYOQX2PWNRR3DFBMJLIUQ45IZIMCL5Q3MOEZC72VPSG6MTDHJIRA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.536854595 (23h4m3.576427715s) [noise=true , address=77.90.31.18:4805 , piece_id=5HIXWZFYKCYCVJO7SCNCK4EGH5WLBD4IBOMLNPWLXNIFVKJO7V3Q, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.563355926 (23h4m3.602929036s) [noise=true , address=45.91.116.12:4226 , piece_id=HMUGW35FHAMUHRVKAIFMOR2PJHDFJILZZPWHRHN7252NZNTAGXKA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.579270911 (23h4m3.618844031s) [noise=true , address=45.91.116.8:4045 , piece_id=A2WPFGBEILYT2GPJ2CIPXZJJ6DF7E25DPJBR7KGOXS2WNUQK5BGA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.596306278 (23h4m3.635879398s) [noise=true , address=77.90.31.12:4590 , piece_id=H7EHZ76GBVNA7T5ECSBZ6EV6UPLMTU4XRQDILCILS6L3JQTCU3DQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.596332468 (23h4m3.635905588s) [seg_pos={0 0}, upload=20341]: Segment reached optimal threshold of 13 pieces. Sep 7 12:32:27.596482004 (23h4m3.636055124s) [noise=true , address=77.90.31.16:4735 , piece_id=J36NH3XBSV2G4M263IK4KSKFQ37CU7FLUX3CTWXRXNKCMAVN5ULQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Sep 7 12:32:27.596621401 (23h4m3.636194511s) [noise=true , address=45.91.116.6:4974 , piece_id=H4W6QGMIHMT3FZ3BQ52ABQVFOLJ4TRJD3F3VBCLCE6VC6OGWSLSQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Sep 7 12:32:27.596667589 (23h4m3.636240709s) [noise=true , address=45.91.116.4:4937 , piece_id=XDDLQBKVLHHK3I4C6SCSRUG6CCL6CCRXC3XTOBKB2SUAV7MEF32Q, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection Sep 7 12:32:27.596683911 (23h4m3.636257031s) [seg_pos={0 0}, upload=20341]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:32:27.596751590 (23h4m3.63632471s) [upload=20341]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:32:27.596757722 (23h4m3.636330842s) [upload=20341]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:32:27.904840427 (23h4m3.944413547s) [upload=20341]: Done uploading segments... Sep 7 12:32:27.904858301 (23h4m3.944431421s) [upload=20341]: Upload finished. err: <nil> Duration: 6.42124331s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X...........X.....X.....................X.........] duration: 13.623656961s FAIL (2025-09-07 12:39:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:32:24.971479778 (23h4m1.011052898s) [upload=20341]: Starting upload Sep 7 12:32:24.971535354 (23h4m1.011108474s) [upload=20341]: Done starting upload Sep 7 12:32:24.971630667 (23h4m1.011203787s) [upload=20341]: Uploading segments... Sep 7 12:32:24.971695100 (23h4m1.01126822s) [upload=20341]: Got next segment. Inline: false Sep 7 12:32:24.971723614 (23h4m1.011296735s) [upload=20341]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:32:24.971728113 (23h4m1.011301233s) [upload=20341]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:32:25.259639680 (23h4m1.2992128s) [seg_pos={0 0}, upload=20341]: Begin upload segment... Sep 7 12:32:25.259682412 (23h4m1.299255532s) [seg_pos={0 0}, upload=20341]: Done begin upload segment. Sep 7 12:32:25.259731626 (23h4m1.299304756s) [noise=true , address=45.91.116.9:4263 , piece_id=VSYUTOSVLDLAOB74XZFESXWLL5EZN5Z25LYDH3WZQN5LMSDSGELQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.259743980 (23h4m1.2993171s) [noise=true , address=45.91.116.12:4226 , piece_id=HMUGW35FHAMUHRVKAIFMOR2PJHDFJILZZPWHRHN7252NZNTAGXKA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260150017 (23h4m1.299723137s) [noise=true , address=45.91.116.13:4155 , piece_id=MYOQX2PWNRR3DFBMJLIUQ45IZIMCL5Q3MOEZC72VPSG6MTDHJIRA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260179443 (23h4m1.299752563s) [noise=true , address=45.91.116.6:4974 , piece_id=H4W6QGMIHMT3FZ3BQ52ABQVFOLJ4TRJD3F3VBCLCE6VC6OGWSLSQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260415534 (23h4m1.299988654s) [noise=true , address=45.91.116.5:4299 , piece_id=2C24J3IYULB7NANTJ4EKWBIXVK2LU6ZH7TT4KVDFMBJ25LMYPHUQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260440222 (23h4m1.300013342s) [noise=true , address=77.90.31.16:4735 , piece_id=J36NH3XBSV2G4M263IK4KSKFQ37CU7FLUX3CTWXRXNKCMAVN5ULQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260672537 (23h4m1.300245677s) [noise=true , address=45.91.116.5:4298 , piece_id=35TNWDJSGJPFJLK7YHLPV7I7HOBUXVIBTF7MMN5KSDJXWUSTCCBA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260696543 (23h4m1.300269663s) [noise=true , address=45.91.116.11:4118 , piece_id=MZA3HUVQCAVINXEJ7MKPRWCBUEWGGQREVNPMXGAWLOPMDXJL52SQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260923076 (23h4m1.300496196s) [noise=true , address=45.91.116.8:4048 , piece_id=GLZM2GCPYEYOGFYZOVRUXASMAOZC6KEU5HJQDHFNNFD2GT2JR3IQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261018137 (23h4m1.300591257s) [noise=true , address=45.91.116.4:4937 , piece_id=XDDLQBKVLHHK3I4C6SCSRUG6CCL6CCRXC3XTOBKB2SUAV7MEF32Q, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261248188 (23h4m1.300821308s) [noise=true , address=77.90.31.18:4805 , piece_id=5HIXWZFYKCYCVJO7SCNCK4EGH5WLBD4IBOMLNPWLXNIFVKJO7V3Q, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261263337 (23h4m1.300836457s) [noise=true , address=45.91.116.6:4975 , piece_id=YNEVVYRYEBHPY33K3LSHPL7QQBLVN3A7MCTFDWC7KPC6MOYTXDMQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261474661 (23h4m1.301047781s) [noise=true , address=45.91.116.8:4045 , piece_id=A2WPFGBEILYT2GPJ2CIPXZJJ6DF7E25DPJBR7KGOXS2WNUQK5BGA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261495622 (23h4m1.301068732s) [noise=true , address=77.90.31.12:4590 , piece_id=H7EHZ76GBVNA7T5ECSBZ6EV6UPLMTU4XRQDILCILS6L3JQTCU3DQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261742434 (23h4m1.301315554s) [noise=true , address=77.90.31.21:4877 , piece_id=R3JXTAY54SDCPJTFCBOEQOZXSZINIAY624OSOFD3SMOIVXJHIWEQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261965330 (23h4m1.30153845s) [noise=true , address=45.91.116.4:4940 , piece_id=Z4A5UBCYQHLULLTWUTB66QGY5GXDRZKWA4XJHYBBJIDDTYAGY7CA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:27.083171850 (23h4m3.12274497s) [upload=20341]: Next returned nil segment Sep 7 12:32:27.083189514 (23h4m3.122762624s) [upload=20341]: Waiting for error group managing segments... Sep 7 12:32:27.397997027 (23h4m3.437570137s) [noise=true , address=77.90.31.21:4877 , piece_id=R3JXTAY54SDCPJTFCBOEQOZXSZINIAY624OSOFD3SMOIVXJHIWEQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.422075642 (23h4m3.461648762s) [noise=true , address=45.91.116.4:4940 , piece_id=Z4A5UBCYQHLULLTWUTB66QGY5GXDRZKWA4XJHYBBJIDDTYAGY7CA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.422196694 (23h4m3.461769804s) [noise=true , address=45.91.116.8:4048 , piece_id=GLZM2GCPYEYOGFYZOVRUXASMAOZC6KEU5HJQDHFNNFD2GT2JR3IQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.426177602 (23h4m3.465750722s) [noise=true , address=45.91.116.9:4263 , piece_id=VSYUTOSVLDLAOB74XZFESXWLL5EZN5Z25LYDH3WZQN5LMSDSGELQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.433234004 (23h4m3.472807124s) [noise=true , address=45.91.116.5:4298 , piece_id=35TNWDJSGJPFJLK7YHLPV7I7HOBUXVIBTF7MMN5KSDJXWUSTCCBA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.442931240 (23h4m3.48250436s) [noise=true , address=45.91.116.6:4975 , piece_id=YNEVVYRYEBHPY33K3LSHPL7QQBLVN3A7MCTFDWC7KPC6MOYTXDMQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.452017787 (23h4m3.491590907s) [noise=true , address=45.91.116.5:4299 , piece_id=2C24J3IYULB7NANTJ4EKWBIXVK2LU6ZH7TT4KVDFMBJ25LMYPHUQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.471965458 (23h4m3.511538568s) [noise=true , address=45.91.116.11:4118 , piece_id=MZA3HUVQCAVINXEJ7MKPRWCBUEWGGQREVNPMXGAWLOPMDXJL52SQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.523292133 (23h4m3.562865253s) [noise=true , address=45.91.116.13:4155 , piece_id=MYOQX2PWNRR3DFBMJLIUQ45IZIMCL5Q3MOEZC72VPSG6MTDHJIRA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.536854595 (23h4m3.576427715s) [noise=true , address=77.90.31.18:4805 , piece_id=5HIXWZFYKCYCVJO7SCNCK4EGH5WLBD4IBOMLNPWLXNIFVKJO7V3Q, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.563355926 (23h4m3.602929036s) [noise=true , address=45.91.116.12:4226 , piece_id=HMUGW35FHAMUHRVKAIFMOR2PJHDFJILZZPWHRHN7252NZNTAGXKA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.579270911 (23h4m3.618844031s) [noise=true , address=45.91.116.8:4045 , piece_id=A2WPFGBEILYT2GPJ2CIPXZJJ6DF7E25DPJBR7KGOXS2WNUQK5BGA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.596306278 (23h4m3.635879398s) [noise=true , address=77.90.31.12:4590 , piece_id=H7EHZ76GBVNA7T5ECSBZ6EV6UPLMTU4XRQDILCILS6L3JQTCU3DQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.596332468 (23h4m3.635905588s) [seg_pos={0 0}, upload=20341]: Segment reached optimal threshold of 13 pieces. Sep 7 12:32:27.596482004 (23h4m3.636055124s) [noise=true , address=77.90.31.16:4735 , piece_id=J36NH3XBSV2G4M263IK4KSKFQ37CU7FLUX3CTWXRXNKCMAVN5ULQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Sep 7 12:32:27.596621401 (23h4m3.636194511s) [noise=true , address=45.91.116.6:4974 , piece_id=H4W6QGMIHMT3FZ3BQ52ABQVFOLJ4TRJD3F3VBCLCE6VC6OGWSLSQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Sep 7 12:32:27.596667589 (23h4m3.636240709s) [noise=true , address=45.91.116.4:4937 , piece_id=XDDLQBKVLHHK3I4C6SCSRUG6CCL6CCRXC3XTOBKB2SUAV7MEF32Q, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection Sep 7 12:32:27.596683911 (23h4m3.636257031s) [seg_pos={0 0}, upload=20341]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:32:27.596751590 (23h4m3.63632471s) [upload=20341]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:32:27.596757722 (23h4m3.636330842s) [upload=20341]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:32:27.904840427 (23h4m3.944413547s) [upload=20341]: Done uploading segments... Sep 7 12:32:27.904858301 (23h4m3.944431421s) [upload=20341]: Upload finished. err: <nil> Duration: 13.623684914s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X...........X.....X.....................X........] duration: 6.83811228s FAIL (2025-09-07 12:38:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:32:24.971479778 (23h4m1.011052898s) [upload=20341]: Starting upload Sep 7 12:32:24.971535354 (23h4m1.011108474s) [upload=20341]: Done starting upload Sep 7 12:32:24.971630667 (23h4m1.011203787s) [upload=20341]: Uploading segments... Sep 7 12:32:24.971695100 (23h4m1.01126822s) [upload=20341]: Got next segment. Inline: false Sep 7 12:32:24.971723614 (23h4m1.011296735s) [upload=20341]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:32:24.971728113 (23h4m1.011301233s) [upload=20341]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:32:25.259639680 (23h4m1.2992128s) [seg_pos={0 0}, upload=20341]: Begin upload segment... Sep 7 12:32:25.259682412 (23h4m1.299255532s) [seg_pos={0 0}, upload=20341]: Done begin upload segment. Sep 7 12:32:25.259731626 (23h4m1.299304756s) [noise=true , address=45.91.116.9:4263 , piece_id=VSYUTOSVLDLAOB74XZFESXWLL5EZN5Z25LYDH3WZQN5LMSDSGELQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.259743980 (23h4m1.2993171s) [noise=true , address=45.91.116.12:4226 , piece_id=HMUGW35FHAMUHRVKAIFMOR2PJHDFJILZZPWHRHN7252NZNTAGXKA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260150017 (23h4m1.299723137s) [noise=true , address=45.91.116.13:4155 , piece_id=MYOQX2PWNRR3DFBMJLIUQ45IZIMCL5Q3MOEZC72VPSG6MTDHJIRA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260179443 (23h4m1.299752563s) [noise=true , address=45.91.116.6:4974 , piece_id=H4W6QGMIHMT3FZ3BQ52ABQVFOLJ4TRJD3F3VBCLCE6VC6OGWSLSQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260415534 (23h4m1.299988654s) [noise=true , address=45.91.116.5:4299 , piece_id=2C24J3IYULB7NANTJ4EKWBIXVK2LU6ZH7TT4KVDFMBJ25LMYPHUQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260440222 (23h4m1.300013342s) [noise=true , address=77.90.31.16:4735 , piece_id=J36NH3XBSV2G4M263IK4KSKFQ37CU7FLUX3CTWXRXNKCMAVN5ULQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260672537 (23h4m1.300245677s) [noise=true , address=45.91.116.5:4298 , piece_id=35TNWDJSGJPFJLK7YHLPV7I7HOBUXVIBTF7MMN5KSDJXWUSTCCBA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260696543 (23h4m1.300269663s) [noise=true , address=45.91.116.11:4118 , piece_id=MZA3HUVQCAVINXEJ7MKPRWCBUEWGGQREVNPMXGAWLOPMDXJL52SQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260923076 (23h4m1.300496196s) [noise=true , address=45.91.116.8:4048 , piece_id=GLZM2GCPYEYOGFYZOVRUXASMAOZC6KEU5HJQDHFNNFD2GT2JR3IQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261018137 (23h4m1.300591257s) [noise=true , address=45.91.116.4:4937 , piece_id=XDDLQBKVLHHK3I4C6SCSRUG6CCL6CCRXC3XTOBKB2SUAV7MEF32Q, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261248188 (23h4m1.300821308s) [noise=true , address=77.90.31.18:4805 , piece_id=5HIXWZFYKCYCVJO7SCNCK4EGH5WLBD4IBOMLNPWLXNIFVKJO7V3Q, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261263337 (23h4m1.300836457s) [noise=true , address=45.91.116.6:4975 , piece_id=YNEVVYRYEBHPY33K3LSHPL7QQBLVN3A7MCTFDWC7KPC6MOYTXDMQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261474661 (23h4m1.301047781s) [noise=true , address=45.91.116.8:4045 , piece_id=A2WPFGBEILYT2GPJ2CIPXZJJ6DF7E25DPJBR7KGOXS2WNUQK5BGA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261495622 (23h4m1.301068732s) [noise=true , address=77.90.31.12:4590 , piece_id=H7EHZ76GBVNA7T5ECSBZ6EV6UPLMTU4XRQDILCILS6L3JQTCU3DQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261742434 (23h4m1.301315554s) [noise=true , address=77.90.31.21:4877 , piece_id=R3JXTAY54SDCPJTFCBOEQOZXSZINIAY624OSOFD3SMOIVXJHIWEQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261965330 (23h4m1.30153845s) [noise=true , address=45.91.116.4:4940 , piece_id=Z4A5UBCYQHLULLTWUTB66QGY5GXDRZKWA4XJHYBBJIDDTYAGY7CA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:27.083171850 (23h4m3.12274497s) [upload=20341]: Next returned nil segment Sep 7 12:32:27.083189514 (23h4m3.122762624s) [upload=20341]: Waiting for error group managing segments... Sep 7 12:32:27.397997027 (23h4m3.437570137s) [noise=true , address=77.90.31.21:4877 , piece_id=R3JXTAY54SDCPJTFCBOEQOZXSZINIAY624OSOFD3SMOIVXJHIWEQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.422075642 (23h4m3.461648762s) [noise=true , address=45.91.116.4:4940 , piece_id=Z4A5UBCYQHLULLTWUTB66QGY5GXDRZKWA4XJHYBBJIDDTYAGY7CA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.422196694 (23h4m3.461769804s) [noise=true , address=45.91.116.8:4048 , piece_id=GLZM2GCPYEYOGFYZOVRUXASMAOZC6KEU5HJQDHFNNFD2GT2JR3IQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.426177602 (23h4m3.465750722s) [noise=true , address=45.91.116.9:4263 , piece_id=VSYUTOSVLDLAOB74XZFESXWLL5EZN5Z25LYDH3WZQN5LMSDSGELQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.433234004 (23h4m3.472807124s) [noise=true , address=45.91.116.5:4298 , piece_id=35TNWDJSGJPFJLK7YHLPV7I7HOBUXVIBTF7MMN5KSDJXWUSTCCBA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.442931240 (23h4m3.48250436s) [noise=true , address=45.91.116.6:4975 , piece_id=YNEVVYRYEBHPY33K3LSHPL7QQBLVN3A7MCTFDWC7KPC6MOYTXDMQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.452017787 (23h4m3.491590907s) [noise=true , address=45.91.116.5:4299 , piece_id=2C24J3IYULB7NANTJ4EKWBIXVK2LU6ZH7TT4KVDFMBJ25LMYPHUQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.471965458 (23h4m3.511538568s) [noise=true , address=45.91.116.11:4118 , piece_id=MZA3HUVQCAVINXEJ7MKPRWCBUEWGGQREVNPMXGAWLOPMDXJL52SQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.523292133 (23h4m3.562865253s) [noise=true , address=45.91.116.13:4155 , piece_id=MYOQX2PWNRR3DFBMJLIUQ45IZIMCL5Q3MOEZC72VPSG6MTDHJIRA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.536854595 (23h4m3.576427715s) [noise=true , address=77.90.31.18:4805 , piece_id=5HIXWZFYKCYCVJO7SCNCK4EGH5WLBD4IBOMLNPWLXNIFVKJO7V3Q, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.563355926 (23h4m3.602929036s) [noise=true , address=45.91.116.12:4226 , piece_id=HMUGW35FHAMUHRVKAIFMOR2PJHDFJILZZPWHRHN7252NZNTAGXKA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.579270911 (23h4m3.618844031s) [noise=true , address=45.91.116.8:4045 , piece_id=A2WPFGBEILYT2GPJ2CIPXZJJ6DF7E25DPJBR7KGOXS2WNUQK5BGA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.596306278 (23h4m3.635879398s) [noise=true , address=77.90.31.12:4590 , piece_id=H7EHZ76GBVNA7T5ECSBZ6EV6UPLMTU4XRQDILCILS6L3JQTCU3DQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.596332468 (23h4m3.635905588s) [seg_pos={0 0}, upload=20341]: Segment reached optimal threshold of 13 pieces. Sep 7 12:32:27.596482004 (23h4m3.636055124s) [noise=true , address=77.90.31.16:4735 , piece_id=J36NH3XBSV2G4M263IK4KSKFQ37CU7FLUX3CTWXRXNKCMAVN5ULQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Sep 7 12:32:27.596621401 (23h4m3.636194511s) [noise=true , address=45.91.116.6:4974 , piece_id=H4W6QGMIHMT3FZ3BQ52ABQVFOLJ4TRJD3F3VBCLCE6VC6OGWSLSQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Sep 7 12:32:27.596667589 (23h4m3.636240709s) [noise=true , address=45.91.116.4:4937 , piece_id=XDDLQBKVLHHK3I4C6SCSRUG6CCL6CCRXC3XTOBKB2SUAV7MEF32Q, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection Sep 7 12:32:27.596683911 (23h4m3.636257031s) [seg_pos={0 0}, upload=20341]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:32:27.596751590 (23h4m3.63632471s) [upload=20341]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:32:27.596757722 (23h4m3.636330842s) [upload=20341]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:32:27.904840427 (23h4m3.944413547s) [upload=20341]: Done uploading segments... Sep 7 12:32:27.904858301 (23h4m3.944431421s) [upload=20341]: Upload finished. err: <nil> Duration: 6.838141906s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X...........X.....X.....................X.......] duration: 10.512553566s FAIL (2025-09-07 12:38:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:32:24.971479778 (23h4m1.011052898s) [upload=20341]: Starting upload Sep 7 12:32:24.971535354 (23h4m1.011108474s) [upload=20341]: Done starting upload Sep 7 12:32:24.971630667 (23h4m1.011203787s) [upload=20341]: Uploading segments... Sep 7 12:32:24.971695100 (23h4m1.01126822s) [upload=20341]: Got next segment. Inline: false Sep 7 12:32:24.971723614 (23h4m1.011296735s) [upload=20341]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:32:24.971728113 (23h4m1.011301233s) [upload=20341]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:32:25.259639680 (23h4m1.2992128s) [seg_pos={0 0}, upload=20341]: Begin upload segment... Sep 7 12:32:25.259682412 (23h4m1.299255532s) [seg_pos={0 0}, upload=20341]: Done begin upload segment. Sep 7 12:32:25.259731626 (23h4m1.299304756s) [noise=true , address=45.91.116.9:4263 , piece_id=VSYUTOSVLDLAOB74XZFESXWLL5EZN5Z25LYDH3WZQN5LMSDSGELQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.259743980 (23h4m1.2993171s) [noise=true , address=45.91.116.12:4226 , piece_id=HMUGW35FHAMUHRVKAIFMOR2PJHDFJILZZPWHRHN7252NZNTAGXKA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260150017 (23h4m1.299723137s) [noise=true , address=45.91.116.13:4155 , piece_id=MYOQX2PWNRR3DFBMJLIUQ45IZIMCL5Q3MOEZC72VPSG6MTDHJIRA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260179443 (23h4m1.299752563s) [noise=true , address=45.91.116.6:4974 , piece_id=H4W6QGMIHMT3FZ3BQ52ABQVFOLJ4TRJD3F3VBCLCE6VC6OGWSLSQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260415534 (23h4m1.299988654s) [noise=true , address=45.91.116.5:4299 , piece_id=2C24J3IYULB7NANTJ4EKWBIXVK2LU6ZH7TT4KVDFMBJ25LMYPHUQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260440222 (23h4m1.300013342s) [noise=true , address=77.90.31.16:4735 , piece_id=J36NH3XBSV2G4M263IK4KSKFQ37CU7FLUX3CTWXRXNKCMAVN5ULQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260672537 (23h4m1.300245677s) [noise=true , address=45.91.116.5:4298 , piece_id=35TNWDJSGJPFJLK7YHLPV7I7HOBUXVIBTF7MMN5KSDJXWUSTCCBA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260696543 (23h4m1.300269663s) [noise=true , address=45.91.116.11:4118 , piece_id=MZA3HUVQCAVINXEJ7MKPRWCBUEWGGQREVNPMXGAWLOPMDXJL52SQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260923076 (23h4m1.300496196s) [noise=true , address=45.91.116.8:4048 , piece_id=GLZM2GCPYEYOGFYZOVRUXASMAOZC6KEU5HJQDHFNNFD2GT2JR3IQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261018137 (23h4m1.300591257s) [noise=true , address=45.91.116.4:4937 , piece_id=XDDLQBKVLHHK3I4C6SCSRUG6CCL6CCRXC3XTOBKB2SUAV7MEF32Q, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261248188 (23h4m1.300821308s) [noise=true , address=77.90.31.18:4805 , piece_id=5HIXWZFYKCYCVJO7SCNCK4EGH5WLBD4IBOMLNPWLXNIFVKJO7V3Q, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261263337 (23h4m1.300836457s) [noise=true , address=45.91.116.6:4975 , piece_id=YNEVVYRYEBHPY33K3LSHPL7QQBLVN3A7MCTFDWC7KPC6MOYTXDMQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261474661 (23h4m1.301047781s) [noise=true , address=45.91.116.8:4045 , piece_id=A2WPFGBEILYT2GPJ2CIPXZJJ6DF7E25DPJBR7KGOXS2WNUQK5BGA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261495622 (23h4m1.301068732s) [noise=true , address=77.90.31.12:4590 , piece_id=H7EHZ76GBVNA7T5ECSBZ6EV6UPLMTU4XRQDILCILS6L3JQTCU3DQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261742434 (23h4m1.301315554s) [noise=true , address=77.90.31.21:4877 , piece_id=R3JXTAY54SDCPJTFCBOEQOZXSZINIAY624OSOFD3SMOIVXJHIWEQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261965330 (23h4m1.30153845s) [noise=true , address=45.91.116.4:4940 , piece_id=Z4A5UBCYQHLULLTWUTB66QGY5GXDRZKWA4XJHYBBJIDDTYAGY7CA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:27.083171850 (23h4m3.12274497s) [upload=20341]: Next returned nil segment Sep 7 12:32:27.083189514 (23h4m3.122762624s) [upload=20341]: Waiting for error group managing segments... Sep 7 12:32:27.397997027 (23h4m3.437570137s) [noise=true , address=77.90.31.21:4877 , piece_id=R3JXTAY54SDCPJTFCBOEQOZXSZINIAY624OSOFD3SMOIVXJHIWEQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.422075642 (23h4m3.461648762s) [noise=true , address=45.91.116.4:4940 , piece_id=Z4A5UBCYQHLULLTWUTB66QGY5GXDRZKWA4XJHYBBJIDDTYAGY7CA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.422196694 (23h4m3.461769804s) [noise=true , address=45.91.116.8:4048 , piece_id=GLZM2GCPYEYOGFYZOVRUXASMAOZC6KEU5HJQDHFNNFD2GT2JR3IQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.426177602 (23h4m3.465750722s) [noise=true , address=45.91.116.9:4263 , piece_id=VSYUTOSVLDLAOB74XZFESXWLL5EZN5Z25LYDH3WZQN5LMSDSGELQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.433234004 (23h4m3.472807124s) [noise=true , address=45.91.116.5:4298 , piece_id=35TNWDJSGJPFJLK7YHLPV7I7HOBUXVIBTF7MMN5KSDJXWUSTCCBA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.442931240 (23h4m3.48250436s) [noise=true , address=45.91.116.6:4975 , piece_id=YNEVVYRYEBHPY33K3LSHPL7QQBLVN3A7MCTFDWC7KPC6MOYTXDMQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.452017787 (23h4m3.491590907s) [noise=true , address=45.91.116.5:4299 , piece_id=2C24J3IYULB7NANTJ4EKWBIXVK2LU6ZH7TT4KVDFMBJ25LMYPHUQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.471965458 (23h4m3.511538568s) [noise=true , address=45.91.116.11:4118 , piece_id=MZA3HUVQCAVINXEJ7MKPRWCBUEWGGQREVNPMXGAWLOPMDXJL52SQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.523292133 (23h4m3.562865253s) [noise=true , address=45.91.116.13:4155 , piece_id=MYOQX2PWNRR3DFBMJLIUQ45IZIMCL5Q3MOEZC72VPSG6MTDHJIRA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.536854595 (23h4m3.576427715s) [noise=true , address=77.90.31.18:4805 , piece_id=5HIXWZFYKCYCVJO7SCNCK4EGH5WLBD4IBOMLNPWLXNIFVKJO7V3Q, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.563355926 (23h4m3.602929036s) [noise=true , address=45.91.116.12:4226 , piece_id=HMUGW35FHAMUHRVKAIFMOR2PJHDFJILZZPWHRHN7252NZNTAGXKA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.579270911 (23h4m3.618844031s) [noise=true , address=45.91.116.8:4045 , piece_id=A2WPFGBEILYT2GPJ2CIPXZJJ6DF7E25DPJBR7KGOXS2WNUQK5BGA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.596306278 (23h4m3.635879398s) [noise=true , address=77.90.31.12:4590 , piece_id=H7EHZ76GBVNA7T5ECSBZ6EV6UPLMTU4XRQDILCILS6L3JQTCU3DQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.596332468 (23h4m3.635905588s) [seg_pos={0 0}, upload=20341]: Segment reached optimal threshold of 13 pieces. Sep 7 12:32:27.596482004 (23h4m3.636055124s) [noise=true , address=77.90.31.16:4735 , piece_id=J36NH3XBSV2G4M263IK4KSKFQ37CU7FLUX3CTWXRXNKCMAVN5ULQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Sep 7 12:32:27.596621401 (23h4m3.636194511s) [noise=true , address=45.91.116.6:4974 , piece_id=H4W6QGMIHMT3FZ3BQ52ABQVFOLJ4TRJD3F3VBCLCE6VC6OGWSLSQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Sep 7 12:32:27.596667589 (23h4m3.636240709s) [noise=true , address=45.91.116.4:4937 , piece_id=XDDLQBKVLHHK3I4C6SCSRUG6CCL6CCRXC3XTOBKB2SUAV7MEF32Q, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection Sep 7 12:32:27.596683911 (23h4m3.636257031s) [seg_pos={0 0}, upload=20341]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:32:27.596751590 (23h4m3.63632471s) [upload=20341]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:32:27.596757722 (23h4m3.636330842s) [upload=20341]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:32:27.904840427 (23h4m3.944413547s) [upload=20341]: Done uploading segments... Sep 7 12:32:27.904858301 (23h4m3.944431421s) [upload=20341]: Upload finished. err: <nil> Duration: 10.512582572s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............X...........X.....X.....................X......] duration: 7.212954322s FAIL (2025-09-07 12:37:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:32:24.971479778 (23h4m1.011052898s) [upload=20341]: Starting upload Sep 7 12:32:24.971535354 (23h4m1.011108474s) [upload=20341]: Done starting upload Sep 7 12:32:24.971630667 (23h4m1.011203787s) [upload=20341]: Uploading segments... Sep 7 12:32:24.971695100 (23h4m1.01126822s) [upload=20341]: Got next segment. Inline: false Sep 7 12:32:24.971723614 (23h4m1.011296735s) [upload=20341]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:32:24.971728113 (23h4m1.011301233s) [upload=20341]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:32:25.259639680 (23h4m1.2992128s) [seg_pos={0 0}, upload=20341]: Begin upload segment... Sep 7 12:32:25.259682412 (23h4m1.299255532s) [seg_pos={0 0}, upload=20341]: Done begin upload segment. Sep 7 12:32:25.259731626 (23h4m1.299304756s) [noise=true , address=45.91.116.9:4263 , piece_id=VSYUTOSVLDLAOB74XZFESXWLL5EZN5Z25LYDH3WZQN5LMSDSGELQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.259743980 (23h4m1.2993171s) [noise=true , address=45.91.116.12:4226 , piece_id=HMUGW35FHAMUHRVKAIFMOR2PJHDFJILZZPWHRHN7252NZNTAGXKA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260150017 (23h4m1.299723137s) [noise=true , address=45.91.116.13:4155 , piece_id=MYOQX2PWNRR3DFBMJLIUQ45IZIMCL5Q3MOEZC72VPSG6MTDHJIRA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260179443 (23h4m1.299752563s) [noise=true , address=45.91.116.6:4974 , piece_id=H4W6QGMIHMT3FZ3BQ52ABQVFOLJ4TRJD3F3VBCLCE6VC6OGWSLSQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260415534 (23h4m1.299988654s) [noise=true , address=45.91.116.5:4299 , piece_id=2C24J3IYULB7NANTJ4EKWBIXVK2LU6ZH7TT4KVDFMBJ25LMYPHUQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260440222 (23h4m1.300013342s) [noise=true , address=77.90.31.16:4735 , piece_id=J36NH3XBSV2G4M263IK4KSKFQ37CU7FLUX3CTWXRXNKCMAVN5ULQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260672537 (23h4m1.300245677s) [noise=true , address=45.91.116.5:4298 , piece_id=35TNWDJSGJPFJLK7YHLPV7I7HOBUXVIBTF7MMN5KSDJXWUSTCCBA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260696543 (23h4m1.300269663s) [noise=true , address=45.91.116.11:4118 , piece_id=MZA3HUVQCAVINXEJ7MKPRWCBUEWGGQREVNPMXGAWLOPMDXJL52SQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260923076 (23h4m1.300496196s) [noise=true , address=45.91.116.8:4048 , piece_id=GLZM2GCPYEYOGFYZOVRUXASMAOZC6KEU5HJQDHFNNFD2GT2JR3IQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261018137 (23h4m1.300591257s) [noise=true , address=45.91.116.4:4937 , piece_id=XDDLQBKVLHHK3I4C6SCSRUG6CCL6CCRXC3XTOBKB2SUAV7MEF32Q, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261248188 (23h4m1.300821308s) [noise=true , address=77.90.31.18:4805 , piece_id=5HIXWZFYKCYCVJO7SCNCK4EGH5WLBD4IBOMLNPWLXNIFVKJO7V3Q, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261263337 (23h4m1.300836457s) [noise=true , address=45.91.116.6:4975 , piece_id=YNEVVYRYEBHPY33K3LSHPL7QQBLVN3A7MCTFDWC7KPC6MOYTXDMQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261474661 (23h4m1.301047781s) [noise=true , address=45.91.116.8:4045 , piece_id=A2WPFGBEILYT2GPJ2CIPXZJJ6DF7E25DPJBR7KGOXS2WNUQK5BGA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261495622 (23h4m1.301068732s) [noise=true , address=77.90.31.12:4590 , piece_id=H7EHZ76GBVNA7T5ECSBZ6EV6UPLMTU4XRQDILCILS6L3JQTCU3DQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261742434 (23h4m1.301315554s) [noise=true , address=77.90.31.21:4877 , piece_id=R3JXTAY54SDCPJTFCBOEQOZXSZINIAY624OSOFD3SMOIVXJHIWEQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261965330 (23h4m1.30153845s) [noise=true , address=45.91.116.4:4940 , piece_id=Z4A5UBCYQHLULLTWUTB66QGY5GXDRZKWA4XJHYBBJIDDTYAGY7CA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:27.083171850 (23h4m3.12274497s) [upload=20341]: Next returned nil segment Sep 7 12:32:27.083189514 (23h4m3.122762624s) [upload=20341]: Waiting for error group managing segments... Sep 7 12:32:27.397997027 (23h4m3.437570137s) [noise=true , address=77.90.31.21:4877 , piece_id=R3JXTAY54SDCPJTFCBOEQOZXSZINIAY624OSOFD3SMOIVXJHIWEQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.422075642 (23h4m3.461648762s) [noise=true , address=45.91.116.4:4940 , piece_id=Z4A5UBCYQHLULLTWUTB66QGY5GXDRZKWA4XJHYBBJIDDTYAGY7CA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.422196694 (23h4m3.461769804s) [noise=true , address=45.91.116.8:4048 , piece_id=GLZM2GCPYEYOGFYZOVRUXASMAOZC6KEU5HJQDHFNNFD2GT2JR3IQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.426177602 (23h4m3.465750722s) [noise=true , address=45.91.116.9:4263 , piece_id=VSYUTOSVLDLAOB74XZFESXWLL5EZN5Z25LYDH3WZQN5LMSDSGELQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.433234004 (23h4m3.472807124s) [noise=true , address=45.91.116.5:4298 , piece_id=35TNWDJSGJPFJLK7YHLPV7I7HOBUXVIBTF7MMN5KSDJXWUSTCCBA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.442931240 (23h4m3.48250436s) [noise=true , address=45.91.116.6:4975 , piece_id=YNEVVYRYEBHPY33K3LSHPL7QQBLVN3A7MCTFDWC7KPC6MOYTXDMQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.452017787 (23h4m3.491590907s) [noise=true , address=45.91.116.5:4299 , piece_id=2C24J3IYULB7NANTJ4EKWBIXVK2LU6ZH7TT4KVDFMBJ25LMYPHUQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.471965458 (23h4m3.511538568s) [noise=true , address=45.91.116.11:4118 , piece_id=MZA3HUVQCAVINXEJ7MKPRWCBUEWGGQREVNPMXGAWLOPMDXJL52SQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.523292133 (23h4m3.562865253s) [noise=true , address=45.91.116.13:4155 , piece_id=MYOQX2PWNRR3DFBMJLIUQ45IZIMCL5Q3MOEZC72VPSG6MTDHJIRA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.536854595 (23h4m3.576427715s) [noise=true , address=77.90.31.18:4805 , piece_id=5HIXWZFYKCYCVJO7SCNCK4EGH5WLBD4IBOMLNPWLXNIFVKJO7V3Q, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.563355926 (23h4m3.602929036s) [noise=true , address=45.91.116.12:4226 , piece_id=HMUGW35FHAMUHRVKAIFMOR2PJHDFJILZZPWHRHN7252NZNTAGXKA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.579270911 (23h4m3.618844031s) [noise=true , address=45.91.116.8:4045 , piece_id=A2WPFGBEILYT2GPJ2CIPXZJJ6DF7E25DPJBR7KGOXS2WNUQK5BGA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.596306278 (23h4m3.635879398s) [noise=true , address=77.90.31.12:4590 , piece_id=H7EHZ76GBVNA7T5ECSBZ6EV6UPLMTU4XRQDILCILS6L3JQTCU3DQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.596332468 (23h4m3.635905588s) [seg_pos={0 0}, upload=20341]: Segment reached optimal threshold of 13 pieces. Sep 7 12:32:27.596482004 (23h4m3.636055124s) [noise=true , address=77.90.31.16:4735 , piece_id=J36NH3XBSV2G4M263IK4KSKFQ37CU7FLUX3CTWXRXNKCMAVN5ULQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Sep 7 12:32:27.596621401 (23h4m3.636194511s) [noise=true , address=45.91.116.6:4974 , piece_id=H4W6QGMIHMT3FZ3BQ52ABQVFOLJ4TRJD3F3VBCLCE6VC6OGWSLSQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Sep 7 12:32:27.596667589 (23h4m3.636240709s) [noise=true , address=45.91.116.4:4937 , piece_id=XDDLQBKVLHHK3I4C6SCSRUG6CCL6CCRXC3XTOBKB2SUAV7MEF32Q, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection Sep 7 12:32:27.596683911 (23h4m3.636257031s) [seg_pos={0 0}, upload=20341]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:32:27.596751590 (23h4m3.63632471s) [upload=20341]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:32:27.596757722 (23h4m3.636330842s) [upload=20341]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:32:27.904840427 (23h4m3.944413547s) [upload=20341]: Done uploading segments... Sep 7 12:32:27.904858301 (23h4m3.944431421s) [upload=20341]: Upload finished. err: <nil> Duration: 7.212978949s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............X...........X.....X.....................X.....] duration: 17.302583892s FAIL (2025-09-07 12:36:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:32:24.971479778 (23h4m1.011052898s) [upload=20341]: Starting upload Sep 7 12:32:24.971535354 (23h4m1.011108474s) [upload=20341]: Done starting upload Sep 7 12:32:24.971630667 (23h4m1.011203787s) [upload=20341]: Uploading segments... Sep 7 12:32:24.971695100 (23h4m1.01126822s) [upload=20341]: Got next segment. Inline: false Sep 7 12:32:24.971723614 (23h4m1.011296735s) [upload=20341]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:32:24.971728113 (23h4m1.011301233s) [upload=20341]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:32:25.259639680 (23h4m1.2992128s) [seg_pos={0 0}, upload=20341]: Begin upload segment... Sep 7 12:32:25.259682412 (23h4m1.299255532s) [seg_pos={0 0}, upload=20341]: Done begin upload segment. Sep 7 12:32:25.259731626 (23h4m1.299304756s) [noise=true , address=45.91.116.9:4263 , piece_id=VSYUTOSVLDLAOB74XZFESXWLL5EZN5Z25LYDH3WZQN5LMSDSGELQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.259743980 (23h4m1.2993171s) [noise=true , address=45.91.116.12:4226 , piece_id=HMUGW35FHAMUHRVKAIFMOR2PJHDFJILZZPWHRHN7252NZNTAGXKA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260150017 (23h4m1.299723137s) [noise=true , address=45.91.116.13:4155 , piece_id=MYOQX2PWNRR3DFBMJLIUQ45IZIMCL5Q3MOEZC72VPSG6MTDHJIRA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260179443 (23h4m1.299752563s) [noise=true , address=45.91.116.6:4974 , piece_id=H4W6QGMIHMT3FZ3BQ52ABQVFOLJ4TRJD3F3VBCLCE6VC6OGWSLSQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260415534 (23h4m1.299988654s) [noise=true , address=45.91.116.5:4299 , piece_id=2C24J3IYULB7NANTJ4EKWBIXVK2LU6ZH7TT4KVDFMBJ25LMYPHUQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260440222 (23h4m1.300013342s) [noise=true , address=77.90.31.16:4735 , piece_id=J36NH3XBSV2G4M263IK4KSKFQ37CU7FLUX3CTWXRXNKCMAVN5ULQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260672537 (23h4m1.300245677s) [noise=true , address=45.91.116.5:4298 , piece_id=35TNWDJSGJPFJLK7YHLPV7I7HOBUXVIBTF7MMN5KSDJXWUSTCCBA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260696543 (23h4m1.300269663s) [noise=true , address=45.91.116.11:4118 , piece_id=MZA3HUVQCAVINXEJ7MKPRWCBUEWGGQREVNPMXGAWLOPMDXJL52SQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260923076 (23h4m1.300496196s) [noise=true , address=45.91.116.8:4048 , piece_id=GLZM2GCPYEYOGFYZOVRUXASMAOZC6KEU5HJQDHFNNFD2GT2JR3IQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261018137 (23h4m1.300591257s) [noise=true , address=45.91.116.4:4937 , piece_id=XDDLQBKVLHHK3I4C6SCSRUG6CCL6CCRXC3XTOBKB2SUAV7MEF32Q, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261248188 (23h4m1.300821308s) [noise=true , address=77.90.31.18:4805 , piece_id=5HIXWZFYKCYCVJO7SCNCK4EGH5WLBD4IBOMLNPWLXNIFVKJO7V3Q, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261263337 (23h4m1.300836457s) [noise=true , address=45.91.116.6:4975 , piece_id=YNEVVYRYEBHPY33K3LSHPL7QQBLVN3A7MCTFDWC7KPC6MOYTXDMQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261474661 (23h4m1.301047781s) [noise=true , address=45.91.116.8:4045 , piece_id=A2WPFGBEILYT2GPJ2CIPXZJJ6DF7E25DPJBR7KGOXS2WNUQK5BGA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261495622 (23h4m1.301068732s) [noise=true , address=77.90.31.12:4590 , piece_id=H7EHZ76GBVNA7T5ECSBZ6EV6UPLMTU4XRQDILCILS6L3JQTCU3DQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261742434 (23h4m1.301315554s) [noise=true , address=77.90.31.21:4877 , piece_id=R3JXTAY54SDCPJTFCBOEQOZXSZINIAY624OSOFD3SMOIVXJHIWEQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261965330 (23h4m1.30153845s) [noise=true , address=45.91.116.4:4940 , piece_id=Z4A5UBCYQHLULLTWUTB66QGY5GXDRZKWA4XJHYBBJIDDTYAGY7CA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:27.083171850 (23h4m3.12274497s) [upload=20341]: Next returned nil segment Sep 7 12:32:27.083189514 (23h4m3.122762624s) [upload=20341]: Waiting for error group managing segments... Sep 7 12:32:27.397997027 (23h4m3.437570137s) [noise=true , address=77.90.31.21:4877 , piece_id=R3JXTAY54SDCPJTFCBOEQOZXSZINIAY624OSOFD3SMOIVXJHIWEQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.422075642 (23h4m3.461648762s) [noise=true , address=45.91.116.4:4940 , piece_id=Z4A5UBCYQHLULLTWUTB66QGY5GXDRZKWA4XJHYBBJIDDTYAGY7CA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.422196694 (23h4m3.461769804s) [noise=true , address=45.91.116.8:4048 , piece_id=GLZM2GCPYEYOGFYZOVRUXASMAOZC6KEU5HJQDHFNNFD2GT2JR3IQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.426177602 (23h4m3.465750722s) [noise=true , address=45.91.116.9:4263 , piece_id=VSYUTOSVLDLAOB74XZFESXWLL5EZN5Z25LYDH3WZQN5LMSDSGELQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.433234004 (23h4m3.472807124s) [noise=true , address=45.91.116.5:4298 , piece_id=35TNWDJSGJPFJLK7YHLPV7I7HOBUXVIBTF7MMN5KSDJXWUSTCCBA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.442931240 (23h4m3.48250436s) [noise=true , address=45.91.116.6:4975 , piece_id=YNEVVYRYEBHPY33K3LSHPL7QQBLVN3A7MCTFDWC7KPC6MOYTXDMQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.452017787 (23h4m3.491590907s) [noise=true , address=45.91.116.5:4299 , piece_id=2C24J3IYULB7NANTJ4EKWBIXVK2LU6ZH7TT4KVDFMBJ25LMYPHUQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.471965458 (23h4m3.511538568s) [noise=true , address=45.91.116.11:4118 , piece_id=MZA3HUVQCAVINXEJ7MKPRWCBUEWGGQREVNPMXGAWLOPMDXJL52SQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.523292133 (23h4m3.562865253s) [noise=true , address=45.91.116.13:4155 , piece_id=MYOQX2PWNRR3DFBMJLIUQ45IZIMCL5Q3MOEZC72VPSG6MTDHJIRA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.536854595 (23h4m3.576427715s) [noise=true , address=77.90.31.18:4805 , piece_id=5HIXWZFYKCYCVJO7SCNCK4EGH5WLBD4IBOMLNPWLXNIFVKJO7V3Q, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.563355926 (23h4m3.602929036s) [noise=true , address=45.91.116.12:4226 , piece_id=HMUGW35FHAMUHRVKAIFMOR2PJHDFJILZZPWHRHN7252NZNTAGXKA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.579270911 (23h4m3.618844031s) [noise=true , address=45.91.116.8:4045 , piece_id=A2WPFGBEILYT2GPJ2CIPXZJJ6DF7E25DPJBR7KGOXS2WNUQK5BGA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.596306278 (23h4m3.635879398s) [noise=true , address=77.90.31.12:4590 , piece_id=H7EHZ76GBVNA7T5ECSBZ6EV6UPLMTU4XRQDILCILS6L3JQTCU3DQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.596332468 (23h4m3.635905588s) [seg_pos={0 0}, upload=20341]: Segment reached optimal threshold of 13 pieces. Sep 7 12:32:27.596482004 (23h4m3.636055124s) [noise=true , address=77.90.31.16:4735 , piece_id=J36NH3XBSV2G4M263IK4KSKFQ37CU7FLUX3CTWXRXNKCMAVN5ULQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Sep 7 12:32:27.596621401 (23h4m3.636194511s) [noise=true , address=45.91.116.6:4974 , piece_id=H4W6QGMIHMT3FZ3BQ52ABQVFOLJ4TRJD3F3VBCLCE6VC6OGWSLSQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Sep 7 12:32:27.596667589 (23h4m3.636240709s) [noise=true , address=45.91.116.4:4937 , piece_id=XDDLQBKVLHHK3I4C6SCSRUG6CCL6CCRXC3XTOBKB2SUAV7MEF32Q, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection Sep 7 12:32:27.596683911 (23h4m3.636257031s) [seg_pos={0 0}, upload=20341]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:32:27.596751590 (23h4m3.63632471s) [upload=20341]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:32:27.596757722 (23h4m3.636330842s) [upload=20341]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:32:27.904840427 (23h4m3.944413547s) [upload=20341]: Done uploading segments... Sep 7 12:32:27.904858301 (23h4m3.944431421s) [upload=20341]: Upload finished. err: <nil> Duration: 17.302609583s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............X...........X.....X.....................X....] duration: 7.195743216s FAIL (2025-09-07 12:36:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:32:24.971479778 (23h4m1.011052898s) [upload=20341]: Starting upload Sep 7 12:32:24.971535354 (23h4m1.011108474s) [upload=20341]: Done starting upload Sep 7 12:32:24.971630667 (23h4m1.011203787s) [upload=20341]: Uploading segments... Sep 7 12:32:24.971695100 (23h4m1.01126822s) [upload=20341]: Got next segment. Inline: false Sep 7 12:32:24.971723614 (23h4m1.011296735s) [upload=20341]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:32:24.971728113 (23h4m1.011301233s) [upload=20341]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:32:25.259639680 (23h4m1.2992128s) [seg_pos={0 0}, upload=20341]: Begin upload segment... Sep 7 12:32:25.259682412 (23h4m1.299255532s) [seg_pos={0 0}, upload=20341]: Done begin upload segment. Sep 7 12:32:25.259731626 (23h4m1.299304756s) [noise=true , address=45.91.116.9:4263 , piece_id=VSYUTOSVLDLAOB74XZFESXWLL5EZN5Z25LYDH3WZQN5LMSDSGELQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.259743980 (23h4m1.2993171s) [noise=true , address=45.91.116.12:4226 , piece_id=HMUGW35FHAMUHRVKAIFMOR2PJHDFJILZZPWHRHN7252NZNTAGXKA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260150017 (23h4m1.299723137s) [noise=true , address=45.91.116.13:4155 , piece_id=MYOQX2PWNRR3DFBMJLIUQ45IZIMCL5Q3MOEZC72VPSG6MTDHJIRA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260179443 (23h4m1.299752563s) [noise=true , address=45.91.116.6:4974 , piece_id=H4W6QGMIHMT3FZ3BQ52ABQVFOLJ4TRJD3F3VBCLCE6VC6OGWSLSQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260415534 (23h4m1.299988654s) [noise=true , address=45.91.116.5:4299 , piece_id=2C24J3IYULB7NANTJ4EKWBIXVK2LU6ZH7TT4KVDFMBJ25LMYPHUQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260440222 (23h4m1.300013342s) [noise=true , address=77.90.31.16:4735 , piece_id=J36NH3XBSV2G4M263IK4KSKFQ37CU7FLUX3CTWXRXNKCMAVN5ULQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260672537 (23h4m1.300245677s) [noise=true , address=45.91.116.5:4298 , piece_id=35TNWDJSGJPFJLK7YHLPV7I7HOBUXVIBTF7MMN5KSDJXWUSTCCBA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260696543 (23h4m1.300269663s) [noise=true , address=45.91.116.11:4118 , piece_id=MZA3HUVQCAVINXEJ7MKPRWCBUEWGGQREVNPMXGAWLOPMDXJL52SQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260923076 (23h4m1.300496196s) [noise=true , address=45.91.116.8:4048 , piece_id=GLZM2GCPYEYOGFYZOVRUXASMAOZC6KEU5HJQDHFNNFD2GT2JR3IQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261018137 (23h4m1.300591257s) [noise=true , address=45.91.116.4:4937 , piece_id=XDDLQBKVLHHK3I4C6SCSRUG6CCL6CCRXC3XTOBKB2SUAV7MEF32Q, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261248188 (23h4m1.300821308s) [noise=true , address=77.90.31.18:4805 , piece_id=5HIXWZFYKCYCVJO7SCNCK4EGH5WLBD4IBOMLNPWLXNIFVKJO7V3Q, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261263337 (23h4m1.300836457s) [noise=true , address=45.91.116.6:4975 , piece_id=YNEVVYRYEBHPY33K3LSHPL7QQBLVN3A7MCTFDWC7KPC6MOYTXDMQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261474661 (23h4m1.301047781s) [noise=true , address=45.91.116.8:4045 , piece_id=A2WPFGBEILYT2GPJ2CIPXZJJ6DF7E25DPJBR7KGOXS2WNUQK5BGA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261495622 (23h4m1.301068732s) [noise=true , address=77.90.31.12:4590 , piece_id=H7EHZ76GBVNA7T5ECSBZ6EV6UPLMTU4XRQDILCILS6L3JQTCU3DQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261742434 (23h4m1.301315554s) [noise=true , address=77.90.31.21:4877 , piece_id=R3JXTAY54SDCPJTFCBOEQOZXSZINIAY624OSOFD3SMOIVXJHIWEQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261965330 (23h4m1.30153845s) [noise=true , address=45.91.116.4:4940 , piece_id=Z4A5UBCYQHLULLTWUTB66QGY5GXDRZKWA4XJHYBBJIDDTYAGY7CA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:27.083171850 (23h4m3.12274497s) [upload=20341]: Next returned nil segment Sep 7 12:32:27.083189514 (23h4m3.122762624s) [upload=20341]: Waiting for error group managing segments... Sep 7 12:32:27.397997027 (23h4m3.437570137s) [noise=true , address=77.90.31.21:4877 , piece_id=R3JXTAY54SDCPJTFCBOEQOZXSZINIAY624OSOFD3SMOIVXJHIWEQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.422075642 (23h4m3.461648762s) [noise=true , address=45.91.116.4:4940 , piece_id=Z4A5UBCYQHLULLTWUTB66QGY5GXDRZKWA4XJHYBBJIDDTYAGY7CA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.422196694 (23h4m3.461769804s) [noise=true , address=45.91.116.8:4048 , piece_id=GLZM2GCPYEYOGFYZOVRUXASMAOZC6KEU5HJQDHFNNFD2GT2JR3IQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.426177602 (23h4m3.465750722s) [noise=true , address=45.91.116.9:4263 , piece_id=VSYUTOSVLDLAOB74XZFESXWLL5EZN5Z25LYDH3WZQN5LMSDSGELQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.433234004 (23h4m3.472807124s) [noise=true , address=45.91.116.5:4298 , piece_id=35TNWDJSGJPFJLK7YHLPV7I7HOBUXVIBTF7MMN5KSDJXWUSTCCBA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.442931240 (23h4m3.48250436s) [noise=true , address=45.91.116.6:4975 , piece_id=YNEVVYRYEBHPY33K3LSHPL7QQBLVN3A7MCTFDWC7KPC6MOYTXDMQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.452017787 (23h4m3.491590907s) [noise=true , address=45.91.116.5:4299 , piece_id=2C24J3IYULB7NANTJ4EKWBIXVK2LU6ZH7TT4KVDFMBJ25LMYPHUQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.471965458 (23h4m3.511538568s) [noise=true , address=45.91.116.11:4118 , piece_id=MZA3HUVQCAVINXEJ7MKPRWCBUEWGGQREVNPMXGAWLOPMDXJL52SQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.523292133 (23h4m3.562865253s) [noise=true , address=45.91.116.13:4155 , piece_id=MYOQX2PWNRR3DFBMJLIUQ45IZIMCL5Q3MOEZC72VPSG6MTDHJIRA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.536854595 (23h4m3.576427715s) [noise=true , address=77.90.31.18:4805 , piece_id=5HIXWZFYKCYCVJO7SCNCK4EGH5WLBD4IBOMLNPWLXNIFVKJO7V3Q, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.563355926 (23h4m3.602929036s) [noise=true , address=45.91.116.12:4226 , piece_id=HMUGW35FHAMUHRVKAIFMOR2PJHDFJILZZPWHRHN7252NZNTAGXKA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.579270911 (23h4m3.618844031s) [noise=true , address=45.91.116.8:4045 , piece_id=A2WPFGBEILYT2GPJ2CIPXZJJ6DF7E25DPJBR7KGOXS2WNUQK5BGA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.596306278 (23h4m3.635879398s) [noise=true , address=77.90.31.12:4590 , piece_id=H7EHZ76GBVNA7T5ECSBZ6EV6UPLMTU4XRQDILCILS6L3JQTCU3DQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.596332468 (23h4m3.635905588s) [seg_pos={0 0}, upload=20341]: Segment reached optimal threshold of 13 pieces. Sep 7 12:32:27.596482004 (23h4m3.636055124s) [noise=true , address=77.90.31.16:4735 , piece_id=J36NH3XBSV2G4M263IK4KSKFQ37CU7FLUX3CTWXRXNKCMAVN5ULQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Sep 7 12:32:27.596621401 (23h4m3.636194511s) [noise=true , address=45.91.116.6:4974 , piece_id=H4W6QGMIHMT3FZ3BQ52ABQVFOLJ4TRJD3F3VBCLCE6VC6OGWSLSQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Sep 7 12:32:27.596667589 (23h4m3.636240709s) [noise=true , address=45.91.116.4:4937 , piece_id=XDDLQBKVLHHK3I4C6SCSRUG6CCL6CCRXC3XTOBKB2SUAV7MEF32Q, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection Sep 7 12:32:27.596683911 (23h4m3.636257031s) [seg_pos={0 0}, upload=20341]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:32:27.596751590 (23h4m3.63632471s) [upload=20341]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:32:27.596757722 (23h4m3.636330842s) [upload=20341]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:32:27.904840427 (23h4m3.944413547s) [upload=20341]: Done uploading segments... Sep 7 12:32:27.904858301 (23h4m3.944431421s) [upload=20341]: Upload finished. err: <nil> Duration: 7.195774548s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................X...........X.....X.....................X...] duration: 7.955234225s FAIL (2025-09-07 12:34:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:32:24.971479778 (23h4m1.011052898s) [upload=20341]: Starting upload Sep 7 12:32:24.971535354 (23h4m1.011108474s) [upload=20341]: Done starting upload Sep 7 12:32:24.971630667 (23h4m1.011203787s) [upload=20341]: Uploading segments... Sep 7 12:32:24.971695100 (23h4m1.01126822s) [upload=20341]: Got next segment. Inline: false Sep 7 12:32:24.971723614 (23h4m1.011296735s) [upload=20341]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:32:24.971728113 (23h4m1.011301233s) [upload=20341]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:32:25.259639680 (23h4m1.2992128s) [seg_pos={0 0}, upload=20341]: Begin upload segment... Sep 7 12:32:25.259682412 (23h4m1.299255532s) [seg_pos={0 0}, upload=20341]: Done begin upload segment. Sep 7 12:32:25.259731626 (23h4m1.299304756s) [noise=true , address=45.91.116.9:4263 , piece_id=VSYUTOSVLDLAOB74XZFESXWLL5EZN5Z25LYDH3WZQN5LMSDSGELQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.259743980 (23h4m1.2993171s) [noise=true , address=45.91.116.12:4226 , piece_id=HMUGW35FHAMUHRVKAIFMOR2PJHDFJILZZPWHRHN7252NZNTAGXKA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260150017 (23h4m1.299723137s) [noise=true , address=45.91.116.13:4155 , piece_id=MYOQX2PWNRR3DFBMJLIUQ45IZIMCL5Q3MOEZC72VPSG6MTDHJIRA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260179443 (23h4m1.299752563s) [noise=true , address=45.91.116.6:4974 , piece_id=H4W6QGMIHMT3FZ3BQ52ABQVFOLJ4TRJD3F3VBCLCE6VC6OGWSLSQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260415534 (23h4m1.299988654s) [noise=true , address=45.91.116.5:4299 , piece_id=2C24J3IYULB7NANTJ4EKWBIXVK2LU6ZH7TT4KVDFMBJ25LMYPHUQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260440222 (23h4m1.300013342s) [noise=true , address=77.90.31.16:4735 , piece_id=J36NH3XBSV2G4M263IK4KSKFQ37CU7FLUX3CTWXRXNKCMAVN5ULQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260672537 (23h4m1.300245677s) [noise=true , address=45.91.116.5:4298 , piece_id=35TNWDJSGJPFJLK7YHLPV7I7HOBUXVIBTF7MMN5KSDJXWUSTCCBA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260696543 (23h4m1.300269663s) [noise=true , address=45.91.116.11:4118 , piece_id=MZA3HUVQCAVINXEJ7MKPRWCBUEWGGQREVNPMXGAWLOPMDXJL52SQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260923076 (23h4m1.300496196s) [noise=true , address=45.91.116.8:4048 , piece_id=GLZM2GCPYEYOGFYZOVRUXASMAOZC6KEU5HJQDHFNNFD2GT2JR3IQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261018137 (23h4m1.300591257s) [noise=true , address=45.91.116.4:4937 , piece_id=XDDLQBKVLHHK3I4C6SCSRUG6CCL6CCRXC3XTOBKB2SUAV7MEF32Q, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261248188 (23h4m1.300821308s) [noise=true , address=77.90.31.18:4805 , piece_id=5HIXWZFYKCYCVJO7SCNCK4EGH5WLBD4IBOMLNPWLXNIFVKJO7V3Q, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261263337 (23h4m1.300836457s) [noise=true , address=45.91.116.6:4975 , piece_id=YNEVVYRYEBHPY33K3LSHPL7QQBLVN3A7MCTFDWC7KPC6MOYTXDMQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261474661 (23h4m1.301047781s) [noise=true , address=45.91.116.8:4045 , piece_id=A2WPFGBEILYT2GPJ2CIPXZJJ6DF7E25DPJBR7KGOXS2WNUQK5BGA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261495622 (23h4m1.301068732s) [noise=true , address=77.90.31.12:4590 , piece_id=H7EHZ76GBVNA7T5ECSBZ6EV6UPLMTU4XRQDILCILS6L3JQTCU3DQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261742434 (23h4m1.301315554s) [noise=true , address=77.90.31.21:4877 , piece_id=R3JXTAY54SDCPJTFCBOEQOZXSZINIAY624OSOFD3SMOIVXJHIWEQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261965330 (23h4m1.30153845s) [noise=true , address=45.91.116.4:4940 , piece_id=Z4A5UBCYQHLULLTWUTB66QGY5GXDRZKWA4XJHYBBJIDDTYAGY7CA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:27.083171850 (23h4m3.12274497s) [upload=20341]: Next returned nil segment Sep 7 12:32:27.083189514 (23h4m3.122762624s) [upload=20341]: Waiting for error group managing segments... Sep 7 12:32:27.397997027 (23h4m3.437570137s) [noise=true , address=77.90.31.21:4877 , piece_id=R3JXTAY54SDCPJTFCBOEQOZXSZINIAY624OSOFD3SMOIVXJHIWEQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.422075642 (23h4m3.461648762s) [noise=true , address=45.91.116.4:4940 , piece_id=Z4A5UBCYQHLULLTWUTB66QGY5GXDRZKWA4XJHYBBJIDDTYAGY7CA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.422196694 (23h4m3.461769804s) [noise=true , address=45.91.116.8:4048 , piece_id=GLZM2GCPYEYOGFYZOVRUXASMAOZC6KEU5HJQDHFNNFD2GT2JR3IQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.426177602 (23h4m3.465750722s) [noise=true , address=45.91.116.9:4263 , piece_id=VSYUTOSVLDLAOB74XZFESXWLL5EZN5Z25LYDH3WZQN5LMSDSGELQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.433234004 (23h4m3.472807124s) [noise=true , address=45.91.116.5:4298 , piece_id=35TNWDJSGJPFJLK7YHLPV7I7HOBUXVIBTF7MMN5KSDJXWUSTCCBA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.442931240 (23h4m3.48250436s) [noise=true , address=45.91.116.6:4975 , piece_id=YNEVVYRYEBHPY33K3LSHPL7QQBLVN3A7MCTFDWC7KPC6MOYTXDMQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.452017787 (23h4m3.491590907s) [noise=true , address=45.91.116.5:4299 , piece_id=2C24J3IYULB7NANTJ4EKWBIXVK2LU6ZH7TT4KVDFMBJ25LMYPHUQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.471965458 (23h4m3.511538568s) [noise=true , address=45.91.116.11:4118 , piece_id=MZA3HUVQCAVINXEJ7MKPRWCBUEWGGQREVNPMXGAWLOPMDXJL52SQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.523292133 (23h4m3.562865253s) [noise=true , address=45.91.116.13:4155 , piece_id=MYOQX2PWNRR3DFBMJLIUQ45IZIMCL5Q3MOEZC72VPSG6MTDHJIRA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.536854595 (23h4m3.576427715s) [noise=true , address=77.90.31.18:4805 , piece_id=5HIXWZFYKCYCVJO7SCNCK4EGH5WLBD4IBOMLNPWLXNIFVKJO7V3Q, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.563355926 (23h4m3.602929036s) [noise=true , address=45.91.116.12:4226 , piece_id=HMUGW35FHAMUHRVKAIFMOR2PJHDFJILZZPWHRHN7252NZNTAGXKA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.579270911 (23h4m3.618844031s) [noise=true , address=45.91.116.8:4045 , piece_id=A2WPFGBEILYT2GPJ2CIPXZJJ6DF7E25DPJBR7KGOXS2WNUQK5BGA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.596306278 (23h4m3.635879398s) [noise=true , address=77.90.31.12:4590 , piece_id=H7EHZ76GBVNA7T5ECSBZ6EV6UPLMTU4XRQDILCILS6L3JQTCU3DQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.596332468 (23h4m3.635905588s) [seg_pos={0 0}, upload=20341]: Segment reached optimal threshold of 13 pieces. Sep 7 12:32:27.596482004 (23h4m3.636055124s) [noise=true , address=77.90.31.16:4735 , piece_id=J36NH3XBSV2G4M263IK4KSKFQ37CU7FLUX3CTWXRXNKCMAVN5ULQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Sep 7 12:32:27.596621401 (23h4m3.636194511s) [noise=true , address=45.91.116.6:4974 , piece_id=H4W6QGMIHMT3FZ3BQ52ABQVFOLJ4TRJD3F3VBCLCE6VC6OGWSLSQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Sep 7 12:32:27.596667589 (23h4m3.636240709s) [noise=true , address=45.91.116.4:4937 , piece_id=XDDLQBKVLHHK3I4C6SCSRUG6CCL6CCRXC3XTOBKB2SUAV7MEF32Q, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection Sep 7 12:32:27.596683911 (23h4m3.636257031s) [seg_pos={0 0}, upload=20341]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:32:27.596751590 (23h4m3.63632471s) [upload=20341]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:32:27.596757722 (23h4m3.636330842s) [upload=20341]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:32:27.904840427 (23h4m3.944413547s) [upload=20341]: Done uploading segments... Sep 7 12:32:27.904858301 (23h4m3.944431421s) [upload=20341]: Upload finished. err: <nil> Duration: 7.95526268s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................X...........X.....X.....................X..] duration: 10.633005899s FAIL (2025-09-07 12:33:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:32:24.971479778 (23h4m1.011052898s) [upload=20341]: Starting upload Sep 7 12:32:24.971535354 (23h4m1.011108474s) [upload=20341]: Done starting upload Sep 7 12:32:24.971630667 (23h4m1.011203787s) [upload=20341]: Uploading segments... Sep 7 12:32:24.971695100 (23h4m1.01126822s) [upload=20341]: Got next segment. Inline: false Sep 7 12:32:24.971723614 (23h4m1.011296735s) [upload=20341]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:32:24.971728113 (23h4m1.011301233s) [upload=20341]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:32:25.259639680 (23h4m1.2992128s) [seg_pos={0 0}, upload=20341]: Begin upload segment... Sep 7 12:32:25.259682412 (23h4m1.299255532s) [seg_pos={0 0}, upload=20341]: Done begin upload segment. Sep 7 12:32:25.259731626 (23h4m1.299304756s) [noise=true , address=45.91.116.9:4263 , piece_id=VSYUTOSVLDLAOB74XZFESXWLL5EZN5Z25LYDH3WZQN5LMSDSGELQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.259743980 (23h4m1.2993171s) [noise=true , address=45.91.116.12:4226 , piece_id=HMUGW35FHAMUHRVKAIFMOR2PJHDFJILZZPWHRHN7252NZNTAGXKA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260150017 (23h4m1.299723137s) [noise=true , address=45.91.116.13:4155 , piece_id=MYOQX2PWNRR3DFBMJLIUQ45IZIMCL5Q3MOEZC72VPSG6MTDHJIRA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260179443 (23h4m1.299752563s) [noise=true , address=45.91.116.6:4974 , piece_id=H4W6QGMIHMT3FZ3BQ52ABQVFOLJ4TRJD3F3VBCLCE6VC6OGWSLSQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260415534 (23h4m1.299988654s) [noise=true , address=45.91.116.5:4299 , piece_id=2C24J3IYULB7NANTJ4EKWBIXVK2LU6ZH7TT4KVDFMBJ25LMYPHUQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260440222 (23h4m1.300013342s) [noise=true , address=77.90.31.16:4735 , piece_id=J36NH3XBSV2G4M263IK4KSKFQ37CU7FLUX3CTWXRXNKCMAVN5ULQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260672537 (23h4m1.300245677s) [noise=true , address=45.91.116.5:4298 , piece_id=35TNWDJSGJPFJLK7YHLPV7I7HOBUXVIBTF7MMN5KSDJXWUSTCCBA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260696543 (23h4m1.300269663s) [noise=true , address=45.91.116.11:4118 , piece_id=MZA3HUVQCAVINXEJ7MKPRWCBUEWGGQREVNPMXGAWLOPMDXJL52SQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260923076 (23h4m1.300496196s) [noise=true , address=45.91.116.8:4048 , piece_id=GLZM2GCPYEYOGFYZOVRUXASMAOZC6KEU5HJQDHFNNFD2GT2JR3IQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261018137 (23h4m1.300591257s) [noise=true , address=45.91.116.4:4937 , piece_id=XDDLQBKVLHHK3I4C6SCSRUG6CCL6CCRXC3XTOBKB2SUAV7MEF32Q, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261248188 (23h4m1.300821308s) [noise=true , address=77.90.31.18:4805 , piece_id=5HIXWZFYKCYCVJO7SCNCK4EGH5WLBD4IBOMLNPWLXNIFVKJO7V3Q, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261263337 (23h4m1.300836457s) [noise=true , address=45.91.116.6:4975 , piece_id=YNEVVYRYEBHPY33K3LSHPL7QQBLVN3A7MCTFDWC7KPC6MOYTXDMQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261474661 (23h4m1.301047781s) [noise=true , address=45.91.116.8:4045 , piece_id=A2WPFGBEILYT2GPJ2CIPXZJJ6DF7E25DPJBR7KGOXS2WNUQK5BGA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261495622 (23h4m1.301068732s) [noise=true , address=77.90.31.12:4590 , piece_id=H7EHZ76GBVNA7T5ECSBZ6EV6UPLMTU4XRQDILCILS6L3JQTCU3DQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261742434 (23h4m1.301315554s) [noise=true , address=77.90.31.21:4877 , piece_id=R3JXTAY54SDCPJTFCBOEQOZXSZINIAY624OSOFD3SMOIVXJHIWEQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261965330 (23h4m1.30153845s) [noise=true , address=45.91.116.4:4940 , piece_id=Z4A5UBCYQHLULLTWUTB66QGY5GXDRZKWA4XJHYBBJIDDTYAGY7CA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:27.083171850 (23h4m3.12274497s) [upload=20341]: Next returned nil segment Sep 7 12:32:27.083189514 (23h4m3.122762624s) [upload=20341]: Waiting for error group managing segments... Sep 7 12:32:27.397997027 (23h4m3.437570137s) [noise=true , address=77.90.31.21:4877 , piece_id=R3JXTAY54SDCPJTFCBOEQOZXSZINIAY624OSOFD3SMOIVXJHIWEQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.422075642 (23h4m3.461648762s) [noise=true , address=45.91.116.4:4940 , piece_id=Z4A5UBCYQHLULLTWUTB66QGY5GXDRZKWA4XJHYBBJIDDTYAGY7CA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.422196694 (23h4m3.461769804s) [noise=true , address=45.91.116.8:4048 , piece_id=GLZM2GCPYEYOGFYZOVRUXASMAOZC6KEU5HJQDHFNNFD2GT2JR3IQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.426177602 (23h4m3.465750722s) [noise=true , address=45.91.116.9:4263 , piece_id=VSYUTOSVLDLAOB74XZFESXWLL5EZN5Z25LYDH3WZQN5LMSDSGELQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.433234004 (23h4m3.472807124s) [noise=true , address=45.91.116.5:4298 , piece_id=35TNWDJSGJPFJLK7YHLPV7I7HOBUXVIBTF7MMN5KSDJXWUSTCCBA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.442931240 (23h4m3.48250436s) [noise=true , address=45.91.116.6:4975 , piece_id=YNEVVYRYEBHPY33K3LSHPL7QQBLVN3A7MCTFDWC7KPC6MOYTXDMQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.452017787 (23h4m3.491590907s) [noise=true , address=45.91.116.5:4299 , piece_id=2C24J3IYULB7NANTJ4EKWBIXVK2LU6ZH7TT4KVDFMBJ25LMYPHUQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.471965458 (23h4m3.511538568s) [noise=true , address=45.91.116.11:4118 , piece_id=MZA3HUVQCAVINXEJ7MKPRWCBUEWGGQREVNPMXGAWLOPMDXJL52SQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.523292133 (23h4m3.562865253s) [noise=true , address=45.91.116.13:4155 , piece_id=MYOQX2PWNRR3DFBMJLIUQ45IZIMCL5Q3MOEZC72VPSG6MTDHJIRA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.536854595 (23h4m3.576427715s) [noise=true , address=77.90.31.18:4805 , piece_id=5HIXWZFYKCYCVJO7SCNCK4EGH5WLBD4IBOMLNPWLXNIFVKJO7V3Q, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.563355926 (23h4m3.602929036s) [noise=true , address=45.91.116.12:4226 , piece_id=HMUGW35FHAMUHRVKAIFMOR2PJHDFJILZZPWHRHN7252NZNTAGXKA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.579270911 (23h4m3.618844031s) [noise=true , address=45.91.116.8:4045 , piece_id=A2WPFGBEILYT2GPJ2CIPXZJJ6DF7E25DPJBR7KGOXS2WNUQK5BGA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.596306278 (23h4m3.635879398s) [noise=true , address=77.90.31.12:4590 , piece_id=H7EHZ76GBVNA7T5ECSBZ6EV6UPLMTU4XRQDILCILS6L3JQTCU3DQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.596332468 (23h4m3.635905588s) [seg_pos={0 0}, upload=20341]: Segment reached optimal threshold of 13 pieces. Sep 7 12:32:27.596482004 (23h4m3.636055124s) [noise=true , address=77.90.31.16:4735 , piece_id=J36NH3XBSV2G4M263IK4KSKFQ37CU7FLUX3CTWXRXNKCMAVN5ULQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Sep 7 12:32:27.596621401 (23h4m3.636194511s) [noise=true , address=45.91.116.6:4974 , piece_id=H4W6QGMIHMT3FZ3BQ52ABQVFOLJ4TRJD3F3VBCLCE6VC6OGWSLSQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Sep 7 12:32:27.596667589 (23h4m3.636240709s) [noise=true , address=45.91.116.4:4937 , piece_id=XDDLQBKVLHHK3I4C6SCSRUG6CCL6CCRXC3XTOBKB2SUAV7MEF32Q, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection Sep 7 12:32:27.596683911 (23h4m3.636257031s) [seg_pos={0 0}, upload=20341]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:32:27.596751590 (23h4m3.63632471s) [upload=20341]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:32:27.596757722 (23h4m3.636330842s) [upload=20341]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:32:27.904840427 (23h4m3.944413547s) [upload=20341]: Done uploading segments... Sep 7 12:32:27.904858301 (23h4m3.944431421s) [upload=20341]: Upload finished. err: <nil> Duration: 10.633034784s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................X...........X.....X.....................X.] duration: 9.302076686s FAIL (2025-09-07 12:33:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:32:24.971479778 (23h4m1.011052898s) [upload=20341]: Starting upload Sep 7 12:32:24.971535354 (23h4m1.011108474s) [upload=20341]: Done starting upload Sep 7 12:32:24.971630667 (23h4m1.011203787s) [upload=20341]: Uploading segments... Sep 7 12:32:24.971695100 (23h4m1.01126822s) [upload=20341]: Got next segment. Inline: false Sep 7 12:32:24.971723614 (23h4m1.011296735s) [upload=20341]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:32:24.971728113 (23h4m1.011301233s) [upload=20341]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:32:25.259639680 (23h4m1.2992128s) [seg_pos={0 0}, upload=20341]: Begin upload segment... Sep 7 12:32:25.259682412 (23h4m1.299255532s) [seg_pos={0 0}, upload=20341]: Done begin upload segment. Sep 7 12:32:25.259731626 (23h4m1.299304756s) [noise=true , address=45.91.116.9:4263 , piece_id=VSYUTOSVLDLAOB74XZFESXWLL5EZN5Z25LYDH3WZQN5LMSDSGELQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.259743980 (23h4m1.2993171s) [noise=true , address=45.91.116.12:4226 , piece_id=HMUGW35FHAMUHRVKAIFMOR2PJHDFJILZZPWHRHN7252NZNTAGXKA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260150017 (23h4m1.299723137s) [noise=true , address=45.91.116.13:4155 , piece_id=MYOQX2PWNRR3DFBMJLIUQ45IZIMCL5Q3MOEZC72VPSG6MTDHJIRA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260179443 (23h4m1.299752563s) [noise=true , address=45.91.116.6:4974 , piece_id=H4W6QGMIHMT3FZ3BQ52ABQVFOLJ4TRJD3F3VBCLCE6VC6OGWSLSQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260415534 (23h4m1.299988654s) [noise=true , address=45.91.116.5:4299 , piece_id=2C24J3IYULB7NANTJ4EKWBIXVK2LU6ZH7TT4KVDFMBJ25LMYPHUQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260440222 (23h4m1.300013342s) [noise=true , address=77.90.31.16:4735 , piece_id=J36NH3XBSV2G4M263IK4KSKFQ37CU7FLUX3CTWXRXNKCMAVN5ULQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260672537 (23h4m1.300245677s) [noise=true , address=45.91.116.5:4298 , piece_id=35TNWDJSGJPFJLK7YHLPV7I7HOBUXVIBTF7MMN5KSDJXWUSTCCBA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260696543 (23h4m1.300269663s) [noise=true , address=45.91.116.11:4118 , piece_id=MZA3HUVQCAVINXEJ7MKPRWCBUEWGGQREVNPMXGAWLOPMDXJL52SQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260923076 (23h4m1.300496196s) [noise=true , address=45.91.116.8:4048 , piece_id=GLZM2GCPYEYOGFYZOVRUXASMAOZC6KEU5HJQDHFNNFD2GT2JR3IQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261018137 (23h4m1.300591257s) [noise=true , address=45.91.116.4:4937 , piece_id=XDDLQBKVLHHK3I4C6SCSRUG6CCL6CCRXC3XTOBKB2SUAV7MEF32Q, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261248188 (23h4m1.300821308s) [noise=true , address=77.90.31.18:4805 , piece_id=5HIXWZFYKCYCVJO7SCNCK4EGH5WLBD4IBOMLNPWLXNIFVKJO7V3Q, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261263337 (23h4m1.300836457s) [noise=true , address=45.91.116.6:4975 , piece_id=YNEVVYRYEBHPY33K3LSHPL7QQBLVN3A7MCTFDWC7KPC6MOYTXDMQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261474661 (23h4m1.301047781s) [noise=true , address=45.91.116.8:4045 , piece_id=A2WPFGBEILYT2GPJ2CIPXZJJ6DF7E25DPJBR7KGOXS2WNUQK5BGA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261495622 (23h4m1.301068732s) [noise=true , address=77.90.31.12:4590 , piece_id=H7EHZ76GBVNA7T5ECSBZ6EV6UPLMTU4XRQDILCILS6L3JQTCU3DQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261742434 (23h4m1.301315554s) [noise=true , address=77.90.31.21:4877 , piece_id=R3JXTAY54SDCPJTFCBOEQOZXSZINIAY624OSOFD3SMOIVXJHIWEQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261965330 (23h4m1.30153845s) [noise=true , address=45.91.116.4:4940 , piece_id=Z4A5UBCYQHLULLTWUTB66QGY5GXDRZKWA4XJHYBBJIDDTYAGY7CA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:27.083171850 (23h4m3.12274497s) [upload=20341]: Next returned nil segment Sep 7 12:32:27.083189514 (23h4m3.122762624s) [upload=20341]: Waiting for error group managing segments... Sep 7 12:32:27.397997027 (23h4m3.437570137s) [noise=true , address=77.90.31.21:4877 , piece_id=R3JXTAY54SDCPJTFCBOEQOZXSZINIAY624OSOFD3SMOIVXJHIWEQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.422075642 (23h4m3.461648762s) [noise=true , address=45.91.116.4:4940 , piece_id=Z4A5UBCYQHLULLTWUTB66QGY5GXDRZKWA4XJHYBBJIDDTYAGY7CA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.422196694 (23h4m3.461769804s) [noise=true , address=45.91.116.8:4048 , piece_id=GLZM2GCPYEYOGFYZOVRUXASMAOZC6KEU5HJQDHFNNFD2GT2JR3IQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.426177602 (23h4m3.465750722s) [noise=true , address=45.91.116.9:4263 , piece_id=VSYUTOSVLDLAOB74XZFESXWLL5EZN5Z25LYDH3WZQN5LMSDSGELQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.433234004 (23h4m3.472807124s) [noise=true , address=45.91.116.5:4298 , piece_id=35TNWDJSGJPFJLK7YHLPV7I7HOBUXVIBTF7MMN5KSDJXWUSTCCBA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.442931240 (23h4m3.48250436s) [noise=true , address=45.91.116.6:4975 , piece_id=YNEVVYRYEBHPY33K3LSHPL7QQBLVN3A7MCTFDWC7KPC6MOYTXDMQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.452017787 (23h4m3.491590907s) [noise=true , address=45.91.116.5:4299 , piece_id=2C24J3IYULB7NANTJ4EKWBIXVK2LU6ZH7TT4KVDFMBJ25LMYPHUQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.471965458 (23h4m3.511538568s) [noise=true , address=45.91.116.11:4118 , piece_id=MZA3HUVQCAVINXEJ7MKPRWCBUEWGGQREVNPMXGAWLOPMDXJL52SQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.523292133 (23h4m3.562865253s) [noise=true , address=45.91.116.13:4155 , piece_id=MYOQX2PWNRR3DFBMJLIUQ45IZIMCL5Q3MOEZC72VPSG6MTDHJIRA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.536854595 (23h4m3.576427715s) [noise=true , address=77.90.31.18:4805 , piece_id=5HIXWZFYKCYCVJO7SCNCK4EGH5WLBD4IBOMLNPWLXNIFVKJO7V3Q, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.563355926 (23h4m3.602929036s) [noise=true , address=45.91.116.12:4226 , piece_id=HMUGW35FHAMUHRVKAIFMOR2PJHDFJILZZPWHRHN7252NZNTAGXKA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.579270911 (23h4m3.618844031s) [noise=true , address=45.91.116.8:4045 , piece_id=A2WPFGBEILYT2GPJ2CIPXZJJ6DF7E25DPJBR7KGOXS2WNUQK5BGA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.596306278 (23h4m3.635879398s) [noise=true , address=77.90.31.12:4590 , piece_id=H7EHZ76GBVNA7T5ECSBZ6EV6UPLMTU4XRQDILCILS6L3JQTCU3DQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.596332468 (23h4m3.635905588s) [seg_pos={0 0}, upload=20341]: Segment reached optimal threshold of 13 pieces. Sep 7 12:32:27.596482004 (23h4m3.636055124s) [noise=true , address=77.90.31.16:4735 , piece_id=J36NH3XBSV2G4M263IK4KSKFQ37CU7FLUX3CTWXRXNKCMAVN5ULQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Sep 7 12:32:27.596621401 (23h4m3.636194511s) [noise=true , address=45.91.116.6:4974 , piece_id=H4W6QGMIHMT3FZ3BQ52ABQVFOLJ4TRJD3F3VBCLCE6VC6OGWSLSQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Sep 7 12:32:27.596667589 (23h4m3.636240709s) [noise=true , address=45.91.116.4:4937 , piece_id=XDDLQBKVLHHK3I4C6SCSRUG6CCL6CCRXC3XTOBKB2SUAV7MEF32Q, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection Sep 7 12:32:27.596683911 (23h4m3.636257031s) [seg_pos={0 0}, upload=20341]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:32:27.596751590 (23h4m3.63632471s) [upload=20341]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:32:27.596757722 (23h4m3.636330842s) [upload=20341]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:32:27.904840427 (23h4m3.944413547s) [upload=20341]: Done uploading segments... Sep 7 12:32:27.904858301 (23h4m3.944431421s) [upload=20341]: Upload finished. err: <nil> Duration: 9.302102366s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................X...........X.....X.....................X] duration: 30.001864877s FAIL (2025-09-07 12:32:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:32:24.971479778 (23h4m1.011052898s) [upload=20341]: Starting upload Sep 7 12:32:24.971535354 (23h4m1.011108474s) [upload=20341]: Done starting upload Sep 7 12:32:24.971630667 (23h4m1.011203787s) [upload=20341]: Uploading segments... Sep 7 12:32:24.971695100 (23h4m1.01126822s) [upload=20341]: Got next segment. Inline: false Sep 7 12:32:24.971723614 (23h4m1.011296735s) [upload=20341]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:32:24.971728113 (23h4m1.011301233s) [upload=20341]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:32:25.259639680 (23h4m1.2992128s) [seg_pos={0 0}, upload=20341]: Begin upload segment... Sep 7 12:32:25.259682412 (23h4m1.299255532s) [seg_pos={0 0}, upload=20341]: Done begin upload segment. Sep 7 12:32:25.259731626 (23h4m1.299304756s) [noise=true , address=45.91.116.9:4263 , piece_id=VSYUTOSVLDLAOB74XZFESXWLL5EZN5Z25LYDH3WZQN5LMSDSGELQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.259743980 (23h4m1.2993171s) [noise=true , address=45.91.116.12:4226 , piece_id=HMUGW35FHAMUHRVKAIFMOR2PJHDFJILZZPWHRHN7252NZNTAGXKA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260150017 (23h4m1.299723137s) [noise=true , address=45.91.116.13:4155 , piece_id=MYOQX2PWNRR3DFBMJLIUQ45IZIMCL5Q3MOEZC72VPSG6MTDHJIRA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260179443 (23h4m1.299752563s) [noise=true , address=45.91.116.6:4974 , piece_id=H4W6QGMIHMT3FZ3BQ52ABQVFOLJ4TRJD3F3VBCLCE6VC6OGWSLSQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260415534 (23h4m1.299988654s) [noise=true , address=45.91.116.5:4299 , piece_id=2C24J3IYULB7NANTJ4EKWBIXVK2LU6ZH7TT4KVDFMBJ25LMYPHUQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260440222 (23h4m1.300013342s) [noise=true , address=77.90.31.16:4735 , piece_id=J36NH3XBSV2G4M263IK4KSKFQ37CU7FLUX3CTWXRXNKCMAVN5ULQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260672537 (23h4m1.300245677s) [noise=true , address=45.91.116.5:4298 , piece_id=35TNWDJSGJPFJLK7YHLPV7I7HOBUXVIBTF7MMN5KSDJXWUSTCCBA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260696543 (23h4m1.300269663s) [noise=true , address=45.91.116.11:4118 , piece_id=MZA3HUVQCAVINXEJ7MKPRWCBUEWGGQREVNPMXGAWLOPMDXJL52SQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.260923076 (23h4m1.300496196s) [noise=true , address=45.91.116.8:4048 , piece_id=GLZM2GCPYEYOGFYZOVRUXASMAOZC6KEU5HJQDHFNNFD2GT2JR3IQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261018137 (23h4m1.300591257s) [noise=true , address=45.91.116.4:4937 , piece_id=XDDLQBKVLHHK3I4C6SCSRUG6CCL6CCRXC3XTOBKB2SUAV7MEF32Q, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261248188 (23h4m1.300821308s) [noise=true , address=77.90.31.18:4805 , piece_id=5HIXWZFYKCYCVJO7SCNCK4EGH5WLBD4IBOMLNPWLXNIFVKJO7V3Q, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261263337 (23h4m1.300836457s) [noise=true , address=45.91.116.6:4975 , piece_id=YNEVVYRYEBHPY33K3LSHPL7QQBLVN3A7MCTFDWC7KPC6MOYTXDMQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261474661 (23h4m1.301047781s) [noise=true , address=45.91.116.8:4045 , piece_id=A2WPFGBEILYT2GPJ2CIPXZJJ6DF7E25DPJBR7KGOXS2WNUQK5BGA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261495622 (23h4m1.301068732s) [noise=true , address=77.90.31.12:4590 , piece_id=H7EHZ76GBVNA7T5ECSBZ6EV6UPLMTU4XRQDILCILS6L3JQTCU3DQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261742434 (23h4m1.301315554s) [noise=true , address=77.90.31.21:4877 , piece_id=R3JXTAY54SDCPJTFCBOEQOZXSZINIAY624OSOFD3SMOIVXJHIWEQ, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:25.261965330 (23h4m1.30153845s) [noise=true , address=45.91.116.4:4940 , piece_id=Z4A5UBCYQHLULLTWUTB66QGY5GXDRZKWA4XJHYBBJIDDTYAGY7CA, seg_pos={0 0}, upload=20341]: Uploading piece... Sep 7 12:32:27.083171850 (23h4m3.12274497s) [upload=20341]: Next returned nil segment Sep 7 12:32:27.083189514 (23h4m3.122762624s) [upload=20341]: Waiting for error group managing segments... Sep 7 12:32:27.397997027 (23h4m3.437570137s) [noise=true , address=77.90.31.21:4877 , piece_id=R3JXTAY54SDCPJTFCBOEQOZXSZINIAY624OSOFD3SMOIVXJHIWEQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.422075642 (23h4m3.461648762s) [noise=true , address=45.91.116.4:4940 , piece_id=Z4A5UBCYQHLULLTWUTB66QGY5GXDRZKWA4XJHYBBJIDDTYAGY7CA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.422196694 (23h4m3.461769804s) [noise=true , address=45.91.116.8:4048 , piece_id=GLZM2GCPYEYOGFYZOVRUXASMAOZC6KEU5HJQDHFNNFD2GT2JR3IQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.426177602 (23h4m3.465750722s) [noise=true , address=45.91.116.9:4263 , piece_id=VSYUTOSVLDLAOB74XZFESXWLL5EZN5Z25LYDH3WZQN5LMSDSGELQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.433234004 (23h4m3.472807124s) [noise=true , address=45.91.116.5:4298 , piece_id=35TNWDJSGJPFJLK7YHLPV7I7HOBUXVIBTF7MMN5KSDJXWUSTCCBA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.442931240 (23h4m3.48250436s) [noise=true , address=45.91.116.6:4975 , piece_id=YNEVVYRYEBHPY33K3LSHPL7QQBLVN3A7MCTFDWC7KPC6MOYTXDMQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.452017787 (23h4m3.491590907s) [noise=true , address=45.91.116.5:4299 , piece_id=2C24J3IYULB7NANTJ4EKWBIXVK2LU6ZH7TT4KVDFMBJ25LMYPHUQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.471965458 (23h4m3.511538568s) [noise=true , address=45.91.116.11:4118 , piece_id=MZA3HUVQCAVINXEJ7MKPRWCBUEWGGQREVNPMXGAWLOPMDXJL52SQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.523292133 (23h4m3.562865253s) [noise=true , address=45.91.116.13:4155 , piece_id=MYOQX2PWNRR3DFBMJLIUQ45IZIMCL5Q3MOEZC72VPSG6MTDHJIRA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.536854595 (23h4m3.576427715s) [noise=true , address=77.90.31.18:4805 , piece_id=5HIXWZFYKCYCVJO7SCNCK4EGH5WLBD4IBOMLNPWLXNIFVKJO7V3Q, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.563355926 (23h4m3.602929036s) [noise=true , address=45.91.116.12:4226 , piece_id=HMUGW35FHAMUHRVKAIFMOR2PJHDFJILZZPWHRHN7252NZNTAGXKA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.579270911 (23h4m3.618844031s) [noise=true , address=45.91.116.8:4045 , piece_id=A2WPFGBEILYT2GPJ2CIPXZJJ6DF7E25DPJBR7KGOXS2WNUQK5BGA, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.596306278 (23h4m3.635879398s) [noise=true , address=77.90.31.12:4590 , piece_id=H7EHZ76GBVNA7T5ECSBZ6EV6UPLMTU4XRQDILCILS6L3JQTCU3DQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: <nil> Sep 7 12:32:27.596332468 (23h4m3.635905588s) [seg_pos={0 0}, upload=20341]: Segment reached optimal threshold of 13 pieces. Sep 7 12:32:27.596482004 (23h4m3.636055124s) [noise=true , address=77.90.31.16:4735 , piece_id=J36NH3XBSV2G4M263IK4KSKFQ37CU7FLUX3CTWXRXNKCMAVN5ULQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Sep 7 12:32:27.596621401 (23h4m3.636194511s) [noise=true , address=45.91.116.6:4974 , piece_id=H4W6QGMIHMT3FZ3BQ52ABQVFOLJ4TRJD3F3VBCLCE6VC6OGWSLSQ, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Sep 7 12:32:27.596667589 (23h4m3.636240709s) [noise=true , address=45.91.116.4:4937 , piece_id=XDDLQBKVLHHK3I4C6SCSRUG6CCL6CCRXC3XTOBKB2SUAV7MEF32Q, seg_pos={0 0}, upload=20341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41170->45.91.116.4:4937: use of closed network connection Sep 7 12:32:27.596683911 (23h4m3.636257031s) [seg_pos={0 0}, upload=20341]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:32:27.596751590 (23h4m3.63632471s) [upload=20341]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:32:27.596757722 (23h4m3.636330842s) [upload=20341]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:32:27.904840427 (23h4m3.944413547s) [upload=20341]: Done uploading segments... Sep 7 12:32:27.904858301 (23h4m3.944431421s) [upload=20341]: Upload finished. err: <nil> Duration: 30.001885647s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X...........X.....X.....................] duration: 10.147249176s FAIL (2025-09-07 12:32:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:18:13.478410628 (22h49m49.517983748s) [upload=20123]: Starting upload Sep 7 12:18:13.478510909 (22h49m49.518084029s) [upload=20123]: Done starting upload Sep 7 12:18:13.478657580 (22h49m49.5182307s) [upload=20123]: Uploading segments... Sep 7 12:18:13.478782378 (22h49m49.518355548s) [upload=20123]: Got next segment. Inline: false Sep 7 12:18:13.478835610 (22h49m49.51840874s) [upload=20123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:18:13.478843154 (22h49m49.518416274s) [upload=20123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:18:14.234039783 (22h49m50.273612903s) [seg_pos={0 0}, upload=20123]: Begin upload segment... Sep 7 12:18:14.234085281 (22h49m50.273658391s) [seg_pos={0 0}, upload=20123]: Done begin upload segment. Sep 7 12:18:14.234094378 (22h49m50.273667488s) [noise=true , address=45.91.116.13:4154 , piece_id=OTDUTZW26I73ADPVJ3ZPDUI4MYIFJQQI24VA4EGJLS5L3T2PJ5DQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234103195 (22h49m50.273676315s) [noise=true , address=45.91.116.11:4118 , piece_id=E5LRFS4MKVCFO5YQSOXRYARST5SRLZZWJVJO5V3DH4VZZL2AKZGQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234504242 (22h49m50.274077402s) [noise=true , address=45.91.116.12:4226 , piece_id=66XXLFWFQ5BOJL3CPTNRHBVPNBO7DSQJSXKY6PFP4OGZWED6CWSQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234967248 (22h49m50.274540368s) [noise=true , address=45.91.116.5:4298 , piece_id=Y7KFGSE7HWFF33QW3NQ6GGKTDNTKQPPSRNOF3C6EKKNG5PHZ6YBA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235273535 (22h49m50.274846655s) [noise=true , address=45.91.116.7:4011 , piece_id=YWBVLWEYCCPEC7BJXZJAQO66NMEBOHY57KJR4G2DYX36JAYGAF3Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235545195 (22h49m50.275118305s) [noise=true , address=45.91.116.11:4117 , piece_id=47ZZ52FK7HR565KFDSQBJNSMNCONMY7RH7KTBR4R4FMUWOVH7SDA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235830380 (22h49m50.2754035s) [noise=true , address=45.91.116.5:4297 , piece_id=NH4AMQD4XIDTW57WLTQPXM6S5KM4N3SI7G44C4HGXZGEQKWHJ3FA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236049159 (22h49m50.275622279s) [noise=true , address=45.91.116.9:4263 , piece_id=CACXWKVZN72HAWYJUZXS7L6RJMIQ4GPW5NEXBCBC4M7BNKGAN66Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236280301 (22h49m50.275853421s) [noise=true , address=45.91.116.6:4975 , piece_id=J5MP3EAEGKOAZUOIOWI6FFGQVN5JZWWO7F475YCVYDVKGWUBMPNQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236514008 (22h49m50.276087128s) [noise=true , address=45.91.116.12:4227 , piece_id=2VE4AKBVU6EJMEIYWZODUTUZMDSLSQSSXSZQH7FDFR6MWEIW3I7Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236736895 (22h49m50.276310015s) [noise=true , address=45.91.116.4:4937 , piece_id=VVXAX3UIDDFNTRAGMG2WSPTPJHXMQX3QJF2NYDY77E2BWNVZKN6A, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236938962 (22h49m50.276512082s) [noise=true , address=45.91.116.7:4009 , piece_id=JNMKW2PJDIF4COW4M5G5EG5AVILORIGP6WDBG4CMO6XVWUBGMNJQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237347714 (22h49m50.276920844s) [noise=true , address=77.90.31.12:4590 , piece_id=KL5A25JVCSNY62ZHGS5AX7ECOVSOU6Y2AXKC2XR5EUG2OOCOTOIA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237629593 (22h49m50.277202713s) [noise=true , address=45.91.116.13:4155 , piece_id=LIVS7QLGIGQIE4BBY3QDKEXA2YOXLUCVX4DCME6DRUYF76AXEDQQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237935188 (22h49m50.277508308s) [noise=true , address=77.90.31.17:4771 , piece_id=ARA7GHD4BTCULSXW27C44BAJH5ELDT2XJ3MDK6M5KJG5LY35FFBA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.238210925 (22h49m50.277784045s) [noise=true , address=45.91.116.5:4299 , piece_id=URTUCQECDZMVGHHYHG5HBE3AQUCJOHAHDWO74DFKMAHXZ3NIULGQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:16.085425613 (22h49m52.124998733s) [upload=20123]: Next returned nil segment Sep 7 12:18:16.085448186 (22h49m52.125021296s) [upload=20123]: Waiting for error group managing segments... Sep 7 12:18:16.382438546 (22h49m52.422011666s) [noise=true , address=45.91.116.5:4298 , piece_id=Y7KFGSE7HWFF33QW3NQ6GGKTDNTKQPPSRNOF3C6EKKNG5PHZ6YBA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.399416514 (22h49m52.438989634s) [noise=true , address=45.91.116.13:4155 , piece_id=LIVS7QLGIGQIE4BBY3QDKEXA2YOXLUCVX4DCME6DRUYF76AXEDQQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.413524900 (22h49m52.45309803s) [noise=true , address=45.91.116.6:4975 , piece_id=J5MP3EAEGKOAZUOIOWI6FFGQVN5JZWWO7F475YCVYDVKGWUBMPNQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.416076483 (22h49m52.455649633s) [noise=true , address=45.91.116.11:4118 , piece_id=E5LRFS4MKVCFO5YQSOXRYARST5SRLZZWJVJO5V3DH4VZZL2AKZGQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.427664638 (22h49m52.467237778s) [noise=true , address=45.91.116.11:4117 , piece_id=47ZZ52FK7HR565KFDSQBJNSMNCONMY7RH7KTBR4R4FMUWOVH7SDA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.430874259 (22h49m52.470447379s) [noise=true , address=45.91.116.13:4154 , piece_id=OTDUTZW26I73ADPVJ3ZPDUI4MYIFJQQI24VA4EGJLS5L3T2PJ5DQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.433357641 (22h49m52.472930761s) [noise=true , address=45.91.116.7:4009 , piece_id=JNMKW2PJDIF4COW4M5G5EG5AVILORIGP6WDBG4CMO6XVWUBGMNJQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.435201369 (22h49m52.474774479s) [noise=true , address=45.91.116.12:4226 , piece_id=66XXLFWFQ5BOJL3CPTNRHBVPNBO7DSQJSXKY6PFP4OGZWED6CWSQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.435517494 (22h49m52.475090614s) [noise=true , address=45.91.116.5:4297 , piece_id=NH4AMQD4XIDTW57WLTQPXM6S5KM4N3SI7G44C4HGXZGEQKWHJ3FA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.455335767 (22h49m52.494908877s) [noise=true , address=45.91.116.5:4299 , piece_id=URTUCQECDZMVGHHYHG5HBE3AQUCJOHAHDWO74DFKMAHXZ3NIULGQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.458056703 (22h49m52.497629823s) [noise=true , address=45.91.116.7:4011 , piece_id=YWBVLWEYCCPEC7BJXZJAQO66NMEBOHY57KJR4G2DYX36JAYGAF3Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.506740612 (22h49m52.546313722s) [noise=true , address=45.91.116.4:4937 , piece_id=VVXAX3UIDDFNTRAGMG2WSPTPJHXMQX3QJF2NYDY77E2BWNVZKN6A, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.524160735 (22h49m52.563733855s) [noise=true , address=45.91.116.9:4263 , piece_id=CACXWKVZN72HAWYJUZXS7L6RJMIQ4GPW5NEXBCBC4M7BNKGAN66Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.524187065 (22h49m52.563760185s) [seg_pos={0 0}, upload=20123]: Segment reached optimal threshold of 13 pieces. Sep 7 12:18:16.524292918 (22h49m52.563866038s) [noise=true , address=45.91.116.12:4227 , piece_id=2VE4AKBVU6EJMEIYWZODUTUZMDSLSQSSXSZQH7FDFR6MWEIW3I7Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524351069 (22h49m52.563924189s) [noise=true , address=77.90.31.12:4590 , piece_id=KL5A25JVCSNY62ZHGS5AX7ECOVSOU6Y2AXKC2XR5EUG2OOCOTOIA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524431704 (22h49m52.564004814s) [noise=true , address=77.90.31.17:4771 , piece_id=ARA7GHD4BTCULSXW27C44BAJH5ELDT2XJ3MDK6M5KJG5LY35FFBA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524477071 (22h49m52.564050191s) [seg_pos={0 0}, upload=20123]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:18:16.524563867 (22h49m52.564136987s) [upload=20123]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:18:16.524569207 (22h49m52.564142327s) [upload=20123]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:18:16.874668283 (22h49m52.914241404s) [upload=20123]: Done uploading segments... Sep 7 12:18:16.874686709 (22h49m52.914259829s) [upload=20123]: Upload finished. err: <nil> Duration: 10.147342164s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X...........X.....X....................] duration: 10.41627204s FAIL (2025-09-07 12:31:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:18:13.478410628 (22h49m49.517983748s) [upload=20123]: Starting upload Sep 7 12:18:13.478510909 (22h49m49.518084029s) [upload=20123]: Done starting upload Sep 7 12:18:13.478657580 (22h49m49.5182307s) [upload=20123]: Uploading segments... Sep 7 12:18:13.478782378 (22h49m49.518355548s) [upload=20123]: Got next segment. Inline: false Sep 7 12:18:13.478835610 (22h49m49.51840874s) [upload=20123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:18:13.478843154 (22h49m49.518416274s) [upload=20123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:18:14.234039783 (22h49m50.273612903s) [seg_pos={0 0}, upload=20123]: Begin upload segment... Sep 7 12:18:14.234085281 (22h49m50.273658391s) [seg_pos={0 0}, upload=20123]: Done begin upload segment. Sep 7 12:18:14.234094378 (22h49m50.273667488s) [noise=true , address=45.91.116.13:4154 , piece_id=OTDUTZW26I73ADPVJ3ZPDUI4MYIFJQQI24VA4EGJLS5L3T2PJ5DQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234103195 (22h49m50.273676315s) [noise=true , address=45.91.116.11:4118 , piece_id=E5LRFS4MKVCFO5YQSOXRYARST5SRLZZWJVJO5V3DH4VZZL2AKZGQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234504242 (22h49m50.274077402s) [noise=true , address=45.91.116.12:4226 , piece_id=66XXLFWFQ5BOJL3CPTNRHBVPNBO7DSQJSXKY6PFP4OGZWED6CWSQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234967248 (22h49m50.274540368s) [noise=true , address=45.91.116.5:4298 , piece_id=Y7KFGSE7HWFF33QW3NQ6GGKTDNTKQPPSRNOF3C6EKKNG5PHZ6YBA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235273535 (22h49m50.274846655s) [noise=true , address=45.91.116.7:4011 , piece_id=YWBVLWEYCCPEC7BJXZJAQO66NMEBOHY57KJR4G2DYX36JAYGAF3Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235545195 (22h49m50.275118305s) [noise=true , address=45.91.116.11:4117 , piece_id=47ZZ52FK7HR565KFDSQBJNSMNCONMY7RH7KTBR4R4FMUWOVH7SDA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235830380 (22h49m50.2754035s) [noise=true , address=45.91.116.5:4297 , piece_id=NH4AMQD4XIDTW57WLTQPXM6S5KM4N3SI7G44C4HGXZGEQKWHJ3FA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236049159 (22h49m50.275622279s) [noise=true , address=45.91.116.9:4263 , piece_id=CACXWKVZN72HAWYJUZXS7L6RJMIQ4GPW5NEXBCBC4M7BNKGAN66Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236280301 (22h49m50.275853421s) [noise=true , address=45.91.116.6:4975 , piece_id=J5MP3EAEGKOAZUOIOWI6FFGQVN5JZWWO7F475YCVYDVKGWUBMPNQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236514008 (22h49m50.276087128s) [noise=true , address=45.91.116.12:4227 , piece_id=2VE4AKBVU6EJMEIYWZODUTUZMDSLSQSSXSZQH7FDFR6MWEIW3I7Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236736895 (22h49m50.276310015s) [noise=true , address=45.91.116.4:4937 , piece_id=VVXAX3UIDDFNTRAGMG2WSPTPJHXMQX3QJF2NYDY77E2BWNVZKN6A, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236938962 (22h49m50.276512082s) [noise=true , address=45.91.116.7:4009 , piece_id=JNMKW2PJDIF4COW4M5G5EG5AVILORIGP6WDBG4CMO6XVWUBGMNJQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237347714 (22h49m50.276920844s) [noise=true , address=77.90.31.12:4590 , piece_id=KL5A25JVCSNY62ZHGS5AX7ECOVSOU6Y2AXKC2XR5EUG2OOCOTOIA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237629593 (22h49m50.277202713s) [noise=true , address=45.91.116.13:4155 , piece_id=LIVS7QLGIGQIE4BBY3QDKEXA2YOXLUCVX4DCME6DRUYF76AXEDQQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237935188 (22h49m50.277508308s) [noise=true , address=77.90.31.17:4771 , piece_id=ARA7GHD4BTCULSXW27C44BAJH5ELDT2XJ3MDK6M5KJG5LY35FFBA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.238210925 (22h49m50.277784045s) [noise=true , address=45.91.116.5:4299 , piece_id=URTUCQECDZMVGHHYHG5HBE3AQUCJOHAHDWO74DFKMAHXZ3NIULGQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:16.085425613 (22h49m52.124998733s) [upload=20123]: Next returned nil segment Sep 7 12:18:16.085448186 (22h49m52.125021296s) [upload=20123]: Waiting for error group managing segments... Sep 7 12:18:16.382438546 (22h49m52.422011666s) [noise=true , address=45.91.116.5:4298 , piece_id=Y7KFGSE7HWFF33QW3NQ6GGKTDNTKQPPSRNOF3C6EKKNG5PHZ6YBA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.399416514 (22h49m52.438989634s) [noise=true , address=45.91.116.13:4155 , piece_id=LIVS7QLGIGQIE4BBY3QDKEXA2YOXLUCVX4DCME6DRUYF76AXEDQQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.413524900 (22h49m52.45309803s) [noise=true , address=45.91.116.6:4975 , piece_id=J5MP3EAEGKOAZUOIOWI6FFGQVN5JZWWO7F475YCVYDVKGWUBMPNQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.416076483 (22h49m52.455649633s) [noise=true , address=45.91.116.11:4118 , piece_id=E5LRFS4MKVCFO5YQSOXRYARST5SRLZZWJVJO5V3DH4VZZL2AKZGQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.427664638 (22h49m52.467237778s) [noise=true , address=45.91.116.11:4117 , piece_id=47ZZ52FK7HR565KFDSQBJNSMNCONMY7RH7KTBR4R4FMUWOVH7SDA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.430874259 (22h49m52.470447379s) [noise=true , address=45.91.116.13:4154 , piece_id=OTDUTZW26I73ADPVJ3ZPDUI4MYIFJQQI24VA4EGJLS5L3T2PJ5DQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.433357641 (22h49m52.472930761s) [noise=true , address=45.91.116.7:4009 , piece_id=JNMKW2PJDIF4COW4M5G5EG5AVILORIGP6WDBG4CMO6XVWUBGMNJQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.435201369 (22h49m52.474774479s) [noise=true , address=45.91.116.12:4226 , piece_id=66XXLFWFQ5BOJL3CPTNRHBVPNBO7DSQJSXKY6PFP4OGZWED6CWSQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.435517494 (22h49m52.475090614s) [noise=true , address=45.91.116.5:4297 , piece_id=NH4AMQD4XIDTW57WLTQPXM6S5KM4N3SI7G44C4HGXZGEQKWHJ3FA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.455335767 (22h49m52.494908877s) [noise=true , address=45.91.116.5:4299 , piece_id=URTUCQECDZMVGHHYHG5HBE3AQUCJOHAHDWO74DFKMAHXZ3NIULGQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.458056703 (22h49m52.497629823s) [noise=true , address=45.91.116.7:4011 , piece_id=YWBVLWEYCCPEC7BJXZJAQO66NMEBOHY57KJR4G2DYX36JAYGAF3Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.506740612 (22h49m52.546313722s) [noise=true , address=45.91.116.4:4937 , piece_id=VVXAX3UIDDFNTRAGMG2WSPTPJHXMQX3QJF2NYDY77E2BWNVZKN6A, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.524160735 (22h49m52.563733855s) [noise=true , address=45.91.116.9:4263 , piece_id=CACXWKVZN72HAWYJUZXS7L6RJMIQ4GPW5NEXBCBC4M7BNKGAN66Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.524187065 (22h49m52.563760185s) [seg_pos={0 0}, upload=20123]: Segment reached optimal threshold of 13 pieces. Sep 7 12:18:16.524292918 (22h49m52.563866038s) [noise=true , address=45.91.116.12:4227 , piece_id=2VE4AKBVU6EJMEIYWZODUTUZMDSLSQSSXSZQH7FDFR6MWEIW3I7Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524351069 (22h49m52.563924189s) [noise=true , address=77.90.31.12:4590 , piece_id=KL5A25JVCSNY62ZHGS5AX7ECOVSOU6Y2AXKC2XR5EUG2OOCOTOIA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524431704 (22h49m52.564004814s) [noise=true , address=77.90.31.17:4771 , piece_id=ARA7GHD4BTCULSXW27C44BAJH5ELDT2XJ3MDK6M5KJG5LY35FFBA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524477071 (22h49m52.564050191s) [seg_pos={0 0}, upload=20123]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:18:16.524563867 (22h49m52.564136987s) [upload=20123]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:18:16.524569207 (22h49m52.564142327s) [upload=20123]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:18:16.874668283 (22h49m52.914241404s) [upload=20123]: Done uploading segments... Sep 7 12:18:16.874686709 (22h49m52.914259829s) [upload=20123]: Upload finished. err: <nil> Duration: 10.416296848s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X...........X.....X...................] duration: 10.100091099s FAIL (2025-09-07 12:30:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:18:13.478410628 (22h49m49.517983748s) [upload=20123]: Starting upload Sep 7 12:18:13.478510909 (22h49m49.518084029s) [upload=20123]: Done starting upload Sep 7 12:18:13.478657580 (22h49m49.5182307s) [upload=20123]: Uploading segments... Sep 7 12:18:13.478782378 (22h49m49.518355548s) [upload=20123]: Got next segment. Inline: false Sep 7 12:18:13.478835610 (22h49m49.51840874s) [upload=20123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:18:13.478843154 (22h49m49.518416274s) [upload=20123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:18:14.234039783 (22h49m50.273612903s) [seg_pos={0 0}, upload=20123]: Begin upload segment... Sep 7 12:18:14.234085281 (22h49m50.273658391s) [seg_pos={0 0}, upload=20123]: Done begin upload segment. Sep 7 12:18:14.234094378 (22h49m50.273667488s) [noise=true , address=45.91.116.13:4154 , piece_id=OTDUTZW26I73ADPVJ3ZPDUI4MYIFJQQI24VA4EGJLS5L3T2PJ5DQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234103195 (22h49m50.273676315s) [noise=true , address=45.91.116.11:4118 , piece_id=E5LRFS4MKVCFO5YQSOXRYARST5SRLZZWJVJO5V3DH4VZZL2AKZGQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234504242 (22h49m50.274077402s) [noise=true , address=45.91.116.12:4226 , piece_id=66XXLFWFQ5BOJL3CPTNRHBVPNBO7DSQJSXKY6PFP4OGZWED6CWSQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234967248 (22h49m50.274540368s) [noise=true , address=45.91.116.5:4298 , piece_id=Y7KFGSE7HWFF33QW3NQ6GGKTDNTKQPPSRNOF3C6EKKNG5PHZ6YBA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235273535 (22h49m50.274846655s) [noise=true , address=45.91.116.7:4011 , piece_id=YWBVLWEYCCPEC7BJXZJAQO66NMEBOHY57KJR4G2DYX36JAYGAF3Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235545195 (22h49m50.275118305s) [noise=true , address=45.91.116.11:4117 , piece_id=47ZZ52FK7HR565KFDSQBJNSMNCONMY7RH7KTBR4R4FMUWOVH7SDA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235830380 (22h49m50.2754035s) [noise=true , address=45.91.116.5:4297 , piece_id=NH4AMQD4XIDTW57WLTQPXM6S5KM4N3SI7G44C4HGXZGEQKWHJ3FA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236049159 (22h49m50.275622279s) [noise=true , address=45.91.116.9:4263 , piece_id=CACXWKVZN72HAWYJUZXS7L6RJMIQ4GPW5NEXBCBC4M7BNKGAN66Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236280301 (22h49m50.275853421s) [noise=true , address=45.91.116.6:4975 , piece_id=J5MP3EAEGKOAZUOIOWI6FFGQVN5JZWWO7F475YCVYDVKGWUBMPNQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236514008 (22h49m50.276087128s) [noise=true , address=45.91.116.12:4227 , piece_id=2VE4AKBVU6EJMEIYWZODUTUZMDSLSQSSXSZQH7FDFR6MWEIW3I7Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236736895 (22h49m50.276310015s) [noise=true , address=45.91.116.4:4937 , piece_id=VVXAX3UIDDFNTRAGMG2WSPTPJHXMQX3QJF2NYDY77E2BWNVZKN6A, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236938962 (22h49m50.276512082s) [noise=true , address=45.91.116.7:4009 , piece_id=JNMKW2PJDIF4COW4M5G5EG5AVILORIGP6WDBG4CMO6XVWUBGMNJQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237347714 (22h49m50.276920844s) [noise=true , address=77.90.31.12:4590 , piece_id=KL5A25JVCSNY62ZHGS5AX7ECOVSOU6Y2AXKC2XR5EUG2OOCOTOIA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237629593 (22h49m50.277202713s) [noise=true , address=45.91.116.13:4155 , piece_id=LIVS7QLGIGQIE4BBY3QDKEXA2YOXLUCVX4DCME6DRUYF76AXEDQQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237935188 (22h49m50.277508308s) [noise=true , address=77.90.31.17:4771 , piece_id=ARA7GHD4BTCULSXW27C44BAJH5ELDT2XJ3MDK6M5KJG5LY35FFBA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.238210925 (22h49m50.277784045s) [noise=true , address=45.91.116.5:4299 , piece_id=URTUCQECDZMVGHHYHG5HBE3AQUCJOHAHDWO74DFKMAHXZ3NIULGQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:16.085425613 (22h49m52.124998733s) [upload=20123]: Next returned nil segment Sep 7 12:18:16.085448186 (22h49m52.125021296s) [upload=20123]: Waiting for error group managing segments... Sep 7 12:18:16.382438546 (22h49m52.422011666s) [noise=true , address=45.91.116.5:4298 , piece_id=Y7KFGSE7HWFF33QW3NQ6GGKTDNTKQPPSRNOF3C6EKKNG5PHZ6YBA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.399416514 (22h49m52.438989634s) [noise=true , address=45.91.116.13:4155 , piece_id=LIVS7QLGIGQIE4BBY3QDKEXA2YOXLUCVX4DCME6DRUYF76AXEDQQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.413524900 (22h49m52.45309803s) [noise=true , address=45.91.116.6:4975 , piece_id=J5MP3EAEGKOAZUOIOWI6FFGQVN5JZWWO7F475YCVYDVKGWUBMPNQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.416076483 (22h49m52.455649633s) [noise=true , address=45.91.116.11:4118 , piece_id=E5LRFS4MKVCFO5YQSOXRYARST5SRLZZWJVJO5V3DH4VZZL2AKZGQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.427664638 (22h49m52.467237778s) [noise=true , address=45.91.116.11:4117 , piece_id=47ZZ52FK7HR565KFDSQBJNSMNCONMY7RH7KTBR4R4FMUWOVH7SDA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.430874259 (22h49m52.470447379s) [noise=true , address=45.91.116.13:4154 , piece_id=OTDUTZW26I73ADPVJ3ZPDUI4MYIFJQQI24VA4EGJLS5L3T2PJ5DQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.433357641 (22h49m52.472930761s) [noise=true , address=45.91.116.7:4009 , piece_id=JNMKW2PJDIF4COW4M5G5EG5AVILORIGP6WDBG4CMO6XVWUBGMNJQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.435201369 (22h49m52.474774479s) [noise=true , address=45.91.116.12:4226 , piece_id=66XXLFWFQ5BOJL3CPTNRHBVPNBO7DSQJSXKY6PFP4OGZWED6CWSQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.435517494 (22h49m52.475090614s) [noise=true , address=45.91.116.5:4297 , piece_id=NH4AMQD4XIDTW57WLTQPXM6S5KM4N3SI7G44C4HGXZGEQKWHJ3FA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.455335767 (22h49m52.494908877s) [noise=true , address=45.91.116.5:4299 , piece_id=URTUCQECDZMVGHHYHG5HBE3AQUCJOHAHDWO74DFKMAHXZ3NIULGQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.458056703 (22h49m52.497629823s) [noise=true , address=45.91.116.7:4011 , piece_id=YWBVLWEYCCPEC7BJXZJAQO66NMEBOHY57KJR4G2DYX36JAYGAF3Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.506740612 (22h49m52.546313722s) [noise=true , address=45.91.116.4:4937 , piece_id=VVXAX3UIDDFNTRAGMG2WSPTPJHXMQX3QJF2NYDY77E2BWNVZKN6A, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.524160735 (22h49m52.563733855s) [noise=true , address=45.91.116.9:4263 , piece_id=CACXWKVZN72HAWYJUZXS7L6RJMIQ4GPW5NEXBCBC4M7BNKGAN66Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.524187065 (22h49m52.563760185s) [seg_pos={0 0}, upload=20123]: Segment reached optimal threshold of 13 pieces. Sep 7 12:18:16.524292918 (22h49m52.563866038s) [noise=true , address=45.91.116.12:4227 , piece_id=2VE4AKBVU6EJMEIYWZODUTUZMDSLSQSSXSZQH7FDFR6MWEIW3I7Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524351069 (22h49m52.563924189s) [noise=true , address=77.90.31.12:4590 , piece_id=KL5A25JVCSNY62ZHGS5AX7ECOVSOU6Y2AXKC2XR5EUG2OOCOTOIA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524431704 (22h49m52.564004814s) [noise=true , address=77.90.31.17:4771 , piece_id=ARA7GHD4BTCULSXW27C44BAJH5ELDT2XJ3MDK6M5KJG5LY35FFBA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524477071 (22h49m52.564050191s) [seg_pos={0 0}, upload=20123]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:18:16.524563867 (22h49m52.564136987s) [upload=20123]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:18:16.524569207 (22h49m52.564142327s) [upload=20123]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:18:16.874668283 (22h49m52.914241404s) [upload=20123]: Done uploading segments... Sep 7 12:18:16.874686709 (22h49m52.914259829s) [upload=20123]: Upload finished. err: <nil> Duration: 10.100115246s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X...........X.....X..................] duration: 11.083710543s FAIL (2025-09-07 12:29:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:18:13.478410628 (22h49m49.517983748s) [upload=20123]: Starting upload Sep 7 12:18:13.478510909 (22h49m49.518084029s) [upload=20123]: Done starting upload Sep 7 12:18:13.478657580 (22h49m49.5182307s) [upload=20123]: Uploading segments... Sep 7 12:18:13.478782378 (22h49m49.518355548s) [upload=20123]: Got next segment. Inline: false Sep 7 12:18:13.478835610 (22h49m49.51840874s) [upload=20123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:18:13.478843154 (22h49m49.518416274s) [upload=20123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:18:14.234039783 (22h49m50.273612903s) [seg_pos={0 0}, upload=20123]: Begin upload segment... Sep 7 12:18:14.234085281 (22h49m50.273658391s) [seg_pos={0 0}, upload=20123]: Done begin upload segment. Sep 7 12:18:14.234094378 (22h49m50.273667488s) [noise=true , address=45.91.116.13:4154 , piece_id=OTDUTZW26I73ADPVJ3ZPDUI4MYIFJQQI24VA4EGJLS5L3T2PJ5DQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234103195 (22h49m50.273676315s) [noise=true , address=45.91.116.11:4118 , piece_id=E5LRFS4MKVCFO5YQSOXRYARST5SRLZZWJVJO5V3DH4VZZL2AKZGQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234504242 (22h49m50.274077402s) [noise=true , address=45.91.116.12:4226 , piece_id=66XXLFWFQ5BOJL3CPTNRHBVPNBO7DSQJSXKY6PFP4OGZWED6CWSQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234967248 (22h49m50.274540368s) [noise=true , address=45.91.116.5:4298 , piece_id=Y7KFGSE7HWFF33QW3NQ6GGKTDNTKQPPSRNOF3C6EKKNG5PHZ6YBA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235273535 (22h49m50.274846655s) [noise=true , address=45.91.116.7:4011 , piece_id=YWBVLWEYCCPEC7BJXZJAQO66NMEBOHY57KJR4G2DYX36JAYGAF3Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235545195 (22h49m50.275118305s) [noise=true , address=45.91.116.11:4117 , piece_id=47ZZ52FK7HR565KFDSQBJNSMNCONMY7RH7KTBR4R4FMUWOVH7SDA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235830380 (22h49m50.2754035s) [noise=true , address=45.91.116.5:4297 , piece_id=NH4AMQD4XIDTW57WLTQPXM6S5KM4N3SI7G44C4HGXZGEQKWHJ3FA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236049159 (22h49m50.275622279s) [noise=true , address=45.91.116.9:4263 , piece_id=CACXWKVZN72HAWYJUZXS7L6RJMIQ4GPW5NEXBCBC4M7BNKGAN66Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236280301 (22h49m50.275853421s) [noise=true , address=45.91.116.6:4975 , piece_id=J5MP3EAEGKOAZUOIOWI6FFGQVN5JZWWO7F475YCVYDVKGWUBMPNQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236514008 (22h49m50.276087128s) [noise=true , address=45.91.116.12:4227 , piece_id=2VE4AKBVU6EJMEIYWZODUTUZMDSLSQSSXSZQH7FDFR6MWEIW3I7Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236736895 (22h49m50.276310015s) [noise=true , address=45.91.116.4:4937 , piece_id=VVXAX3UIDDFNTRAGMG2WSPTPJHXMQX3QJF2NYDY77E2BWNVZKN6A, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236938962 (22h49m50.276512082s) [noise=true , address=45.91.116.7:4009 , piece_id=JNMKW2PJDIF4COW4M5G5EG5AVILORIGP6WDBG4CMO6XVWUBGMNJQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237347714 (22h49m50.276920844s) [noise=true , address=77.90.31.12:4590 , piece_id=KL5A25JVCSNY62ZHGS5AX7ECOVSOU6Y2AXKC2XR5EUG2OOCOTOIA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237629593 (22h49m50.277202713s) [noise=true , address=45.91.116.13:4155 , piece_id=LIVS7QLGIGQIE4BBY3QDKEXA2YOXLUCVX4DCME6DRUYF76AXEDQQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237935188 (22h49m50.277508308s) [noise=true , address=77.90.31.17:4771 , piece_id=ARA7GHD4BTCULSXW27C44BAJH5ELDT2XJ3MDK6M5KJG5LY35FFBA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.238210925 (22h49m50.277784045s) [noise=true , address=45.91.116.5:4299 , piece_id=URTUCQECDZMVGHHYHG5HBE3AQUCJOHAHDWO74DFKMAHXZ3NIULGQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:16.085425613 (22h49m52.124998733s) [upload=20123]: Next returned nil segment Sep 7 12:18:16.085448186 (22h49m52.125021296s) [upload=20123]: Waiting for error group managing segments... Sep 7 12:18:16.382438546 (22h49m52.422011666s) [noise=true , address=45.91.116.5:4298 , piece_id=Y7KFGSE7HWFF33QW3NQ6GGKTDNTKQPPSRNOF3C6EKKNG5PHZ6YBA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.399416514 (22h49m52.438989634s) [noise=true , address=45.91.116.13:4155 , piece_id=LIVS7QLGIGQIE4BBY3QDKEXA2YOXLUCVX4DCME6DRUYF76AXEDQQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.413524900 (22h49m52.45309803s) [noise=true , address=45.91.116.6:4975 , piece_id=J5MP3EAEGKOAZUOIOWI6FFGQVN5JZWWO7F475YCVYDVKGWUBMPNQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.416076483 (22h49m52.455649633s) [noise=true , address=45.91.116.11:4118 , piece_id=E5LRFS4MKVCFO5YQSOXRYARST5SRLZZWJVJO5V3DH4VZZL2AKZGQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.427664638 (22h49m52.467237778s) [noise=true , address=45.91.116.11:4117 , piece_id=47ZZ52FK7HR565KFDSQBJNSMNCONMY7RH7KTBR4R4FMUWOVH7SDA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.430874259 (22h49m52.470447379s) [noise=true , address=45.91.116.13:4154 , piece_id=OTDUTZW26I73ADPVJ3ZPDUI4MYIFJQQI24VA4EGJLS5L3T2PJ5DQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.433357641 (22h49m52.472930761s) [noise=true , address=45.91.116.7:4009 , piece_id=JNMKW2PJDIF4COW4M5G5EG5AVILORIGP6WDBG4CMO6XVWUBGMNJQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.435201369 (22h49m52.474774479s) [noise=true , address=45.91.116.12:4226 , piece_id=66XXLFWFQ5BOJL3CPTNRHBVPNBO7DSQJSXKY6PFP4OGZWED6CWSQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.435517494 (22h49m52.475090614s) [noise=true , address=45.91.116.5:4297 , piece_id=NH4AMQD4XIDTW57WLTQPXM6S5KM4N3SI7G44C4HGXZGEQKWHJ3FA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.455335767 (22h49m52.494908877s) [noise=true , address=45.91.116.5:4299 , piece_id=URTUCQECDZMVGHHYHG5HBE3AQUCJOHAHDWO74DFKMAHXZ3NIULGQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.458056703 (22h49m52.497629823s) [noise=true , address=45.91.116.7:4011 , piece_id=YWBVLWEYCCPEC7BJXZJAQO66NMEBOHY57KJR4G2DYX36JAYGAF3Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.506740612 (22h49m52.546313722s) [noise=true , address=45.91.116.4:4937 , piece_id=VVXAX3UIDDFNTRAGMG2WSPTPJHXMQX3QJF2NYDY77E2BWNVZKN6A, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.524160735 (22h49m52.563733855s) [noise=true , address=45.91.116.9:4263 , piece_id=CACXWKVZN72HAWYJUZXS7L6RJMIQ4GPW5NEXBCBC4M7BNKGAN66Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.524187065 (22h49m52.563760185s) [seg_pos={0 0}, upload=20123]: Segment reached optimal threshold of 13 pieces. Sep 7 12:18:16.524292918 (22h49m52.563866038s) [noise=true , address=45.91.116.12:4227 , piece_id=2VE4AKBVU6EJMEIYWZODUTUZMDSLSQSSXSZQH7FDFR6MWEIW3I7Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524351069 (22h49m52.563924189s) [noise=true , address=77.90.31.12:4590 , piece_id=KL5A25JVCSNY62ZHGS5AX7ECOVSOU6Y2AXKC2XR5EUG2OOCOTOIA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524431704 (22h49m52.564004814s) [noise=true , address=77.90.31.17:4771 , piece_id=ARA7GHD4BTCULSXW27C44BAJH5ELDT2XJ3MDK6M5KJG5LY35FFBA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524477071 (22h49m52.564050191s) [seg_pos={0 0}, upload=20123]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:18:16.524563867 (22h49m52.564136987s) [upload=20123]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:18:16.524569207 (22h49m52.564142327s) [upload=20123]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:18:16.874668283 (22h49m52.914241404s) [upload=20123]: Done uploading segments... Sep 7 12:18:16.874686709 (22h49m52.914259829s) [upload=20123]: Upload finished. err: <nil> Duration: 11.083737655s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X...........X.....X.................] duration: 10.450763251s FAIL (2025-09-07 12:28:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:18:13.478410628 (22h49m49.517983748s) [upload=20123]: Starting upload Sep 7 12:18:13.478510909 (22h49m49.518084029s) [upload=20123]: Done starting upload Sep 7 12:18:13.478657580 (22h49m49.5182307s) [upload=20123]: Uploading segments... Sep 7 12:18:13.478782378 (22h49m49.518355548s) [upload=20123]: Got next segment. Inline: false Sep 7 12:18:13.478835610 (22h49m49.51840874s) [upload=20123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:18:13.478843154 (22h49m49.518416274s) [upload=20123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:18:14.234039783 (22h49m50.273612903s) [seg_pos={0 0}, upload=20123]: Begin upload segment... Sep 7 12:18:14.234085281 (22h49m50.273658391s) [seg_pos={0 0}, upload=20123]: Done begin upload segment. Sep 7 12:18:14.234094378 (22h49m50.273667488s) [noise=true , address=45.91.116.13:4154 , piece_id=OTDUTZW26I73ADPVJ3ZPDUI4MYIFJQQI24VA4EGJLS5L3T2PJ5DQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234103195 (22h49m50.273676315s) [noise=true , address=45.91.116.11:4118 , piece_id=E5LRFS4MKVCFO5YQSOXRYARST5SRLZZWJVJO5V3DH4VZZL2AKZGQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234504242 (22h49m50.274077402s) [noise=true , address=45.91.116.12:4226 , piece_id=66XXLFWFQ5BOJL3CPTNRHBVPNBO7DSQJSXKY6PFP4OGZWED6CWSQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234967248 (22h49m50.274540368s) [noise=true , address=45.91.116.5:4298 , piece_id=Y7KFGSE7HWFF33QW3NQ6GGKTDNTKQPPSRNOF3C6EKKNG5PHZ6YBA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235273535 (22h49m50.274846655s) [noise=true , address=45.91.116.7:4011 , piece_id=YWBVLWEYCCPEC7BJXZJAQO66NMEBOHY57KJR4G2DYX36JAYGAF3Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235545195 (22h49m50.275118305s) [noise=true , address=45.91.116.11:4117 , piece_id=47ZZ52FK7HR565KFDSQBJNSMNCONMY7RH7KTBR4R4FMUWOVH7SDA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235830380 (22h49m50.2754035s) [noise=true , address=45.91.116.5:4297 , piece_id=NH4AMQD4XIDTW57WLTQPXM6S5KM4N3SI7G44C4HGXZGEQKWHJ3FA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236049159 (22h49m50.275622279s) [noise=true , address=45.91.116.9:4263 , piece_id=CACXWKVZN72HAWYJUZXS7L6RJMIQ4GPW5NEXBCBC4M7BNKGAN66Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236280301 (22h49m50.275853421s) [noise=true , address=45.91.116.6:4975 , piece_id=J5MP3EAEGKOAZUOIOWI6FFGQVN5JZWWO7F475YCVYDVKGWUBMPNQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236514008 (22h49m50.276087128s) [noise=true , address=45.91.116.12:4227 , piece_id=2VE4AKBVU6EJMEIYWZODUTUZMDSLSQSSXSZQH7FDFR6MWEIW3I7Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236736895 (22h49m50.276310015s) [noise=true , address=45.91.116.4:4937 , piece_id=VVXAX3UIDDFNTRAGMG2WSPTPJHXMQX3QJF2NYDY77E2BWNVZKN6A, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236938962 (22h49m50.276512082s) [noise=true , address=45.91.116.7:4009 , piece_id=JNMKW2PJDIF4COW4M5G5EG5AVILORIGP6WDBG4CMO6XVWUBGMNJQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237347714 (22h49m50.276920844s) [noise=true , address=77.90.31.12:4590 , piece_id=KL5A25JVCSNY62ZHGS5AX7ECOVSOU6Y2AXKC2XR5EUG2OOCOTOIA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237629593 (22h49m50.277202713s) [noise=true , address=45.91.116.13:4155 , piece_id=LIVS7QLGIGQIE4BBY3QDKEXA2YOXLUCVX4DCME6DRUYF76AXEDQQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237935188 (22h49m50.277508308s) [noise=true , address=77.90.31.17:4771 , piece_id=ARA7GHD4BTCULSXW27C44BAJH5ELDT2XJ3MDK6M5KJG5LY35FFBA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.238210925 (22h49m50.277784045s) [noise=true , address=45.91.116.5:4299 , piece_id=URTUCQECDZMVGHHYHG5HBE3AQUCJOHAHDWO74DFKMAHXZ3NIULGQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:16.085425613 (22h49m52.124998733s) [upload=20123]: Next returned nil segment Sep 7 12:18:16.085448186 (22h49m52.125021296s) [upload=20123]: Waiting for error group managing segments... Sep 7 12:18:16.382438546 (22h49m52.422011666s) [noise=true , address=45.91.116.5:4298 , piece_id=Y7KFGSE7HWFF33QW3NQ6GGKTDNTKQPPSRNOF3C6EKKNG5PHZ6YBA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.399416514 (22h49m52.438989634s) [noise=true , address=45.91.116.13:4155 , piece_id=LIVS7QLGIGQIE4BBY3QDKEXA2YOXLUCVX4DCME6DRUYF76AXEDQQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.413524900 (22h49m52.45309803s) [noise=true , address=45.91.116.6:4975 , piece_id=J5MP3EAEGKOAZUOIOWI6FFGQVN5JZWWO7F475YCVYDVKGWUBMPNQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.416076483 (22h49m52.455649633s) [noise=true , address=45.91.116.11:4118 , piece_id=E5LRFS4MKVCFO5YQSOXRYARST5SRLZZWJVJO5V3DH4VZZL2AKZGQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.427664638 (22h49m52.467237778s) [noise=true , address=45.91.116.11:4117 , piece_id=47ZZ52FK7HR565KFDSQBJNSMNCONMY7RH7KTBR4R4FMUWOVH7SDA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.430874259 (22h49m52.470447379s) [noise=true , address=45.91.116.13:4154 , piece_id=OTDUTZW26I73ADPVJ3ZPDUI4MYIFJQQI24VA4EGJLS5L3T2PJ5DQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.433357641 (22h49m52.472930761s) [noise=true , address=45.91.116.7:4009 , piece_id=JNMKW2PJDIF4COW4M5G5EG5AVILORIGP6WDBG4CMO6XVWUBGMNJQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.435201369 (22h49m52.474774479s) [noise=true , address=45.91.116.12:4226 , piece_id=66XXLFWFQ5BOJL3CPTNRHBVPNBO7DSQJSXKY6PFP4OGZWED6CWSQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.435517494 (22h49m52.475090614s) [noise=true , address=45.91.116.5:4297 , piece_id=NH4AMQD4XIDTW57WLTQPXM6S5KM4N3SI7G44C4HGXZGEQKWHJ3FA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.455335767 (22h49m52.494908877s) [noise=true , address=45.91.116.5:4299 , piece_id=URTUCQECDZMVGHHYHG5HBE3AQUCJOHAHDWO74DFKMAHXZ3NIULGQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.458056703 (22h49m52.497629823s) [noise=true , address=45.91.116.7:4011 , piece_id=YWBVLWEYCCPEC7BJXZJAQO66NMEBOHY57KJR4G2DYX36JAYGAF3Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.506740612 (22h49m52.546313722s) [noise=true , address=45.91.116.4:4937 , piece_id=VVXAX3UIDDFNTRAGMG2WSPTPJHXMQX3QJF2NYDY77E2BWNVZKN6A, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.524160735 (22h49m52.563733855s) [noise=true , address=45.91.116.9:4263 , piece_id=CACXWKVZN72HAWYJUZXS7L6RJMIQ4GPW5NEXBCBC4M7BNKGAN66Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.524187065 (22h49m52.563760185s) [seg_pos={0 0}, upload=20123]: Segment reached optimal threshold of 13 pieces. Sep 7 12:18:16.524292918 (22h49m52.563866038s) [noise=true , address=45.91.116.12:4227 , piece_id=2VE4AKBVU6EJMEIYWZODUTUZMDSLSQSSXSZQH7FDFR6MWEIW3I7Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524351069 (22h49m52.563924189s) [noise=true , address=77.90.31.12:4590 , piece_id=KL5A25JVCSNY62ZHGS5AX7ECOVSOU6Y2AXKC2XR5EUG2OOCOTOIA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524431704 (22h49m52.564004814s) [noise=true , address=77.90.31.17:4771 , piece_id=ARA7GHD4BTCULSXW27C44BAJH5ELDT2XJ3MDK6M5KJG5LY35FFBA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524477071 (22h49m52.564050191s) [seg_pos={0 0}, upload=20123]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:18:16.524563867 (22h49m52.564136987s) [upload=20123]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:18:16.524569207 (22h49m52.564142327s) [upload=20123]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:18:16.874668283 (22h49m52.914241404s) [upload=20123]: Done uploading segments... Sep 7 12:18:16.874686709 (22h49m52.914259829s) [upload=20123]: Upload finished. err: <nil> Duration: 10.450792526s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X...........X.....X................] duration: 6.993927205s FAIL (2025-09-07 12:28:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:18:13.478410628 (22h49m49.517983748s) [upload=20123]: Starting upload Sep 7 12:18:13.478510909 (22h49m49.518084029s) [upload=20123]: Done starting upload Sep 7 12:18:13.478657580 (22h49m49.5182307s) [upload=20123]: Uploading segments... Sep 7 12:18:13.478782378 (22h49m49.518355548s) [upload=20123]: Got next segment. Inline: false Sep 7 12:18:13.478835610 (22h49m49.51840874s) [upload=20123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:18:13.478843154 (22h49m49.518416274s) [upload=20123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:18:14.234039783 (22h49m50.273612903s) [seg_pos={0 0}, upload=20123]: Begin upload segment... Sep 7 12:18:14.234085281 (22h49m50.273658391s) [seg_pos={0 0}, upload=20123]: Done begin upload segment. Sep 7 12:18:14.234094378 (22h49m50.273667488s) [noise=true , address=45.91.116.13:4154 , piece_id=OTDUTZW26I73ADPVJ3ZPDUI4MYIFJQQI24VA4EGJLS5L3T2PJ5DQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234103195 (22h49m50.273676315s) [noise=true , address=45.91.116.11:4118 , piece_id=E5LRFS4MKVCFO5YQSOXRYARST5SRLZZWJVJO5V3DH4VZZL2AKZGQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234504242 (22h49m50.274077402s) [noise=true , address=45.91.116.12:4226 , piece_id=66XXLFWFQ5BOJL3CPTNRHBVPNBO7DSQJSXKY6PFP4OGZWED6CWSQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234967248 (22h49m50.274540368s) [noise=true , address=45.91.116.5:4298 , piece_id=Y7KFGSE7HWFF33QW3NQ6GGKTDNTKQPPSRNOF3C6EKKNG5PHZ6YBA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235273535 (22h49m50.274846655s) [noise=true , address=45.91.116.7:4011 , piece_id=YWBVLWEYCCPEC7BJXZJAQO66NMEBOHY57KJR4G2DYX36JAYGAF3Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235545195 (22h49m50.275118305s) [noise=true , address=45.91.116.11:4117 , piece_id=47ZZ52FK7HR565KFDSQBJNSMNCONMY7RH7KTBR4R4FMUWOVH7SDA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235830380 (22h49m50.2754035s) [noise=true , address=45.91.116.5:4297 , piece_id=NH4AMQD4XIDTW57WLTQPXM6S5KM4N3SI7G44C4HGXZGEQKWHJ3FA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236049159 (22h49m50.275622279s) [noise=true , address=45.91.116.9:4263 , piece_id=CACXWKVZN72HAWYJUZXS7L6RJMIQ4GPW5NEXBCBC4M7BNKGAN66Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236280301 (22h49m50.275853421s) [noise=true , address=45.91.116.6:4975 , piece_id=J5MP3EAEGKOAZUOIOWI6FFGQVN5JZWWO7F475YCVYDVKGWUBMPNQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236514008 (22h49m50.276087128s) [noise=true , address=45.91.116.12:4227 , piece_id=2VE4AKBVU6EJMEIYWZODUTUZMDSLSQSSXSZQH7FDFR6MWEIW3I7Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236736895 (22h49m50.276310015s) [noise=true , address=45.91.116.4:4937 , piece_id=VVXAX3UIDDFNTRAGMG2WSPTPJHXMQX3QJF2NYDY77E2BWNVZKN6A, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236938962 (22h49m50.276512082s) [noise=true , address=45.91.116.7:4009 , piece_id=JNMKW2PJDIF4COW4M5G5EG5AVILORIGP6WDBG4CMO6XVWUBGMNJQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237347714 (22h49m50.276920844s) [noise=true , address=77.90.31.12:4590 , piece_id=KL5A25JVCSNY62ZHGS5AX7ECOVSOU6Y2AXKC2XR5EUG2OOCOTOIA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237629593 (22h49m50.277202713s) [noise=true , address=45.91.116.13:4155 , piece_id=LIVS7QLGIGQIE4BBY3QDKEXA2YOXLUCVX4DCME6DRUYF76AXEDQQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237935188 (22h49m50.277508308s) [noise=true , address=77.90.31.17:4771 , piece_id=ARA7GHD4BTCULSXW27C44BAJH5ELDT2XJ3MDK6M5KJG5LY35FFBA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.238210925 (22h49m50.277784045s) [noise=true , address=45.91.116.5:4299 , piece_id=URTUCQECDZMVGHHYHG5HBE3AQUCJOHAHDWO74DFKMAHXZ3NIULGQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:16.085425613 (22h49m52.124998733s) [upload=20123]: Next returned nil segment Sep 7 12:18:16.085448186 (22h49m52.125021296s) [upload=20123]: Waiting for error group managing segments... Sep 7 12:18:16.382438546 (22h49m52.422011666s) [noise=true , address=45.91.116.5:4298 , piece_id=Y7KFGSE7HWFF33QW3NQ6GGKTDNTKQPPSRNOF3C6EKKNG5PHZ6YBA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.399416514 (22h49m52.438989634s) [noise=true , address=45.91.116.13:4155 , piece_id=LIVS7QLGIGQIE4BBY3QDKEXA2YOXLUCVX4DCME6DRUYF76AXEDQQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.413524900 (22h49m52.45309803s) [noise=true , address=45.91.116.6:4975 , piece_id=J5MP3EAEGKOAZUOIOWI6FFGQVN5JZWWO7F475YCVYDVKGWUBMPNQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.416076483 (22h49m52.455649633s) [noise=true , address=45.91.116.11:4118 , piece_id=E5LRFS4MKVCFO5YQSOXRYARST5SRLZZWJVJO5V3DH4VZZL2AKZGQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.427664638 (22h49m52.467237778s) [noise=true , address=45.91.116.11:4117 , piece_id=47ZZ52FK7HR565KFDSQBJNSMNCONMY7RH7KTBR4R4FMUWOVH7SDA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.430874259 (22h49m52.470447379s) [noise=true , address=45.91.116.13:4154 , piece_id=OTDUTZW26I73ADPVJ3ZPDUI4MYIFJQQI24VA4EGJLS5L3T2PJ5DQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.433357641 (22h49m52.472930761s) [noise=true , address=45.91.116.7:4009 , piece_id=JNMKW2PJDIF4COW4M5G5EG5AVILORIGP6WDBG4CMO6XVWUBGMNJQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.435201369 (22h49m52.474774479s) [noise=true , address=45.91.116.12:4226 , piece_id=66XXLFWFQ5BOJL3CPTNRHBVPNBO7DSQJSXKY6PFP4OGZWED6CWSQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.435517494 (22h49m52.475090614s) [noise=true , address=45.91.116.5:4297 , piece_id=NH4AMQD4XIDTW57WLTQPXM6S5KM4N3SI7G44C4HGXZGEQKWHJ3FA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.455335767 (22h49m52.494908877s) [noise=true , address=45.91.116.5:4299 , piece_id=URTUCQECDZMVGHHYHG5HBE3AQUCJOHAHDWO74DFKMAHXZ3NIULGQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.458056703 (22h49m52.497629823s) [noise=true , address=45.91.116.7:4011 , piece_id=YWBVLWEYCCPEC7BJXZJAQO66NMEBOHY57KJR4G2DYX36JAYGAF3Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.506740612 (22h49m52.546313722s) [noise=true , address=45.91.116.4:4937 , piece_id=VVXAX3UIDDFNTRAGMG2WSPTPJHXMQX3QJF2NYDY77E2BWNVZKN6A, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.524160735 (22h49m52.563733855s) [noise=true , address=45.91.116.9:4263 , piece_id=CACXWKVZN72HAWYJUZXS7L6RJMIQ4GPW5NEXBCBC4M7BNKGAN66Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.524187065 (22h49m52.563760185s) [seg_pos={0 0}, upload=20123]: Segment reached optimal threshold of 13 pieces. Sep 7 12:18:16.524292918 (22h49m52.563866038s) [noise=true , address=45.91.116.12:4227 , piece_id=2VE4AKBVU6EJMEIYWZODUTUZMDSLSQSSXSZQH7FDFR6MWEIW3I7Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524351069 (22h49m52.563924189s) [noise=true , address=77.90.31.12:4590 , piece_id=KL5A25JVCSNY62ZHGS5AX7ECOVSOU6Y2AXKC2XR5EUG2OOCOTOIA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524431704 (22h49m52.564004814s) [noise=true , address=77.90.31.17:4771 , piece_id=ARA7GHD4BTCULSXW27C44BAJH5ELDT2XJ3MDK6M5KJG5LY35FFBA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524477071 (22h49m52.564050191s) [seg_pos={0 0}, upload=20123]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:18:16.524563867 (22h49m52.564136987s) [upload=20123]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:18:16.524569207 (22h49m52.564142327s) [upload=20123]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:18:16.874668283 (22h49m52.914241404s) [upload=20123]: Done uploading segments... Sep 7 12:18:16.874686709 (22h49m52.914259829s) [upload=20123]: Upload finished. err: <nil> Duration: 6.993955049s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................X...........X.....X...............] duration: 6.567903755s FAIL (2025-09-07 12:27:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:18:13.478410628 (22h49m49.517983748s) [upload=20123]: Starting upload Sep 7 12:18:13.478510909 (22h49m49.518084029s) [upload=20123]: Done starting upload Sep 7 12:18:13.478657580 (22h49m49.5182307s) [upload=20123]: Uploading segments... Sep 7 12:18:13.478782378 (22h49m49.518355548s) [upload=20123]: Got next segment. Inline: false Sep 7 12:18:13.478835610 (22h49m49.51840874s) [upload=20123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:18:13.478843154 (22h49m49.518416274s) [upload=20123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:18:14.234039783 (22h49m50.273612903s) [seg_pos={0 0}, upload=20123]: Begin upload segment... Sep 7 12:18:14.234085281 (22h49m50.273658391s) [seg_pos={0 0}, upload=20123]: Done begin upload segment. Sep 7 12:18:14.234094378 (22h49m50.273667488s) [noise=true , address=45.91.116.13:4154 , piece_id=OTDUTZW26I73ADPVJ3ZPDUI4MYIFJQQI24VA4EGJLS5L3T2PJ5DQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234103195 (22h49m50.273676315s) [noise=true , address=45.91.116.11:4118 , piece_id=E5LRFS4MKVCFO5YQSOXRYARST5SRLZZWJVJO5V3DH4VZZL2AKZGQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234504242 (22h49m50.274077402s) [noise=true , address=45.91.116.12:4226 , piece_id=66XXLFWFQ5BOJL3CPTNRHBVPNBO7DSQJSXKY6PFP4OGZWED6CWSQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234967248 (22h49m50.274540368s) [noise=true , address=45.91.116.5:4298 , piece_id=Y7KFGSE7HWFF33QW3NQ6GGKTDNTKQPPSRNOF3C6EKKNG5PHZ6YBA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235273535 (22h49m50.274846655s) [noise=true , address=45.91.116.7:4011 , piece_id=YWBVLWEYCCPEC7BJXZJAQO66NMEBOHY57KJR4G2DYX36JAYGAF3Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235545195 (22h49m50.275118305s) [noise=true , address=45.91.116.11:4117 , piece_id=47ZZ52FK7HR565KFDSQBJNSMNCONMY7RH7KTBR4R4FMUWOVH7SDA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235830380 (22h49m50.2754035s) [noise=true , address=45.91.116.5:4297 , piece_id=NH4AMQD4XIDTW57WLTQPXM6S5KM4N3SI7G44C4HGXZGEQKWHJ3FA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236049159 (22h49m50.275622279s) [noise=true , address=45.91.116.9:4263 , piece_id=CACXWKVZN72HAWYJUZXS7L6RJMIQ4GPW5NEXBCBC4M7BNKGAN66Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236280301 (22h49m50.275853421s) [noise=true , address=45.91.116.6:4975 , piece_id=J5MP3EAEGKOAZUOIOWI6FFGQVN5JZWWO7F475YCVYDVKGWUBMPNQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236514008 (22h49m50.276087128s) [noise=true , address=45.91.116.12:4227 , piece_id=2VE4AKBVU6EJMEIYWZODUTUZMDSLSQSSXSZQH7FDFR6MWEIW3I7Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236736895 (22h49m50.276310015s) [noise=true , address=45.91.116.4:4937 , piece_id=VVXAX3UIDDFNTRAGMG2WSPTPJHXMQX3QJF2NYDY77E2BWNVZKN6A, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236938962 (22h49m50.276512082s) [noise=true , address=45.91.116.7:4009 , piece_id=JNMKW2PJDIF4COW4M5G5EG5AVILORIGP6WDBG4CMO6XVWUBGMNJQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237347714 (22h49m50.276920844s) [noise=true , address=77.90.31.12:4590 , piece_id=KL5A25JVCSNY62ZHGS5AX7ECOVSOU6Y2AXKC2XR5EUG2OOCOTOIA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237629593 (22h49m50.277202713s) [noise=true , address=45.91.116.13:4155 , piece_id=LIVS7QLGIGQIE4BBY3QDKEXA2YOXLUCVX4DCME6DRUYF76AXEDQQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237935188 (22h49m50.277508308s) [noise=true , address=77.90.31.17:4771 , piece_id=ARA7GHD4BTCULSXW27C44BAJH5ELDT2XJ3MDK6M5KJG5LY35FFBA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.238210925 (22h49m50.277784045s) [noise=true , address=45.91.116.5:4299 , piece_id=URTUCQECDZMVGHHYHG5HBE3AQUCJOHAHDWO74DFKMAHXZ3NIULGQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:16.085425613 (22h49m52.124998733s) [upload=20123]: Next returned nil segment Sep 7 12:18:16.085448186 (22h49m52.125021296s) [upload=20123]: Waiting for error group managing segments... Sep 7 12:18:16.382438546 (22h49m52.422011666s) [noise=true , address=45.91.116.5:4298 , piece_id=Y7KFGSE7HWFF33QW3NQ6GGKTDNTKQPPSRNOF3C6EKKNG5PHZ6YBA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.399416514 (22h49m52.438989634s) [noise=true , address=45.91.116.13:4155 , piece_id=LIVS7QLGIGQIE4BBY3QDKEXA2YOXLUCVX4DCME6DRUYF76AXEDQQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.413524900 (22h49m52.45309803s) [noise=true , address=45.91.116.6:4975 , piece_id=J5MP3EAEGKOAZUOIOWI6FFGQVN5JZWWO7F475YCVYDVKGWUBMPNQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.416076483 (22h49m52.455649633s) [noise=true , address=45.91.116.11:4118 , piece_id=E5LRFS4MKVCFO5YQSOXRYARST5SRLZZWJVJO5V3DH4VZZL2AKZGQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.427664638 (22h49m52.467237778s) [noise=true , address=45.91.116.11:4117 , piece_id=47ZZ52FK7HR565KFDSQBJNSMNCONMY7RH7KTBR4R4FMUWOVH7SDA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.430874259 (22h49m52.470447379s) [noise=true , address=45.91.116.13:4154 , piece_id=OTDUTZW26I73ADPVJ3ZPDUI4MYIFJQQI24VA4EGJLS5L3T2PJ5DQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.433357641 (22h49m52.472930761s) [noise=true , address=45.91.116.7:4009 , piece_id=JNMKW2PJDIF4COW4M5G5EG5AVILORIGP6WDBG4CMO6XVWUBGMNJQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.435201369 (22h49m52.474774479s) [noise=true , address=45.91.116.12:4226 , piece_id=66XXLFWFQ5BOJL3CPTNRHBVPNBO7DSQJSXKY6PFP4OGZWED6CWSQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.435517494 (22h49m52.475090614s) [noise=true , address=45.91.116.5:4297 , piece_id=NH4AMQD4XIDTW57WLTQPXM6S5KM4N3SI7G44C4HGXZGEQKWHJ3FA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.455335767 (22h49m52.494908877s) [noise=true , address=45.91.116.5:4299 , piece_id=URTUCQECDZMVGHHYHG5HBE3AQUCJOHAHDWO74DFKMAHXZ3NIULGQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.458056703 (22h49m52.497629823s) [noise=true , address=45.91.116.7:4011 , piece_id=YWBVLWEYCCPEC7BJXZJAQO66NMEBOHY57KJR4G2DYX36JAYGAF3Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.506740612 (22h49m52.546313722s) [noise=true , address=45.91.116.4:4937 , piece_id=VVXAX3UIDDFNTRAGMG2WSPTPJHXMQX3QJF2NYDY77E2BWNVZKN6A, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.524160735 (22h49m52.563733855s) [noise=true , address=45.91.116.9:4263 , piece_id=CACXWKVZN72HAWYJUZXS7L6RJMIQ4GPW5NEXBCBC4M7BNKGAN66Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.524187065 (22h49m52.563760185s) [seg_pos={0 0}, upload=20123]: Segment reached optimal threshold of 13 pieces. Sep 7 12:18:16.524292918 (22h49m52.563866038s) [noise=true , address=45.91.116.12:4227 , piece_id=2VE4AKBVU6EJMEIYWZODUTUZMDSLSQSSXSZQH7FDFR6MWEIW3I7Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524351069 (22h49m52.563924189s) [noise=true , address=77.90.31.12:4590 , piece_id=KL5A25JVCSNY62ZHGS5AX7ECOVSOU6Y2AXKC2XR5EUG2OOCOTOIA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524431704 (22h49m52.564004814s) [noise=true , address=77.90.31.17:4771 , piece_id=ARA7GHD4BTCULSXW27C44BAJH5ELDT2XJ3MDK6M5KJG5LY35FFBA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524477071 (22h49m52.564050191s) [seg_pos={0 0}, upload=20123]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:18:16.524563867 (22h49m52.564136987s) [upload=20123]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:18:16.524569207 (22h49m52.564142327s) [upload=20123]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:18:16.874668283 (22h49m52.914241404s) [upload=20123]: Done uploading segments... Sep 7 12:18:16.874686709 (22h49m52.914259829s) [upload=20123]: Upload finished. err: <nil> Duration: 6.567935967s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................X...........X.....X..............] duration: 7.252206604s FAIL (2025-09-07 12:26:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:18:13.478410628 (22h49m49.517983748s) [upload=20123]: Starting upload Sep 7 12:18:13.478510909 (22h49m49.518084029s) [upload=20123]: Done starting upload Sep 7 12:18:13.478657580 (22h49m49.5182307s) [upload=20123]: Uploading segments... Sep 7 12:18:13.478782378 (22h49m49.518355548s) [upload=20123]: Got next segment. Inline: false Sep 7 12:18:13.478835610 (22h49m49.51840874s) [upload=20123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:18:13.478843154 (22h49m49.518416274s) [upload=20123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:18:14.234039783 (22h49m50.273612903s) [seg_pos={0 0}, upload=20123]: Begin upload segment... Sep 7 12:18:14.234085281 (22h49m50.273658391s) [seg_pos={0 0}, upload=20123]: Done begin upload segment. Sep 7 12:18:14.234094378 (22h49m50.273667488s) [noise=true , address=45.91.116.13:4154 , piece_id=OTDUTZW26I73ADPVJ3ZPDUI4MYIFJQQI24VA4EGJLS5L3T2PJ5DQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234103195 (22h49m50.273676315s) [noise=true , address=45.91.116.11:4118 , piece_id=E5LRFS4MKVCFO5YQSOXRYARST5SRLZZWJVJO5V3DH4VZZL2AKZGQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234504242 (22h49m50.274077402s) [noise=true , address=45.91.116.12:4226 , piece_id=66XXLFWFQ5BOJL3CPTNRHBVPNBO7DSQJSXKY6PFP4OGZWED6CWSQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234967248 (22h49m50.274540368s) [noise=true , address=45.91.116.5:4298 , piece_id=Y7KFGSE7HWFF33QW3NQ6GGKTDNTKQPPSRNOF3C6EKKNG5PHZ6YBA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235273535 (22h49m50.274846655s) [noise=true , address=45.91.116.7:4011 , piece_id=YWBVLWEYCCPEC7BJXZJAQO66NMEBOHY57KJR4G2DYX36JAYGAF3Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235545195 (22h49m50.275118305s) [noise=true , address=45.91.116.11:4117 , piece_id=47ZZ52FK7HR565KFDSQBJNSMNCONMY7RH7KTBR4R4FMUWOVH7SDA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235830380 (22h49m50.2754035s) [noise=true , address=45.91.116.5:4297 , piece_id=NH4AMQD4XIDTW57WLTQPXM6S5KM4N3SI7G44C4HGXZGEQKWHJ3FA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236049159 (22h49m50.275622279s) [noise=true , address=45.91.116.9:4263 , piece_id=CACXWKVZN72HAWYJUZXS7L6RJMIQ4GPW5NEXBCBC4M7BNKGAN66Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236280301 (22h49m50.275853421s) [noise=true , address=45.91.116.6:4975 , piece_id=J5MP3EAEGKOAZUOIOWI6FFGQVN5JZWWO7F475YCVYDVKGWUBMPNQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236514008 (22h49m50.276087128s) [noise=true , address=45.91.116.12:4227 , piece_id=2VE4AKBVU6EJMEIYWZODUTUZMDSLSQSSXSZQH7FDFR6MWEIW3I7Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236736895 (22h49m50.276310015s) [noise=true , address=45.91.116.4:4937 , piece_id=VVXAX3UIDDFNTRAGMG2WSPTPJHXMQX3QJF2NYDY77E2BWNVZKN6A, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236938962 (22h49m50.276512082s) [noise=true , address=45.91.116.7:4009 , piece_id=JNMKW2PJDIF4COW4M5G5EG5AVILORIGP6WDBG4CMO6XVWUBGMNJQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237347714 (22h49m50.276920844s) [noise=true , address=77.90.31.12:4590 , piece_id=KL5A25JVCSNY62ZHGS5AX7ECOVSOU6Y2AXKC2XR5EUG2OOCOTOIA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237629593 (22h49m50.277202713s) [noise=true , address=45.91.116.13:4155 , piece_id=LIVS7QLGIGQIE4BBY3QDKEXA2YOXLUCVX4DCME6DRUYF76AXEDQQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237935188 (22h49m50.277508308s) [noise=true , address=77.90.31.17:4771 , piece_id=ARA7GHD4BTCULSXW27C44BAJH5ELDT2XJ3MDK6M5KJG5LY35FFBA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.238210925 (22h49m50.277784045s) [noise=true , address=45.91.116.5:4299 , piece_id=URTUCQECDZMVGHHYHG5HBE3AQUCJOHAHDWO74DFKMAHXZ3NIULGQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:16.085425613 (22h49m52.124998733s) [upload=20123]: Next returned nil segment Sep 7 12:18:16.085448186 (22h49m52.125021296s) [upload=20123]: Waiting for error group managing segments... Sep 7 12:18:16.382438546 (22h49m52.422011666s) [noise=true , address=45.91.116.5:4298 , piece_id=Y7KFGSE7HWFF33QW3NQ6GGKTDNTKQPPSRNOF3C6EKKNG5PHZ6YBA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.399416514 (22h49m52.438989634s) [noise=true , address=45.91.116.13:4155 , piece_id=LIVS7QLGIGQIE4BBY3QDKEXA2YOXLUCVX4DCME6DRUYF76AXEDQQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.413524900 (22h49m52.45309803s) [noise=true , address=45.91.116.6:4975 , piece_id=J5MP3EAEGKOAZUOIOWI6FFGQVN5JZWWO7F475YCVYDVKGWUBMPNQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.416076483 (22h49m52.455649633s) [noise=true , address=45.91.116.11:4118 , piece_id=E5LRFS4MKVCFO5YQSOXRYARST5SRLZZWJVJO5V3DH4VZZL2AKZGQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.427664638 (22h49m52.467237778s) [noise=true , address=45.91.116.11:4117 , piece_id=47ZZ52FK7HR565KFDSQBJNSMNCONMY7RH7KTBR4R4FMUWOVH7SDA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.430874259 (22h49m52.470447379s) [noise=true , address=45.91.116.13:4154 , piece_id=OTDUTZW26I73ADPVJ3ZPDUI4MYIFJQQI24VA4EGJLS5L3T2PJ5DQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.433357641 (22h49m52.472930761s) [noise=true , address=45.91.116.7:4009 , piece_id=JNMKW2PJDIF4COW4M5G5EG5AVILORIGP6WDBG4CMO6XVWUBGMNJQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.435201369 (22h49m52.474774479s) [noise=true , address=45.91.116.12:4226 , piece_id=66XXLFWFQ5BOJL3CPTNRHBVPNBO7DSQJSXKY6PFP4OGZWED6CWSQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.435517494 (22h49m52.475090614s) [noise=true , address=45.91.116.5:4297 , piece_id=NH4AMQD4XIDTW57WLTQPXM6S5KM4N3SI7G44C4HGXZGEQKWHJ3FA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.455335767 (22h49m52.494908877s) [noise=true , address=45.91.116.5:4299 , piece_id=URTUCQECDZMVGHHYHG5HBE3AQUCJOHAHDWO74DFKMAHXZ3NIULGQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.458056703 (22h49m52.497629823s) [noise=true , address=45.91.116.7:4011 , piece_id=YWBVLWEYCCPEC7BJXZJAQO66NMEBOHY57KJR4G2DYX36JAYGAF3Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.506740612 (22h49m52.546313722s) [noise=true , address=45.91.116.4:4937 , piece_id=VVXAX3UIDDFNTRAGMG2WSPTPJHXMQX3QJF2NYDY77E2BWNVZKN6A, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.524160735 (22h49m52.563733855s) [noise=true , address=45.91.116.9:4263 , piece_id=CACXWKVZN72HAWYJUZXS7L6RJMIQ4GPW5NEXBCBC4M7BNKGAN66Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.524187065 (22h49m52.563760185s) [seg_pos={0 0}, upload=20123]: Segment reached optimal threshold of 13 pieces. Sep 7 12:18:16.524292918 (22h49m52.563866038s) [noise=true , address=45.91.116.12:4227 , piece_id=2VE4AKBVU6EJMEIYWZODUTUZMDSLSQSSXSZQH7FDFR6MWEIW3I7Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524351069 (22h49m52.563924189s) [noise=true , address=77.90.31.12:4590 , piece_id=KL5A25JVCSNY62ZHGS5AX7ECOVSOU6Y2AXKC2XR5EUG2OOCOTOIA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524431704 (22h49m52.564004814s) [noise=true , address=77.90.31.17:4771 , piece_id=ARA7GHD4BTCULSXW27C44BAJH5ELDT2XJ3MDK6M5KJG5LY35FFBA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524477071 (22h49m52.564050191s) [seg_pos={0 0}, upload=20123]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:18:16.524563867 (22h49m52.564136987s) [upload=20123]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:18:16.524569207 (22h49m52.564142327s) [upload=20123]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:18:16.874668283 (22h49m52.914241404s) [upload=20123]: Done uploading segments... Sep 7 12:18:16.874686709 (22h49m52.914259829s) [upload=20123]: Upload finished. err: <nil> Duration: 7.252227282s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................X...........X.....X.............] duration: 6.910307308s FAIL (2025-09-07 12:26:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:18:13.478410628 (22h49m49.517983748s) [upload=20123]: Starting upload Sep 7 12:18:13.478510909 (22h49m49.518084029s) [upload=20123]: Done starting upload Sep 7 12:18:13.478657580 (22h49m49.5182307s) [upload=20123]: Uploading segments... Sep 7 12:18:13.478782378 (22h49m49.518355548s) [upload=20123]: Got next segment. Inline: false Sep 7 12:18:13.478835610 (22h49m49.51840874s) [upload=20123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:18:13.478843154 (22h49m49.518416274s) [upload=20123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:18:14.234039783 (22h49m50.273612903s) [seg_pos={0 0}, upload=20123]: Begin upload segment... Sep 7 12:18:14.234085281 (22h49m50.273658391s) [seg_pos={0 0}, upload=20123]: Done begin upload segment. Sep 7 12:18:14.234094378 (22h49m50.273667488s) [noise=true , address=45.91.116.13:4154 , piece_id=OTDUTZW26I73ADPVJ3ZPDUI4MYIFJQQI24VA4EGJLS5L3T2PJ5DQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234103195 (22h49m50.273676315s) [noise=true , address=45.91.116.11:4118 , piece_id=E5LRFS4MKVCFO5YQSOXRYARST5SRLZZWJVJO5V3DH4VZZL2AKZGQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234504242 (22h49m50.274077402s) [noise=true , address=45.91.116.12:4226 , piece_id=66XXLFWFQ5BOJL3CPTNRHBVPNBO7DSQJSXKY6PFP4OGZWED6CWSQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234967248 (22h49m50.274540368s) [noise=true , address=45.91.116.5:4298 , piece_id=Y7KFGSE7HWFF33QW3NQ6GGKTDNTKQPPSRNOF3C6EKKNG5PHZ6YBA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235273535 (22h49m50.274846655s) [noise=true , address=45.91.116.7:4011 , piece_id=YWBVLWEYCCPEC7BJXZJAQO66NMEBOHY57KJR4G2DYX36JAYGAF3Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235545195 (22h49m50.275118305s) [noise=true , address=45.91.116.11:4117 , piece_id=47ZZ52FK7HR565KFDSQBJNSMNCONMY7RH7KTBR4R4FMUWOVH7SDA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235830380 (22h49m50.2754035s) [noise=true , address=45.91.116.5:4297 , piece_id=NH4AMQD4XIDTW57WLTQPXM6S5KM4N3SI7G44C4HGXZGEQKWHJ3FA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236049159 (22h49m50.275622279s) [noise=true , address=45.91.116.9:4263 , piece_id=CACXWKVZN72HAWYJUZXS7L6RJMIQ4GPW5NEXBCBC4M7BNKGAN66Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236280301 (22h49m50.275853421s) [noise=true , address=45.91.116.6:4975 , piece_id=J5MP3EAEGKOAZUOIOWI6FFGQVN5JZWWO7F475YCVYDVKGWUBMPNQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236514008 (22h49m50.276087128s) [noise=true , address=45.91.116.12:4227 , piece_id=2VE4AKBVU6EJMEIYWZODUTUZMDSLSQSSXSZQH7FDFR6MWEIW3I7Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236736895 (22h49m50.276310015s) [noise=true , address=45.91.116.4:4937 , piece_id=VVXAX3UIDDFNTRAGMG2WSPTPJHXMQX3QJF2NYDY77E2BWNVZKN6A, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236938962 (22h49m50.276512082s) [noise=true , address=45.91.116.7:4009 , piece_id=JNMKW2PJDIF4COW4M5G5EG5AVILORIGP6WDBG4CMO6XVWUBGMNJQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237347714 (22h49m50.276920844s) [noise=true , address=77.90.31.12:4590 , piece_id=KL5A25JVCSNY62ZHGS5AX7ECOVSOU6Y2AXKC2XR5EUG2OOCOTOIA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237629593 (22h49m50.277202713s) [noise=true , address=45.91.116.13:4155 , piece_id=LIVS7QLGIGQIE4BBY3QDKEXA2YOXLUCVX4DCME6DRUYF76AXEDQQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237935188 (22h49m50.277508308s) [noise=true , address=77.90.31.17:4771 , piece_id=ARA7GHD4BTCULSXW27C44BAJH5ELDT2XJ3MDK6M5KJG5LY35FFBA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.238210925 (22h49m50.277784045s) [noise=true , address=45.91.116.5:4299 , piece_id=URTUCQECDZMVGHHYHG5HBE3AQUCJOHAHDWO74DFKMAHXZ3NIULGQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:16.085425613 (22h49m52.124998733s) [upload=20123]: Next returned nil segment Sep 7 12:18:16.085448186 (22h49m52.125021296s) [upload=20123]: Waiting for error group managing segments... Sep 7 12:18:16.382438546 (22h49m52.422011666s) [noise=true , address=45.91.116.5:4298 , piece_id=Y7KFGSE7HWFF33QW3NQ6GGKTDNTKQPPSRNOF3C6EKKNG5PHZ6YBA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.399416514 (22h49m52.438989634s) [noise=true , address=45.91.116.13:4155 , piece_id=LIVS7QLGIGQIE4BBY3QDKEXA2YOXLUCVX4DCME6DRUYF76AXEDQQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.413524900 (22h49m52.45309803s) [noise=true , address=45.91.116.6:4975 , piece_id=J5MP3EAEGKOAZUOIOWI6FFGQVN5JZWWO7F475YCVYDVKGWUBMPNQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.416076483 (22h49m52.455649633s) [noise=true , address=45.91.116.11:4118 , piece_id=E5LRFS4MKVCFO5YQSOXRYARST5SRLZZWJVJO5V3DH4VZZL2AKZGQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.427664638 (22h49m52.467237778s) [noise=true , address=45.91.116.11:4117 , piece_id=47ZZ52FK7HR565KFDSQBJNSMNCONMY7RH7KTBR4R4FMUWOVH7SDA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.430874259 (22h49m52.470447379s) [noise=true , address=45.91.116.13:4154 , piece_id=OTDUTZW26I73ADPVJ3ZPDUI4MYIFJQQI24VA4EGJLS5L3T2PJ5DQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.433357641 (22h49m52.472930761s) [noise=true , address=45.91.116.7:4009 , piece_id=JNMKW2PJDIF4COW4M5G5EG5AVILORIGP6WDBG4CMO6XVWUBGMNJQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.435201369 (22h49m52.474774479s) [noise=true , address=45.91.116.12:4226 , piece_id=66XXLFWFQ5BOJL3CPTNRHBVPNBO7DSQJSXKY6PFP4OGZWED6CWSQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.435517494 (22h49m52.475090614s) [noise=true , address=45.91.116.5:4297 , piece_id=NH4AMQD4XIDTW57WLTQPXM6S5KM4N3SI7G44C4HGXZGEQKWHJ3FA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.455335767 (22h49m52.494908877s) [noise=true , address=45.91.116.5:4299 , piece_id=URTUCQECDZMVGHHYHG5HBE3AQUCJOHAHDWO74DFKMAHXZ3NIULGQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.458056703 (22h49m52.497629823s) [noise=true , address=45.91.116.7:4011 , piece_id=YWBVLWEYCCPEC7BJXZJAQO66NMEBOHY57KJR4G2DYX36JAYGAF3Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.506740612 (22h49m52.546313722s) [noise=true , address=45.91.116.4:4937 , piece_id=VVXAX3UIDDFNTRAGMG2WSPTPJHXMQX3QJF2NYDY77E2BWNVZKN6A, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.524160735 (22h49m52.563733855s) [noise=true , address=45.91.116.9:4263 , piece_id=CACXWKVZN72HAWYJUZXS7L6RJMIQ4GPW5NEXBCBC4M7BNKGAN66Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.524187065 (22h49m52.563760185s) [seg_pos={0 0}, upload=20123]: Segment reached optimal threshold of 13 pieces. Sep 7 12:18:16.524292918 (22h49m52.563866038s) [noise=true , address=45.91.116.12:4227 , piece_id=2VE4AKBVU6EJMEIYWZODUTUZMDSLSQSSXSZQH7FDFR6MWEIW3I7Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524351069 (22h49m52.563924189s) [noise=true , address=77.90.31.12:4590 , piece_id=KL5A25JVCSNY62ZHGS5AX7ECOVSOU6Y2AXKC2XR5EUG2OOCOTOIA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524431704 (22h49m52.564004814s) [noise=true , address=77.90.31.17:4771 , piece_id=ARA7GHD4BTCULSXW27C44BAJH5ELDT2XJ3MDK6M5KJG5LY35FFBA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524477071 (22h49m52.564050191s) [seg_pos={0 0}, upload=20123]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:18:16.524563867 (22h49m52.564136987s) [upload=20123]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:18:16.524569207 (22h49m52.564142327s) [upload=20123]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:18:16.874668283 (22h49m52.914241404s) [upload=20123]: Done uploading segments... Sep 7 12:18:16.874686709 (22h49m52.914259829s) [upload=20123]: Upload finished. err: <nil> Duration: 6.910329361s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................X...........X.....X............] duration: 9.085899606s FAIL (2025-09-07 12:25:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:18:13.478410628 (22h49m49.517983748s) [upload=20123]: Starting upload Sep 7 12:18:13.478510909 (22h49m49.518084029s) [upload=20123]: Done starting upload Sep 7 12:18:13.478657580 (22h49m49.5182307s) [upload=20123]: Uploading segments... Sep 7 12:18:13.478782378 (22h49m49.518355548s) [upload=20123]: Got next segment. Inline: false Sep 7 12:18:13.478835610 (22h49m49.51840874s) [upload=20123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:18:13.478843154 (22h49m49.518416274s) [upload=20123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:18:14.234039783 (22h49m50.273612903s) [seg_pos={0 0}, upload=20123]: Begin upload segment... Sep 7 12:18:14.234085281 (22h49m50.273658391s) [seg_pos={0 0}, upload=20123]: Done begin upload segment. Sep 7 12:18:14.234094378 (22h49m50.273667488s) [noise=true , address=45.91.116.13:4154 , piece_id=OTDUTZW26I73ADPVJ3ZPDUI4MYIFJQQI24VA4EGJLS5L3T2PJ5DQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234103195 (22h49m50.273676315s) [noise=true , address=45.91.116.11:4118 , piece_id=E5LRFS4MKVCFO5YQSOXRYARST5SRLZZWJVJO5V3DH4VZZL2AKZGQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234504242 (22h49m50.274077402s) [noise=true , address=45.91.116.12:4226 , piece_id=66XXLFWFQ5BOJL3CPTNRHBVPNBO7DSQJSXKY6PFP4OGZWED6CWSQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234967248 (22h49m50.274540368s) [noise=true , address=45.91.116.5:4298 , piece_id=Y7KFGSE7HWFF33QW3NQ6GGKTDNTKQPPSRNOF3C6EKKNG5PHZ6YBA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235273535 (22h49m50.274846655s) [noise=true , address=45.91.116.7:4011 , piece_id=YWBVLWEYCCPEC7BJXZJAQO66NMEBOHY57KJR4G2DYX36JAYGAF3Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235545195 (22h49m50.275118305s) [noise=true , address=45.91.116.11:4117 , piece_id=47ZZ52FK7HR565KFDSQBJNSMNCONMY7RH7KTBR4R4FMUWOVH7SDA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235830380 (22h49m50.2754035s) [noise=true , address=45.91.116.5:4297 , piece_id=NH4AMQD4XIDTW57WLTQPXM6S5KM4N3SI7G44C4HGXZGEQKWHJ3FA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236049159 (22h49m50.275622279s) [noise=true , address=45.91.116.9:4263 , piece_id=CACXWKVZN72HAWYJUZXS7L6RJMIQ4GPW5NEXBCBC4M7BNKGAN66Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236280301 (22h49m50.275853421s) [noise=true , address=45.91.116.6:4975 , piece_id=J5MP3EAEGKOAZUOIOWI6FFGQVN5JZWWO7F475YCVYDVKGWUBMPNQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236514008 (22h49m50.276087128s) [noise=true , address=45.91.116.12:4227 , piece_id=2VE4AKBVU6EJMEIYWZODUTUZMDSLSQSSXSZQH7FDFR6MWEIW3I7Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236736895 (22h49m50.276310015s) [noise=true , address=45.91.116.4:4937 , piece_id=VVXAX3UIDDFNTRAGMG2WSPTPJHXMQX3QJF2NYDY77E2BWNVZKN6A, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236938962 (22h49m50.276512082s) [noise=true , address=45.91.116.7:4009 , piece_id=JNMKW2PJDIF4COW4M5G5EG5AVILORIGP6WDBG4CMO6XVWUBGMNJQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237347714 (22h49m50.276920844s) [noise=true , address=77.90.31.12:4590 , piece_id=KL5A25JVCSNY62ZHGS5AX7ECOVSOU6Y2AXKC2XR5EUG2OOCOTOIA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237629593 (22h49m50.277202713s) [noise=true , address=45.91.116.13:4155 , piece_id=LIVS7QLGIGQIE4BBY3QDKEXA2YOXLUCVX4DCME6DRUYF76AXEDQQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237935188 (22h49m50.277508308s) [noise=true , address=77.90.31.17:4771 , piece_id=ARA7GHD4BTCULSXW27C44BAJH5ELDT2XJ3MDK6M5KJG5LY35FFBA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.238210925 (22h49m50.277784045s) [noise=true , address=45.91.116.5:4299 , piece_id=URTUCQECDZMVGHHYHG5HBE3AQUCJOHAHDWO74DFKMAHXZ3NIULGQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:16.085425613 (22h49m52.124998733s) [upload=20123]: Next returned nil segment Sep 7 12:18:16.085448186 (22h49m52.125021296s) [upload=20123]: Waiting for error group managing segments... Sep 7 12:18:16.382438546 (22h49m52.422011666s) [noise=true , address=45.91.116.5:4298 , piece_id=Y7KFGSE7HWFF33QW3NQ6GGKTDNTKQPPSRNOF3C6EKKNG5PHZ6YBA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.399416514 (22h49m52.438989634s) [noise=true , address=45.91.116.13:4155 , piece_id=LIVS7QLGIGQIE4BBY3QDKEXA2YOXLUCVX4DCME6DRUYF76AXEDQQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.413524900 (22h49m52.45309803s) [noise=true , address=45.91.116.6:4975 , piece_id=J5MP3EAEGKOAZUOIOWI6FFGQVN5JZWWO7F475YCVYDVKGWUBMPNQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.416076483 (22h49m52.455649633s) [noise=true , address=45.91.116.11:4118 , piece_id=E5LRFS4MKVCFO5YQSOXRYARST5SRLZZWJVJO5V3DH4VZZL2AKZGQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.427664638 (22h49m52.467237778s) [noise=true , address=45.91.116.11:4117 , piece_id=47ZZ52FK7HR565KFDSQBJNSMNCONMY7RH7KTBR4R4FMUWOVH7SDA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.430874259 (22h49m52.470447379s) [noise=true , address=45.91.116.13:4154 , piece_id=OTDUTZW26I73ADPVJ3ZPDUI4MYIFJQQI24VA4EGJLS5L3T2PJ5DQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.433357641 (22h49m52.472930761s) [noise=true , address=45.91.116.7:4009 , piece_id=JNMKW2PJDIF4COW4M5G5EG5AVILORIGP6WDBG4CMO6XVWUBGMNJQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.435201369 (22h49m52.474774479s) [noise=true , address=45.91.116.12:4226 , piece_id=66XXLFWFQ5BOJL3CPTNRHBVPNBO7DSQJSXKY6PFP4OGZWED6CWSQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.435517494 (22h49m52.475090614s) [noise=true , address=45.91.116.5:4297 , piece_id=NH4AMQD4XIDTW57WLTQPXM6S5KM4N3SI7G44C4HGXZGEQKWHJ3FA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.455335767 (22h49m52.494908877s) [noise=true , address=45.91.116.5:4299 , piece_id=URTUCQECDZMVGHHYHG5HBE3AQUCJOHAHDWO74DFKMAHXZ3NIULGQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.458056703 (22h49m52.497629823s) [noise=true , address=45.91.116.7:4011 , piece_id=YWBVLWEYCCPEC7BJXZJAQO66NMEBOHY57KJR4G2DYX36JAYGAF3Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.506740612 (22h49m52.546313722s) [noise=true , address=45.91.116.4:4937 , piece_id=VVXAX3UIDDFNTRAGMG2WSPTPJHXMQX3QJF2NYDY77E2BWNVZKN6A, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.524160735 (22h49m52.563733855s) [noise=true , address=45.91.116.9:4263 , piece_id=CACXWKVZN72HAWYJUZXS7L6RJMIQ4GPW5NEXBCBC4M7BNKGAN66Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.524187065 (22h49m52.563760185s) [seg_pos={0 0}, upload=20123]: Segment reached optimal threshold of 13 pieces. Sep 7 12:18:16.524292918 (22h49m52.563866038s) [noise=true , address=45.91.116.12:4227 , piece_id=2VE4AKBVU6EJMEIYWZODUTUZMDSLSQSSXSZQH7FDFR6MWEIW3I7Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524351069 (22h49m52.563924189s) [noise=true , address=77.90.31.12:4590 , piece_id=KL5A25JVCSNY62ZHGS5AX7ECOVSOU6Y2AXKC2XR5EUG2OOCOTOIA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524431704 (22h49m52.564004814s) [noise=true , address=77.90.31.17:4771 , piece_id=ARA7GHD4BTCULSXW27C44BAJH5ELDT2XJ3MDK6M5KJG5LY35FFBA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524477071 (22h49m52.564050191s) [seg_pos={0 0}, upload=20123]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:18:16.524563867 (22h49m52.564136987s) [upload=20123]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:18:16.524569207 (22h49m52.564142327s) [upload=20123]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:18:16.874668283 (22h49m52.914241404s) [upload=20123]: Done uploading segments... Sep 7 12:18:16.874686709 (22h49m52.914259829s) [upload=20123]: Upload finished. err: <nil> Duration: 9.085926077s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................X...........X.....X...........] duration: 12.659049394s FAIL (2025-09-07 12:25:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:18:13.478410628 (22h49m49.517983748s) [upload=20123]: Starting upload Sep 7 12:18:13.478510909 (22h49m49.518084029s) [upload=20123]: Done starting upload Sep 7 12:18:13.478657580 (22h49m49.5182307s) [upload=20123]: Uploading segments... Sep 7 12:18:13.478782378 (22h49m49.518355548s) [upload=20123]: Got next segment. Inline: false Sep 7 12:18:13.478835610 (22h49m49.51840874s) [upload=20123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:18:13.478843154 (22h49m49.518416274s) [upload=20123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:18:14.234039783 (22h49m50.273612903s) [seg_pos={0 0}, upload=20123]: Begin upload segment... Sep 7 12:18:14.234085281 (22h49m50.273658391s) [seg_pos={0 0}, upload=20123]: Done begin upload segment. Sep 7 12:18:14.234094378 (22h49m50.273667488s) [noise=true , address=45.91.116.13:4154 , piece_id=OTDUTZW26I73ADPVJ3ZPDUI4MYIFJQQI24VA4EGJLS5L3T2PJ5DQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234103195 (22h49m50.273676315s) [noise=true , address=45.91.116.11:4118 , piece_id=E5LRFS4MKVCFO5YQSOXRYARST5SRLZZWJVJO5V3DH4VZZL2AKZGQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234504242 (22h49m50.274077402s) [noise=true , address=45.91.116.12:4226 , piece_id=66XXLFWFQ5BOJL3CPTNRHBVPNBO7DSQJSXKY6PFP4OGZWED6CWSQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234967248 (22h49m50.274540368s) [noise=true , address=45.91.116.5:4298 , piece_id=Y7KFGSE7HWFF33QW3NQ6GGKTDNTKQPPSRNOF3C6EKKNG5PHZ6YBA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235273535 (22h49m50.274846655s) [noise=true , address=45.91.116.7:4011 , piece_id=YWBVLWEYCCPEC7BJXZJAQO66NMEBOHY57KJR4G2DYX36JAYGAF3Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235545195 (22h49m50.275118305s) [noise=true , address=45.91.116.11:4117 , piece_id=47ZZ52FK7HR565KFDSQBJNSMNCONMY7RH7KTBR4R4FMUWOVH7SDA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235830380 (22h49m50.2754035s) [noise=true , address=45.91.116.5:4297 , piece_id=NH4AMQD4XIDTW57WLTQPXM6S5KM4N3SI7G44C4HGXZGEQKWHJ3FA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236049159 (22h49m50.275622279s) [noise=true , address=45.91.116.9:4263 , piece_id=CACXWKVZN72HAWYJUZXS7L6RJMIQ4GPW5NEXBCBC4M7BNKGAN66Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236280301 (22h49m50.275853421s) [noise=true , address=45.91.116.6:4975 , piece_id=J5MP3EAEGKOAZUOIOWI6FFGQVN5JZWWO7F475YCVYDVKGWUBMPNQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236514008 (22h49m50.276087128s) [noise=true , address=45.91.116.12:4227 , piece_id=2VE4AKBVU6EJMEIYWZODUTUZMDSLSQSSXSZQH7FDFR6MWEIW3I7Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236736895 (22h49m50.276310015s) [noise=true , address=45.91.116.4:4937 , piece_id=VVXAX3UIDDFNTRAGMG2WSPTPJHXMQX3QJF2NYDY77E2BWNVZKN6A, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236938962 (22h49m50.276512082s) [noise=true , address=45.91.116.7:4009 , piece_id=JNMKW2PJDIF4COW4M5G5EG5AVILORIGP6WDBG4CMO6XVWUBGMNJQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237347714 (22h49m50.276920844s) [noise=true , address=77.90.31.12:4590 , piece_id=KL5A25JVCSNY62ZHGS5AX7ECOVSOU6Y2AXKC2XR5EUG2OOCOTOIA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237629593 (22h49m50.277202713s) [noise=true , address=45.91.116.13:4155 , piece_id=LIVS7QLGIGQIE4BBY3QDKEXA2YOXLUCVX4DCME6DRUYF76AXEDQQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237935188 (22h49m50.277508308s) [noise=true , address=77.90.31.17:4771 , piece_id=ARA7GHD4BTCULSXW27C44BAJH5ELDT2XJ3MDK6M5KJG5LY35FFBA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.238210925 (22h49m50.277784045s) [noise=true , address=45.91.116.5:4299 , piece_id=URTUCQECDZMVGHHYHG5HBE3AQUCJOHAHDWO74DFKMAHXZ3NIULGQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:16.085425613 (22h49m52.124998733s) [upload=20123]: Next returned nil segment Sep 7 12:18:16.085448186 (22h49m52.125021296s) [upload=20123]: Waiting for error group managing segments... Sep 7 12:18:16.382438546 (22h49m52.422011666s) [noise=true , address=45.91.116.5:4298 , piece_id=Y7KFGSE7HWFF33QW3NQ6GGKTDNTKQPPSRNOF3C6EKKNG5PHZ6YBA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.399416514 (22h49m52.438989634s) [noise=true , address=45.91.116.13:4155 , piece_id=LIVS7QLGIGQIE4BBY3QDKEXA2YOXLUCVX4DCME6DRUYF76AXEDQQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.413524900 (22h49m52.45309803s) [noise=true , address=45.91.116.6:4975 , piece_id=J5MP3EAEGKOAZUOIOWI6FFGQVN5JZWWO7F475YCVYDVKGWUBMPNQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.416076483 (22h49m52.455649633s) [noise=true , address=45.91.116.11:4118 , piece_id=E5LRFS4MKVCFO5YQSOXRYARST5SRLZZWJVJO5V3DH4VZZL2AKZGQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.427664638 (22h49m52.467237778s) [noise=true , address=45.91.116.11:4117 , piece_id=47ZZ52FK7HR565KFDSQBJNSMNCONMY7RH7KTBR4R4FMUWOVH7SDA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.430874259 (22h49m52.470447379s) [noise=true , address=45.91.116.13:4154 , piece_id=OTDUTZW26I73ADPVJ3ZPDUI4MYIFJQQI24VA4EGJLS5L3T2PJ5DQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.433357641 (22h49m52.472930761s) [noise=true , address=45.91.116.7:4009 , piece_id=JNMKW2PJDIF4COW4M5G5EG5AVILORIGP6WDBG4CMO6XVWUBGMNJQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.435201369 (22h49m52.474774479s) [noise=true , address=45.91.116.12:4226 , piece_id=66XXLFWFQ5BOJL3CPTNRHBVPNBO7DSQJSXKY6PFP4OGZWED6CWSQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.435517494 (22h49m52.475090614s) [noise=true , address=45.91.116.5:4297 , piece_id=NH4AMQD4XIDTW57WLTQPXM6S5KM4N3SI7G44C4HGXZGEQKWHJ3FA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.455335767 (22h49m52.494908877s) [noise=true , address=45.91.116.5:4299 , piece_id=URTUCQECDZMVGHHYHG5HBE3AQUCJOHAHDWO74DFKMAHXZ3NIULGQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.458056703 (22h49m52.497629823s) [noise=true , address=45.91.116.7:4011 , piece_id=YWBVLWEYCCPEC7BJXZJAQO66NMEBOHY57KJR4G2DYX36JAYGAF3Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.506740612 (22h49m52.546313722s) [noise=true , address=45.91.116.4:4937 , piece_id=VVXAX3UIDDFNTRAGMG2WSPTPJHXMQX3QJF2NYDY77E2BWNVZKN6A, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.524160735 (22h49m52.563733855s) [noise=true , address=45.91.116.9:4263 , piece_id=CACXWKVZN72HAWYJUZXS7L6RJMIQ4GPW5NEXBCBC4M7BNKGAN66Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.524187065 (22h49m52.563760185s) [seg_pos={0 0}, upload=20123]: Segment reached optimal threshold of 13 pieces. Sep 7 12:18:16.524292918 (22h49m52.563866038s) [noise=true , address=45.91.116.12:4227 , piece_id=2VE4AKBVU6EJMEIYWZODUTUZMDSLSQSSXSZQH7FDFR6MWEIW3I7Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524351069 (22h49m52.563924189s) [noise=true , address=77.90.31.12:4590 , piece_id=KL5A25JVCSNY62ZHGS5AX7ECOVSOU6Y2AXKC2XR5EUG2OOCOTOIA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524431704 (22h49m52.564004814s) [noise=true , address=77.90.31.17:4771 , piece_id=ARA7GHD4BTCULSXW27C44BAJH5ELDT2XJ3MDK6M5KJG5LY35FFBA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524477071 (22h49m52.564050191s) [seg_pos={0 0}, upload=20123]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:18:16.524563867 (22h49m52.564136987s) [upload=20123]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:18:16.524569207 (22h49m52.564142327s) [upload=20123]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:18:16.874668283 (22h49m52.914241404s) [upload=20123]: Done uploading segments... Sep 7 12:18:16.874686709 (22h49m52.914259829s) [upload=20123]: Upload finished. err: <nil> Duration: 12.659081256s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................X...........X.....X..........] duration: 26.351096969s FAIL (2025-09-07 12:24:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:18:13.478410628 (22h49m49.517983748s) [upload=20123]: Starting upload Sep 7 12:18:13.478510909 (22h49m49.518084029s) [upload=20123]: Done starting upload Sep 7 12:18:13.478657580 (22h49m49.5182307s) [upload=20123]: Uploading segments... Sep 7 12:18:13.478782378 (22h49m49.518355548s) [upload=20123]: Got next segment. Inline: false Sep 7 12:18:13.478835610 (22h49m49.51840874s) [upload=20123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:18:13.478843154 (22h49m49.518416274s) [upload=20123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:18:14.234039783 (22h49m50.273612903s) [seg_pos={0 0}, upload=20123]: Begin upload segment... Sep 7 12:18:14.234085281 (22h49m50.273658391s) [seg_pos={0 0}, upload=20123]: Done begin upload segment. Sep 7 12:18:14.234094378 (22h49m50.273667488s) [noise=true , address=45.91.116.13:4154 , piece_id=OTDUTZW26I73ADPVJ3ZPDUI4MYIFJQQI24VA4EGJLS5L3T2PJ5DQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234103195 (22h49m50.273676315s) [noise=true , address=45.91.116.11:4118 , piece_id=E5LRFS4MKVCFO5YQSOXRYARST5SRLZZWJVJO5V3DH4VZZL2AKZGQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234504242 (22h49m50.274077402s) [noise=true , address=45.91.116.12:4226 , piece_id=66XXLFWFQ5BOJL3CPTNRHBVPNBO7DSQJSXKY6PFP4OGZWED6CWSQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234967248 (22h49m50.274540368s) [noise=true , address=45.91.116.5:4298 , piece_id=Y7KFGSE7HWFF33QW3NQ6GGKTDNTKQPPSRNOF3C6EKKNG5PHZ6YBA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235273535 (22h49m50.274846655s) [noise=true , address=45.91.116.7:4011 , piece_id=YWBVLWEYCCPEC7BJXZJAQO66NMEBOHY57KJR4G2DYX36JAYGAF3Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235545195 (22h49m50.275118305s) [noise=true , address=45.91.116.11:4117 , piece_id=47ZZ52FK7HR565KFDSQBJNSMNCONMY7RH7KTBR4R4FMUWOVH7SDA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235830380 (22h49m50.2754035s) [noise=true , address=45.91.116.5:4297 , piece_id=NH4AMQD4XIDTW57WLTQPXM6S5KM4N3SI7G44C4HGXZGEQKWHJ3FA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236049159 (22h49m50.275622279s) [noise=true , address=45.91.116.9:4263 , piece_id=CACXWKVZN72HAWYJUZXS7L6RJMIQ4GPW5NEXBCBC4M7BNKGAN66Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236280301 (22h49m50.275853421s) [noise=true , address=45.91.116.6:4975 , piece_id=J5MP3EAEGKOAZUOIOWI6FFGQVN5JZWWO7F475YCVYDVKGWUBMPNQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236514008 (22h49m50.276087128s) [noise=true , address=45.91.116.12:4227 , piece_id=2VE4AKBVU6EJMEIYWZODUTUZMDSLSQSSXSZQH7FDFR6MWEIW3I7Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236736895 (22h49m50.276310015s) [noise=true , address=45.91.116.4:4937 , piece_id=VVXAX3UIDDFNTRAGMG2WSPTPJHXMQX3QJF2NYDY77E2BWNVZKN6A, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236938962 (22h49m50.276512082s) [noise=true , address=45.91.116.7:4009 , piece_id=JNMKW2PJDIF4COW4M5G5EG5AVILORIGP6WDBG4CMO6XVWUBGMNJQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237347714 (22h49m50.276920844s) [noise=true , address=77.90.31.12:4590 , piece_id=KL5A25JVCSNY62ZHGS5AX7ECOVSOU6Y2AXKC2XR5EUG2OOCOTOIA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237629593 (22h49m50.277202713s) [noise=true , address=45.91.116.13:4155 , piece_id=LIVS7QLGIGQIE4BBY3QDKEXA2YOXLUCVX4DCME6DRUYF76AXEDQQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237935188 (22h49m50.277508308s) [noise=true , address=77.90.31.17:4771 , piece_id=ARA7GHD4BTCULSXW27C44BAJH5ELDT2XJ3MDK6M5KJG5LY35FFBA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.238210925 (22h49m50.277784045s) [noise=true , address=45.91.116.5:4299 , piece_id=URTUCQECDZMVGHHYHG5HBE3AQUCJOHAHDWO74DFKMAHXZ3NIULGQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:16.085425613 (22h49m52.124998733s) [upload=20123]: Next returned nil segment Sep 7 12:18:16.085448186 (22h49m52.125021296s) [upload=20123]: Waiting for error group managing segments... Sep 7 12:18:16.382438546 (22h49m52.422011666s) [noise=true , address=45.91.116.5:4298 , piece_id=Y7KFGSE7HWFF33QW3NQ6GGKTDNTKQPPSRNOF3C6EKKNG5PHZ6YBA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.399416514 (22h49m52.438989634s) [noise=true , address=45.91.116.13:4155 , piece_id=LIVS7QLGIGQIE4BBY3QDKEXA2YOXLUCVX4DCME6DRUYF76AXEDQQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.413524900 (22h49m52.45309803s) [noise=true , address=45.91.116.6:4975 , piece_id=J5MP3EAEGKOAZUOIOWI6FFGQVN5JZWWO7F475YCVYDVKGWUBMPNQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.416076483 (22h49m52.455649633s) [noise=true , address=45.91.116.11:4118 , piece_id=E5LRFS4MKVCFO5YQSOXRYARST5SRLZZWJVJO5V3DH4VZZL2AKZGQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.427664638 (22h49m52.467237778s) [noise=true , address=45.91.116.11:4117 , piece_id=47ZZ52FK7HR565KFDSQBJNSMNCONMY7RH7KTBR4R4FMUWOVH7SDA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.430874259 (22h49m52.470447379s) [noise=true , address=45.91.116.13:4154 , piece_id=OTDUTZW26I73ADPVJ3ZPDUI4MYIFJQQI24VA4EGJLS5L3T2PJ5DQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.433357641 (22h49m52.472930761s) [noise=true , address=45.91.116.7:4009 , piece_id=JNMKW2PJDIF4COW4M5G5EG5AVILORIGP6WDBG4CMO6XVWUBGMNJQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.435201369 (22h49m52.474774479s) [noise=true , address=45.91.116.12:4226 , piece_id=66XXLFWFQ5BOJL3CPTNRHBVPNBO7DSQJSXKY6PFP4OGZWED6CWSQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.435517494 (22h49m52.475090614s) [noise=true , address=45.91.116.5:4297 , piece_id=NH4AMQD4XIDTW57WLTQPXM6S5KM4N3SI7G44C4HGXZGEQKWHJ3FA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.455335767 (22h49m52.494908877s) [noise=true , address=45.91.116.5:4299 , piece_id=URTUCQECDZMVGHHYHG5HBE3AQUCJOHAHDWO74DFKMAHXZ3NIULGQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.458056703 (22h49m52.497629823s) [noise=true , address=45.91.116.7:4011 , piece_id=YWBVLWEYCCPEC7BJXZJAQO66NMEBOHY57KJR4G2DYX36JAYGAF3Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.506740612 (22h49m52.546313722s) [noise=true , address=45.91.116.4:4937 , piece_id=VVXAX3UIDDFNTRAGMG2WSPTPJHXMQX3QJF2NYDY77E2BWNVZKN6A, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.524160735 (22h49m52.563733855s) [noise=true , address=45.91.116.9:4263 , piece_id=CACXWKVZN72HAWYJUZXS7L6RJMIQ4GPW5NEXBCBC4M7BNKGAN66Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.524187065 (22h49m52.563760185s) [seg_pos={0 0}, upload=20123]: Segment reached optimal threshold of 13 pieces. Sep 7 12:18:16.524292918 (22h49m52.563866038s) [noise=true , address=45.91.116.12:4227 , piece_id=2VE4AKBVU6EJMEIYWZODUTUZMDSLSQSSXSZQH7FDFR6MWEIW3I7Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524351069 (22h49m52.563924189s) [noise=true , address=77.90.31.12:4590 , piece_id=KL5A25JVCSNY62ZHGS5AX7ECOVSOU6Y2AXKC2XR5EUG2OOCOTOIA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524431704 (22h49m52.564004814s) [noise=true , address=77.90.31.17:4771 , piece_id=ARA7GHD4BTCULSXW27C44BAJH5ELDT2XJ3MDK6M5KJG5LY35FFBA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524477071 (22h49m52.564050191s) [seg_pos={0 0}, upload=20123]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:18:16.524563867 (22h49m52.564136987s) [upload=20123]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:18:16.524569207 (22h49m52.564142327s) [upload=20123]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:18:16.874668283 (22h49m52.914241404s) [upload=20123]: Done uploading segments... Sep 7 12:18:16.874686709 (22h49m52.914259829s) [upload=20123]: Upload finished. err: <nil> Duration: 26.351126426s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................X...........X.....X.........] duration: 14.60966963s FAIL (2025-09-07 12:23:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:18:13.478410628 (22h49m49.517983748s) [upload=20123]: Starting upload Sep 7 12:18:13.478510909 (22h49m49.518084029s) [upload=20123]: Done starting upload Sep 7 12:18:13.478657580 (22h49m49.5182307s) [upload=20123]: Uploading segments... Sep 7 12:18:13.478782378 (22h49m49.518355548s) [upload=20123]: Got next segment. Inline: false Sep 7 12:18:13.478835610 (22h49m49.51840874s) [upload=20123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:18:13.478843154 (22h49m49.518416274s) [upload=20123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:18:14.234039783 (22h49m50.273612903s) [seg_pos={0 0}, upload=20123]: Begin upload segment... Sep 7 12:18:14.234085281 (22h49m50.273658391s) [seg_pos={0 0}, upload=20123]: Done begin upload segment. Sep 7 12:18:14.234094378 (22h49m50.273667488s) [noise=true , address=45.91.116.13:4154 , piece_id=OTDUTZW26I73ADPVJ3ZPDUI4MYIFJQQI24VA4EGJLS5L3T2PJ5DQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234103195 (22h49m50.273676315s) [noise=true , address=45.91.116.11:4118 , piece_id=E5LRFS4MKVCFO5YQSOXRYARST5SRLZZWJVJO5V3DH4VZZL2AKZGQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234504242 (22h49m50.274077402s) [noise=true , address=45.91.116.12:4226 , piece_id=66XXLFWFQ5BOJL3CPTNRHBVPNBO7DSQJSXKY6PFP4OGZWED6CWSQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234967248 (22h49m50.274540368s) [noise=true , address=45.91.116.5:4298 , piece_id=Y7KFGSE7HWFF33QW3NQ6GGKTDNTKQPPSRNOF3C6EKKNG5PHZ6YBA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235273535 (22h49m50.274846655s) [noise=true , address=45.91.116.7:4011 , piece_id=YWBVLWEYCCPEC7BJXZJAQO66NMEBOHY57KJR4G2DYX36JAYGAF3Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235545195 (22h49m50.275118305s) [noise=true , address=45.91.116.11:4117 , piece_id=47ZZ52FK7HR565KFDSQBJNSMNCONMY7RH7KTBR4R4FMUWOVH7SDA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235830380 (22h49m50.2754035s) [noise=true , address=45.91.116.5:4297 , piece_id=NH4AMQD4XIDTW57WLTQPXM6S5KM4N3SI7G44C4HGXZGEQKWHJ3FA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236049159 (22h49m50.275622279s) [noise=true , address=45.91.116.9:4263 , piece_id=CACXWKVZN72HAWYJUZXS7L6RJMIQ4GPW5NEXBCBC4M7BNKGAN66Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236280301 (22h49m50.275853421s) [noise=true , address=45.91.116.6:4975 , piece_id=J5MP3EAEGKOAZUOIOWI6FFGQVN5JZWWO7F475YCVYDVKGWUBMPNQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236514008 (22h49m50.276087128s) [noise=true , address=45.91.116.12:4227 , piece_id=2VE4AKBVU6EJMEIYWZODUTUZMDSLSQSSXSZQH7FDFR6MWEIW3I7Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236736895 (22h49m50.276310015s) [noise=true , address=45.91.116.4:4937 , piece_id=VVXAX3UIDDFNTRAGMG2WSPTPJHXMQX3QJF2NYDY77E2BWNVZKN6A, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236938962 (22h49m50.276512082s) [noise=true , address=45.91.116.7:4009 , piece_id=JNMKW2PJDIF4COW4M5G5EG5AVILORIGP6WDBG4CMO6XVWUBGMNJQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237347714 (22h49m50.276920844s) [noise=true , address=77.90.31.12:4590 , piece_id=KL5A25JVCSNY62ZHGS5AX7ECOVSOU6Y2AXKC2XR5EUG2OOCOTOIA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237629593 (22h49m50.277202713s) [noise=true , address=45.91.116.13:4155 , piece_id=LIVS7QLGIGQIE4BBY3QDKEXA2YOXLUCVX4DCME6DRUYF76AXEDQQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237935188 (22h49m50.277508308s) [noise=true , address=77.90.31.17:4771 , piece_id=ARA7GHD4BTCULSXW27C44BAJH5ELDT2XJ3MDK6M5KJG5LY35FFBA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.238210925 (22h49m50.277784045s) [noise=true , address=45.91.116.5:4299 , piece_id=URTUCQECDZMVGHHYHG5HBE3AQUCJOHAHDWO74DFKMAHXZ3NIULGQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:16.085425613 (22h49m52.124998733s) [upload=20123]: Next returned nil segment Sep 7 12:18:16.085448186 (22h49m52.125021296s) [upload=20123]: Waiting for error group managing segments... Sep 7 12:18:16.382438546 (22h49m52.422011666s) [noise=true , address=45.91.116.5:4298 , piece_id=Y7KFGSE7HWFF33QW3NQ6GGKTDNTKQPPSRNOF3C6EKKNG5PHZ6YBA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.399416514 (22h49m52.438989634s) [noise=true , address=45.91.116.13:4155 , piece_id=LIVS7QLGIGQIE4BBY3QDKEXA2YOXLUCVX4DCME6DRUYF76AXEDQQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.413524900 (22h49m52.45309803s) [noise=true , address=45.91.116.6:4975 , piece_id=J5MP3EAEGKOAZUOIOWI6FFGQVN5JZWWO7F475YCVYDVKGWUBMPNQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.416076483 (22h49m52.455649633s) [noise=true , address=45.91.116.11:4118 , piece_id=E5LRFS4MKVCFO5YQSOXRYARST5SRLZZWJVJO5V3DH4VZZL2AKZGQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.427664638 (22h49m52.467237778s) [noise=true , address=45.91.116.11:4117 , piece_id=47ZZ52FK7HR565KFDSQBJNSMNCONMY7RH7KTBR4R4FMUWOVH7SDA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.430874259 (22h49m52.470447379s) [noise=true , address=45.91.116.13:4154 , piece_id=OTDUTZW26I73ADPVJ3ZPDUI4MYIFJQQI24VA4EGJLS5L3T2PJ5DQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.433357641 (22h49m52.472930761s) [noise=true , address=45.91.116.7:4009 , piece_id=JNMKW2PJDIF4COW4M5G5EG5AVILORIGP6WDBG4CMO6XVWUBGMNJQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.435201369 (22h49m52.474774479s) [noise=true , address=45.91.116.12:4226 , piece_id=66XXLFWFQ5BOJL3CPTNRHBVPNBO7DSQJSXKY6PFP4OGZWED6CWSQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.435517494 (22h49m52.475090614s) [noise=true , address=45.91.116.5:4297 , piece_id=NH4AMQD4XIDTW57WLTQPXM6S5KM4N3SI7G44C4HGXZGEQKWHJ3FA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.455335767 (22h49m52.494908877s) [noise=true , address=45.91.116.5:4299 , piece_id=URTUCQECDZMVGHHYHG5HBE3AQUCJOHAHDWO74DFKMAHXZ3NIULGQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.458056703 (22h49m52.497629823s) [noise=true , address=45.91.116.7:4011 , piece_id=YWBVLWEYCCPEC7BJXZJAQO66NMEBOHY57KJR4G2DYX36JAYGAF3Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.506740612 (22h49m52.546313722s) [noise=true , address=45.91.116.4:4937 , piece_id=VVXAX3UIDDFNTRAGMG2WSPTPJHXMQX3QJF2NYDY77E2BWNVZKN6A, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.524160735 (22h49m52.563733855s) [noise=true , address=45.91.116.9:4263 , piece_id=CACXWKVZN72HAWYJUZXS7L6RJMIQ4GPW5NEXBCBC4M7BNKGAN66Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.524187065 (22h49m52.563760185s) [seg_pos={0 0}, upload=20123]: Segment reached optimal threshold of 13 pieces. Sep 7 12:18:16.524292918 (22h49m52.563866038s) [noise=true , address=45.91.116.12:4227 , piece_id=2VE4AKBVU6EJMEIYWZODUTUZMDSLSQSSXSZQH7FDFR6MWEIW3I7Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524351069 (22h49m52.563924189s) [noise=true , address=77.90.31.12:4590 , piece_id=KL5A25JVCSNY62ZHGS5AX7ECOVSOU6Y2AXKC2XR5EUG2OOCOTOIA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524431704 (22h49m52.564004814s) [noise=true , address=77.90.31.17:4771 , piece_id=ARA7GHD4BTCULSXW27C44BAJH5ELDT2XJ3MDK6M5KJG5LY35FFBA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524477071 (22h49m52.564050191s) [seg_pos={0 0}, upload=20123]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:18:16.524563867 (22h49m52.564136987s) [upload=20123]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:18:16.524569207 (22h49m52.564142327s) [upload=20123]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:18:16.874668283 (22h49m52.914241404s) [upload=20123]: Done uploading segments... Sep 7 12:18:16.874686709 (22h49m52.914259829s) [upload=20123]: Upload finished. err: <nil> Duration: 14.609696352s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................X...........X.....X........] duration: 14.526746534s FAIL (2025-09-07 12:23:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:18:13.478410628 (22h49m49.517983748s) [upload=20123]: Starting upload Sep 7 12:18:13.478510909 (22h49m49.518084029s) [upload=20123]: Done starting upload Sep 7 12:18:13.478657580 (22h49m49.5182307s) [upload=20123]: Uploading segments... Sep 7 12:18:13.478782378 (22h49m49.518355548s) [upload=20123]: Got next segment. Inline: false Sep 7 12:18:13.478835610 (22h49m49.51840874s) [upload=20123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:18:13.478843154 (22h49m49.518416274s) [upload=20123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:18:14.234039783 (22h49m50.273612903s) [seg_pos={0 0}, upload=20123]: Begin upload segment... Sep 7 12:18:14.234085281 (22h49m50.273658391s) [seg_pos={0 0}, upload=20123]: Done begin upload segment. Sep 7 12:18:14.234094378 (22h49m50.273667488s) [noise=true , address=45.91.116.13:4154 , piece_id=OTDUTZW26I73ADPVJ3ZPDUI4MYIFJQQI24VA4EGJLS5L3T2PJ5DQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234103195 (22h49m50.273676315s) [noise=true , address=45.91.116.11:4118 , piece_id=E5LRFS4MKVCFO5YQSOXRYARST5SRLZZWJVJO5V3DH4VZZL2AKZGQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234504242 (22h49m50.274077402s) [noise=true , address=45.91.116.12:4226 , piece_id=66XXLFWFQ5BOJL3CPTNRHBVPNBO7DSQJSXKY6PFP4OGZWED6CWSQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234967248 (22h49m50.274540368s) [noise=true , address=45.91.116.5:4298 , piece_id=Y7KFGSE7HWFF33QW3NQ6GGKTDNTKQPPSRNOF3C6EKKNG5PHZ6YBA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235273535 (22h49m50.274846655s) [noise=true , address=45.91.116.7:4011 , piece_id=YWBVLWEYCCPEC7BJXZJAQO66NMEBOHY57KJR4G2DYX36JAYGAF3Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235545195 (22h49m50.275118305s) [noise=true , address=45.91.116.11:4117 , piece_id=47ZZ52FK7HR565KFDSQBJNSMNCONMY7RH7KTBR4R4FMUWOVH7SDA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235830380 (22h49m50.2754035s) [noise=true , address=45.91.116.5:4297 , piece_id=NH4AMQD4XIDTW57WLTQPXM6S5KM4N3SI7G44C4HGXZGEQKWHJ3FA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236049159 (22h49m50.275622279s) [noise=true , address=45.91.116.9:4263 , piece_id=CACXWKVZN72HAWYJUZXS7L6RJMIQ4GPW5NEXBCBC4M7BNKGAN66Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236280301 (22h49m50.275853421s) [noise=true , address=45.91.116.6:4975 , piece_id=J5MP3EAEGKOAZUOIOWI6FFGQVN5JZWWO7F475YCVYDVKGWUBMPNQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236514008 (22h49m50.276087128s) [noise=true , address=45.91.116.12:4227 , piece_id=2VE4AKBVU6EJMEIYWZODUTUZMDSLSQSSXSZQH7FDFR6MWEIW3I7Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236736895 (22h49m50.276310015s) [noise=true , address=45.91.116.4:4937 , piece_id=VVXAX3UIDDFNTRAGMG2WSPTPJHXMQX3QJF2NYDY77E2BWNVZKN6A, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236938962 (22h49m50.276512082s) [noise=true , address=45.91.116.7:4009 , piece_id=JNMKW2PJDIF4COW4M5G5EG5AVILORIGP6WDBG4CMO6XVWUBGMNJQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237347714 (22h49m50.276920844s) [noise=true , address=77.90.31.12:4590 , piece_id=KL5A25JVCSNY62ZHGS5AX7ECOVSOU6Y2AXKC2XR5EUG2OOCOTOIA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237629593 (22h49m50.277202713s) [noise=true , address=45.91.116.13:4155 , piece_id=LIVS7QLGIGQIE4BBY3QDKEXA2YOXLUCVX4DCME6DRUYF76AXEDQQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237935188 (22h49m50.277508308s) [noise=true , address=77.90.31.17:4771 , piece_id=ARA7GHD4BTCULSXW27C44BAJH5ELDT2XJ3MDK6M5KJG5LY35FFBA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.238210925 (22h49m50.277784045s) [noise=true , address=45.91.116.5:4299 , piece_id=URTUCQECDZMVGHHYHG5HBE3AQUCJOHAHDWO74DFKMAHXZ3NIULGQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:16.085425613 (22h49m52.124998733s) [upload=20123]: Next returned nil segment Sep 7 12:18:16.085448186 (22h49m52.125021296s) [upload=20123]: Waiting for error group managing segments... Sep 7 12:18:16.382438546 (22h49m52.422011666s) [noise=true , address=45.91.116.5:4298 , piece_id=Y7KFGSE7HWFF33QW3NQ6GGKTDNTKQPPSRNOF3C6EKKNG5PHZ6YBA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.399416514 (22h49m52.438989634s) [noise=true , address=45.91.116.13:4155 , piece_id=LIVS7QLGIGQIE4BBY3QDKEXA2YOXLUCVX4DCME6DRUYF76AXEDQQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.413524900 (22h49m52.45309803s) [noise=true , address=45.91.116.6:4975 , piece_id=J5MP3EAEGKOAZUOIOWI6FFGQVN5JZWWO7F475YCVYDVKGWUBMPNQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.416076483 (22h49m52.455649633s) [noise=true , address=45.91.116.11:4118 , piece_id=E5LRFS4MKVCFO5YQSOXRYARST5SRLZZWJVJO5V3DH4VZZL2AKZGQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.427664638 (22h49m52.467237778s) [noise=true , address=45.91.116.11:4117 , piece_id=47ZZ52FK7HR565KFDSQBJNSMNCONMY7RH7KTBR4R4FMUWOVH7SDA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.430874259 (22h49m52.470447379s) [noise=true , address=45.91.116.13:4154 , piece_id=OTDUTZW26I73ADPVJ3ZPDUI4MYIFJQQI24VA4EGJLS5L3T2PJ5DQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.433357641 (22h49m52.472930761s) [noise=true , address=45.91.116.7:4009 , piece_id=JNMKW2PJDIF4COW4M5G5EG5AVILORIGP6WDBG4CMO6XVWUBGMNJQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.435201369 (22h49m52.474774479s) [noise=true , address=45.91.116.12:4226 , piece_id=66XXLFWFQ5BOJL3CPTNRHBVPNBO7DSQJSXKY6PFP4OGZWED6CWSQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.435517494 (22h49m52.475090614s) [noise=true , address=45.91.116.5:4297 , piece_id=NH4AMQD4XIDTW57WLTQPXM6S5KM4N3SI7G44C4HGXZGEQKWHJ3FA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.455335767 (22h49m52.494908877s) [noise=true , address=45.91.116.5:4299 , piece_id=URTUCQECDZMVGHHYHG5HBE3AQUCJOHAHDWO74DFKMAHXZ3NIULGQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.458056703 (22h49m52.497629823s) [noise=true , address=45.91.116.7:4011 , piece_id=YWBVLWEYCCPEC7BJXZJAQO66NMEBOHY57KJR4G2DYX36JAYGAF3Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.506740612 (22h49m52.546313722s) [noise=true , address=45.91.116.4:4937 , piece_id=VVXAX3UIDDFNTRAGMG2WSPTPJHXMQX3QJF2NYDY77E2BWNVZKN6A, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.524160735 (22h49m52.563733855s) [noise=true , address=45.91.116.9:4263 , piece_id=CACXWKVZN72HAWYJUZXS7L6RJMIQ4GPW5NEXBCBC4M7BNKGAN66Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.524187065 (22h49m52.563760185s) [seg_pos={0 0}, upload=20123]: Segment reached optimal threshold of 13 pieces. Sep 7 12:18:16.524292918 (22h49m52.563866038s) [noise=true , address=45.91.116.12:4227 , piece_id=2VE4AKBVU6EJMEIYWZODUTUZMDSLSQSSXSZQH7FDFR6MWEIW3I7Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524351069 (22h49m52.563924189s) [noise=true , address=77.90.31.12:4590 , piece_id=KL5A25JVCSNY62ZHGS5AX7ECOVSOU6Y2AXKC2XR5EUG2OOCOTOIA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524431704 (22h49m52.564004814s) [noise=true , address=77.90.31.17:4771 , piece_id=ARA7GHD4BTCULSXW27C44BAJH5ELDT2XJ3MDK6M5KJG5LY35FFBA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524477071 (22h49m52.564050191s) [seg_pos={0 0}, upload=20123]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:18:16.524563867 (22h49m52.564136987s) [upload=20123]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:18:16.524569207 (22h49m52.564142327s) [upload=20123]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:18:16.874668283 (22h49m52.914241404s) [upload=20123]: Done uploading segments... Sep 7 12:18:16.874686709 (22h49m52.914259829s) [upload=20123]: Upload finished. err: <nil> Duration: 14.526773115s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................X...........X.....X.......] duration: 13.149519081s FAIL (2025-09-07 12:22:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:18:13.478410628 (22h49m49.517983748s) [upload=20123]: Starting upload Sep 7 12:18:13.478510909 (22h49m49.518084029s) [upload=20123]: Done starting upload Sep 7 12:18:13.478657580 (22h49m49.5182307s) [upload=20123]: Uploading segments... Sep 7 12:18:13.478782378 (22h49m49.518355548s) [upload=20123]: Got next segment. Inline: false Sep 7 12:18:13.478835610 (22h49m49.51840874s) [upload=20123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:18:13.478843154 (22h49m49.518416274s) [upload=20123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:18:14.234039783 (22h49m50.273612903s) [seg_pos={0 0}, upload=20123]: Begin upload segment... Sep 7 12:18:14.234085281 (22h49m50.273658391s) [seg_pos={0 0}, upload=20123]: Done begin upload segment. Sep 7 12:18:14.234094378 (22h49m50.273667488s) [noise=true , address=45.91.116.13:4154 , piece_id=OTDUTZW26I73ADPVJ3ZPDUI4MYIFJQQI24VA4EGJLS5L3T2PJ5DQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234103195 (22h49m50.273676315s) [noise=true , address=45.91.116.11:4118 , piece_id=E5LRFS4MKVCFO5YQSOXRYARST5SRLZZWJVJO5V3DH4VZZL2AKZGQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234504242 (22h49m50.274077402s) [noise=true , address=45.91.116.12:4226 , piece_id=66XXLFWFQ5BOJL3CPTNRHBVPNBO7DSQJSXKY6PFP4OGZWED6CWSQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234967248 (22h49m50.274540368s) [noise=true , address=45.91.116.5:4298 , piece_id=Y7KFGSE7HWFF33QW3NQ6GGKTDNTKQPPSRNOF3C6EKKNG5PHZ6YBA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235273535 (22h49m50.274846655s) [noise=true , address=45.91.116.7:4011 , piece_id=YWBVLWEYCCPEC7BJXZJAQO66NMEBOHY57KJR4G2DYX36JAYGAF3Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235545195 (22h49m50.275118305s) [noise=true , address=45.91.116.11:4117 , piece_id=47ZZ52FK7HR565KFDSQBJNSMNCONMY7RH7KTBR4R4FMUWOVH7SDA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235830380 (22h49m50.2754035s) [noise=true , address=45.91.116.5:4297 , piece_id=NH4AMQD4XIDTW57WLTQPXM6S5KM4N3SI7G44C4HGXZGEQKWHJ3FA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236049159 (22h49m50.275622279s) [noise=true , address=45.91.116.9:4263 , piece_id=CACXWKVZN72HAWYJUZXS7L6RJMIQ4GPW5NEXBCBC4M7BNKGAN66Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236280301 (22h49m50.275853421s) [noise=true , address=45.91.116.6:4975 , piece_id=J5MP3EAEGKOAZUOIOWI6FFGQVN5JZWWO7F475YCVYDVKGWUBMPNQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236514008 (22h49m50.276087128s) [noise=true , address=45.91.116.12:4227 , piece_id=2VE4AKBVU6EJMEIYWZODUTUZMDSLSQSSXSZQH7FDFR6MWEIW3I7Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236736895 (22h49m50.276310015s) [noise=true , address=45.91.116.4:4937 , piece_id=VVXAX3UIDDFNTRAGMG2WSPTPJHXMQX3QJF2NYDY77E2BWNVZKN6A, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236938962 (22h49m50.276512082s) [noise=true , address=45.91.116.7:4009 , piece_id=JNMKW2PJDIF4COW4M5G5EG5AVILORIGP6WDBG4CMO6XVWUBGMNJQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237347714 (22h49m50.276920844s) [noise=true , address=77.90.31.12:4590 , piece_id=KL5A25JVCSNY62ZHGS5AX7ECOVSOU6Y2AXKC2XR5EUG2OOCOTOIA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237629593 (22h49m50.277202713s) [noise=true , address=45.91.116.13:4155 , piece_id=LIVS7QLGIGQIE4BBY3QDKEXA2YOXLUCVX4DCME6DRUYF76AXEDQQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237935188 (22h49m50.277508308s) [noise=true , address=77.90.31.17:4771 , piece_id=ARA7GHD4BTCULSXW27C44BAJH5ELDT2XJ3MDK6M5KJG5LY35FFBA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.238210925 (22h49m50.277784045s) [noise=true , address=45.91.116.5:4299 , piece_id=URTUCQECDZMVGHHYHG5HBE3AQUCJOHAHDWO74DFKMAHXZ3NIULGQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:16.085425613 (22h49m52.124998733s) [upload=20123]: Next returned nil segment Sep 7 12:18:16.085448186 (22h49m52.125021296s) [upload=20123]: Waiting for error group managing segments... Sep 7 12:18:16.382438546 (22h49m52.422011666s) [noise=true , address=45.91.116.5:4298 , piece_id=Y7KFGSE7HWFF33QW3NQ6GGKTDNTKQPPSRNOF3C6EKKNG5PHZ6YBA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.399416514 (22h49m52.438989634s) [noise=true , address=45.91.116.13:4155 , piece_id=LIVS7QLGIGQIE4BBY3QDKEXA2YOXLUCVX4DCME6DRUYF76AXEDQQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.413524900 (22h49m52.45309803s) [noise=true , address=45.91.116.6:4975 , piece_id=J5MP3EAEGKOAZUOIOWI6FFGQVN5JZWWO7F475YCVYDVKGWUBMPNQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.416076483 (22h49m52.455649633s) [noise=true , address=45.91.116.11:4118 , piece_id=E5LRFS4MKVCFO5YQSOXRYARST5SRLZZWJVJO5V3DH4VZZL2AKZGQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.427664638 (22h49m52.467237778s) [noise=true , address=45.91.116.11:4117 , piece_id=47ZZ52FK7HR565KFDSQBJNSMNCONMY7RH7KTBR4R4FMUWOVH7SDA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.430874259 (22h49m52.470447379s) [noise=true , address=45.91.116.13:4154 , piece_id=OTDUTZW26I73ADPVJ3ZPDUI4MYIFJQQI24VA4EGJLS5L3T2PJ5DQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.433357641 (22h49m52.472930761s) [noise=true , address=45.91.116.7:4009 , piece_id=JNMKW2PJDIF4COW4M5G5EG5AVILORIGP6WDBG4CMO6XVWUBGMNJQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.435201369 (22h49m52.474774479s) [noise=true , address=45.91.116.12:4226 , piece_id=66XXLFWFQ5BOJL3CPTNRHBVPNBO7DSQJSXKY6PFP4OGZWED6CWSQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.435517494 (22h49m52.475090614s) [noise=true , address=45.91.116.5:4297 , piece_id=NH4AMQD4XIDTW57WLTQPXM6S5KM4N3SI7G44C4HGXZGEQKWHJ3FA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.455335767 (22h49m52.494908877s) [noise=true , address=45.91.116.5:4299 , piece_id=URTUCQECDZMVGHHYHG5HBE3AQUCJOHAHDWO74DFKMAHXZ3NIULGQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.458056703 (22h49m52.497629823s) [noise=true , address=45.91.116.7:4011 , piece_id=YWBVLWEYCCPEC7BJXZJAQO66NMEBOHY57KJR4G2DYX36JAYGAF3Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.506740612 (22h49m52.546313722s) [noise=true , address=45.91.116.4:4937 , piece_id=VVXAX3UIDDFNTRAGMG2WSPTPJHXMQX3QJF2NYDY77E2BWNVZKN6A, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.524160735 (22h49m52.563733855s) [noise=true , address=45.91.116.9:4263 , piece_id=CACXWKVZN72HAWYJUZXS7L6RJMIQ4GPW5NEXBCBC4M7BNKGAN66Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.524187065 (22h49m52.563760185s) [seg_pos={0 0}, upload=20123]: Segment reached optimal threshold of 13 pieces. Sep 7 12:18:16.524292918 (22h49m52.563866038s) [noise=true , address=45.91.116.12:4227 , piece_id=2VE4AKBVU6EJMEIYWZODUTUZMDSLSQSSXSZQH7FDFR6MWEIW3I7Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524351069 (22h49m52.563924189s) [noise=true , address=77.90.31.12:4590 , piece_id=KL5A25JVCSNY62ZHGS5AX7ECOVSOU6Y2AXKC2XR5EUG2OOCOTOIA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524431704 (22h49m52.564004814s) [noise=true , address=77.90.31.17:4771 , piece_id=ARA7GHD4BTCULSXW27C44BAJH5ELDT2XJ3MDK6M5KJG5LY35FFBA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524477071 (22h49m52.564050191s) [seg_pos={0 0}, upload=20123]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:18:16.524563867 (22h49m52.564136987s) [upload=20123]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:18:16.524569207 (22h49m52.564142327s) [upload=20123]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:18:16.874668283 (22h49m52.914241404s) [upload=20123]: Done uploading segments... Sep 7 12:18:16.874686709 (22h49m52.914259829s) [upload=20123]: Upload finished. err: <nil> Duration: 13.149541555s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................X...........X.....X......] duration: 10.259836843s FAIL (2025-09-07 12:21:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:18:13.478410628 (22h49m49.517983748s) [upload=20123]: Starting upload Sep 7 12:18:13.478510909 (22h49m49.518084029s) [upload=20123]: Done starting upload Sep 7 12:18:13.478657580 (22h49m49.5182307s) [upload=20123]: Uploading segments... Sep 7 12:18:13.478782378 (22h49m49.518355548s) [upload=20123]: Got next segment. Inline: false Sep 7 12:18:13.478835610 (22h49m49.51840874s) [upload=20123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:18:13.478843154 (22h49m49.518416274s) [upload=20123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:18:14.234039783 (22h49m50.273612903s) [seg_pos={0 0}, upload=20123]: Begin upload segment... Sep 7 12:18:14.234085281 (22h49m50.273658391s) [seg_pos={0 0}, upload=20123]: Done begin upload segment. Sep 7 12:18:14.234094378 (22h49m50.273667488s) [noise=true , address=45.91.116.13:4154 , piece_id=OTDUTZW26I73ADPVJ3ZPDUI4MYIFJQQI24VA4EGJLS5L3T2PJ5DQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234103195 (22h49m50.273676315s) [noise=true , address=45.91.116.11:4118 , piece_id=E5LRFS4MKVCFO5YQSOXRYARST5SRLZZWJVJO5V3DH4VZZL2AKZGQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234504242 (22h49m50.274077402s) [noise=true , address=45.91.116.12:4226 , piece_id=66XXLFWFQ5BOJL3CPTNRHBVPNBO7DSQJSXKY6PFP4OGZWED6CWSQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234967248 (22h49m50.274540368s) [noise=true , address=45.91.116.5:4298 , piece_id=Y7KFGSE7HWFF33QW3NQ6GGKTDNTKQPPSRNOF3C6EKKNG5PHZ6YBA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235273535 (22h49m50.274846655s) [noise=true , address=45.91.116.7:4011 , piece_id=YWBVLWEYCCPEC7BJXZJAQO66NMEBOHY57KJR4G2DYX36JAYGAF3Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235545195 (22h49m50.275118305s) [noise=true , address=45.91.116.11:4117 , piece_id=47ZZ52FK7HR565KFDSQBJNSMNCONMY7RH7KTBR4R4FMUWOVH7SDA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235830380 (22h49m50.2754035s) [noise=true , address=45.91.116.5:4297 , piece_id=NH4AMQD4XIDTW57WLTQPXM6S5KM4N3SI7G44C4HGXZGEQKWHJ3FA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236049159 (22h49m50.275622279s) [noise=true , address=45.91.116.9:4263 , piece_id=CACXWKVZN72HAWYJUZXS7L6RJMIQ4GPW5NEXBCBC4M7BNKGAN66Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236280301 (22h49m50.275853421s) [noise=true , address=45.91.116.6:4975 , piece_id=J5MP3EAEGKOAZUOIOWI6FFGQVN5JZWWO7F475YCVYDVKGWUBMPNQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236514008 (22h49m50.276087128s) [noise=true , address=45.91.116.12:4227 , piece_id=2VE4AKBVU6EJMEIYWZODUTUZMDSLSQSSXSZQH7FDFR6MWEIW3I7Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236736895 (22h49m50.276310015s) [noise=true , address=45.91.116.4:4937 , piece_id=VVXAX3UIDDFNTRAGMG2WSPTPJHXMQX3QJF2NYDY77E2BWNVZKN6A, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236938962 (22h49m50.276512082s) [noise=true , address=45.91.116.7:4009 , piece_id=JNMKW2PJDIF4COW4M5G5EG5AVILORIGP6WDBG4CMO6XVWUBGMNJQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237347714 (22h49m50.276920844s) [noise=true , address=77.90.31.12:4590 , piece_id=KL5A25JVCSNY62ZHGS5AX7ECOVSOU6Y2AXKC2XR5EUG2OOCOTOIA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237629593 (22h49m50.277202713s) [noise=true , address=45.91.116.13:4155 , piece_id=LIVS7QLGIGQIE4BBY3QDKEXA2YOXLUCVX4DCME6DRUYF76AXEDQQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237935188 (22h49m50.277508308s) [noise=true , address=77.90.31.17:4771 , piece_id=ARA7GHD4BTCULSXW27C44BAJH5ELDT2XJ3MDK6M5KJG5LY35FFBA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.238210925 (22h49m50.277784045s) [noise=true , address=45.91.116.5:4299 , piece_id=URTUCQECDZMVGHHYHG5HBE3AQUCJOHAHDWO74DFKMAHXZ3NIULGQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:16.085425613 (22h49m52.124998733s) [upload=20123]: Next returned nil segment Sep 7 12:18:16.085448186 (22h49m52.125021296s) [upload=20123]: Waiting for error group managing segments... Sep 7 12:18:16.382438546 (22h49m52.422011666s) [noise=true , address=45.91.116.5:4298 , piece_id=Y7KFGSE7HWFF33QW3NQ6GGKTDNTKQPPSRNOF3C6EKKNG5PHZ6YBA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.399416514 (22h49m52.438989634s) [noise=true , address=45.91.116.13:4155 , piece_id=LIVS7QLGIGQIE4BBY3QDKEXA2YOXLUCVX4DCME6DRUYF76AXEDQQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.413524900 (22h49m52.45309803s) [noise=true , address=45.91.116.6:4975 , piece_id=J5MP3EAEGKOAZUOIOWI6FFGQVN5JZWWO7F475YCVYDVKGWUBMPNQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.416076483 (22h49m52.455649633s) [noise=true , address=45.91.116.11:4118 , piece_id=E5LRFS4MKVCFO5YQSOXRYARST5SRLZZWJVJO5V3DH4VZZL2AKZGQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.427664638 (22h49m52.467237778s) [noise=true , address=45.91.116.11:4117 , piece_id=47ZZ52FK7HR565KFDSQBJNSMNCONMY7RH7KTBR4R4FMUWOVH7SDA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.430874259 (22h49m52.470447379s) [noise=true , address=45.91.116.13:4154 , piece_id=OTDUTZW26I73ADPVJ3ZPDUI4MYIFJQQI24VA4EGJLS5L3T2PJ5DQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.433357641 (22h49m52.472930761s) [noise=true , address=45.91.116.7:4009 , piece_id=JNMKW2PJDIF4COW4M5G5EG5AVILORIGP6WDBG4CMO6XVWUBGMNJQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.435201369 (22h49m52.474774479s) [noise=true , address=45.91.116.12:4226 , piece_id=66XXLFWFQ5BOJL3CPTNRHBVPNBO7DSQJSXKY6PFP4OGZWED6CWSQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.435517494 (22h49m52.475090614s) [noise=true , address=45.91.116.5:4297 , piece_id=NH4AMQD4XIDTW57WLTQPXM6S5KM4N3SI7G44C4HGXZGEQKWHJ3FA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.455335767 (22h49m52.494908877s) [noise=true , address=45.91.116.5:4299 , piece_id=URTUCQECDZMVGHHYHG5HBE3AQUCJOHAHDWO74DFKMAHXZ3NIULGQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.458056703 (22h49m52.497629823s) [noise=true , address=45.91.116.7:4011 , piece_id=YWBVLWEYCCPEC7BJXZJAQO66NMEBOHY57KJR4G2DYX36JAYGAF3Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.506740612 (22h49m52.546313722s) [noise=true , address=45.91.116.4:4937 , piece_id=VVXAX3UIDDFNTRAGMG2WSPTPJHXMQX3QJF2NYDY77E2BWNVZKN6A, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.524160735 (22h49m52.563733855s) [noise=true , address=45.91.116.9:4263 , piece_id=CACXWKVZN72HAWYJUZXS7L6RJMIQ4GPW5NEXBCBC4M7BNKGAN66Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.524187065 (22h49m52.563760185s) [seg_pos={0 0}, upload=20123]: Segment reached optimal threshold of 13 pieces. Sep 7 12:18:16.524292918 (22h49m52.563866038s) [noise=true , address=45.91.116.12:4227 , piece_id=2VE4AKBVU6EJMEIYWZODUTUZMDSLSQSSXSZQH7FDFR6MWEIW3I7Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524351069 (22h49m52.563924189s) [noise=true , address=77.90.31.12:4590 , piece_id=KL5A25JVCSNY62ZHGS5AX7ECOVSOU6Y2AXKC2XR5EUG2OOCOTOIA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524431704 (22h49m52.564004814s) [noise=true , address=77.90.31.17:4771 , piece_id=ARA7GHD4BTCULSXW27C44BAJH5ELDT2XJ3MDK6M5KJG5LY35FFBA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524477071 (22h49m52.564050191s) [seg_pos={0 0}, upload=20123]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:18:16.524563867 (22h49m52.564136987s) [upload=20123]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:18:16.524569207 (22h49m52.564142327s) [upload=20123]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:18:16.874668283 (22h49m52.914241404s) [upload=20123]: Done uploading segments... Sep 7 12:18:16.874686709 (22h49m52.914259829s) [upload=20123]: Upload finished. err: <nil> Duration: 10.259865888s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................X...........X.....X.....] duration: 7.147402248s FAIL (2025-09-07 12:21:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:18:13.478410628 (22h49m49.517983748s) [upload=20123]: Starting upload Sep 7 12:18:13.478510909 (22h49m49.518084029s) [upload=20123]: Done starting upload Sep 7 12:18:13.478657580 (22h49m49.5182307s) [upload=20123]: Uploading segments... Sep 7 12:18:13.478782378 (22h49m49.518355548s) [upload=20123]: Got next segment. Inline: false Sep 7 12:18:13.478835610 (22h49m49.51840874s) [upload=20123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:18:13.478843154 (22h49m49.518416274s) [upload=20123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:18:14.234039783 (22h49m50.273612903s) [seg_pos={0 0}, upload=20123]: Begin upload segment... Sep 7 12:18:14.234085281 (22h49m50.273658391s) [seg_pos={0 0}, upload=20123]: Done begin upload segment. Sep 7 12:18:14.234094378 (22h49m50.273667488s) [noise=true , address=45.91.116.13:4154 , piece_id=OTDUTZW26I73ADPVJ3ZPDUI4MYIFJQQI24VA4EGJLS5L3T2PJ5DQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234103195 (22h49m50.273676315s) [noise=true , address=45.91.116.11:4118 , piece_id=E5LRFS4MKVCFO5YQSOXRYARST5SRLZZWJVJO5V3DH4VZZL2AKZGQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234504242 (22h49m50.274077402s) [noise=true , address=45.91.116.12:4226 , piece_id=66XXLFWFQ5BOJL3CPTNRHBVPNBO7DSQJSXKY6PFP4OGZWED6CWSQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234967248 (22h49m50.274540368s) [noise=true , address=45.91.116.5:4298 , piece_id=Y7KFGSE7HWFF33QW3NQ6GGKTDNTKQPPSRNOF3C6EKKNG5PHZ6YBA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235273535 (22h49m50.274846655s) [noise=true , address=45.91.116.7:4011 , piece_id=YWBVLWEYCCPEC7BJXZJAQO66NMEBOHY57KJR4G2DYX36JAYGAF3Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235545195 (22h49m50.275118305s) [noise=true , address=45.91.116.11:4117 , piece_id=47ZZ52FK7HR565KFDSQBJNSMNCONMY7RH7KTBR4R4FMUWOVH7SDA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235830380 (22h49m50.2754035s) [noise=true , address=45.91.116.5:4297 , piece_id=NH4AMQD4XIDTW57WLTQPXM6S5KM4N3SI7G44C4HGXZGEQKWHJ3FA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236049159 (22h49m50.275622279s) [noise=true , address=45.91.116.9:4263 , piece_id=CACXWKVZN72HAWYJUZXS7L6RJMIQ4GPW5NEXBCBC4M7BNKGAN66Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236280301 (22h49m50.275853421s) [noise=true , address=45.91.116.6:4975 , piece_id=J5MP3EAEGKOAZUOIOWI6FFGQVN5JZWWO7F475YCVYDVKGWUBMPNQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236514008 (22h49m50.276087128s) [noise=true , address=45.91.116.12:4227 , piece_id=2VE4AKBVU6EJMEIYWZODUTUZMDSLSQSSXSZQH7FDFR6MWEIW3I7Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236736895 (22h49m50.276310015s) [noise=true , address=45.91.116.4:4937 , piece_id=VVXAX3UIDDFNTRAGMG2WSPTPJHXMQX3QJF2NYDY77E2BWNVZKN6A, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236938962 (22h49m50.276512082s) [noise=true , address=45.91.116.7:4009 , piece_id=JNMKW2PJDIF4COW4M5G5EG5AVILORIGP6WDBG4CMO6XVWUBGMNJQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237347714 (22h49m50.276920844s) [noise=true , address=77.90.31.12:4590 , piece_id=KL5A25JVCSNY62ZHGS5AX7ECOVSOU6Y2AXKC2XR5EUG2OOCOTOIA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237629593 (22h49m50.277202713s) [noise=true , address=45.91.116.13:4155 , piece_id=LIVS7QLGIGQIE4BBY3QDKEXA2YOXLUCVX4DCME6DRUYF76AXEDQQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237935188 (22h49m50.277508308s) [noise=true , address=77.90.31.17:4771 , piece_id=ARA7GHD4BTCULSXW27C44BAJH5ELDT2XJ3MDK6M5KJG5LY35FFBA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.238210925 (22h49m50.277784045s) [noise=true , address=45.91.116.5:4299 , piece_id=URTUCQECDZMVGHHYHG5HBE3AQUCJOHAHDWO74DFKMAHXZ3NIULGQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:16.085425613 (22h49m52.124998733s) [upload=20123]: Next returned nil segment Sep 7 12:18:16.085448186 (22h49m52.125021296s) [upload=20123]: Waiting for error group managing segments... Sep 7 12:18:16.382438546 (22h49m52.422011666s) [noise=true , address=45.91.116.5:4298 , piece_id=Y7KFGSE7HWFF33QW3NQ6GGKTDNTKQPPSRNOF3C6EKKNG5PHZ6YBA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.399416514 (22h49m52.438989634s) [noise=true , address=45.91.116.13:4155 , piece_id=LIVS7QLGIGQIE4BBY3QDKEXA2YOXLUCVX4DCME6DRUYF76AXEDQQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.413524900 (22h49m52.45309803s) [noise=true , address=45.91.116.6:4975 , piece_id=J5MP3EAEGKOAZUOIOWI6FFGQVN5JZWWO7F475YCVYDVKGWUBMPNQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.416076483 (22h49m52.455649633s) [noise=true , address=45.91.116.11:4118 , piece_id=E5LRFS4MKVCFO5YQSOXRYARST5SRLZZWJVJO5V3DH4VZZL2AKZGQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.427664638 (22h49m52.467237778s) [noise=true , address=45.91.116.11:4117 , piece_id=47ZZ52FK7HR565KFDSQBJNSMNCONMY7RH7KTBR4R4FMUWOVH7SDA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.430874259 (22h49m52.470447379s) [noise=true , address=45.91.116.13:4154 , piece_id=OTDUTZW26I73ADPVJ3ZPDUI4MYIFJQQI24VA4EGJLS5L3T2PJ5DQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.433357641 (22h49m52.472930761s) [noise=true , address=45.91.116.7:4009 , piece_id=JNMKW2PJDIF4COW4M5G5EG5AVILORIGP6WDBG4CMO6XVWUBGMNJQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.435201369 (22h49m52.474774479s) [noise=true , address=45.91.116.12:4226 , piece_id=66XXLFWFQ5BOJL3CPTNRHBVPNBO7DSQJSXKY6PFP4OGZWED6CWSQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.435517494 (22h49m52.475090614s) [noise=true , address=45.91.116.5:4297 , piece_id=NH4AMQD4XIDTW57WLTQPXM6S5KM4N3SI7G44C4HGXZGEQKWHJ3FA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.455335767 (22h49m52.494908877s) [noise=true , address=45.91.116.5:4299 , piece_id=URTUCQECDZMVGHHYHG5HBE3AQUCJOHAHDWO74DFKMAHXZ3NIULGQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.458056703 (22h49m52.497629823s) [noise=true , address=45.91.116.7:4011 , piece_id=YWBVLWEYCCPEC7BJXZJAQO66NMEBOHY57KJR4G2DYX36JAYGAF3Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.506740612 (22h49m52.546313722s) [noise=true , address=45.91.116.4:4937 , piece_id=VVXAX3UIDDFNTRAGMG2WSPTPJHXMQX3QJF2NYDY77E2BWNVZKN6A, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.524160735 (22h49m52.563733855s) [noise=true , address=45.91.116.9:4263 , piece_id=CACXWKVZN72HAWYJUZXS7L6RJMIQ4GPW5NEXBCBC4M7BNKGAN66Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.524187065 (22h49m52.563760185s) [seg_pos={0 0}, upload=20123]: Segment reached optimal threshold of 13 pieces. Sep 7 12:18:16.524292918 (22h49m52.563866038s) [noise=true , address=45.91.116.12:4227 , piece_id=2VE4AKBVU6EJMEIYWZODUTUZMDSLSQSSXSZQH7FDFR6MWEIW3I7Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524351069 (22h49m52.563924189s) [noise=true , address=77.90.31.12:4590 , piece_id=KL5A25JVCSNY62ZHGS5AX7ECOVSOU6Y2AXKC2XR5EUG2OOCOTOIA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524431704 (22h49m52.564004814s) [noise=true , address=77.90.31.17:4771 , piece_id=ARA7GHD4BTCULSXW27C44BAJH5ELDT2XJ3MDK6M5KJG5LY35FFBA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524477071 (22h49m52.564050191s) [seg_pos={0 0}, upload=20123]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:18:16.524563867 (22h49m52.564136987s) [upload=20123]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:18:16.524569207 (22h49m52.564142327s) [upload=20123]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:18:16.874668283 (22h49m52.914241404s) [upload=20123]: Done uploading segments... Sep 7 12:18:16.874686709 (22h49m52.914259829s) [upload=20123]: Upload finished. err: <nil> Duration: 7.147432355s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................X...........X.....X....] duration: 7.812564285s FAIL (2025-09-07 12:20:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:18:13.478410628 (22h49m49.517983748s) [upload=20123]: Starting upload Sep 7 12:18:13.478510909 (22h49m49.518084029s) [upload=20123]: Done starting upload Sep 7 12:18:13.478657580 (22h49m49.5182307s) [upload=20123]: Uploading segments... Sep 7 12:18:13.478782378 (22h49m49.518355548s) [upload=20123]: Got next segment. Inline: false Sep 7 12:18:13.478835610 (22h49m49.51840874s) [upload=20123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:18:13.478843154 (22h49m49.518416274s) [upload=20123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:18:14.234039783 (22h49m50.273612903s) [seg_pos={0 0}, upload=20123]: Begin upload segment... Sep 7 12:18:14.234085281 (22h49m50.273658391s) [seg_pos={0 0}, upload=20123]: Done begin upload segment. Sep 7 12:18:14.234094378 (22h49m50.273667488s) [noise=true , address=45.91.116.13:4154 , piece_id=OTDUTZW26I73ADPVJ3ZPDUI4MYIFJQQI24VA4EGJLS5L3T2PJ5DQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234103195 (22h49m50.273676315s) [noise=true , address=45.91.116.11:4118 , piece_id=E5LRFS4MKVCFO5YQSOXRYARST5SRLZZWJVJO5V3DH4VZZL2AKZGQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234504242 (22h49m50.274077402s) [noise=true , address=45.91.116.12:4226 , piece_id=66XXLFWFQ5BOJL3CPTNRHBVPNBO7DSQJSXKY6PFP4OGZWED6CWSQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234967248 (22h49m50.274540368s) [noise=true , address=45.91.116.5:4298 , piece_id=Y7KFGSE7HWFF33QW3NQ6GGKTDNTKQPPSRNOF3C6EKKNG5PHZ6YBA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235273535 (22h49m50.274846655s) [noise=true , address=45.91.116.7:4011 , piece_id=YWBVLWEYCCPEC7BJXZJAQO66NMEBOHY57KJR4G2DYX36JAYGAF3Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235545195 (22h49m50.275118305s) [noise=true , address=45.91.116.11:4117 , piece_id=47ZZ52FK7HR565KFDSQBJNSMNCONMY7RH7KTBR4R4FMUWOVH7SDA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235830380 (22h49m50.2754035s) [noise=true , address=45.91.116.5:4297 , piece_id=NH4AMQD4XIDTW57WLTQPXM6S5KM4N3SI7G44C4HGXZGEQKWHJ3FA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236049159 (22h49m50.275622279s) [noise=true , address=45.91.116.9:4263 , piece_id=CACXWKVZN72HAWYJUZXS7L6RJMIQ4GPW5NEXBCBC4M7BNKGAN66Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236280301 (22h49m50.275853421s) [noise=true , address=45.91.116.6:4975 , piece_id=J5MP3EAEGKOAZUOIOWI6FFGQVN5JZWWO7F475YCVYDVKGWUBMPNQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236514008 (22h49m50.276087128s) [noise=true , address=45.91.116.12:4227 , piece_id=2VE4AKBVU6EJMEIYWZODUTUZMDSLSQSSXSZQH7FDFR6MWEIW3I7Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236736895 (22h49m50.276310015s) [noise=true , address=45.91.116.4:4937 , piece_id=VVXAX3UIDDFNTRAGMG2WSPTPJHXMQX3QJF2NYDY77E2BWNVZKN6A, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236938962 (22h49m50.276512082s) [noise=true , address=45.91.116.7:4009 , piece_id=JNMKW2PJDIF4COW4M5G5EG5AVILORIGP6WDBG4CMO6XVWUBGMNJQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237347714 (22h49m50.276920844s) [noise=true , address=77.90.31.12:4590 , piece_id=KL5A25JVCSNY62ZHGS5AX7ECOVSOU6Y2AXKC2XR5EUG2OOCOTOIA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237629593 (22h49m50.277202713s) [noise=true , address=45.91.116.13:4155 , piece_id=LIVS7QLGIGQIE4BBY3QDKEXA2YOXLUCVX4DCME6DRUYF76AXEDQQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237935188 (22h49m50.277508308s) [noise=true , address=77.90.31.17:4771 , piece_id=ARA7GHD4BTCULSXW27C44BAJH5ELDT2XJ3MDK6M5KJG5LY35FFBA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.238210925 (22h49m50.277784045s) [noise=true , address=45.91.116.5:4299 , piece_id=URTUCQECDZMVGHHYHG5HBE3AQUCJOHAHDWO74DFKMAHXZ3NIULGQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:16.085425613 (22h49m52.124998733s) [upload=20123]: Next returned nil segment Sep 7 12:18:16.085448186 (22h49m52.125021296s) [upload=20123]: Waiting for error group managing segments... Sep 7 12:18:16.382438546 (22h49m52.422011666s) [noise=true , address=45.91.116.5:4298 , piece_id=Y7KFGSE7HWFF33QW3NQ6GGKTDNTKQPPSRNOF3C6EKKNG5PHZ6YBA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.399416514 (22h49m52.438989634s) [noise=true , address=45.91.116.13:4155 , piece_id=LIVS7QLGIGQIE4BBY3QDKEXA2YOXLUCVX4DCME6DRUYF76AXEDQQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.413524900 (22h49m52.45309803s) [noise=true , address=45.91.116.6:4975 , piece_id=J5MP3EAEGKOAZUOIOWI6FFGQVN5JZWWO7F475YCVYDVKGWUBMPNQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.416076483 (22h49m52.455649633s) [noise=true , address=45.91.116.11:4118 , piece_id=E5LRFS4MKVCFO5YQSOXRYARST5SRLZZWJVJO5V3DH4VZZL2AKZGQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.427664638 (22h49m52.467237778s) [noise=true , address=45.91.116.11:4117 , piece_id=47ZZ52FK7HR565KFDSQBJNSMNCONMY7RH7KTBR4R4FMUWOVH7SDA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.430874259 (22h49m52.470447379s) [noise=true , address=45.91.116.13:4154 , piece_id=OTDUTZW26I73ADPVJ3ZPDUI4MYIFJQQI24VA4EGJLS5L3T2PJ5DQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.433357641 (22h49m52.472930761s) [noise=true , address=45.91.116.7:4009 , piece_id=JNMKW2PJDIF4COW4M5G5EG5AVILORIGP6WDBG4CMO6XVWUBGMNJQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.435201369 (22h49m52.474774479s) [noise=true , address=45.91.116.12:4226 , piece_id=66XXLFWFQ5BOJL3CPTNRHBVPNBO7DSQJSXKY6PFP4OGZWED6CWSQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.435517494 (22h49m52.475090614s) [noise=true , address=45.91.116.5:4297 , piece_id=NH4AMQD4XIDTW57WLTQPXM6S5KM4N3SI7G44C4HGXZGEQKWHJ3FA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.455335767 (22h49m52.494908877s) [noise=true , address=45.91.116.5:4299 , piece_id=URTUCQECDZMVGHHYHG5HBE3AQUCJOHAHDWO74DFKMAHXZ3NIULGQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.458056703 (22h49m52.497629823s) [noise=true , address=45.91.116.7:4011 , piece_id=YWBVLWEYCCPEC7BJXZJAQO66NMEBOHY57KJR4G2DYX36JAYGAF3Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.506740612 (22h49m52.546313722s) [noise=true , address=45.91.116.4:4937 , piece_id=VVXAX3UIDDFNTRAGMG2WSPTPJHXMQX3QJF2NYDY77E2BWNVZKN6A, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.524160735 (22h49m52.563733855s) [noise=true , address=45.91.116.9:4263 , piece_id=CACXWKVZN72HAWYJUZXS7L6RJMIQ4GPW5NEXBCBC4M7BNKGAN66Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.524187065 (22h49m52.563760185s) [seg_pos={0 0}, upload=20123]: Segment reached optimal threshold of 13 pieces. Sep 7 12:18:16.524292918 (22h49m52.563866038s) [noise=true , address=45.91.116.12:4227 , piece_id=2VE4AKBVU6EJMEIYWZODUTUZMDSLSQSSXSZQH7FDFR6MWEIW3I7Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524351069 (22h49m52.563924189s) [noise=true , address=77.90.31.12:4590 , piece_id=KL5A25JVCSNY62ZHGS5AX7ECOVSOU6Y2AXKC2XR5EUG2OOCOTOIA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524431704 (22h49m52.564004814s) [noise=true , address=77.90.31.17:4771 , piece_id=ARA7GHD4BTCULSXW27C44BAJH5ELDT2XJ3MDK6M5KJG5LY35FFBA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524477071 (22h49m52.564050191s) [seg_pos={0 0}, upload=20123]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:18:16.524563867 (22h49m52.564136987s) [upload=20123]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:18:16.524569207 (22h49m52.564142327s) [upload=20123]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:18:16.874668283 (22h49m52.914241404s) [upload=20123]: Done uploading segments... Sep 7 12:18:16.874686709 (22h49m52.914259829s) [upload=20123]: Upload finished. err: <nil> Duration: 7.812582689s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................X...........X.....X...] duration: 28.972825484s FAIL (2025-09-07 12:20:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:18:13.478410628 (22h49m49.517983748s) [upload=20123]: Starting upload Sep 7 12:18:13.478510909 (22h49m49.518084029s) [upload=20123]: Done starting upload Sep 7 12:18:13.478657580 (22h49m49.5182307s) [upload=20123]: Uploading segments... Sep 7 12:18:13.478782378 (22h49m49.518355548s) [upload=20123]: Got next segment. Inline: false Sep 7 12:18:13.478835610 (22h49m49.51840874s) [upload=20123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:18:13.478843154 (22h49m49.518416274s) [upload=20123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:18:14.234039783 (22h49m50.273612903s) [seg_pos={0 0}, upload=20123]: Begin upload segment... Sep 7 12:18:14.234085281 (22h49m50.273658391s) [seg_pos={0 0}, upload=20123]: Done begin upload segment. Sep 7 12:18:14.234094378 (22h49m50.273667488s) [noise=true , address=45.91.116.13:4154 , piece_id=OTDUTZW26I73ADPVJ3ZPDUI4MYIFJQQI24VA4EGJLS5L3T2PJ5DQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234103195 (22h49m50.273676315s) [noise=true , address=45.91.116.11:4118 , piece_id=E5LRFS4MKVCFO5YQSOXRYARST5SRLZZWJVJO5V3DH4VZZL2AKZGQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234504242 (22h49m50.274077402s) [noise=true , address=45.91.116.12:4226 , piece_id=66XXLFWFQ5BOJL3CPTNRHBVPNBO7DSQJSXKY6PFP4OGZWED6CWSQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234967248 (22h49m50.274540368s) [noise=true , address=45.91.116.5:4298 , piece_id=Y7KFGSE7HWFF33QW3NQ6GGKTDNTKQPPSRNOF3C6EKKNG5PHZ6YBA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235273535 (22h49m50.274846655s) [noise=true , address=45.91.116.7:4011 , piece_id=YWBVLWEYCCPEC7BJXZJAQO66NMEBOHY57KJR4G2DYX36JAYGAF3Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235545195 (22h49m50.275118305s) [noise=true , address=45.91.116.11:4117 , piece_id=47ZZ52FK7HR565KFDSQBJNSMNCONMY7RH7KTBR4R4FMUWOVH7SDA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235830380 (22h49m50.2754035s) [noise=true , address=45.91.116.5:4297 , piece_id=NH4AMQD4XIDTW57WLTQPXM6S5KM4N3SI7G44C4HGXZGEQKWHJ3FA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236049159 (22h49m50.275622279s) [noise=true , address=45.91.116.9:4263 , piece_id=CACXWKVZN72HAWYJUZXS7L6RJMIQ4GPW5NEXBCBC4M7BNKGAN66Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236280301 (22h49m50.275853421s) [noise=true , address=45.91.116.6:4975 , piece_id=J5MP3EAEGKOAZUOIOWI6FFGQVN5JZWWO7F475YCVYDVKGWUBMPNQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236514008 (22h49m50.276087128s) [noise=true , address=45.91.116.12:4227 , piece_id=2VE4AKBVU6EJMEIYWZODUTUZMDSLSQSSXSZQH7FDFR6MWEIW3I7Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236736895 (22h49m50.276310015s) [noise=true , address=45.91.116.4:4937 , piece_id=VVXAX3UIDDFNTRAGMG2WSPTPJHXMQX3QJF2NYDY77E2BWNVZKN6A, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236938962 (22h49m50.276512082s) [noise=true , address=45.91.116.7:4009 , piece_id=JNMKW2PJDIF4COW4M5G5EG5AVILORIGP6WDBG4CMO6XVWUBGMNJQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237347714 (22h49m50.276920844s) [noise=true , address=77.90.31.12:4590 , piece_id=KL5A25JVCSNY62ZHGS5AX7ECOVSOU6Y2AXKC2XR5EUG2OOCOTOIA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237629593 (22h49m50.277202713s) [noise=true , address=45.91.116.13:4155 , piece_id=LIVS7QLGIGQIE4BBY3QDKEXA2YOXLUCVX4DCME6DRUYF76AXEDQQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237935188 (22h49m50.277508308s) [noise=true , address=77.90.31.17:4771 , piece_id=ARA7GHD4BTCULSXW27C44BAJH5ELDT2XJ3MDK6M5KJG5LY35FFBA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.238210925 (22h49m50.277784045s) [noise=true , address=45.91.116.5:4299 , piece_id=URTUCQECDZMVGHHYHG5HBE3AQUCJOHAHDWO74DFKMAHXZ3NIULGQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:16.085425613 (22h49m52.124998733s) [upload=20123]: Next returned nil segment Sep 7 12:18:16.085448186 (22h49m52.125021296s) [upload=20123]: Waiting for error group managing segments... Sep 7 12:18:16.382438546 (22h49m52.422011666s) [noise=true , address=45.91.116.5:4298 , piece_id=Y7KFGSE7HWFF33QW3NQ6GGKTDNTKQPPSRNOF3C6EKKNG5PHZ6YBA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.399416514 (22h49m52.438989634s) [noise=true , address=45.91.116.13:4155 , piece_id=LIVS7QLGIGQIE4BBY3QDKEXA2YOXLUCVX4DCME6DRUYF76AXEDQQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.413524900 (22h49m52.45309803s) [noise=true , address=45.91.116.6:4975 , piece_id=J5MP3EAEGKOAZUOIOWI6FFGQVN5JZWWO7F475YCVYDVKGWUBMPNQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.416076483 (22h49m52.455649633s) [noise=true , address=45.91.116.11:4118 , piece_id=E5LRFS4MKVCFO5YQSOXRYARST5SRLZZWJVJO5V3DH4VZZL2AKZGQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.427664638 (22h49m52.467237778s) [noise=true , address=45.91.116.11:4117 , piece_id=47ZZ52FK7HR565KFDSQBJNSMNCONMY7RH7KTBR4R4FMUWOVH7SDA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.430874259 (22h49m52.470447379s) [noise=true , address=45.91.116.13:4154 , piece_id=OTDUTZW26I73ADPVJ3ZPDUI4MYIFJQQI24VA4EGJLS5L3T2PJ5DQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.433357641 (22h49m52.472930761s) [noise=true , address=45.91.116.7:4009 , piece_id=JNMKW2PJDIF4COW4M5G5EG5AVILORIGP6WDBG4CMO6XVWUBGMNJQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.435201369 (22h49m52.474774479s) [noise=true , address=45.91.116.12:4226 , piece_id=66XXLFWFQ5BOJL3CPTNRHBVPNBO7DSQJSXKY6PFP4OGZWED6CWSQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.435517494 (22h49m52.475090614s) [noise=true , address=45.91.116.5:4297 , piece_id=NH4AMQD4XIDTW57WLTQPXM6S5KM4N3SI7G44C4HGXZGEQKWHJ3FA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.455335767 (22h49m52.494908877s) [noise=true , address=45.91.116.5:4299 , piece_id=URTUCQECDZMVGHHYHG5HBE3AQUCJOHAHDWO74DFKMAHXZ3NIULGQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.458056703 (22h49m52.497629823s) [noise=true , address=45.91.116.7:4011 , piece_id=YWBVLWEYCCPEC7BJXZJAQO66NMEBOHY57KJR4G2DYX36JAYGAF3Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.506740612 (22h49m52.546313722s) [noise=true , address=45.91.116.4:4937 , piece_id=VVXAX3UIDDFNTRAGMG2WSPTPJHXMQX3QJF2NYDY77E2BWNVZKN6A, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.524160735 (22h49m52.563733855s) [noise=true , address=45.91.116.9:4263 , piece_id=CACXWKVZN72HAWYJUZXS7L6RJMIQ4GPW5NEXBCBC4M7BNKGAN66Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.524187065 (22h49m52.563760185s) [seg_pos={0 0}, upload=20123]: Segment reached optimal threshold of 13 pieces. Sep 7 12:18:16.524292918 (22h49m52.563866038s) [noise=true , address=45.91.116.12:4227 , piece_id=2VE4AKBVU6EJMEIYWZODUTUZMDSLSQSSXSZQH7FDFR6MWEIW3I7Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524351069 (22h49m52.563924189s) [noise=true , address=77.90.31.12:4590 , piece_id=KL5A25JVCSNY62ZHGS5AX7ECOVSOU6Y2AXKC2XR5EUG2OOCOTOIA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524431704 (22h49m52.564004814s) [noise=true , address=77.90.31.17:4771 , piece_id=ARA7GHD4BTCULSXW27C44BAJH5ELDT2XJ3MDK6M5KJG5LY35FFBA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524477071 (22h49m52.564050191s) [seg_pos={0 0}, upload=20123]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:18:16.524563867 (22h49m52.564136987s) [upload=20123]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:18:16.524569207 (22h49m52.564142327s) [upload=20123]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:18:16.874668283 (22h49m52.914241404s) [upload=20123]: Done uploading segments... Sep 7 12:18:16.874686709 (22h49m52.914259829s) [upload=20123]: Upload finished. err: <nil> Duration: 28.972893454s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................X...........X.....X..] duration: 19.539917141s FAIL (2025-09-07 12:19:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:18:13.478410628 (22h49m49.517983748s) [upload=20123]: Starting upload Sep 7 12:18:13.478510909 (22h49m49.518084029s) [upload=20123]: Done starting upload Sep 7 12:18:13.478657580 (22h49m49.5182307s) [upload=20123]: Uploading segments... Sep 7 12:18:13.478782378 (22h49m49.518355548s) [upload=20123]: Got next segment. Inline: false Sep 7 12:18:13.478835610 (22h49m49.51840874s) [upload=20123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:18:13.478843154 (22h49m49.518416274s) [upload=20123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:18:14.234039783 (22h49m50.273612903s) [seg_pos={0 0}, upload=20123]: Begin upload segment... Sep 7 12:18:14.234085281 (22h49m50.273658391s) [seg_pos={0 0}, upload=20123]: Done begin upload segment. Sep 7 12:18:14.234094378 (22h49m50.273667488s) [noise=true , address=45.91.116.13:4154 , piece_id=OTDUTZW26I73ADPVJ3ZPDUI4MYIFJQQI24VA4EGJLS5L3T2PJ5DQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234103195 (22h49m50.273676315s) [noise=true , address=45.91.116.11:4118 , piece_id=E5LRFS4MKVCFO5YQSOXRYARST5SRLZZWJVJO5V3DH4VZZL2AKZGQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234504242 (22h49m50.274077402s) [noise=true , address=45.91.116.12:4226 , piece_id=66XXLFWFQ5BOJL3CPTNRHBVPNBO7DSQJSXKY6PFP4OGZWED6CWSQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234967248 (22h49m50.274540368s) [noise=true , address=45.91.116.5:4298 , piece_id=Y7KFGSE7HWFF33QW3NQ6GGKTDNTKQPPSRNOF3C6EKKNG5PHZ6YBA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235273535 (22h49m50.274846655s) [noise=true , address=45.91.116.7:4011 , piece_id=YWBVLWEYCCPEC7BJXZJAQO66NMEBOHY57KJR4G2DYX36JAYGAF3Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235545195 (22h49m50.275118305s) [noise=true , address=45.91.116.11:4117 , piece_id=47ZZ52FK7HR565KFDSQBJNSMNCONMY7RH7KTBR4R4FMUWOVH7SDA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235830380 (22h49m50.2754035s) [noise=true , address=45.91.116.5:4297 , piece_id=NH4AMQD4XIDTW57WLTQPXM6S5KM4N3SI7G44C4HGXZGEQKWHJ3FA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236049159 (22h49m50.275622279s) [noise=true , address=45.91.116.9:4263 , piece_id=CACXWKVZN72HAWYJUZXS7L6RJMIQ4GPW5NEXBCBC4M7BNKGAN66Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236280301 (22h49m50.275853421s) [noise=true , address=45.91.116.6:4975 , piece_id=J5MP3EAEGKOAZUOIOWI6FFGQVN5JZWWO7F475YCVYDVKGWUBMPNQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236514008 (22h49m50.276087128s) [noise=true , address=45.91.116.12:4227 , piece_id=2VE4AKBVU6EJMEIYWZODUTUZMDSLSQSSXSZQH7FDFR6MWEIW3I7Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236736895 (22h49m50.276310015s) [noise=true , address=45.91.116.4:4937 , piece_id=VVXAX3UIDDFNTRAGMG2WSPTPJHXMQX3QJF2NYDY77E2BWNVZKN6A, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236938962 (22h49m50.276512082s) [noise=true , address=45.91.116.7:4009 , piece_id=JNMKW2PJDIF4COW4M5G5EG5AVILORIGP6WDBG4CMO6XVWUBGMNJQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237347714 (22h49m50.276920844s) [noise=true , address=77.90.31.12:4590 , piece_id=KL5A25JVCSNY62ZHGS5AX7ECOVSOU6Y2AXKC2XR5EUG2OOCOTOIA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237629593 (22h49m50.277202713s) [noise=true , address=45.91.116.13:4155 , piece_id=LIVS7QLGIGQIE4BBY3QDKEXA2YOXLUCVX4DCME6DRUYF76AXEDQQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237935188 (22h49m50.277508308s) [noise=true , address=77.90.31.17:4771 , piece_id=ARA7GHD4BTCULSXW27C44BAJH5ELDT2XJ3MDK6M5KJG5LY35FFBA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.238210925 (22h49m50.277784045s) [noise=true , address=45.91.116.5:4299 , piece_id=URTUCQECDZMVGHHYHG5HBE3AQUCJOHAHDWO74DFKMAHXZ3NIULGQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:16.085425613 (22h49m52.124998733s) [upload=20123]: Next returned nil segment Sep 7 12:18:16.085448186 (22h49m52.125021296s) [upload=20123]: Waiting for error group managing segments... Sep 7 12:18:16.382438546 (22h49m52.422011666s) [noise=true , address=45.91.116.5:4298 , piece_id=Y7KFGSE7HWFF33QW3NQ6GGKTDNTKQPPSRNOF3C6EKKNG5PHZ6YBA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.399416514 (22h49m52.438989634s) [noise=true , address=45.91.116.13:4155 , piece_id=LIVS7QLGIGQIE4BBY3QDKEXA2YOXLUCVX4DCME6DRUYF76AXEDQQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.413524900 (22h49m52.45309803s) [noise=true , address=45.91.116.6:4975 , piece_id=J5MP3EAEGKOAZUOIOWI6FFGQVN5JZWWO7F475YCVYDVKGWUBMPNQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.416076483 (22h49m52.455649633s) [noise=true , address=45.91.116.11:4118 , piece_id=E5LRFS4MKVCFO5YQSOXRYARST5SRLZZWJVJO5V3DH4VZZL2AKZGQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.427664638 (22h49m52.467237778s) [noise=true , address=45.91.116.11:4117 , piece_id=47ZZ52FK7HR565KFDSQBJNSMNCONMY7RH7KTBR4R4FMUWOVH7SDA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.430874259 (22h49m52.470447379s) [noise=true , address=45.91.116.13:4154 , piece_id=OTDUTZW26I73ADPVJ3ZPDUI4MYIFJQQI24VA4EGJLS5L3T2PJ5DQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.433357641 (22h49m52.472930761s) [noise=true , address=45.91.116.7:4009 , piece_id=JNMKW2PJDIF4COW4M5G5EG5AVILORIGP6WDBG4CMO6XVWUBGMNJQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.435201369 (22h49m52.474774479s) [noise=true , address=45.91.116.12:4226 , piece_id=66XXLFWFQ5BOJL3CPTNRHBVPNBO7DSQJSXKY6PFP4OGZWED6CWSQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.435517494 (22h49m52.475090614s) [noise=true , address=45.91.116.5:4297 , piece_id=NH4AMQD4XIDTW57WLTQPXM6S5KM4N3SI7G44C4HGXZGEQKWHJ3FA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.455335767 (22h49m52.494908877s) [noise=true , address=45.91.116.5:4299 , piece_id=URTUCQECDZMVGHHYHG5HBE3AQUCJOHAHDWO74DFKMAHXZ3NIULGQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.458056703 (22h49m52.497629823s) [noise=true , address=45.91.116.7:4011 , piece_id=YWBVLWEYCCPEC7BJXZJAQO66NMEBOHY57KJR4G2DYX36JAYGAF3Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.506740612 (22h49m52.546313722s) [noise=true , address=45.91.116.4:4937 , piece_id=VVXAX3UIDDFNTRAGMG2WSPTPJHXMQX3QJF2NYDY77E2BWNVZKN6A, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.524160735 (22h49m52.563733855s) [noise=true , address=45.91.116.9:4263 , piece_id=CACXWKVZN72HAWYJUZXS7L6RJMIQ4GPW5NEXBCBC4M7BNKGAN66Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.524187065 (22h49m52.563760185s) [seg_pos={0 0}, upload=20123]: Segment reached optimal threshold of 13 pieces. Sep 7 12:18:16.524292918 (22h49m52.563866038s) [noise=true , address=45.91.116.12:4227 , piece_id=2VE4AKBVU6EJMEIYWZODUTUZMDSLSQSSXSZQH7FDFR6MWEIW3I7Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524351069 (22h49m52.563924189s) [noise=true , address=77.90.31.12:4590 , piece_id=KL5A25JVCSNY62ZHGS5AX7ECOVSOU6Y2AXKC2XR5EUG2OOCOTOIA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524431704 (22h49m52.564004814s) [noise=true , address=77.90.31.17:4771 , piece_id=ARA7GHD4BTCULSXW27C44BAJH5ELDT2XJ3MDK6M5KJG5LY35FFBA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524477071 (22h49m52.564050191s) [seg_pos={0 0}, upload=20123]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:18:16.524563867 (22h49m52.564136987s) [upload=20123]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:18:16.524569207 (22h49m52.564142327s) [upload=20123]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:18:16.874668283 (22h49m52.914241404s) [upload=20123]: Done uploading segments... Sep 7 12:18:16.874686709 (22h49m52.914259829s) [upload=20123]: Upload finished. err: <nil> Duration: 19.540011262s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................X...........X.....X.] duration: 7.023810171s FAIL (2025-09-07 12:18:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:18:13.478410628 (22h49m49.517983748s) [upload=20123]: Starting upload Sep 7 12:18:13.478510909 (22h49m49.518084029s) [upload=20123]: Done starting upload Sep 7 12:18:13.478657580 (22h49m49.5182307s) [upload=20123]: Uploading segments... Sep 7 12:18:13.478782378 (22h49m49.518355548s) [upload=20123]: Got next segment. Inline: false Sep 7 12:18:13.478835610 (22h49m49.51840874s) [upload=20123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:18:13.478843154 (22h49m49.518416274s) [upload=20123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:18:14.234039783 (22h49m50.273612903s) [seg_pos={0 0}, upload=20123]: Begin upload segment... Sep 7 12:18:14.234085281 (22h49m50.273658391s) [seg_pos={0 0}, upload=20123]: Done begin upload segment. Sep 7 12:18:14.234094378 (22h49m50.273667488s) [noise=true , address=45.91.116.13:4154 , piece_id=OTDUTZW26I73ADPVJ3ZPDUI4MYIFJQQI24VA4EGJLS5L3T2PJ5DQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234103195 (22h49m50.273676315s) [noise=true , address=45.91.116.11:4118 , piece_id=E5LRFS4MKVCFO5YQSOXRYARST5SRLZZWJVJO5V3DH4VZZL2AKZGQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234504242 (22h49m50.274077402s) [noise=true , address=45.91.116.12:4226 , piece_id=66XXLFWFQ5BOJL3CPTNRHBVPNBO7DSQJSXKY6PFP4OGZWED6CWSQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234967248 (22h49m50.274540368s) [noise=true , address=45.91.116.5:4298 , piece_id=Y7KFGSE7HWFF33QW3NQ6GGKTDNTKQPPSRNOF3C6EKKNG5PHZ6YBA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235273535 (22h49m50.274846655s) [noise=true , address=45.91.116.7:4011 , piece_id=YWBVLWEYCCPEC7BJXZJAQO66NMEBOHY57KJR4G2DYX36JAYGAF3Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235545195 (22h49m50.275118305s) [noise=true , address=45.91.116.11:4117 , piece_id=47ZZ52FK7HR565KFDSQBJNSMNCONMY7RH7KTBR4R4FMUWOVH7SDA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235830380 (22h49m50.2754035s) [noise=true , address=45.91.116.5:4297 , piece_id=NH4AMQD4XIDTW57WLTQPXM6S5KM4N3SI7G44C4HGXZGEQKWHJ3FA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236049159 (22h49m50.275622279s) [noise=true , address=45.91.116.9:4263 , piece_id=CACXWKVZN72HAWYJUZXS7L6RJMIQ4GPW5NEXBCBC4M7BNKGAN66Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236280301 (22h49m50.275853421s) [noise=true , address=45.91.116.6:4975 , piece_id=J5MP3EAEGKOAZUOIOWI6FFGQVN5JZWWO7F475YCVYDVKGWUBMPNQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236514008 (22h49m50.276087128s) [noise=true , address=45.91.116.12:4227 , piece_id=2VE4AKBVU6EJMEIYWZODUTUZMDSLSQSSXSZQH7FDFR6MWEIW3I7Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236736895 (22h49m50.276310015s) [noise=true , address=45.91.116.4:4937 , piece_id=VVXAX3UIDDFNTRAGMG2WSPTPJHXMQX3QJF2NYDY77E2BWNVZKN6A, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236938962 (22h49m50.276512082s) [noise=true , address=45.91.116.7:4009 , piece_id=JNMKW2PJDIF4COW4M5G5EG5AVILORIGP6WDBG4CMO6XVWUBGMNJQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237347714 (22h49m50.276920844s) [noise=true , address=77.90.31.12:4590 , piece_id=KL5A25JVCSNY62ZHGS5AX7ECOVSOU6Y2AXKC2XR5EUG2OOCOTOIA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237629593 (22h49m50.277202713s) [noise=true , address=45.91.116.13:4155 , piece_id=LIVS7QLGIGQIE4BBY3QDKEXA2YOXLUCVX4DCME6DRUYF76AXEDQQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237935188 (22h49m50.277508308s) [noise=true , address=77.90.31.17:4771 , piece_id=ARA7GHD4BTCULSXW27C44BAJH5ELDT2XJ3MDK6M5KJG5LY35FFBA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.238210925 (22h49m50.277784045s) [noise=true , address=45.91.116.5:4299 , piece_id=URTUCQECDZMVGHHYHG5HBE3AQUCJOHAHDWO74DFKMAHXZ3NIULGQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:16.085425613 (22h49m52.124998733s) [upload=20123]: Next returned nil segment Sep 7 12:18:16.085448186 (22h49m52.125021296s) [upload=20123]: Waiting for error group managing segments... Sep 7 12:18:16.382438546 (22h49m52.422011666s) [noise=true , address=45.91.116.5:4298 , piece_id=Y7KFGSE7HWFF33QW3NQ6GGKTDNTKQPPSRNOF3C6EKKNG5PHZ6YBA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.399416514 (22h49m52.438989634s) [noise=true , address=45.91.116.13:4155 , piece_id=LIVS7QLGIGQIE4BBY3QDKEXA2YOXLUCVX4DCME6DRUYF76AXEDQQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.413524900 (22h49m52.45309803s) [noise=true , address=45.91.116.6:4975 , piece_id=J5MP3EAEGKOAZUOIOWI6FFGQVN5JZWWO7F475YCVYDVKGWUBMPNQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.416076483 (22h49m52.455649633s) [noise=true , address=45.91.116.11:4118 , piece_id=E5LRFS4MKVCFO5YQSOXRYARST5SRLZZWJVJO5V3DH4VZZL2AKZGQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.427664638 (22h49m52.467237778s) [noise=true , address=45.91.116.11:4117 , piece_id=47ZZ52FK7HR565KFDSQBJNSMNCONMY7RH7KTBR4R4FMUWOVH7SDA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.430874259 (22h49m52.470447379s) [noise=true , address=45.91.116.13:4154 , piece_id=OTDUTZW26I73ADPVJ3ZPDUI4MYIFJQQI24VA4EGJLS5L3T2PJ5DQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.433357641 (22h49m52.472930761s) [noise=true , address=45.91.116.7:4009 , piece_id=JNMKW2PJDIF4COW4M5G5EG5AVILORIGP6WDBG4CMO6XVWUBGMNJQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.435201369 (22h49m52.474774479s) [noise=true , address=45.91.116.12:4226 , piece_id=66XXLFWFQ5BOJL3CPTNRHBVPNBO7DSQJSXKY6PFP4OGZWED6CWSQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.435517494 (22h49m52.475090614s) [noise=true , address=45.91.116.5:4297 , piece_id=NH4AMQD4XIDTW57WLTQPXM6S5KM4N3SI7G44C4HGXZGEQKWHJ3FA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.455335767 (22h49m52.494908877s) [noise=true , address=45.91.116.5:4299 , piece_id=URTUCQECDZMVGHHYHG5HBE3AQUCJOHAHDWO74DFKMAHXZ3NIULGQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.458056703 (22h49m52.497629823s) [noise=true , address=45.91.116.7:4011 , piece_id=YWBVLWEYCCPEC7BJXZJAQO66NMEBOHY57KJR4G2DYX36JAYGAF3Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.506740612 (22h49m52.546313722s) [noise=true , address=45.91.116.4:4937 , piece_id=VVXAX3UIDDFNTRAGMG2WSPTPJHXMQX3QJF2NYDY77E2BWNVZKN6A, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.524160735 (22h49m52.563733855s) [noise=true , address=45.91.116.9:4263 , piece_id=CACXWKVZN72HAWYJUZXS7L6RJMIQ4GPW5NEXBCBC4M7BNKGAN66Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.524187065 (22h49m52.563760185s) [seg_pos={0 0}, upload=20123]: Segment reached optimal threshold of 13 pieces. Sep 7 12:18:16.524292918 (22h49m52.563866038s) [noise=true , address=45.91.116.12:4227 , piece_id=2VE4AKBVU6EJMEIYWZODUTUZMDSLSQSSXSZQH7FDFR6MWEIW3I7Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524351069 (22h49m52.563924189s) [noise=true , address=77.90.31.12:4590 , piece_id=KL5A25JVCSNY62ZHGS5AX7ECOVSOU6Y2AXKC2XR5EUG2OOCOTOIA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524431704 (22h49m52.564004814s) [noise=true , address=77.90.31.17:4771 , piece_id=ARA7GHD4BTCULSXW27C44BAJH5ELDT2XJ3MDK6M5KJG5LY35FFBA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524477071 (22h49m52.564050191s) [seg_pos={0 0}, upload=20123]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:18:16.524563867 (22h49m52.564136987s) [upload=20123]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:18:16.524569207 (22h49m52.564142327s) [upload=20123]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:18:16.874668283 (22h49m52.914241404s) [upload=20123]: Done uploading segments... Sep 7 12:18:16.874686709 (22h49m52.914259829s) [upload=20123]: Upload finished. err: <nil> Duration: 7.02382511s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X........................................X...........X.....X] duration: 30.002464136s FAIL (2025-09-07 12:18:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:18:13.478410628 (22h49m49.517983748s) [upload=20123]: Starting upload Sep 7 12:18:13.478510909 (22h49m49.518084029s) [upload=20123]: Done starting upload Sep 7 12:18:13.478657580 (22h49m49.5182307s) [upload=20123]: Uploading segments... Sep 7 12:18:13.478782378 (22h49m49.518355548s) [upload=20123]: Got next segment. Inline: false Sep 7 12:18:13.478835610 (22h49m49.51840874s) [upload=20123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:18:13.478843154 (22h49m49.518416274s) [upload=20123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:18:14.234039783 (22h49m50.273612903s) [seg_pos={0 0}, upload=20123]: Begin upload segment... Sep 7 12:18:14.234085281 (22h49m50.273658391s) [seg_pos={0 0}, upload=20123]: Done begin upload segment. Sep 7 12:18:14.234094378 (22h49m50.273667488s) [noise=true , address=45.91.116.13:4154 , piece_id=OTDUTZW26I73ADPVJ3ZPDUI4MYIFJQQI24VA4EGJLS5L3T2PJ5DQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234103195 (22h49m50.273676315s) [noise=true , address=45.91.116.11:4118 , piece_id=E5LRFS4MKVCFO5YQSOXRYARST5SRLZZWJVJO5V3DH4VZZL2AKZGQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234504242 (22h49m50.274077402s) [noise=true , address=45.91.116.12:4226 , piece_id=66XXLFWFQ5BOJL3CPTNRHBVPNBO7DSQJSXKY6PFP4OGZWED6CWSQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.234967248 (22h49m50.274540368s) [noise=true , address=45.91.116.5:4298 , piece_id=Y7KFGSE7HWFF33QW3NQ6GGKTDNTKQPPSRNOF3C6EKKNG5PHZ6YBA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235273535 (22h49m50.274846655s) [noise=true , address=45.91.116.7:4011 , piece_id=YWBVLWEYCCPEC7BJXZJAQO66NMEBOHY57KJR4G2DYX36JAYGAF3Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235545195 (22h49m50.275118305s) [noise=true , address=45.91.116.11:4117 , piece_id=47ZZ52FK7HR565KFDSQBJNSMNCONMY7RH7KTBR4R4FMUWOVH7SDA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.235830380 (22h49m50.2754035s) [noise=true , address=45.91.116.5:4297 , piece_id=NH4AMQD4XIDTW57WLTQPXM6S5KM4N3SI7G44C4HGXZGEQKWHJ3FA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236049159 (22h49m50.275622279s) [noise=true , address=45.91.116.9:4263 , piece_id=CACXWKVZN72HAWYJUZXS7L6RJMIQ4GPW5NEXBCBC4M7BNKGAN66Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236280301 (22h49m50.275853421s) [noise=true , address=45.91.116.6:4975 , piece_id=J5MP3EAEGKOAZUOIOWI6FFGQVN5JZWWO7F475YCVYDVKGWUBMPNQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236514008 (22h49m50.276087128s) [noise=true , address=45.91.116.12:4227 , piece_id=2VE4AKBVU6EJMEIYWZODUTUZMDSLSQSSXSZQH7FDFR6MWEIW3I7Q, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236736895 (22h49m50.276310015s) [noise=true , address=45.91.116.4:4937 , piece_id=VVXAX3UIDDFNTRAGMG2WSPTPJHXMQX3QJF2NYDY77E2BWNVZKN6A, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.236938962 (22h49m50.276512082s) [noise=true , address=45.91.116.7:4009 , piece_id=JNMKW2PJDIF4COW4M5G5EG5AVILORIGP6WDBG4CMO6XVWUBGMNJQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237347714 (22h49m50.276920844s) [noise=true , address=77.90.31.12:4590 , piece_id=KL5A25JVCSNY62ZHGS5AX7ECOVSOU6Y2AXKC2XR5EUG2OOCOTOIA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237629593 (22h49m50.277202713s) [noise=true , address=45.91.116.13:4155 , piece_id=LIVS7QLGIGQIE4BBY3QDKEXA2YOXLUCVX4DCME6DRUYF76AXEDQQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.237935188 (22h49m50.277508308s) [noise=true , address=77.90.31.17:4771 , piece_id=ARA7GHD4BTCULSXW27C44BAJH5ELDT2XJ3MDK6M5KJG5LY35FFBA, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:14.238210925 (22h49m50.277784045s) [noise=true , address=45.91.116.5:4299 , piece_id=URTUCQECDZMVGHHYHG5HBE3AQUCJOHAHDWO74DFKMAHXZ3NIULGQ, seg_pos={0 0}, upload=20123]: Uploading piece... Sep 7 12:18:16.085425613 (22h49m52.124998733s) [upload=20123]: Next returned nil segment Sep 7 12:18:16.085448186 (22h49m52.125021296s) [upload=20123]: Waiting for error group managing segments... Sep 7 12:18:16.382438546 (22h49m52.422011666s) [noise=true , address=45.91.116.5:4298 , piece_id=Y7KFGSE7HWFF33QW3NQ6GGKTDNTKQPPSRNOF3C6EKKNG5PHZ6YBA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.399416514 (22h49m52.438989634s) [noise=true , address=45.91.116.13:4155 , piece_id=LIVS7QLGIGQIE4BBY3QDKEXA2YOXLUCVX4DCME6DRUYF76AXEDQQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.413524900 (22h49m52.45309803s) [noise=true , address=45.91.116.6:4975 , piece_id=J5MP3EAEGKOAZUOIOWI6FFGQVN5JZWWO7F475YCVYDVKGWUBMPNQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.416076483 (22h49m52.455649633s) [noise=true , address=45.91.116.11:4118 , piece_id=E5LRFS4MKVCFO5YQSOXRYARST5SRLZZWJVJO5V3DH4VZZL2AKZGQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.427664638 (22h49m52.467237778s) [noise=true , address=45.91.116.11:4117 , piece_id=47ZZ52FK7HR565KFDSQBJNSMNCONMY7RH7KTBR4R4FMUWOVH7SDA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.430874259 (22h49m52.470447379s) [noise=true , address=45.91.116.13:4154 , piece_id=OTDUTZW26I73ADPVJ3ZPDUI4MYIFJQQI24VA4EGJLS5L3T2PJ5DQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.433357641 (22h49m52.472930761s) [noise=true , address=45.91.116.7:4009 , piece_id=JNMKW2PJDIF4COW4M5G5EG5AVILORIGP6WDBG4CMO6XVWUBGMNJQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.435201369 (22h49m52.474774479s) [noise=true , address=45.91.116.12:4226 , piece_id=66XXLFWFQ5BOJL3CPTNRHBVPNBO7DSQJSXKY6PFP4OGZWED6CWSQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.435517494 (22h49m52.475090614s) [noise=true , address=45.91.116.5:4297 , piece_id=NH4AMQD4XIDTW57WLTQPXM6S5KM4N3SI7G44C4HGXZGEQKWHJ3FA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.455335767 (22h49m52.494908877s) [noise=true , address=45.91.116.5:4299 , piece_id=URTUCQECDZMVGHHYHG5HBE3AQUCJOHAHDWO74DFKMAHXZ3NIULGQ, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.458056703 (22h49m52.497629823s) [noise=true , address=45.91.116.7:4011 , piece_id=YWBVLWEYCCPEC7BJXZJAQO66NMEBOHY57KJR4G2DYX36JAYGAF3Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.506740612 (22h49m52.546313722s) [noise=true , address=45.91.116.4:4937 , piece_id=VVXAX3UIDDFNTRAGMG2WSPTPJHXMQX3QJF2NYDY77E2BWNVZKN6A, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.524160735 (22h49m52.563733855s) [noise=true , address=45.91.116.9:4263 , piece_id=CACXWKVZN72HAWYJUZXS7L6RJMIQ4GPW5NEXBCBC4M7BNKGAN66Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: <nil> Sep 7 12:18:16.524187065 (22h49m52.563760185s) [seg_pos={0 0}, upload=20123]: Segment reached optimal threshold of 13 pieces. Sep 7 12:18:16.524292918 (22h49m52.563866038s) [noise=true , address=45.91.116.12:4227 , piece_id=2VE4AKBVU6EJMEIYWZODUTUZMDSLSQSSXSZQH7FDFR6MWEIW3I7Q, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524351069 (22h49m52.563924189s) [noise=true , address=77.90.31.12:4590 , piece_id=KL5A25JVCSNY62ZHGS5AX7ECOVSOU6Y2AXKC2XR5EUG2OOCOTOIA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524431704 (22h49m52.564004814s) [noise=true , address=77.90.31.17:4771 , piece_id=ARA7GHD4BTCULSXW27C44BAJH5ELDT2XJ3MDK6M5KJG5LY35FFBA, seg_pos={0 0}, upload=20123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Sep 7 12:18:16.524477071 (22h49m52.564050191s) [seg_pos={0 0}, upload=20123]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:18:16.524563867 (22h49m52.564136987s) [upload=20123]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:18:16.524569207 (22h49m52.564142327s) [upload=20123]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:18:16.874668283 (22h49m52.914241404s) [upload=20123]: Done uploading segments... Sep 7 12:18:16.874686709 (22h49m52.914259829s) [upload=20123]: Upload finished. err: <nil> Duration: 30.002484294s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X........................................X...........X.....] duration: 6.613554927s FAIL (2025-09-07 12:18:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:14:40.856932950 (22h46m16.89650607s) [upload=20069]: Starting upload Sep 7 12:14:40.856994037 (22h46m16.896567157s) [upload=20069]: Done starting upload Sep 7 12:14:40.857086575 (22h46m16.896659695s) [upload=20069]: Uploading segments... Sep 7 12:14:40.857171938 (22h46m16.896745058s) [upload=20069]: Got next segment. Inline: false Sep 7 12:14:40.857200673 (22h46m16.896773793s) [upload=20069]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:14:40.857205502 (22h46m16.896778622s) [upload=20069]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:14:42.120233071 (22h46m18.159806191s) [seg_pos={0 0}, upload=20069]: Begin upload segment... Sep 7 12:14:42.120272818 (22h46m18.159845938s) [seg_pos={0 0}, upload=20069]: Done begin upload segment. Sep 7 12:14:42.120293797 (22h46m18.159866917s) [noise=true , address=77.90.31.17:4772 , piece_id=DNMMY5VLQL6I3KSHCN425ZRXNLMP4U7HFEKGTPJRAWQEDVUYX6OQ, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.120352219 (22h46m18.159925339s) [noise=true , address=77.90.31.10:4517 , piece_id=DBN4ESGYA3E4XREB6ELWHGCSFY75IMEC2PZBDP72UH3CMT275ZIQ, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.120699504 (22h46m18.160272614s) [noise=true , address=45.91.116.8:4047 , piece_id=3M6HNAWRK7U5QTZ5YLMWRTU7YSKXL253X3HA7UK2I5FKAM6GXTCQ, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.120797101 (22h46m18.160370211s) [noise=true , address=45.91.116.10:4083 , piece_id=FAQIG2C27JZDH52CNQ3YUVIJXKVOAVV2NFHMRNWBIKOQO2F6NUZA, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.121037181 (22h46m18.160610301s) [noise=true , address=45.91.116.7:4009 , piece_id=VZ7VU2JBS3NJIUANO5CJOUC5Z4TMNLZ5757HKNTMZU5RQOXHAZWA, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.121089830 (22h46m18.16066294s) [noise=true , address=45.91.116.5:4299 , piece_id=YYNTEKRGUSIU332RXAJRMDITDUAGR4VB4LCWYHLCTG5GA3L5OLPQ, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.121462484 (22h46m18.161035604s) [noise=true , address=45.91.116.7:4011 , piece_id=Q72DS5LTF4KPUV7YO5R573VPOHXYUN7U5LFEUJ7H2NTHRJYLYSDA, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.121499795 (22h46m18.161072915s) [noise=true , address=45.91.116.6:4974 , piece_id=VQJ2SEXLWFTHTG64MGAKIZE2TTIY4T4CFSUKN2S67SSMPJPUZDMA, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.121789109 (22h46m18.161362229s) [noise=true , address=45.91.116.4:4940 , piece_id=5XFGG36XMTCIZGD5RWVXAADC6JSNHWSZBDQ6BH7AM6I3ZGGL67OA, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.121839626 (22h46m18.161412746s) [noise=true , address=45.91.116.6:4975 , piece_id=6OER362GG3P7EM5ICLYMTNTNFNCRRTRAQ6LHD7IYO5XVCM43YXPQ, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.122066399 (22h46m18.161639519s) [noise=true , address=45.91.116.14:4190 , piece_id=R775GNQVAYCFFXIZYE42VHXHME2ZUNFZG3PGFQU72CCKKDVIRD7A, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.122106396 (22h46m18.161679516s) [noise=true , address=45.91.116.13:4153 , piece_id=EHYIHEPY5XVEAGNAGYVVTEOAXEGKCHFVMOO3AEGCLTAVJG7E6O3A, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.122278445 (22h46m18.161851555s) [noise=true , address=45.91.116.8:4045 , piece_id=C474QMRWKSEBPCU2WTJSLK2HSLKM4IRD5YW436EUN4INXLCQ5L4Q, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.122446136 (22h46m18.162019256s) [noise=true , address=45.91.116.14:4191 , piece_id=ZK4ILC43X3IKG5BF23KGL4CSFAQF3QE4I3KUVVUFFOV3GKT34M2A, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.122491443 (22h46m18.162064563s) [noise=true , address=45.91.116.10:4082 , piece_id=HRIWTYC2CTFFCKA5ZUJOXC36JNB7QDHYAZ7UQTXWHXMJEWANS42A, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.122727184 (22h46m18.162300304s) [noise=true , address=77.90.31.16:4734 , piece_id=2XV5L6XTKOSOVUQY4QRDNI6KUDLGKWUPOQ2GOD6F5GYOZ3RD74PQ, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:43.923014265 (22h46m19.962587385s) [upload=20069]: Next returned nil segment Sep 7 12:14:43.923031448 (22h46m19.962604568s) [upload=20069]: Waiting for error group managing segments... Sep 7 12:14:44.244799200 (22h46m20.28437232s) [noise=true , address=45.91.116.8:4047 , piece_id=3M6HNAWRK7U5QTZ5YLMWRTU7YSKXL253X3HA7UK2I5FKAM6GXTCQ, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.253947506 (22h46m20.293520626s) [noise=true , address=45.91.116.6:4974 , piece_id=VQJ2SEXLWFTHTG64MGAKIZE2TTIY4T4CFSUKN2S67SSMPJPUZDMA, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.258645795 (22h46m20.298218905s) [noise=true , address=45.91.116.5:4299 , piece_id=YYNTEKRGUSIU332RXAJRMDITDUAGR4VB4LCWYHLCTG5GA3L5OLPQ, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.259059357 (22h46m20.298632477s) [noise=true , address=45.91.116.8:4045 , piece_id=C474QMRWKSEBPCU2WTJSLK2HSLKM4IRD5YW436EUN4INXLCQ5L4Q, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.260936849 (22h46m20.300509969s) [noise=true , address=45.91.116.7:4009 , piece_id=VZ7VU2JBS3NJIUANO5CJOUC5Z4TMNLZ5757HKNTMZU5RQOXHAZWA, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.281681895 (22h46m20.321255015s) [noise=true , address=45.91.116.13:4153 , piece_id=EHYIHEPY5XVEAGNAGYVVTEOAXEGKCHFVMOO3AEGCLTAVJG7E6O3A, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.283288820 (22h46m20.32286194s) [noise=true , address=45.91.116.7:4011 , piece_id=Q72DS5LTF4KPUV7YO5R573VPOHXYUN7U5LFEUJ7H2NTHRJYLYSDA, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.288615273 (22h46m20.328188393s) [noise=true , address=45.91.116.10:4082 , piece_id=HRIWTYC2CTFFCKA5ZUJOXC36JNB7QDHYAZ7UQTXWHXMJEWANS42A, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.298688398 (22h46m20.338261528s) [noise=true , address=77.90.31.17:4772 , piece_id=DNMMY5VLQL6I3KSHCN425ZRXNLMP4U7HFEKGTPJRAWQEDVUYX6OQ, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.315911695 (22h46m20.355484815s) [noise=true , address=45.91.116.6:4975 , piece_id=6OER362GG3P7EM5ICLYMTNTNFNCRRTRAQ6LHD7IYO5XVCM43YXPQ, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.328711859 (22h46m20.368284979s) [noise=true , address=77.90.31.10:4517 , piece_id=DBN4ESGYA3E4XREB6ELWHGCSFY75IMEC2PZBDP72UH3CMT275ZIQ, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.355695773 (22h46m20.395268883s) [noise=true , address=45.91.116.4:4940 , piece_id=5XFGG36XMTCIZGD5RWVXAADC6JSNHWSZBDQ6BH7AM6I3ZGGL67OA, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.360864083 (22h46m20.400437203s) [noise=true , address=45.91.116.10:4083 , piece_id=FAQIG2C27JZDH52CNQ3YUVIJXKVOAVV2NFHMRNWBIKOQO2F6NUZA, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.360887418 (22h46m20.400460538s) [seg_pos={0 0}, upload=20069]: Segment reached optimal threshold of 13 pieces. Sep 7 12:14:44.361008330 (22h46m20.40058145s) [noise=true , address=77.90.31.16:4734 , piece_id=2XV5L6XTKOSOVUQY4QRDNI6KUDLGKWUPOQ2GOD6F5GYOZ3RD74PQ, seg_pos={0 0}, upload=20069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: expected piece hash; context canceled; EOF Sep 7 12:14:44.361140151 (22h46m20.400713271s) [noise=true , address=45.91.116.14:4190 , piece_id=R775GNQVAYCFFXIZYE42VHXHME2ZUNFZG3PGFQU72CCKKDVIRD7A, seg_pos={0 0}, upload=20069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Sep 7 12:14:44.361168666 (22h46m20.400741786s) [noise=true , address=45.91.116.14:4191 , piece_id=ZK4ILC43X3IKG5BF23KGL4CSFAQF3QE4I3KUVVUFFOV3GKT34M2A, seg_pos={0 0}, upload=20069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Sep 7 12:14:44.361184867 (22h46m20.400757977s) [seg_pos={0 0}, upload=20069]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:14:44.361299416 (22h46m20.400872536s) [upload=20069]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:14:44.361315918 (22h46m20.400889048s) [upload=20069]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:14:44.694804187 (22h46m20.734377307s) [upload=20069]: Done uploading segments... Sep 7 12:14:44.694824897 (22h46m20.734398057s) [upload=20069]: Upload finished. err: <nil> Duration: 6.613591917s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X........................................X...........X....] duration: 7.978743385s FAIL (2025-09-07 12:17:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:14:40.856932950 (22h46m16.89650607s) [upload=20069]: Starting upload Sep 7 12:14:40.856994037 (22h46m16.896567157s) [upload=20069]: Done starting upload Sep 7 12:14:40.857086575 (22h46m16.896659695s) [upload=20069]: Uploading segments... Sep 7 12:14:40.857171938 (22h46m16.896745058s) [upload=20069]: Got next segment. Inline: false Sep 7 12:14:40.857200673 (22h46m16.896773793s) [upload=20069]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:14:40.857205502 (22h46m16.896778622s) [upload=20069]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:14:42.120233071 (22h46m18.159806191s) [seg_pos={0 0}, upload=20069]: Begin upload segment... Sep 7 12:14:42.120272818 (22h46m18.159845938s) [seg_pos={0 0}, upload=20069]: Done begin upload segment. Sep 7 12:14:42.120293797 (22h46m18.159866917s) [noise=true , address=77.90.31.17:4772 , piece_id=DNMMY5VLQL6I3KSHCN425ZRXNLMP4U7HFEKGTPJRAWQEDVUYX6OQ, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.120352219 (22h46m18.159925339s) [noise=true , address=77.90.31.10:4517 , piece_id=DBN4ESGYA3E4XREB6ELWHGCSFY75IMEC2PZBDP72UH3CMT275ZIQ, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.120699504 (22h46m18.160272614s) [noise=true , address=45.91.116.8:4047 , piece_id=3M6HNAWRK7U5QTZ5YLMWRTU7YSKXL253X3HA7UK2I5FKAM6GXTCQ, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.120797101 (22h46m18.160370211s) [noise=true , address=45.91.116.10:4083 , piece_id=FAQIG2C27JZDH52CNQ3YUVIJXKVOAVV2NFHMRNWBIKOQO2F6NUZA, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.121037181 (22h46m18.160610301s) [noise=true , address=45.91.116.7:4009 , piece_id=VZ7VU2JBS3NJIUANO5CJOUC5Z4TMNLZ5757HKNTMZU5RQOXHAZWA, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.121089830 (22h46m18.16066294s) [noise=true , address=45.91.116.5:4299 , piece_id=YYNTEKRGUSIU332RXAJRMDITDUAGR4VB4LCWYHLCTG5GA3L5OLPQ, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.121462484 (22h46m18.161035604s) [noise=true , address=45.91.116.7:4011 , piece_id=Q72DS5LTF4KPUV7YO5R573VPOHXYUN7U5LFEUJ7H2NTHRJYLYSDA, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.121499795 (22h46m18.161072915s) [noise=true , address=45.91.116.6:4974 , piece_id=VQJ2SEXLWFTHTG64MGAKIZE2TTIY4T4CFSUKN2S67SSMPJPUZDMA, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.121789109 (22h46m18.161362229s) [noise=true , address=45.91.116.4:4940 , piece_id=5XFGG36XMTCIZGD5RWVXAADC6JSNHWSZBDQ6BH7AM6I3ZGGL67OA, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.121839626 (22h46m18.161412746s) [noise=true , address=45.91.116.6:4975 , piece_id=6OER362GG3P7EM5ICLYMTNTNFNCRRTRAQ6LHD7IYO5XVCM43YXPQ, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.122066399 (22h46m18.161639519s) [noise=true , address=45.91.116.14:4190 , piece_id=R775GNQVAYCFFXIZYE42VHXHME2ZUNFZG3PGFQU72CCKKDVIRD7A, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.122106396 (22h46m18.161679516s) [noise=true , address=45.91.116.13:4153 , piece_id=EHYIHEPY5XVEAGNAGYVVTEOAXEGKCHFVMOO3AEGCLTAVJG7E6O3A, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.122278445 (22h46m18.161851555s) [noise=true , address=45.91.116.8:4045 , piece_id=C474QMRWKSEBPCU2WTJSLK2HSLKM4IRD5YW436EUN4INXLCQ5L4Q, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.122446136 (22h46m18.162019256s) [noise=true , address=45.91.116.14:4191 , piece_id=ZK4ILC43X3IKG5BF23KGL4CSFAQF3QE4I3KUVVUFFOV3GKT34M2A, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.122491443 (22h46m18.162064563s) [noise=true , address=45.91.116.10:4082 , piece_id=HRIWTYC2CTFFCKA5ZUJOXC36JNB7QDHYAZ7UQTXWHXMJEWANS42A, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.122727184 (22h46m18.162300304s) [noise=true , address=77.90.31.16:4734 , piece_id=2XV5L6XTKOSOVUQY4QRDNI6KUDLGKWUPOQ2GOD6F5GYOZ3RD74PQ, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:43.923014265 (22h46m19.962587385s) [upload=20069]: Next returned nil segment Sep 7 12:14:43.923031448 (22h46m19.962604568s) [upload=20069]: Waiting for error group managing segments... Sep 7 12:14:44.244799200 (22h46m20.28437232s) [noise=true , address=45.91.116.8:4047 , piece_id=3M6HNAWRK7U5QTZ5YLMWRTU7YSKXL253X3HA7UK2I5FKAM6GXTCQ, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.253947506 (22h46m20.293520626s) [noise=true , address=45.91.116.6:4974 , piece_id=VQJ2SEXLWFTHTG64MGAKIZE2TTIY4T4CFSUKN2S67SSMPJPUZDMA, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.258645795 (22h46m20.298218905s) [noise=true , address=45.91.116.5:4299 , piece_id=YYNTEKRGUSIU332RXAJRMDITDUAGR4VB4LCWYHLCTG5GA3L5OLPQ, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.259059357 (22h46m20.298632477s) [noise=true , address=45.91.116.8:4045 , piece_id=C474QMRWKSEBPCU2WTJSLK2HSLKM4IRD5YW436EUN4INXLCQ5L4Q, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.260936849 (22h46m20.300509969s) [noise=true , address=45.91.116.7:4009 , piece_id=VZ7VU2JBS3NJIUANO5CJOUC5Z4TMNLZ5757HKNTMZU5RQOXHAZWA, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.281681895 (22h46m20.321255015s) [noise=true , address=45.91.116.13:4153 , piece_id=EHYIHEPY5XVEAGNAGYVVTEOAXEGKCHFVMOO3AEGCLTAVJG7E6O3A, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.283288820 (22h46m20.32286194s) [noise=true , address=45.91.116.7:4011 , piece_id=Q72DS5LTF4KPUV7YO5R573VPOHXYUN7U5LFEUJ7H2NTHRJYLYSDA, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.288615273 (22h46m20.328188393s) [noise=true , address=45.91.116.10:4082 , piece_id=HRIWTYC2CTFFCKA5ZUJOXC36JNB7QDHYAZ7UQTXWHXMJEWANS42A, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.298688398 (22h46m20.338261528s) [noise=true , address=77.90.31.17:4772 , piece_id=DNMMY5VLQL6I3KSHCN425ZRXNLMP4U7HFEKGTPJRAWQEDVUYX6OQ, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.315911695 (22h46m20.355484815s) [noise=true , address=45.91.116.6:4975 , piece_id=6OER362GG3P7EM5ICLYMTNTNFNCRRTRAQ6LHD7IYO5XVCM43YXPQ, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.328711859 (22h46m20.368284979s) [noise=true , address=77.90.31.10:4517 , piece_id=DBN4ESGYA3E4XREB6ELWHGCSFY75IMEC2PZBDP72UH3CMT275ZIQ, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.355695773 (22h46m20.395268883s) [noise=true , address=45.91.116.4:4940 , piece_id=5XFGG36XMTCIZGD5RWVXAADC6JSNHWSZBDQ6BH7AM6I3ZGGL67OA, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.360864083 (22h46m20.400437203s) [noise=true , address=45.91.116.10:4083 , piece_id=FAQIG2C27JZDH52CNQ3YUVIJXKVOAVV2NFHMRNWBIKOQO2F6NUZA, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.360887418 (22h46m20.400460538s) [seg_pos={0 0}, upload=20069]: Segment reached optimal threshold of 13 pieces. Sep 7 12:14:44.361008330 (22h46m20.40058145s) [noise=true , address=77.90.31.16:4734 , piece_id=2XV5L6XTKOSOVUQY4QRDNI6KUDLGKWUPOQ2GOD6F5GYOZ3RD74PQ, seg_pos={0 0}, upload=20069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: expected piece hash; context canceled; EOF Sep 7 12:14:44.361140151 (22h46m20.400713271s) [noise=true , address=45.91.116.14:4190 , piece_id=R775GNQVAYCFFXIZYE42VHXHME2ZUNFZG3PGFQU72CCKKDVIRD7A, seg_pos={0 0}, upload=20069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Sep 7 12:14:44.361168666 (22h46m20.400741786s) [noise=true , address=45.91.116.14:4191 , piece_id=ZK4ILC43X3IKG5BF23KGL4CSFAQF3QE4I3KUVVUFFOV3GKT34M2A, seg_pos={0 0}, upload=20069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Sep 7 12:14:44.361184867 (22h46m20.400757977s) [seg_pos={0 0}, upload=20069]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:14:44.361299416 (22h46m20.400872536s) [upload=20069]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:14:44.361315918 (22h46m20.400889048s) [upload=20069]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:14:44.694804187 (22h46m20.734377307s) [upload=20069]: Done uploading segments... Sep 7 12:14:44.694824897 (22h46m20.734398057s) [upload=20069]: Upload finished. err: <nil> Duration: 7.978770086s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X........................................X...........X...] duration: 6.382442937s FAIL (2025-09-07 12:16:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:14:40.856932950 (22h46m16.89650607s) [upload=20069]: Starting upload Sep 7 12:14:40.856994037 (22h46m16.896567157s) [upload=20069]: Done starting upload Sep 7 12:14:40.857086575 (22h46m16.896659695s) [upload=20069]: Uploading segments... Sep 7 12:14:40.857171938 (22h46m16.896745058s) [upload=20069]: Got next segment. Inline: false Sep 7 12:14:40.857200673 (22h46m16.896773793s) [upload=20069]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:14:40.857205502 (22h46m16.896778622s) [upload=20069]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:14:42.120233071 (22h46m18.159806191s) [seg_pos={0 0}, upload=20069]: Begin upload segment... Sep 7 12:14:42.120272818 (22h46m18.159845938s) [seg_pos={0 0}, upload=20069]: Done begin upload segment. Sep 7 12:14:42.120293797 (22h46m18.159866917s) [noise=true , address=77.90.31.17:4772 , piece_id=DNMMY5VLQL6I3KSHCN425ZRXNLMP4U7HFEKGTPJRAWQEDVUYX6OQ, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.120352219 (22h46m18.159925339s) [noise=true , address=77.90.31.10:4517 , piece_id=DBN4ESGYA3E4XREB6ELWHGCSFY75IMEC2PZBDP72UH3CMT275ZIQ, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.120699504 (22h46m18.160272614s) [noise=true , address=45.91.116.8:4047 , piece_id=3M6HNAWRK7U5QTZ5YLMWRTU7YSKXL253X3HA7UK2I5FKAM6GXTCQ, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.120797101 (22h46m18.160370211s) [noise=true , address=45.91.116.10:4083 , piece_id=FAQIG2C27JZDH52CNQ3YUVIJXKVOAVV2NFHMRNWBIKOQO2F6NUZA, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.121037181 (22h46m18.160610301s) [noise=true , address=45.91.116.7:4009 , piece_id=VZ7VU2JBS3NJIUANO5CJOUC5Z4TMNLZ5757HKNTMZU5RQOXHAZWA, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.121089830 (22h46m18.16066294s) [noise=true , address=45.91.116.5:4299 , piece_id=YYNTEKRGUSIU332RXAJRMDITDUAGR4VB4LCWYHLCTG5GA3L5OLPQ, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.121462484 (22h46m18.161035604s) [noise=true , address=45.91.116.7:4011 , piece_id=Q72DS5LTF4KPUV7YO5R573VPOHXYUN7U5LFEUJ7H2NTHRJYLYSDA, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.121499795 (22h46m18.161072915s) [noise=true , address=45.91.116.6:4974 , piece_id=VQJ2SEXLWFTHTG64MGAKIZE2TTIY4T4CFSUKN2S67SSMPJPUZDMA, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.121789109 (22h46m18.161362229s) [noise=true , address=45.91.116.4:4940 , piece_id=5XFGG36XMTCIZGD5RWVXAADC6JSNHWSZBDQ6BH7AM6I3ZGGL67OA, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.121839626 (22h46m18.161412746s) [noise=true , address=45.91.116.6:4975 , piece_id=6OER362GG3P7EM5ICLYMTNTNFNCRRTRAQ6LHD7IYO5XVCM43YXPQ, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.122066399 (22h46m18.161639519s) [noise=true , address=45.91.116.14:4190 , piece_id=R775GNQVAYCFFXIZYE42VHXHME2ZUNFZG3PGFQU72CCKKDVIRD7A, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.122106396 (22h46m18.161679516s) [noise=true , address=45.91.116.13:4153 , piece_id=EHYIHEPY5XVEAGNAGYVVTEOAXEGKCHFVMOO3AEGCLTAVJG7E6O3A, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.122278445 (22h46m18.161851555s) [noise=true , address=45.91.116.8:4045 , piece_id=C474QMRWKSEBPCU2WTJSLK2HSLKM4IRD5YW436EUN4INXLCQ5L4Q, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.122446136 (22h46m18.162019256s) [noise=true , address=45.91.116.14:4191 , piece_id=ZK4ILC43X3IKG5BF23KGL4CSFAQF3QE4I3KUVVUFFOV3GKT34M2A, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.122491443 (22h46m18.162064563s) [noise=true , address=45.91.116.10:4082 , piece_id=HRIWTYC2CTFFCKA5ZUJOXC36JNB7QDHYAZ7UQTXWHXMJEWANS42A, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.122727184 (22h46m18.162300304s) [noise=true , address=77.90.31.16:4734 , piece_id=2XV5L6XTKOSOVUQY4QRDNI6KUDLGKWUPOQ2GOD6F5GYOZ3RD74PQ, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:43.923014265 (22h46m19.962587385s) [upload=20069]: Next returned nil segment Sep 7 12:14:43.923031448 (22h46m19.962604568s) [upload=20069]: Waiting for error group managing segments... Sep 7 12:14:44.244799200 (22h46m20.28437232s) [noise=true , address=45.91.116.8:4047 , piece_id=3M6HNAWRK7U5QTZ5YLMWRTU7YSKXL253X3HA7UK2I5FKAM6GXTCQ, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.253947506 (22h46m20.293520626s) [noise=true , address=45.91.116.6:4974 , piece_id=VQJ2SEXLWFTHTG64MGAKIZE2TTIY4T4CFSUKN2S67SSMPJPUZDMA, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.258645795 (22h46m20.298218905s) [noise=true , address=45.91.116.5:4299 , piece_id=YYNTEKRGUSIU332RXAJRMDITDUAGR4VB4LCWYHLCTG5GA3L5OLPQ, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.259059357 (22h46m20.298632477s) [noise=true , address=45.91.116.8:4045 , piece_id=C474QMRWKSEBPCU2WTJSLK2HSLKM4IRD5YW436EUN4INXLCQ5L4Q, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.260936849 (22h46m20.300509969s) [noise=true , address=45.91.116.7:4009 , piece_id=VZ7VU2JBS3NJIUANO5CJOUC5Z4TMNLZ5757HKNTMZU5RQOXHAZWA, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.281681895 (22h46m20.321255015s) [noise=true , address=45.91.116.13:4153 , piece_id=EHYIHEPY5XVEAGNAGYVVTEOAXEGKCHFVMOO3AEGCLTAVJG7E6O3A, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.283288820 (22h46m20.32286194s) [noise=true , address=45.91.116.7:4011 , piece_id=Q72DS5LTF4KPUV7YO5R573VPOHXYUN7U5LFEUJ7H2NTHRJYLYSDA, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.288615273 (22h46m20.328188393s) [noise=true , address=45.91.116.10:4082 , piece_id=HRIWTYC2CTFFCKA5ZUJOXC36JNB7QDHYAZ7UQTXWHXMJEWANS42A, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.298688398 (22h46m20.338261528s) [noise=true , address=77.90.31.17:4772 , piece_id=DNMMY5VLQL6I3KSHCN425ZRXNLMP4U7HFEKGTPJRAWQEDVUYX6OQ, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.315911695 (22h46m20.355484815s) [noise=true , address=45.91.116.6:4975 , piece_id=6OER362GG3P7EM5ICLYMTNTNFNCRRTRAQ6LHD7IYO5XVCM43YXPQ, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.328711859 (22h46m20.368284979s) [noise=true , address=77.90.31.10:4517 , piece_id=DBN4ESGYA3E4XREB6ELWHGCSFY75IMEC2PZBDP72UH3CMT275ZIQ, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.355695773 (22h46m20.395268883s) [noise=true , address=45.91.116.4:4940 , piece_id=5XFGG36XMTCIZGD5RWVXAADC6JSNHWSZBDQ6BH7AM6I3ZGGL67OA, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.360864083 (22h46m20.400437203s) [noise=true , address=45.91.116.10:4083 , piece_id=FAQIG2C27JZDH52CNQ3YUVIJXKVOAVV2NFHMRNWBIKOQO2F6NUZA, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.360887418 (22h46m20.400460538s) [seg_pos={0 0}, upload=20069]: Segment reached optimal threshold of 13 pieces. Sep 7 12:14:44.361008330 (22h46m20.40058145s) [noise=true , address=77.90.31.16:4734 , piece_id=2XV5L6XTKOSOVUQY4QRDNI6KUDLGKWUPOQ2GOD6F5GYOZ3RD74PQ, seg_pos={0 0}, upload=20069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: expected piece hash; context canceled; EOF Sep 7 12:14:44.361140151 (22h46m20.400713271s) [noise=true , address=45.91.116.14:4190 , piece_id=R775GNQVAYCFFXIZYE42VHXHME2ZUNFZG3PGFQU72CCKKDVIRD7A, seg_pos={0 0}, upload=20069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Sep 7 12:14:44.361168666 (22h46m20.400741786s) [noise=true , address=45.91.116.14:4191 , piece_id=ZK4ILC43X3IKG5BF23KGL4CSFAQF3QE4I3KUVVUFFOV3GKT34M2A, seg_pos={0 0}, upload=20069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Sep 7 12:14:44.361184867 (22h46m20.400757977s) [seg_pos={0 0}, upload=20069]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:14:44.361299416 (22h46m20.400872536s) [upload=20069]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:14:44.361315918 (22h46m20.400889048s) [upload=20069]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:14:44.694804187 (22h46m20.734377307s) [upload=20069]: Done uploading segments... Sep 7 12:14:44.694824897 (22h46m20.734398057s) [upload=20069]: Upload finished. err: <nil> Duration: 6.382468968s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X........................................X...........X..] duration: 11.662995175s FAIL (2025-09-07 12:15:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:14:40.856932950 (22h46m16.89650607s) [upload=20069]: Starting upload Sep 7 12:14:40.856994037 (22h46m16.896567157s) [upload=20069]: Done starting upload Sep 7 12:14:40.857086575 (22h46m16.896659695s) [upload=20069]: Uploading segments... Sep 7 12:14:40.857171938 (22h46m16.896745058s) [upload=20069]: Got next segment. Inline: false Sep 7 12:14:40.857200673 (22h46m16.896773793s) [upload=20069]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:14:40.857205502 (22h46m16.896778622s) [upload=20069]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:14:42.120233071 (22h46m18.159806191s) [seg_pos={0 0}, upload=20069]: Begin upload segment... Sep 7 12:14:42.120272818 (22h46m18.159845938s) [seg_pos={0 0}, upload=20069]: Done begin upload segment. Sep 7 12:14:42.120293797 (22h46m18.159866917s) [noise=true , address=77.90.31.17:4772 , piece_id=DNMMY5VLQL6I3KSHCN425ZRXNLMP4U7HFEKGTPJRAWQEDVUYX6OQ, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.120352219 (22h46m18.159925339s) [noise=true , address=77.90.31.10:4517 , piece_id=DBN4ESGYA3E4XREB6ELWHGCSFY75IMEC2PZBDP72UH3CMT275ZIQ, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.120699504 (22h46m18.160272614s) [noise=true , address=45.91.116.8:4047 , piece_id=3M6HNAWRK7U5QTZ5YLMWRTU7YSKXL253X3HA7UK2I5FKAM6GXTCQ, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.120797101 (22h46m18.160370211s) [noise=true , address=45.91.116.10:4083 , piece_id=FAQIG2C27JZDH52CNQ3YUVIJXKVOAVV2NFHMRNWBIKOQO2F6NUZA, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.121037181 (22h46m18.160610301s) [noise=true , address=45.91.116.7:4009 , piece_id=VZ7VU2JBS3NJIUANO5CJOUC5Z4TMNLZ5757HKNTMZU5RQOXHAZWA, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.121089830 (22h46m18.16066294s) [noise=true , address=45.91.116.5:4299 , piece_id=YYNTEKRGUSIU332RXAJRMDITDUAGR4VB4LCWYHLCTG5GA3L5OLPQ, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.121462484 (22h46m18.161035604s) [noise=true , address=45.91.116.7:4011 , piece_id=Q72DS5LTF4KPUV7YO5R573VPOHXYUN7U5LFEUJ7H2NTHRJYLYSDA, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.121499795 (22h46m18.161072915s) [noise=true , address=45.91.116.6:4974 , piece_id=VQJ2SEXLWFTHTG64MGAKIZE2TTIY4T4CFSUKN2S67SSMPJPUZDMA, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.121789109 (22h46m18.161362229s) [noise=true , address=45.91.116.4:4940 , piece_id=5XFGG36XMTCIZGD5RWVXAADC6JSNHWSZBDQ6BH7AM6I3ZGGL67OA, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.121839626 (22h46m18.161412746s) [noise=true , address=45.91.116.6:4975 , piece_id=6OER362GG3P7EM5ICLYMTNTNFNCRRTRAQ6LHD7IYO5XVCM43YXPQ, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.122066399 (22h46m18.161639519s) [noise=true , address=45.91.116.14:4190 , piece_id=R775GNQVAYCFFXIZYE42VHXHME2ZUNFZG3PGFQU72CCKKDVIRD7A, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.122106396 (22h46m18.161679516s) [noise=true , address=45.91.116.13:4153 , piece_id=EHYIHEPY5XVEAGNAGYVVTEOAXEGKCHFVMOO3AEGCLTAVJG7E6O3A, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.122278445 (22h46m18.161851555s) [noise=true , address=45.91.116.8:4045 , piece_id=C474QMRWKSEBPCU2WTJSLK2HSLKM4IRD5YW436EUN4INXLCQ5L4Q, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.122446136 (22h46m18.162019256s) [noise=true , address=45.91.116.14:4191 , piece_id=ZK4ILC43X3IKG5BF23KGL4CSFAQF3QE4I3KUVVUFFOV3GKT34M2A, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.122491443 (22h46m18.162064563s) [noise=true , address=45.91.116.10:4082 , piece_id=HRIWTYC2CTFFCKA5ZUJOXC36JNB7QDHYAZ7UQTXWHXMJEWANS42A, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.122727184 (22h46m18.162300304s) [noise=true , address=77.90.31.16:4734 , piece_id=2XV5L6XTKOSOVUQY4QRDNI6KUDLGKWUPOQ2GOD6F5GYOZ3RD74PQ, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:43.923014265 (22h46m19.962587385s) [upload=20069]: Next returned nil segment Sep 7 12:14:43.923031448 (22h46m19.962604568s) [upload=20069]: Waiting for error group managing segments... Sep 7 12:14:44.244799200 (22h46m20.28437232s) [noise=true , address=45.91.116.8:4047 , piece_id=3M6HNAWRK7U5QTZ5YLMWRTU7YSKXL253X3HA7UK2I5FKAM6GXTCQ, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.253947506 (22h46m20.293520626s) [noise=true , address=45.91.116.6:4974 , piece_id=VQJ2SEXLWFTHTG64MGAKIZE2TTIY4T4CFSUKN2S67SSMPJPUZDMA, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.258645795 (22h46m20.298218905s) [noise=true , address=45.91.116.5:4299 , piece_id=YYNTEKRGUSIU332RXAJRMDITDUAGR4VB4LCWYHLCTG5GA3L5OLPQ, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.259059357 (22h46m20.298632477s) [noise=true , address=45.91.116.8:4045 , piece_id=C474QMRWKSEBPCU2WTJSLK2HSLKM4IRD5YW436EUN4INXLCQ5L4Q, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.260936849 (22h46m20.300509969s) [noise=true , address=45.91.116.7:4009 , piece_id=VZ7VU2JBS3NJIUANO5CJOUC5Z4TMNLZ5757HKNTMZU5RQOXHAZWA, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.281681895 (22h46m20.321255015s) [noise=true , address=45.91.116.13:4153 , piece_id=EHYIHEPY5XVEAGNAGYVVTEOAXEGKCHFVMOO3AEGCLTAVJG7E6O3A, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.283288820 (22h46m20.32286194s) [noise=true , address=45.91.116.7:4011 , piece_id=Q72DS5LTF4KPUV7YO5R573VPOHXYUN7U5LFEUJ7H2NTHRJYLYSDA, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.288615273 (22h46m20.328188393s) [noise=true , address=45.91.116.10:4082 , piece_id=HRIWTYC2CTFFCKA5ZUJOXC36JNB7QDHYAZ7UQTXWHXMJEWANS42A, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.298688398 (22h46m20.338261528s) [noise=true , address=77.90.31.17:4772 , piece_id=DNMMY5VLQL6I3KSHCN425ZRXNLMP4U7HFEKGTPJRAWQEDVUYX6OQ, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.315911695 (22h46m20.355484815s) [noise=true , address=45.91.116.6:4975 , piece_id=6OER362GG3P7EM5ICLYMTNTNFNCRRTRAQ6LHD7IYO5XVCM43YXPQ, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.328711859 (22h46m20.368284979s) [noise=true , address=77.90.31.10:4517 , piece_id=DBN4ESGYA3E4XREB6ELWHGCSFY75IMEC2PZBDP72UH3CMT275ZIQ, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.355695773 (22h46m20.395268883s) [noise=true , address=45.91.116.4:4940 , piece_id=5XFGG36XMTCIZGD5RWVXAADC6JSNHWSZBDQ6BH7AM6I3ZGGL67OA, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.360864083 (22h46m20.400437203s) [noise=true , address=45.91.116.10:4083 , piece_id=FAQIG2C27JZDH52CNQ3YUVIJXKVOAVV2NFHMRNWBIKOQO2F6NUZA, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.360887418 (22h46m20.400460538s) [seg_pos={0 0}, upload=20069]: Segment reached optimal threshold of 13 pieces. Sep 7 12:14:44.361008330 (22h46m20.40058145s) [noise=true , address=77.90.31.16:4734 , piece_id=2XV5L6XTKOSOVUQY4QRDNI6KUDLGKWUPOQ2GOD6F5GYOZ3RD74PQ, seg_pos={0 0}, upload=20069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: expected piece hash; context canceled; EOF Sep 7 12:14:44.361140151 (22h46m20.400713271s) [noise=true , address=45.91.116.14:4190 , piece_id=R775GNQVAYCFFXIZYE42VHXHME2ZUNFZG3PGFQU72CCKKDVIRD7A, seg_pos={0 0}, upload=20069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Sep 7 12:14:44.361168666 (22h46m20.400741786s) [noise=true , address=45.91.116.14:4191 , piece_id=ZK4ILC43X3IKG5BF23KGL4CSFAQF3QE4I3KUVVUFFOV3GKT34M2A, seg_pos={0 0}, upload=20069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Sep 7 12:14:44.361184867 (22h46m20.400757977s) [seg_pos={0 0}, upload=20069]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:14:44.361299416 (22h46m20.400872536s) [upload=20069]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:14:44.361315918 (22h46m20.400889048s) [upload=20069]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:14:44.694804187 (22h46m20.734377307s) [upload=20069]: Done uploading segments... Sep 7 12:14:44.694824897 (22h46m20.734398057s) [upload=20069]: Upload finished. err: <nil> Duration: 11.663022608s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X....X........................................X...........X.] duration: 13.689885548s FAIL (2025-09-07 12:15:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:14:40.856932950 (22h46m16.89650607s) [upload=20069]: Starting upload Sep 7 12:14:40.856994037 (22h46m16.896567157s) [upload=20069]: Done starting upload Sep 7 12:14:40.857086575 (22h46m16.896659695s) [upload=20069]: Uploading segments... Sep 7 12:14:40.857171938 (22h46m16.896745058s) [upload=20069]: Got next segment. Inline: false Sep 7 12:14:40.857200673 (22h46m16.896773793s) [upload=20069]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:14:40.857205502 (22h46m16.896778622s) [upload=20069]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:14:42.120233071 (22h46m18.159806191s) [seg_pos={0 0}, upload=20069]: Begin upload segment... Sep 7 12:14:42.120272818 (22h46m18.159845938s) [seg_pos={0 0}, upload=20069]: Done begin upload segment. Sep 7 12:14:42.120293797 (22h46m18.159866917s) [noise=true , address=77.90.31.17:4772 , piece_id=DNMMY5VLQL6I3KSHCN425ZRXNLMP4U7HFEKGTPJRAWQEDVUYX6OQ, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.120352219 (22h46m18.159925339s) [noise=true , address=77.90.31.10:4517 , piece_id=DBN4ESGYA3E4XREB6ELWHGCSFY75IMEC2PZBDP72UH3CMT275ZIQ, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.120699504 (22h46m18.160272614s) [noise=true , address=45.91.116.8:4047 , piece_id=3M6HNAWRK7U5QTZ5YLMWRTU7YSKXL253X3HA7UK2I5FKAM6GXTCQ, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.120797101 (22h46m18.160370211s) [noise=true , address=45.91.116.10:4083 , piece_id=FAQIG2C27JZDH52CNQ3YUVIJXKVOAVV2NFHMRNWBIKOQO2F6NUZA, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.121037181 (22h46m18.160610301s) [noise=true , address=45.91.116.7:4009 , piece_id=VZ7VU2JBS3NJIUANO5CJOUC5Z4TMNLZ5757HKNTMZU5RQOXHAZWA, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.121089830 (22h46m18.16066294s) [noise=true , address=45.91.116.5:4299 , piece_id=YYNTEKRGUSIU332RXAJRMDITDUAGR4VB4LCWYHLCTG5GA3L5OLPQ, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.121462484 (22h46m18.161035604s) [noise=true , address=45.91.116.7:4011 , piece_id=Q72DS5LTF4KPUV7YO5R573VPOHXYUN7U5LFEUJ7H2NTHRJYLYSDA, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.121499795 (22h46m18.161072915s) [noise=true , address=45.91.116.6:4974 , piece_id=VQJ2SEXLWFTHTG64MGAKIZE2TTIY4T4CFSUKN2S67SSMPJPUZDMA, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.121789109 (22h46m18.161362229s) [noise=true , address=45.91.116.4:4940 , piece_id=5XFGG36XMTCIZGD5RWVXAADC6JSNHWSZBDQ6BH7AM6I3ZGGL67OA, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.121839626 (22h46m18.161412746s) [noise=true , address=45.91.116.6:4975 , piece_id=6OER362GG3P7EM5ICLYMTNTNFNCRRTRAQ6LHD7IYO5XVCM43YXPQ, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.122066399 (22h46m18.161639519s) [noise=true , address=45.91.116.14:4190 , piece_id=R775GNQVAYCFFXIZYE42VHXHME2ZUNFZG3PGFQU72CCKKDVIRD7A, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.122106396 (22h46m18.161679516s) [noise=true , address=45.91.116.13:4153 , piece_id=EHYIHEPY5XVEAGNAGYVVTEOAXEGKCHFVMOO3AEGCLTAVJG7E6O3A, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.122278445 (22h46m18.161851555s) [noise=true , address=45.91.116.8:4045 , piece_id=C474QMRWKSEBPCU2WTJSLK2HSLKM4IRD5YW436EUN4INXLCQ5L4Q, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.122446136 (22h46m18.162019256s) [noise=true , address=45.91.116.14:4191 , piece_id=ZK4ILC43X3IKG5BF23KGL4CSFAQF3QE4I3KUVVUFFOV3GKT34M2A, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.122491443 (22h46m18.162064563s) [noise=true , address=45.91.116.10:4082 , piece_id=HRIWTYC2CTFFCKA5ZUJOXC36JNB7QDHYAZ7UQTXWHXMJEWANS42A, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.122727184 (22h46m18.162300304s) [noise=true , address=77.90.31.16:4734 , piece_id=2XV5L6XTKOSOVUQY4QRDNI6KUDLGKWUPOQ2GOD6F5GYOZ3RD74PQ, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:43.923014265 (22h46m19.962587385s) [upload=20069]: Next returned nil segment Sep 7 12:14:43.923031448 (22h46m19.962604568s) [upload=20069]: Waiting for error group managing segments... Sep 7 12:14:44.244799200 (22h46m20.28437232s) [noise=true , address=45.91.116.8:4047 , piece_id=3M6HNAWRK7U5QTZ5YLMWRTU7YSKXL253X3HA7UK2I5FKAM6GXTCQ, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.253947506 (22h46m20.293520626s) [noise=true , address=45.91.116.6:4974 , piece_id=VQJ2SEXLWFTHTG64MGAKIZE2TTIY4T4CFSUKN2S67SSMPJPUZDMA, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.258645795 (22h46m20.298218905s) [noise=true , address=45.91.116.5:4299 , piece_id=YYNTEKRGUSIU332RXAJRMDITDUAGR4VB4LCWYHLCTG5GA3L5OLPQ, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.259059357 (22h46m20.298632477s) [noise=true , address=45.91.116.8:4045 , piece_id=C474QMRWKSEBPCU2WTJSLK2HSLKM4IRD5YW436EUN4INXLCQ5L4Q, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.260936849 (22h46m20.300509969s) [noise=true , address=45.91.116.7:4009 , piece_id=VZ7VU2JBS3NJIUANO5CJOUC5Z4TMNLZ5757HKNTMZU5RQOXHAZWA, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.281681895 (22h46m20.321255015s) [noise=true , address=45.91.116.13:4153 , piece_id=EHYIHEPY5XVEAGNAGYVVTEOAXEGKCHFVMOO3AEGCLTAVJG7E6O3A, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.283288820 (22h46m20.32286194s) [noise=true , address=45.91.116.7:4011 , piece_id=Q72DS5LTF4KPUV7YO5R573VPOHXYUN7U5LFEUJ7H2NTHRJYLYSDA, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.288615273 (22h46m20.328188393s) [noise=true , address=45.91.116.10:4082 , piece_id=HRIWTYC2CTFFCKA5ZUJOXC36JNB7QDHYAZ7UQTXWHXMJEWANS42A, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.298688398 (22h46m20.338261528s) [noise=true , address=77.90.31.17:4772 , piece_id=DNMMY5VLQL6I3KSHCN425ZRXNLMP4U7HFEKGTPJRAWQEDVUYX6OQ, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.315911695 (22h46m20.355484815s) [noise=true , address=45.91.116.6:4975 , piece_id=6OER362GG3P7EM5ICLYMTNTNFNCRRTRAQ6LHD7IYO5XVCM43YXPQ, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.328711859 (22h46m20.368284979s) [noise=true , address=77.90.31.10:4517 , piece_id=DBN4ESGYA3E4XREB6ELWHGCSFY75IMEC2PZBDP72UH3CMT275ZIQ, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.355695773 (22h46m20.395268883s) [noise=true , address=45.91.116.4:4940 , piece_id=5XFGG36XMTCIZGD5RWVXAADC6JSNHWSZBDQ6BH7AM6I3ZGGL67OA, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.360864083 (22h46m20.400437203s) [noise=true , address=45.91.116.10:4083 , piece_id=FAQIG2C27JZDH52CNQ3YUVIJXKVOAVV2NFHMRNWBIKOQO2F6NUZA, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.360887418 (22h46m20.400460538s) [seg_pos={0 0}, upload=20069]: Segment reached optimal threshold of 13 pieces. Sep 7 12:14:44.361008330 (22h46m20.40058145s) [noise=true , address=77.90.31.16:4734 , piece_id=2XV5L6XTKOSOVUQY4QRDNI6KUDLGKWUPOQ2GOD6F5GYOZ3RD74PQ, seg_pos={0 0}, upload=20069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: expected piece hash; context canceled; EOF Sep 7 12:14:44.361140151 (22h46m20.400713271s) [noise=true , address=45.91.116.14:4190 , piece_id=R775GNQVAYCFFXIZYE42VHXHME2ZUNFZG3PGFQU72CCKKDVIRD7A, seg_pos={0 0}, upload=20069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Sep 7 12:14:44.361168666 (22h46m20.400741786s) [noise=true , address=45.91.116.14:4191 , piece_id=ZK4ILC43X3IKG5BF23KGL4CSFAQF3QE4I3KUVVUFFOV3GKT34M2A, seg_pos={0 0}, upload=20069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Sep 7 12:14:44.361184867 (22h46m20.400757977s) [seg_pos={0 0}, upload=20069]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:14:44.361299416 (22h46m20.400872536s) [upload=20069]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:14:44.361315918 (22h46m20.400889048s) [upload=20069]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:14:44.694804187 (22h46m20.734377307s) [upload=20069]: Done uploading segments... Sep 7 12:14:44.694824897 (22h46m20.734398057s) [upload=20069]: Upload finished. err: <nil> Duration: 13.689901098s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X....X........................................X...........X] duration: 30.002120698s FAIL (2025-09-07 12:15:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 7 12:14:40.856932950 (22h46m16.89650607s) [upload=20069]: Starting upload Sep 7 12:14:40.856994037 (22h46m16.896567157s) [upload=20069]: Done starting upload Sep 7 12:14:40.857086575 (22h46m16.896659695s) [upload=20069]: Uploading segments... Sep 7 12:14:40.857171938 (22h46m16.896745058s) [upload=20069]: Got next segment. Inline: false Sep 7 12:14:40.857200673 (22h46m16.896773793s) [upload=20069]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 7 12:14:40.857205502 (22h46m16.896778622s) [upload=20069]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 7 12:14:42.120233071 (22h46m18.159806191s) [seg_pos={0 0}, upload=20069]: Begin upload segment... Sep 7 12:14:42.120272818 (22h46m18.159845938s) [seg_pos={0 0}, upload=20069]: Done begin upload segment. Sep 7 12:14:42.120293797 (22h46m18.159866917s) [noise=true , address=77.90.31.17:4772 , piece_id=DNMMY5VLQL6I3KSHCN425ZRXNLMP4U7HFEKGTPJRAWQEDVUYX6OQ, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.120352219 (22h46m18.159925339s) [noise=true , address=77.90.31.10:4517 , piece_id=DBN4ESGYA3E4XREB6ELWHGCSFY75IMEC2PZBDP72UH3CMT275ZIQ, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.120699504 (22h46m18.160272614s) [noise=true , address=45.91.116.8:4047 , piece_id=3M6HNAWRK7U5QTZ5YLMWRTU7YSKXL253X3HA7UK2I5FKAM6GXTCQ, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.120797101 (22h46m18.160370211s) [noise=true , address=45.91.116.10:4083 , piece_id=FAQIG2C27JZDH52CNQ3YUVIJXKVOAVV2NFHMRNWBIKOQO2F6NUZA, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.121037181 (22h46m18.160610301s) [noise=true , address=45.91.116.7:4009 , piece_id=VZ7VU2JBS3NJIUANO5CJOUC5Z4TMNLZ5757HKNTMZU5RQOXHAZWA, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.121089830 (22h46m18.16066294s) [noise=true , address=45.91.116.5:4299 , piece_id=YYNTEKRGUSIU332RXAJRMDITDUAGR4VB4LCWYHLCTG5GA3L5OLPQ, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.121462484 (22h46m18.161035604s) [noise=true , address=45.91.116.7:4011 , piece_id=Q72DS5LTF4KPUV7YO5R573VPOHXYUN7U5LFEUJ7H2NTHRJYLYSDA, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.121499795 (22h46m18.161072915s) [noise=true , address=45.91.116.6:4974 , piece_id=VQJ2SEXLWFTHTG64MGAKIZE2TTIY4T4CFSUKN2S67SSMPJPUZDMA, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.121789109 (22h46m18.161362229s) [noise=true , address=45.91.116.4:4940 , piece_id=5XFGG36XMTCIZGD5RWVXAADC6JSNHWSZBDQ6BH7AM6I3ZGGL67OA, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.121839626 (22h46m18.161412746s) [noise=true , address=45.91.116.6:4975 , piece_id=6OER362GG3P7EM5ICLYMTNTNFNCRRTRAQ6LHD7IYO5XVCM43YXPQ, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.122066399 (22h46m18.161639519s) [noise=true , address=45.91.116.14:4190 , piece_id=R775GNQVAYCFFXIZYE42VHXHME2ZUNFZG3PGFQU72CCKKDVIRD7A, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.122106396 (22h46m18.161679516s) [noise=true , address=45.91.116.13:4153 , piece_id=EHYIHEPY5XVEAGNAGYVVTEOAXEGKCHFVMOO3AEGCLTAVJG7E6O3A, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.122278445 (22h46m18.161851555s) [noise=true , address=45.91.116.8:4045 , piece_id=C474QMRWKSEBPCU2WTJSLK2HSLKM4IRD5YW436EUN4INXLCQ5L4Q, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.122446136 (22h46m18.162019256s) [noise=true , address=45.91.116.14:4191 , piece_id=ZK4ILC43X3IKG5BF23KGL4CSFAQF3QE4I3KUVVUFFOV3GKT34M2A, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.122491443 (22h46m18.162064563s) [noise=true , address=45.91.116.10:4082 , piece_id=HRIWTYC2CTFFCKA5ZUJOXC36JNB7QDHYAZ7UQTXWHXMJEWANS42A, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:42.122727184 (22h46m18.162300304s) [noise=true , address=77.90.31.16:4734 , piece_id=2XV5L6XTKOSOVUQY4QRDNI6KUDLGKWUPOQ2GOD6F5GYOZ3RD74PQ, seg_pos={0 0}, upload=20069]: Uploading piece... Sep 7 12:14:43.923014265 (22h46m19.962587385s) [upload=20069]: Next returned nil segment Sep 7 12:14:43.923031448 (22h46m19.962604568s) [upload=20069]: Waiting for error group managing segments... Sep 7 12:14:44.244799200 (22h46m20.28437232s) [noise=true , address=45.91.116.8:4047 , piece_id=3M6HNAWRK7U5QTZ5YLMWRTU7YSKXL253X3HA7UK2I5FKAM6GXTCQ, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.253947506 (22h46m20.293520626s) [noise=true , address=45.91.116.6:4974 , piece_id=VQJ2SEXLWFTHTG64MGAKIZE2TTIY4T4CFSUKN2S67SSMPJPUZDMA, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.258645795 (22h46m20.298218905s) [noise=true , address=45.91.116.5:4299 , piece_id=YYNTEKRGUSIU332RXAJRMDITDUAGR4VB4LCWYHLCTG5GA3L5OLPQ, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.259059357 (22h46m20.298632477s) [noise=true , address=45.91.116.8:4045 , piece_id=C474QMRWKSEBPCU2WTJSLK2HSLKM4IRD5YW436EUN4INXLCQ5L4Q, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.260936849 (22h46m20.300509969s) [noise=true , address=45.91.116.7:4009 , piece_id=VZ7VU2JBS3NJIUANO5CJOUC5Z4TMNLZ5757HKNTMZU5RQOXHAZWA, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.281681895 (22h46m20.321255015s) [noise=true , address=45.91.116.13:4153 , piece_id=EHYIHEPY5XVEAGNAGYVVTEOAXEGKCHFVMOO3AEGCLTAVJG7E6O3A, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.283288820 (22h46m20.32286194s) [noise=true , address=45.91.116.7:4011 , piece_id=Q72DS5LTF4KPUV7YO5R573VPOHXYUN7U5LFEUJ7H2NTHRJYLYSDA, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.288615273 (22h46m20.328188393s) [noise=true , address=45.91.116.10:4082 , piece_id=HRIWTYC2CTFFCKA5ZUJOXC36JNB7QDHYAZ7UQTXWHXMJEWANS42A, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.298688398 (22h46m20.338261528s) [noise=true , address=77.90.31.17:4772 , piece_id=DNMMY5VLQL6I3KSHCN425ZRXNLMP4U7HFEKGTPJRAWQEDVUYX6OQ, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.315911695 (22h46m20.355484815s) [noise=true , address=45.91.116.6:4975 , piece_id=6OER362GG3P7EM5ICLYMTNTNFNCRRTRAQ6LHD7IYO5XVCM43YXPQ, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.328711859 (22h46m20.368284979s) [noise=true , address=77.90.31.10:4517 , piece_id=DBN4ESGYA3E4XREB6ELWHGCSFY75IMEC2PZBDP72UH3CMT275ZIQ, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.355695773 (22h46m20.395268883s) [noise=true , address=45.91.116.4:4940 , piece_id=5XFGG36XMTCIZGD5RWVXAADC6JSNHWSZBDQ6BH7AM6I3ZGGL67OA, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.360864083 (22h46m20.400437203s) [noise=true , address=45.91.116.10:4083 , piece_id=FAQIG2C27JZDH52CNQ3YUVIJXKVOAVV2NFHMRNWBIKOQO2F6NUZA, seg_pos={0 0}, upload=20069]: Done uploading piece. err: <nil> Sep 7 12:14:44.360887418 (22h46m20.400460538s) [seg_pos={0 0}, upload=20069]: Segment reached optimal threshold of 13 pieces. Sep 7 12:14:44.361008330 (22h46m20.40058145s) [noise=true , address=77.90.31.16:4734 , piece_id=2XV5L6XTKOSOVUQY4QRDNI6KUDLGKWUPOQ2GOD6F5GYOZ3RD74PQ, seg_pos={0 0}, upload=20069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: expected piece hash; context canceled; EOF Sep 7 12:14:44.361140151 (22h46m20.400713271s) [noise=true , address=45.91.116.14:4190 , piece_id=R775GNQVAYCFFXIZYE42VHXHME2ZUNFZG3PGFQU72CCKKDVIRD7A, seg_pos={0 0}, upload=20069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Sep 7 12:14:44.361168666 (22h46m20.400741786s) [noise=true , address=45.91.116.14:4191 , piece_id=ZK4ILC43X3IKG5BF23KGL4CSFAQF3QE4I3KUVVUFFOV3GKT34M2A, seg_pos={0 0}, upload=20069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Sep 7 12:14:44.361184867 (22h46m20.400757977s) [seg_pos={0 0}, upload=20069]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 7 12:14:44.361299416 (22h46m20.400872536s) [upload=20069]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 7 12:14:44.361315918 (22h46m20.400889048s) [upload=20069]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 7 12:14:44.694804187 (22h46m20.734377307s) [upload=20069]: Done uploading segments... Sep 7 12:14:44.694824897 (22h46m20.734398057s) [upload=20069]: Upload finished. err: <nil> Duration: 30.00214307s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)