Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.195519046s FAIL (2025-10-26 14:27:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 6.195547188s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.X.........................................................] duration: 6.014595149s FAIL (2025-10-26 14:27:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 6.014625257s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.X........................................................] duration: 6.021476628s FAIL (2025-10-26 14:26:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 6.021512276s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X.X.......................................................] duration: 7.117869451s FAIL (2025-10-26 14:25:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 7.117902804s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X.X......................................................] duration: 5.962829128s FAIL (2025-10-26 14:24:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 5.962860507s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.754577846s FAIL (2025-10-26 14:24:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.754915738s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X.X.....................................................] duration: 5.807217114s FAIL (2025-10-26 14:23:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 5.807248302s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.762733541s FAIL (2025-10-26 14:23:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.762759821s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X.X....................................................] duration: 5.809913687s FAIL (2025-10-26 14:23:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 5.809944967s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.73513864s FAIL (2025-10-26 14:22:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.735194987s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X.X...................................................] duration: 5.912363503s FAIL (2025-10-26 14:22:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 5.912393548s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X.X..................................................] duration: 5.902842775s FAIL (2025-10-26 14:21:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 5.902868184s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.......X.X.................................................] duration: 5.809381948s FAIL (2025-10-26 14:21:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 5.809424509s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.664903555s FAIL (2025-10-26 14:21:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.66494324s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.69667208s FAIL (2025-10-26 14:20:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.697045541s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X.......X.X................................................] duration: 5.844981321s FAIL (2025-10-26 14:20:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 5.845015376s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [X....X......................................................] duration: 5.778767569s FAIL (2025-10-26 14:20:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.778804818s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X.......X.X...............................................] duration: 6.487351872s FAIL (2025-10-26 14:20:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 6.487393371s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X.......X.X..............................................] duration: 5.816413535s FAIL (2025-10-26 14:19:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 5.816448332s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.X....X.....................................................] duration: 5.774875206s FAIL (2025-10-26 14:19:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.774905694s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X.......X.X.............................................] duration: 5.939940826s FAIL (2025-10-26 14:18:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 5.939973117s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..X....X....................................................] duration: 5.786405774s FAIL (2025-10-26 14:18:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.786435199s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X.......X.X............................................] duration: 5.750841111s FAIL (2025-10-26 14:18:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 5.750897177s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...X....X...................................................] duration: 5.8056396s FAIL (2025-10-26 14:17:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.805673736s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X.......X.X...........................................] duration: 6.723604062s FAIL (2025-10-26 14:17:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 6.723638557s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....X....X..................................................] duration: 5.628224445s FAIL (2025-10-26 14:16:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.628257768s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X.......X.X..........................................] duration: 5.977715057s FAIL (2025-10-26 14:16:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 5.977752709s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X.......X.X.........................................] duration: 6.727385383s FAIL (2025-10-26 14:16:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 6.727412615s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....X....X.................................................] duration: 5.703929741s FAIL (2025-10-26 14:15:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.703953386s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X.......X.X........................................] duration: 5.924240034s FAIL (2025-10-26 14:15:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 5.924276814s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......X....X................................................] duration: 5.773401838s FAIL (2025-10-26 14:15:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.773430013s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......X....X...............................................] duration: 5.707063188s FAIL (2025-10-26 14:14:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.707086033s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X.......X.X.......................................] duration: 6.800173775s FAIL (2025-10-26 14:14:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 6.800210375s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........X....X..............................................] duration: 5.789459333s FAIL (2025-10-26 14:14:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.789487778s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X.......X.X......................................] duration: 6.00533007s FAIL (2025-10-26 14:13:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 6.005353444s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........X....X.............................................] duration: 5.617531576s FAIL (2025-10-26 14:13:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.617557686s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X.......X.X.....................................] duration: 5.978023437s FAIL (2025-10-26 14:13:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 5.978053674s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X.......X.X....................................] duration: 5.957913659s FAIL (2025-10-26 14:12:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 5.957947574s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........X....X............................................] duration: 5.593726219s FAIL (2025-10-26 14:12:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.593787676s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X.......X.X...................................] duration: 6.136590438s FAIL (2025-10-26 14:11:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 6.136626607s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........X....X...........................................] duration: 5.661056048s FAIL (2025-10-26 14:11:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.661087096s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X.......X.X..................................] duration: 5.768633238s FAIL (2025-10-26 14:10:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 5.768668285s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X.......X.X.................................] duration: 5.775824821s FAIL (2025-10-26 14:10:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 5.775854489s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............X....X..........................................] duration: 5.797694387s FAIL (2025-10-26 14:10:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.797788525s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X.......X.X................................] duration: 5.901661615s FAIL (2025-10-26 14:09:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 5.901698836s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X.......X.X...............................] duration: 5.806379222s FAIL (2025-10-26 14:09:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 5.806412256s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............X....X.........................................] duration: 5.81631315s FAIL (2025-10-26 14:09:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.816411907s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X.......X.X..............................] duration: 6.108077521s FAIL (2025-10-26 14:08:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 6.108111776s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X.......X.X.............................] duration: 6.003398128s FAIL (2025-10-26 14:08:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 6.003451058s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............X....X........................................] duration: 5.729351275s FAIL (2025-10-26 14:08:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.729470092s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)