Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....X....X..................................................] duration: 5.628224445s FAIL (2025-10-26 14:16:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.628257768s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X.......X.X..........................................] duration: 5.977715057s FAIL (2025-10-26 14:16:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 5.977752709s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X.......X.X.........................................] duration: 6.727385383s FAIL (2025-10-26 14:16:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 6.727412615s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....X....X.................................................] duration: 5.703929741s FAIL (2025-10-26 14:15:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.703953386s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X.......X.X........................................] duration: 5.924240034s FAIL (2025-10-26 14:15:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 5.924276814s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......X....X................................................] duration: 5.773401838s FAIL (2025-10-26 14:15:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.773430013s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......X....X...............................................] duration: 5.707063188s FAIL (2025-10-26 14:14:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.707086033s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X.......X.X.......................................] duration: 6.800173775s FAIL (2025-10-26 14:14:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 6.800210375s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........X....X..............................................] duration: 5.789459333s FAIL (2025-10-26 14:14:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.789487778s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X.......X.X......................................] duration: 6.00533007s FAIL (2025-10-26 14:13:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 6.005353444s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........X....X.............................................] duration: 5.617531576s FAIL (2025-10-26 14:13:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.617557686s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X.......X.X.....................................] duration: 5.978023437s FAIL (2025-10-26 14:13:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 5.978053674s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X.......X.X....................................] duration: 5.957913659s FAIL (2025-10-26 14:12:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 5.957947574s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........X....X............................................] duration: 5.593726219s FAIL (2025-10-26 14:12:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.593787676s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X.......X.X...................................] duration: 6.136590438s FAIL (2025-10-26 14:11:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 6.136626607s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........X....X...........................................] duration: 5.661056048s FAIL (2025-10-26 14:11:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.661087096s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X.......X.X..................................] duration: 5.768633238s FAIL (2025-10-26 14:10:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 5.768668285s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X.......X.X.................................] duration: 5.775824821s FAIL (2025-10-26 14:10:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 5.775854489s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............X....X..........................................] duration: 5.797694387s FAIL (2025-10-26 14:10:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.797788525s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X.......X.X................................] duration: 5.901661615s FAIL (2025-10-26 14:09:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 5.901698836s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X.......X.X...............................] duration: 5.806379222s FAIL (2025-10-26 14:09:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 5.806412256s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............X....X.........................................] duration: 5.81631315s FAIL (2025-10-26 14:09:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.816411907s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X.......X.X..............................] duration: 6.108077521s FAIL (2025-10-26 14:08:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 6.108111776s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X.......X.X.............................] duration: 6.003398128s FAIL (2025-10-26 14:08:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 6.003451058s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............X....X........................................] duration: 5.729351275s FAIL (2025-10-26 14:08:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.729470092s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X.......X.X............................] duration: 6.399222062s FAIL (2025-10-26 14:07:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 6.399255857s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............X....X.......................................] duration: 5.659276271s FAIL (2025-10-26 14:07:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.659325344s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X.......X.X...........................] duration: 6.002812173s FAIL (2025-10-26 14:06:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 6.002839697s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................X....X......................................] duration: 5.723658095s FAIL (2025-10-26 14:06:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.723675077s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X.......X.X..........................] duration: 6.809992222s FAIL (2025-10-26 14:06:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 6.810173707s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X.......X.X.........................] duration: 5.845880702s FAIL (2025-10-26 14:05:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 5.845934243s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................X....X.....................................] duration: 5.889764034s FAIL (2025-10-26 14:05:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.889791878s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X.......X.X........................] duration: 5.953126601s FAIL (2025-10-26 14:05:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 5.953167276s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................X....X....................................] duration: 5.811558445s FAIL (2025-10-26 14:04:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.811649748s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................X....X...................................] duration: 5.634882556s FAIL (2025-10-26 14:04:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.634909497s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................X.......X.X.......................] duration: 5.80574783s FAIL (2025-10-26 14:03:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 5.805775391s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................X....X..................................] duration: 8.34193786s FAIL (2025-10-26 14:03:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.341970462s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................X.......X.X......................] duration: 5.842744273s FAIL (2025-10-26 14:03:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 5.842781264s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................X....X.................................] duration: 5.630261034s FAIL (2025-10-26 14:02:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.630288727s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................X.......X.X.....................] duration: 5.859067764s FAIL (2025-10-26 14:02:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 5.859109854s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................X....X................................] duration: 5.650492593s FAIL (2025-10-26 14:01:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.650526095s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................X.......X.X....................] duration: 5.914269461s FAIL (2025-10-26 14:01:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 5.914303234s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................X.......X.X...................] duration: 6.131820811s FAIL (2025-10-26 14:00:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 6.131884785s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................X....X...............................] duration: 5.74742782s FAIL (2025-10-26 14:00:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.74745931s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................X....X..............................] duration: 5.810810719s FAIL (2025-10-26 14:00:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.810841488s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................X.......X.X..................] duration: 6.225412126s FAIL (2025-10-26 14:00:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 6.225460048s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........................X....X.............................] duration: 22.023393091s FAIL (2025-10-26 13:59:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.023424762s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................X.......X.X.................] duration: 6.123953661s FAIL (2025-10-26 13:59:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 6.124002153s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................X....X............................] duration: 17.737309006s FAIL (2025-10-26 13:58:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.737336418s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................X.......X.X................] duration: 10.534654822s FAIL (2025-10-26 13:58:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ) context deadline exceeded; piecestore: (Node ID: 12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, Piece ID: C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA) context deadline exceeded ======LOGS====== 46.163905297 (792h16m31.62599778s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 13:47:46.415507525 (792h16m31.877600008s) [seg_pos={0 0}, upload=696923]: Begin upload segment... Oct 26 13:47:46.415570155 (792h16m31.877662638s) [seg_pos={0 0}, upload=696923]: Done begin upload segment. Oct 26 13:47:46.415601114 (792h16m31.877693587s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.415609439 (792h16m31.877701922s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416219802 (792h16m31.878312275s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416325654 (792h16m31.878418127s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416790368 (792h16m31.878882841s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.416898424 (792h16m31.878990907s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417153720 (792h16m31.879246203s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417479861 (792h16m31.879572334s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.417580132 (792h16m31.879672615s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418046931 (792h16m31.880139404s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418075976 (792h16m31.880168459s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418561250 (792h16m31.880653723s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.418583743 (792h16m31.880676226s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419167324 (792h16m31.881259807s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.419776234 (792h16m31.881868717s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:46.420331061 (792h16m31.882423544s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Uploading piece... Oct 26 13:47:48.164291102 (792h16m33.626383585s) [upload=696923]: Next returned nil segment Oct 26 13:47:48.164311901 (792h16m33.626404384s) [upload=696923]: Waiting for error group managing segments... Oct 26 13:47:48.445892354 (792h16m33.907984827s) [noise=true , address=145.79.227.13:4801 , piece_id=SHHQPXNSLTCIQTCN2YKQXIPXHXFJWTLEQQWXFRRVUO4LKZUUII6Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.450335054 (792h16m33.912427537s) [noise=true , address=145.79.227.15:4876 , piece_id=S35OYD4JPNT3B5HOBK52HIWVRYU3YDQFL6NFIJAPCX2CSUMNNUFQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.453409137 (792h16m33.91550161s) [noise=true , address=145.79.227.23:4545 , piece_id=LQBBNGUF4NPZSKKRFLHX5SKJXZOXVDJP5HK5G7PS4DF2ULHSU2RQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.460362688 (792h16m33.922455171s) [noise=true , address=145.79.227.22:4511 , piece_id=CQTBNFS3II3CH6WKVZF6DCNFEY24MNA25XPVCEJODHYVGJPMNBKQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.465456998 (792h16m33.927549472s) [noise=true , address=145.79.227.17:4971 , piece_id=N36JVGZXZXW5ZLW34U7OQVW5EOGEU7ZMUB6AZUSXLLP6OWPNUX7A, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.483352365 (792h16m33.945444838s) [noise=true , address=145.79.227.24:4583 , piece_id=T6PBGXVYE6W4FCSF7BFRAPRARZVDYIPOA2NWA7ICWB5IOEQJBVFA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.485927960 (792h16m33.948020443s) [noise=true , address=145.79.227.16:4911 , piece_id=IEMZXRFII4YRQXO3OPE7TFL2DSY56ILH3QAMDZASUQJNXSVD5CVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.495117638 (792h16m33.957210121s) [noise=true , address=145.79.227.8:4329 , piece_id=GXVGLDCHCRDPJU3PZMAY2L7FAYOA2RSKEFXRIRAH4XZI6Z536KCQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.519274919 (792h16m33.981367392s) [noise=true , address=145.79.227.9:4367 , piece_id=ADMPXE7P3DD5YUMPQ45U232EPMH45FTPLJPXBSFACTRCQKCQ6EVQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.707682332 (792h16m34.169774815s) [noise=true , address=77.90.31.18:4807 , piece_id=C2QLRDTBWJPZAAJFW532LXSMNM7TK2AVGLK3X7NYILCCYD2HIFJA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.716174933 (792h16m34.178267416s) [noise=true , address=77.90.31.19:4843 , piece_id=VKRUCH25NDBQXIPU6ELBC5MWRNZHLVPMS7GUUY42AGL3VQEMPIIA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.751215507 (792h16m34.21330798s) [noise=true , address=77.90.31.13:4627 , piece_id=HZPRWSTMN6G4RKMQM2HSZWS324DE5B53LK64N3ARWWZNYAVOEEQQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800219771 (792h16m34.262312244s) [noise=true , address=77.90.31.10:4519 , piece_id=UIO56OMGKB6UIOPJH7NYGT6CMTPGJVWWTYQJF6QNN3X4ETSANZRQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: <nil> Oct 26 13:47:48.800251962 (792h16m34.262344435s) [seg_pos={0 0}, upload=696923]: Segment reached optimal threshold of 13 pieces. Oct 26 13:47:48.800440811 (792h16m34.262533285s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35544->77.90.31.21:4878: use of closed network connection Oct 26 13:47:48.800451963 (792h16m34.262544436s) [noise=true , address=77.90.31.21:4878 , piece_id=XIDVOPLYOMVZRXIWAQTRFXIRX2XVBDOMGSGTY6M6TR6UBOPFC6PQ, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800509893 (792h16m34.262602376s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43428->77.90.31.11:4553: use of closed network connection Oct 26 13:47:48.800521125 (792h16m34.262613598s) [noise=true , address=77.90.31.11:4553 , piece_id=OSKXDLRMJGG536GZZWXYDTJOAXUV6C4PXVJDNAEB5IY6JQCPXR2Q, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800560450 (792h16m34.262652933s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->77.90.31.17:4772: use of closed network connection Oct 26 13:47:48.800568795 (792h16m34.262661279s) [noise=true , address=77.90.31.17:4772 , piece_id=A46PDJCIELRNQWHQDLAKXBYLMOXQC4SM6WHJ7ANXUO3A6PHWRFOA, seg_pos={0 0}, upload=696923]: Long tail cancellation due to optimal threshold Oct 26 13:47:48.800590437 (792h16m34.26268292s) [seg_pos={0 0}, upload=696923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 13:47:48.800731355 (792h16m34.262823838s) [upload=696923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 13:47:48.800739050 (792h16m34.262831523s) [upload=696923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 13:47:49.092388375 (792h16m34.554480858s) [upload=696923]: Done uploading segments... Oct 26 13:47:49.092408874 (792h16m34.554501347s) [upload=696923]: Upload finished. err: <nil> Duration:* 10.534684808s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)