Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............X.......X....................................] duration: 5.632122023s FAIL (2025-07-09 23:46:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.632141642s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................X.......X...................................] duration: 25.642943787s FAIL (2025-07-09 23:45:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.642963106s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [X.......X............X......................................] duration: 21.270193366s FAIL (2025-07-09 23:45:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== uestItem_SegmentBegin Jul 9 23:12:42.134881962 (398h2m16.905523297s) [seg_pos={0 0}, upload=318452]: Begin upload segment... Jul 9 23:12:42.134950294 (398h2m16.905591629s) [seg_pos={0 0}, upload=318452]: Done begin upload segment. Jul 9 23:12:42.134981082 (398h2m16.905622417s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135020638 (398h2m16.905661983s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135575439 (398h2m16.906216774s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135895081 (398h2m16.906536416s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135916562 (398h2m16.906557897s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136222628 (398h2m16.906863963s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136501170 (398h2m16.907142506s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136773411 (398h2m16.907414746s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136994234 (398h2m16.907635559s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137216439 (398h2m16.907857774s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137458321 (398h2m16.908099656s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137623086 (398h2m16.908264421s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138134475 (398h2m16.90877581s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138499122 (398h2m16.909140458s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138736497 (398h2m16.909377822s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138951127 (398h2m16.909592452s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139222767 (398h2m16.909864102s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139541577 (398h2m16.910182912s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:43.510699884 (398h2m18.28134122s) [upload=318452]: Next returned nil segment Jul 9 23:12:43.510713381 (398h2m18.281354716s) [upload=318452]: Waiting for error group managing segments... Jul 9 23:12:44.093679677 (398h2m18.864321012s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139156961 (398h2m18.909798286s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139550674 (398h2m18.910192019s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139723775 (398h2m18.91036512s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.140606163 (398h2m18.911247489s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.141871074 (398h2m18.912512409s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143296170 (398h2m18.913937495s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143572189 (398h2m18.914213524s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:45.311783220 (398h2m20.082424555s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:47.074380105 (398h2m21.84502144s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:52.162572575 (398h2m26.93321391s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:13.763376135 (398h2m48.53401746s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:29.504846258 (398h3m4.275487583s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:41.049567150 (398h3m15.820208505s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:12cgYc8zVZh7w7iTTDC5zq1xK12wa5357zdAH3F3bZ21n1Y8uLg, address:104.36.82.17:4350): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049619941 (398h3m15.820261276s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1mLbGHntspYmsTVg8433HYSdAEMrXjGnDhmmvX1JmmgcxtdLgk, address:104.36.82.23:4464): protocol: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection Jul 9 23:13:41.049694543 (398h3m15.820335878s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:17SUmQEJJ3Qwe9yRq5DFTGaZBqGnz5Pu5YDri93Wk5NP3rcm5c, address:45.91.95.4:4023): protocol: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection Jul 9 23:13:41.049733979 (398h3m15.820375314s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1R6A6f6LTL3VDDg5gbhvQE55bRLJPDfAjViixeNX4KjVJpYFDE, address:45.91.95.3:4017): protocol: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection Jul 9 23:13:41.049758877 (398h3m15.820400212s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1m82Tdkxzub83FtYpd2Abhgf6StE1eFtxn1ESBi8BdHGGVhCu8, address:45.91.95.5:4025): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049823370 (398h3m15.820464705s) [seg_pos={0 0}, upload=318452]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 23:13:41.285661371 (398h3m16.056302706s) [upload=318452]: Done uploading segments... Jul 9 23:13:41.285688192 (398h3m16.056329517s) [upload=318452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 21.270219186s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................X.......X..................................] duration: 5.588925255s FAIL (2025-07-09 23:44:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.588941736s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.X.......X............X.....................................] duration: 25.35637731s FAIL (2025-07-09 23:44:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== uestItem_SegmentBegin Jul 9 23:12:42.134881962 (398h2m16.905523297s) [seg_pos={0 0}, upload=318452]: Begin upload segment... Jul 9 23:12:42.134950294 (398h2m16.905591629s) [seg_pos={0 0}, upload=318452]: Done begin upload segment. Jul 9 23:12:42.134981082 (398h2m16.905622417s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135020638 (398h2m16.905661983s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135575439 (398h2m16.906216774s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135895081 (398h2m16.906536416s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135916562 (398h2m16.906557897s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136222628 (398h2m16.906863963s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136501170 (398h2m16.907142506s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136773411 (398h2m16.907414746s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136994234 (398h2m16.907635559s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137216439 (398h2m16.907857774s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137458321 (398h2m16.908099656s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137623086 (398h2m16.908264421s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138134475 (398h2m16.90877581s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138499122 (398h2m16.909140458s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138736497 (398h2m16.909377822s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138951127 (398h2m16.909592452s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139222767 (398h2m16.909864102s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139541577 (398h2m16.910182912s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:43.510699884 (398h2m18.28134122s) [upload=318452]: Next returned nil segment Jul 9 23:12:43.510713381 (398h2m18.281354716s) [upload=318452]: Waiting for error group managing segments... Jul 9 23:12:44.093679677 (398h2m18.864321012s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139156961 (398h2m18.909798286s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139550674 (398h2m18.910192019s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139723775 (398h2m18.91036512s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.140606163 (398h2m18.911247489s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.141871074 (398h2m18.912512409s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143296170 (398h2m18.913937495s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143572189 (398h2m18.914213524s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:45.311783220 (398h2m20.082424555s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:47.074380105 (398h2m21.84502144s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:52.162572575 (398h2m26.93321391s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:13.763376135 (398h2m48.53401746s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:29.504846258 (398h3m4.275487583s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:41.049567150 (398h3m15.820208505s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:12cgYc8zVZh7w7iTTDC5zq1xK12wa5357zdAH3F3bZ21n1Y8uLg, address:104.36.82.17:4350): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049619941 (398h3m15.820261276s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1mLbGHntspYmsTVg8433HYSdAEMrXjGnDhmmvX1JmmgcxtdLgk, address:104.36.82.23:4464): protocol: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection Jul 9 23:13:41.049694543 (398h3m15.820335878s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:17SUmQEJJ3Qwe9yRq5DFTGaZBqGnz5Pu5YDri93Wk5NP3rcm5c, address:45.91.95.4:4023): protocol: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection Jul 9 23:13:41.049733979 (398h3m15.820375314s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1R6A6f6LTL3VDDg5gbhvQE55bRLJPDfAjViixeNX4KjVJpYFDE, address:45.91.95.3:4017): protocol: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection Jul 9 23:13:41.049758877 (398h3m15.820400212s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1m82Tdkxzub83FtYpd2Abhgf6StE1eFtxn1ESBi8BdHGGVhCu8, address:45.91.95.5:4025): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049823370 (398h3m15.820464705s) [seg_pos={0 0}, upload=318452]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 23:13:41.285661371 (398h3m16.056302706s) [upload=318452]: Done uploading segments... Jul 9 23:13:41.285688192 (398h3m16.056329517s) [upload=318452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 25.356401375s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................X.......X.................................] duration: 12.512090391s FAIL (2025-07-09 23:43:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.51211528s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..X.......X............X....................................] duration: 7.645380339s FAIL (2025-07-09 23:43:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== uestItem_SegmentBegin Jul 9 23:12:42.134881962 (398h2m16.905523297s) [seg_pos={0 0}, upload=318452]: Begin upload segment... Jul 9 23:12:42.134950294 (398h2m16.905591629s) [seg_pos={0 0}, upload=318452]: Done begin upload segment. Jul 9 23:12:42.134981082 (398h2m16.905622417s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135020638 (398h2m16.905661983s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135575439 (398h2m16.906216774s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135895081 (398h2m16.906536416s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135916562 (398h2m16.906557897s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136222628 (398h2m16.906863963s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136501170 (398h2m16.907142506s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136773411 (398h2m16.907414746s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136994234 (398h2m16.907635559s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137216439 (398h2m16.907857774s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137458321 (398h2m16.908099656s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137623086 (398h2m16.908264421s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138134475 (398h2m16.90877581s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138499122 (398h2m16.909140458s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138736497 (398h2m16.909377822s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138951127 (398h2m16.909592452s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139222767 (398h2m16.909864102s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139541577 (398h2m16.910182912s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:43.510699884 (398h2m18.28134122s) [upload=318452]: Next returned nil segment Jul 9 23:12:43.510713381 (398h2m18.281354716s) [upload=318452]: Waiting for error group managing segments... Jul 9 23:12:44.093679677 (398h2m18.864321012s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139156961 (398h2m18.909798286s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139550674 (398h2m18.910192019s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139723775 (398h2m18.91036512s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.140606163 (398h2m18.911247489s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.141871074 (398h2m18.912512409s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143296170 (398h2m18.913937495s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143572189 (398h2m18.914213524s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:45.311783220 (398h2m20.082424555s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:47.074380105 (398h2m21.84502144s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:52.162572575 (398h2m26.93321391s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:13.763376135 (398h2m48.53401746s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:29.504846258 (398h3m4.275487583s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:41.049567150 (398h3m15.820208505s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:12cgYc8zVZh7w7iTTDC5zq1xK12wa5357zdAH3F3bZ21n1Y8uLg, address:104.36.82.17:4350): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049619941 (398h3m15.820261276s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1mLbGHntspYmsTVg8433HYSdAEMrXjGnDhmmvX1JmmgcxtdLgk, address:104.36.82.23:4464): protocol: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection Jul 9 23:13:41.049694543 (398h3m15.820335878s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:17SUmQEJJ3Qwe9yRq5DFTGaZBqGnz5Pu5YDri93Wk5NP3rcm5c, address:45.91.95.4:4023): protocol: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection Jul 9 23:13:41.049733979 (398h3m15.820375314s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1R6A6f6LTL3VDDg5gbhvQE55bRLJPDfAjViixeNX4KjVJpYFDE, address:45.91.95.3:4017): protocol: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection Jul 9 23:13:41.049758877 (398h3m15.820400212s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1m82Tdkxzub83FtYpd2Abhgf6StE1eFtxn1ESBi8BdHGGVhCu8, address:45.91.95.5:4025): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049823370 (398h3m15.820464705s) [seg_pos={0 0}, upload=318452]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 23:13:41.285661371 (398h3m16.056302706s) [upload=318452]: Done uploading segments... Jul 9 23:13:41.285688192 (398h3m16.056329517s) [upload=318452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 7.645425655s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...X.......X............X...................................] duration: 26.739584086s FAIL (2025-07-09 23:42:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== uestItem_SegmentBegin Jul 9 23:12:42.134881962 (398h2m16.905523297s) [seg_pos={0 0}, upload=318452]: Begin upload segment... Jul 9 23:12:42.134950294 (398h2m16.905591629s) [seg_pos={0 0}, upload=318452]: Done begin upload segment. Jul 9 23:12:42.134981082 (398h2m16.905622417s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135020638 (398h2m16.905661983s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135575439 (398h2m16.906216774s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135895081 (398h2m16.906536416s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135916562 (398h2m16.906557897s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136222628 (398h2m16.906863963s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136501170 (398h2m16.907142506s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136773411 (398h2m16.907414746s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136994234 (398h2m16.907635559s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137216439 (398h2m16.907857774s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137458321 (398h2m16.908099656s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137623086 (398h2m16.908264421s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138134475 (398h2m16.90877581s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138499122 (398h2m16.909140458s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138736497 (398h2m16.909377822s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138951127 (398h2m16.909592452s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139222767 (398h2m16.909864102s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139541577 (398h2m16.910182912s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:43.510699884 (398h2m18.28134122s) [upload=318452]: Next returned nil segment Jul 9 23:12:43.510713381 (398h2m18.281354716s) [upload=318452]: Waiting for error group managing segments... Jul 9 23:12:44.093679677 (398h2m18.864321012s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139156961 (398h2m18.909798286s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139550674 (398h2m18.910192019s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139723775 (398h2m18.91036512s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.140606163 (398h2m18.911247489s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.141871074 (398h2m18.912512409s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143296170 (398h2m18.913937495s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143572189 (398h2m18.914213524s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:45.311783220 (398h2m20.082424555s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:47.074380105 (398h2m21.84502144s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:52.162572575 (398h2m26.93321391s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:13.763376135 (398h2m48.53401746s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:29.504846258 (398h3m4.275487583s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:41.049567150 (398h3m15.820208505s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:12cgYc8zVZh7w7iTTDC5zq1xK12wa5357zdAH3F3bZ21n1Y8uLg, address:104.36.82.17:4350): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049619941 (398h3m15.820261276s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1mLbGHntspYmsTVg8433HYSdAEMrXjGnDhmmvX1JmmgcxtdLgk, address:104.36.82.23:4464): protocol: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection Jul 9 23:13:41.049694543 (398h3m15.820335878s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:17SUmQEJJ3Qwe9yRq5DFTGaZBqGnz5Pu5YDri93Wk5NP3rcm5c, address:45.91.95.4:4023): protocol: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection Jul 9 23:13:41.049733979 (398h3m15.820375314s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1R6A6f6LTL3VDDg5gbhvQE55bRLJPDfAjViixeNX4KjVJpYFDE, address:45.91.95.3:4017): protocol: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection Jul 9 23:13:41.049758877 (398h3m15.820400212s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1m82Tdkxzub83FtYpd2Abhgf6StE1eFtxn1ESBi8BdHGGVhCu8, address:45.91.95.5:4025): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049823370 (398h3m15.820464705s) [seg_pos={0 0}, upload=318452]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 23:13:41.285661371 (398h3m16.056302706s) [upload=318452]: Done uploading segments... Jul 9 23:13:41.285688192 (398h3m16.056329517s) [upload=318452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 26.739607933s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................X.......X................................] duration: 13.636938887s FAIL (2025-07-09 23:42:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.636961099s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................X.......X...............................] duration: 11.720678733s FAIL (2025-07-09 23:42:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.72070801s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [....X.......X............X..................................] duration: 11.684411909s FAIL (2025-07-09 23:41:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== uestItem_SegmentBegin Jul 9 23:12:42.134881962 (398h2m16.905523297s) [seg_pos={0 0}, upload=318452]: Begin upload segment... Jul 9 23:12:42.134950294 (398h2m16.905591629s) [seg_pos={0 0}, upload=318452]: Done begin upload segment. Jul 9 23:12:42.134981082 (398h2m16.905622417s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135020638 (398h2m16.905661983s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135575439 (398h2m16.906216774s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135895081 (398h2m16.906536416s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135916562 (398h2m16.906557897s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136222628 (398h2m16.906863963s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136501170 (398h2m16.907142506s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136773411 (398h2m16.907414746s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136994234 (398h2m16.907635559s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137216439 (398h2m16.907857774s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137458321 (398h2m16.908099656s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137623086 (398h2m16.908264421s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138134475 (398h2m16.90877581s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138499122 (398h2m16.909140458s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138736497 (398h2m16.909377822s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138951127 (398h2m16.909592452s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139222767 (398h2m16.909864102s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139541577 (398h2m16.910182912s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:43.510699884 (398h2m18.28134122s) [upload=318452]: Next returned nil segment Jul 9 23:12:43.510713381 (398h2m18.281354716s) [upload=318452]: Waiting for error group managing segments... Jul 9 23:12:44.093679677 (398h2m18.864321012s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139156961 (398h2m18.909798286s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139550674 (398h2m18.910192019s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139723775 (398h2m18.91036512s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.140606163 (398h2m18.911247489s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.141871074 (398h2m18.912512409s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143296170 (398h2m18.913937495s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143572189 (398h2m18.914213524s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:45.311783220 (398h2m20.082424555s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:47.074380105 (398h2m21.84502144s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:52.162572575 (398h2m26.93321391s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:13.763376135 (398h2m48.53401746s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:29.504846258 (398h3m4.275487583s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:41.049567150 (398h3m15.820208505s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:12cgYc8zVZh7w7iTTDC5zq1xK12wa5357zdAH3F3bZ21n1Y8uLg, address:104.36.82.17:4350): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049619941 (398h3m15.820261276s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1mLbGHntspYmsTVg8433HYSdAEMrXjGnDhmmvX1JmmgcxtdLgk, address:104.36.82.23:4464): protocol: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection Jul 9 23:13:41.049694543 (398h3m15.820335878s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:17SUmQEJJ3Qwe9yRq5DFTGaZBqGnz5Pu5YDri93Wk5NP3rcm5c, address:45.91.95.4:4023): protocol: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection Jul 9 23:13:41.049733979 (398h3m15.820375314s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1R6A6f6LTL3VDDg5gbhvQE55bRLJPDfAjViixeNX4KjVJpYFDE, address:45.91.95.3:4017): protocol: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection Jul 9 23:13:41.049758877 (398h3m15.820400212s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1m82Tdkxzub83FtYpd2Abhgf6StE1eFtxn1ESBi8BdHGGVhCu8, address:45.91.95.5:4025): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049823370 (398h3m15.820464705s) [seg_pos={0 0}, upload=318452]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 23:13:41.285661371 (398h3m16.056302706s) [upload=318452]: Done uploading segments... Jul 9 23:13:41.285688192 (398h3m16.056329517s) [upload=318452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 11.684455793s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.....X.......X............X.................................] duration: 15.6667307s FAIL (2025-07-09 23:41:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== uestItem_SegmentBegin Jul 9 23:12:42.134881962 (398h2m16.905523297s) [seg_pos={0 0}, upload=318452]: Begin upload segment... Jul 9 23:12:42.134950294 (398h2m16.905591629s) [seg_pos={0 0}, upload=318452]: Done begin upload segment. Jul 9 23:12:42.134981082 (398h2m16.905622417s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135020638 (398h2m16.905661983s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135575439 (398h2m16.906216774s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135895081 (398h2m16.906536416s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135916562 (398h2m16.906557897s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136222628 (398h2m16.906863963s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136501170 (398h2m16.907142506s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136773411 (398h2m16.907414746s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136994234 (398h2m16.907635559s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137216439 (398h2m16.907857774s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137458321 (398h2m16.908099656s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137623086 (398h2m16.908264421s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138134475 (398h2m16.90877581s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138499122 (398h2m16.909140458s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138736497 (398h2m16.909377822s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138951127 (398h2m16.909592452s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139222767 (398h2m16.909864102s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139541577 (398h2m16.910182912s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:43.510699884 (398h2m18.28134122s) [upload=318452]: Next returned nil segment Jul 9 23:12:43.510713381 (398h2m18.281354716s) [upload=318452]: Waiting for error group managing segments... Jul 9 23:12:44.093679677 (398h2m18.864321012s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139156961 (398h2m18.909798286s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139550674 (398h2m18.910192019s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139723775 (398h2m18.91036512s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.140606163 (398h2m18.911247489s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.141871074 (398h2m18.912512409s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143296170 (398h2m18.913937495s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143572189 (398h2m18.914213524s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:45.311783220 (398h2m20.082424555s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:47.074380105 (398h2m21.84502144s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:52.162572575 (398h2m26.93321391s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:13.763376135 (398h2m48.53401746s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:29.504846258 (398h3m4.275487583s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:41.049567150 (398h3m15.820208505s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:12cgYc8zVZh7w7iTTDC5zq1xK12wa5357zdAH3F3bZ21n1Y8uLg, address:104.36.82.17:4350): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049619941 (398h3m15.820261276s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1mLbGHntspYmsTVg8433HYSdAEMrXjGnDhmmvX1JmmgcxtdLgk, address:104.36.82.23:4464): protocol: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection Jul 9 23:13:41.049694543 (398h3m15.820335878s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:17SUmQEJJ3Qwe9yRq5DFTGaZBqGnz5Pu5YDri93Wk5NP3rcm5c, address:45.91.95.4:4023): protocol: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection Jul 9 23:13:41.049733979 (398h3m15.820375314s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1R6A6f6LTL3VDDg5gbhvQE55bRLJPDfAjViixeNX4KjVJpYFDE, address:45.91.95.3:4017): protocol: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection Jul 9 23:13:41.049758877 (398h3m15.820400212s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1m82Tdkxzub83FtYpd2Abhgf6StE1eFtxn1ESBi8BdHGGVhCu8, address:45.91.95.5:4025): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049823370 (398h3m15.820464705s) [seg_pos={0 0}, upload=318452]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 23:13:41.285661371 (398h3m16.056302706s) [upload=318452]: Done uploading segments... Jul 9 23:13:41.285688192 (398h3m16.056329517s) [upload=318452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 15.66676235s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.......X..............................] duration: 17.683378118s FAIL (2025-07-09 23:41:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.683403326s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [......X.......X............X................................] duration: 28.485713624s FAIL (2025-07-09 23:40:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== uestItem_SegmentBegin Jul 9 23:12:42.134881962 (398h2m16.905523297s) [seg_pos={0 0}, upload=318452]: Begin upload segment... Jul 9 23:12:42.134950294 (398h2m16.905591629s) [seg_pos={0 0}, upload=318452]: Done begin upload segment. Jul 9 23:12:42.134981082 (398h2m16.905622417s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135020638 (398h2m16.905661983s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135575439 (398h2m16.906216774s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135895081 (398h2m16.906536416s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135916562 (398h2m16.906557897s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136222628 (398h2m16.906863963s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136501170 (398h2m16.907142506s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136773411 (398h2m16.907414746s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136994234 (398h2m16.907635559s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137216439 (398h2m16.907857774s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137458321 (398h2m16.908099656s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137623086 (398h2m16.908264421s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138134475 (398h2m16.90877581s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138499122 (398h2m16.909140458s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138736497 (398h2m16.909377822s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138951127 (398h2m16.909592452s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139222767 (398h2m16.909864102s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139541577 (398h2m16.910182912s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:43.510699884 (398h2m18.28134122s) [upload=318452]: Next returned nil segment Jul 9 23:12:43.510713381 (398h2m18.281354716s) [upload=318452]: Waiting for error group managing segments... Jul 9 23:12:44.093679677 (398h2m18.864321012s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139156961 (398h2m18.909798286s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139550674 (398h2m18.910192019s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139723775 (398h2m18.91036512s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.140606163 (398h2m18.911247489s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.141871074 (398h2m18.912512409s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143296170 (398h2m18.913937495s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143572189 (398h2m18.914213524s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:45.311783220 (398h2m20.082424555s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:47.074380105 (398h2m21.84502144s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:52.162572575 (398h2m26.93321391s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:13.763376135 (398h2m48.53401746s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:29.504846258 (398h3m4.275487583s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:41.049567150 (398h3m15.820208505s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:12cgYc8zVZh7w7iTTDC5zq1xK12wa5357zdAH3F3bZ21n1Y8uLg, address:104.36.82.17:4350): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049619941 (398h3m15.820261276s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1mLbGHntspYmsTVg8433HYSdAEMrXjGnDhmmvX1JmmgcxtdLgk, address:104.36.82.23:4464): protocol: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection Jul 9 23:13:41.049694543 (398h3m15.820335878s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:17SUmQEJJ3Qwe9yRq5DFTGaZBqGnz5Pu5YDri93Wk5NP3rcm5c, address:45.91.95.4:4023): protocol: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection Jul 9 23:13:41.049733979 (398h3m15.820375314s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1R6A6f6LTL3VDDg5gbhvQE55bRLJPDfAjViixeNX4KjVJpYFDE, address:45.91.95.3:4017): protocol: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection Jul 9 23:13:41.049758877 (398h3m15.820400212s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1m82Tdkxzub83FtYpd2Abhgf6StE1eFtxn1ESBi8BdHGGVhCu8, address:45.91.95.5:4025): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049823370 (398h3m15.820464705s) [seg_pos={0 0}, upload=318452]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 23:13:41.285661371 (398h3m16.056302706s) [upload=318452]: Done uploading segments... Jul 9 23:13:41.285688192 (398h3m16.056329517s) [upload=318452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 28.485739472s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................X.......X.............................] duration: 19.751011669s FAIL (2025-07-09 23:40:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.751036976s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................X.......X............................] duration: 5.597507912s FAIL (2025-07-09 23:40:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.597612272s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.......X.......X............X...............................] duration: 26.226279587s FAIL (2025-07-09 23:39:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== uestItem_SegmentBegin Jul 9 23:12:42.134881962 (398h2m16.905523297s) [seg_pos={0 0}, upload=318452]: Begin upload segment... Jul 9 23:12:42.134950294 (398h2m16.905591629s) [seg_pos={0 0}, upload=318452]: Done begin upload segment. Jul 9 23:12:42.134981082 (398h2m16.905622417s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135020638 (398h2m16.905661983s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135575439 (398h2m16.906216774s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135895081 (398h2m16.906536416s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135916562 (398h2m16.906557897s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136222628 (398h2m16.906863963s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136501170 (398h2m16.907142506s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136773411 (398h2m16.907414746s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136994234 (398h2m16.907635559s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137216439 (398h2m16.907857774s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137458321 (398h2m16.908099656s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137623086 (398h2m16.908264421s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138134475 (398h2m16.90877581s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138499122 (398h2m16.909140458s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138736497 (398h2m16.909377822s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138951127 (398h2m16.909592452s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139222767 (398h2m16.909864102s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139541577 (398h2m16.910182912s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:43.510699884 (398h2m18.28134122s) [upload=318452]: Next returned nil segment Jul 9 23:12:43.510713381 (398h2m18.281354716s) [upload=318452]: Waiting for error group managing segments... Jul 9 23:12:44.093679677 (398h2m18.864321012s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139156961 (398h2m18.909798286s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139550674 (398h2m18.910192019s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139723775 (398h2m18.91036512s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.140606163 (398h2m18.911247489s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.141871074 (398h2m18.912512409s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143296170 (398h2m18.913937495s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143572189 (398h2m18.914213524s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:45.311783220 (398h2m20.082424555s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:47.074380105 (398h2m21.84502144s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:52.162572575 (398h2m26.93321391s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:13.763376135 (398h2m48.53401746s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:29.504846258 (398h3m4.275487583s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:41.049567150 (398h3m15.820208505s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:12cgYc8zVZh7w7iTTDC5zq1xK12wa5357zdAH3F3bZ21n1Y8uLg, address:104.36.82.17:4350): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049619941 (398h3m15.820261276s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1mLbGHntspYmsTVg8433HYSdAEMrXjGnDhmmvX1JmmgcxtdLgk, address:104.36.82.23:4464): protocol: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection Jul 9 23:13:41.049694543 (398h3m15.820335878s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:17SUmQEJJ3Qwe9yRq5DFTGaZBqGnz5Pu5YDri93Wk5NP3rcm5c, address:45.91.95.4:4023): protocol: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection Jul 9 23:13:41.049733979 (398h3m15.820375314s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1R6A6f6LTL3VDDg5gbhvQE55bRLJPDfAjViixeNX4KjVJpYFDE, address:45.91.95.3:4017): protocol: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection Jul 9 23:13:41.049758877 (398h3m15.820400212s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1m82Tdkxzub83FtYpd2Abhgf6StE1eFtxn1ESBi8BdHGGVhCu8, address:45.91.95.5:4025): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049823370 (398h3m15.820464705s) [seg_pos={0 0}, upload=318452]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 23:13:41.285661371 (398h3m16.056302706s) [upload=318452]: Done uploading segments... Jul 9 23:13:41.285688192 (398h3m16.056329517s) [upload=318452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 26.226315174s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................X.......X...........................] duration: 14.98935587s FAIL (2025-07-09 23:39:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.989390938s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [X.......X.......X............X..............................] duration: 9.850937096s FAIL (2025-07-09 23:38:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== uestItem_SegmentBegin Jul 9 23:12:42.134881962 (398h2m16.905523297s) [seg_pos={0 0}, upload=318452]: Begin upload segment... Jul 9 23:12:42.134950294 (398h2m16.905591629s) [seg_pos={0 0}, upload=318452]: Done begin upload segment. Jul 9 23:12:42.134981082 (398h2m16.905622417s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135020638 (398h2m16.905661983s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135575439 (398h2m16.906216774s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135895081 (398h2m16.906536416s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135916562 (398h2m16.906557897s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136222628 (398h2m16.906863963s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136501170 (398h2m16.907142506s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136773411 (398h2m16.907414746s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136994234 (398h2m16.907635559s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137216439 (398h2m16.907857774s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137458321 (398h2m16.908099656s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137623086 (398h2m16.908264421s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138134475 (398h2m16.90877581s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138499122 (398h2m16.909140458s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138736497 (398h2m16.909377822s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138951127 (398h2m16.909592452s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139222767 (398h2m16.909864102s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139541577 (398h2m16.910182912s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:43.510699884 (398h2m18.28134122s) [upload=318452]: Next returned nil segment Jul 9 23:12:43.510713381 (398h2m18.281354716s) [upload=318452]: Waiting for error group managing segments... Jul 9 23:12:44.093679677 (398h2m18.864321012s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139156961 (398h2m18.909798286s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139550674 (398h2m18.910192019s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139723775 (398h2m18.91036512s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.140606163 (398h2m18.911247489s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.141871074 (398h2m18.912512409s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143296170 (398h2m18.913937495s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143572189 (398h2m18.914213524s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:45.311783220 (398h2m20.082424555s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:47.074380105 (398h2m21.84502144s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:52.162572575 (398h2m26.93321391s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:13.763376135 (398h2m48.53401746s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:29.504846258 (398h3m4.275487583s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:41.049567150 (398h3m15.820208505s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:12cgYc8zVZh7w7iTTDC5zq1xK12wa5357zdAH3F3bZ21n1Y8uLg, address:104.36.82.17:4350): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049619941 (398h3m15.820261276s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1mLbGHntspYmsTVg8433HYSdAEMrXjGnDhmmvX1JmmgcxtdLgk, address:104.36.82.23:4464): protocol: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection Jul 9 23:13:41.049694543 (398h3m15.820335878s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:17SUmQEJJ3Qwe9yRq5DFTGaZBqGnz5Pu5YDri93Wk5NP3rcm5c, address:45.91.95.4:4023): protocol: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection Jul 9 23:13:41.049733979 (398h3m15.820375314s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1R6A6f6LTL3VDDg5gbhvQE55bRLJPDfAjViixeNX4KjVJpYFDE, address:45.91.95.3:4017): protocol: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection Jul 9 23:13:41.049758877 (398h3m15.820400212s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1m82Tdkxzub83FtYpd2Abhgf6StE1eFtxn1ESBi8BdHGGVhCu8, address:45.91.95.5:4025): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049823370 (398h3m15.820464705s) [seg_pos={0 0}, upload=318452]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 23:13:41.285661371 (398h3m16.056302706s) [upload=318452]: Done uploading segments... Jul 9 23:13:41.285688192 (398h3m16.056329517s) [upload=318452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 9.850954038s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........................X.......X..........................] duration: 5.740176017s FAIL (2025-07-09 23:38:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.740190356s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.X.......X.......X............X.............................] duration: 35.938225339s FAIL (2025-07-09 23:38:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== uestItem_SegmentBegin Jul 9 23:12:42.134881962 (398h2m16.905523297s) [seg_pos={0 0}, upload=318452]: Begin upload segment... Jul 9 23:12:42.134950294 (398h2m16.905591629s) [seg_pos={0 0}, upload=318452]: Done begin upload segment. Jul 9 23:12:42.134981082 (398h2m16.905622417s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135020638 (398h2m16.905661983s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135575439 (398h2m16.906216774s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135895081 (398h2m16.906536416s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135916562 (398h2m16.906557897s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136222628 (398h2m16.906863963s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136501170 (398h2m16.907142506s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136773411 (398h2m16.907414746s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136994234 (398h2m16.907635559s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137216439 (398h2m16.907857774s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137458321 (398h2m16.908099656s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137623086 (398h2m16.908264421s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138134475 (398h2m16.90877581s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138499122 (398h2m16.909140458s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138736497 (398h2m16.909377822s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138951127 (398h2m16.909592452s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139222767 (398h2m16.909864102s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139541577 (398h2m16.910182912s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:43.510699884 (398h2m18.28134122s) [upload=318452]: Next returned nil segment Jul 9 23:12:43.510713381 (398h2m18.281354716s) [upload=318452]: Waiting for error group managing segments... Jul 9 23:12:44.093679677 (398h2m18.864321012s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139156961 (398h2m18.909798286s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139550674 (398h2m18.910192019s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139723775 (398h2m18.91036512s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.140606163 (398h2m18.911247489s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.141871074 (398h2m18.912512409s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143296170 (398h2m18.913937495s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143572189 (398h2m18.914213524s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:45.311783220 (398h2m20.082424555s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:47.074380105 (398h2m21.84502144s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:52.162572575 (398h2m26.93321391s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:13.763376135 (398h2m48.53401746s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:29.504846258 (398h3m4.275487583s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:41.049567150 (398h3m15.820208505s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:12cgYc8zVZh7w7iTTDC5zq1xK12wa5357zdAH3F3bZ21n1Y8uLg, address:104.36.82.17:4350): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049619941 (398h3m15.820261276s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1mLbGHntspYmsTVg8433HYSdAEMrXjGnDhmmvX1JmmgcxtdLgk, address:104.36.82.23:4464): protocol: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection Jul 9 23:13:41.049694543 (398h3m15.820335878s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:17SUmQEJJ3Qwe9yRq5DFTGaZBqGnz5Pu5YDri93Wk5NP3rcm5c, address:45.91.95.4:4023): protocol: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection Jul 9 23:13:41.049733979 (398h3m15.820375314s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1R6A6f6LTL3VDDg5gbhvQE55bRLJPDfAjViixeNX4KjVJpYFDE, address:45.91.95.3:4017): protocol: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection Jul 9 23:13:41.049758877 (398h3m15.820400212s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1m82Tdkxzub83FtYpd2Abhgf6StE1eFtxn1ESBi8BdHGGVhCu8, address:45.91.95.5:4025): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049823370 (398h3m15.820464705s) [seg_pos={0 0}, upload=318452]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 23:13:41.285661371 (398h3m16.056302706s) [upload=318452]: Done uploading segments... Jul 9 23:13:41.285688192 (398h3m16.056329517s) [upload=318452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 35.938253563s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................X.......X.........................] duration: 28.63736327s FAIL (2025-07-09 23:37:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 28.63738338s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..X.......X.......X............X............................] duration: 20.162500503s FAIL (2025-07-09 23:36:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== uestItem_SegmentBegin Jul 9 23:12:42.134881962 (398h2m16.905523297s) [seg_pos={0 0}, upload=318452]: Begin upload segment... Jul 9 23:12:42.134950294 (398h2m16.905591629s) [seg_pos={0 0}, upload=318452]: Done begin upload segment. Jul 9 23:12:42.134981082 (398h2m16.905622417s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135020638 (398h2m16.905661983s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135575439 (398h2m16.906216774s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135895081 (398h2m16.906536416s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135916562 (398h2m16.906557897s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136222628 (398h2m16.906863963s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136501170 (398h2m16.907142506s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136773411 (398h2m16.907414746s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136994234 (398h2m16.907635559s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137216439 (398h2m16.907857774s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137458321 (398h2m16.908099656s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137623086 (398h2m16.908264421s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138134475 (398h2m16.90877581s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138499122 (398h2m16.909140458s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138736497 (398h2m16.909377822s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138951127 (398h2m16.909592452s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139222767 (398h2m16.909864102s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139541577 (398h2m16.910182912s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:43.510699884 (398h2m18.28134122s) [upload=318452]: Next returned nil segment Jul 9 23:12:43.510713381 (398h2m18.281354716s) [upload=318452]: Waiting for error group managing segments... Jul 9 23:12:44.093679677 (398h2m18.864321012s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139156961 (398h2m18.909798286s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139550674 (398h2m18.910192019s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139723775 (398h2m18.91036512s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.140606163 (398h2m18.911247489s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.141871074 (398h2m18.912512409s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143296170 (398h2m18.913937495s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143572189 (398h2m18.914213524s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:45.311783220 (398h2m20.082424555s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:47.074380105 (398h2m21.84502144s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:52.162572575 (398h2m26.93321391s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:13.763376135 (398h2m48.53401746s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:29.504846258 (398h3m4.275487583s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:41.049567150 (398h3m15.820208505s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:12cgYc8zVZh7w7iTTDC5zq1xK12wa5357zdAH3F3bZ21n1Y8uLg, address:104.36.82.17:4350): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049619941 (398h3m15.820261276s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1mLbGHntspYmsTVg8433HYSdAEMrXjGnDhmmvX1JmmgcxtdLgk, address:104.36.82.23:4464): protocol: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection Jul 9 23:13:41.049694543 (398h3m15.820335878s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:17SUmQEJJ3Qwe9yRq5DFTGaZBqGnz5Pu5YDri93Wk5NP3rcm5c, address:45.91.95.4:4023): protocol: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection Jul 9 23:13:41.049733979 (398h3m15.820375314s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1R6A6f6LTL3VDDg5gbhvQE55bRLJPDfAjViixeNX4KjVJpYFDE, address:45.91.95.3:4017): protocol: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection Jul 9 23:13:41.049758877 (398h3m15.820400212s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1m82Tdkxzub83FtYpd2Abhgf6StE1eFtxn1ESBi8BdHGGVhCu8, address:45.91.95.5:4025): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049823370 (398h3m15.820464705s) [seg_pos={0 0}, upload=318452]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 23:13:41.285661371 (398h3m16.056302706s) [upload=318452]: Done uploading segments... Jul 9 23:13:41.285688192 (398h3m16.056329517s) [upload=318452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 20.162517725s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.566960068s FAIL (2025-07-09 23:36:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.566988594s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........................X.......X........................] duration: 13.499062278s FAIL (2025-07-09 23:36:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.499085202s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [X..X.......X.......X............X...........................] duration: 50.118594684s FAIL (2025-07-09 23:36:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== uestItem_SegmentBegin Jul 9 23:12:42.134881962 (398h2m16.905523297s) [seg_pos={0 0}, upload=318452]: Begin upload segment... Jul 9 23:12:42.134950294 (398h2m16.905591629s) [seg_pos={0 0}, upload=318452]: Done begin upload segment. Jul 9 23:12:42.134981082 (398h2m16.905622417s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135020638 (398h2m16.905661983s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135575439 (398h2m16.906216774s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135895081 (398h2m16.906536416s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135916562 (398h2m16.906557897s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136222628 (398h2m16.906863963s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136501170 (398h2m16.907142506s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136773411 (398h2m16.907414746s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136994234 (398h2m16.907635559s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137216439 (398h2m16.907857774s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137458321 (398h2m16.908099656s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137623086 (398h2m16.908264421s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138134475 (398h2m16.90877581s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138499122 (398h2m16.909140458s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138736497 (398h2m16.909377822s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138951127 (398h2m16.909592452s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139222767 (398h2m16.909864102s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139541577 (398h2m16.910182912s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:43.510699884 (398h2m18.28134122s) [upload=318452]: Next returned nil segment Jul 9 23:12:43.510713381 (398h2m18.281354716s) [upload=318452]: Waiting for error group managing segments... Jul 9 23:12:44.093679677 (398h2m18.864321012s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139156961 (398h2m18.909798286s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139550674 (398h2m18.910192019s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139723775 (398h2m18.91036512s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.140606163 (398h2m18.911247489s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.141871074 (398h2m18.912512409s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143296170 (398h2m18.913937495s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143572189 (398h2m18.914213524s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:45.311783220 (398h2m20.082424555s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:47.074380105 (398h2m21.84502144s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:52.162572575 (398h2m26.93321391s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:13.763376135 (398h2m48.53401746s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:29.504846258 (398h3m4.275487583s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:41.049567150 (398h3m15.820208505s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:12cgYc8zVZh7w7iTTDC5zq1xK12wa5357zdAH3F3bZ21n1Y8uLg, address:104.36.82.17:4350): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049619941 (398h3m15.820261276s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1mLbGHntspYmsTVg8433HYSdAEMrXjGnDhmmvX1JmmgcxtdLgk, address:104.36.82.23:4464): protocol: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection Jul 9 23:13:41.049694543 (398h3m15.820335878s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:17SUmQEJJ3Qwe9yRq5DFTGaZBqGnz5Pu5YDri93Wk5NP3rcm5c, address:45.91.95.4:4023): protocol: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection Jul 9 23:13:41.049733979 (398h3m15.820375314s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1R6A6f6LTL3VDDg5gbhvQE55bRLJPDfAjViixeNX4KjVJpYFDE, address:45.91.95.3:4017): protocol: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection Jul 9 23:13:41.049758877 (398h3m15.820400212s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1m82Tdkxzub83FtYpd2Abhgf6StE1eFtxn1ESBi8BdHGGVhCu8, address:45.91.95.5:4025): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049823370 (398h3m15.820464705s) [seg_pos={0 0}, upload=318452]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 23:13:41.285661371 (398h3m16.056302706s) [upload=318452]: Done uploading segments... Jul 9 23:13:41.285688192 (398h3m16.056329517s) [upload=318452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 50.118615763s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............................X.......X.......................] duration: 22.029823864s FAIL (2025-07-09 23:36:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.029837971s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.173901968s FAIL (2025-07-09 23:35:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.173923178s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.X..X.......X.......X............X..........................] duration: 5.586539418s FAIL (2025-07-09 23:34:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== uestItem_SegmentBegin Jul 9 23:12:42.134881962 (398h2m16.905523297s) [seg_pos={0 0}, upload=318452]: Begin upload segment... Jul 9 23:12:42.134950294 (398h2m16.905591629s) [seg_pos={0 0}, upload=318452]: Done begin upload segment. Jul 9 23:12:42.134981082 (398h2m16.905622417s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135020638 (398h2m16.905661983s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135575439 (398h2m16.906216774s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135895081 (398h2m16.906536416s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135916562 (398h2m16.906557897s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136222628 (398h2m16.906863963s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136501170 (398h2m16.907142506s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136773411 (398h2m16.907414746s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136994234 (398h2m16.907635559s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137216439 (398h2m16.907857774s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137458321 (398h2m16.908099656s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137623086 (398h2m16.908264421s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138134475 (398h2m16.90877581s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138499122 (398h2m16.909140458s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138736497 (398h2m16.909377822s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138951127 (398h2m16.909592452s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139222767 (398h2m16.909864102s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139541577 (398h2m16.910182912s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:43.510699884 (398h2m18.28134122s) [upload=318452]: Next returned nil segment Jul 9 23:12:43.510713381 (398h2m18.281354716s) [upload=318452]: Waiting for error group managing segments... Jul 9 23:12:44.093679677 (398h2m18.864321012s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139156961 (398h2m18.909798286s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139550674 (398h2m18.910192019s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139723775 (398h2m18.91036512s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.140606163 (398h2m18.911247489s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.141871074 (398h2m18.912512409s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143296170 (398h2m18.913937495s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143572189 (398h2m18.914213524s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:45.311783220 (398h2m20.082424555s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:47.074380105 (398h2m21.84502144s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:52.162572575 (398h2m26.93321391s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:13.763376135 (398h2m48.53401746s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:29.504846258 (398h3m4.275487583s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:41.049567150 (398h3m15.820208505s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:12cgYc8zVZh7w7iTTDC5zq1xK12wa5357zdAH3F3bZ21n1Y8uLg, address:104.36.82.17:4350): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049619941 (398h3m15.820261276s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1mLbGHntspYmsTVg8433HYSdAEMrXjGnDhmmvX1JmmgcxtdLgk, address:104.36.82.23:4464): protocol: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection Jul 9 23:13:41.049694543 (398h3m15.820335878s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:17SUmQEJJ3Qwe9yRq5DFTGaZBqGnz5Pu5YDri93Wk5NP3rcm5c, address:45.91.95.4:4023): protocol: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection Jul 9 23:13:41.049733979 (398h3m15.820375314s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1R6A6f6LTL3VDDg5gbhvQE55bRLJPDfAjViixeNX4KjVJpYFDE, address:45.91.95.3:4017): protocol: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection Jul 9 23:13:41.049758877 (398h3m15.820400212s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1m82Tdkxzub83FtYpd2Abhgf6StE1eFtxn1ESBi8BdHGGVhCu8, address:45.91.95.5:4025): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049823370 (398h3m15.820464705s) [seg_pos={0 0}, upload=318452]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 23:13:41.285661371 (398h3m16.056302706s) [upload=318452]: Done uploading segments... Jul 9 23:13:41.285688192 (398h3m16.056329517s) [upload=318452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.586565477s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............................X.......X......................] duration: 6.550644502s FAIL (2025-07-09 23:34:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.550690571s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.185691173s FAIL (2025-07-09 23:34:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.185719046s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............................X.......X.....................] duration: 5.644049582s FAIL (2025-07-09 23:34:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.644071364s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..X..X.......X.......X............X.........................] duration: 7.058446885s FAIL (2025-07-09 23:34:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== uestItem_SegmentBegin Jul 9 23:12:42.134881962 (398h2m16.905523297s) [seg_pos={0 0}, upload=318452]: Begin upload segment... Jul 9 23:12:42.134950294 (398h2m16.905591629s) [seg_pos={0 0}, upload=318452]: Done begin upload segment. Jul 9 23:12:42.134981082 (398h2m16.905622417s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135020638 (398h2m16.905661983s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135575439 (398h2m16.906216774s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135895081 (398h2m16.906536416s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135916562 (398h2m16.906557897s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136222628 (398h2m16.906863963s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136501170 (398h2m16.907142506s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136773411 (398h2m16.907414746s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136994234 (398h2m16.907635559s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137216439 (398h2m16.907857774s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137458321 (398h2m16.908099656s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137623086 (398h2m16.908264421s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138134475 (398h2m16.90877581s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138499122 (398h2m16.909140458s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138736497 (398h2m16.909377822s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138951127 (398h2m16.909592452s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139222767 (398h2m16.909864102s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139541577 (398h2m16.910182912s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:43.510699884 (398h2m18.28134122s) [upload=318452]: Next returned nil segment Jul 9 23:12:43.510713381 (398h2m18.281354716s) [upload=318452]: Waiting for error group managing segments... Jul 9 23:12:44.093679677 (398h2m18.864321012s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139156961 (398h2m18.909798286s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139550674 (398h2m18.910192019s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139723775 (398h2m18.91036512s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.140606163 (398h2m18.911247489s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.141871074 (398h2m18.912512409s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143296170 (398h2m18.913937495s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143572189 (398h2m18.914213524s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:45.311783220 (398h2m20.082424555s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:47.074380105 (398h2m21.84502144s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:52.162572575 (398h2m26.93321391s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:13.763376135 (398h2m48.53401746s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:29.504846258 (398h3m4.275487583s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:41.049567150 (398h3m15.820208505s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:12cgYc8zVZh7w7iTTDC5zq1xK12wa5357zdAH3F3bZ21n1Y8uLg, address:104.36.82.17:4350): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049619941 (398h3m15.820261276s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1mLbGHntspYmsTVg8433HYSdAEMrXjGnDhmmvX1JmmgcxtdLgk, address:104.36.82.23:4464): protocol: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection Jul 9 23:13:41.049694543 (398h3m15.820335878s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:17SUmQEJJ3Qwe9yRq5DFTGaZBqGnz5Pu5YDri93Wk5NP3rcm5c, address:45.91.95.4:4023): protocol: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection Jul 9 23:13:41.049733979 (398h3m15.820375314s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1R6A6f6LTL3VDDg5gbhvQE55bRLJPDfAjViixeNX4KjVJpYFDE, address:45.91.95.3:4017): protocol: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection Jul 9 23:13:41.049758877 (398h3m15.820400212s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1m82Tdkxzub83FtYpd2Abhgf6StE1eFtxn1ESBi8BdHGGVhCu8, address:45.91.95.5:4025): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049823370 (398h3m15.820464705s) [seg_pos={0 0}, upload=318452]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 23:13:41.285661371 (398h3m16.056302706s) [upload=318452]: Done uploading segments... Jul 9 23:13:41.285688192 (398h3m16.056329517s) [upload=318452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 7.058478786s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.93610707s FAIL (2025-07-09 23:33:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.936188437s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...X..X.......X.......X............X........................] duration: 5.86602267s FAIL (2025-07-09 23:33:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== uestItem_SegmentBegin Jul 9 23:12:42.134881962 (398h2m16.905523297s) [seg_pos={0 0}, upload=318452]: Begin upload segment... Jul 9 23:12:42.134950294 (398h2m16.905591629s) [seg_pos={0 0}, upload=318452]: Done begin upload segment. Jul 9 23:12:42.134981082 (398h2m16.905622417s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135020638 (398h2m16.905661983s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135575439 (398h2m16.906216774s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135895081 (398h2m16.906536416s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135916562 (398h2m16.906557897s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136222628 (398h2m16.906863963s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136501170 (398h2m16.907142506s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136773411 (398h2m16.907414746s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136994234 (398h2m16.907635559s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137216439 (398h2m16.907857774s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137458321 (398h2m16.908099656s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137623086 (398h2m16.908264421s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138134475 (398h2m16.90877581s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138499122 (398h2m16.909140458s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138736497 (398h2m16.909377822s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138951127 (398h2m16.909592452s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139222767 (398h2m16.909864102s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139541577 (398h2m16.910182912s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:43.510699884 (398h2m18.28134122s) [upload=318452]: Next returned nil segment Jul 9 23:12:43.510713381 (398h2m18.281354716s) [upload=318452]: Waiting for error group managing segments... Jul 9 23:12:44.093679677 (398h2m18.864321012s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139156961 (398h2m18.909798286s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139550674 (398h2m18.910192019s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139723775 (398h2m18.91036512s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.140606163 (398h2m18.911247489s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.141871074 (398h2m18.912512409s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143296170 (398h2m18.913937495s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143572189 (398h2m18.914213524s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:45.311783220 (398h2m20.082424555s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:47.074380105 (398h2m21.84502144s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:52.162572575 (398h2m26.93321391s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:13.763376135 (398h2m48.53401746s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:29.504846258 (398h3m4.275487583s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:41.049567150 (398h3m15.820208505s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:12cgYc8zVZh7w7iTTDC5zq1xK12wa5357zdAH3F3bZ21n1Y8uLg, address:104.36.82.17:4350): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049619941 (398h3m15.820261276s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1mLbGHntspYmsTVg8433HYSdAEMrXjGnDhmmvX1JmmgcxtdLgk, address:104.36.82.23:4464): protocol: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection Jul 9 23:13:41.049694543 (398h3m15.820335878s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:17SUmQEJJ3Qwe9yRq5DFTGaZBqGnz5Pu5YDri93Wk5NP3rcm5c, address:45.91.95.4:4023): protocol: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection Jul 9 23:13:41.049733979 (398h3m15.820375314s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1R6A6f6LTL3VDDg5gbhvQE55bRLJPDfAjViixeNX4KjVJpYFDE, address:45.91.95.3:4017): protocol: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection Jul 9 23:13:41.049758877 (398h3m15.820400212s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1m82Tdkxzub83FtYpd2Abhgf6StE1eFtxn1ESBi8BdHGGVhCu8, address:45.91.95.5:4025): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049823370 (398h3m15.820464705s) [seg_pos={0 0}, upload=318452]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 23:13:41.285661371 (398h3m16.056302706s) [upload=318452]: Done uploading segments... Jul 9 23:13:41.285688192 (398h3m16.056329517s) [upload=318452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.866048989s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............................X.......X....................] duration: 5.661510759s FAIL (2025-07-09 23:33:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.661531829s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X...X.......................................................] duration: 6.155267197s FAIL (2025-07-09 23:32:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.15528883s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [....X..X.......X.......X............X.......................] duration: 8.363448997s FAIL (2025-07-09 23:32:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== uestItem_SegmentBegin Jul 9 23:12:42.134881962 (398h2m16.905523297s) [seg_pos={0 0}, upload=318452]: Begin upload segment... Jul 9 23:12:42.134950294 (398h2m16.905591629s) [seg_pos={0 0}, upload=318452]: Done begin upload segment. Jul 9 23:12:42.134981082 (398h2m16.905622417s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135020638 (398h2m16.905661983s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135575439 (398h2m16.906216774s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135895081 (398h2m16.906536416s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135916562 (398h2m16.906557897s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136222628 (398h2m16.906863963s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136501170 (398h2m16.907142506s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136773411 (398h2m16.907414746s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136994234 (398h2m16.907635559s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137216439 (398h2m16.907857774s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137458321 (398h2m16.908099656s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137623086 (398h2m16.908264421s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138134475 (398h2m16.90877581s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138499122 (398h2m16.909140458s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138736497 (398h2m16.909377822s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138951127 (398h2m16.909592452s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139222767 (398h2m16.909864102s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139541577 (398h2m16.910182912s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:43.510699884 (398h2m18.28134122s) [upload=318452]: Next returned nil segment Jul 9 23:12:43.510713381 (398h2m18.281354716s) [upload=318452]: Waiting for error group managing segments... Jul 9 23:12:44.093679677 (398h2m18.864321012s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139156961 (398h2m18.909798286s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139550674 (398h2m18.910192019s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139723775 (398h2m18.91036512s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.140606163 (398h2m18.911247489s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.141871074 (398h2m18.912512409s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143296170 (398h2m18.913937495s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143572189 (398h2m18.914213524s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:45.311783220 (398h2m20.082424555s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:47.074380105 (398h2m21.84502144s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:52.162572575 (398h2m26.93321391s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:13.763376135 (398h2m48.53401746s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:29.504846258 (398h3m4.275487583s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:41.049567150 (398h3m15.820208505s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:12cgYc8zVZh7w7iTTDC5zq1xK12wa5357zdAH3F3bZ21n1Y8uLg, address:104.36.82.17:4350): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049619941 (398h3m15.820261276s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1mLbGHntspYmsTVg8433HYSdAEMrXjGnDhmmvX1JmmgcxtdLgk, address:104.36.82.23:4464): protocol: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection Jul 9 23:13:41.049694543 (398h3m15.820335878s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:17SUmQEJJ3Qwe9yRq5DFTGaZBqGnz5Pu5YDri93Wk5NP3rcm5c, address:45.91.95.4:4023): protocol: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection Jul 9 23:13:41.049733979 (398h3m15.820375314s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1R6A6f6LTL3VDDg5gbhvQE55bRLJPDfAjViixeNX4KjVJpYFDE, address:45.91.95.3:4017): protocol: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection Jul 9 23:13:41.049758877 (398h3m15.820400212s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1m82Tdkxzub83FtYpd2Abhgf6StE1eFtxn1ESBi8BdHGGVhCu8, address:45.91.95.5:4025): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049823370 (398h3m15.820464705s) [seg_pos={0 0}, upload=318452]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 23:13:41.285661371 (398h3m16.056302706s) [upload=318452]: Done uploading segments... Jul 9 23:13:41.285688192 (398h3m16.056329517s) [upload=318452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 8.363478354s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................................X.......X...................] duration: 7.390298806s FAIL (2025-07-09 23:32:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.390321018s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................................X.......X..................] duration: 5.545386914s FAIL (2025-07-09 23:31:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.545408998s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X...X......................................................] duration: 6.169455858s FAIL (2025-07-09 23:31:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.169480075s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.....X..X.......X.......X............X......................] duration: 11.368386144s FAIL (2025-07-09 23:31:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== uestItem_SegmentBegin Jul 9 23:12:42.134881962 (398h2m16.905523297s) [seg_pos={0 0}, upload=318452]: Begin upload segment... Jul 9 23:12:42.134950294 (398h2m16.905591629s) [seg_pos={0 0}, upload=318452]: Done begin upload segment. Jul 9 23:12:42.134981082 (398h2m16.905622417s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135020638 (398h2m16.905661983s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135575439 (398h2m16.906216774s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135895081 (398h2m16.906536416s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135916562 (398h2m16.906557897s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136222628 (398h2m16.906863963s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136501170 (398h2m16.907142506s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136773411 (398h2m16.907414746s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136994234 (398h2m16.907635559s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137216439 (398h2m16.907857774s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137458321 (398h2m16.908099656s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137623086 (398h2m16.908264421s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138134475 (398h2m16.90877581s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138499122 (398h2m16.909140458s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138736497 (398h2m16.909377822s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138951127 (398h2m16.909592452s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139222767 (398h2m16.909864102s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139541577 (398h2m16.910182912s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:43.510699884 (398h2m18.28134122s) [upload=318452]: Next returned nil segment Jul 9 23:12:43.510713381 (398h2m18.281354716s) [upload=318452]: Waiting for error group managing segments... Jul 9 23:12:44.093679677 (398h2m18.864321012s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139156961 (398h2m18.909798286s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139550674 (398h2m18.910192019s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139723775 (398h2m18.91036512s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.140606163 (398h2m18.911247489s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.141871074 (398h2m18.912512409s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143296170 (398h2m18.913937495s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143572189 (398h2m18.914213524s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:45.311783220 (398h2m20.082424555s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:47.074380105 (398h2m21.84502144s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:52.162572575 (398h2m26.93321391s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:13.763376135 (398h2m48.53401746s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:29.504846258 (398h3m4.275487583s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:41.049567150 (398h3m15.820208505s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:12cgYc8zVZh7w7iTTDC5zq1xK12wa5357zdAH3F3bZ21n1Y8uLg, address:104.36.82.17:4350): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049619941 (398h3m15.820261276s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1mLbGHntspYmsTVg8433HYSdAEMrXjGnDhmmvX1JmmgcxtdLgk, address:104.36.82.23:4464): protocol: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection Jul 9 23:13:41.049694543 (398h3m15.820335878s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:17SUmQEJJ3Qwe9yRq5DFTGaZBqGnz5Pu5YDri93Wk5NP3rcm5c, address:45.91.95.4:4023): protocol: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection Jul 9 23:13:41.049733979 (398h3m15.820375314s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1R6A6f6LTL3VDDg5gbhvQE55bRLJPDfAjViixeNX4KjVJpYFDE, address:45.91.95.3:4017): protocol: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection Jul 9 23:13:41.049758877 (398h3m15.820400212s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1m82Tdkxzub83FtYpd2Abhgf6StE1eFtxn1ESBi8BdHGGVhCu8, address:45.91.95.5:4025): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049823370 (398h3m15.820464705s) [seg_pos={0 0}, upload=318452]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 23:13:41.285661371 (398h3m16.056302706s) [upload=318452]: Done uploading segments... Jul 9 23:13:41.285688192 (398h3m16.056329517s) [upload=318452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 11.368411412s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X...X.....................................................] duration: 6.331943625s FAIL (2025-07-09 23:31:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.33196703s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [......X..X.......X.......X............X.....................] duration: 6.97168185s FAIL (2025-07-09 23:30:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== uestItem_SegmentBegin Jul 9 23:12:42.134881962 (398h2m16.905523297s) [seg_pos={0 0}, upload=318452]: Begin upload segment... Jul 9 23:12:42.134950294 (398h2m16.905591629s) [seg_pos={0 0}, upload=318452]: Done begin upload segment. Jul 9 23:12:42.134981082 (398h2m16.905622417s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135020638 (398h2m16.905661983s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135575439 (398h2m16.906216774s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135895081 (398h2m16.906536416s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135916562 (398h2m16.906557897s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136222628 (398h2m16.906863963s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136501170 (398h2m16.907142506s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136773411 (398h2m16.907414746s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136994234 (398h2m16.907635559s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137216439 (398h2m16.907857774s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137458321 (398h2m16.908099656s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137623086 (398h2m16.908264421s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138134475 (398h2m16.90877581s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138499122 (398h2m16.909140458s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138736497 (398h2m16.909377822s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138951127 (398h2m16.909592452s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139222767 (398h2m16.909864102s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139541577 (398h2m16.910182912s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:43.510699884 (398h2m18.28134122s) [upload=318452]: Next returned nil segment Jul 9 23:12:43.510713381 (398h2m18.281354716s) [upload=318452]: Waiting for error group managing segments... Jul 9 23:12:44.093679677 (398h2m18.864321012s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139156961 (398h2m18.909798286s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139550674 (398h2m18.910192019s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139723775 (398h2m18.91036512s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.140606163 (398h2m18.911247489s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.141871074 (398h2m18.912512409s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143296170 (398h2m18.913937495s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143572189 (398h2m18.914213524s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:45.311783220 (398h2m20.082424555s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:47.074380105 (398h2m21.84502144s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:52.162572575 (398h2m26.93321391s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:13.763376135 (398h2m48.53401746s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:29.504846258 (398h3m4.275487583s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:41.049567150 (398h3m15.820208505s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:12cgYc8zVZh7w7iTTDC5zq1xK12wa5357zdAH3F3bZ21n1Y8uLg, address:104.36.82.17:4350): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049619941 (398h3m15.820261276s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1mLbGHntspYmsTVg8433HYSdAEMrXjGnDhmmvX1JmmgcxtdLgk, address:104.36.82.23:4464): protocol: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection Jul 9 23:13:41.049694543 (398h3m15.820335878s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:17SUmQEJJ3Qwe9yRq5DFTGaZBqGnz5Pu5YDri93Wk5NP3rcm5c, address:45.91.95.4:4023): protocol: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection Jul 9 23:13:41.049733979 (398h3m15.820375314s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1R6A6f6LTL3VDDg5gbhvQE55bRLJPDfAjViixeNX4KjVJpYFDE, address:45.91.95.3:4017): protocol: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection Jul 9 23:13:41.049758877 (398h3m15.820400212s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1m82Tdkxzub83FtYpd2Abhgf6StE1eFtxn1ESBi8BdHGGVhCu8, address:45.91.95.5:4025): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049823370 (398h3m15.820464705s) [seg_pos={0 0}, upload=318452]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 23:13:41.285661371 (398h3m16.056302706s) [upload=318452]: Done uploading segments... Jul 9 23:13:41.285688192 (398h3m16.056329517s) [upload=318452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.971701187s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................................X.......X.................] duration: 6.293788649s FAIL (2025-07-09 23:30:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.293829568s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X...X....................................................] duration: 6.177986584s FAIL (2025-07-09 23:30:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.178011723s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................................X.......X................] duration: 5.862230315s FAIL (2025-07-09 23:30:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.86224828s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.......X..X.......X.......X............X....................] duration: 13.681401553s FAIL (2025-07-09 23:29:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== uestItem_SegmentBegin Jul 9 23:12:42.134881962 (398h2m16.905523297s) [seg_pos={0 0}, upload=318452]: Begin upload segment... Jul 9 23:12:42.134950294 (398h2m16.905591629s) [seg_pos={0 0}, upload=318452]: Done begin upload segment. Jul 9 23:12:42.134981082 (398h2m16.905622417s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135020638 (398h2m16.905661983s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135575439 (398h2m16.906216774s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135895081 (398h2m16.906536416s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135916562 (398h2m16.906557897s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136222628 (398h2m16.906863963s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136501170 (398h2m16.907142506s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136773411 (398h2m16.907414746s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136994234 (398h2m16.907635559s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137216439 (398h2m16.907857774s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137458321 (398h2m16.908099656s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137623086 (398h2m16.908264421s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138134475 (398h2m16.90877581s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138499122 (398h2m16.909140458s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138736497 (398h2m16.909377822s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138951127 (398h2m16.909592452s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139222767 (398h2m16.909864102s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139541577 (398h2m16.910182912s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:43.510699884 (398h2m18.28134122s) [upload=318452]: Next returned nil segment Jul 9 23:12:43.510713381 (398h2m18.281354716s) [upload=318452]: Waiting for error group managing segments... Jul 9 23:12:44.093679677 (398h2m18.864321012s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139156961 (398h2m18.909798286s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139550674 (398h2m18.910192019s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139723775 (398h2m18.91036512s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.140606163 (398h2m18.911247489s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.141871074 (398h2m18.912512409s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143296170 (398h2m18.913937495s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143572189 (398h2m18.914213524s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:45.311783220 (398h2m20.082424555s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:47.074380105 (398h2m21.84502144s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:52.162572575 (398h2m26.93321391s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:13.763376135 (398h2m48.53401746s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:29.504846258 (398h3m4.275487583s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:41.049567150 (398h3m15.820208505s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:12cgYc8zVZh7w7iTTDC5zq1xK12wa5357zdAH3F3bZ21n1Y8uLg, address:104.36.82.17:4350): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049619941 (398h3m15.820261276s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1mLbGHntspYmsTVg8433HYSdAEMrXjGnDhmmvX1JmmgcxtdLgk, address:104.36.82.23:4464): protocol: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection Jul 9 23:13:41.049694543 (398h3m15.820335878s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:17SUmQEJJ3Qwe9yRq5DFTGaZBqGnz5Pu5YDri93Wk5NP3rcm5c, address:45.91.95.4:4023): protocol: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection Jul 9 23:13:41.049733979 (398h3m15.820375314s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1R6A6f6LTL3VDDg5gbhvQE55bRLJPDfAjViixeNX4KjVJpYFDE, address:45.91.95.3:4017): protocol: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection Jul 9 23:13:41.049758877 (398h3m15.820400212s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1m82Tdkxzub83FtYpd2Abhgf6StE1eFtxn1ESBi8BdHGGVhCu8, address:45.91.95.5:4025): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049823370 (398h3m15.820464705s) [seg_pos={0 0}, upload=318452]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 23:13:41.285661371 (398h3m16.056302706s) [upload=318452]: Done uploading segments... Jul 9 23:13:41.285688192 (398h3m16.056329517s) [upload=318452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 13.68142635s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................................X.......X...............] duration: 7.654025636s FAIL (2025-07-09 23:29:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.654068478s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X...X...................................................] duration: 6.429185259s FAIL (2025-07-09 23:29:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.429208324s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)