Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X...X........................................] duration: 5.958760403s FAIL (2025-07-09 23:21:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.958782985s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.................X..X.......X.......X............X..........] duration: 26.038912355s FAIL (2025-07-09 23:21:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== uestItem_SegmentBegin Jul 9 23:12:42.134881962 (398h2m16.905523297s) [seg_pos={0 0}, upload=318452]: Begin upload segment... Jul 9 23:12:42.134950294 (398h2m16.905591629s) [seg_pos={0 0}, upload=318452]: Done begin upload segment. Jul 9 23:12:42.134981082 (398h2m16.905622417s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135020638 (398h2m16.905661983s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135575439 (398h2m16.906216774s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135895081 (398h2m16.906536416s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135916562 (398h2m16.906557897s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136222628 (398h2m16.906863963s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136501170 (398h2m16.907142506s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136773411 (398h2m16.907414746s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136994234 (398h2m16.907635559s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137216439 (398h2m16.907857774s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137458321 (398h2m16.908099656s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137623086 (398h2m16.908264421s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138134475 (398h2m16.90877581s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138499122 (398h2m16.909140458s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138736497 (398h2m16.909377822s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138951127 (398h2m16.909592452s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139222767 (398h2m16.909864102s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139541577 (398h2m16.910182912s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:43.510699884 (398h2m18.28134122s) [upload=318452]: Next returned nil segment Jul 9 23:12:43.510713381 (398h2m18.281354716s) [upload=318452]: Waiting for error group managing segments... Jul 9 23:12:44.093679677 (398h2m18.864321012s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139156961 (398h2m18.909798286s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139550674 (398h2m18.910192019s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139723775 (398h2m18.91036512s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.140606163 (398h2m18.911247489s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.141871074 (398h2m18.912512409s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143296170 (398h2m18.913937495s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143572189 (398h2m18.914213524s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:45.311783220 (398h2m20.082424555s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:47.074380105 (398h2m21.84502144s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:52.162572575 (398h2m26.93321391s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:13.763376135 (398h2m48.53401746s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:29.504846258 (398h3m4.275487583s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:41.049567150 (398h3m15.820208505s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:12cgYc8zVZh7w7iTTDC5zq1xK12wa5357zdAH3F3bZ21n1Y8uLg, address:104.36.82.17:4350): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049619941 (398h3m15.820261276s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1mLbGHntspYmsTVg8433HYSdAEMrXjGnDhmmvX1JmmgcxtdLgk, address:104.36.82.23:4464): protocol: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection Jul 9 23:13:41.049694543 (398h3m15.820335878s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:17SUmQEJJ3Qwe9yRq5DFTGaZBqGnz5Pu5YDri93Wk5NP3rcm5c, address:45.91.95.4:4023): protocol: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection Jul 9 23:13:41.049733979 (398h3m15.820375314s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1R6A6f6LTL3VDDg5gbhvQE55bRLJPDfAjViixeNX4KjVJpYFDE, address:45.91.95.3:4017): protocol: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection Jul 9 23:13:41.049758877 (398h3m15.820400212s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1m82Tdkxzub83FtYpd2Abhgf6StE1eFtxn1ESBi8BdHGGVhCu8, address:45.91.95.5:4025): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049823370 (398h3m15.820464705s) [seg_pos={0 0}, upload=318452]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 23:13:41.285661371 (398h3m16.056302706s) [upload=318452]: Done uploading segments... Jul 9 23:13:41.285688192 (398h3m16.056329517s) [upload=318452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 26.03893096s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X.......X.....] duration: 22.640052373s FAIL (2025-07-09 23:21:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.640667409s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X...............X...X.......................................] duration: 20.257619648s FAIL (2025-07-09 23:21:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 20.257669304s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X.......X....] duration: 29.268112746s FAIL (2025-07-09 23:20:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 29.268138735s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..................X..X.......X.......X............X.........] duration: 22.896659668s FAIL (2025-07-09 23:20:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== uestItem_SegmentBegin Jul 9 23:12:42.134881962 (398h2m16.905523297s) [seg_pos={0 0}, upload=318452]: Begin upload segment... Jul 9 23:12:42.134950294 (398h2m16.905591629s) [seg_pos={0 0}, upload=318452]: Done begin upload segment. Jul 9 23:12:42.134981082 (398h2m16.905622417s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135020638 (398h2m16.905661983s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135575439 (398h2m16.906216774s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135895081 (398h2m16.906536416s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135916562 (398h2m16.906557897s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136222628 (398h2m16.906863963s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136501170 (398h2m16.907142506s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136773411 (398h2m16.907414746s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136994234 (398h2m16.907635559s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137216439 (398h2m16.907857774s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137458321 (398h2m16.908099656s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137623086 (398h2m16.908264421s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138134475 (398h2m16.90877581s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138499122 (398h2m16.909140458s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138736497 (398h2m16.909377822s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138951127 (398h2m16.909592452s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139222767 (398h2m16.909864102s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139541577 (398h2m16.910182912s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:43.510699884 (398h2m18.28134122s) [upload=318452]: Next returned nil segment Jul 9 23:12:43.510713381 (398h2m18.281354716s) [upload=318452]: Waiting for error group managing segments... Jul 9 23:12:44.093679677 (398h2m18.864321012s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139156961 (398h2m18.909798286s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139550674 (398h2m18.910192019s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139723775 (398h2m18.91036512s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.140606163 (398h2m18.911247489s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.141871074 (398h2m18.912512409s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143296170 (398h2m18.913937495s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143572189 (398h2m18.914213524s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:45.311783220 (398h2m20.082424555s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:47.074380105 (398h2m21.84502144s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:52.162572575 (398h2m26.93321391s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:13.763376135 (398h2m48.53401746s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:29.504846258 (398h3m4.275487583s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:41.049567150 (398h3m15.820208505s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:12cgYc8zVZh7w7iTTDC5zq1xK12wa5357zdAH3F3bZ21n1Y8uLg, address:104.36.82.17:4350): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049619941 (398h3m15.820261276s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1mLbGHntspYmsTVg8433HYSdAEMrXjGnDhmmvX1JmmgcxtdLgk, address:104.36.82.23:4464): protocol: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection Jul 9 23:13:41.049694543 (398h3m15.820335878s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:17SUmQEJJ3Qwe9yRq5DFTGaZBqGnz5Pu5YDri93Wk5NP3rcm5c, address:45.91.95.4:4023): protocol: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection Jul 9 23:13:41.049733979 (398h3m15.820375314s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1R6A6f6LTL3VDDg5gbhvQE55bRLJPDfAjViixeNX4KjVJpYFDE, address:45.91.95.3:4017): protocol: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection Jul 9 23:13:41.049758877 (398h3m15.820400212s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1m82Tdkxzub83FtYpd2Abhgf6StE1eFtxn1ESBi8BdHGGVhCu8, address:45.91.95.5:4025): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049823370 (398h3m15.820464705s) [seg_pos={0 0}, upload=318452]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 23:13:41.285661371 (398h3m16.056302706s) [upload=318452]: Done uploading segments... Jul 9 23:13:41.285688192 (398h3m16.056329517s) [upload=318452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 22.89668166s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................................................X.......X...] duration: 12.942410161s FAIL (2025-07-09 23:20:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.942430441s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...................X..X.......X.......X............X........] duration: 9.285851082s FAIL (2025-07-09 23:20:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== uestItem_SegmentBegin Jul 9 23:12:42.134881962 (398h2m16.905523297s) [seg_pos={0 0}, upload=318452]: Begin upload segment... Jul 9 23:12:42.134950294 (398h2m16.905591629s) [seg_pos={0 0}, upload=318452]: Done begin upload segment. Jul 9 23:12:42.134981082 (398h2m16.905622417s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135020638 (398h2m16.905661983s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135575439 (398h2m16.906216774s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135895081 (398h2m16.906536416s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135916562 (398h2m16.906557897s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136222628 (398h2m16.906863963s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136501170 (398h2m16.907142506s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136773411 (398h2m16.907414746s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136994234 (398h2m16.907635559s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137216439 (398h2m16.907857774s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137458321 (398h2m16.908099656s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137623086 (398h2m16.908264421s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138134475 (398h2m16.90877581s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138499122 (398h2m16.909140458s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138736497 (398h2m16.909377822s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138951127 (398h2m16.909592452s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139222767 (398h2m16.909864102s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139541577 (398h2m16.910182912s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:43.510699884 (398h2m18.28134122s) [upload=318452]: Next returned nil segment Jul 9 23:12:43.510713381 (398h2m18.281354716s) [upload=318452]: Waiting for error group managing segments... Jul 9 23:12:44.093679677 (398h2m18.864321012s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139156961 (398h2m18.909798286s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139550674 (398h2m18.910192019s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139723775 (398h2m18.91036512s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.140606163 (398h2m18.911247489s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.141871074 (398h2m18.912512409s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143296170 (398h2m18.913937495s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143572189 (398h2m18.914213524s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:45.311783220 (398h2m20.082424555s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:47.074380105 (398h2m21.84502144s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:52.162572575 (398h2m26.93321391s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:13.763376135 (398h2m48.53401746s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:29.504846258 (398h3m4.275487583s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:41.049567150 (398h3m15.820208505s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:12cgYc8zVZh7w7iTTDC5zq1xK12wa5357zdAH3F3bZ21n1Y8uLg, address:104.36.82.17:4350): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049619941 (398h3m15.820261276s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1mLbGHntspYmsTVg8433HYSdAEMrXjGnDhmmvX1JmmgcxtdLgk, address:104.36.82.23:4464): protocol: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection Jul 9 23:13:41.049694543 (398h3m15.820335878s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:17SUmQEJJ3Qwe9yRq5DFTGaZBqGnz5Pu5YDri93Wk5NP3rcm5c, address:45.91.95.4:4023): protocol: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection Jul 9 23:13:41.049733979 (398h3m15.820375314s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1R6A6f6LTL3VDDg5gbhvQE55bRLJPDfAjViixeNX4KjVJpYFDE, address:45.91.95.3:4017): protocol: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection Jul 9 23:13:41.049758877 (398h3m15.820400212s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1m82Tdkxzub83FtYpd2Abhgf6StE1eFtxn1ESBi8BdHGGVhCu8, address:45.91.95.5:4025): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049823370 (398h3m15.820464705s) [seg_pos={0 0}, upload=318452]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 23:13:41.285661371 (398h3m16.056302706s) [upload=318452]: Done uploading segments... Jul 9 23:13:41.285688192 (398h3m16.056329517s) [upload=318452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 9.285896668s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X...............X...X......................................] duration: 6.254070872s FAIL (2025-07-09 23:19:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.254092582s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X...............X...X.....................................] duration: 6.399236926s FAIL (2025-07-09 23:19:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.399257635s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X.......X..] duration: 9.300286923s FAIL (2025-07-09 23:19:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.300326027s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [....................X..X.......X.......X............X.......] duration: 15.469307346s FAIL (2025-07-09 23:19:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== uestItem_SegmentBegin Jul 9 23:12:42.134881962 (398h2m16.905523297s) [seg_pos={0 0}, upload=318452]: Begin upload segment... Jul 9 23:12:42.134950294 (398h2m16.905591629s) [seg_pos={0 0}, upload=318452]: Done begin upload segment. Jul 9 23:12:42.134981082 (398h2m16.905622417s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135020638 (398h2m16.905661983s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135575439 (398h2m16.906216774s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135895081 (398h2m16.906536416s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135916562 (398h2m16.906557897s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136222628 (398h2m16.906863963s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136501170 (398h2m16.907142506s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136773411 (398h2m16.907414746s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136994234 (398h2m16.907635559s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137216439 (398h2m16.907857774s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137458321 (398h2m16.908099656s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137623086 (398h2m16.908264421s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138134475 (398h2m16.90877581s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138499122 (398h2m16.909140458s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138736497 (398h2m16.909377822s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138951127 (398h2m16.909592452s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139222767 (398h2m16.909864102s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139541577 (398h2m16.910182912s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:43.510699884 (398h2m18.28134122s) [upload=318452]: Next returned nil segment Jul 9 23:12:43.510713381 (398h2m18.281354716s) [upload=318452]: Waiting for error group managing segments... Jul 9 23:12:44.093679677 (398h2m18.864321012s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139156961 (398h2m18.909798286s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139550674 (398h2m18.910192019s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139723775 (398h2m18.91036512s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.140606163 (398h2m18.911247489s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.141871074 (398h2m18.912512409s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143296170 (398h2m18.913937495s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143572189 (398h2m18.914213524s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:45.311783220 (398h2m20.082424555s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:47.074380105 (398h2m21.84502144s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:52.162572575 (398h2m26.93321391s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:13.763376135 (398h2m48.53401746s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:29.504846258 (398h3m4.275487583s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:41.049567150 (398h3m15.820208505s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:12cgYc8zVZh7w7iTTDC5zq1xK12wa5357zdAH3F3bZ21n1Y8uLg, address:104.36.82.17:4350): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049619941 (398h3m15.820261276s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1mLbGHntspYmsTVg8433HYSdAEMrXjGnDhmmvX1JmmgcxtdLgk, address:104.36.82.23:4464): protocol: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection Jul 9 23:13:41.049694543 (398h3m15.820335878s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:17SUmQEJJ3Qwe9yRq5DFTGaZBqGnz5Pu5YDri93Wk5NP3rcm5c, address:45.91.95.4:4023): protocol: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection Jul 9 23:13:41.049733979 (398h3m15.820375314s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1R6A6f6LTL3VDDg5gbhvQE55bRLJPDfAjViixeNX4KjVJpYFDE, address:45.91.95.3:4017): protocol: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection Jul 9 23:13:41.049758877 (398h3m15.820400212s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1m82Tdkxzub83FtYpd2Abhgf6StE1eFtxn1ESBi8BdHGGVhCu8, address:45.91.95.5:4025): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049823370 (398h3m15.820464705s) [seg_pos={0 0}, upload=318452]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 23:13:41.285661371 (398h3m16.056302706s) [upload=318452]: Done uploading segments... Jul 9 23:13:41.285688192 (398h3m16.056329517s) [upload=318452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 15.469333526s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.....................X..X.......X.......X............X......] duration: 50.724351692s FAIL (2025-07-09 23:18:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== uestItem_SegmentBegin Jul 9 23:12:42.134881962 (398h2m16.905523297s) [seg_pos={0 0}, upload=318452]: Begin upload segment... Jul 9 23:12:42.134950294 (398h2m16.905591629s) [seg_pos={0 0}, upload=318452]: Done begin upload segment. Jul 9 23:12:42.134981082 (398h2m16.905622417s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135020638 (398h2m16.905661983s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135575439 (398h2m16.906216774s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135895081 (398h2m16.906536416s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135916562 (398h2m16.906557897s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136222628 (398h2m16.906863963s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136501170 (398h2m16.907142506s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136773411 (398h2m16.907414746s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136994234 (398h2m16.907635559s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137216439 (398h2m16.907857774s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137458321 (398h2m16.908099656s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137623086 (398h2m16.908264421s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138134475 (398h2m16.90877581s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138499122 (398h2m16.909140458s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138736497 (398h2m16.909377822s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138951127 (398h2m16.909592452s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139222767 (398h2m16.909864102s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139541577 (398h2m16.910182912s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:43.510699884 (398h2m18.28134122s) [upload=318452]: Next returned nil segment Jul 9 23:12:43.510713381 (398h2m18.281354716s) [upload=318452]: Waiting for error group managing segments... Jul 9 23:12:44.093679677 (398h2m18.864321012s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139156961 (398h2m18.909798286s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139550674 (398h2m18.910192019s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139723775 (398h2m18.91036512s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.140606163 (398h2m18.911247489s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.141871074 (398h2m18.912512409s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143296170 (398h2m18.913937495s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143572189 (398h2m18.914213524s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:45.311783220 (398h2m20.082424555s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:47.074380105 (398h2m21.84502144s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:52.162572575 (398h2m26.93321391s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:13.763376135 (398h2m48.53401746s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:29.504846258 (398h3m4.275487583s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:41.049567150 (398h3m15.820208505s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:12cgYc8zVZh7w7iTTDC5zq1xK12wa5357zdAH3F3bZ21n1Y8uLg, address:104.36.82.17:4350): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049619941 (398h3m15.820261276s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1mLbGHntspYmsTVg8433HYSdAEMrXjGnDhmmvX1JmmgcxtdLgk, address:104.36.82.23:4464): protocol: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection Jul 9 23:13:41.049694543 (398h3m15.820335878s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:17SUmQEJJ3Qwe9yRq5DFTGaZBqGnz5Pu5YDri93Wk5NP3rcm5c, address:45.91.95.4:4023): protocol: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection Jul 9 23:13:41.049733979 (398h3m15.820375314s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1R6A6f6LTL3VDDg5gbhvQE55bRLJPDfAjViixeNX4KjVJpYFDE, address:45.91.95.3:4017): protocol: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection Jul 9 23:13:41.049758877 (398h3m15.820400212s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1m82Tdkxzub83FtYpd2Abhgf6StE1eFtxn1ESBi8BdHGGVhCu8, address:45.91.95.5:4025): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049823370 (398h3m15.820464705s) [seg_pos={0 0}, upload=318452]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 23:13:41.285661371 (398h3m16.056302706s) [upload=318452]: Done uploading segments... Jul 9 23:13:41.285688192 (398h3m16.056329517s) [upload=318452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 50.724379555s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X.......X.] duration: 18.543445373s FAIL (2025-07-09 23:18:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.543457617s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X...............X...X....................................] duration: 6.141714638s FAIL (2025-07-09 23:18:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.142433965s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X.......X] duration: 1m0.000303369s FAIL (2025-07-09 23:18:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000316664s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X...............X...X...................................] duration: 5.901108836s FAIL (2025-07-09 23:18:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.901128925s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X...............X...X..................................] duration: 5.921507278s FAIL (2025-07-09 23:17:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.921527878s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [......................X..X.......X.......X............X.....] duration: 37.219764295s FAIL (2025-07-09 23:17:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== uestItem_SegmentBegin Jul 9 23:12:42.134881962 (398h2m16.905523297s) [seg_pos={0 0}, upload=318452]: Begin upload segment... Jul 9 23:12:42.134950294 (398h2m16.905591629s) [seg_pos={0 0}, upload=318452]: Done begin upload segment. Jul 9 23:12:42.134981082 (398h2m16.905622417s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135020638 (398h2m16.905661983s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135575439 (398h2m16.906216774s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135895081 (398h2m16.906536416s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135916562 (398h2m16.906557897s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136222628 (398h2m16.906863963s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136501170 (398h2m16.907142506s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136773411 (398h2m16.907414746s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136994234 (398h2m16.907635559s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137216439 (398h2m16.907857774s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137458321 (398h2m16.908099656s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137623086 (398h2m16.908264421s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138134475 (398h2m16.90877581s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138499122 (398h2m16.909140458s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138736497 (398h2m16.909377822s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138951127 (398h2m16.909592452s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139222767 (398h2m16.909864102s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139541577 (398h2m16.910182912s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:43.510699884 (398h2m18.28134122s) [upload=318452]: Next returned nil segment Jul 9 23:12:43.510713381 (398h2m18.281354716s) [upload=318452]: Waiting for error group managing segments... Jul 9 23:12:44.093679677 (398h2m18.864321012s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139156961 (398h2m18.909798286s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139550674 (398h2m18.910192019s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139723775 (398h2m18.91036512s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.140606163 (398h2m18.911247489s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.141871074 (398h2m18.912512409s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143296170 (398h2m18.913937495s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143572189 (398h2m18.914213524s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:45.311783220 (398h2m20.082424555s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:47.074380105 (398h2m21.84502144s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:52.162572575 (398h2m26.93321391s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:13.763376135 (398h2m48.53401746s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:29.504846258 (398h3m4.275487583s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:41.049567150 (398h3m15.820208505s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:12cgYc8zVZh7w7iTTDC5zq1xK12wa5357zdAH3F3bZ21n1Y8uLg, address:104.36.82.17:4350): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049619941 (398h3m15.820261276s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1mLbGHntspYmsTVg8433HYSdAEMrXjGnDhmmvX1JmmgcxtdLgk, address:104.36.82.23:4464): protocol: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection Jul 9 23:13:41.049694543 (398h3m15.820335878s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:17SUmQEJJ3Qwe9yRq5DFTGaZBqGnz5Pu5YDri93Wk5NP3rcm5c, address:45.91.95.4:4023): protocol: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection Jul 9 23:13:41.049733979 (398h3m15.820375314s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1R6A6f6LTL3VDDg5gbhvQE55bRLJPDfAjViixeNX4KjVJpYFDE, address:45.91.95.3:4017): protocol: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection Jul 9 23:13:41.049758877 (398h3m15.820400212s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1m82Tdkxzub83FtYpd2Abhgf6StE1eFtxn1ESBi8BdHGGVhCu8, address:45.91.95.5:4025): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049823370 (398h3m15.820464705s) [seg_pos={0 0}, upload=318452]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 23:13:41.285661371 (398h3m16.056302706s) [upload=318452]: Done uploading segments... Jul 9 23:13:41.285688192 (398h3m16.056329517s) [upload=318452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 37.219795195s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X...............X...X.................................] duration: 6.180343058s FAIL (2025-07-09 23:17:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.18038643s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [X...................................................X.......] duration: 16.710132022s FAIL (2025-07-09 23:16:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.710143354s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.......................X..X.......X.......X............X....] duration: 13.456922466s FAIL (2025-07-09 23:16:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== uestItem_SegmentBegin Jul 9 23:12:42.134881962 (398h2m16.905523297s) [seg_pos={0 0}, upload=318452]: Begin upload segment... Jul 9 23:12:42.134950294 (398h2m16.905591629s) [seg_pos={0 0}, upload=318452]: Done begin upload segment. Jul 9 23:12:42.134981082 (398h2m16.905622417s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135020638 (398h2m16.905661983s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135575439 (398h2m16.906216774s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135895081 (398h2m16.906536416s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135916562 (398h2m16.906557897s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136222628 (398h2m16.906863963s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136501170 (398h2m16.907142506s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136773411 (398h2m16.907414746s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136994234 (398h2m16.907635559s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137216439 (398h2m16.907857774s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137458321 (398h2m16.908099656s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137623086 (398h2m16.908264421s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138134475 (398h2m16.90877581s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138499122 (398h2m16.909140458s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138736497 (398h2m16.909377822s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138951127 (398h2m16.909592452s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139222767 (398h2m16.909864102s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139541577 (398h2m16.910182912s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:43.510699884 (398h2m18.28134122s) [upload=318452]: Next returned nil segment Jul 9 23:12:43.510713381 (398h2m18.281354716s) [upload=318452]: Waiting for error group managing segments... Jul 9 23:12:44.093679677 (398h2m18.864321012s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139156961 (398h2m18.909798286s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139550674 (398h2m18.910192019s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139723775 (398h2m18.91036512s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.140606163 (398h2m18.911247489s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.141871074 (398h2m18.912512409s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143296170 (398h2m18.913937495s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143572189 (398h2m18.914213524s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:45.311783220 (398h2m20.082424555s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:47.074380105 (398h2m21.84502144s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:52.162572575 (398h2m26.93321391s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:13.763376135 (398h2m48.53401746s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:29.504846258 (398h3m4.275487583s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:41.049567150 (398h3m15.820208505s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:12cgYc8zVZh7w7iTTDC5zq1xK12wa5357zdAH3F3bZ21n1Y8uLg, address:104.36.82.17:4350): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049619941 (398h3m15.820261276s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1mLbGHntspYmsTVg8433HYSdAEMrXjGnDhmmvX1JmmgcxtdLgk, address:104.36.82.23:4464): protocol: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection Jul 9 23:13:41.049694543 (398h3m15.820335878s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:17SUmQEJJ3Qwe9yRq5DFTGaZBqGnz5Pu5YDri93Wk5NP3rcm5c, address:45.91.95.4:4023): protocol: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection Jul 9 23:13:41.049733979 (398h3m15.820375314s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1R6A6f6LTL3VDDg5gbhvQE55bRLJPDfAjViixeNX4KjVJpYFDE, address:45.91.95.3:4017): protocol: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection Jul 9 23:13:41.049758877 (398h3m15.820400212s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1m82Tdkxzub83FtYpd2Abhgf6StE1eFtxn1ESBi8BdHGGVhCu8, address:45.91.95.5:4025): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049823370 (398h3m15.820464705s) [seg_pos={0 0}, upload=318452]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 23:13:41.285661371 (398h3m16.056302706s) [upload=318452]: Done uploading segments... Jul 9 23:13:41.285688192 (398h3m16.056329517s) [upload=318452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 13.456946271s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X...............X...X................................] duration: 6.230971472s FAIL (2025-07-09 23:16:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.230987071s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.X...................................................X......] duration: 36.464057333s FAIL (2025-07-09 23:16:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 36.464074516s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [........................X..X.......X.......X............X...] duration: 16.437067421s FAIL (2025-07-09 23:16:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== uestItem_SegmentBegin Jul 9 23:12:42.134881962 (398h2m16.905523297s) [seg_pos={0 0}, upload=318452]: Begin upload segment... Jul 9 23:12:42.134950294 (398h2m16.905591629s) [seg_pos={0 0}, upload=318452]: Done begin upload segment. Jul 9 23:12:42.134981082 (398h2m16.905622417s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135020638 (398h2m16.905661983s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135575439 (398h2m16.906216774s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135895081 (398h2m16.906536416s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135916562 (398h2m16.906557897s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136222628 (398h2m16.906863963s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136501170 (398h2m16.907142506s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136773411 (398h2m16.907414746s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136994234 (398h2m16.907635559s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137216439 (398h2m16.907857774s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137458321 (398h2m16.908099656s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137623086 (398h2m16.908264421s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138134475 (398h2m16.90877581s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138499122 (398h2m16.909140458s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138736497 (398h2m16.909377822s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138951127 (398h2m16.909592452s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139222767 (398h2m16.909864102s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139541577 (398h2m16.910182912s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:43.510699884 (398h2m18.28134122s) [upload=318452]: Next returned nil segment Jul 9 23:12:43.510713381 (398h2m18.281354716s) [upload=318452]: Waiting for error group managing segments... Jul 9 23:12:44.093679677 (398h2m18.864321012s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139156961 (398h2m18.909798286s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139550674 (398h2m18.910192019s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139723775 (398h2m18.91036512s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.140606163 (398h2m18.911247489s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.141871074 (398h2m18.912512409s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143296170 (398h2m18.913937495s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143572189 (398h2m18.914213524s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:45.311783220 (398h2m20.082424555s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:47.074380105 (398h2m21.84502144s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:52.162572575 (398h2m26.93321391s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:13.763376135 (398h2m48.53401746s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:29.504846258 (398h3m4.275487583s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:41.049567150 (398h3m15.820208505s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:12cgYc8zVZh7w7iTTDC5zq1xK12wa5357zdAH3F3bZ21n1Y8uLg, address:104.36.82.17:4350): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049619941 (398h3m15.820261276s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1mLbGHntspYmsTVg8433HYSdAEMrXjGnDhmmvX1JmmgcxtdLgk, address:104.36.82.23:4464): protocol: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection Jul 9 23:13:41.049694543 (398h3m15.820335878s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:17SUmQEJJ3Qwe9yRq5DFTGaZBqGnz5Pu5YDri93Wk5NP3rcm5c, address:45.91.95.4:4023): protocol: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection Jul 9 23:13:41.049733979 (398h3m15.820375314s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1R6A6f6LTL3VDDg5gbhvQE55bRLJPDfAjViixeNX4KjVJpYFDE, address:45.91.95.3:4017): protocol: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection Jul 9 23:13:41.049758877 (398h3m15.820400212s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1m82Tdkxzub83FtYpd2Abhgf6StE1eFtxn1ESBi8BdHGGVhCu8, address:45.91.95.5:4025): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049823370 (398h3m15.820464705s) [seg_pos={0 0}, upload=318452]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 23:13:41.285661371 (398h3m16.056302706s) [upload=318452]: Done uploading segments... Jul 9 23:13:41.285688192 (398h3m16.056329517s) [upload=318452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 16.437098361s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X...............X...X...............................] duration: 6.210860949s FAIL (2025-07-09 23:15:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.210884724s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.........................X..X.......X.......X............X..] duration: 27.827226636s FAIL (2025-07-09 23:15:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== uestItem_SegmentBegin Jul 9 23:12:42.134881962 (398h2m16.905523297s) [seg_pos={0 0}, upload=318452]: Begin upload segment... Jul 9 23:12:42.134950294 (398h2m16.905591629s) [seg_pos={0 0}, upload=318452]: Done begin upload segment. Jul 9 23:12:42.134981082 (398h2m16.905622417s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135020638 (398h2m16.905661983s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135575439 (398h2m16.906216774s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135895081 (398h2m16.906536416s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135916562 (398h2m16.906557897s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136222628 (398h2m16.906863963s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136501170 (398h2m16.907142506s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136773411 (398h2m16.907414746s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136994234 (398h2m16.907635559s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137216439 (398h2m16.907857774s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137458321 (398h2m16.908099656s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137623086 (398h2m16.908264421s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138134475 (398h2m16.90877581s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138499122 (398h2m16.909140458s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138736497 (398h2m16.909377822s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138951127 (398h2m16.909592452s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139222767 (398h2m16.909864102s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139541577 (398h2m16.910182912s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:43.510699884 (398h2m18.28134122s) [upload=318452]: Next returned nil segment Jul 9 23:12:43.510713381 (398h2m18.281354716s) [upload=318452]: Waiting for error group managing segments... Jul 9 23:12:44.093679677 (398h2m18.864321012s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139156961 (398h2m18.909798286s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139550674 (398h2m18.910192019s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139723775 (398h2m18.91036512s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.140606163 (398h2m18.911247489s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.141871074 (398h2m18.912512409s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143296170 (398h2m18.913937495s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143572189 (398h2m18.914213524s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:45.311783220 (398h2m20.082424555s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:47.074380105 (398h2m21.84502144s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:52.162572575 (398h2m26.93321391s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:13.763376135 (398h2m48.53401746s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:29.504846258 (398h3m4.275487583s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:41.049567150 (398h3m15.820208505s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:12cgYc8zVZh7w7iTTDC5zq1xK12wa5357zdAH3F3bZ21n1Y8uLg, address:104.36.82.17:4350): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049619941 (398h3m15.820261276s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1mLbGHntspYmsTVg8433HYSdAEMrXjGnDhmmvX1JmmgcxtdLgk, address:104.36.82.23:4464): protocol: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection Jul 9 23:13:41.049694543 (398h3m15.820335878s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:17SUmQEJJ3Qwe9yRq5DFTGaZBqGnz5Pu5YDri93Wk5NP3rcm5c, address:45.91.95.4:4023): protocol: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection Jul 9 23:13:41.049733979 (398h3m15.820375314s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1R6A6f6LTL3VDDg5gbhvQE55bRLJPDfAjViixeNX4KjVJpYFDE, address:45.91.95.3:4017): protocol: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection Jul 9 23:13:41.049758877 (398h3m15.820400212s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1m82Tdkxzub83FtYpd2Abhgf6StE1eFtxn1ESBi8BdHGGVhCu8, address:45.91.95.5:4025): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049823370 (398h3m15.820464705s) [seg_pos={0 0}, upload=318452]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 23:13:41.285661371 (398h3m16.056302706s) [upload=318452]: Done uploading segments... Jul 9 23:13:41.285688192 (398h3m16.056329517s) [upload=318452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 27.827251742s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..X...................................................X.....] duration: 21.657075364s FAIL (2025-07-09 23:15:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.657099541s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X...............X...X..............................] duration: 6.375479732s FAIL (2025-07-09 23:15:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.375502054s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...X...................................................X....] duration: 25.869171327s FAIL (2025-07-09 23:14:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.869182459s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X...............X...X.............................] duration: 6.000317666s FAIL (2025-07-09 23:13:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.00034053s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..........................X..X.......X.......X............X.] duration: 15.500218879s FAIL (2025-07-09 23:13:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== uestItem_SegmentBegin Jul 9 23:12:42.134881962 (398h2m16.905523297s) [seg_pos={0 0}, upload=318452]: Begin upload segment... Jul 9 23:12:42.134950294 (398h2m16.905591629s) [seg_pos={0 0}, upload=318452]: Done begin upload segment. Jul 9 23:12:42.134981082 (398h2m16.905622417s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135020638 (398h2m16.905661983s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135575439 (398h2m16.906216774s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135895081 (398h2m16.906536416s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135916562 (398h2m16.906557897s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136222628 (398h2m16.906863963s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136501170 (398h2m16.907142506s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136773411 (398h2m16.907414746s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136994234 (398h2m16.907635559s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137216439 (398h2m16.907857774s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137458321 (398h2m16.908099656s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137623086 (398h2m16.908264421s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138134475 (398h2m16.90877581s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138499122 (398h2m16.909140458s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138736497 (398h2m16.909377822s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138951127 (398h2m16.909592452s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139222767 (398h2m16.909864102s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139541577 (398h2m16.910182912s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:43.510699884 (398h2m18.28134122s) [upload=318452]: Next returned nil segment Jul 9 23:12:43.510713381 (398h2m18.281354716s) [upload=318452]: Waiting for error group managing segments... Jul 9 23:12:44.093679677 (398h2m18.864321012s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139156961 (398h2m18.909798286s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139550674 (398h2m18.910192019s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139723775 (398h2m18.91036512s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.140606163 (398h2m18.911247489s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.141871074 (398h2m18.912512409s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143296170 (398h2m18.913937495s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143572189 (398h2m18.914213524s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:45.311783220 (398h2m20.082424555s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:47.074380105 (398h2m21.84502144s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:52.162572575 (398h2m26.93321391s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:13.763376135 (398h2m48.53401746s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:29.504846258 (398h3m4.275487583s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:41.049567150 (398h3m15.820208505s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:12cgYc8zVZh7w7iTTDC5zq1xK12wa5357zdAH3F3bZ21n1Y8uLg, address:104.36.82.17:4350): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049619941 (398h3m15.820261276s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1mLbGHntspYmsTVg8433HYSdAEMrXjGnDhmmvX1JmmgcxtdLgk, address:104.36.82.23:4464): protocol: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection Jul 9 23:13:41.049694543 (398h3m15.820335878s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:17SUmQEJJ3Qwe9yRq5DFTGaZBqGnz5Pu5YDri93Wk5NP3rcm5c, address:45.91.95.4:4023): protocol: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection Jul 9 23:13:41.049733979 (398h3m15.820375314s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1R6A6f6LTL3VDDg5gbhvQE55bRLJPDfAjViixeNX4KjVJpYFDE, address:45.91.95.3:4017): protocol: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection Jul 9 23:13:41.049758877 (398h3m15.820400212s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1m82Tdkxzub83FtYpd2Abhgf6StE1eFtxn1ESBi8BdHGGVhCu8, address:45.91.95.5:4025): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049823370 (398h3m15.820464705s) [seg_pos={0 0}, upload=318452]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 23:13:41.285661371 (398h3m16.056302706s) [upload=318452]: Done uploading segments... Jul 9 23:13:41.285688192 (398h3m16.056329517s) [upload=318452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 15.500236122s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....X...................................................X...] duration: 28.6926422s FAIL (2025-07-09 23:13:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 28.692990778s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...........................X..X.......X.......X............X] duration: 1m0.237956371s FAIL (2025-07-09 23:13:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== uestItem_SegmentBegin Jul 9 23:12:42.134881962 (398h2m16.905523297s) [seg_pos={0 0}, upload=318452]: Begin upload segment... Jul 9 23:12:42.134950294 (398h2m16.905591629s) [seg_pos={0 0}, upload=318452]: Done begin upload segment. Jul 9 23:12:42.134981082 (398h2m16.905622417s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135020638 (398h2m16.905661983s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135575439 (398h2m16.906216774s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135895081 (398h2m16.906536416s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.135916562 (398h2m16.906557897s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136222628 (398h2m16.906863963s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136501170 (398h2m16.907142506s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136773411 (398h2m16.907414746s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.136994234 (398h2m16.907635559s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137216439 (398h2m16.907857774s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137458321 (398h2m16.908099656s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.137623086 (398h2m16.908264421s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138134475 (398h2m16.90877581s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138499122 (398h2m16.909140458s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138736497 (398h2m16.909377822s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.138951127 (398h2m16.909592452s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139222767 (398h2m16.909864102s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:42.139541577 (398h2m16.910182912s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Uploading piece... Jul 9 23:12:43.510699884 (398h2m18.28134122s) [upload=318452]: Next returned nil segment Jul 9 23:12:43.510713381 (398h2m18.281354716s) [upload=318452]: Waiting for error group managing segments... Jul 9 23:12:44.093679677 (398h2m18.864321012s) [noise=true , address=79.127.220.98:4399 , piece_id=JBYTJ2J2D7SM4H3RV3EQH5ABL4ZKNAIADKQK34F7QNCWEHYTSUWQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139156961 (398h2m18.909798286s) [noise=true , address=77.90.31.13:4635 , piece_id=FWYYW5PBLVBCDN7NVSBINFBXSEV4ILKI5C3V5PPPEBBDBNEBYN4A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139550674 (398h2m18.910192019s) [noise=true , address=77.90.31.16:4751 , piece_id=JW4FBKCDMFQTWPBDZMEFXRDEMIXYODHJE7HLM77MCRD4LFDXNZPQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.139723775 (398h2m18.91036512s) [noise=true , address=77.90.31.17:4798 , piece_id=TBDKFQBEJ7WT36EXSJG3P6PNW3XIZONBUZTOGTXY2VUFLF22REVQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.140606163 (398h2m18.911247489s) [noise=true , address=77.90.31.21:4903 , piece_id=R5QAKYLI5REVR225XMMC36NWZJUKA5OEYGGFXYSTEFM2AYULUA6A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.141871074 (398h2m18.912512409s) [noise=true , address=77.90.31.12:4614 , piece_id=NI2FQSAPIWR5MX7AFFEXYAQ7QRU2ZUJXOYVP5HWPQETSFDZ2KW4Q, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143296170 (398h2m18.913937495s) [noise=true , address=77.90.31.15:4707 , piece_id=DRQVQ2WANPEU45AEHTUG4NJ3L77HRJ6EAZA2VAGWNXIK7D52IMBQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:44.143572189 (398h2m18.914213524s) [noise=true , address=77.90.31.10:4544 , piece_id=RGK2KCBCURPJ4VCSVTJG3UDA6DRJ7FV3ZD4MI6RFLZJD4P5ZBT5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:45.311783220 (398h2m20.082424555s) [noise=true , address=79.127.201.215:4358 , piece_id=5CVNUXO5Y372KUW36TNVR3MC46O4E3ACKW5SX6SGW326XE37HBRA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:47.074380105 (398h2m21.84502144s) [noise=true , address=79.127.219.38:4026 , piece_id=A2UHMA5EFI43SAKWXDNQLL2PL3XPMNQTATSOJTIBXECPXRZRPOIA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:12:52.162572575 (398h2m26.93321391s) [noise=true , address=104.36.82.8:4980 , piece_id=SYG55RNPHNX4QPSLUD2RFJYJZSUR6LFINSBEZWJYSVW5XNLSEJSQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:13.763376135 (398h2m48.53401746s) [noise=true , address=45.91.95.2:4026 , piece_id=JGENHL3OPL53656GHVPHLIQBDXJJE5HTNTBBXESVVOSJJGFSLPCQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:29.504846258 (398h3m4.275487583s) [noise=true , address=104.36.82.19:4425 , piece_id=KU7X5R54DEHQI4P6MND6DBKFWQAEPBNL2WT7WWNPD67MC75MA73A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: <nil> Jul 9 23:13:41.049567150 (398h3m15.820208505s) [noise=true , address=104.36.82.17:4350 , piece_id=7S7QDNFA53XHNBTVWE4XBXB6WO3DEQVCITWIX6C7UVP4L4YIXQ5A, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:12cgYc8zVZh7w7iTTDC5zq1xK12wa5357zdAH3F3bZ21n1Y8uLg, address:104.36.82.17:4350): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049619941 (398h3m15.820261276s) [noise=true , address=104.36.82.23:4464 , piece_id=HKQ2EFJ5DDRTLMACCVLXHEH64IAY3KBCFATSKHKHY3TKG7ANW3UQ, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1mLbGHntspYmsTVg8433HYSdAEMrXjGnDhmmvX1JmmgcxtdLgk, address:104.36.82.23:4464): protocol: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43978->104.36.82.23:4464: use of closed network connection Jul 9 23:13:41.049694543 (398h3m15.820335878s) [noise=true , address=45.91.95.4:4023 , piece_id=CJSW4BYN6KT7AHN7NGCXY23JLGR6CMHTU722HSCGUKU46CG4BEWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:17SUmQEJJ3Qwe9yRq5DFTGaZBqGnz5Pu5YDri93Wk5NP3rcm5c, address:45.91.95.4:4023): protocol: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40226->45.91.95.4:4023: use of closed network connection Jul 9 23:13:41.049733979 (398h3m15.820375314s) [noise=true , address=45.91.95.3:4017 , piece_id=VH7DZMSQMOZ5365UHMZ5V3ZQXKI5DMTW24E7ZIWAKSLPWDZILVWA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1R6A6f6LTL3VDDg5gbhvQE55bRLJPDfAjViixeNX4KjVJpYFDE, address:45.91.95.3:4017): protocol: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48304->45.91.95.3:4017: use of closed network connection Jul 9 23:13:41.049758877 (398h3m15.820400212s) [noise=true , address=45.91.95.5:4025 , piece_id=FRE5AKNIJYUIZH5JCJFCJVP7V3K5NS4STFYI4YSZGEFZNCSVROFA, seg_pos={0 0}, upload=318452]: Done uploading piece. err: ecclient: upload failed (node:1m82Tdkxzub83FtYpd2Abhgf6StE1eFtxn1ESBi8BdHGGVhCu8, address:45.91.95.5:4025): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:13:41.049823370 (398h3m15.820464705s) [seg_pos={0 0}, upload=318452]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 23:13:41.285661371 (398h3m16.056302706s) [upload=318452]: Done uploading segments... Jul 9 23:13:41.285688192 (398h3m16.056329517s) [upload=318452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.237981798s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....X...................................................X..] duration: 50.81535118s FAIL (2025-07-09 23:13:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 50.815369924s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X...............X...X............................] duration: 6.164162748s FAIL (2025-07-09 23:13:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.16418473s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......X...................................................X.] duration: 17.718352613s FAIL (2025-07-09 23:12:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.718364617s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......X...................................................X] duration: 1m0.00149442s FAIL (2025-07-09 23:11:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001551329s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [............................X..X.......X.......X............] duration: 15.37501234s FAIL (2025-07-09 23:11:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 0 (397h51m47.070257915s) [noise=true , address=77.90.31.16:4738 , piece_id=EOD3ITGEW6B5TKWAAX5OLRSBHWLUSZ3HP6WRRC5JRNYOBURXJLYA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.299863513 (397h51m47.070504848s) [noise=true , address=45.91.95.3:4009 , piece_id=M6DJJMELQOAIEXU2OY2BUSGY63LI6WY6SFHVG6DFO64RS63J3SQA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.300155401 (397h51m47.070796736s) [noise=true , address=45.91.95.4:4001 , piece_id=DWNFECN6K3SEVJGBBKAF3QGK5TG4NMFGIZSKDB2NPV5APFBIEYRA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.300500541 (397h51m47.071141886s) [noise=true , address=79.127.220.101:4071 , piece_id=PCSVC42EYQVKAAHWVZOA4QNW3GRGOZINTK3A37OUKMEM37L2TURA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.300707928 (397h51m47.071349263s) [noise=true , address=104.36.82.36:4624 , piece_id=3MJAMTZ3D5HURGA37NIOBVLFCYUDNGK3X42KAJ2I6RG2DB5Q3YSQ, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.300895386 (397h51m47.071536712s) [noise=true , address=79.127.219.38:4031 , piece_id=CPSX3N3IDEWKUMHBQ6H6WMBFWDG3TGBOFV4AF464NRE4BTMRWYKQ, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.301093135 (397h51m47.07173447s) [noise=true , address=77.90.31.17:4802 , piece_id=OL6BCBP4YTNGDX23VQXEU7MBPWG42BYZ3AZYJ5DOWMOPYIZ7HKJA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.301289260 (397h51m47.071930595s) [noise=true , address=77.90.31.19:4876 , piece_id=MFCUKJNLJOGD6BSOJX34NF6ETPQB6GKHKHZZOBE27BIFS5C5QQ4A, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.301483822 (397h51m47.072125147s) [noise=true , address=104.36.82.23:4447 , piece_id=QJ7UQG24TH5QAPDOSOCMCI7B4S5V7ZCLDTIBQ7T5I5IWOLINDZWQ, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.301668996 (397h51m47.072310332s) [noise=true , address=104.36.82.19:4410 , piece_id=GIE3YQ3NHF44OHPJXNZQCCQLAYMH7WVHU5RYCC6UU3A5L4MNBDVQ, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.301881594 (397h51m47.072522929s) [noise=true , address=45.91.95.5:4022 , piece_id=63X6LG462THOR7RUHRZ4MGYRZDJZUD3CFUIE57GY4ENGMNR72GGA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.302036139 (397h51m47.072677484s) [noise=true , address=45.91.95.2:4015 , piece_id=SSMTWH5WQCMLFZOXRZX4WGCQDSHMUTD2W555XO6AA7OJFBUV72YA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.302541906 (397h51m47.073183241s) [noise=true , address=104.36.82.44:4899 , piece_id=BJ6QCUENRVC2MLUQRAL5NL7KP6OELTJWZ2NBYZJ3HPIJH3EGIKFQ, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.302904690 (397h51m47.073546025s) [noise=true , address=77.90.31.14:4688 , piece_id=BH5GKPDTKYKS7LMOSBFGCAK3GPTMOHS2XVHPBQ3IIXYUR5KOMNQQ, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.306087348 (397h51m47.076728683s) [noise=true , address=77.90.31.13:4645 , piece_id=SCDGEUTMKVRVHG4YA7DC6OYBRBEQQ7AJ3TJUGS3OXBAQ6E6VUZMQ, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.306494538 (397h51m47.077135883s) [noise=true , address=77.90.31.21:4884 , piece_id=Y6KXVSWCMBNCUKEXVDYT5MJTSHXQ6BTWQMQ755BPYXJZ54F6HPZA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.306742372 (397h51m47.077383707s) [noise=true , address=77.90.31.11:4575 , piece_id=QKIXONTL5DHGG2U6FCE4G5S2MGMLF6M342LSA5PIUEYAFRILAVEA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:13.699557207 (397h51m48.470198542s) [upload=318314]: Next returned nil segment Jul 9 23:02:13.699575412 (397h51m48.470216747s) [upload=318314]: Waiting for error group managing segments... Jul 9 23:02:14.004198502 (397h51m48.774839857s) [noise=true , address=79.127.220.101:4071 , piece_id=PCSVC42EYQVKAAHWVZOA4QNW3GRGOZINTK3A37OUKMEM37L2TURA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:14.317361159 (397h51m49.088002494s) [noise=true , address=77.90.31.17:4802 , piece_id=OL6BCBP4YTNGDX23VQXEU7MBPWG42BYZ3AZYJ5DOWMOPYIZ7HKJA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:14.318855398 (397h51m49.089496723s) [noise=true , address=77.90.31.14:4688 , piece_id=BH5GKPDTKYKS7LMOSBFGCAK3GPTMOHS2XVHPBQ3IIXYUR5KOMNQQ, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:14.321324141 (397h51m49.091965476s) [noise=true , address=77.90.31.11:4575 , piece_id=QKIXONTL5DHGG2U6FCE4G5S2MGMLF6M342LSA5PIUEYAFRILAVEA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:14.322031012 (397h51m49.092672357s) [noise=true , address=77.90.31.12:4602 , piece_id=YALPUB6SRLSX54PKVC4UOKREEAXYZ22JE3FDX4OS3I6RUC7B5KOQ, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:14.323073236 (397h51m49.093714572s) [noise=true , address=77.90.31.21:4884 , piece_id=Y6KXVSWCMBNCUKEXVDYT5MJTSHXQ6BTWQMQ755BPYXJZ54F6HPZA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:14.323756533 (397h51m49.094397868s) [noise=true , address=77.90.31.13:4645 , piece_id=SCDGEUTMKVRVHG4YA7DC6OYBRBEQQ7AJ3TJUGS3OXBAQ6E6VUZMQ, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:14.326098634 (397h51m49.096739969s) [noise=true , address=77.90.31.16:4738 , piece_id=EOD3ITGEW6B5TKWAAX5OLRSBHWLUSZ3HP6WRRC5JRNYOBURXJLYA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:14.329541540 (397h51m49.100182875s) [noise=true , address=77.90.31.19:4876 , piece_id=MFCUKJNLJOGD6BSOJX34NF6ETPQB6GKHKHZZOBE27BIFS5C5QQ4A, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:22.918099454 (397h51m57.688740789s) [noise=true , address=79.127.219.38:4031 , piece_id=CPSX3N3IDEWKUMHBQ6H6WMBFWDG3TGBOFV4AF464NRE4BTMRWYKQ, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:03:11.197265107 (397h52m45.967906442s) [noise=true , address=45.91.95.2:4015 , piece_id=SSMTWH5WQCMLFZOXRZX4WGCQDSHMUTD2W555XO6AA7OJFBUV72YA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: ecclient: upload failed (node:1swt4fnsU2PmLbnozfXxbQPNbdDVjgdfpxFFR4VsvAdKcf5ZUj, address:45.91.95.2:4015): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:03:11.197291919 (397h52m45.967933254s) [noise=true , address=45.91.95.3:4009 , piece_id=M6DJJMELQOAIEXU2OY2BUSGY63LI6WY6SFHVG6DFO64RS63J3SQA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: ecclient: upload failed (node:12Yb4nEC9PfMTDs4iPU4WmSCHJrALQSDzKSwyD2z5g6WxM5Q5cB, address:45.91.95.3:4009): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:03:11.197383253 (397h52m45.968024588s) [noise=true , address=104.36.82.23:4447 , piece_id=QJ7UQG24TH5QAPDOSOCMCI7B4S5V7ZCLDTIBQ7T5I5IWOLINDZWQ, seg_pos={0 0}, upload=318314]: Done uploading piece. err: ecclient: upload failed (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, address:104.36.82.23:4447): protocol: write tcp 188.166.225.170:48822->104.36.82.23:4447: use of closed network connection; write tcp 188.166.225.170:48822->104.36.82.23:4447: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48822->104.36.82.23:4447: use of closed network connection Jul 9 23:03:11.197407600 (397h52m45.968048935s) [noise=true , address=45.91.95.5:4022 , piece_id=63X6LG462THOR7RUHRZ4MGYRZDJZUD3CFUIE57GY4ENGMNR72GGA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: ecclient: upload failed (node:12X8Et5FNyAsE7ifEckfQemZkfnvYUuZ2Rn2xdxNxhmCmuDST2M, address:45.91.95.5:4022): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:03:11.197423410 (397h52m45.968064745s) [noise=true , address=45.91.95.4:4001 , piece_id=DWNFECN6K3SEVJGBBKAF3QGK5TG4NMFGIZSKDB2NPV5APFBIEYRA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: ecclient: upload failed (node:12PdimTdB4ysD3SQ8hKMAGAXkzzC71irLv6rW9FArf5qmrvZ5MY, address:45.91.95.4:4001): protocol: write tcp 188.166.225.170:33470->45.91.95.4:4001: use of closed network connection; write tcp 188.166.225.170:33470->45.91.95.4:4001: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33470->45.91.95.4:4001: use of closed network connection Jul 9 23:03:11.197480189 (397h52m45.968121524s) [noise=true , address=104.36.82.36:4624 , piece_id=3MJAMTZ3D5HURGA37NIOBVLFCYUDNGK3X42KAJ2I6RG2DB5Q3YSQ, seg_pos={0 0}, upload=318314]: Done uploading piece. err: ecclient: upload failed (node:1GG1c98sVuVmeJgmwXuVqxm7hKZ7Wv3LgwQGzgzGtYETe4RbF1, address:104.36.82.36:4624): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:03:11.197527659 (397h52m45.968168994s) [noise=true , address=104.36.82.44:4899 , piece_id=BJ6QCUENRVC2MLUQRAL5NL7KP6OELTJWZ2NBYZJ3HPIJH3EGIKFQ, seg_pos={0 0}, upload=318314]: Done uploading piece. err: ecclient: upload failed (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw, address:104.36.82.44:4899): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:03:11.197541516 (397h52m45.968182841s) [noise=true , address=104.36.82.19:4410 , piece_id=GIE3YQ3NHF44OHPJXNZQCCQLAYMH7WVHU5RYCC6UU3A5L4MNBDVQ, seg_pos={0 0}, upload=318314]: Done uploading piece. err: ecclient: upload failed (node:12hLhHpBmtWRfwU7osDv82ncUNjTXWR4JQfpPxtoLKMNyeaUSSA, address:104.36.82.19:4410): protocol: expected piece hash; write tcp 188.166.225.170:35686->104.36.82.19:4410: use of closed network connection; write tcp 188.166.225.170:35686->104.36.82.19:4410: use of closed network connection; EOF Jul 9 23:03:11.197562597 (397h52m45.968203932s) [seg_pos={0 0}, upload=318314]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 9 23:03:11.438173501 (397h52m46.208814836s) [upload=318314]: Done uploading segments... Jul 9 23:03:11.438197957 (397h52m46.208839292s) [upload=318314]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 15.37502885s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X...............X...X...........................] duration: 6.252449795s FAIL (2025-07-09 23:11:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.252465113s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.............................X..X.......X.......X...........] duration: 53.997611594s FAIL (2025-07-09 23:11:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 0 (397h51m47.070257915s) [noise=true , address=77.90.31.16:4738 , piece_id=EOD3ITGEW6B5TKWAAX5OLRSBHWLUSZ3HP6WRRC5JRNYOBURXJLYA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.299863513 (397h51m47.070504848s) [noise=true , address=45.91.95.3:4009 , piece_id=M6DJJMELQOAIEXU2OY2BUSGY63LI6WY6SFHVG6DFO64RS63J3SQA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.300155401 (397h51m47.070796736s) [noise=true , address=45.91.95.4:4001 , piece_id=DWNFECN6K3SEVJGBBKAF3QGK5TG4NMFGIZSKDB2NPV5APFBIEYRA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.300500541 (397h51m47.071141886s) [noise=true , address=79.127.220.101:4071 , piece_id=PCSVC42EYQVKAAHWVZOA4QNW3GRGOZINTK3A37OUKMEM37L2TURA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.300707928 (397h51m47.071349263s) [noise=true , address=104.36.82.36:4624 , piece_id=3MJAMTZ3D5HURGA37NIOBVLFCYUDNGK3X42KAJ2I6RG2DB5Q3YSQ, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.300895386 (397h51m47.071536712s) [noise=true , address=79.127.219.38:4031 , piece_id=CPSX3N3IDEWKUMHBQ6H6WMBFWDG3TGBOFV4AF464NRE4BTMRWYKQ, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.301093135 (397h51m47.07173447s) [noise=true , address=77.90.31.17:4802 , piece_id=OL6BCBP4YTNGDX23VQXEU7MBPWG42BYZ3AZYJ5DOWMOPYIZ7HKJA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.301289260 (397h51m47.071930595s) [noise=true , address=77.90.31.19:4876 , piece_id=MFCUKJNLJOGD6BSOJX34NF6ETPQB6GKHKHZZOBE27BIFS5C5QQ4A, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.301483822 (397h51m47.072125147s) [noise=true , address=104.36.82.23:4447 , piece_id=QJ7UQG24TH5QAPDOSOCMCI7B4S5V7ZCLDTIBQ7T5I5IWOLINDZWQ, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.301668996 (397h51m47.072310332s) [noise=true , address=104.36.82.19:4410 , piece_id=GIE3YQ3NHF44OHPJXNZQCCQLAYMH7WVHU5RYCC6UU3A5L4MNBDVQ, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.301881594 (397h51m47.072522929s) [noise=true , address=45.91.95.5:4022 , piece_id=63X6LG462THOR7RUHRZ4MGYRZDJZUD3CFUIE57GY4ENGMNR72GGA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.302036139 (397h51m47.072677484s) [noise=true , address=45.91.95.2:4015 , piece_id=SSMTWH5WQCMLFZOXRZX4WGCQDSHMUTD2W555XO6AA7OJFBUV72YA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.302541906 (397h51m47.073183241s) [noise=true , address=104.36.82.44:4899 , piece_id=BJ6QCUENRVC2MLUQRAL5NL7KP6OELTJWZ2NBYZJ3HPIJH3EGIKFQ, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.302904690 (397h51m47.073546025s) [noise=true , address=77.90.31.14:4688 , piece_id=BH5GKPDTKYKS7LMOSBFGCAK3GPTMOHS2XVHPBQ3IIXYUR5KOMNQQ, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.306087348 (397h51m47.076728683s) [noise=true , address=77.90.31.13:4645 , piece_id=SCDGEUTMKVRVHG4YA7DC6OYBRBEQQ7AJ3TJUGS3OXBAQ6E6VUZMQ, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.306494538 (397h51m47.077135883s) [noise=true , address=77.90.31.21:4884 , piece_id=Y6KXVSWCMBNCUKEXVDYT5MJTSHXQ6BTWQMQ755BPYXJZ54F6HPZA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.306742372 (397h51m47.077383707s) [noise=true , address=77.90.31.11:4575 , piece_id=QKIXONTL5DHGG2U6FCE4G5S2MGMLF6M342LSA5PIUEYAFRILAVEA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:13.699557207 (397h51m48.470198542s) [upload=318314]: Next returned nil segment Jul 9 23:02:13.699575412 (397h51m48.470216747s) [upload=318314]: Waiting for error group managing segments... Jul 9 23:02:14.004198502 (397h51m48.774839857s) [noise=true , address=79.127.220.101:4071 , piece_id=PCSVC42EYQVKAAHWVZOA4QNW3GRGOZINTK3A37OUKMEM37L2TURA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:14.317361159 (397h51m49.088002494s) [noise=true , address=77.90.31.17:4802 , piece_id=OL6BCBP4YTNGDX23VQXEU7MBPWG42BYZ3AZYJ5DOWMOPYIZ7HKJA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:14.318855398 (397h51m49.089496723s) [noise=true , address=77.90.31.14:4688 , piece_id=BH5GKPDTKYKS7LMOSBFGCAK3GPTMOHS2XVHPBQ3IIXYUR5KOMNQQ, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:14.321324141 (397h51m49.091965476s) [noise=true , address=77.90.31.11:4575 , piece_id=QKIXONTL5DHGG2U6FCE4G5S2MGMLF6M342LSA5PIUEYAFRILAVEA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:14.322031012 (397h51m49.092672357s) [noise=true , address=77.90.31.12:4602 , piece_id=YALPUB6SRLSX54PKVC4UOKREEAXYZ22JE3FDX4OS3I6RUC7B5KOQ, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:14.323073236 (397h51m49.093714572s) [noise=true , address=77.90.31.21:4884 , piece_id=Y6KXVSWCMBNCUKEXVDYT5MJTSHXQ6BTWQMQ755BPYXJZ54F6HPZA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:14.323756533 (397h51m49.094397868s) [noise=true , address=77.90.31.13:4645 , piece_id=SCDGEUTMKVRVHG4YA7DC6OYBRBEQQ7AJ3TJUGS3OXBAQ6E6VUZMQ, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:14.326098634 (397h51m49.096739969s) [noise=true , address=77.90.31.16:4738 , piece_id=EOD3ITGEW6B5TKWAAX5OLRSBHWLUSZ3HP6WRRC5JRNYOBURXJLYA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:14.329541540 (397h51m49.100182875s) [noise=true , address=77.90.31.19:4876 , piece_id=MFCUKJNLJOGD6BSOJX34NF6ETPQB6GKHKHZZOBE27BIFS5C5QQ4A, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:22.918099454 (397h51m57.688740789s) [noise=true , address=79.127.219.38:4031 , piece_id=CPSX3N3IDEWKUMHBQ6H6WMBFWDG3TGBOFV4AF464NRE4BTMRWYKQ, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:03:11.197265107 (397h52m45.967906442s) [noise=true , address=45.91.95.2:4015 , piece_id=SSMTWH5WQCMLFZOXRZX4WGCQDSHMUTD2W555XO6AA7OJFBUV72YA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: ecclient: upload failed (node:1swt4fnsU2PmLbnozfXxbQPNbdDVjgdfpxFFR4VsvAdKcf5ZUj, address:45.91.95.2:4015): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:03:11.197291919 (397h52m45.967933254s) [noise=true , address=45.91.95.3:4009 , piece_id=M6DJJMELQOAIEXU2OY2BUSGY63LI6WY6SFHVG6DFO64RS63J3SQA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: ecclient: upload failed (node:12Yb4nEC9PfMTDs4iPU4WmSCHJrALQSDzKSwyD2z5g6WxM5Q5cB, address:45.91.95.3:4009): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:03:11.197383253 (397h52m45.968024588s) [noise=true , address=104.36.82.23:4447 , piece_id=QJ7UQG24TH5QAPDOSOCMCI7B4S5V7ZCLDTIBQ7T5I5IWOLINDZWQ, seg_pos={0 0}, upload=318314]: Done uploading piece. err: ecclient: upload failed (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, address:104.36.82.23:4447): protocol: write tcp 188.166.225.170:48822->104.36.82.23:4447: use of closed network connection; write tcp 188.166.225.170:48822->104.36.82.23:4447: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48822->104.36.82.23:4447: use of closed network connection Jul 9 23:03:11.197407600 (397h52m45.968048935s) [noise=true , address=45.91.95.5:4022 , piece_id=63X6LG462THOR7RUHRZ4MGYRZDJZUD3CFUIE57GY4ENGMNR72GGA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: ecclient: upload failed (node:12X8Et5FNyAsE7ifEckfQemZkfnvYUuZ2Rn2xdxNxhmCmuDST2M, address:45.91.95.5:4022): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:03:11.197423410 (397h52m45.968064745s) [noise=true , address=45.91.95.4:4001 , piece_id=DWNFECN6K3SEVJGBBKAF3QGK5TG4NMFGIZSKDB2NPV5APFBIEYRA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: ecclient: upload failed (node:12PdimTdB4ysD3SQ8hKMAGAXkzzC71irLv6rW9FArf5qmrvZ5MY, address:45.91.95.4:4001): protocol: write tcp 188.166.225.170:33470->45.91.95.4:4001: use of closed network connection; write tcp 188.166.225.170:33470->45.91.95.4:4001: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33470->45.91.95.4:4001: use of closed network connection Jul 9 23:03:11.197480189 (397h52m45.968121524s) [noise=true , address=104.36.82.36:4624 , piece_id=3MJAMTZ3D5HURGA37NIOBVLFCYUDNGK3X42KAJ2I6RG2DB5Q3YSQ, seg_pos={0 0}, upload=318314]: Done uploading piece. err: ecclient: upload failed (node:1GG1c98sVuVmeJgmwXuVqxm7hKZ7Wv3LgwQGzgzGtYETe4RbF1, address:104.36.82.36:4624): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:03:11.197527659 (397h52m45.968168994s) [noise=true , address=104.36.82.44:4899 , piece_id=BJ6QCUENRVC2MLUQRAL5NL7KP6OELTJWZ2NBYZJ3HPIJH3EGIKFQ, seg_pos={0 0}, upload=318314]: Done uploading piece. err: ecclient: upload failed (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw, address:104.36.82.44:4899): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:03:11.197541516 (397h52m45.968182841s) [noise=true , address=104.36.82.19:4410 , piece_id=GIE3YQ3NHF44OHPJXNZQCCQLAYMH7WVHU5RYCC6UU3A5L4MNBDVQ, seg_pos={0 0}, upload=318314]: Done uploading piece. err: ecclient: upload failed (node:12hLhHpBmtWRfwU7osDv82ncUNjTXWR4JQfpPxtoLKMNyeaUSSA, address:104.36.82.19:4410): protocol: expected piece hash; write tcp 188.166.225.170:35686->104.36.82.19:4410: use of closed network connection; write tcp 188.166.225.170:35686->104.36.82.19:4410: use of closed network connection; EOF Jul 9 23:03:11.197562597 (397h52m45.968203932s) [seg_pos={0 0}, upload=318314]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 9 23:03:11.438173501 (397h52m46.208814836s) [upload=318314]: Done uploading segments... Jul 9 23:03:11.438197957 (397h52m46.208839292s) [upload=318314]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 53.997636762s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X...............X...X..........................] duration: 5.933871509s FAIL (2025-07-09 23:10:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.933904462s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 12.88024557s FAIL (2025-07-09 23:10:10 UTC) orbiter warning: context deadline exceeded Duration: 12.880261291s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..............................X..X.......X.......X..........] duration: 13.863647553s FAIL (2025-07-09 23:10:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 0 (397h51m47.070257915s) [noise=true , address=77.90.31.16:4738 , piece_id=EOD3ITGEW6B5TKWAAX5OLRSBHWLUSZ3HP6WRRC5JRNYOBURXJLYA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.299863513 (397h51m47.070504848s) [noise=true , address=45.91.95.3:4009 , piece_id=M6DJJMELQOAIEXU2OY2BUSGY63LI6WY6SFHVG6DFO64RS63J3SQA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.300155401 (397h51m47.070796736s) [noise=true , address=45.91.95.4:4001 , piece_id=DWNFECN6K3SEVJGBBKAF3QGK5TG4NMFGIZSKDB2NPV5APFBIEYRA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.300500541 (397h51m47.071141886s) [noise=true , address=79.127.220.101:4071 , piece_id=PCSVC42EYQVKAAHWVZOA4QNW3GRGOZINTK3A37OUKMEM37L2TURA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.300707928 (397h51m47.071349263s) [noise=true , address=104.36.82.36:4624 , piece_id=3MJAMTZ3D5HURGA37NIOBVLFCYUDNGK3X42KAJ2I6RG2DB5Q3YSQ, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.300895386 (397h51m47.071536712s) [noise=true , address=79.127.219.38:4031 , piece_id=CPSX3N3IDEWKUMHBQ6H6WMBFWDG3TGBOFV4AF464NRE4BTMRWYKQ, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.301093135 (397h51m47.07173447s) [noise=true , address=77.90.31.17:4802 , piece_id=OL6BCBP4YTNGDX23VQXEU7MBPWG42BYZ3AZYJ5DOWMOPYIZ7HKJA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.301289260 (397h51m47.071930595s) [noise=true , address=77.90.31.19:4876 , piece_id=MFCUKJNLJOGD6BSOJX34NF6ETPQB6GKHKHZZOBE27BIFS5C5QQ4A, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.301483822 (397h51m47.072125147s) [noise=true , address=104.36.82.23:4447 , piece_id=QJ7UQG24TH5QAPDOSOCMCI7B4S5V7ZCLDTIBQ7T5I5IWOLINDZWQ, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.301668996 (397h51m47.072310332s) [noise=true , address=104.36.82.19:4410 , piece_id=GIE3YQ3NHF44OHPJXNZQCCQLAYMH7WVHU5RYCC6UU3A5L4MNBDVQ, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.301881594 (397h51m47.072522929s) [noise=true , address=45.91.95.5:4022 , piece_id=63X6LG462THOR7RUHRZ4MGYRZDJZUD3CFUIE57GY4ENGMNR72GGA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.302036139 (397h51m47.072677484s) [noise=true , address=45.91.95.2:4015 , piece_id=SSMTWH5WQCMLFZOXRZX4WGCQDSHMUTD2W555XO6AA7OJFBUV72YA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.302541906 (397h51m47.073183241s) [noise=true , address=104.36.82.44:4899 , piece_id=BJ6QCUENRVC2MLUQRAL5NL7KP6OELTJWZ2NBYZJ3HPIJH3EGIKFQ, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.302904690 (397h51m47.073546025s) [noise=true , address=77.90.31.14:4688 , piece_id=BH5GKPDTKYKS7LMOSBFGCAK3GPTMOHS2XVHPBQ3IIXYUR5KOMNQQ, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.306087348 (397h51m47.076728683s) [noise=true , address=77.90.31.13:4645 , piece_id=SCDGEUTMKVRVHG4YA7DC6OYBRBEQQ7AJ3TJUGS3OXBAQ6E6VUZMQ, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.306494538 (397h51m47.077135883s) [noise=true , address=77.90.31.21:4884 , piece_id=Y6KXVSWCMBNCUKEXVDYT5MJTSHXQ6BTWQMQ755BPYXJZ54F6HPZA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.306742372 (397h51m47.077383707s) [noise=true , address=77.90.31.11:4575 , piece_id=QKIXONTL5DHGG2U6FCE4G5S2MGMLF6M342LSA5PIUEYAFRILAVEA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:13.699557207 (397h51m48.470198542s) [upload=318314]: Next returned nil segment Jul 9 23:02:13.699575412 (397h51m48.470216747s) [upload=318314]: Waiting for error group managing segments... Jul 9 23:02:14.004198502 (397h51m48.774839857s) [noise=true , address=79.127.220.101:4071 , piece_id=PCSVC42EYQVKAAHWVZOA4QNW3GRGOZINTK3A37OUKMEM37L2TURA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:14.317361159 (397h51m49.088002494s) [noise=true , address=77.90.31.17:4802 , piece_id=OL6BCBP4YTNGDX23VQXEU7MBPWG42BYZ3AZYJ5DOWMOPYIZ7HKJA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:14.318855398 (397h51m49.089496723s) [noise=true , address=77.90.31.14:4688 , piece_id=BH5GKPDTKYKS7LMOSBFGCAK3GPTMOHS2XVHPBQ3IIXYUR5KOMNQQ, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:14.321324141 (397h51m49.091965476s) [noise=true , address=77.90.31.11:4575 , piece_id=QKIXONTL5DHGG2U6FCE4G5S2MGMLF6M342LSA5PIUEYAFRILAVEA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:14.322031012 (397h51m49.092672357s) [noise=true , address=77.90.31.12:4602 , piece_id=YALPUB6SRLSX54PKVC4UOKREEAXYZ22JE3FDX4OS3I6RUC7B5KOQ, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:14.323073236 (397h51m49.093714572s) [noise=true , address=77.90.31.21:4884 , piece_id=Y6KXVSWCMBNCUKEXVDYT5MJTSHXQ6BTWQMQ755BPYXJZ54F6HPZA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:14.323756533 (397h51m49.094397868s) [noise=true , address=77.90.31.13:4645 , piece_id=SCDGEUTMKVRVHG4YA7DC6OYBRBEQQ7AJ3TJUGS3OXBAQ6E6VUZMQ, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:14.326098634 (397h51m49.096739969s) [noise=true , address=77.90.31.16:4738 , piece_id=EOD3ITGEW6B5TKWAAX5OLRSBHWLUSZ3HP6WRRC5JRNYOBURXJLYA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:14.329541540 (397h51m49.100182875s) [noise=true , address=77.90.31.19:4876 , piece_id=MFCUKJNLJOGD6BSOJX34NF6ETPQB6GKHKHZZOBE27BIFS5C5QQ4A, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:22.918099454 (397h51m57.688740789s) [noise=true , address=79.127.219.38:4031 , piece_id=CPSX3N3IDEWKUMHBQ6H6WMBFWDG3TGBOFV4AF464NRE4BTMRWYKQ, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:03:11.197265107 (397h52m45.967906442s) [noise=true , address=45.91.95.2:4015 , piece_id=SSMTWH5WQCMLFZOXRZX4WGCQDSHMUTD2W555XO6AA7OJFBUV72YA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: ecclient: upload failed (node:1swt4fnsU2PmLbnozfXxbQPNbdDVjgdfpxFFR4VsvAdKcf5ZUj, address:45.91.95.2:4015): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:03:11.197291919 (397h52m45.967933254s) [noise=true , address=45.91.95.3:4009 , piece_id=M6DJJMELQOAIEXU2OY2BUSGY63LI6WY6SFHVG6DFO64RS63J3SQA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: ecclient: upload failed (node:12Yb4nEC9PfMTDs4iPU4WmSCHJrALQSDzKSwyD2z5g6WxM5Q5cB, address:45.91.95.3:4009): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:03:11.197383253 (397h52m45.968024588s) [noise=true , address=104.36.82.23:4447 , piece_id=QJ7UQG24TH5QAPDOSOCMCI7B4S5V7ZCLDTIBQ7T5I5IWOLINDZWQ, seg_pos={0 0}, upload=318314]: Done uploading piece. err: ecclient: upload failed (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, address:104.36.82.23:4447): protocol: write tcp 188.166.225.170:48822->104.36.82.23:4447: use of closed network connection; write tcp 188.166.225.170:48822->104.36.82.23:4447: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48822->104.36.82.23:4447: use of closed network connection Jul 9 23:03:11.197407600 (397h52m45.968048935s) [noise=true , address=45.91.95.5:4022 , piece_id=63X6LG462THOR7RUHRZ4MGYRZDJZUD3CFUIE57GY4ENGMNR72GGA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: ecclient: upload failed (node:12X8Et5FNyAsE7ifEckfQemZkfnvYUuZ2Rn2xdxNxhmCmuDST2M, address:45.91.95.5:4022): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:03:11.197423410 (397h52m45.968064745s) [noise=true , address=45.91.95.4:4001 , piece_id=DWNFECN6K3SEVJGBBKAF3QGK5TG4NMFGIZSKDB2NPV5APFBIEYRA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: ecclient: upload failed (node:12PdimTdB4ysD3SQ8hKMAGAXkzzC71irLv6rW9FArf5qmrvZ5MY, address:45.91.95.4:4001): protocol: write tcp 188.166.225.170:33470->45.91.95.4:4001: use of closed network connection; write tcp 188.166.225.170:33470->45.91.95.4:4001: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33470->45.91.95.4:4001: use of closed network connection Jul 9 23:03:11.197480189 (397h52m45.968121524s) [noise=true , address=104.36.82.36:4624 , piece_id=3MJAMTZ3D5HURGA37NIOBVLFCYUDNGK3X42KAJ2I6RG2DB5Q3YSQ, seg_pos={0 0}, upload=318314]: Done uploading piece. err: ecclient: upload failed (node:1GG1c98sVuVmeJgmwXuVqxm7hKZ7Wv3LgwQGzgzGtYETe4RbF1, address:104.36.82.36:4624): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:03:11.197527659 (397h52m45.968168994s) [noise=true , address=104.36.82.44:4899 , piece_id=BJ6QCUENRVC2MLUQRAL5NL7KP6OELTJWZ2NBYZJ3HPIJH3EGIKFQ, seg_pos={0 0}, upload=318314]: Done uploading piece. err: ecclient: upload failed (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw, address:104.36.82.44:4899): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:03:11.197541516 (397h52m45.968182841s) [noise=true , address=104.36.82.19:4410 , piece_id=GIE3YQ3NHF44OHPJXNZQCCQLAYMH7WVHU5RYCC6UU3A5L4MNBDVQ, seg_pos={0 0}, upload=318314]: Done uploading piece. err: ecclient: upload failed (node:12hLhHpBmtWRfwU7osDv82ncUNjTXWR4JQfpPxtoLKMNyeaUSSA, address:104.36.82.19:4410): protocol: expected piece hash; write tcp 188.166.225.170:35686->104.36.82.19:4410: use of closed network connection; write tcp 188.166.225.170:35686->104.36.82.19:4410: use of closed network connection; EOF Jul 9 23:03:11.197562597 (397h52m45.968203932s) [seg_pos={0 0}, upload=318314]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 9 23:03:11.438173501 (397h52m46.208814836s) [upload=318314]: Done uploading segments... Jul 9 23:03:11.438197957 (397h52m46.208839292s) [upload=318314]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 13.863669135s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...............................X..X.......X.......X.........] duration: 32.121170501s FAIL (2025-07-09 23:09:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 0 (397h51m47.070257915s) [noise=true , address=77.90.31.16:4738 , piece_id=EOD3ITGEW6B5TKWAAX5OLRSBHWLUSZ3HP6WRRC5JRNYOBURXJLYA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.299863513 (397h51m47.070504848s) [noise=true , address=45.91.95.3:4009 , piece_id=M6DJJMELQOAIEXU2OY2BUSGY63LI6WY6SFHVG6DFO64RS63J3SQA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.300155401 (397h51m47.070796736s) [noise=true , address=45.91.95.4:4001 , piece_id=DWNFECN6K3SEVJGBBKAF3QGK5TG4NMFGIZSKDB2NPV5APFBIEYRA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.300500541 (397h51m47.071141886s) [noise=true , address=79.127.220.101:4071 , piece_id=PCSVC42EYQVKAAHWVZOA4QNW3GRGOZINTK3A37OUKMEM37L2TURA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.300707928 (397h51m47.071349263s) [noise=true , address=104.36.82.36:4624 , piece_id=3MJAMTZ3D5HURGA37NIOBVLFCYUDNGK3X42KAJ2I6RG2DB5Q3YSQ, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.300895386 (397h51m47.071536712s) [noise=true , address=79.127.219.38:4031 , piece_id=CPSX3N3IDEWKUMHBQ6H6WMBFWDG3TGBOFV4AF464NRE4BTMRWYKQ, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.301093135 (397h51m47.07173447s) [noise=true , address=77.90.31.17:4802 , piece_id=OL6BCBP4YTNGDX23VQXEU7MBPWG42BYZ3AZYJ5DOWMOPYIZ7HKJA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.301289260 (397h51m47.071930595s) [noise=true , address=77.90.31.19:4876 , piece_id=MFCUKJNLJOGD6BSOJX34NF6ETPQB6GKHKHZZOBE27BIFS5C5QQ4A, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.301483822 (397h51m47.072125147s) [noise=true , address=104.36.82.23:4447 , piece_id=QJ7UQG24TH5QAPDOSOCMCI7B4S5V7ZCLDTIBQ7T5I5IWOLINDZWQ, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.301668996 (397h51m47.072310332s) [noise=true , address=104.36.82.19:4410 , piece_id=GIE3YQ3NHF44OHPJXNZQCCQLAYMH7WVHU5RYCC6UU3A5L4MNBDVQ, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.301881594 (397h51m47.072522929s) [noise=true , address=45.91.95.5:4022 , piece_id=63X6LG462THOR7RUHRZ4MGYRZDJZUD3CFUIE57GY4ENGMNR72GGA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.302036139 (397h51m47.072677484s) [noise=true , address=45.91.95.2:4015 , piece_id=SSMTWH5WQCMLFZOXRZX4WGCQDSHMUTD2W555XO6AA7OJFBUV72YA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.302541906 (397h51m47.073183241s) [noise=true , address=104.36.82.44:4899 , piece_id=BJ6QCUENRVC2MLUQRAL5NL7KP6OELTJWZ2NBYZJ3HPIJH3EGIKFQ, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.302904690 (397h51m47.073546025s) [noise=true , address=77.90.31.14:4688 , piece_id=BH5GKPDTKYKS7LMOSBFGCAK3GPTMOHS2XVHPBQ3IIXYUR5KOMNQQ, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.306087348 (397h51m47.076728683s) [noise=true , address=77.90.31.13:4645 , piece_id=SCDGEUTMKVRVHG4YA7DC6OYBRBEQQ7AJ3TJUGS3OXBAQ6E6VUZMQ, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.306494538 (397h51m47.077135883s) [noise=true , address=77.90.31.21:4884 , piece_id=Y6KXVSWCMBNCUKEXVDYT5MJTSHXQ6BTWQMQ755BPYXJZ54F6HPZA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.306742372 (397h51m47.077383707s) [noise=true , address=77.90.31.11:4575 , piece_id=QKIXONTL5DHGG2U6FCE4G5S2MGMLF6M342LSA5PIUEYAFRILAVEA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:13.699557207 (397h51m48.470198542s) [upload=318314]: Next returned nil segment Jul 9 23:02:13.699575412 (397h51m48.470216747s) [upload=318314]: Waiting for error group managing segments... Jul 9 23:02:14.004198502 (397h51m48.774839857s) [noise=true , address=79.127.220.101:4071 , piece_id=PCSVC42EYQVKAAHWVZOA4QNW3GRGOZINTK3A37OUKMEM37L2TURA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:14.317361159 (397h51m49.088002494s) [noise=true , address=77.90.31.17:4802 , piece_id=OL6BCBP4YTNGDX23VQXEU7MBPWG42BYZ3AZYJ5DOWMOPYIZ7HKJA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:14.318855398 (397h51m49.089496723s) [noise=true , address=77.90.31.14:4688 , piece_id=BH5GKPDTKYKS7LMOSBFGCAK3GPTMOHS2XVHPBQ3IIXYUR5KOMNQQ, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:14.321324141 (397h51m49.091965476s) [noise=true , address=77.90.31.11:4575 , piece_id=QKIXONTL5DHGG2U6FCE4G5S2MGMLF6M342LSA5PIUEYAFRILAVEA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:14.322031012 (397h51m49.092672357s) [noise=true , address=77.90.31.12:4602 , piece_id=YALPUB6SRLSX54PKVC4UOKREEAXYZ22JE3FDX4OS3I6RUC7B5KOQ, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:14.323073236 (397h51m49.093714572s) [noise=true , address=77.90.31.21:4884 , piece_id=Y6KXVSWCMBNCUKEXVDYT5MJTSHXQ6BTWQMQ755BPYXJZ54F6HPZA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:14.323756533 (397h51m49.094397868s) [noise=true , address=77.90.31.13:4645 , piece_id=SCDGEUTMKVRVHG4YA7DC6OYBRBEQQ7AJ3TJUGS3OXBAQ6E6VUZMQ, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:14.326098634 (397h51m49.096739969s) [noise=true , address=77.90.31.16:4738 , piece_id=EOD3ITGEW6B5TKWAAX5OLRSBHWLUSZ3HP6WRRC5JRNYOBURXJLYA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:14.329541540 (397h51m49.100182875s) [noise=true , address=77.90.31.19:4876 , piece_id=MFCUKJNLJOGD6BSOJX34NF6ETPQB6GKHKHZZOBE27BIFS5C5QQ4A, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:22.918099454 (397h51m57.688740789s) [noise=true , address=79.127.219.38:4031 , piece_id=CPSX3N3IDEWKUMHBQ6H6WMBFWDG3TGBOFV4AF464NRE4BTMRWYKQ, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:03:11.197265107 (397h52m45.967906442s) [noise=true , address=45.91.95.2:4015 , piece_id=SSMTWH5WQCMLFZOXRZX4WGCQDSHMUTD2W555XO6AA7OJFBUV72YA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: ecclient: upload failed (node:1swt4fnsU2PmLbnozfXxbQPNbdDVjgdfpxFFR4VsvAdKcf5ZUj, address:45.91.95.2:4015): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:03:11.197291919 (397h52m45.967933254s) [noise=true , address=45.91.95.3:4009 , piece_id=M6DJJMELQOAIEXU2OY2BUSGY63LI6WY6SFHVG6DFO64RS63J3SQA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: ecclient: upload failed (node:12Yb4nEC9PfMTDs4iPU4WmSCHJrALQSDzKSwyD2z5g6WxM5Q5cB, address:45.91.95.3:4009): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:03:11.197383253 (397h52m45.968024588s) [noise=true , address=104.36.82.23:4447 , piece_id=QJ7UQG24TH5QAPDOSOCMCI7B4S5V7ZCLDTIBQ7T5I5IWOLINDZWQ, seg_pos={0 0}, upload=318314]: Done uploading piece. err: ecclient: upload failed (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, address:104.36.82.23:4447): protocol: write tcp 188.166.225.170:48822->104.36.82.23:4447: use of closed network connection; write tcp 188.166.225.170:48822->104.36.82.23:4447: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48822->104.36.82.23:4447: use of closed network connection Jul 9 23:03:11.197407600 (397h52m45.968048935s) [noise=true , address=45.91.95.5:4022 , piece_id=63X6LG462THOR7RUHRZ4MGYRZDJZUD3CFUIE57GY4ENGMNR72GGA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: ecclient: upload failed (node:12X8Et5FNyAsE7ifEckfQemZkfnvYUuZ2Rn2xdxNxhmCmuDST2M, address:45.91.95.5:4022): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:03:11.197423410 (397h52m45.968064745s) [noise=true , address=45.91.95.4:4001 , piece_id=DWNFECN6K3SEVJGBBKAF3QGK5TG4NMFGIZSKDB2NPV5APFBIEYRA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: ecclient: upload failed (node:12PdimTdB4ysD3SQ8hKMAGAXkzzC71irLv6rW9FArf5qmrvZ5MY, address:45.91.95.4:4001): protocol: write tcp 188.166.225.170:33470->45.91.95.4:4001: use of closed network connection; write tcp 188.166.225.170:33470->45.91.95.4:4001: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33470->45.91.95.4:4001: use of closed network connection Jul 9 23:03:11.197480189 (397h52m45.968121524s) [noise=true , address=104.36.82.36:4624 , piece_id=3MJAMTZ3D5HURGA37NIOBVLFCYUDNGK3X42KAJ2I6RG2DB5Q3YSQ, seg_pos={0 0}, upload=318314]: Done uploading piece. err: ecclient: upload failed (node:1GG1c98sVuVmeJgmwXuVqxm7hKZ7Wv3LgwQGzgzGtYETe4RbF1, address:104.36.82.36:4624): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:03:11.197527659 (397h52m45.968168994s) [noise=true , address=104.36.82.44:4899 , piece_id=BJ6QCUENRVC2MLUQRAL5NL7KP6OELTJWZ2NBYZJ3HPIJH3EGIKFQ, seg_pos={0 0}, upload=318314]: Done uploading piece. err: ecclient: upload failed (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw, address:104.36.82.44:4899): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:03:11.197541516 (397h52m45.968182841s) [noise=true , address=104.36.82.19:4410 , piece_id=GIE3YQ3NHF44OHPJXNZQCCQLAYMH7WVHU5RYCC6UU3A5L4MNBDVQ, seg_pos={0 0}, upload=318314]: Done uploading piece. err: ecclient: upload failed (node:12hLhHpBmtWRfwU7osDv82ncUNjTXWR4JQfpPxtoLKMNyeaUSSA, address:104.36.82.19:4410): protocol: expected piece hash; write tcp 188.166.225.170:35686->104.36.82.19:4410: use of closed network connection; write tcp 188.166.225.170:35686->104.36.82.19:4410: use of closed network connection; EOF Jul 9 23:03:11.197562597 (397h52m45.968203932s) [seg_pos={0 0}, upload=318314]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 9 23:03:11.438173501 (397h52m46.208814836s) [upload=318314]: Done uploading segments... Jul 9 23:03:11.438197957 (397h52m46.208839292s) [upload=318314]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 32.121188656s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 53.783960115s FAIL (2025-07-09 23:09:52 UTC) orbiter warning: context deadline exceeded Duration: 53.783983039s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X...............X...X.........................] duration: 6.065433301s FAIL (2025-07-09 23:09:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.06546894s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X...............X...X........................] duration: 5.980600511s FAIL (2025-07-09 23:09:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.980621861s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [X...............................X..X.......X.......X........] duration: 11.279828023s FAIL (2025-07-09 23:08:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 0 (397h51m47.070257915s) [noise=true , address=77.90.31.16:4738 , piece_id=EOD3ITGEW6B5TKWAAX5OLRSBHWLUSZ3HP6WRRC5JRNYOBURXJLYA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.299863513 (397h51m47.070504848s) [noise=true , address=45.91.95.3:4009 , piece_id=M6DJJMELQOAIEXU2OY2BUSGY63LI6WY6SFHVG6DFO64RS63J3SQA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.300155401 (397h51m47.070796736s) [noise=true , address=45.91.95.4:4001 , piece_id=DWNFECN6K3SEVJGBBKAF3QGK5TG4NMFGIZSKDB2NPV5APFBIEYRA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.300500541 (397h51m47.071141886s) [noise=true , address=79.127.220.101:4071 , piece_id=PCSVC42EYQVKAAHWVZOA4QNW3GRGOZINTK3A37OUKMEM37L2TURA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.300707928 (397h51m47.071349263s) [noise=true , address=104.36.82.36:4624 , piece_id=3MJAMTZ3D5HURGA37NIOBVLFCYUDNGK3X42KAJ2I6RG2DB5Q3YSQ, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.300895386 (397h51m47.071536712s) [noise=true , address=79.127.219.38:4031 , piece_id=CPSX3N3IDEWKUMHBQ6H6WMBFWDG3TGBOFV4AF464NRE4BTMRWYKQ, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.301093135 (397h51m47.07173447s) [noise=true , address=77.90.31.17:4802 , piece_id=OL6BCBP4YTNGDX23VQXEU7MBPWG42BYZ3AZYJ5DOWMOPYIZ7HKJA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.301289260 (397h51m47.071930595s) [noise=true , address=77.90.31.19:4876 , piece_id=MFCUKJNLJOGD6BSOJX34NF6ETPQB6GKHKHZZOBE27BIFS5C5QQ4A, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.301483822 (397h51m47.072125147s) [noise=true , address=104.36.82.23:4447 , piece_id=QJ7UQG24TH5QAPDOSOCMCI7B4S5V7ZCLDTIBQ7T5I5IWOLINDZWQ, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.301668996 (397h51m47.072310332s) [noise=true , address=104.36.82.19:4410 , piece_id=GIE3YQ3NHF44OHPJXNZQCCQLAYMH7WVHU5RYCC6UU3A5L4MNBDVQ, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.301881594 (397h51m47.072522929s) [noise=true , address=45.91.95.5:4022 , piece_id=63X6LG462THOR7RUHRZ4MGYRZDJZUD3CFUIE57GY4ENGMNR72GGA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.302036139 (397h51m47.072677484s) [noise=true , address=45.91.95.2:4015 , piece_id=SSMTWH5WQCMLFZOXRZX4WGCQDSHMUTD2W555XO6AA7OJFBUV72YA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.302541906 (397h51m47.073183241s) [noise=true , address=104.36.82.44:4899 , piece_id=BJ6QCUENRVC2MLUQRAL5NL7KP6OELTJWZ2NBYZJ3HPIJH3EGIKFQ, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.302904690 (397h51m47.073546025s) [noise=true , address=77.90.31.14:4688 , piece_id=BH5GKPDTKYKS7LMOSBFGCAK3GPTMOHS2XVHPBQ3IIXYUR5KOMNQQ, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.306087348 (397h51m47.076728683s) [noise=true , address=77.90.31.13:4645 , piece_id=SCDGEUTMKVRVHG4YA7DC6OYBRBEQQ7AJ3TJUGS3OXBAQ6E6VUZMQ, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.306494538 (397h51m47.077135883s) [noise=true , address=77.90.31.21:4884 , piece_id=Y6KXVSWCMBNCUKEXVDYT5MJTSHXQ6BTWQMQ755BPYXJZ54F6HPZA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:12.306742372 (397h51m47.077383707s) [noise=true , address=77.90.31.11:4575 , piece_id=QKIXONTL5DHGG2U6FCE4G5S2MGMLF6M342LSA5PIUEYAFRILAVEA, seg_pos={0 0}, upload=318314]: Uploading piece... Jul 9 23:02:13.699557207 (397h51m48.470198542s) [upload=318314]: Next returned nil segment Jul 9 23:02:13.699575412 (397h51m48.470216747s) [upload=318314]: Waiting for error group managing segments... Jul 9 23:02:14.004198502 (397h51m48.774839857s) [noise=true , address=79.127.220.101:4071 , piece_id=PCSVC42EYQVKAAHWVZOA4QNW3GRGOZINTK3A37OUKMEM37L2TURA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:14.317361159 (397h51m49.088002494s) [noise=true , address=77.90.31.17:4802 , piece_id=OL6BCBP4YTNGDX23VQXEU7MBPWG42BYZ3AZYJ5DOWMOPYIZ7HKJA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:14.318855398 (397h51m49.089496723s) [noise=true , address=77.90.31.14:4688 , piece_id=BH5GKPDTKYKS7LMOSBFGCAK3GPTMOHS2XVHPBQ3IIXYUR5KOMNQQ, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:14.321324141 (397h51m49.091965476s) [noise=true , address=77.90.31.11:4575 , piece_id=QKIXONTL5DHGG2U6FCE4G5S2MGMLF6M342LSA5PIUEYAFRILAVEA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:14.322031012 (397h51m49.092672357s) [noise=true , address=77.90.31.12:4602 , piece_id=YALPUB6SRLSX54PKVC4UOKREEAXYZ22JE3FDX4OS3I6RUC7B5KOQ, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:14.323073236 (397h51m49.093714572s) [noise=true , address=77.90.31.21:4884 , piece_id=Y6KXVSWCMBNCUKEXVDYT5MJTSHXQ6BTWQMQ755BPYXJZ54F6HPZA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:14.323756533 (397h51m49.094397868s) [noise=true , address=77.90.31.13:4645 , piece_id=SCDGEUTMKVRVHG4YA7DC6OYBRBEQQ7AJ3TJUGS3OXBAQ6E6VUZMQ, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:14.326098634 (397h51m49.096739969s) [noise=true , address=77.90.31.16:4738 , piece_id=EOD3ITGEW6B5TKWAAX5OLRSBHWLUSZ3HP6WRRC5JRNYOBURXJLYA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:14.329541540 (397h51m49.100182875s) [noise=true , address=77.90.31.19:4876 , piece_id=MFCUKJNLJOGD6BSOJX34NF6ETPQB6GKHKHZZOBE27BIFS5C5QQ4A, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:02:22.918099454 (397h51m57.688740789s) [noise=true , address=79.127.219.38:4031 , piece_id=CPSX3N3IDEWKUMHBQ6H6WMBFWDG3TGBOFV4AF464NRE4BTMRWYKQ, seg_pos={0 0}, upload=318314]: Done uploading piece. err: <nil> Jul 9 23:03:11.197265107 (397h52m45.967906442s) [noise=true , address=45.91.95.2:4015 , piece_id=SSMTWH5WQCMLFZOXRZX4WGCQDSHMUTD2W555XO6AA7OJFBUV72YA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: ecclient: upload failed (node:1swt4fnsU2PmLbnozfXxbQPNbdDVjgdfpxFFR4VsvAdKcf5ZUj, address:45.91.95.2:4015): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:03:11.197291919 (397h52m45.967933254s) [noise=true , address=45.91.95.3:4009 , piece_id=M6DJJMELQOAIEXU2OY2BUSGY63LI6WY6SFHVG6DFO64RS63J3SQA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: ecclient: upload failed (node:12Yb4nEC9PfMTDs4iPU4WmSCHJrALQSDzKSwyD2z5g6WxM5Q5cB, address:45.91.95.3:4009): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:03:11.197383253 (397h52m45.968024588s) [noise=true , address=104.36.82.23:4447 , piece_id=QJ7UQG24TH5QAPDOSOCMCI7B4S5V7ZCLDTIBQ7T5I5IWOLINDZWQ, seg_pos={0 0}, upload=318314]: Done uploading piece. err: ecclient: upload failed (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, address:104.36.82.23:4447): protocol: write tcp 188.166.225.170:48822->104.36.82.23:4447: use of closed network connection; write tcp 188.166.225.170:48822->104.36.82.23:4447: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48822->104.36.82.23:4447: use of closed network connection Jul 9 23:03:11.197407600 (397h52m45.968048935s) [noise=true , address=45.91.95.5:4022 , piece_id=63X6LG462THOR7RUHRZ4MGYRZDJZUD3CFUIE57GY4ENGMNR72GGA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: ecclient: upload failed (node:12X8Et5FNyAsE7ifEckfQemZkfnvYUuZ2Rn2xdxNxhmCmuDST2M, address:45.91.95.5:4022): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:03:11.197423410 (397h52m45.968064745s) [noise=true , address=45.91.95.4:4001 , piece_id=DWNFECN6K3SEVJGBBKAF3QGK5TG4NMFGIZSKDB2NPV5APFBIEYRA, seg_pos={0 0}, upload=318314]: Done uploading piece. err: ecclient: upload failed (node:12PdimTdB4ysD3SQ8hKMAGAXkzzC71irLv6rW9FArf5qmrvZ5MY, address:45.91.95.4:4001): protocol: write tcp 188.166.225.170:33470->45.91.95.4:4001: use of closed network connection; write tcp 188.166.225.170:33470->45.91.95.4:4001: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33470->45.91.95.4:4001: use of closed network connection Jul 9 23:03:11.197480189 (397h52m45.968121524s) [noise=true , address=104.36.82.36:4624 , piece_id=3MJAMTZ3D5HURGA37NIOBVLFCYUDNGK3X42KAJ2I6RG2DB5Q3YSQ, seg_pos={0 0}, upload=318314]: Done uploading piece. err: ecclient: upload failed (node:1GG1c98sVuVmeJgmwXuVqxm7hKZ7Wv3LgwQGzgzGtYETe4RbF1, address:104.36.82.36:4624): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:03:11.197527659 (397h52m45.968168994s) [noise=true , address=104.36.82.44:4899 , piece_id=BJ6QCUENRVC2MLUQRAL5NL7KP6OELTJWZ2NBYZJ3HPIJH3EGIKFQ, seg_pos={0 0}, upload=318314]: Done uploading piece. err: ecclient: upload failed (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw, address:104.36.82.44:4899): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 23:03:11.197541516 (397h52m45.968182841s) [noise=true , address=104.36.82.19:4410 , piece_id=GIE3YQ3NHF44OHPJXNZQCCQLAYMH7WVHU5RYCC6UU3A5L4MNBDVQ, seg_pos={0 0}, upload=318314]: Done uploading piece. err: ecclient: upload failed (node:12hLhHpBmtWRfwU7osDv82ncUNjTXWR4JQfpPxtoLKMNyeaUSSA, address:104.36.82.19:4410): protocol: expected piece hash; write tcp 188.166.225.170:35686->104.36.82.19:4410: use of closed network connection; write tcp 188.166.225.170:35686->104.36.82.19:4410: use of closed network connection; EOF Jul 9 23:03:11.197562597 (397h52m45.968203932s) [seg_pos={0 0}, upload=318314]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 9 23:03:11.438173501 (397h52m46.208814836s) [upload=318314]: Done uploading segments... Jul 9 23:03:11.438197957 (397h52m46.208839292s) [upload=318314]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 11.279844586s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 27.197640544s FAIL (2025-07-09 23:08:18 UTC) orbiter warning: context deadline exceeded Duration: 27.198024046s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)