Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [X....X..X.........X...............X..X....X........X........] duration: 28.473549891s FAIL (2025-07-09 21:55:10 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 28.473605397s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [...........X....X...............X..X.X...X...X.......X......] duration: 33.865917148s FAIL (2025-07-09 21:54:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== SegmentBegin Jul 9 21:49:40.272144307 (396h39m15.042785642s) [seg_pos={0 0}, upload=317350]: Begin upload segment... Jul 9 21:49:40.272207637 (396h39m15.042848972s) [seg_pos={0 0}, upload=317350]: Done begin upload segment. Jul 9 21:49:40.272228447 (396h39m15.042869772s) [noise=true , address=45.91.116.8:4072 , piece_id=2THWBWLISKLKJFVUAESYPDVBGOSZ2M6AVHMUN2C22S46FYBNE5RA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.272260418 (396h39m15.042901743s) [noise=true , address=104.36.82.20:4451 , piece_id=CVO53ZISRV2ON542CVCHUAUDVZQI2L5KGCRCMZLP4POOBKLZNVFA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.272657718 (396h39m15.043299043s) [noise=true , address=104.36.82.34:4564 , piece_id=CXLC3UUVVIGKA2OP2BQXE7XH7HNHXOJ3O6BQVUNS5NVY4KRPUWIA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.272906815 (396h39m15.04354815s) [noise=true , address=104.36.82.45:4905 , piece_id=YQWZK3R425VYYPSXLTBCNZQUN2IPWYZTNU7NNR5EW5ONIOH7BG3A, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.273256314 (396h39m15.043897649s) [noise=true , address=45.91.95.3:4021 , piece_id=W7TAKLK7PEUHTITB74T7HJGOU6WTBPRTFHPQF6VTVM4OTPNYTRWA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.273291210 (396h39m15.043932535s) [noise=true , address=45.91.95.2:4017 , piece_id=W52X6YGCDZBSRQEB63HLCQV6CN5JL6JYC2SML5QIRBO5W7CIKDRQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.273553742 (396h39m15.044195077s) [noise=true , address=79.127.219.38:4022 , piece_id=72QFHVOHWUNOSLO65KGIXMA7BYR4DTRPVLVAD4EHHZKXX3GAQMFQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.273793651 (396h39m15.044434986s) [noise=true , address=104.36.82.44:4885 , piece_id=OZVJDJJLCK46MBKO6TGGHQFSW2QUDQ6GDUHTBZLI7Y5LWBXVPJCQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.273834831 (396h39m15.044476156s) [noise=true , address=77.90.31.11:4573 , piece_id=G76LQFRBIKPUKF7CI7DXGL6JRYJOOZJ6SLPAM4Z64CXYJVPW42DA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.273869627 (396h39m15.044510962s) [noise=true , address=77.90.31.18:4827 , piece_id=MQJMN3CW2AEP5T74I7WUUBIHFZOLMMKCIN7IOBWZT2VYQARYLIIQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.274155423 (396h39m15.044796758s) [noise=true , address=77.90.31.19:4876 , piece_id=ROM6VJDIDOEUEGP3PMEHKEDM5POGHMFNKPYRC3LCIAWIVRUGVBUQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.274314117 (396h39m15.044955452s) [noise=true , address=79.127.220.97:4371 , piece_id=XNPEHYQWPTFHLU5TJEO7HZC3MLOLE26GQWI4PI2CUCIL6QTAO7TQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.274414660 (396h39m15.045055995s) [noise=true , address=45.91.95.5:4006 , piece_id=JGKTT6OVQ7T2C22UOWG5HPQ67DBF3NPA3OAMALDLQGATY5LT63PA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.274653466 (396h39m15.045294791s) [noise=true , address=77.90.31.10:4528 , piece_id=XA7ZR76AK6TYNZDCMKPWNMZELNRGXFUU53L34IFMDWRB4LN3W4RA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.274681149 (396h39m15.045322484s) [noise=true , address=45.91.95.4:4021 , piece_id=PFQ3VSEQPTNB4DERVNGT4BCSMZGY3C3OWWJJHVLQTTOCB6TWCMIA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.274898425 (396h39m15.04553976s) [noise=true , address=77.90.31.21:4903 , piece_id=LYVWE3A5IAAEZNHB6DZ63XGBH2UQRJRYMJDMRBOJ2BIN3WESS6RQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.275008706 (396h39m15.045650041s) [noise=true , address=77.90.31.13:4652 , piece_id=IVLFU33BFOUXWJEYUUS2AEOZLAVTOKDMEK7RDPEBPOWF34SWNR6Q, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.275332215 (396h39m15.04597355s) [noise=true , address=45.91.116.7:4041 , piece_id=5JXKDZGV2EPLVSDUVP5LDFOGQ66FXPR2GXYEBFF5KWQXUPJNKPMA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:41.657764345 (396h39m16.42840568s) [upload=317350]: Next returned nil segment Jul 9 21:49:41.657771018 (396h39m16.428412353s) [upload=317350]: Waiting for error group managing segments... Jul 9 21:49:41.690644980 (396h39m16.461286315s) [noise=true , address=104.36.82.45:4905 , piece_id=YQWZK3R425VYYPSXLTBCNZQUN2IPWYZTNU7NNR5EW5ONIOH7BG3A, seg_pos={0 0}, upload=317350]: Done uploading piece. err: ecclient: upload failed (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, address:104.36.82.45:4905): protocol: untrusted: invalid order limit signature: rpcpeer: drpc transport does not have required methods Jul 9 21:49:42.092098451 (396h39m16.862739786s) [noise=true , address=79.127.220.97:4371 , piece_id=XNPEHYQWPTFHLU5TJEO7HZC3MLOLE26GQWI4PI2CUCIL6QTAO7TQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:42.266002837 (396h39m17.036644172s) [noise=true , address=77.90.31.19:4876 , piece_id=ROM6VJDIDOEUEGP3PMEHKEDM5POGHMFNKPYRC3LCIAWIVRUGVBUQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:42.269167210 (396h39m17.039808535s) [noise=true , address=77.90.31.13:4652 , piece_id=IVLFU33BFOUXWJEYUUS2AEOZLAVTOKDMEK7RDPEBPOWF34SWNR6Q, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:42.278551023 (396h39m17.049192358s) [noise=true , address=77.90.31.10:4528 , piece_id=XA7ZR76AK6TYNZDCMKPWNMZELNRGXFUU53L34IFMDWRB4LN3W4RA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:42.280070640 (396h39m17.050711975s) [noise=true , address=77.90.31.21:4903 , piece_id=LYVWE3A5IAAEZNHB6DZ63XGBH2UQRJRYMJDMRBOJ2BIN3WESS6RQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:42.296867786 (396h39m17.067509111s) [noise=true , address=77.90.31.11:4573 , piece_id=G76LQFRBIKPUKF7CI7DXGL6JRYJOOZJ6SLPAM4Z64CXYJVPW42DA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:42.303116681 (396h39m17.073758016s) [noise=true , address=77.90.31.18:4827 , piece_id=MQJMN3CW2AEP5T74I7WUUBIHFZOLMMKCIN7IOBWZT2VYQARYLIIQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:44.408881294 (396h39m19.179522629s) [noise=true , address=45.91.116.7:4041 , piece_id=5JXKDZGV2EPLVSDUVP5LDFOGQ66FXPR2GXYEBFF5KWQXUPJNKPMA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:44.871597160 (396h39m19.642238495s) [noise=true , address=45.91.95.5:4006 , piece_id=JGKTT6OVQ7T2C22UOWG5HPQ67DBF3NPA3OAMALDLQGATY5LT63PA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: ecclient: upload failed (node:12ViW64JDJxzvRU39MsLp5JZKAmHZmxCAJuLVGi2E1E1r9gfc9A, address:45.91.95.5:4006): protocol: storage node overloaded, request limit: 2000 Jul 9 21:49:51.702994736 (396h39m26.473636071s) [noise=true , address=45.91.116.8:4072 , piece_id=2THWBWLISKLKJFVUAESYPDVBGOSZ2M6AVHMUN2C22S46FYBNE5RA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:55.468073925 (396h39m30.23871526s) [noise=true , address=79.127.219.38:4022 , piece_id=72QFHVOHWUNOSLO65KGIXMA7BYR4DTRPVLVAD4EHHZKXX3GAQMFQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:50:22.315419748 (396h39m57.086061083s) [noise=true , address=104.36.82.34:4564 , piece_id=CXLC3UUVVIGKA2OP2BQXE7XH7HNHXOJ3O6BQVUNS5NVY4KRPUWIA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:50:32.620803316 (396h40m7.391444651s) [noise=true , address=104.36.82.20:4451 , piece_id=CVO53ZISRV2ON542CVCHUAUDVZQI2L5KGCRCMZLP4POOBKLZNVFA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:50:39.176702995 (396h40m13.94734433s) [noise=true , address=45.91.95.4:4021 , piece_id=PFQ3VSEQPTNB4DERVNGT4BCSMZGY3C3OWWJJHVLQTTOCB6TWCMIA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: ecclient: upload failed (node:1dXfFE3nY9ZSEixFePGxdptsrybf38kYjYgV61MzV8kHtc12w, address:45.91.95.4:4021): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:50:39.176724405 (396h40m13.94736573s) [noise=true , address=104.36.82.44:4885 , piece_id=OZVJDJJLCK46MBKO6TGGHQFSW2QUDQ6GDUHTBZLI7Y5LWBXVPJCQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:104.36.82.44:4885): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:50:39.176800822 (396h40m13.947442157s) [noise=true , address=45.91.95.3:4021 , piece_id=W7TAKLK7PEUHTITB74T7HJGOU6WTBPRTFHPQF6VTVM4OTPNYTRWA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: ecclient: upload failed (node:131LBDdhw5s1dsLF2CnAPWaL3Amwm4kvfjyqfTvjkrHSQUjJ2q, address:45.91.95.3:4021): protocol: write tcp 188.166.225.170:52410->45.91.95.3:4021: use of closed network connection; write tcp 188.166.225.170:52410->45.91.95.3:4021: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52410->45.91.95.3:4021: use of closed network connection Jul 9 21:50:39.176914419 (396h40m13.947555744s) [noise=true , address=45.91.95.2:4017 , piece_id=W52X6YGCDZBSRQEB63HLCQV6CN5JL6JYC2SML5QIRBO5W7CIKDRQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: ecclient: upload failed (node:1vtusfmiftx4cbYbLEzPEpgvmaGQAD9wsJGBCk8r4f1MkY1RL6, address:45.91.95.2:4017): protocol: write tcp 188.166.225.170:33416->45.91.95.2:4017: use of closed network connection; write tcp 188.166.225.170:33416->45.91.95.2:4017: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33416->45.91.95.2:4017: use of closed network connection Jul 9 21:50:39.176950669 (396h40m13.947591994s) [seg_pos={0 0}, upload=317350]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 21:50:39.412544674 (396h40m14.183186009s) [upload=317350]: Done uploading segments... Jul 9 21:50:39.412568239 (396h40m14.183209574s) [upload=317350]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 33.865931416s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.X....X..X.........X...............X..X....X........X.......] duration: 30.105733304s FAIL (2025-07-09 21:54:11 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.105744105s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [............X....X...............X..X.X...X...X.......X.....] duration: 24.175672911s FAIL (2025-07-09 21:54:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== SegmentBegin Jul 9 21:49:40.272144307 (396h39m15.042785642s) [seg_pos={0 0}, upload=317350]: Begin upload segment... Jul 9 21:49:40.272207637 (396h39m15.042848972s) [seg_pos={0 0}, upload=317350]: Done begin upload segment. Jul 9 21:49:40.272228447 (396h39m15.042869772s) [noise=true , address=45.91.116.8:4072 , piece_id=2THWBWLISKLKJFVUAESYPDVBGOSZ2M6AVHMUN2C22S46FYBNE5RA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.272260418 (396h39m15.042901743s) [noise=true , address=104.36.82.20:4451 , piece_id=CVO53ZISRV2ON542CVCHUAUDVZQI2L5KGCRCMZLP4POOBKLZNVFA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.272657718 (396h39m15.043299043s) [noise=true , address=104.36.82.34:4564 , piece_id=CXLC3UUVVIGKA2OP2BQXE7XH7HNHXOJ3O6BQVUNS5NVY4KRPUWIA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.272906815 (396h39m15.04354815s) [noise=true , address=104.36.82.45:4905 , piece_id=YQWZK3R425VYYPSXLTBCNZQUN2IPWYZTNU7NNR5EW5ONIOH7BG3A, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.273256314 (396h39m15.043897649s) [noise=true , address=45.91.95.3:4021 , piece_id=W7TAKLK7PEUHTITB74T7HJGOU6WTBPRTFHPQF6VTVM4OTPNYTRWA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.273291210 (396h39m15.043932535s) [noise=true , address=45.91.95.2:4017 , piece_id=W52X6YGCDZBSRQEB63HLCQV6CN5JL6JYC2SML5QIRBO5W7CIKDRQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.273553742 (396h39m15.044195077s) [noise=true , address=79.127.219.38:4022 , piece_id=72QFHVOHWUNOSLO65KGIXMA7BYR4DTRPVLVAD4EHHZKXX3GAQMFQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.273793651 (396h39m15.044434986s) [noise=true , address=104.36.82.44:4885 , piece_id=OZVJDJJLCK46MBKO6TGGHQFSW2QUDQ6GDUHTBZLI7Y5LWBXVPJCQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.273834831 (396h39m15.044476156s) [noise=true , address=77.90.31.11:4573 , piece_id=G76LQFRBIKPUKF7CI7DXGL6JRYJOOZJ6SLPAM4Z64CXYJVPW42DA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.273869627 (396h39m15.044510962s) [noise=true , address=77.90.31.18:4827 , piece_id=MQJMN3CW2AEP5T74I7WUUBIHFZOLMMKCIN7IOBWZT2VYQARYLIIQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.274155423 (396h39m15.044796758s) [noise=true , address=77.90.31.19:4876 , piece_id=ROM6VJDIDOEUEGP3PMEHKEDM5POGHMFNKPYRC3LCIAWIVRUGVBUQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.274314117 (396h39m15.044955452s) [noise=true , address=79.127.220.97:4371 , piece_id=XNPEHYQWPTFHLU5TJEO7HZC3MLOLE26GQWI4PI2CUCIL6QTAO7TQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.274414660 (396h39m15.045055995s) [noise=true , address=45.91.95.5:4006 , piece_id=JGKTT6OVQ7T2C22UOWG5HPQ67DBF3NPA3OAMALDLQGATY5LT63PA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.274653466 (396h39m15.045294791s) [noise=true , address=77.90.31.10:4528 , piece_id=XA7ZR76AK6TYNZDCMKPWNMZELNRGXFUU53L34IFMDWRB4LN3W4RA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.274681149 (396h39m15.045322484s) [noise=true , address=45.91.95.4:4021 , piece_id=PFQ3VSEQPTNB4DERVNGT4BCSMZGY3C3OWWJJHVLQTTOCB6TWCMIA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.274898425 (396h39m15.04553976s) [noise=true , address=77.90.31.21:4903 , piece_id=LYVWE3A5IAAEZNHB6DZ63XGBH2UQRJRYMJDMRBOJ2BIN3WESS6RQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.275008706 (396h39m15.045650041s) [noise=true , address=77.90.31.13:4652 , piece_id=IVLFU33BFOUXWJEYUUS2AEOZLAVTOKDMEK7RDPEBPOWF34SWNR6Q, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.275332215 (396h39m15.04597355s) [noise=true , address=45.91.116.7:4041 , piece_id=5JXKDZGV2EPLVSDUVP5LDFOGQ66FXPR2GXYEBFF5KWQXUPJNKPMA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:41.657764345 (396h39m16.42840568s) [upload=317350]: Next returned nil segment Jul 9 21:49:41.657771018 (396h39m16.428412353s) [upload=317350]: Waiting for error group managing segments... Jul 9 21:49:41.690644980 (396h39m16.461286315s) [noise=true , address=104.36.82.45:4905 , piece_id=YQWZK3R425VYYPSXLTBCNZQUN2IPWYZTNU7NNR5EW5ONIOH7BG3A, seg_pos={0 0}, upload=317350]: Done uploading piece. err: ecclient: upload failed (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, address:104.36.82.45:4905): protocol: untrusted: invalid order limit signature: rpcpeer: drpc transport does not have required methods Jul 9 21:49:42.092098451 (396h39m16.862739786s) [noise=true , address=79.127.220.97:4371 , piece_id=XNPEHYQWPTFHLU5TJEO7HZC3MLOLE26GQWI4PI2CUCIL6QTAO7TQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:42.266002837 (396h39m17.036644172s) [noise=true , address=77.90.31.19:4876 , piece_id=ROM6VJDIDOEUEGP3PMEHKEDM5POGHMFNKPYRC3LCIAWIVRUGVBUQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:42.269167210 (396h39m17.039808535s) [noise=true , address=77.90.31.13:4652 , piece_id=IVLFU33BFOUXWJEYUUS2AEOZLAVTOKDMEK7RDPEBPOWF34SWNR6Q, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:42.278551023 (396h39m17.049192358s) [noise=true , address=77.90.31.10:4528 , piece_id=XA7ZR76AK6TYNZDCMKPWNMZELNRGXFUU53L34IFMDWRB4LN3W4RA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:42.280070640 (396h39m17.050711975s) [noise=true , address=77.90.31.21:4903 , piece_id=LYVWE3A5IAAEZNHB6DZ63XGBH2UQRJRYMJDMRBOJ2BIN3WESS6RQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:42.296867786 (396h39m17.067509111s) [noise=true , address=77.90.31.11:4573 , piece_id=G76LQFRBIKPUKF7CI7DXGL6JRYJOOZJ6SLPAM4Z64CXYJVPW42DA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:42.303116681 (396h39m17.073758016s) [noise=true , address=77.90.31.18:4827 , piece_id=MQJMN3CW2AEP5T74I7WUUBIHFZOLMMKCIN7IOBWZT2VYQARYLIIQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:44.408881294 (396h39m19.179522629s) [noise=true , address=45.91.116.7:4041 , piece_id=5JXKDZGV2EPLVSDUVP5LDFOGQ66FXPR2GXYEBFF5KWQXUPJNKPMA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:44.871597160 (396h39m19.642238495s) [noise=true , address=45.91.95.5:4006 , piece_id=JGKTT6OVQ7T2C22UOWG5HPQ67DBF3NPA3OAMALDLQGATY5LT63PA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: ecclient: upload failed (node:12ViW64JDJxzvRU39MsLp5JZKAmHZmxCAJuLVGi2E1E1r9gfc9A, address:45.91.95.5:4006): protocol: storage node overloaded, request limit: 2000 Jul 9 21:49:51.702994736 (396h39m26.473636071s) [noise=true , address=45.91.116.8:4072 , piece_id=2THWBWLISKLKJFVUAESYPDVBGOSZ2M6AVHMUN2C22S46FYBNE5RA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:55.468073925 (396h39m30.23871526s) [noise=true , address=79.127.219.38:4022 , piece_id=72QFHVOHWUNOSLO65KGIXMA7BYR4DTRPVLVAD4EHHZKXX3GAQMFQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:50:22.315419748 (396h39m57.086061083s) [noise=true , address=104.36.82.34:4564 , piece_id=CXLC3UUVVIGKA2OP2BQXE7XH7HNHXOJ3O6BQVUNS5NVY4KRPUWIA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:50:32.620803316 (396h40m7.391444651s) [noise=true , address=104.36.82.20:4451 , piece_id=CVO53ZISRV2ON542CVCHUAUDVZQI2L5KGCRCMZLP4POOBKLZNVFA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:50:39.176702995 (396h40m13.94734433s) [noise=true , address=45.91.95.4:4021 , piece_id=PFQ3VSEQPTNB4DERVNGT4BCSMZGY3C3OWWJJHVLQTTOCB6TWCMIA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: ecclient: upload failed (node:1dXfFE3nY9ZSEixFePGxdptsrybf38kYjYgV61MzV8kHtc12w, address:45.91.95.4:4021): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:50:39.176724405 (396h40m13.94736573s) [noise=true , address=104.36.82.44:4885 , piece_id=OZVJDJJLCK46MBKO6TGGHQFSW2QUDQ6GDUHTBZLI7Y5LWBXVPJCQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:104.36.82.44:4885): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:50:39.176800822 (396h40m13.947442157s) [noise=true , address=45.91.95.3:4021 , piece_id=W7TAKLK7PEUHTITB74T7HJGOU6WTBPRTFHPQF6VTVM4OTPNYTRWA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: ecclient: upload failed (node:131LBDdhw5s1dsLF2CnAPWaL3Amwm4kvfjyqfTvjkrHSQUjJ2q, address:45.91.95.3:4021): protocol: write tcp 188.166.225.170:52410->45.91.95.3:4021: use of closed network connection; write tcp 188.166.225.170:52410->45.91.95.3:4021: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52410->45.91.95.3:4021: use of closed network connection Jul 9 21:50:39.176914419 (396h40m13.947555744s) [noise=true , address=45.91.95.2:4017 , piece_id=W52X6YGCDZBSRQEB63HLCQV6CN5JL6JYC2SML5QIRBO5W7CIKDRQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: ecclient: upload failed (node:1vtusfmiftx4cbYbLEzPEpgvmaGQAD9wsJGBCk8r4f1MkY1RL6, address:45.91.95.2:4017): protocol: write tcp 188.166.225.170:33416->45.91.95.2:4017: use of closed network connection; write tcp 188.166.225.170:33416->45.91.95.2:4017: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33416->45.91.95.2:4017: use of closed network connection Jul 9 21:50:39.176950669 (396h40m13.947591994s) [seg_pos={0 0}, upload=317350]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 21:50:39.412544674 (396h40m14.183186009s) [upload=317350]: Done uploading segments... Jul 9 21:50:39.412568239 (396h40m14.183209574s) [upload=317350]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 24.175691396s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.............X....X...............X..X.X...X...X.......X....] duration: 49.539210983s FAIL (2025-07-09 21:53:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== SegmentBegin Jul 9 21:49:40.272144307 (396h39m15.042785642s) [seg_pos={0 0}, upload=317350]: Begin upload segment... Jul 9 21:49:40.272207637 (396h39m15.042848972s) [seg_pos={0 0}, upload=317350]: Done begin upload segment. Jul 9 21:49:40.272228447 (396h39m15.042869772s) [noise=true , address=45.91.116.8:4072 , piece_id=2THWBWLISKLKJFVUAESYPDVBGOSZ2M6AVHMUN2C22S46FYBNE5RA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.272260418 (396h39m15.042901743s) [noise=true , address=104.36.82.20:4451 , piece_id=CVO53ZISRV2ON542CVCHUAUDVZQI2L5KGCRCMZLP4POOBKLZNVFA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.272657718 (396h39m15.043299043s) [noise=true , address=104.36.82.34:4564 , piece_id=CXLC3UUVVIGKA2OP2BQXE7XH7HNHXOJ3O6BQVUNS5NVY4KRPUWIA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.272906815 (396h39m15.04354815s) [noise=true , address=104.36.82.45:4905 , piece_id=YQWZK3R425VYYPSXLTBCNZQUN2IPWYZTNU7NNR5EW5ONIOH7BG3A, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.273256314 (396h39m15.043897649s) [noise=true , address=45.91.95.3:4021 , piece_id=W7TAKLK7PEUHTITB74T7HJGOU6WTBPRTFHPQF6VTVM4OTPNYTRWA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.273291210 (396h39m15.043932535s) [noise=true , address=45.91.95.2:4017 , piece_id=W52X6YGCDZBSRQEB63HLCQV6CN5JL6JYC2SML5QIRBO5W7CIKDRQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.273553742 (396h39m15.044195077s) [noise=true , address=79.127.219.38:4022 , piece_id=72QFHVOHWUNOSLO65KGIXMA7BYR4DTRPVLVAD4EHHZKXX3GAQMFQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.273793651 (396h39m15.044434986s) [noise=true , address=104.36.82.44:4885 , piece_id=OZVJDJJLCK46MBKO6TGGHQFSW2QUDQ6GDUHTBZLI7Y5LWBXVPJCQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.273834831 (396h39m15.044476156s) [noise=true , address=77.90.31.11:4573 , piece_id=G76LQFRBIKPUKF7CI7DXGL6JRYJOOZJ6SLPAM4Z64CXYJVPW42DA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.273869627 (396h39m15.044510962s) [noise=true , address=77.90.31.18:4827 , piece_id=MQJMN3CW2AEP5T74I7WUUBIHFZOLMMKCIN7IOBWZT2VYQARYLIIQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.274155423 (396h39m15.044796758s) [noise=true , address=77.90.31.19:4876 , piece_id=ROM6VJDIDOEUEGP3PMEHKEDM5POGHMFNKPYRC3LCIAWIVRUGVBUQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.274314117 (396h39m15.044955452s) [noise=true , address=79.127.220.97:4371 , piece_id=XNPEHYQWPTFHLU5TJEO7HZC3MLOLE26GQWI4PI2CUCIL6QTAO7TQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.274414660 (396h39m15.045055995s) [noise=true , address=45.91.95.5:4006 , piece_id=JGKTT6OVQ7T2C22UOWG5HPQ67DBF3NPA3OAMALDLQGATY5LT63PA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.274653466 (396h39m15.045294791s) [noise=true , address=77.90.31.10:4528 , piece_id=XA7ZR76AK6TYNZDCMKPWNMZELNRGXFUU53L34IFMDWRB4LN3W4RA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.274681149 (396h39m15.045322484s) [noise=true , address=45.91.95.4:4021 , piece_id=PFQ3VSEQPTNB4DERVNGT4BCSMZGY3C3OWWJJHVLQTTOCB6TWCMIA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.274898425 (396h39m15.04553976s) [noise=true , address=77.90.31.21:4903 , piece_id=LYVWE3A5IAAEZNHB6DZ63XGBH2UQRJRYMJDMRBOJ2BIN3WESS6RQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.275008706 (396h39m15.045650041s) [noise=true , address=77.90.31.13:4652 , piece_id=IVLFU33BFOUXWJEYUUS2AEOZLAVTOKDMEK7RDPEBPOWF34SWNR6Q, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.275332215 (396h39m15.04597355s) [noise=true , address=45.91.116.7:4041 , piece_id=5JXKDZGV2EPLVSDUVP5LDFOGQ66FXPR2GXYEBFF5KWQXUPJNKPMA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:41.657764345 (396h39m16.42840568s) [upload=317350]: Next returned nil segment Jul 9 21:49:41.657771018 (396h39m16.428412353s) [upload=317350]: Waiting for error group managing segments... Jul 9 21:49:41.690644980 (396h39m16.461286315s) [noise=true , address=104.36.82.45:4905 , piece_id=YQWZK3R425VYYPSXLTBCNZQUN2IPWYZTNU7NNR5EW5ONIOH7BG3A, seg_pos={0 0}, upload=317350]: Done uploading piece. err: ecclient: upload failed (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, address:104.36.82.45:4905): protocol: untrusted: invalid order limit signature: rpcpeer: drpc transport does not have required methods Jul 9 21:49:42.092098451 (396h39m16.862739786s) [noise=true , address=79.127.220.97:4371 , piece_id=XNPEHYQWPTFHLU5TJEO7HZC3MLOLE26GQWI4PI2CUCIL6QTAO7TQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:42.266002837 (396h39m17.036644172s) [noise=true , address=77.90.31.19:4876 , piece_id=ROM6VJDIDOEUEGP3PMEHKEDM5POGHMFNKPYRC3LCIAWIVRUGVBUQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:42.269167210 (396h39m17.039808535s) [noise=true , address=77.90.31.13:4652 , piece_id=IVLFU33BFOUXWJEYUUS2AEOZLAVTOKDMEK7RDPEBPOWF34SWNR6Q, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:42.278551023 (396h39m17.049192358s) [noise=true , address=77.90.31.10:4528 , piece_id=XA7ZR76AK6TYNZDCMKPWNMZELNRGXFUU53L34IFMDWRB4LN3W4RA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:42.280070640 (396h39m17.050711975s) [noise=true , address=77.90.31.21:4903 , piece_id=LYVWE3A5IAAEZNHB6DZ63XGBH2UQRJRYMJDMRBOJ2BIN3WESS6RQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:42.296867786 (396h39m17.067509111s) [noise=true , address=77.90.31.11:4573 , piece_id=G76LQFRBIKPUKF7CI7DXGL6JRYJOOZJ6SLPAM4Z64CXYJVPW42DA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:42.303116681 (396h39m17.073758016s) [noise=true , address=77.90.31.18:4827 , piece_id=MQJMN3CW2AEP5T74I7WUUBIHFZOLMMKCIN7IOBWZT2VYQARYLIIQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:44.408881294 (396h39m19.179522629s) [noise=true , address=45.91.116.7:4041 , piece_id=5JXKDZGV2EPLVSDUVP5LDFOGQ66FXPR2GXYEBFF5KWQXUPJNKPMA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:44.871597160 (396h39m19.642238495s) [noise=true , address=45.91.95.5:4006 , piece_id=JGKTT6OVQ7T2C22UOWG5HPQ67DBF3NPA3OAMALDLQGATY5LT63PA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: ecclient: upload failed (node:12ViW64JDJxzvRU39MsLp5JZKAmHZmxCAJuLVGi2E1E1r9gfc9A, address:45.91.95.5:4006): protocol: storage node overloaded, request limit: 2000 Jul 9 21:49:51.702994736 (396h39m26.473636071s) [noise=true , address=45.91.116.8:4072 , piece_id=2THWBWLISKLKJFVUAESYPDVBGOSZ2M6AVHMUN2C22S46FYBNE5RA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:55.468073925 (396h39m30.23871526s) [noise=true , address=79.127.219.38:4022 , piece_id=72QFHVOHWUNOSLO65KGIXMA7BYR4DTRPVLVAD4EHHZKXX3GAQMFQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:50:22.315419748 (396h39m57.086061083s) [noise=true , address=104.36.82.34:4564 , piece_id=CXLC3UUVVIGKA2OP2BQXE7XH7HNHXOJ3O6BQVUNS5NVY4KRPUWIA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:50:32.620803316 (396h40m7.391444651s) [noise=true , address=104.36.82.20:4451 , piece_id=CVO53ZISRV2ON542CVCHUAUDVZQI2L5KGCRCMZLP4POOBKLZNVFA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:50:39.176702995 (396h40m13.94734433s) [noise=true , address=45.91.95.4:4021 , piece_id=PFQ3VSEQPTNB4DERVNGT4BCSMZGY3C3OWWJJHVLQTTOCB6TWCMIA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: ecclient: upload failed (node:1dXfFE3nY9ZSEixFePGxdptsrybf38kYjYgV61MzV8kHtc12w, address:45.91.95.4:4021): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:50:39.176724405 (396h40m13.94736573s) [noise=true , address=104.36.82.44:4885 , piece_id=OZVJDJJLCK46MBKO6TGGHQFSW2QUDQ6GDUHTBZLI7Y5LWBXVPJCQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:104.36.82.44:4885): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:50:39.176800822 (396h40m13.947442157s) [noise=true , address=45.91.95.3:4021 , piece_id=W7TAKLK7PEUHTITB74T7HJGOU6WTBPRTFHPQF6VTVM4OTPNYTRWA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: ecclient: upload failed (node:131LBDdhw5s1dsLF2CnAPWaL3Amwm4kvfjyqfTvjkrHSQUjJ2q, address:45.91.95.3:4021): protocol: write tcp 188.166.225.170:52410->45.91.95.3:4021: use of closed network connection; write tcp 188.166.225.170:52410->45.91.95.3:4021: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52410->45.91.95.3:4021: use of closed network connection Jul 9 21:50:39.176914419 (396h40m13.947555744s) [noise=true , address=45.91.95.2:4017 , piece_id=W52X6YGCDZBSRQEB63HLCQV6CN5JL6JYC2SML5QIRBO5W7CIKDRQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: ecclient: upload failed (node:1vtusfmiftx4cbYbLEzPEpgvmaGQAD9wsJGBCk8r4f1MkY1RL6, address:45.91.95.2:4017): protocol: write tcp 188.166.225.170:33416->45.91.95.2:4017: use of closed network connection; write tcp 188.166.225.170:33416->45.91.95.2:4017: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33416->45.91.95.2:4017: use of closed network connection Jul 9 21:50:39.176950669 (396h40m13.947591994s) [seg_pos={0 0}, upload=317350]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 21:50:39.412544674 (396h40m14.183186009s) [upload=317350]: Done uploading segments... Jul 9 21:50:39.412568239 (396h40m14.183209574s) [upload=317350]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 49.539231101s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [..X....X..X.........X...............X..X....X........X......] duration: 26.359940723s FAIL (2025-07-09 21:53:40 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 26.359955571s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [...X....X..X.........X...............X..X....X........X.....] duration: 26.457219993s FAIL (2025-07-09 21:52:48 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 26.457232657s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [..............X....X...............X..X.X...X...X.......X...] duration: 19.11442128s FAIL (2025-07-09 21:52:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== SegmentBegin Jul 9 21:49:40.272144307 (396h39m15.042785642s) [seg_pos={0 0}, upload=317350]: Begin upload segment... Jul 9 21:49:40.272207637 (396h39m15.042848972s) [seg_pos={0 0}, upload=317350]: Done begin upload segment. Jul 9 21:49:40.272228447 (396h39m15.042869772s) [noise=true , address=45.91.116.8:4072 , piece_id=2THWBWLISKLKJFVUAESYPDVBGOSZ2M6AVHMUN2C22S46FYBNE5RA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.272260418 (396h39m15.042901743s) [noise=true , address=104.36.82.20:4451 , piece_id=CVO53ZISRV2ON542CVCHUAUDVZQI2L5KGCRCMZLP4POOBKLZNVFA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.272657718 (396h39m15.043299043s) [noise=true , address=104.36.82.34:4564 , piece_id=CXLC3UUVVIGKA2OP2BQXE7XH7HNHXOJ3O6BQVUNS5NVY4KRPUWIA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.272906815 (396h39m15.04354815s) [noise=true , address=104.36.82.45:4905 , piece_id=YQWZK3R425VYYPSXLTBCNZQUN2IPWYZTNU7NNR5EW5ONIOH7BG3A, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.273256314 (396h39m15.043897649s) [noise=true , address=45.91.95.3:4021 , piece_id=W7TAKLK7PEUHTITB74T7HJGOU6WTBPRTFHPQF6VTVM4OTPNYTRWA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.273291210 (396h39m15.043932535s) [noise=true , address=45.91.95.2:4017 , piece_id=W52X6YGCDZBSRQEB63HLCQV6CN5JL6JYC2SML5QIRBO5W7CIKDRQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.273553742 (396h39m15.044195077s) [noise=true , address=79.127.219.38:4022 , piece_id=72QFHVOHWUNOSLO65KGIXMA7BYR4DTRPVLVAD4EHHZKXX3GAQMFQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.273793651 (396h39m15.044434986s) [noise=true , address=104.36.82.44:4885 , piece_id=OZVJDJJLCK46MBKO6TGGHQFSW2QUDQ6GDUHTBZLI7Y5LWBXVPJCQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.273834831 (396h39m15.044476156s) [noise=true , address=77.90.31.11:4573 , piece_id=G76LQFRBIKPUKF7CI7DXGL6JRYJOOZJ6SLPAM4Z64CXYJVPW42DA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.273869627 (396h39m15.044510962s) [noise=true , address=77.90.31.18:4827 , piece_id=MQJMN3CW2AEP5T74I7WUUBIHFZOLMMKCIN7IOBWZT2VYQARYLIIQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.274155423 (396h39m15.044796758s) [noise=true , address=77.90.31.19:4876 , piece_id=ROM6VJDIDOEUEGP3PMEHKEDM5POGHMFNKPYRC3LCIAWIVRUGVBUQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.274314117 (396h39m15.044955452s) [noise=true , address=79.127.220.97:4371 , piece_id=XNPEHYQWPTFHLU5TJEO7HZC3MLOLE26GQWI4PI2CUCIL6QTAO7TQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.274414660 (396h39m15.045055995s) [noise=true , address=45.91.95.5:4006 , piece_id=JGKTT6OVQ7T2C22UOWG5HPQ67DBF3NPA3OAMALDLQGATY5LT63PA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.274653466 (396h39m15.045294791s) [noise=true , address=77.90.31.10:4528 , piece_id=XA7ZR76AK6TYNZDCMKPWNMZELNRGXFUU53L34IFMDWRB4LN3W4RA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.274681149 (396h39m15.045322484s) [noise=true , address=45.91.95.4:4021 , piece_id=PFQ3VSEQPTNB4DERVNGT4BCSMZGY3C3OWWJJHVLQTTOCB6TWCMIA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.274898425 (396h39m15.04553976s) [noise=true , address=77.90.31.21:4903 , piece_id=LYVWE3A5IAAEZNHB6DZ63XGBH2UQRJRYMJDMRBOJ2BIN3WESS6RQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.275008706 (396h39m15.045650041s) [noise=true , address=77.90.31.13:4652 , piece_id=IVLFU33BFOUXWJEYUUS2AEOZLAVTOKDMEK7RDPEBPOWF34SWNR6Q, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.275332215 (396h39m15.04597355s) [noise=true , address=45.91.116.7:4041 , piece_id=5JXKDZGV2EPLVSDUVP5LDFOGQ66FXPR2GXYEBFF5KWQXUPJNKPMA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:41.657764345 (396h39m16.42840568s) [upload=317350]: Next returned nil segment Jul 9 21:49:41.657771018 (396h39m16.428412353s) [upload=317350]: Waiting for error group managing segments... Jul 9 21:49:41.690644980 (396h39m16.461286315s) [noise=true , address=104.36.82.45:4905 , piece_id=YQWZK3R425VYYPSXLTBCNZQUN2IPWYZTNU7NNR5EW5ONIOH7BG3A, seg_pos={0 0}, upload=317350]: Done uploading piece. err: ecclient: upload failed (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, address:104.36.82.45:4905): protocol: untrusted: invalid order limit signature: rpcpeer: drpc transport does not have required methods Jul 9 21:49:42.092098451 (396h39m16.862739786s) [noise=true , address=79.127.220.97:4371 , piece_id=XNPEHYQWPTFHLU5TJEO7HZC3MLOLE26GQWI4PI2CUCIL6QTAO7TQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:42.266002837 (396h39m17.036644172s) [noise=true , address=77.90.31.19:4876 , piece_id=ROM6VJDIDOEUEGP3PMEHKEDM5POGHMFNKPYRC3LCIAWIVRUGVBUQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:42.269167210 (396h39m17.039808535s) [noise=true , address=77.90.31.13:4652 , piece_id=IVLFU33BFOUXWJEYUUS2AEOZLAVTOKDMEK7RDPEBPOWF34SWNR6Q, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:42.278551023 (396h39m17.049192358s) [noise=true , address=77.90.31.10:4528 , piece_id=XA7ZR76AK6TYNZDCMKPWNMZELNRGXFUU53L34IFMDWRB4LN3W4RA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:42.280070640 (396h39m17.050711975s) [noise=true , address=77.90.31.21:4903 , piece_id=LYVWE3A5IAAEZNHB6DZ63XGBH2UQRJRYMJDMRBOJ2BIN3WESS6RQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:42.296867786 (396h39m17.067509111s) [noise=true , address=77.90.31.11:4573 , piece_id=G76LQFRBIKPUKF7CI7DXGL6JRYJOOZJ6SLPAM4Z64CXYJVPW42DA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:42.303116681 (396h39m17.073758016s) [noise=true , address=77.90.31.18:4827 , piece_id=MQJMN3CW2AEP5T74I7WUUBIHFZOLMMKCIN7IOBWZT2VYQARYLIIQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:44.408881294 (396h39m19.179522629s) [noise=true , address=45.91.116.7:4041 , piece_id=5JXKDZGV2EPLVSDUVP5LDFOGQ66FXPR2GXYEBFF5KWQXUPJNKPMA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:44.871597160 (396h39m19.642238495s) [noise=true , address=45.91.95.5:4006 , piece_id=JGKTT6OVQ7T2C22UOWG5HPQ67DBF3NPA3OAMALDLQGATY5LT63PA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: ecclient: upload failed (node:12ViW64JDJxzvRU39MsLp5JZKAmHZmxCAJuLVGi2E1E1r9gfc9A, address:45.91.95.5:4006): protocol: storage node overloaded, request limit: 2000 Jul 9 21:49:51.702994736 (396h39m26.473636071s) [noise=true , address=45.91.116.8:4072 , piece_id=2THWBWLISKLKJFVUAESYPDVBGOSZ2M6AVHMUN2C22S46FYBNE5RA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:55.468073925 (396h39m30.23871526s) [noise=true , address=79.127.219.38:4022 , piece_id=72QFHVOHWUNOSLO65KGIXMA7BYR4DTRPVLVAD4EHHZKXX3GAQMFQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:50:22.315419748 (396h39m57.086061083s) [noise=true , address=104.36.82.34:4564 , piece_id=CXLC3UUVVIGKA2OP2BQXE7XH7HNHXOJ3O6BQVUNS5NVY4KRPUWIA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:50:32.620803316 (396h40m7.391444651s) [noise=true , address=104.36.82.20:4451 , piece_id=CVO53ZISRV2ON542CVCHUAUDVZQI2L5KGCRCMZLP4POOBKLZNVFA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:50:39.176702995 (396h40m13.94734433s) [noise=true , address=45.91.95.4:4021 , piece_id=PFQ3VSEQPTNB4DERVNGT4BCSMZGY3C3OWWJJHVLQTTOCB6TWCMIA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: ecclient: upload failed (node:1dXfFE3nY9ZSEixFePGxdptsrybf38kYjYgV61MzV8kHtc12w, address:45.91.95.4:4021): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:50:39.176724405 (396h40m13.94736573s) [noise=true , address=104.36.82.44:4885 , piece_id=OZVJDJJLCK46MBKO6TGGHQFSW2QUDQ6GDUHTBZLI7Y5LWBXVPJCQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:104.36.82.44:4885): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:50:39.176800822 (396h40m13.947442157s) [noise=true , address=45.91.95.3:4021 , piece_id=W7TAKLK7PEUHTITB74T7HJGOU6WTBPRTFHPQF6VTVM4OTPNYTRWA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: ecclient: upload failed (node:131LBDdhw5s1dsLF2CnAPWaL3Amwm4kvfjyqfTvjkrHSQUjJ2q, address:45.91.95.3:4021): protocol: write tcp 188.166.225.170:52410->45.91.95.3:4021: use of closed network connection; write tcp 188.166.225.170:52410->45.91.95.3:4021: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52410->45.91.95.3:4021: use of closed network connection Jul 9 21:50:39.176914419 (396h40m13.947555744s) [noise=true , address=45.91.95.2:4017 , piece_id=W52X6YGCDZBSRQEB63HLCQV6CN5JL6JYC2SML5QIRBO5W7CIKDRQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: ecclient: upload failed (node:1vtusfmiftx4cbYbLEzPEpgvmaGQAD9wsJGBCk8r4f1MkY1RL6, address:45.91.95.2:4017): protocol: write tcp 188.166.225.170:33416->45.91.95.2:4017: use of closed network connection; write tcp 188.166.225.170:33416->45.91.95.2:4017: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33416->45.91.95.2:4017: use of closed network connection Jul 9 21:50:39.176950669 (396h40m13.947591994s) [seg_pos={0 0}, upload=317350]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 21:50:39.412544674 (396h40m14.183186009s) [upload=317350]: Done uploading segments... Jul 9 21:50:39.412568239 (396h40m14.183209574s) [upload=317350]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 19.114439965s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [....X....X..X.........X...............X..X....X........X....] duration: 30.999076968s FAIL (2025-07-09 21:52:21 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.999101133s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [...............X....X...............X..X.X...X...X.......X..] duration: 36.667380411s FAIL (2025-07-09 21:52:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== SegmentBegin Jul 9 21:49:40.272144307 (396h39m15.042785642s) [seg_pos={0 0}, upload=317350]: Begin upload segment... Jul 9 21:49:40.272207637 (396h39m15.042848972s) [seg_pos={0 0}, upload=317350]: Done begin upload segment. Jul 9 21:49:40.272228447 (396h39m15.042869772s) [noise=true , address=45.91.116.8:4072 , piece_id=2THWBWLISKLKJFVUAESYPDVBGOSZ2M6AVHMUN2C22S46FYBNE5RA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.272260418 (396h39m15.042901743s) [noise=true , address=104.36.82.20:4451 , piece_id=CVO53ZISRV2ON542CVCHUAUDVZQI2L5KGCRCMZLP4POOBKLZNVFA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.272657718 (396h39m15.043299043s) [noise=true , address=104.36.82.34:4564 , piece_id=CXLC3UUVVIGKA2OP2BQXE7XH7HNHXOJ3O6BQVUNS5NVY4KRPUWIA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.272906815 (396h39m15.04354815s) [noise=true , address=104.36.82.45:4905 , piece_id=YQWZK3R425VYYPSXLTBCNZQUN2IPWYZTNU7NNR5EW5ONIOH7BG3A, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.273256314 (396h39m15.043897649s) [noise=true , address=45.91.95.3:4021 , piece_id=W7TAKLK7PEUHTITB74T7HJGOU6WTBPRTFHPQF6VTVM4OTPNYTRWA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.273291210 (396h39m15.043932535s) [noise=true , address=45.91.95.2:4017 , piece_id=W52X6YGCDZBSRQEB63HLCQV6CN5JL6JYC2SML5QIRBO5W7CIKDRQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.273553742 (396h39m15.044195077s) [noise=true , address=79.127.219.38:4022 , piece_id=72QFHVOHWUNOSLO65KGIXMA7BYR4DTRPVLVAD4EHHZKXX3GAQMFQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.273793651 (396h39m15.044434986s) [noise=true , address=104.36.82.44:4885 , piece_id=OZVJDJJLCK46MBKO6TGGHQFSW2QUDQ6GDUHTBZLI7Y5LWBXVPJCQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.273834831 (396h39m15.044476156s) [noise=true , address=77.90.31.11:4573 , piece_id=G76LQFRBIKPUKF7CI7DXGL6JRYJOOZJ6SLPAM4Z64CXYJVPW42DA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.273869627 (396h39m15.044510962s) [noise=true , address=77.90.31.18:4827 , piece_id=MQJMN3CW2AEP5T74I7WUUBIHFZOLMMKCIN7IOBWZT2VYQARYLIIQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.274155423 (396h39m15.044796758s) [noise=true , address=77.90.31.19:4876 , piece_id=ROM6VJDIDOEUEGP3PMEHKEDM5POGHMFNKPYRC3LCIAWIVRUGVBUQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.274314117 (396h39m15.044955452s) [noise=true , address=79.127.220.97:4371 , piece_id=XNPEHYQWPTFHLU5TJEO7HZC3MLOLE26GQWI4PI2CUCIL6QTAO7TQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.274414660 (396h39m15.045055995s) [noise=true , address=45.91.95.5:4006 , piece_id=JGKTT6OVQ7T2C22UOWG5HPQ67DBF3NPA3OAMALDLQGATY5LT63PA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.274653466 (396h39m15.045294791s) [noise=true , address=77.90.31.10:4528 , piece_id=XA7ZR76AK6TYNZDCMKPWNMZELNRGXFUU53L34IFMDWRB4LN3W4RA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.274681149 (396h39m15.045322484s) [noise=true , address=45.91.95.4:4021 , piece_id=PFQ3VSEQPTNB4DERVNGT4BCSMZGY3C3OWWJJHVLQTTOCB6TWCMIA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.274898425 (396h39m15.04553976s) [noise=true , address=77.90.31.21:4903 , piece_id=LYVWE3A5IAAEZNHB6DZ63XGBH2UQRJRYMJDMRBOJ2BIN3WESS6RQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.275008706 (396h39m15.045650041s) [noise=true , address=77.90.31.13:4652 , piece_id=IVLFU33BFOUXWJEYUUS2AEOZLAVTOKDMEK7RDPEBPOWF34SWNR6Q, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.275332215 (396h39m15.04597355s) [noise=true , address=45.91.116.7:4041 , piece_id=5JXKDZGV2EPLVSDUVP5LDFOGQ66FXPR2GXYEBFF5KWQXUPJNKPMA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:41.657764345 (396h39m16.42840568s) [upload=317350]: Next returned nil segment Jul 9 21:49:41.657771018 (396h39m16.428412353s) [upload=317350]: Waiting for error group managing segments... Jul 9 21:49:41.690644980 (396h39m16.461286315s) [noise=true , address=104.36.82.45:4905 , piece_id=YQWZK3R425VYYPSXLTBCNZQUN2IPWYZTNU7NNR5EW5ONIOH7BG3A, seg_pos={0 0}, upload=317350]: Done uploading piece. err: ecclient: upload failed (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, address:104.36.82.45:4905): protocol: untrusted: invalid order limit signature: rpcpeer: drpc transport does not have required methods Jul 9 21:49:42.092098451 (396h39m16.862739786s) [noise=true , address=79.127.220.97:4371 , piece_id=XNPEHYQWPTFHLU5TJEO7HZC3MLOLE26GQWI4PI2CUCIL6QTAO7TQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:42.266002837 (396h39m17.036644172s) [noise=true , address=77.90.31.19:4876 , piece_id=ROM6VJDIDOEUEGP3PMEHKEDM5POGHMFNKPYRC3LCIAWIVRUGVBUQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:42.269167210 (396h39m17.039808535s) [noise=true , address=77.90.31.13:4652 , piece_id=IVLFU33BFOUXWJEYUUS2AEOZLAVTOKDMEK7RDPEBPOWF34SWNR6Q, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:42.278551023 (396h39m17.049192358s) [noise=true , address=77.90.31.10:4528 , piece_id=XA7ZR76AK6TYNZDCMKPWNMZELNRGXFUU53L34IFMDWRB4LN3W4RA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:42.280070640 (396h39m17.050711975s) [noise=true , address=77.90.31.21:4903 , piece_id=LYVWE3A5IAAEZNHB6DZ63XGBH2UQRJRYMJDMRBOJ2BIN3WESS6RQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:42.296867786 (396h39m17.067509111s) [noise=true , address=77.90.31.11:4573 , piece_id=G76LQFRBIKPUKF7CI7DXGL6JRYJOOZJ6SLPAM4Z64CXYJVPW42DA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:42.303116681 (396h39m17.073758016s) [noise=true , address=77.90.31.18:4827 , piece_id=MQJMN3CW2AEP5T74I7WUUBIHFZOLMMKCIN7IOBWZT2VYQARYLIIQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:44.408881294 (396h39m19.179522629s) [noise=true , address=45.91.116.7:4041 , piece_id=5JXKDZGV2EPLVSDUVP5LDFOGQ66FXPR2GXYEBFF5KWQXUPJNKPMA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:44.871597160 (396h39m19.642238495s) [noise=true , address=45.91.95.5:4006 , piece_id=JGKTT6OVQ7T2C22UOWG5HPQ67DBF3NPA3OAMALDLQGATY5LT63PA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: ecclient: upload failed (node:12ViW64JDJxzvRU39MsLp5JZKAmHZmxCAJuLVGi2E1E1r9gfc9A, address:45.91.95.5:4006): protocol: storage node overloaded, request limit: 2000 Jul 9 21:49:51.702994736 (396h39m26.473636071s) [noise=true , address=45.91.116.8:4072 , piece_id=2THWBWLISKLKJFVUAESYPDVBGOSZ2M6AVHMUN2C22S46FYBNE5RA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:55.468073925 (396h39m30.23871526s) [noise=true , address=79.127.219.38:4022 , piece_id=72QFHVOHWUNOSLO65KGIXMA7BYR4DTRPVLVAD4EHHZKXX3GAQMFQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:50:22.315419748 (396h39m57.086061083s) [noise=true , address=104.36.82.34:4564 , piece_id=CXLC3UUVVIGKA2OP2BQXE7XH7HNHXOJ3O6BQVUNS5NVY4KRPUWIA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:50:32.620803316 (396h40m7.391444651s) [noise=true , address=104.36.82.20:4451 , piece_id=CVO53ZISRV2ON542CVCHUAUDVZQI2L5KGCRCMZLP4POOBKLZNVFA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:50:39.176702995 (396h40m13.94734433s) [noise=true , address=45.91.95.4:4021 , piece_id=PFQ3VSEQPTNB4DERVNGT4BCSMZGY3C3OWWJJHVLQTTOCB6TWCMIA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: ecclient: upload failed (node:1dXfFE3nY9ZSEixFePGxdptsrybf38kYjYgV61MzV8kHtc12w, address:45.91.95.4:4021): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:50:39.176724405 (396h40m13.94736573s) [noise=true , address=104.36.82.44:4885 , piece_id=OZVJDJJLCK46MBKO6TGGHQFSW2QUDQ6GDUHTBZLI7Y5LWBXVPJCQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:104.36.82.44:4885): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:50:39.176800822 (396h40m13.947442157s) [noise=true , address=45.91.95.3:4021 , piece_id=W7TAKLK7PEUHTITB74T7HJGOU6WTBPRTFHPQF6VTVM4OTPNYTRWA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: ecclient: upload failed (node:131LBDdhw5s1dsLF2CnAPWaL3Amwm4kvfjyqfTvjkrHSQUjJ2q, address:45.91.95.3:4021): protocol: write tcp 188.166.225.170:52410->45.91.95.3:4021: use of closed network connection; write tcp 188.166.225.170:52410->45.91.95.3:4021: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52410->45.91.95.3:4021: use of closed network connection Jul 9 21:50:39.176914419 (396h40m13.947555744s) [noise=true , address=45.91.95.2:4017 , piece_id=W52X6YGCDZBSRQEB63HLCQV6CN5JL6JYC2SML5QIRBO5W7CIKDRQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: ecclient: upload failed (node:1vtusfmiftx4cbYbLEzPEpgvmaGQAD9wsJGBCk8r4f1MkY1RL6, address:45.91.95.2:4017): protocol: write tcp 188.166.225.170:33416->45.91.95.2:4017: use of closed network connection; write tcp 188.166.225.170:33416->45.91.95.2:4017: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33416->45.91.95.2:4017: use of closed network connection Jul 9 21:50:39.176950669 (396h40m13.947591994s) [seg_pos={0 0}, upload=317350]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 21:50:39.412544674 (396h40m14.183186009s) [upload=317350]: Done uploading segments... Jul 9 21:50:39.412568239 (396h40m14.183209574s) [upload=317350]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 36.667402774s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.....X....X..X.........X...............X..X....X........X...] duration: 48.593266693s FAIL (2025-07-09 21:51:44 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 48.593287805s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [................X....X...............X..X.X...X...X.......X.] duration: 30.624740929s FAIL (2025-07-09 21:51:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== SegmentBegin Jul 9 21:49:40.272144307 (396h39m15.042785642s) [seg_pos={0 0}, upload=317350]: Begin upload segment... Jul 9 21:49:40.272207637 (396h39m15.042848972s) [seg_pos={0 0}, upload=317350]: Done begin upload segment. Jul 9 21:49:40.272228447 (396h39m15.042869772s) [noise=true , address=45.91.116.8:4072 , piece_id=2THWBWLISKLKJFVUAESYPDVBGOSZ2M6AVHMUN2C22S46FYBNE5RA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.272260418 (396h39m15.042901743s) [noise=true , address=104.36.82.20:4451 , piece_id=CVO53ZISRV2ON542CVCHUAUDVZQI2L5KGCRCMZLP4POOBKLZNVFA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.272657718 (396h39m15.043299043s) [noise=true , address=104.36.82.34:4564 , piece_id=CXLC3UUVVIGKA2OP2BQXE7XH7HNHXOJ3O6BQVUNS5NVY4KRPUWIA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.272906815 (396h39m15.04354815s) [noise=true , address=104.36.82.45:4905 , piece_id=YQWZK3R425VYYPSXLTBCNZQUN2IPWYZTNU7NNR5EW5ONIOH7BG3A, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.273256314 (396h39m15.043897649s) [noise=true , address=45.91.95.3:4021 , piece_id=W7TAKLK7PEUHTITB74T7HJGOU6WTBPRTFHPQF6VTVM4OTPNYTRWA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.273291210 (396h39m15.043932535s) [noise=true , address=45.91.95.2:4017 , piece_id=W52X6YGCDZBSRQEB63HLCQV6CN5JL6JYC2SML5QIRBO5W7CIKDRQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.273553742 (396h39m15.044195077s) [noise=true , address=79.127.219.38:4022 , piece_id=72QFHVOHWUNOSLO65KGIXMA7BYR4DTRPVLVAD4EHHZKXX3GAQMFQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.273793651 (396h39m15.044434986s) [noise=true , address=104.36.82.44:4885 , piece_id=OZVJDJJLCK46MBKO6TGGHQFSW2QUDQ6GDUHTBZLI7Y5LWBXVPJCQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.273834831 (396h39m15.044476156s) [noise=true , address=77.90.31.11:4573 , piece_id=G76LQFRBIKPUKF7CI7DXGL6JRYJOOZJ6SLPAM4Z64CXYJVPW42DA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.273869627 (396h39m15.044510962s) [noise=true , address=77.90.31.18:4827 , piece_id=MQJMN3CW2AEP5T74I7WUUBIHFZOLMMKCIN7IOBWZT2VYQARYLIIQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.274155423 (396h39m15.044796758s) [noise=true , address=77.90.31.19:4876 , piece_id=ROM6VJDIDOEUEGP3PMEHKEDM5POGHMFNKPYRC3LCIAWIVRUGVBUQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.274314117 (396h39m15.044955452s) [noise=true , address=79.127.220.97:4371 , piece_id=XNPEHYQWPTFHLU5TJEO7HZC3MLOLE26GQWI4PI2CUCIL6QTAO7TQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.274414660 (396h39m15.045055995s) [noise=true , address=45.91.95.5:4006 , piece_id=JGKTT6OVQ7T2C22UOWG5HPQ67DBF3NPA3OAMALDLQGATY5LT63PA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.274653466 (396h39m15.045294791s) [noise=true , address=77.90.31.10:4528 , piece_id=XA7ZR76AK6TYNZDCMKPWNMZELNRGXFUU53L34IFMDWRB4LN3W4RA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.274681149 (396h39m15.045322484s) [noise=true , address=45.91.95.4:4021 , piece_id=PFQ3VSEQPTNB4DERVNGT4BCSMZGY3C3OWWJJHVLQTTOCB6TWCMIA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.274898425 (396h39m15.04553976s) [noise=true , address=77.90.31.21:4903 , piece_id=LYVWE3A5IAAEZNHB6DZ63XGBH2UQRJRYMJDMRBOJ2BIN3WESS6RQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.275008706 (396h39m15.045650041s) [noise=true , address=77.90.31.13:4652 , piece_id=IVLFU33BFOUXWJEYUUS2AEOZLAVTOKDMEK7RDPEBPOWF34SWNR6Q, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.275332215 (396h39m15.04597355s) [noise=true , address=45.91.116.7:4041 , piece_id=5JXKDZGV2EPLVSDUVP5LDFOGQ66FXPR2GXYEBFF5KWQXUPJNKPMA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:41.657764345 (396h39m16.42840568s) [upload=317350]: Next returned nil segment Jul 9 21:49:41.657771018 (396h39m16.428412353s) [upload=317350]: Waiting for error group managing segments... Jul 9 21:49:41.690644980 (396h39m16.461286315s) [noise=true , address=104.36.82.45:4905 , piece_id=YQWZK3R425VYYPSXLTBCNZQUN2IPWYZTNU7NNR5EW5ONIOH7BG3A, seg_pos={0 0}, upload=317350]: Done uploading piece. err: ecclient: upload failed (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, address:104.36.82.45:4905): protocol: untrusted: invalid order limit signature: rpcpeer: drpc transport does not have required methods Jul 9 21:49:42.092098451 (396h39m16.862739786s) [noise=true , address=79.127.220.97:4371 , piece_id=XNPEHYQWPTFHLU5TJEO7HZC3MLOLE26GQWI4PI2CUCIL6QTAO7TQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:42.266002837 (396h39m17.036644172s) [noise=true , address=77.90.31.19:4876 , piece_id=ROM6VJDIDOEUEGP3PMEHKEDM5POGHMFNKPYRC3LCIAWIVRUGVBUQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:42.269167210 (396h39m17.039808535s) [noise=true , address=77.90.31.13:4652 , piece_id=IVLFU33BFOUXWJEYUUS2AEOZLAVTOKDMEK7RDPEBPOWF34SWNR6Q, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:42.278551023 (396h39m17.049192358s) [noise=true , address=77.90.31.10:4528 , piece_id=XA7ZR76AK6TYNZDCMKPWNMZELNRGXFUU53L34IFMDWRB4LN3W4RA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:42.280070640 (396h39m17.050711975s) [noise=true , address=77.90.31.21:4903 , piece_id=LYVWE3A5IAAEZNHB6DZ63XGBH2UQRJRYMJDMRBOJ2BIN3WESS6RQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:42.296867786 (396h39m17.067509111s) [noise=true , address=77.90.31.11:4573 , piece_id=G76LQFRBIKPUKF7CI7DXGL6JRYJOOZJ6SLPAM4Z64CXYJVPW42DA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:42.303116681 (396h39m17.073758016s) [noise=true , address=77.90.31.18:4827 , piece_id=MQJMN3CW2AEP5T74I7WUUBIHFZOLMMKCIN7IOBWZT2VYQARYLIIQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:44.408881294 (396h39m19.179522629s) [noise=true , address=45.91.116.7:4041 , piece_id=5JXKDZGV2EPLVSDUVP5LDFOGQ66FXPR2GXYEBFF5KWQXUPJNKPMA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:44.871597160 (396h39m19.642238495s) [noise=true , address=45.91.95.5:4006 , piece_id=JGKTT6OVQ7T2C22UOWG5HPQ67DBF3NPA3OAMALDLQGATY5LT63PA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: ecclient: upload failed (node:12ViW64JDJxzvRU39MsLp5JZKAmHZmxCAJuLVGi2E1E1r9gfc9A, address:45.91.95.5:4006): protocol: storage node overloaded, request limit: 2000 Jul 9 21:49:51.702994736 (396h39m26.473636071s) [noise=true , address=45.91.116.8:4072 , piece_id=2THWBWLISKLKJFVUAESYPDVBGOSZ2M6AVHMUN2C22S46FYBNE5RA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:55.468073925 (396h39m30.23871526s) [noise=true , address=79.127.219.38:4022 , piece_id=72QFHVOHWUNOSLO65KGIXMA7BYR4DTRPVLVAD4EHHZKXX3GAQMFQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:50:22.315419748 (396h39m57.086061083s) [noise=true , address=104.36.82.34:4564 , piece_id=CXLC3UUVVIGKA2OP2BQXE7XH7HNHXOJ3O6BQVUNS5NVY4KRPUWIA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:50:32.620803316 (396h40m7.391444651s) [noise=true , address=104.36.82.20:4451 , piece_id=CVO53ZISRV2ON542CVCHUAUDVZQI2L5KGCRCMZLP4POOBKLZNVFA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:50:39.176702995 (396h40m13.94734433s) [noise=true , address=45.91.95.4:4021 , piece_id=PFQ3VSEQPTNB4DERVNGT4BCSMZGY3C3OWWJJHVLQTTOCB6TWCMIA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: ecclient: upload failed (node:1dXfFE3nY9ZSEixFePGxdptsrybf38kYjYgV61MzV8kHtc12w, address:45.91.95.4:4021): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:50:39.176724405 (396h40m13.94736573s) [noise=true , address=104.36.82.44:4885 , piece_id=OZVJDJJLCK46MBKO6TGGHQFSW2QUDQ6GDUHTBZLI7Y5LWBXVPJCQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:104.36.82.44:4885): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:50:39.176800822 (396h40m13.947442157s) [noise=true , address=45.91.95.3:4021 , piece_id=W7TAKLK7PEUHTITB74T7HJGOU6WTBPRTFHPQF6VTVM4OTPNYTRWA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: ecclient: upload failed (node:131LBDdhw5s1dsLF2CnAPWaL3Amwm4kvfjyqfTvjkrHSQUjJ2q, address:45.91.95.3:4021): protocol: write tcp 188.166.225.170:52410->45.91.95.3:4021: use of closed network connection; write tcp 188.166.225.170:52410->45.91.95.3:4021: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52410->45.91.95.3:4021: use of closed network connection Jul 9 21:50:39.176914419 (396h40m13.947555744s) [noise=true , address=45.91.95.2:4017 , piece_id=W52X6YGCDZBSRQEB63HLCQV6CN5JL6JYC2SML5QIRBO5W7CIKDRQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: ecclient: upload failed (node:1vtusfmiftx4cbYbLEzPEpgvmaGQAD9wsJGBCk8r4f1MkY1RL6, address:45.91.95.2:4017): protocol: write tcp 188.166.225.170:33416->45.91.95.2:4017: use of closed network connection; write tcp 188.166.225.170:33416->45.91.95.2:4017: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33416->45.91.95.2:4017: use of closed network connection Jul 9 21:50:39.176950669 (396h40m13.947591994s) [seg_pos={0 0}, upload=317350]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 21:50:39.412544674 (396h40m14.183186009s) [upload=317350]: Done uploading segments... Jul 9 21:50:39.412568239 (396h40m14.183209574s) [upload=317350]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 30.624758173s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [......X....X..X.........X...............X..X....X........X..] duration: 57.067785042s FAIL (2025-07-09 21:50:44 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 57.067798547s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [X................X....X...............X..X.X...X...X.......X] duration: 1m0.237112493s FAIL (2025-07-09 21:50:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== SegmentBegin Jul 9 21:49:40.272144307 (396h39m15.042785642s) [seg_pos={0 0}, upload=317350]: Begin upload segment... Jul 9 21:49:40.272207637 (396h39m15.042848972s) [seg_pos={0 0}, upload=317350]: Done begin upload segment. Jul 9 21:49:40.272228447 (396h39m15.042869772s) [noise=true , address=45.91.116.8:4072 , piece_id=2THWBWLISKLKJFVUAESYPDVBGOSZ2M6AVHMUN2C22S46FYBNE5RA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.272260418 (396h39m15.042901743s) [noise=true , address=104.36.82.20:4451 , piece_id=CVO53ZISRV2ON542CVCHUAUDVZQI2L5KGCRCMZLP4POOBKLZNVFA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.272657718 (396h39m15.043299043s) [noise=true , address=104.36.82.34:4564 , piece_id=CXLC3UUVVIGKA2OP2BQXE7XH7HNHXOJ3O6BQVUNS5NVY4KRPUWIA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.272906815 (396h39m15.04354815s) [noise=true , address=104.36.82.45:4905 , piece_id=YQWZK3R425VYYPSXLTBCNZQUN2IPWYZTNU7NNR5EW5ONIOH7BG3A, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.273256314 (396h39m15.043897649s) [noise=true , address=45.91.95.3:4021 , piece_id=W7TAKLK7PEUHTITB74T7HJGOU6WTBPRTFHPQF6VTVM4OTPNYTRWA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.273291210 (396h39m15.043932535s) [noise=true , address=45.91.95.2:4017 , piece_id=W52X6YGCDZBSRQEB63HLCQV6CN5JL6JYC2SML5QIRBO5W7CIKDRQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.273553742 (396h39m15.044195077s) [noise=true , address=79.127.219.38:4022 , piece_id=72QFHVOHWUNOSLO65KGIXMA7BYR4DTRPVLVAD4EHHZKXX3GAQMFQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.273793651 (396h39m15.044434986s) [noise=true , address=104.36.82.44:4885 , piece_id=OZVJDJJLCK46MBKO6TGGHQFSW2QUDQ6GDUHTBZLI7Y5LWBXVPJCQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.273834831 (396h39m15.044476156s) [noise=true , address=77.90.31.11:4573 , piece_id=G76LQFRBIKPUKF7CI7DXGL6JRYJOOZJ6SLPAM4Z64CXYJVPW42DA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.273869627 (396h39m15.044510962s) [noise=true , address=77.90.31.18:4827 , piece_id=MQJMN3CW2AEP5T74I7WUUBIHFZOLMMKCIN7IOBWZT2VYQARYLIIQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.274155423 (396h39m15.044796758s) [noise=true , address=77.90.31.19:4876 , piece_id=ROM6VJDIDOEUEGP3PMEHKEDM5POGHMFNKPYRC3LCIAWIVRUGVBUQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.274314117 (396h39m15.044955452s) [noise=true , address=79.127.220.97:4371 , piece_id=XNPEHYQWPTFHLU5TJEO7HZC3MLOLE26GQWI4PI2CUCIL6QTAO7TQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.274414660 (396h39m15.045055995s) [noise=true , address=45.91.95.5:4006 , piece_id=JGKTT6OVQ7T2C22UOWG5HPQ67DBF3NPA3OAMALDLQGATY5LT63PA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.274653466 (396h39m15.045294791s) [noise=true , address=77.90.31.10:4528 , piece_id=XA7ZR76AK6TYNZDCMKPWNMZELNRGXFUU53L34IFMDWRB4LN3W4RA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.274681149 (396h39m15.045322484s) [noise=true , address=45.91.95.4:4021 , piece_id=PFQ3VSEQPTNB4DERVNGT4BCSMZGY3C3OWWJJHVLQTTOCB6TWCMIA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.274898425 (396h39m15.04553976s) [noise=true , address=77.90.31.21:4903 , piece_id=LYVWE3A5IAAEZNHB6DZ63XGBH2UQRJRYMJDMRBOJ2BIN3WESS6RQ, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.275008706 (396h39m15.045650041s) [noise=true , address=77.90.31.13:4652 , piece_id=IVLFU33BFOUXWJEYUUS2AEOZLAVTOKDMEK7RDPEBPOWF34SWNR6Q, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:40.275332215 (396h39m15.04597355s) [noise=true , address=45.91.116.7:4041 , piece_id=5JXKDZGV2EPLVSDUVP5LDFOGQ66FXPR2GXYEBFF5KWQXUPJNKPMA, seg_pos={0 0}, upload=317350]: Uploading piece... Jul 9 21:49:41.657764345 (396h39m16.42840568s) [upload=317350]: Next returned nil segment Jul 9 21:49:41.657771018 (396h39m16.428412353s) [upload=317350]: Waiting for error group managing segments... Jul 9 21:49:41.690644980 (396h39m16.461286315s) [noise=true , address=104.36.82.45:4905 , piece_id=YQWZK3R425VYYPSXLTBCNZQUN2IPWYZTNU7NNR5EW5ONIOH7BG3A, seg_pos={0 0}, upload=317350]: Done uploading piece. err: ecclient: upload failed (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, address:104.36.82.45:4905): protocol: untrusted: invalid order limit signature: rpcpeer: drpc transport does not have required methods Jul 9 21:49:42.092098451 (396h39m16.862739786s) [noise=true , address=79.127.220.97:4371 , piece_id=XNPEHYQWPTFHLU5TJEO7HZC3MLOLE26GQWI4PI2CUCIL6QTAO7TQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:42.266002837 (396h39m17.036644172s) [noise=true , address=77.90.31.19:4876 , piece_id=ROM6VJDIDOEUEGP3PMEHKEDM5POGHMFNKPYRC3LCIAWIVRUGVBUQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:42.269167210 (396h39m17.039808535s) [noise=true , address=77.90.31.13:4652 , piece_id=IVLFU33BFOUXWJEYUUS2AEOZLAVTOKDMEK7RDPEBPOWF34SWNR6Q, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:42.278551023 (396h39m17.049192358s) [noise=true , address=77.90.31.10:4528 , piece_id=XA7ZR76AK6TYNZDCMKPWNMZELNRGXFUU53L34IFMDWRB4LN3W4RA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:42.280070640 (396h39m17.050711975s) [noise=true , address=77.90.31.21:4903 , piece_id=LYVWE3A5IAAEZNHB6DZ63XGBH2UQRJRYMJDMRBOJ2BIN3WESS6RQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:42.296867786 (396h39m17.067509111s) [noise=true , address=77.90.31.11:4573 , piece_id=G76LQFRBIKPUKF7CI7DXGL6JRYJOOZJ6SLPAM4Z64CXYJVPW42DA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:42.303116681 (396h39m17.073758016s) [noise=true , address=77.90.31.18:4827 , piece_id=MQJMN3CW2AEP5T74I7WUUBIHFZOLMMKCIN7IOBWZT2VYQARYLIIQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:44.408881294 (396h39m19.179522629s) [noise=true , address=45.91.116.7:4041 , piece_id=5JXKDZGV2EPLVSDUVP5LDFOGQ66FXPR2GXYEBFF5KWQXUPJNKPMA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:44.871597160 (396h39m19.642238495s) [noise=true , address=45.91.95.5:4006 , piece_id=JGKTT6OVQ7T2C22UOWG5HPQ67DBF3NPA3OAMALDLQGATY5LT63PA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: ecclient: upload failed (node:12ViW64JDJxzvRU39MsLp5JZKAmHZmxCAJuLVGi2E1E1r9gfc9A, address:45.91.95.5:4006): protocol: storage node overloaded, request limit: 2000 Jul 9 21:49:51.702994736 (396h39m26.473636071s) [noise=true , address=45.91.116.8:4072 , piece_id=2THWBWLISKLKJFVUAESYPDVBGOSZ2M6AVHMUN2C22S46FYBNE5RA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:49:55.468073925 (396h39m30.23871526s) [noise=true , address=79.127.219.38:4022 , piece_id=72QFHVOHWUNOSLO65KGIXMA7BYR4DTRPVLVAD4EHHZKXX3GAQMFQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:50:22.315419748 (396h39m57.086061083s) [noise=true , address=104.36.82.34:4564 , piece_id=CXLC3UUVVIGKA2OP2BQXE7XH7HNHXOJ3O6BQVUNS5NVY4KRPUWIA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:50:32.620803316 (396h40m7.391444651s) [noise=true , address=104.36.82.20:4451 , piece_id=CVO53ZISRV2ON542CVCHUAUDVZQI2L5KGCRCMZLP4POOBKLZNVFA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: <nil> Jul 9 21:50:39.176702995 (396h40m13.94734433s) [noise=true , address=45.91.95.4:4021 , piece_id=PFQ3VSEQPTNB4DERVNGT4BCSMZGY3C3OWWJJHVLQTTOCB6TWCMIA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: ecclient: upload failed (node:1dXfFE3nY9ZSEixFePGxdptsrybf38kYjYgV61MzV8kHtc12w, address:45.91.95.4:4021): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:50:39.176724405 (396h40m13.94736573s) [noise=true , address=104.36.82.44:4885 , piece_id=OZVJDJJLCK46MBKO6TGGHQFSW2QUDQ6GDUHTBZLI7Y5LWBXVPJCQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:104.36.82.44:4885): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:50:39.176800822 (396h40m13.947442157s) [noise=true , address=45.91.95.3:4021 , piece_id=W7TAKLK7PEUHTITB74T7HJGOU6WTBPRTFHPQF6VTVM4OTPNYTRWA, seg_pos={0 0}, upload=317350]: Done uploading piece. err: ecclient: upload failed (node:131LBDdhw5s1dsLF2CnAPWaL3Amwm4kvfjyqfTvjkrHSQUjJ2q, address:45.91.95.3:4021): protocol: write tcp 188.166.225.170:52410->45.91.95.3:4021: use of closed network connection; write tcp 188.166.225.170:52410->45.91.95.3:4021: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52410->45.91.95.3:4021: use of closed network connection Jul 9 21:50:39.176914419 (396h40m13.947555744s) [noise=true , address=45.91.95.2:4017 , piece_id=W52X6YGCDZBSRQEB63HLCQV6CN5JL6JYC2SML5QIRBO5W7CIKDRQ, seg_pos={0 0}, upload=317350]: Done uploading piece. err: ecclient: upload failed (node:1vtusfmiftx4cbYbLEzPEpgvmaGQAD9wsJGBCk8r4f1MkY1RL6, address:45.91.95.2:4017): protocol: write tcp 188.166.225.170:33416->45.91.95.2:4017: use of closed network connection; write tcp 188.166.225.170:33416->45.91.95.2:4017: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33416->45.91.95.2:4017: use of closed network connection Jul 9 21:50:39.176950669 (396h40m13.947591994s) [seg_pos={0 0}, upload=317350]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 21:50:39.412544674 (396h40m14.183186009s) [upload=317350]: Done uploading segments... Jul 9 21:50:39.412568239 (396h40m14.183209574s) [upload=317350]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.237135427s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [X......X....X..X.........X...............X..X....X........X.] duration: 58.68449521s FAIL (2025-07-09 21:49:47 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 58.684510099s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.X................X....X...............X..X.X...X...X.......] duration: 34.14993293s FAIL (2025-07-09 21:49:17 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 17258]: Got next segment. Inline: false Jul 9 21:42:48.387406701 (396h32m23.158048036s) [upload=317258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:42:48.387413103 (396h32m23.158054439s) [upload=317258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:42:48.834299395 (396h32m23.60494072s) [seg_pos={0 0}, upload=317258]: Begin upload segment... Jul 9 21:42:48.834354481 (396h32m23.604995816s) [seg_pos={0 0}, upload=317258]: Done begin upload segment. Jul 9 21:42:48.834374599 (396h32m23.605015924s) [noise=true , address=77.90.31.10:4523 , piece_id=SKESIP2VBIDJIBXIEGMC2LZESW47O7WJCLSBQBSWLHYWV4PRT7FQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.834761038 (396h32m23.605402363s) [noise=true , address=77.90.31.21:4908 , piece_id=HGKISLD6DZXJ6DL4UCZWUJD2V5CTDYRX2BSKDCDJYXCQQBX6Q3RA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835061884 (396h32m23.605703219s) [noise=true , address=79.127.201.215:4345 , piece_id=WEQTKXLED3S6W4HMRFM7YUEXBWGUUMQL65E4U4FG27AVCZFRK72A, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835344354 (396h32m23.605985689s) [noise=true , address=104.36.82.23:4474 , piece_id=JJNF2423XB5JQXQJ3VWRUYGLZ2QR5SFPN7BUJO4GRPSDST6IFBCA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835641913 (396h32m23.606283248s) [noise=true , address=45.91.95.2:4022 , piece_id=UJ6PKLMNBID76O3ZR4SXRNLL3CCLNGTYBZSN57E2YBD57TDLKTPQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835925006 (396h32m23.606566341s) [noise=true , address=77.90.31.14:4667 , piece_id=3XNOC4TIZJFEMIJQIMYHYNXA3SD72IYKBVBVWZ5J7NKVG5XOQYMA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836153993 (396h32m23.606795328s) [noise=true , address=104.36.82.38:4716 , piece_id=FK7WLNN6677LINQJ26LPGZ5X4JJ7ISGDRUHMVQTGK34OTL44EF6A, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836372001 (396h32m23.607013336s) [noise=true , address=77.90.31.18:4810 , piece_id=33CXTR5YAVMPV3VGG23FMKLGAKR355O66PAHFZPKVQSFJGQIEYLQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836627199 (396h32m23.607268524s) [noise=true , address=77.90.31.16:4768 , piece_id=LOU4TXWNZRUAIB7BVVZXKEVZSBL44DKOIITBGOCPDPM73L7CYRYQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836860434 (396h32m23.607501759s) [noise=true , address=77.90.31.13:4640 , piece_id=XCMFSHLM372AAMNYL2YVGRUSGWT7RB4RE4TMZ6ONHPVNR26NKUAQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837076287 (396h32m23.607717623s) [noise=true , address=45.91.95.5:4026 , piece_id=QQWGIQENU5TGALRAU4SRUFWJDQYOPW552SN46COWCRFNFBRRPBFA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837281300 (396h32m23.607922635s) [noise=true , address=79.127.220.101:4073 , piece_id=NNWGURWIPMP67UIQGVHZPFZJBD2QFOGBQ6KZNNTHLJXCGRIDTXZQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837494658 (396h32m23.608135993s) [noise=true , address=104.36.82.18:4396 , piece_id=2DRRKEK2OLFL54G6BQIIQLEPONIHDUICJUVAEAONC2IOR22POPLA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837859957 (396h32m23.608501292s) [noise=true , address=45.91.95.4:4014 , piece_id=V5ZNHUMTQCZ5YXOYDAYCFKBBSAUKHJ2RJ3FSQNME6IXSGFSLIJ4Q, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838145814 (396h32m23.608787139s) [noise=true , address=77.90.31.18:4838 , piece_id=EGYCA5GBMRI3JPX3SHKQ57GU6ZQV6S33BJNOJVUJHESMEKZU5WTQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838416161 (396h32m23.609057486s) [noise=true , address=45.91.95.3:4019 , piece_id=3XZWJLZH5RXX45ULYQLXAG4SG2RNMOGWXLNFTOVCVKZ2QUQOC6KQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838665709 (396h32m23.609307044s) [noise=true , address=77.90.31.19:4852 , piece_id=MQ7OIWIHD642LXDOEBKZF2DTZRLR5OJT4Q37DCLLDXR7XTJVKXKA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838944833 (396h32m23.609586168s) [noise=true , address=77.90.31.12:4621 , piece_id=CL234SZAWOTG4ONGWQ2ZBXEGWUYTQQVR76BVCHAPXETRGF5TQJBQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:50.183574564 (396h32m24.954215899s) [upload=317258]: Next returned nil segment Jul 9 21:42:50.183596435 (396h32m24.95423777s) [upload=317258]: Waiting for error group managing segments... Jul 9 21:42:50.548527303 (396h32m25.319168638s) [noise=true , address=79.127.220.101:4073 , piece_id=NNWGURWIPMP67UIQGVHZPFZJBD2QFOGBQ6KZNNTHLJXCGRIDTXZQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.840602026 (396h32m25.611243361s) [noise=true , address=77.90.31.18:4838 , piece_id=EGYCA5GBMRI3JPX3SHKQ57GU6ZQV6S33BJNOJVUJHESMEKZU5WTQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.841968832 (396h32m25.612610167s) [noise=true , address=77.90.31.18:4810 , piece_id=33CXTR5YAVMPV3VGG23FMKLGAKR355O66PAHFZPKVQSFJGQIEYLQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.842708556 (396h32m25.613349951s) [noise=true , address=77.90.31.13:4640 , piece_id=XCMFSHLM372AAMNYL2YVGRUSGWT7RB4RE4TMZ6ONHPVNR26NKUAQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.843190929 (396h32m25.613832265s) [noise=true , address=77.90.31.14:4667 , piece_id=3XNOC4TIZJFEMIJQIMYHYNXA3SD72IYKBVBVWZ5J7NKVG5XOQYMA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.844590978 (396h32m25.615232313s) [noise=true , address=77.90.31.10:4523 , piece_id=SKESIP2VBIDJIBXIEGMC2LZESW47O7WJCLSBQBSWLHYWV4PRT7FQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.845022593 (396h32m25.615663928s) [noise=true , address=77.90.31.19:4852 , piece_id=MQ7OIWIHD642LXDOEBKZF2DTZRLR5OJT4Q37DCLLDXR7XTJVKXKA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.846516732 (396h32m25.617158067s) [noise=true , address=77.90.31.12:4621 , piece_id=CL234SZAWOTG4ONGWQ2ZBXEGWUYTQQVR76BVCHAPXETRGF5TQJBQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.847220148 (396h32m25.617861473s) [noise=true , address=77.90.31.21:4908 , piece_id=HGKISLD6DZXJ6DL4UCZWUJD2V5CTDYRX2BSKDCDJYXCQQBX6Q3RA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.847310801 (396h32m25.617952126s) [noise=true , address=77.90.31.16:4768 , piece_id=LOU4TXWNZRUAIB7BVVZXKEVZSBL44DKOIITBGOCPDPM73L7CYRYQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:55.315575878 (396h32m30.086217213s) [noise=true , address=79.127.201.215:4345 , piece_id=WEQTKXLED3S6W4HMRFM7YUEXBWGUUMQL65E4U4FG27AVCZFRK72A, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:12.086886240 (396h32m46.857527576s) [noise=true , address=104.36.82.23:4474 , piece_id=JJNF2423XB5JQXQJ3VWRUYGLZ2QR5SFPN7BUJO4GRPSDST6IFBCA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:29.357614308 (396h33m4.128255643s) [noise=true , address=104.36.82.18:4396 , piece_id=2DRRKEK2OLFL54G6BQIIQLEPONIHDUICJUVAEAONC2IOR22POPLA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:38.089997559 (396h33m12.860638884s) [noise=true , address=45.91.95.4:4014 , piece_id=V5ZNHUMTQCZ5YXOYDAYCFKBBSAUKHJ2RJ3FSQNME6IXSGFSLIJ4Q, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:38.090021875 (396h33m12.86066321s) [seg_pos={0 0}, upload=317258]: Segment reached optimal threshold of 14 pieces. Jul 9 21:43:38.090108922 (396h33m12.860750257s) [noise=true , address=45.91.95.5:4026 , piece_id=QQWGIQENU5TGALRAU4SRUFWJDQYOPW552SN46COWCRFNFBRRPBFA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xxtZ3msZnUkE4kgTFKJmprcNS986Uhi1LjwVyd1DbrTfZS52H): protocol: expected piece hash; context canceled; EOF Jul 9 21:43:38.090133739 (396h33m12.860775074s) [noise=true , address=104.36.82.38:4716 , piece_id=FK7WLNN6677LINQJ26LPGZ5X4JJ7ISGDRUHMVQTGK34OTL44EF6A, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MAHcnhvjvD74dpLS5S3m751NzfjvJ8a9ZvHwA7mdXUEXm9DAX): protocol: expected piece hash; context canceled; EOF Jul 9 21:43:38.090161603 (396h33m12.860802938s) [noise=true , address=45.91.95.3:4019 , piece_id=3XZWJLZH5RXX45ULYQLXAG4SG2RNMOGWXLNFTOVCVKZ2QUQOC6KQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SGjoGjbfugbxFgDctHygi7qo8upC463TRw8sEceUNnti1iGV1): protocol: expected piece hash; context canceled; EOF Jul 9 21:43:38.090225525 (396h33m12.86086685s) [noise=true , address=45.91.95.2:4022 , piece_id=UJ6PKLMNBID76O3ZR4SXRNLL3CCLNGTYBZSN57E2YBD57TDLKTPQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XGBvsxi6S7dz9d93CWmnQhomJVg1x5evu7DxsGsujkejBkZVN): protocol: write tcp 188.166.225.170:50926->45.91.95.2:4022: use of closed network connection; write tcp 188.166.225.170:50926->45.91.95.2:4022: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50926->45.91.95.2:4022: use of closed network connection Jul 9 21:43:38.090246715 (396h33m12.860888051s) [seg_pos={0 0}, upload=317258]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:43:38.090334674 (396h33m12.860976009s) [upload=317258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:43:38.090338631 (396h33m12.860979956s) [upload=317258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:43:38.379746682 (396h33m13.150388017s) [upload=317258]: Done uploading segments... Jul 9 21:43:38.379765919 (396h33m13.150407254s) [upload=317258]: Upload finished. err: <nil> Duration: 34.149946444s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.X......X....X..X.........X...............X..X....X........X] duration: 1m0.000976134s FAIL (2025-07-09 21:48:48 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000987185s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [..X................X....X...............X..X.X...X...X......] duration: 48.26210004s FAIL (2025-07-09 21:48:43 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 17258]: Got next segment. Inline: false Jul 9 21:42:48.387406701 (396h32m23.158048036s) [upload=317258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:42:48.387413103 (396h32m23.158054439s) [upload=317258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:42:48.834299395 (396h32m23.60494072s) [seg_pos={0 0}, upload=317258]: Begin upload segment... Jul 9 21:42:48.834354481 (396h32m23.604995816s) [seg_pos={0 0}, upload=317258]: Done begin upload segment. Jul 9 21:42:48.834374599 (396h32m23.605015924s) [noise=true , address=77.90.31.10:4523 , piece_id=SKESIP2VBIDJIBXIEGMC2LZESW47O7WJCLSBQBSWLHYWV4PRT7FQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.834761038 (396h32m23.605402363s) [noise=true , address=77.90.31.21:4908 , piece_id=HGKISLD6DZXJ6DL4UCZWUJD2V5CTDYRX2BSKDCDJYXCQQBX6Q3RA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835061884 (396h32m23.605703219s) [noise=true , address=79.127.201.215:4345 , piece_id=WEQTKXLED3S6W4HMRFM7YUEXBWGUUMQL65E4U4FG27AVCZFRK72A, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835344354 (396h32m23.605985689s) [noise=true , address=104.36.82.23:4474 , piece_id=JJNF2423XB5JQXQJ3VWRUYGLZ2QR5SFPN7BUJO4GRPSDST6IFBCA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835641913 (396h32m23.606283248s) [noise=true , address=45.91.95.2:4022 , piece_id=UJ6PKLMNBID76O3ZR4SXRNLL3CCLNGTYBZSN57E2YBD57TDLKTPQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835925006 (396h32m23.606566341s) [noise=true , address=77.90.31.14:4667 , piece_id=3XNOC4TIZJFEMIJQIMYHYNXA3SD72IYKBVBVWZ5J7NKVG5XOQYMA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836153993 (396h32m23.606795328s) [noise=true , address=104.36.82.38:4716 , piece_id=FK7WLNN6677LINQJ26LPGZ5X4JJ7ISGDRUHMVQTGK34OTL44EF6A, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836372001 (396h32m23.607013336s) [noise=true , address=77.90.31.18:4810 , piece_id=33CXTR5YAVMPV3VGG23FMKLGAKR355O66PAHFZPKVQSFJGQIEYLQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836627199 (396h32m23.607268524s) [noise=true , address=77.90.31.16:4768 , piece_id=LOU4TXWNZRUAIB7BVVZXKEVZSBL44DKOIITBGOCPDPM73L7CYRYQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836860434 (396h32m23.607501759s) [noise=true , address=77.90.31.13:4640 , piece_id=XCMFSHLM372AAMNYL2YVGRUSGWT7RB4RE4TMZ6ONHPVNR26NKUAQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837076287 (396h32m23.607717623s) [noise=true , address=45.91.95.5:4026 , piece_id=QQWGIQENU5TGALRAU4SRUFWJDQYOPW552SN46COWCRFNFBRRPBFA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837281300 (396h32m23.607922635s) [noise=true , address=79.127.220.101:4073 , piece_id=NNWGURWIPMP67UIQGVHZPFZJBD2QFOGBQ6KZNNTHLJXCGRIDTXZQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837494658 (396h32m23.608135993s) [noise=true , address=104.36.82.18:4396 , piece_id=2DRRKEK2OLFL54G6BQIIQLEPONIHDUICJUVAEAONC2IOR22POPLA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837859957 (396h32m23.608501292s) [noise=true , address=45.91.95.4:4014 , piece_id=V5ZNHUMTQCZ5YXOYDAYCFKBBSAUKHJ2RJ3FSQNME6IXSGFSLIJ4Q, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838145814 (396h32m23.608787139s) [noise=true , address=77.90.31.18:4838 , piece_id=EGYCA5GBMRI3JPX3SHKQ57GU6ZQV6S33BJNOJVUJHESMEKZU5WTQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838416161 (396h32m23.609057486s) [noise=true , address=45.91.95.3:4019 , piece_id=3XZWJLZH5RXX45ULYQLXAG4SG2RNMOGWXLNFTOVCVKZ2QUQOC6KQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838665709 (396h32m23.609307044s) [noise=true , address=77.90.31.19:4852 , piece_id=MQ7OIWIHD642LXDOEBKZF2DTZRLR5OJT4Q37DCLLDXR7XTJVKXKA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838944833 (396h32m23.609586168s) [noise=true , address=77.90.31.12:4621 , piece_id=CL234SZAWOTG4ONGWQ2ZBXEGWUYTQQVR76BVCHAPXETRGF5TQJBQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:50.183574564 (396h32m24.954215899s) [upload=317258]: Next returned nil segment Jul 9 21:42:50.183596435 (396h32m24.95423777s) [upload=317258]: Waiting for error group managing segments... Jul 9 21:42:50.548527303 (396h32m25.319168638s) [noise=true , address=79.127.220.101:4073 , piece_id=NNWGURWIPMP67UIQGVHZPFZJBD2QFOGBQ6KZNNTHLJXCGRIDTXZQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.840602026 (396h32m25.611243361s) [noise=true , address=77.90.31.18:4838 , piece_id=EGYCA5GBMRI3JPX3SHKQ57GU6ZQV6S33BJNOJVUJHESMEKZU5WTQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.841968832 (396h32m25.612610167s) [noise=true , address=77.90.31.18:4810 , piece_id=33CXTR5YAVMPV3VGG23FMKLGAKR355O66PAHFZPKVQSFJGQIEYLQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.842708556 (396h32m25.613349951s) [noise=true , address=77.90.31.13:4640 , piece_id=XCMFSHLM372AAMNYL2YVGRUSGWT7RB4RE4TMZ6ONHPVNR26NKUAQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.843190929 (396h32m25.613832265s) [noise=true , address=77.90.31.14:4667 , piece_id=3XNOC4TIZJFEMIJQIMYHYNXA3SD72IYKBVBVWZ5J7NKVG5XOQYMA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.844590978 (396h32m25.615232313s) [noise=true , address=77.90.31.10:4523 , piece_id=SKESIP2VBIDJIBXIEGMC2LZESW47O7WJCLSBQBSWLHYWV4PRT7FQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.845022593 (396h32m25.615663928s) [noise=true , address=77.90.31.19:4852 , piece_id=MQ7OIWIHD642LXDOEBKZF2DTZRLR5OJT4Q37DCLLDXR7XTJVKXKA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.846516732 (396h32m25.617158067s) [noise=true , address=77.90.31.12:4621 , piece_id=CL234SZAWOTG4ONGWQ2ZBXEGWUYTQQVR76BVCHAPXETRGF5TQJBQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.847220148 (396h32m25.617861473s) [noise=true , address=77.90.31.21:4908 , piece_id=HGKISLD6DZXJ6DL4UCZWUJD2V5CTDYRX2BSKDCDJYXCQQBX6Q3RA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.847310801 (396h32m25.617952126s) [noise=true , address=77.90.31.16:4768 , piece_id=LOU4TXWNZRUAIB7BVVZXKEVZSBL44DKOIITBGOCPDPM73L7CYRYQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:55.315575878 (396h32m30.086217213s) [noise=true , address=79.127.201.215:4345 , piece_id=WEQTKXLED3S6W4HMRFM7YUEXBWGUUMQL65E4U4FG27AVCZFRK72A, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:12.086886240 (396h32m46.857527576s) [noise=true , address=104.36.82.23:4474 , piece_id=JJNF2423XB5JQXQJ3VWRUYGLZ2QR5SFPN7BUJO4GRPSDST6IFBCA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:29.357614308 (396h33m4.128255643s) [noise=true , address=104.36.82.18:4396 , piece_id=2DRRKEK2OLFL54G6BQIIQLEPONIHDUICJUVAEAONC2IOR22POPLA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:38.089997559 (396h33m12.860638884s) [noise=true , address=45.91.95.4:4014 , piece_id=V5ZNHUMTQCZ5YXOYDAYCFKBBSAUKHJ2RJ3FSQNME6IXSGFSLIJ4Q, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:38.090021875 (396h33m12.86066321s) [seg_pos={0 0}, upload=317258]: Segment reached optimal threshold of 14 pieces. Jul 9 21:43:38.090108922 (396h33m12.860750257s) [noise=true , address=45.91.95.5:4026 , piece_id=QQWGIQENU5TGALRAU4SRUFWJDQYOPW552SN46COWCRFNFBRRPBFA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xxtZ3msZnUkE4kgTFKJmprcNS986Uhi1LjwVyd1DbrTfZS52H): protocol: expected piece hash; context canceled; EOF Jul 9 21:43:38.090133739 (396h33m12.860775074s) [noise=true , address=104.36.82.38:4716 , piece_id=FK7WLNN6677LINQJ26LPGZ5X4JJ7ISGDRUHMVQTGK34OTL44EF6A, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MAHcnhvjvD74dpLS5S3m751NzfjvJ8a9ZvHwA7mdXUEXm9DAX): protocol: expected piece hash; context canceled; EOF Jul 9 21:43:38.090161603 (396h33m12.860802938s) [noise=true , address=45.91.95.3:4019 , piece_id=3XZWJLZH5RXX45ULYQLXAG4SG2RNMOGWXLNFTOVCVKZ2QUQOC6KQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SGjoGjbfugbxFgDctHygi7qo8upC463TRw8sEceUNnti1iGV1): protocol: expected piece hash; context canceled; EOF Jul 9 21:43:38.090225525 (396h33m12.86086685s) [noise=true , address=45.91.95.2:4022 , piece_id=UJ6PKLMNBID76O3ZR4SXRNLL3CCLNGTYBZSN57E2YBD57TDLKTPQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XGBvsxi6S7dz9d93CWmnQhomJVg1x5evu7DxsGsujkejBkZVN): protocol: write tcp 188.166.225.170:50926->45.91.95.2:4022: use of closed network connection; write tcp 188.166.225.170:50926->45.91.95.2:4022: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50926->45.91.95.2:4022: use of closed network connection Jul 9 21:43:38.090246715 (396h33m12.860888051s) [seg_pos={0 0}, upload=317258]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:43:38.090334674 (396h33m12.860976009s) [upload=317258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:43:38.090338631 (396h33m12.860979956s) [upload=317258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:43:38.379746682 (396h33m13.150388017s) [upload=317258]: Done uploading segments... Jul 9 21:43:38.379765919 (396h33m13.150407254s) [upload=317258]: Upload finished. err: <nil> Duration: 48.262116471s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [...X................X....X...............X..X.X...X...X.....] duration: 42.4569309s FAIL (2025-07-09 21:47:54 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 17258]: Got next segment. Inline: false Jul 9 21:42:48.387406701 (396h32m23.158048036s) [upload=317258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:42:48.387413103 (396h32m23.158054439s) [upload=317258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:42:48.834299395 (396h32m23.60494072s) [seg_pos={0 0}, upload=317258]: Begin upload segment... Jul 9 21:42:48.834354481 (396h32m23.604995816s) [seg_pos={0 0}, upload=317258]: Done begin upload segment. Jul 9 21:42:48.834374599 (396h32m23.605015924s) [noise=true , address=77.90.31.10:4523 , piece_id=SKESIP2VBIDJIBXIEGMC2LZESW47O7WJCLSBQBSWLHYWV4PRT7FQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.834761038 (396h32m23.605402363s) [noise=true , address=77.90.31.21:4908 , piece_id=HGKISLD6DZXJ6DL4UCZWUJD2V5CTDYRX2BSKDCDJYXCQQBX6Q3RA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835061884 (396h32m23.605703219s) [noise=true , address=79.127.201.215:4345 , piece_id=WEQTKXLED3S6W4HMRFM7YUEXBWGUUMQL65E4U4FG27AVCZFRK72A, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835344354 (396h32m23.605985689s) [noise=true , address=104.36.82.23:4474 , piece_id=JJNF2423XB5JQXQJ3VWRUYGLZ2QR5SFPN7BUJO4GRPSDST6IFBCA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835641913 (396h32m23.606283248s) [noise=true , address=45.91.95.2:4022 , piece_id=UJ6PKLMNBID76O3ZR4SXRNLL3CCLNGTYBZSN57E2YBD57TDLKTPQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835925006 (396h32m23.606566341s) [noise=true , address=77.90.31.14:4667 , piece_id=3XNOC4TIZJFEMIJQIMYHYNXA3SD72IYKBVBVWZ5J7NKVG5XOQYMA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836153993 (396h32m23.606795328s) [noise=true , address=104.36.82.38:4716 , piece_id=FK7WLNN6677LINQJ26LPGZ5X4JJ7ISGDRUHMVQTGK34OTL44EF6A, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836372001 (396h32m23.607013336s) [noise=true , address=77.90.31.18:4810 , piece_id=33CXTR5YAVMPV3VGG23FMKLGAKR355O66PAHFZPKVQSFJGQIEYLQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836627199 (396h32m23.607268524s) [noise=true , address=77.90.31.16:4768 , piece_id=LOU4TXWNZRUAIB7BVVZXKEVZSBL44DKOIITBGOCPDPM73L7CYRYQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836860434 (396h32m23.607501759s) [noise=true , address=77.90.31.13:4640 , piece_id=XCMFSHLM372AAMNYL2YVGRUSGWT7RB4RE4TMZ6ONHPVNR26NKUAQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837076287 (396h32m23.607717623s) [noise=true , address=45.91.95.5:4026 , piece_id=QQWGIQENU5TGALRAU4SRUFWJDQYOPW552SN46COWCRFNFBRRPBFA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837281300 (396h32m23.607922635s) [noise=true , address=79.127.220.101:4073 , piece_id=NNWGURWIPMP67UIQGVHZPFZJBD2QFOGBQ6KZNNTHLJXCGRIDTXZQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837494658 (396h32m23.608135993s) [noise=true , address=104.36.82.18:4396 , piece_id=2DRRKEK2OLFL54G6BQIIQLEPONIHDUICJUVAEAONC2IOR22POPLA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837859957 (396h32m23.608501292s) [noise=true , address=45.91.95.4:4014 , piece_id=V5ZNHUMTQCZ5YXOYDAYCFKBBSAUKHJ2RJ3FSQNME6IXSGFSLIJ4Q, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838145814 (396h32m23.608787139s) [noise=true , address=77.90.31.18:4838 , piece_id=EGYCA5GBMRI3JPX3SHKQ57GU6ZQV6S33BJNOJVUJHESMEKZU5WTQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838416161 (396h32m23.609057486s) [noise=true , address=45.91.95.3:4019 , piece_id=3XZWJLZH5RXX45ULYQLXAG4SG2RNMOGWXLNFTOVCVKZ2QUQOC6KQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838665709 (396h32m23.609307044s) [noise=true , address=77.90.31.19:4852 , piece_id=MQ7OIWIHD642LXDOEBKZF2DTZRLR5OJT4Q37DCLLDXR7XTJVKXKA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838944833 (396h32m23.609586168s) [noise=true , address=77.90.31.12:4621 , piece_id=CL234SZAWOTG4ONGWQ2ZBXEGWUYTQQVR76BVCHAPXETRGF5TQJBQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:50.183574564 (396h32m24.954215899s) [upload=317258]: Next returned nil segment Jul 9 21:42:50.183596435 (396h32m24.95423777s) [upload=317258]: Waiting for error group managing segments... Jul 9 21:42:50.548527303 (396h32m25.319168638s) [noise=true , address=79.127.220.101:4073 , piece_id=NNWGURWIPMP67UIQGVHZPFZJBD2QFOGBQ6KZNNTHLJXCGRIDTXZQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.840602026 (396h32m25.611243361s) [noise=true , address=77.90.31.18:4838 , piece_id=EGYCA5GBMRI3JPX3SHKQ57GU6ZQV6S33BJNOJVUJHESMEKZU5WTQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.841968832 (396h32m25.612610167s) [noise=true , address=77.90.31.18:4810 , piece_id=33CXTR5YAVMPV3VGG23FMKLGAKR355O66PAHFZPKVQSFJGQIEYLQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.842708556 (396h32m25.613349951s) [noise=true , address=77.90.31.13:4640 , piece_id=XCMFSHLM372AAMNYL2YVGRUSGWT7RB4RE4TMZ6ONHPVNR26NKUAQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.843190929 (396h32m25.613832265s) [noise=true , address=77.90.31.14:4667 , piece_id=3XNOC4TIZJFEMIJQIMYHYNXA3SD72IYKBVBVWZ5J7NKVG5XOQYMA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.844590978 (396h32m25.615232313s) [noise=true , address=77.90.31.10:4523 , piece_id=SKESIP2VBIDJIBXIEGMC2LZESW47O7WJCLSBQBSWLHYWV4PRT7FQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.845022593 (396h32m25.615663928s) [noise=true , address=77.90.31.19:4852 , piece_id=MQ7OIWIHD642LXDOEBKZF2DTZRLR5OJT4Q37DCLLDXR7XTJVKXKA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.846516732 (396h32m25.617158067s) [noise=true , address=77.90.31.12:4621 , piece_id=CL234SZAWOTG4ONGWQ2ZBXEGWUYTQQVR76BVCHAPXETRGF5TQJBQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.847220148 (396h32m25.617861473s) [noise=true , address=77.90.31.21:4908 , piece_id=HGKISLD6DZXJ6DL4UCZWUJD2V5CTDYRX2BSKDCDJYXCQQBX6Q3RA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.847310801 (396h32m25.617952126s) [noise=true , address=77.90.31.16:4768 , piece_id=LOU4TXWNZRUAIB7BVVZXKEVZSBL44DKOIITBGOCPDPM73L7CYRYQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:55.315575878 (396h32m30.086217213s) [noise=true , address=79.127.201.215:4345 , piece_id=WEQTKXLED3S6W4HMRFM7YUEXBWGUUMQL65E4U4FG27AVCZFRK72A, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:12.086886240 (396h32m46.857527576s) [noise=true , address=104.36.82.23:4474 , piece_id=JJNF2423XB5JQXQJ3VWRUYGLZ2QR5SFPN7BUJO4GRPSDST6IFBCA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:29.357614308 (396h33m4.128255643s) [noise=true , address=104.36.82.18:4396 , piece_id=2DRRKEK2OLFL54G6BQIIQLEPONIHDUICJUVAEAONC2IOR22POPLA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:38.089997559 (396h33m12.860638884s) [noise=true , address=45.91.95.4:4014 , piece_id=V5ZNHUMTQCZ5YXOYDAYCFKBBSAUKHJ2RJ3FSQNME6IXSGFSLIJ4Q, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:38.090021875 (396h33m12.86066321s) [seg_pos={0 0}, upload=317258]: Segment reached optimal threshold of 14 pieces. Jul 9 21:43:38.090108922 (396h33m12.860750257s) [noise=true , address=45.91.95.5:4026 , piece_id=QQWGIQENU5TGALRAU4SRUFWJDQYOPW552SN46COWCRFNFBRRPBFA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xxtZ3msZnUkE4kgTFKJmprcNS986Uhi1LjwVyd1DbrTfZS52H): protocol: expected piece hash; context canceled; EOF Jul 9 21:43:38.090133739 (396h33m12.860775074s) [noise=true , address=104.36.82.38:4716 , piece_id=FK7WLNN6677LINQJ26LPGZ5X4JJ7ISGDRUHMVQTGK34OTL44EF6A, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MAHcnhvjvD74dpLS5S3m751NzfjvJ8a9ZvHwA7mdXUEXm9DAX): protocol: expected piece hash; context canceled; EOF Jul 9 21:43:38.090161603 (396h33m12.860802938s) [noise=true , address=45.91.95.3:4019 , piece_id=3XZWJLZH5RXX45ULYQLXAG4SG2RNMOGWXLNFTOVCVKZ2QUQOC6KQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SGjoGjbfugbxFgDctHygi7qo8upC463TRw8sEceUNnti1iGV1): protocol: expected piece hash; context canceled; EOF Jul 9 21:43:38.090225525 (396h33m12.86086685s) [noise=true , address=45.91.95.2:4022 , piece_id=UJ6PKLMNBID76O3ZR4SXRNLL3CCLNGTYBZSN57E2YBD57TDLKTPQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XGBvsxi6S7dz9d93CWmnQhomJVg1x5evu7DxsGsujkejBkZVN): protocol: write tcp 188.166.225.170:50926->45.91.95.2:4022: use of closed network connection; write tcp 188.166.225.170:50926->45.91.95.2:4022: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50926->45.91.95.2:4022: use of closed network connection Jul 9 21:43:38.090246715 (396h33m12.860888051s) [seg_pos={0 0}, upload=317258]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:43:38.090334674 (396h33m12.860976009s) [upload=317258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:43:38.090338631 (396h33m12.860979956s) [upload=317258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:43:38.379746682 (396h33m13.150388017s) [upload=317258]: Done uploading segments... Jul 9 21:43:38.379765919 (396h33m13.150407254s) [upload=317258]: Upload finished. err: <nil> Duration: 42.456984302s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [..X......X....X..X.........X...............X..X....X........] duration: 53.38074498s FAIL (2025-07-09 21:47:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 53.380768275s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [X...X................X....X...............X..X.X...X...X....] duration: 51.119191328s FAIL (2025-07-09 21:47:09 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 17258]: Got next segment. Inline: false Jul 9 21:42:48.387406701 (396h32m23.158048036s) [upload=317258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:42:48.387413103 (396h32m23.158054439s) [upload=317258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:42:48.834299395 (396h32m23.60494072s) [seg_pos={0 0}, upload=317258]: Begin upload segment... Jul 9 21:42:48.834354481 (396h32m23.604995816s) [seg_pos={0 0}, upload=317258]: Done begin upload segment. Jul 9 21:42:48.834374599 (396h32m23.605015924s) [noise=true , address=77.90.31.10:4523 , piece_id=SKESIP2VBIDJIBXIEGMC2LZESW47O7WJCLSBQBSWLHYWV4PRT7FQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.834761038 (396h32m23.605402363s) [noise=true , address=77.90.31.21:4908 , piece_id=HGKISLD6DZXJ6DL4UCZWUJD2V5CTDYRX2BSKDCDJYXCQQBX6Q3RA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835061884 (396h32m23.605703219s) [noise=true , address=79.127.201.215:4345 , piece_id=WEQTKXLED3S6W4HMRFM7YUEXBWGUUMQL65E4U4FG27AVCZFRK72A, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835344354 (396h32m23.605985689s) [noise=true , address=104.36.82.23:4474 , piece_id=JJNF2423XB5JQXQJ3VWRUYGLZ2QR5SFPN7BUJO4GRPSDST6IFBCA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835641913 (396h32m23.606283248s) [noise=true , address=45.91.95.2:4022 , piece_id=UJ6PKLMNBID76O3ZR4SXRNLL3CCLNGTYBZSN57E2YBD57TDLKTPQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835925006 (396h32m23.606566341s) [noise=true , address=77.90.31.14:4667 , piece_id=3XNOC4TIZJFEMIJQIMYHYNXA3SD72IYKBVBVWZ5J7NKVG5XOQYMA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836153993 (396h32m23.606795328s) [noise=true , address=104.36.82.38:4716 , piece_id=FK7WLNN6677LINQJ26LPGZ5X4JJ7ISGDRUHMVQTGK34OTL44EF6A, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836372001 (396h32m23.607013336s) [noise=true , address=77.90.31.18:4810 , piece_id=33CXTR5YAVMPV3VGG23FMKLGAKR355O66PAHFZPKVQSFJGQIEYLQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836627199 (396h32m23.607268524s) [noise=true , address=77.90.31.16:4768 , piece_id=LOU4TXWNZRUAIB7BVVZXKEVZSBL44DKOIITBGOCPDPM73L7CYRYQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836860434 (396h32m23.607501759s) [noise=true , address=77.90.31.13:4640 , piece_id=XCMFSHLM372AAMNYL2YVGRUSGWT7RB4RE4TMZ6ONHPVNR26NKUAQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837076287 (396h32m23.607717623s) [noise=true , address=45.91.95.5:4026 , piece_id=QQWGIQENU5TGALRAU4SRUFWJDQYOPW552SN46COWCRFNFBRRPBFA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837281300 (396h32m23.607922635s) [noise=true , address=79.127.220.101:4073 , piece_id=NNWGURWIPMP67UIQGVHZPFZJBD2QFOGBQ6KZNNTHLJXCGRIDTXZQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837494658 (396h32m23.608135993s) [noise=true , address=104.36.82.18:4396 , piece_id=2DRRKEK2OLFL54G6BQIIQLEPONIHDUICJUVAEAONC2IOR22POPLA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837859957 (396h32m23.608501292s) [noise=true , address=45.91.95.4:4014 , piece_id=V5ZNHUMTQCZ5YXOYDAYCFKBBSAUKHJ2RJ3FSQNME6IXSGFSLIJ4Q, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838145814 (396h32m23.608787139s) [noise=true , address=77.90.31.18:4838 , piece_id=EGYCA5GBMRI3JPX3SHKQ57GU6ZQV6S33BJNOJVUJHESMEKZU5WTQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838416161 (396h32m23.609057486s) [noise=true , address=45.91.95.3:4019 , piece_id=3XZWJLZH5RXX45ULYQLXAG4SG2RNMOGWXLNFTOVCVKZ2QUQOC6KQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838665709 (396h32m23.609307044s) [noise=true , address=77.90.31.19:4852 , piece_id=MQ7OIWIHD642LXDOEBKZF2DTZRLR5OJT4Q37DCLLDXR7XTJVKXKA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838944833 (396h32m23.609586168s) [noise=true , address=77.90.31.12:4621 , piece_id=CL234SZAWOTG4ONGWQ2ZBXEGWUYTQQVR76BVCHAPXETRGF5TQJBQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:50.183574564 (396h32m24.954215899s) [upload=317258]: Next returned nil segment Jul 9 21:42:50.183596435 (396h32m24.95423777s) [upload=317258]: Waiting for error group managing segments... Jul 9 21:42:50.548527303 (396h32m25.319168638s) [noise=true , address=79.127.220.101:4073 , piece_id=NNWGURWIPMP67UIQGVHZPFZJBD2QFOGBQ6KZNNTHLJXCGRIDTXZQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.840602026 (396h32m25.611243361s) [noise=true , address=77.90.31.18:4838 , piece_id=EGYCA5GBMRI3JPX3SHKQ57GU6ZQV6S33BJNOJVUJHESMEKZU5WTQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.841968832 (396h32m25.612610167s) [noise=true , address=77.90.31.18:4810 , piece_id=33CXTR5YAVMPV3VGG23FMKLGAKR355O66PAHFZPKVQSFJGQIEYLQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.842708556 (396h32m25.613349951s) [noise=true , address=77.90.31.13:4640 , piece_id=XCMFSHLM372AAMNYL2YVGRUSGWT7RB4RE4TMZ6ONHPVNR26NKUAQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.843190929 (396h32m25.613832265s) [noise=true , address=77.90.31.14:4667 , piece_id=3XNOC4TIZJFEMIJQIMYHYNXA3SD72IYKBVBVWZ5J7NKVG5XOQYMA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.844590978 (396h32m25.615232313s) [noise=true , address=77.90.31.10:4523 , piece_id=SKESIP2VBIDJIBXIEGMC2LZESW47O7WJCLSBQBSWLHYWV4PRT7FQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.845022593 (396h32m25.615663928s) [noise=true , address=77.90.31.19:4852 , piece_id=MQ7OIWIHD642LXDOEBKZF2DTZRLR5OJT4Q37DCLLDXR7XTJVKXKA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.846516732 (396h32m25.617158067s) [noise=true , address=77.90.31.12:4621 , piece_id=CL234SZAWOTG4ONGWQ2ZBXEGWUYTQQVR76BVCHAPXETRGF5TQJBQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.847220148 (396h32m25.617861473s) [noise=true , address=77.90.31.21:4908 , piece_id=HGKISLD6DZXJ6DL4UCZWUJD2V5CTDYRX2BSKDCDJYXCQQBX6Q3RA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.847310801 (396h32m25.617952126s) [noise=true , address=77.90.31.16:4768 , piece_id=LOU4TXWNZRUAIB7BVVZXKEVZSBL44DKOIITBGOCPDPM73L7CYRYQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:55.315575878 (396h32m30.086217213s) [noise=true , address=79.127.201.215:4345 , piece_id=WEQTKXLED3S6W4HMRFM7YUEXBWGUUMQL65E4U4FG27AVCZFRK72A, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:12.086886240 (396h32m46.857527576s) [noise=true , address=104.36.82.23:4474 , piece_id=JJNF2423XB5JQXQJ3VWRUYGLZ2QR5SFPN7BUJO4GRPSDST6IFBCA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:29.357614308 (396h33m4.128255643s) [noise=true , address=104.36.82.18:4396 , piece_id=2DRRKEK2OLFL54G6BQIIQLEPONIHDUICJUVAEAONC2IOR22POPLA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:38.089997559 (396h33m12.860638884s) [noise=true , address=45.91.95.4:4014 , piece_id=V5ZNHUMTQCZ5YXOYDAYCFKBBSAUKHJ2RJ3FSQNME6IXSGFSLIJ4Q, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:38.090021875 (396h33m12.86066321s) [seg_pos={0 0}, upload=317258]: Segment reached optimal threshold of 14 pieces. Jul 9 21:43:38.090108922 (396h33m12.860750257s) [noise=true , address=45.91.95.5:4026 , piece_id=QQWGIQENU5TGALRAU4SRUFWJDQYOPW552SN46COWCRFNFBRRPBFA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xxtZ3msZnUkE4kgTFKJmprcNS986Uhi1LjwVyd1DbrTfZS52H): protocol: expected piece hash; context canceled; EOF Jul 9 21:43:38.090133739 (396h33m12.860775074s) [noise=true , address=104.36.82.38:4716 , piece_id=FK7WLNN6677LINQJ26LPGZ5X4JJ7ISGDRUHMVQTGK34OTL44EF6A, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MAHcnhvjvD74dpLS5S3m751NzfjvJ8a9ZvHwA7mdXUEXm9DAX): protocol: expected piece hash; context canceled; EOF Jul 9 21:43:38.090161603 (396h33m12.860802938s) [noise=true , address=45.91.95.3:4019 , piece_id=3XZWJLZH5RXX45ULYQLXAG4SG2RNMOGWXLNFTOVCVKZ2QUQOC6KQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SGjoGjbfugbxFgDctHygi7qo8upC463TRw8sEceUNnti1iGV1): protocol: expected piece hash; context canceled; EOF Jul 9 21:43:38.090225525 (396h33m12.86086685s) [noise=true , address=45.91.95.2:4022 , piece_id=UJ6PKLMNBID76O3ZR4SXRNLL3CCLNGTYBZSN57E2YBD57TDLKTPQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XGBvsxi6S7dz9d93CWmnQhomJVg1x5evu7DxsGsujkejBkZVN): protocol: write tcp 188.166.225.170:50926->45.91.95.2:4022: use of closed network connection; write tcp 188.166.225.170:50926->45.91.95.2:4022: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50926->45.91.95.2:4022: use of closed network connection Jul 9 21:43:38.090246715 (396h33m12.860888051s) [seg_pos={0 0}, upload=317258]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:43:38.090334674 (396h33m12.860976009s) [upload=317258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:43:38.090338631 (396h33m12.860979956s) [upload=317258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:43:38.379746682 (396h33m13.150388017s) [upload=317258]: Done uploading segments... Jul 9 21:43:38.379765919 (396h33m13.150407254s) [upload=317258]: Upload finished. err: <nil> Duration: 51.119214581s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [...X......X....X..X.........X...............X..X....X.......] duration: 31.059284206s FAIL (2025-07-09 21:46:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 31.059297802s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [....X......X....X..X.........X...............X..X....X......] duration: 40.469081087s FAIL (2025-07-09 21:45:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 40.469101767s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.X...X................X....X...............X..X.X...X...X...] duration: 21.135699781s FAIL (2025-07-09 21:45:28 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 17258]: Got next segment. Inline: false Jul 9 21:42:48.387406701 (396h32m23.158048036s) [upload=317258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:42:48.387413103 (396h32m23.158054439s) [upload=317258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:42:48.834299395 (396h32m23.60494072s) [seg_pos={0 0}, upload=317258]: Begin upload segment... Jul 9 21:42:48.834354481 (396h32m23.604995816s) [seg_pos={0 0}, upload=317258]: Done begin upload segment. Jul 9 21:42:48.834374599 (396h32m23.605015924s) [noise=true , address=77.90.31.10:4523 , piece_id=SKESIP2VBIDJIBXIEGMC2LZESW47O7WJCLSBQBSWLHYWV4PRT7FQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.834761038 (396h32m23.605402363s) [noise=true , address=77.90.31.21:4908 , piece_id=HGKISLD6DZXJ6DL4UCZWUJD2V5CTDYRX2BSKDCDJYXCQQBX6Q3RA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835061884 (396h32m23.605703219s) [noise=true , address=79.127.201.215:4345 , piece_id=WEQTKXLED3S6W4HMRFM7YUEXBWGUUMQL65E4U4FG27AVCZFRK72A, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835344354 (396h32m23.605985689s) [noise=true , address=104.36.82.23:4474 , piece_id=JJNF2423XB5JQXQJ3VWRUYGLZ2QR5SFPN7BUJO4GRPSDST6IFBCA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835641913 (396h32m23.606283248s) [noise=true , address=45.91.95.2:4022 , piece_id=UJ6PKLMNBID76O3ZR4SXRNLL3CCLNGTYBZSN57E2YBD57TDLKTPQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835925006 (396h32m23.606566341s) [noise=true , address=77.90.31.14:4667 , piece_id=3XNOC4TIZJFEMIJQIMYHYNXA3SD72IYKBVBVWZ5J7NKVG5XOQYMA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836153993 (396h32m23.606795328s) [noise=true , address=104.36.82.38:4716 , piece_id=FK7WLNN6677LINQJ26LPGZ5X4JJ7ISGDRUHMVQTGK34OTL44EF6A, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836372001 (396h32m23.607013336s) [noise=true , address=77.90.31.18:4810 , piece_id=33CXTR5YAVMPV3VGG23FMKLGAKR355O66PAHFZPKVQSFJGQIEYLQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836627199 (396h32m23.607268524s) [noise=true , address=77.90.31.16:4768 , piece_id=LOU4TXWNZRUAIB7BVVZXKEVZSBL44DKOIITBGOCPDPM73L7CYRYQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836860434 (396h32m23.607501759s) [noise=true , address=77.90.31.13:4640 , piece_id=XCMFSHLM372AAMNYL2YVGRUSGWT7RB4RE4TMZ6ONHPVNR26NKUAQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837076287 (396h32m23.607717623s) [noise=true , address=45.91.95.5:4026 , piece_id=QQWGIQENU5TGALRAU4SRUFWJDQYOPW552SN46COWCRFNFBRRPBFA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837281300 (396h32m23.607922635s) [noise=true , address=79.127.220.101:4073 , piece_id=NNWGURWIPMP67UIQGVHZPFZJBD2QFOGBQ6KZNNTHLJXCGRIDTXZQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837494658 (396h32m23.608135993s) [noise=true , address=104.36.82.18:4396 , piece_id=2DRRKEK2OLFL54G6BQIIQLEPONIHDUICJUVAEAONC2IOR22POPLA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837859957 (396h32m23.608501292s) [noise=true , address=45.91.95.4:4014 , piece_id=V5ZNHUMTQCZ5YXOYDAYCFKBBSAUKHJ2RJ3FSQNME6IXSGFSLIJ4Q, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838145814 (396h32m23.608787139s) [noise=true , address=77.90.31.18:4838 , piece_id=EGYCA5GBMRI3JPX3SHKQ57GU6ZQV6S33BJNOJVUJHESMEKZU5WTQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838416161 (396h32m23.609057486s) [noise=true , address=45.91.95.3:4019 , piece_id=3XZWJLZH5RXX45ULYQLXAG4SG2RNMOGWXLNFTOVCVKZ2QUQOC6KQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838665709 (396h32m23.609307044s) [noise=true , address=77.90.31.19:4852 , piece_id=MQ7OIWIHD642LXDOEBKZF2DTZRLR5OJT4Q37DCLLDXR7XTJVKXKA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838944833 (396h32m23.609586168s) [noise=true , address=77.90.31.12:4621 , piece_id=CL234SZAWOTG4ONGWQ2ZBXEGWUYTQQVR76BVCHAPXETRGF5TQJBQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:50.183574564 (396h32m24.954215899s) [upload=317258]: Next returned nil segment Jul 9 21:42:50.183596435 (396h32m24.95423777s) [upload=317258]: Waiting for error group managing segments... Jul 9 21:42:50.548527303 (396h32m25.319168638s) [noise=true , address=79.127.220.101:4073 , piece_id=NNWGURWIPMP67UIQGVHZPFZJBD2QFOGBQ6KZNNTHLJXCGRIDTXZQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.840602026 (396h32m25.611243361s) [noise=true , address=77.90.31.18:4838 , piece_id=EGYCA5GBMRI3JPX3SHKQ57GU6ZQV6S33BJNOJVUJHESMEKZU5WTQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.841968832 (396h32m25.612610167s) [noise=true , address=77.90.31.18:4810 , piece_id=33CXTR5YAVMPV3VGG23FMKLGAKR355O66PAHFZPKVQSFJGQIEYLQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.842708556 (396h32m25.613349951s) [noise=true , address=77.90.31.13:4640 , piece_id=XCMFSHLM372AAMNYL2YVGRUSGWT7RB4RE4TMZ6ONHPVNR26NKUAQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.843190929 (396h32m25.613832265s) [noise=true , address=77.90.31.14:4667 , piece_id=3XNOC4TIZJFEMIJQIMYHYNXA3SD72IYKBVBVWZ5J7NKVG5XOQYMA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.844590978 (396h32m25.615232313s) [noise=true , address=77.90.31.10:4523 , piece_id=SKESIP2VBIDJIBXIEGMC2LZESW47O7WJCLSBQBSWLHYWV4PRT7FQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.845022593 (396h32m25.615663928s) [noise=true , address=77.90.31.19:4852 , piece_id=MQ7OIWIHD642LXDOEBKZF2DTZRLR5OJT4Q37DCLLDXR7XTJVKXKA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.846516732 (396h32m25.617158067s) [noise=true , address=77.90.31.12:4621 , piece_id=CL234SZAWOTG4ONGWQ2ZBXEGWUYTQQVR76BVCHAPXETRGF5TQJBQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.847220148 (396h32m25.617861473s) [noise=true , address=77.90.31.21:4908 , piece_id=HGKISLD6DZXJ6DL4UCZWUJD2V5CTDYRX2BSKDCDJYXCQQBX6Q3RA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.847310801 (396h32m25.617952126s) [noise=true , address=77.90.31.16:4768 , piece_id=LOU4TXWNZRUAIB7BVVZXKEVZSBL44DKOIITBGOCPDPM73L7CYRYQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:55.315575878 (396h32m30.086217213s) [noise=true , address=79.127.201.215:4345 , piece_id=WEQTKXLED3S6W4HMRFM7YUEXBWGUUMQL65E4U4FG27AVCZFRK72A, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:12.086886240 (396h32m46.857527576s) [noise=true , address=104.36.82.23:4474 , piece_id=JJNF2423XB5JQXQJ3VWRUYGLZ2QR5SFPN7BUJO4GRPSDST6IFBCA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:29.357614308 (396h33m4.128255643s) [noise=true , address=104.36.82.18:4396 , piece_id=2DRRKEK2OLFL54G6BQIIQLEPONIHDUICJUVAEAONC2IOR22POPLA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:38.089997559 (396h33m12.860638884s) [noise=true , address=45.91.95.4:4014 , piece_id=V5ZNHUMTQCZ5YXOYDAYCFKBBSAUKHJ2RJ3FSQNME6IXSGFSLIJ4Q, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:38.090021875 (396h33m12.86066321s) [seg_pos={0 0}, upload=317258]: Segment reached optimal threshold of 14 pieces. Jul 9 21:43:38.090108922 (396h33m12.860750257s) [noise=true , address=45.91.95.5:4026 , piece_id=QQWGIQENU5TGALRAU4SRUFWJDQYOPW552SN46COWCRFNFBRRPBFA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xxtZ3msZnUkE4kgTFKJmprcNS986Uhi1LjwVyd1DbrTfZS52H): protocol: expected piece hash; context canceled; EOF Jul 9 21:43:38.090133739 (396h33m12.860775074s) [noise=true , address=104.36.82.38:4716 , piece_id=FK7WLNN6677LINQJ26LPGZ5X4JJ7ISGDRUHMVQTGK34OTL44EF6A, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MAHcnhvjvD74dpLS5S3m751NzfjvJ8a9ZvHwA7mdXUEXm9DAX): protocol: expected piece hash; context canceled; EOF Jul 9 21:43:38.090161603 (396h33m12.860802938s) [noise=true , address=45.91.95.3:4019 , piece_id=3XZWJLZH5RXX45ULYQLXAG4SG2RNMOGWXLNFTOVCVKZ2QUQOC6KQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SGjoGjbfugbxFgDctHygi7qo8upC463TRw8sEceUNnti1iGV1): protocol: expected piece hash; context canceled; EOF Jul 9 21:43:38.090225525 (396h33m12.86086685s) [noise=true , address=45.91.95.2:4022 , piece_id=UJ6PKLMNBID76O3ZR4SXRNLL3CCLNGTYBZSN57E2YBD57TDLKTPQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XGBvsxi6S7dz9d93CWmnQhomJVg1x5evu7DxsGsujkejBkZVN): protocol: write tcp 188.166.225.170:50926->45.91.95.2:4022: use of closed network connection; write tcp 188.166.225.170:50926->45.91.95.2:4022: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50926->45.91.95.2:4022: use of closed network connection Jul 9 21:43:38.090246715 (396h33m12.860888051s) [seg_pos={0 0}, upload=317258]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:43:38.090334674 (396h33m12.860976009s) [upload=317258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:43:38.090338631 (396h33m12.860979956s) [upload=317258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:43:38.379746682 (396h33m13.150388017s) [upload=317258]: Done uploading segments... Jul 9 21:43:38.379765919 (396h33m13.150407254s) [upload=317258]: Upload finished. err: <nil> Duration: 21.135715813s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [..X...X................X....X...............X..X.X...X...X..] duration: 40.453250874s FAIL (2025-07-09 21:45:07 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 17258]: Got next segment. Inline: false Jul 9 21:42:48.387406701 (396h32m23.158048036s) [upload=317258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:42:48.387413103 (396h32m23.158054439s) [upload=317258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:42:48.834299395 (396h32m23.60494072s) [seg_pos={0 0}, upload=317258]: Begin upload segment... Jul 9 21:42:48.834354481 (396h32m23.604995816s) [seg_pos={0 0}, upload=317258]: Done begin upload segment. Jul 9 21:42:48.834374599 (396h32m23.605015924s) [noise=true , address=77.90.31.10:4523 , piece_id=SKESIP2VBIDJIBXIEGMC2LZESW47O7WJCLSBQBSWLHYWV4PRT7FQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.834761038 (396h32m23.605402363s) [noise=true , address=77.90.31.21:4908 , piece_id=HGKISLD6DZXJ6DL4UCZWUJD2V5CTDYRX2BSKDCDJYXCQQBX6Q3RA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835061884 (396h32m23.605703219s) [noise=true , address=79.127.201.215:4345 , piece_id=WEQTKXLED3S6W4HMRFM7YUEXBWGUUMQL65E4U4FG27AVCZFRK72A, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835344354 (396h32m23.605985689s) [noise=true , address=104.36.82.23:4474 , piece_id=JJNF2423XB5JQXQJ3VWRUYGLZ2QR5SFPN7BUJO4GRPSDST6IFBCA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835641913 (396h32m23.606283248s) [noise=true , address=45.91.95.2:4022 , piece_id=UJ6PKLMNBID76O3ZR4SXRNLL3CCLNGTYBZSN57E2YBD57TDLKTPQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835925006 (396h32m23.606566341s) [noise=true , address=77.90.31.14:4667 , piece_id=3XNOC4TIZJFEMIJQIMYHYNXA3SD72IYKBVBVWZ5J7NKVG5XOQYMA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836153993 (396h32m23.606795328s) [noise=true , address=104.36.82.38:4716 , piece_id=FK7WLNN6677LINQJ26LPGZ5X4JJ7ISGDRUHMVQTGK34OTL44EF6A, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836372001 (396h32m23.607013336s) [noise=true , address=77.90.31.18:4810 , piece_id=33CXTR5YAVMPV3VGG23FMKLGAKR355O66PAHFZPKVQSFJGQIEYLQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836627199 (396h32m23.607268524s) [noise=true , address=77.90.31.16:4768 , piece_id=LOU4TXWNZRUAIB7BVVZXKEVZSBL44DKOIITBGOCPDPM73L7CYRYQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836860434 (396h32m23.607501759s) [noise=true , address=77.90.31.13:4640 , piece_id=XCMFSHLM372AAMNYL2YVGRUSGWT7RB4RE4TMZ6ONHPVNR26NKUAQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837076287 (396h32m23.607717623s) [noise=true , address=45.91.95.5:4026 , piece_id=QQWGIQENU5TGALRAU4SRUFWJDQYOPW552SN46COWCRFNFBRRPBFA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837281300 (396h32m23.607922635s) [noise=true , address=79.127.220.101:4073 , piece_id=NNWGURWIPMP67UIQGVHZPFZJBD2QFOGBQ6KZNNTHLJXCGRIDTXZQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837494658 (396h32m23.608135993s) [noise=true , address=104.36.82.18:4396 , piece_id=2DRRKEK2OLFL54G6BQIIQLEPONIHDUICJUVAEAONC2IOR22POPLA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837859957 (396h32m23.608501292s) [noise=true , address=45.91.95.4:4014 , piece_id=V5ZNHUMTQCZ5YXOYDAYCFKBBSAUKHJ2RJ3FSQNME6IXSGFSLIJ4Q, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838145814 (396h32m23.608787139s) [noise=true , address=77.90.31.18:4838 , piece_id=EGYCA5GBMRI3JPX3SHKQ57GU6ZQV6S33BJNOJVUJHESMEKZU5WTQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838416161 (396h32m23.609057486s) [noise=true , address=45.91.95.3:4019 , piece_id=3XZWJLZH5RXX45ULYQLXAG4SG2RNMOGWXLNFTOVCVKZ2QUQOC6KQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838665709 (396h32m23.609307044s) [noise=true , address=77.90.31.19:4852 , piece_id=MQ7OIWIHD642LXDOEBKZF2DTZRLR5OJT4Q37DCLLDXR7XTJVKXKA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838944833 (396h32m23.609586168s) [noise=true , address=77.90.31.12:4621 , piece_id=CL234SZAWOTG4ONGWQ2ZBXEGWUYTQQVR76BVCHAPXETRGF5TQJBQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:50.183574564 (396h32m24.954215899s) [upload=317258]: Next returned nil segment Jul 9 21:42:50.183596435 (396h32m24.95423777s) [upload=317258]: Waiting for error group managing segments... Jul 9 21:42:50.548527303 (396h32m25.319168638s) [noise=true , address=79.127.220.101:4073 , piece_id=NNWGURWIPMP67UIQGVHZPFZJBD2QFOGBQ6KZNNTHLJXCGRIDTXZQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.840602026 (396h32m25.611243361s) [noise=true , address=77.90.31.18:4838 , piece_id=EGYCA5GBMRI3JPX3SHKQ57GU6ZQV6S33BJNOJVUJHESMEKZU5WTQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.841968832 (396h32m25.612610167s) [noise=true , address=77.90.31.18:4810 , piece_id=33CXTR5YAVMPV3VGG23FMKLGAKR355O66PAHFZPKVQSFJGQIEYLQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.842708556 (396h32m25.613349951s) [noise=true , address=77.90.31.13:4640 , piece_id=XCMFSHLM372AAMNYL2YVGRUSGWT7RB4RE4TMZ6ONHPVNR26NKUAQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.843190929 (396h32m25.613832265s) [noise=true , address=77.90.31.14:4667 , piece_id=3XNOC4TIZJFEMIJQIMYHYNXA3SD72IYKBVBVWZ5J7NKVG5XOQYMA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.844590978 (396h32m25.615232313s) [noise=true , address=77.90.31.10:4523 , piece_id=SKESIP2VBIDJIBXIEGMC2LZESW47O7WJCLSBQBSWLHYWV4PRT7FQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.845022593 (396h32m25.615663928s) [noise=true , address=77.90.31.19:4852 , piece_id=MQ7OIWIHD642LXDOEBKZF2DTZRLR5OJT4Q37DCLLDXR7XTJVKXKA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.846516732 (396h32m25.617158067s) [noise=true , address=77.90.31.12:4621 , piece_id=CL234SZAWOTG4ONGWQ2ZBXEGWUYTQQVR76BVCHAPXETRGF5TQJBQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.847220148 (396h32m25.617861473s) [noise=true , address=77.90.31.21:4908 , piece_id=HGKISLD6DZXJ6DL4UCZWUJD2V5CTDYRX2BSKDCDJYXCQQBX6Q3RA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.847310801 (396h32m25.617952126s) [noise=true , address=77.90.31.16:4768 , piece_id=LOU4TXWNZRUAIB7BVVZXKEVZSBL44DKOIITBGOCPDPM73L7CYRYQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:55.315575878 (396h32m30.086217213s) [noise=true , address=79.127.201.215:4345 , piece_id=WEQTKXLED3S6W4HMRFM7YUEXBWGUUMQL65E4U4FG27AVCZFRK72A, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:12.086886240 (396h32m46.857527576s) [noise=true , address=104.36.82.23:4474 , piece_id=JJNF2423XB5JQXQJ3VWRUYGLZ2QR5SFPN7BUJO4GRPSDST6IFBCA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:29.357614308 (396h33m4.128255643s) [noise=true , address=104.36.82.18:4396 , piece_id=2DRRKEK2OLFL54G6BQIIQLEPONIHDUICJUVAEAONC2IOR22POPLA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:38.089997559 (396h33m12.860638884s) [noise=true , address=45.91.95.4:4014 , piece_id=V5ZNHUMTQCZ5YXOYDAYCFKBBSAUKHJ2RJ3FSQNME6IXSGFSLIJ4Q, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:38.090021875 (396h33m12.86066321s) [seg_pos={0 0}, upload=317258]: Segment reached optimal threshold of 14 pieces. Jul 9 21:43:38.090108922 (396h33m12.860750257s) [noise=true , address=45.91.95.5:4026 , piece_id=QQWGIQENU5TGALRAU4SRUFWJDQYOPW552SN46COWCRFNFBRRPBFA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xxtZ3msZnUkE4kgTFKJmprcNS986Uhi1LjwVyd1DbrTfZS52H): protocol: expected piece hash; context canceled; EOF Jul 9 21:43:38.090133739 (396h33m12.860775074s) [noise=true , address=104.36.82.38:4716 , piece_id=FK7WLNN6677LINQJ26LPGZ5X4JJ7ISGDRUHMVQTGK34OTL44EF6A, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MAHcnhvjvD74dpLS5S3m751NzfjvJ8a9ZvHwA7mdXUEXm9DAX): protocol: expected piece hash; context canceled; EOF Jul 9 21:43:38.090161603 (396h33m12.860802938s) [noise=true , address=45.91.95.3:4019 , piece_id=3XZWJLZH5RXX45ULYQLXAG4SG2RNMOGWXLNFTOVCVKZ2QUQOC6KQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SGjoGjbfugbxFgDctHygi7qo8upC463TRw8sEceUNnti1iGV1): protocol: expected piece hash; context canceled; EOF Jul 9 21:43:38.090225525 (396h33m12.86086685s) [noise=true , address=45.91.95.2:4022 , piece_id=UJ6PKLMNBID76O3ZR4SXRNLL3CCLNGTYBZSN57E2YBD57TDLKTPQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XGBvsxi6S7dz9d93CWmnQhomJVg1x5evu7DxsGsujkejBkZVN): protocol: write tcp 188.166.225.170:50926->45.91.95.2:4022: use of closed network connection; write tcp 188.166.225.170:50926->45.91.95.2:4022: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50926->45.91.95.2:4022: use of closed network connection Jul 9 21:43:38.090246715 (396h33m12.860888051s) [seg_pos={0 0}, upload=317258]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:43:38.090334674 (396h33m12.860976009s) [upload=317258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:43:38.090338631 (396h33m12.860979956s) [upload=317258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:43:38.379746682 (396h33m13.150388017s) [upload=317258]: Done uploading segments... Jul 9 21:43:38.379765919 (396h33m13.150407254s) [upload=317258]: Upload finished. err: <nil> Duration: 40.45326996s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.....X......X....X..X.........X...............X..X....X.....] duration: 53.010221732s FAIL (2025-07-09 21:45:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 53.010232592s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [...X...X................X....X...............X..X.X...X...X.] duration: 38.662139123s FAIL (2025-07-09 21:44:26 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 17258]: Got next segment. Inline: false Jul 9 21:42:48.387406701 (396h32m23.158048036s) [upload=317258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:42:48.387413103 (396h32m23.158054439s) [upload=317258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:42:48.834299395 (396h32m23.60494072s) [seg_pos={0 0}, upload=317258]: Begin upload segment... Jul 9 21:42:48.834354481 (396h32m23.604995816s) [seg_pos={0 0}, upload=317258]: Done begin upload segment. Jul 9 21:42:48.834374599 (396h32m23.605015924s) [noise=true , address=77.90.31.10:4523 , piece_id=SKESIP2VBIDJIBXIEGMC2LZESW47O7WJCLSBQBSWLHYWV4PRT7FQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.834761038 (396h32m23.605402363s) [noise=true , address=77.90.31.21:4908 , piece_id=HGKISLD6DZXJ6DL4UCZWUJD2V5CTDYRX2BSKDCDJYXCQQBX6Q3RA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835061884 (396h32m23.605703219s) [noise=true , address=79.127.201.215:4345 , piece_id=WEQTKXLED3S6W4HMRFM7YUEXBWGUUMQL65E4U4FG27AVCZFRK72A, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835344354 (396h32m23.605985689s) [noise=true , address=104.36.82.23:4474 , piece_id=JJNF2423XB5JQXQJ3VWRUYGLZ2QR5SFPN7BUJO4GRPSDST6IFBCA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835641913 (396h32m23.606283248s) [noise=true , address=45.91.95.2:4022 , piece_id=UJ6PKLMNBID76O3ZR4SXRNLL3CCLNGTYBZSN57E2YBD57TDLKTPQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835925006 (396h32m23.606566341s) [noise=true , address=77.90.31.14:4667 , piece_id=3XNOC4TIZJFEMIJQIMYHYNXA3SD72IYKBVBVWZ5J7NKVG5XOQYMA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836153993 (396h32m23.606795328s) [noise=true , address=104.36.82.38:4716 , piece_id=FK7WLNN6677LINQJ26LPGZ5X4JJ7ISGDRUHMVQTGK34OTL44EF6A, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836372001 (396h32m23.607013336s) [noise=true , address=77.90.31.18:4810 , piece_id=33CXTR5YAVMPV3VGG23FMKLGAKR355O66PAHFZPKVQSFJGQIEYLQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836627199 (396h32m23.607268524s) [noise=true , address=77.90.31.16:4768 , piece_id=LOU4TXWNZRUAIB7BVVZXKEVZSBL44DKOIITBGOCPDPM73L7CYRYQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836860434 (396h32m23.607501759s) [noise=true , address=77.90.31.13:4640 , piece_id=XCMFSHLM372AAMNYL2YVGRUSGWT7RB4RE4TMZ6ONHPVNR26NKUAQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837076287 (396h32m23.607717623s) [noise=true , address=45.91.95.5:4026 , piece_id=QQWGIQENU5TGALRAU4SRUFWJDQYOPW552SN46COWCRFNFBRRPBFA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837281300 (396h32m23.607922635s) [noise=true , address=79.127.220.101:4073 , piece_id=NNWGURWIPMP67UIQGVHZPFZJBD2QFOGBQ6KZNNTHLJXCGRIDTXZQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837494658 (396h32m23.608135993s) [noise=true , address=104.36.82.18:4396 , piece_id=2DRRKEK2OLFL54G6BQIIQLEPONIHDUICJUVAEAONC2IOR22POPLA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837859957 (396h32m23.608501292s) [noise=true , address=45.91.95.4:4014 , piece_id=V5ZNHUMTQCZ5YXOYDAYCFKBBSAUKHJ2RJ3FSQNME6IXSGFSLIJ4Q, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838145814 (396h32m23.608787139s) [noise=true , address=77.90.31.18:4838 , piece_id=EGYCA5GBMRI3JPX3SHKQ57GU6ZQV6S33BJNOJVUJHESMEKZU5WTQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838416161 (396h32m23.609057486s) [noise=true , address=45.91.95.3:4019 , piece_id=3XZWJLZH5RXX45ULYQLXAG4SG2RNMOGWXLNFTOVCVKZ2QUQOC6KQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838665709 (396h32m23.609307044s) [noise=true , address=77.90.31.19:4852 , piece_id=MQ7OIWIHD642LXDOEBKZF2DTZRLR5OJT4Q37DCLLDXR7XTJVKXKA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838944833 (396h32m23.609586168s) [noise=true , address=77.90.31.12:4621 , piece_id=CL234SZAWOTG4ONGWQ2ZBXEGWUYTQQVR76BVCHAPXETRGF5TQJBQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:50.183574564 (396h32m24.954215899s) [upload=317258]: Next returned nil segment Jul 9 21:42:50.183596435 (396h32m24.95423777s) [upload=317258]: Waiting for error group managing segments... Jul 9 21:42:50.548527303 (396h32m25.319168638s) [noise=true , address=79.127.220.101:4073 , piece_id=NNWGURWIPMP67UIQGVHZPFZJBD2QFOGBQ6KZNNTHLJXCGRIDTXZQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.840602026 (396h32m25.611243361s) [noise=true , address=77.90.31.18:4838 , piece_id=EGYCA5GBMRI3JPX3SHKQ57GU6ZQV6S33BJNOJVUJHESMEKZU5WTQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.841968832 (396h32m25.612610167s) [noise=true , address=77.90.31.18:4810 , piece_id=33CXTR5YAVMPV3VGG23FMKLGAKR355O66PAHFZPKVQSFJGQIEYLQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.842708556 (396h32m25.613349951s) [noise=true , address=77.90.31.13:4640 , piece_id=XCMFSHLM372AAMNYL2YVGRUSGWT7RB4RE4TMZ6ONHPVNR26NKUAQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.843190929 (396h32m25.613832265s) [noise=true , address=77.90.31.14:4667 , piece_id=3XNOC4TIZJFEMIJQIMYHYNXA3SD72IYKBVBVWZ5J7NKVG5XOQYMA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.844590978 (396h32m25.615232313s) [noise=true , address=77.90.31.10:4523 , piece_id=SKESIP2VBIDJIBXIEGMC2LZESW47O7WJCLSBQBSWLHYWV4PRT7FQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.845022593 (396h32m25.615663928s) [noise=true , address=77.90.31.19:4852 , piece_id=MQ7OIWIHD642LXDOEBKZF2DTZRLR5OJT4Q37DCLLDXR7XTJVKXKA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.846516732 (396h32m25.617158067s) [noise=true , address=77.90.31.12:4621 , piece_id=CL234SZAWOTG4ONGWQ2ZBXEGWUYTQQVR76BVCHAPXETRGF5TQJBQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.847220148 (396h32m25.617861473s) [noise=true , address=77.90.31.21:4908 , piece_id=HGKISLD6DZXJ6DL4UCZWUJD2V5CTDYRX2BSKDCDJYXCQQBX6Q3RA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.847310801 (396h32m25.617952126s) [noise=true , address=77.90.31.16:4768 , piece_id=LOU4TXWNZRUAIB7BVVZXKEVZSBL44DKOIITBGOCPDPM73L7CYRYQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:55.315575878 (396h32m30.086217213s) [noise=true , address=79.127.201.215:4345 , piece_id=WEQTKXLED3S6W4HMRFM7YUEXBWGUUMQL65E4U4FG27AVCZFRK72A, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:12.086886240 (396h32m46.857527576s) [noise=true , address=104.36.82.23:4474 , piece_id=JJNF2423XB5JQXQJ3VWRUYGLZ2QR5SFPN7BUJO4GRPSDST6IFBCA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:29.357614308 (396h33m4.128255643s) [noise=true , address=104.36.82.18:4396 , piece_id=2DRRKEK2OLFL54G6BQIIQLEPONIHDUICJUVAEAONC2IOR22POPLA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:38.089997559 (396h33m12.860638884s) [noise=true , address=45.91.95.4:4014 , piece_id=V5ZNHUMTQCZ5YXOYDAYCFKBBSAUKHJ2RJ3FSQNME6IXSGFSLIJ4Q, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:38.090021875 (396h33m12.86066321s) [seg_pos={0 0}, upload=317258]: Segment reached optimal threshold of 14 pieces. Jul 9 21:43:38.090108922 (396h33m12.860750257s) [noise=true , address=45.91.95.5:4026 , piece_id=QQWGIQENU5TGALRAU4SRUFWJDQYOPW552SN46COWCRFNFBRRPBFA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xxtZ3msZnUkE4kgTFKJmprcNS986Uhi1LjwVyd1DbrTfZS52H): protocol: expected piece hash; context canceled; EOF Jul 9 21:43:38.090133739 (396h33m12.860775074s) [noise=true , address=104.36.82.38:4716 , piece_id=FK7WLNN6677LINQJ26LPGZ5X4JJ7ISGDRUHMVQTGK34OTL44EF6A, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MAHcnhvjvD74dpLS5S3m751NzfjvJ8a9ZvHwA7mdXUEXm9DAX): protocol: expected piece hash; context canceled; EOF Jul 9 21:43:38.090161603 (396h33m12.860802938s) [noise=true , address=45.91.95.3:4019 , piece_id=3XZWJLZH5RXX45ULYQLXAG4SG2RNMOGWXLNFTOVCVKZ2QUQOC6KQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SGjoGjbfugbxFgDctHygi7qo8upC463TRw8sEceUNnti1iGV1): protocol: expected piece hash; context canceled; EOF Jul 9 21:43:38.090225525 (396h33m12.86086685s) [noise=true , address=45.91.95.2:4022 , piece_id=UJ6PKLMNBID76O3ZR4SXRNLL3CCLNGTYBZSN57E2YBD57TDLKTPQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XGBvsxi6S7dz9d93CWmnQhomJVg1x5evu7DxsGsujkejBkZVN): protocol: write tcp 188.166.225.170:50926->45.91.95.2:4022: use of closed network connection; write tcp 188.166.225.170:50926->45.91.95.2:4022: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50926->45.91.95.2:4022: use of closed network connection Jul 9 21:43:38.090246715 (396h33m12.860888051s) [seg_pos={0 0}, upload=317258]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:43:38.090334674 (396h33m12.860976009s) [upload=317258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:43:38.090338631 (396h33m12.860979956s) [upload=317258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:43:38.379746682 (396h33m13.150388017s) [upload=317258]: Done uploading segments... Jul 9 21:43:38.379765919 (396h33m13.150407254s) [upload=317258]: Upload finished. err: <nil> Duration: 38.662156465s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [......X......X....X..X.........X...............X..X....X....] duration: 54.550651669s FAIL (2025-07-09 21:44:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 54.550663162s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [....X...X................X....X...............X..X.X...X...X] duration: 1m0.002472839s FAIL (2025-07-09 21:43:47 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 17258]: Got next segment. Inline: false Jul 9 21:42:48.387406701 (396h32m23.158048036s) [upload=317258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:42:48.387413103 (396h32m23.158054439s) [upload=317258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:42:48.834299395 (396h32m23.60494072s) [seg_pos={0 0}, upload=317258]: Begin upload segment... Jul 9 21:42:48.834354481 (396h32m23.604995816s) [seg_pos={0 0}, upload=317258]: Done begin upload segment. Jul 9 21:42:48.834374599 (396h32m23.605015924s) [noise=true , address=77.90.31.10:4523 , piece_id=SKESIP2VBIDJIBXIEGMC2LZESW47O7WJCLSBQBSWLHYWV4PRT7FQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.834761038 (396h32m23.605402363s) [noise=true , address=77.90.31.21:4908 , piece_id=HGKISLD6DZXJ6DL4UCZWUJD2V5CTDYRX2BSKDCDJYXCQQBX6Q3RA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835061884 (396h32m23.605703219s) [noise=true , address=79.127.201.215:4345 , piece_id=WEQTKXLED3S6W4HMRFM7YUEXBWGUUMQL65E4U4FG27AVCZFRK72A, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835344354 (396h32m23.605985689s) [noise=true , address=104.36.82.23:4474 , piece_id=JJNF2423XB5JQXQJ3VWRUYGLZ2QR5SFPN7BUJO4GRPSDST6IFBCA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835641913 (396h32m23.606283248s) [noise=true , address=45.91.95.2:4022 , piece_id=UJ6PKLMNBID76O3ZR4SXRNLL3CCLNGTYBZSN57E2YBD57TDLKTPQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835925006 (396h32m23.606566341s) [noise=true , address=77.90.31.14:4667 , piece_id=3XNOC4TIZJFEMIJQIMYHYNXA3SD72IYKBVBVWZ5J7NKVG5XOQYMA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836153993 (396h32m23.606795328s) [noise=true , address=104.36.82.38:4716 , piece_id=FK7WLNN6677LINQJ26LPGZ5X4JJ7ISGDRUHMVQTGK34OTL44EF6A, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836372001 (396h32m23.607013336s) [noise=true , address=77.90.31.18:4810 , piece_id=33CXTR5YAVMPV3VGG23FMKLGAKR355O66PAHFZPKVQSFJGQIEYLQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836627199 (396h32m23.607268524s) [noise=true , address=77.90.31.16:4768 , piece_id=LOU4TXWNZRUAIB7BVVZXKEVZSBL44DKOIITBGOCPDPM73L7CYRYQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836860434 (396h32m23.607501759s) [noise=true , address=77.90.31.13:4640 , piece_id=XCMFSHLM372AAMNYL2YVGRUSGWT7RB4RE4TMZ6ONHPVNR26NKUAQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837076287 (396h32m23.607717623s) [noise=true , address=45.91.95.5:4026 , piece_id=QQWGIQENU5TGALRAU4SRUFWJDQYOPW552SN46COWCRFNFBRRPBFA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837281300 (396h32m23.607922635s) [noise=true , address=79.127.220.101:4073 , piece_id=NNWGURWIPMP67UIQGVHZPFZJBD2QFOGBQ6KZNNTHLJXCGRIDTXZQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837494658 (396h32m23.608135993s) [noise=true , address=104.36.82.18:4396 , piece_id=2DRRKEK2OLFL54G6BQIIQLEPONIHDUICJUVAEAONC2IOR22POPLA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837859957 (396h32m23.608501292s) [noise=true , address=45.91.95.4:4014 , piece_id=V5ZNHUMTQCZ5YXOYDAYCFKBBSAUKHJ2RJ3FSQNME6IXSGFSLIJ4Q, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838145814 (396h32m23.608787139s) [noise=true , address=77.90.31.18:4838 , piece_id=EGYCA5GBMRI3JPX3SHKQ57GU6ZQV6S33BJNOJVUJHESMEKZU5WTQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838416161 (396h32m23.609057486s) [noise=true , address=45.91.95.3:4019 , piece_id=3XZWJLZH5RXX45ULYQLXAG4SG2RNMOGWXLNFTOVCVKZ2QUQOC6KQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838665709 (396h32m23.609307044s) [noise=true , address=77.90.31.19:4852 , piece_id=MQ7OIWIHD642LXDOEBKZF2DTZRLR5OJT4Q37DCLLDXR7XTJVKXKA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838944833 (396h32m23.609586168s) [noise=true , address=77.90.31.12:4621 , piece_id=CL234SZAWOTG4ONGWQ2ZBXEGWUYTQQVR76BVCHAPXETRGF5TQJBQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:50.183574564 (396h32m24.954215899s) [upload=317258]: Next returned nil segment Jul 9 21:42:50.183596435 (396h32m24.95423777s) [upload=317258]: Waiting for error group managing segments... Jul 9 21:42:50.548527303 (396h32m25.319168638s) [noise=true , address=79.127.220.101:4073 , piece_id=NNWGURWIPMP67UIQGVHZPFZJBD2QFOGBQ6KZNNTHLJXCGRIDTXZQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.840602026 (396h32m25.611243361s) [noise=true , address=77.90.31.18:4838 , piece_id=EGYCA5GBMRI3JPX3SHKQ57GU6ZQV6S33BJNOJVUJHESMEKZU5WTQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.841968832 (396h32m25.612610167s) [noise=true , address=77.90.31.18:4810 , piece_id=33CXTR5YAVMPV3VGG23FMKLGAKR355O66PAHFZPKVQSFJGQIEYLQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.842708556 (396h32m25.613349951s) [noise=true , address=77.90.31.13:4640 , piece_id=XCMFSHLM372AAMNYL2YVGRUSGWT7RB4RE4TMZ6ONHPVNR26NKUAQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.843190929 (396h32m25.613832265s) [noise=true , address=77.90.31.14:4667 , piece_id=3XNOC4TIZJFEMIJQIMYHYNXA3SD72IYKBVBVWZ5J7NKVG5XOQYMA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.844590978 (396h32m25.615232313s) [noise=true , address=77.90.31.10:4523 , piece_id=SKESIP2VBIDJIBXIEGMC2LZESW47O7WJCLSBQBSWLHYWV4PRT7FQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.845022593 (396h32m25.615663928s) [noise=true , address=77.90.31.19:4852 , piece_id=MQ7OIWIHD642LXDOEBKZF2DTZRLR5OJT4Q37DCLLDXR7XTJVKXKA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.846516732 (396h32m25.617158067s) [noise=true , address=77.90.31.12:4621 , piece_id=CL234SZAWOTG4ONGWQ2ZBXEGWUYTQQVR76BVCHAPXETRGF5TQJBQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.847220148 (396h32m25.617861473s) [noise=true , address=77.90.31.21:4908 , piece_id=HGKISLD6DZXJ6DL4UCZWUJD2V5CTDYRX2BSKDCDJYXCQQBX6Q3RA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.847310801 (396h32m25.617952126s) [noise=true , address=77.90.31.16:4768 , piece_id=LOU4TXWNZRUAIB7BVVZXKEVZSBL44DKOIITBGOCPDPM73L7CYRYQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:55.315575878 (396h32m30.086217213s) [noise=true , address=79.127.201.215:4345 , piece_id=WEQTKXLED3S6W4HMRFM7YUEXBWGUUMQL65E4U4FG27AVCZFRK72A, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:12.086886240 (396h32m46.857527576s) [noise=true , address=104.36.82.23:4474 , piece_id=JJNF2423XB5JQXQJ3VWRUYGLZ2QR5SFPN7BUJO4GRPSDST6IFBCA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:29.357614308 (396h33m4.128255643s) [noise=true , address=104.36.82.18:4396 , piece_id=2DRRKEK2OLFL54G6BQIIQLEPONIHDUICJUVAEAONC2IOR22POPLA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:38.089997559 (396h33m12.860638884s) [noise=true , address=45.91.95.4:4014 , piece_id=V5ZNHUMTQCZ5YXOYDAYCFKBBSAUKHJ2RJ3FSQNME6IXSGFSLIJ4Q, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:38.090021875 (396h33m12.86066321s) [seg_pos={0 0}, upload=317258]: Segment reached optimal threshold of 14 pieces. Jul 9 21:43:38.090108922 (396h33m12.860750257s) [noise=true , address=45.91.95.5:4026 , piece_id=QQWGIQENU5TGALRAU4SRUFWJDQYOPW552SN46COWCRFNFBRRPBFA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xxtZ3msZnUkE4kgTFKJmprcNS986Uhi1LjwVyd1DbrTfZS52H): protocol: expected piece hash; context canceled; EOF Jul 9 21:43:38.090133739 (396h33m12.860775074s) [noise=true , address=104.36.82.38:4716 , piece_id=FK7WLNN6677LINQJ26LPGZ5X4JJ7ISGDRUHMVQTGK34OTL44EF6A, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MAHcnhvjvD74dpLS5S3m751NzfjvJ8a9ZvHwA7mdXUEXm9DAX): protocol: expected piece hash; context canceled; EOF Jul 9 21:43:38.090161603 (396h33m12.860802938s) [noise=true , address=45.91.95.3:4019 , piece_id=3XZWJLZH5RXX45ULYQLXAG4SG2RNMOGWXLNFTOVCVKZ2QUQOC6KQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SGjoGjbfugbxFgDctHygi7qo8upC463TRw8sEceUNnti1iGV1): protocol: expected piece hash; context canceled; EOF Jul 9 21:43:38.090225525 (396h33m12.86086685s) [noise=true , address=45.91.95.2:4022 , piece_id=UJ6PKLMNBID76O3ZR4SXRNLL3CCLNGTYBZSN57E2YBD57TDLKTPQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XGBvsxi6S7dz9d93CWmnQhomJVg1x5evu7DxsGsujkejBkZVN): protocol: write tcp 188.166.225.170:50926->45.91.95.2:4022: use of closed network connection; write tcp 188.166.225.170:50926->45.91.95.2:4022: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50926->45.91.95.2:4022: use of closed network connection Jul 9 21:43:38.090246715 (396h33m12.860888051s) [seg_pos={0 0}, upload=317258]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:43:38.090334674 (396h33m12.860976009s) [upload=317258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:43:38.090338631 (396h33m12.860979956s) [upload=317258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:43:38.379746682 (396h33m13.150388017s) [upload=317258]: Done uploading segments... Jul 9 21:43:38.379765919 (396h33m13.150407254s) [upload=317258]: Upload finished. err: <nil> Duration: 1m0.002492859s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.......X......X....X..X.........X...............X..X....X...] duration: 54.018072407s FAIL (2025-07-09 21:43:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 54.018098147s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.....X...X................X....X...............X..X.X...X...] duration: 31.217200502s FAIL (2025-07-09 21:42:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 727635s) [upload=317212]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:39:32.743915214 (396h29m7.514556539s) [seg_pos={0 0}, upload=317212]: Begin upload segment... Jul 9 21:39:32.743955091 (396h29m7.514596416s) [seg_pos={0 0}, upload=317212]: Done begin upload segment. Jul 9 21:39:32.743976182 (396h29m7.514617517s) [noise=true , address=77.90.31.10:4535 , piece_id=KJER6ZFIVPUXCK6WEI5TX5UX3LU3BAO25MYTQW3A7TIHXHLK4ECA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.744312375 (396h29m7.51495371s) [noise=true , address=104.36.82.19:4425 , piece_id=GMMOB7UIYDEB37LG5WCBGSGIH6IT652DHXBTH66MLWHBFF3IQTBA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.744344205 (396h29m7.51498554s) [noise=true , address=77.90.31.12:4616 , piece_id=WYOLXV2SNY3PF3VWQS76KBGNRTOPKDR2DDSMZGMAQCVWKM3JRG7A, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.744776322 (396h29m7.515417667s) [noise=true , address=104.36.82.21:4473 , piece_id=65CNRWVKLCWLW6IVVIMV2O3U3ONRWJWTPZ5WA6AB7W7CSKHJJGHQ, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.744879529 (396h29m7.515520864s) [noise=true , address=77.90.31.18:4831 , piece_id=4CSJQKEAD6XPOQMIIOG6SCTBEBNUPCCED3DL6R367YFKHTKXFAVA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.745527850 (396h29m7.516169185s) [noise=true , address=64.49.13.137:4502 , piece_id=T4BAEHJ2GEW7YVSNSXTCFGX2ANUGGUYMAU7A2AIILIUBHS2AQMJQ, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.745755746 (396h29m7.516397081s) [noise=true , address=45.91.95.3:4004 , piece_id=V3MVFLFQMXW3KO4PSEL3BQUEOWL4OR5HPPWRONMEVFPHOORC5CYQ, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.746225264 (396h29m7.516866599s) [noise=true , address=45.91.95.2:4012 , piece_id=ZNJPEWCNHTPOFCWXHCCTTOCFNDJSZE2NRXJK2SWE5DX2YSSIT2UQ, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.746255782 (396h29m7.516897117s) [noise=true , address=77.90.31.17:4787 , piece_id=SIKBA3Z4FKNVGMLHVE3ODZP2IJU42LJDA26UQQZRGF45GYOICLSA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.746836593 (396h29m7.517477928s) [noise=true , address=45.91.95.4:4036 , piece_id=GSSBZKTAVOFGTBOD2SQ3C5Y6P3KYWZD7UJFJ4L2ZA2EYZ7DQAZKA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.747161365 (396h29m7.5178027s) [noise=true , address=77.90.31.16:4767 , piece_id=B3XFM5FUZQHU3CJ5OM3DJBACYYCZNIVON74C4UUSSK5K6GE24JVA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.747581919 (396h29m7.518223254s) [noise=true , address=104.36.82.23:4452 , piece_id=G4XFNFBBTJBOXSPS7YOTCSGFC7NY3VH5WUQ5BRFWWJ3IWXGPT3SA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.748234087 (396h29m7.518875412s) [noise=true , address=77.90.31.21:4881 , piece_id=DQRBEHZFEHIMGIKCQIJQQXR6LVG67YOF24ESXW3JLWHBVLWPHEHA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.748773640 (396h29m7.519414995s) [noise=true , address=77.90.31.11:4575 , piece_id=3EPIKI2CXDS4JWHP5YOT4HWZBBZBTWSJZ2BCYULSZTQPFULAVKBA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.749532161 (396h29m7.520173496s) [noise=true , address=64.49.13.169:4706 , piece_id=4CU34X6K2PKGF6YFOLFVGF62G4RPZ7BYBOVMVPMLTPYVZ2OAJ4LQ, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.750049140 (396h29m7.520690475s) [noise=true , address=79.127.219.38:4006 , piece_id=6S2FX7PM4HMCULTOQ4WW6TDN52W43CRPCZC5A3DYFI5ON3QJ3WIA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.750308366 (396h29m7.520949701s) [noise=true , address=79.127.201.215:4352 , piece_id=K4L367RZGAISJISAJZRPNAPGRSNAO2QL6YRLVCTRQGDXMT23DKBA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.764751689 (396h29m7.535393024s) [noise=true , address=77.90.31.14:4686 , piece_id=4ARR5GY74KXEVSKHBQRAS2WADAACDNCGO2LPNSWO5LRUZQ7GEQIA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:33.880679709 (396h29m8.651321044s) [upload=317212]: Next returned nil segment Jul 9 21:39:33.880697924 (396h29m8.65133926s) [upload=317212]: Waiting for error group managing segments... Jul 9 21:39:34.264937553 (396h29m9.035578888s) [noise=true , address=64.49.13.137:4502 , piece_id=T4BAEHJ2GEW7YVSNSXTCFGX2ANUGGUYMAU7A2AIILIUBHS2AQMJQ, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.740039134 (396h29m9.510680469s) [noise=true , address=77.90.31.18:4831 , piece_id=4CSJQKEAD6XPOQMIIOG6SCTBEBNUPCCED3DL6R367YFKHTKXFAVA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.742667242 (396h29m9.513308567s) [noise=true , address=77.90.31.16:4767 , piece_id=B3XFM5FUZQHU3CJ5OM3DJBACYYCZNIVON74C4UUSSK5K6GE24JVA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.745658134 (396h29m9.516299469s) [noise=true , address=77.90.31.12:4616 , piece_id=WYOLXV2SNY3PF3VWQS76KBGNRTOPKDR2DDSMZGMAQCVWKM3JRG7A, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.746157450 (396h29m9.516798785s) [noise=true , address=79.127.219.38:4006 , piece_id=6S2FX7PM4HMCULTOQ4WW6TDN52W43CRPCZC5A3DYFI5ON3QJ3WIA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.751901208 (396h29m9.522542533s) [noise=true , address=77.90.31.11:4575 , piece_id=3EPIKI2CXDS4JWHP5YOT4HWZBBZBTWSJZ2BCYULSZTQPFULAVKBA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.757250002 (396h29m9.527891337s) [noise=true , address=77.90.31.10:4535 , piece_id=KJER6ZFIVPUXCK6WEI5TX5UX3LU3BAO25MYTQW3A7TIHXHLK4ECA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.757587377 (396h29m9.528228712s) [noise=true , address=77.90.31.17:4787 , piece_id=SIKBA3Z4FKNVGMLHVE3ODZP2IJU42LJDA26UQQZRGF45GYOICLSA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.761661611 (396h29m9.532302946s) [noise=true , address=77.90.31.21:4881 , piece_id=DQRBEHZFEHIMGIKCQIJQQXR6LVG67YOF24ESXW3JLWHBVLWPHEHA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.765694217 (396h29m9.536335552s) [noise=true , address=77.90.31.14:4686 , piece_id=4ARR5GY74KXEVSKHBQRAS2WADAACDNCGO2LPNSWO5LRUZQ7GEQIA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:40.548306308 (396h29m15.318947683s) [noise=true , address=79.127.201.215:4352 , piece_id=K4L367RZGAISJISAJZRPNAPGRSNAO2QL6YRLVCTRQGDXMT23DKBA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:40:16.301179498 (396h29m51.071820823s) [noise=true , address=64.49.13.169:4706 , piece_id=4CU34X6K2PKGF6YFOLFVGF62G4RPZ7BYBOVMVPMLTPYVZ2OAJ4LQ, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:40:31.659193440 (396h30m6.429834776s) [noise=true , address=104.36.82.23:4452 , piece_id=G4XFNFBBTJBOXSPS7YOTCSGFC7NY3VH5WUQ5BRFWWJ3IWXGPT3SA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:1sAh1mLmzbtmc4o2FAZssW8Y7iDDNmNTCXdCS8xrH8PBVQqxG5, address:104.36.82.23:4452): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:40:31.659249508 (396h30m6.429890833s) [noise=true , address=104.36.82.21:4473 , piece_id=65CNRWVKLCWLW6IVVIMV2O3U3ONRWJWTPZ5WA6AB7W7CSKHJJGHQ, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:104.36.82.21:4473): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:40:31.659428720 (396h30m6.430070055s) [noise=true , address=45.91.95.4:4036 , piece_id=GSSBZKTAVOFGTBOD2SQ3C5Y6P3KYWZD7UJFJ4L2ZA2EYZ7DQAZKA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:12iVWBkBV5xpsCNemxmJasZtAyWa7L2QxMDyvKK85DcHvnsZ4YS, address:45.91.95.4:4036): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:40:31.659540674 (396h30m6.430182009s) [noise=true , address=45.91.95.3:4004 , piece_id=V3MVFLFQMXW3KO4PSEL3BQUEOWL4OR5HPPWRONMEVFPHOORC5CYQ, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:12vyhWVEevgJh7gAqrpfHeHyXeboegjmnKZAZpGKC8nvXdu6fnr, address:45.91.95.3:4004): protocol: write tcp 188.166.225.170:54188->45.91.95.3:4004: use of closed network connection; write tcp 188.166.225.170:54188->45.91.95.3:4004: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54188->45.91.95.3:4004: use of closed network connection Jul 9 21:40:31.659695461 (396h30m6.430336796s) [noise=true , address=104.36.82.19:4425 , piece_id=GMMOB7UIYDEB37LG5WCBGSGIH6IT652DHXBTH66MLWHBFF3IQTBA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:13sAcGFjAXuesAkUn7NuLaaV8TAah3Mr1kqd33wwh9MTELqmpD, address:104.36.82.19:4425): protocol: write tcp 188.166.225.170:47080->104.36.82.19:4425: use of closed network connection; write tcp 188.166.225.170:47080->104.36.82.19:4425: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47080->104.36.82.19:4425: use of closed network connection Jul 9 21:40:31.659717543 (396h30m6.430358868s) [noise=true , address=45.91.95.2:4012 , piece_id=ZNJPEWCNHTPOFCWXHCCTTOCFNDJSZE2NRXJK2SWE5DX2YSSIT2UQ, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:12T59HTbN21RvB3a7g4eniRrCRvFiqe9Jpsncf6yUPzWTmJG9Vu, address:45.91.95.2:4012): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:40:31.659748442 (396h30m6.430389777s) [seg_pos={0 0}, upload=317212]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 21:40:31.902160282 (396h30m6.672801617s) [upload=317212]: Done uploading segments... Jul 9 21:40:31.902184678 (396h30m6.672826013s) [upload=317212]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 31.217222613s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [......X...X................X....X...............X..X.X...X..] duration: 49.118419028s FAIL (2025-07-09 21:42:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 727635s) [upload=317212]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:39:32.743915214 (396h29m7.514556539s) [seg_pos={0 0}, upload=317212]: Begin upload segment... Jul 9 21:39:32.743955091 (396h29m7.514596416s) [seg_pos={0 0}, upload=317212]: Done begin upload segment. Jul 9 21:39:32.743976182 (396h29m7.514617517s) [noise=true , address=77.90.31.10:4535 , piece_id=KJER6ZFIVPUXCK6WEI5TX5UX3LU3BAO25MYTQW3A7TIHXHLK4ECA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.744312375 (396h29m7.51495371s) [noise=true , address=104.36.82.19:4425 , piece_id=GMMOB7UIYDEB37LG5WCBGSGIH6IT652DHXBTH66MLWHBFF3IQTBA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.744344205 (396h29m7.51498554s) [noise=true , address=77.90.31.12:4616 , piece_id=WYOLXV2SNY3PF3VWQS76KBGNRTOPKDR2DDSMZGMAQCVWKM3JRG7A, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.744776322 (396h29m7.515417667s) [noise=true , address=104.36.82.21:4473 , piece_id=65CNRWVKLCWLW6IVVIMV2O3U3ONRWJWTPZ5WA6AB7W7CSKHJJGHQ, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.744879529 (396h29m7.515520864s) [noise=true , address=77.90.31.18:4831 , piece_id=4CSJQKEAD6XPOQMIIOG6SCTBEBNUPCCED3DL6R367YFKHTKXFAVA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.745527850 (396h29m7.516169185s) [noise=true , address=64.49.13.137:4502 , piece_id=T4BAEHJ2GEW7YVSNSXTCFGX2ANUGGUYMAU7A2AIILIUBHS2AQMJQ, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.745755746 (396h29m7.516397081s) [noise=true , address=45.91.95.3:4004 , piece_id=V3MVFLFQMXW3KO4PSEL3BQUEOWL4OR5HPPWRONMEVFPHOORC5CYQ, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.746225264 (396h29m7.516866599s) [noise=true , address=45.91.95.2:4012 , piece_id=ZNJPEWCNHTPOFCWXHCCTTOCFNDJSZE2NRXJK2SWE5DX2YSSIT2UQ, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.746255782 (396h29m7.516897117s) [noise=true , address=77.90.31.17:4787 , piece_id=SIKBA3Z4FKNVGMLHVE3ODZP2IJU42LJDA26UQQZRGF45GYOICLSA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.746836593 (396h29m7.517477928s) [noise=true , address=45.91.95.4:4036 , piece_id=GSSBZKTAVOFGTBOD2SQ3C5Y6P3KYWZD7UJFJ4L2ZA2EYZ7DQAZKA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.747161365 (396h29m7.5178027s) [noise=true , address=77.90.31.16:4767 , piece_id=B3XFM5FUZQHU3CJ5OM3DJBACYYCZNIVON74C4UUSSK5K6GE24JVA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.747581919 (396h29m7.518223254s) [noise=true , address=104.36.82.23:4452 , piece_id=G4XFNFBBTJBOXSPS7YOTCSGFC7NY3VH5WUQ5BRFWWJ3IWXGPT3SA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.748234087 (396h29m7.518875412s) [noise=true , address=77.90.31.21:4881 , piece_id=DQRBEHZFEHIMGIKCQIJQQXR6LVG67YOF24ESXW3JLWHBVLWPHEHA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.748773640 (396h29m7.519414995s) [noise=true , address=77.90.31.11:4575 , piece_id=3EPIKI2CXDS4JWHP5YOT4HWZBBZBTWSJZ2BCYULSZTQPFULAVKBA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.749532161 (396h29m7.520173496s) [noise=true , address=64.49.13.169:4706 , piece_id=4CU34X6K2PKGF6YFOLFVGF62G4RPZ7BYBOVMVPMLTPYVZ2OAJ4LQ, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.750049140 (396h29m7.520690475s) [noise=true , address=79.127.219.38:4006 , piece_id=6S2FX7PM4HMCULTOQ4WW6TDN52W43CRPCZC5A3DYFI5ON3QJ3WIA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.750308366 (396h29m7.520949701s) [noise=true , address=79.127.201.215:4352 , piece_id=K4L367RZGAISJISAJZRPNAPGRSNAO2QL6YRLVCTRQGDXMT23DKBA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.764751689 (396h29m7.535393024s) [noise=true , address=77.90.31.14:4686 , piece_id=4ARR5GY74KXEVSKHBQRAS2WADAACDNCGO2LPNSWO5LRUZQ7GEQIA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:33.880679709 (396h29m8.651321044s) [upload=317212]: Next returned nil segment Jul 9 21:39:33.880697924 (396h29m8.65133926s) [upload=317212]: Waiting for error group managing segments... Jul 9 21:39:34.264937553 (396h29m9.035578888s) [noise=true , address=64.49.13.137:4502 , piece_id=T4BAEHJ2GEW7YVSNSXTCFGX2ANUGGUYMAU7A2AIILIUBHS2AQMJQ, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.740039134 (396h29m9.510680469s) [noise=true , address=77.90.31.18:4831 , piece_id=4CSJQKEAD6XPOQMIIOG6SCTBEBNUPCCED3DL6R367YFKHTKXFAVA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.742667242 (396h29m9.513308567s) [noise=true , address=77.90.31.16:4767 , piece_id=B3XFM5FUZQHU3CJ5OM3DJBACYYCZNIVON74C4UUSSK5K6GE24JVA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.745658134 (396h29m9.516299469s) [noise=true , address=77.90.31.12:4616 , piece_id=WYOLXV2SNY3PF3VWQS76KBGNRTOPKDR2DDSMZGMAQCVWKM3JRG7A, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.746157450 (396h29m9.516798785s) [noise=true , address=79.127.219.38:4006 , piece_id=6S2FX7PM4HMCULTOQ4WW6TDN52W43CRPCZC5A3DYFI5ON3QJ3WIA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.751901208 (396h29m9.522542533s) [noise=true , address=77.90.31.11:4575 , piece_id=3EPIKI2CXDS4JWHP5YOT4HWZBBZBTWSJZ2BCYULSZTQPFULAVKBA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.757250002 (396h29m9.527891337s) [noise=true , address=77.90.31.10:4535 , piece_id=KJER6ZFIVPUXCK6WEI5TX5UX3LU3BAO25MYTQW3A7TIHXHLK4ECA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.757587377 (396h29m9.528228712s) [noise=true , address=77.90.31.17:4787 , piece_id=SIKBA3Z4FKNVGMLHVE3ODZP2IJU42LJDA26UQQZRGF45GYOICLSA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.761661611 (396h29m9.532302946s) [noise=true , address=77.90.31.21:4881 , piece_id=DQRBEHZFEHIMGIKCQIJQQXR6LVG67YOF24ESXW3JLWHBVLWPHEHA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.765694217 (396h29m9.536335552s) [noise=true , address=77.90.31.14:4686 , piece_id=4ARR5GY74KXEVSKHBQRAS2WADAACDNCGO2LPNSWO5LRUZQ7GEQIA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:40.548306308 (396h29m15.318947683s) [noise=true , address=79.127.201.215:4352 , piece_id=K4L367RZGAISJISAJZRPNAPGRSNAO2QL6YRLVCTRQGDXMT23DKBA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:40:16.301179498 (396h29m51.071820823s) [noise=true , address=64.49.13.169:4706 , piece_id=4CU34X6K2PKGF6YFOLFVGF62G4RPZ7BYBOVMVPMLTPYVZ2OAJ4LQ, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:40:31.659193440 (396h30m6.429834776s) [noise=true , address=104.36.82.23:4452 , piece_id=G4XFNFBBTJBOXSPS7YOTCSGFC7NY3VH5WUQ5BRFWWJ3IWXGPT3SA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:1sAh1mLmzbtmc4o2FAZssW8Y7iDDNmNTCXdCS8xrH8PBVQqxG5, address:104.36.82.23:4452): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:40:31.659249508 (396h30m6.429890833s) [noise=true , address=104.36.82.21:4473 , piece_id=65CNRWVKLCWLW6IVVIMV2O3U3ONRWJWTPZ5WA6AB7W7CSKHJJGHQ, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:104.36.82.21:4473): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:40:31.659428720 (396h30m6.430070055s) [noise=true , address=45.91.95.4:4036 , piece_id=GSSBZKTAVOFGTBOD2SQ3C5Y6P3KYWZD7UJFJ4L2ZA2EYZ7DQAZKA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:12iVWBkBV5xpsCNemxmJasZtAyWa7L2QxMDyvKK85DcHvnsZ4YS, address:45.91.95.4:4036): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:40:31.659540674 (396h30m6.430182009s) [noise=true , address=45.91.95.3:4004 , piece_id=V3MVFLFQMXW3KO4PSEL3BQUEOWL4OR5HPPWRONMEVFPHOORC5CYQ, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:12vyhWVEevgJh7gAqrpfHeHyXeboegjmnKZAZpGKC8nvXdu6fnr, address:45.91.95.3:4004): protocol: write tcp 188.166.225.170:54188->45.91.95.3:4004: use of closed network connection; write tcp 188.166.225.170:54188->45.91.95.3:4004: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54188->45.91.95.3:4004: use of closed network connection Jul 9 21:40:31.659695461 (396h30m6.430336796s) [noise=true , address=104.36.82.19:4425 , piece_id=GMMOB7UIYDEB37LG5WCBGSGIH6IT652DHXBTH66MLWHBFF3IQTBA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:13sAcGFjAXuesAkUn7NuLaaV8TAah3Mr1kqd33wwh9MTELqmpD, address:104.36.82.19:4425): protocol: write tcp 188.166.225.170:47080->104.36.82.19:4425: use of closed network connection; write tcp 188.166.225.170:47080->104.36.82.19:4425: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47080->104.36.82.19:4425: use of closed network connection Jul 9 21:40:31.659717543 (396h30m6.430358868s) [noise=true , address=45.91.95.2:4012 , piece_id=ZNJPEWCNHTPOFCWXHCCTTOCFNDJSZE2NRXJK2SWE5DX2YSSIT2UQ, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:12T59HTbN21RvB3a7g4eniRrCRvFiqe9Jpsncf6yUPzWTmJG9Vu, address:45.91.95.2:4012): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:40:31.659748442 (396h30m6.430389777s) [seg_pos={0 0}, upload=317212]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 21:40:31.902160282 (396h30m6.672801617s) [upload=317212]: Done uploading segments... Jul 9 21:40:31.902184678 (396h30m6.672826013s) [upload=317212]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 49.118447514s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [........X......X....X..X.........X...............X..X....X..] duration: 43.514492373s FAIL (2025-07-09 21:41:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 43.514510197s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.........X......X....X..X.........X...............X..X....X.] duration: 29.773434933s FAIL (2025-07-09 21:41:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 29.773445243s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.183115717s FAIL (2025-07-09 21:40:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.183159271s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.......X...X................X....X...............X..X.X...X.] duration: 14.688662298s FAIL (2025-07-09 21:40:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 727635s) [upload=317212]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:39:32.743915214 (396h29m7.514556539s) [seg_pos={0 0}, upload=317212]: Begin upload segment... Jul 9 21:39:32.743955091 (396h29m7.514596416s) [seg_pos={0 0}, upload=317212]: Done begin upload segment. Jul 9 21:39:32.743976182 (396h29m7.514617517s) [noise=true , address=77.90.31.10:4535 , piece_id=KJER6ZFIVPUXCK6WEI5TX5UX3LU3BAO25MYTQW3A7TIHXHLK4ECA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.744312375 (396h29m7.51495371s) [noise=true , address=104.36.82.19:4425 , piece_id=GMMOB7UIYDEB37LG5WCBGSGIH6IT652DHXBTH66MLWHBFF3IQTBA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.744344205 (396h29m7.51498554s) [noise=true , address=77.90.31.12:4616 , piece_id=WYOLXV2SNY3PF3VWQS76KBGNRTOPKDR2DDSMZGMAQCVWKM3JRG7A, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.744776322 (396h29m7.515417667s) [noise=true , address=104.36.82.21:4473 , piece_id=65CNRWVKLCWLW6IVVIMV2O3U3ONRWJWTPZ5WA6AB7W7CSKHJJGHQ, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.744879529 (396h29m7.515520864s) [noise=true , address=77.90.31.18:4831 , piece_id=4CSJQKEAD6XPOQMIIOG6SCTBEBNUPCCED3DL6R367YFKHTKXFAVA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.745527850 (396h29m7.516169185s) [noise=true , address=64.49.13.137:4502 , piece_id=T4BAEHJ2GEW7YVSNSXTCFGX2ANUGGUYMAU7A2AIILIUBHS2AQMJQ, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.745755746 (396h29m7.516397081s) [noise=true , address=45.91.95.3:4004 , piece_id=V3MVFLFQMXW3KO4PSEL3BQUEOWL4OR5HPPWRONMEVFPHOORC5CYQ, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.746225264 (396h29m7.516866599s) [noise=true , address=45.91.95.2:4012 , piece_id=ZNJPEWCNHTPOFCWXHCCTTOCFNDJSZE2NRXJK2SWE5DX2YSSIT2UQ, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.746255782 (396h29m7.516897117s) [noise=true , address=77.90.31.17:4787 , piece_id=SIKBA3Z4FKNVGMLHVE3ODZP2IJU42LJDA26UQQZRGF45GYOICLSA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.746836593 (396h29m7.517477928s) [noise=true , address=45.91.95.4:4036 , piece_id=GSSBZKTAVOFGTBOD2SQ3C5Y6P3KYWZD7UJFJ4L2ZA2EYZ7DQAZKA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.747161365 (396h29m7.5178027s) [noise=true , address=77.90.31.16:4767 , piece_id=B3XFM5FUZQHU3CJ5OM3DJBACYYCZNIVON74C4UUSSK5K6GE24JVA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.747581919 (396h29m7.518223254s) [noise=true , address=104.36.82.23:4452 , piece_id=G4XFNFBBTJBOXSPS7YOTCSGFC7NY3VH5WUQ5BRFWWJ3IWXGPT3SA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.748234087 (396h29m7.518875412s) [noise=true , address=77.90.31.21:4881 , piece_id=DQRBEHZFEHIMGIKCQIJQQXR6LVG67YOF24ESXW3JLWHBVLWPHEHA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.748773640 (396h29m7.519414995s) [noise=true , address=77.90.31.11:4575 , piece_id=3EPIKI2CXDS4JWHP5YOT4HWZBBZBTWSJZ2BCYULSZTQPFULAVKBA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.749532161 (396h29m7.520173496s) [noise=true , address=64.49.13.169:4706 , piece_id=4CU34X6K2PKGF6YFOLFVGF62G4RPZ7BYBOVMVPMLTPYVZ2OAJ4LQ, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.750049140 (396h29m7.520690475s) [noise=true , address=79.127.219.38:4006 , piece_id=6S2FX7PM4HMCULTOQ4WW6TDN52W43CRPCZC5A3DYFI5ON3QJ3WIA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.750308366 (396h29m7.520949701s) [noise=true , address=79.127.201.215:4352 , piece_id=K4L367RZGAISJISAJZRPNAPGRSNAO2QL6YRLVCTRQGDXMT23DKBA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.764751689 (396h29m7.535393024s) [noise=true , address=77.90.31.14:4686 , piece_id=4ARR5GY74KXEVSKHBQRAS2WADAACDNCGO2LPNSWO5LRUZQ7GEQIA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:33.880679709 (396h29m8.651321044s) [upload=317212]: Next returned nil segment Jul 9 21:39:33.880697924 (396h29m8.65133926s) [upload=317212]: Waiting for error group managing segments... Jul 9 21:39:34.264937553 (396h29m9.035578888s) [noise=true , address=64.49.13.137:4502 , piece_id=T4BAEHJ2GEW7YVSNSXTCFGX2ANUGGUYMAU7A2AIILIUBHS2AQMJQ, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.740039134 (396h29m9.510680469s) [noise=true , address=77.90.31.18:4831 , piece_id=4CSJQKEAD6XPOQMIIOG6SCTBEBNUPCCED3DL6R367YFKHTKXFAVA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.742667242 (396h29m9.513308567s) [noise=true , address=77.90.31.16:4767 , piece_id=B3XFM5FUZQHU3CJ5OM3DJBACYYCZNIVON74C4UUSSK5K6GE24JVA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.745658134 (396h29m9.516299469s) [noise=true , address=77.90.31.12:4616 , piece_id=WYOLXV2SNY3PF3VWQS76KBGNRTOPKDR2DDSMZGMAQCVWKM3JRG7A, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.746157450 (396h29m9.516798785s) [noise=true , address=79.127.219.38:4006 , piece_id=6S2FX7PM4HMCULTOQ4WW6TDN52W43CRPCZC5A3DYFI5ON3QJ3WIA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.751901208 (396h29m9.522542533s) [noise=true , address=77.90.31.11:4575 , piece_id=3EPIKI2CXDS4JWHP5YOT4HWZBBZBTWSJZ2BCYULSZTQPFULAVKBA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.757250002 (396h29m9.527891337s) [noise=true , address=77.90.31.10:4535 , piece_id=KJER6ZFIVPUXCK6WEI5TX5UX3LU3BAO25MYTQW3A7TIHXHLK4ECA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.757587377 (396h29m9.528228712s) [noise=true , address=77.90.31.17:4787 , piece_id=SIKBA3Z4FKNVGMLHVE3ODZP2IJU42LJDA26UQQZRGF45GYOICLSA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.761661611 (396h29m9.532302946s) [noise=true , address=77.90.31.21:4881 , piece_id=DQRBEHZFEHIMGIKCQIJQQXR6LVG67YOF24ESXW3JLWHBVLWPHEHA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.765694217 (396h29m9.536335552s) [noise=true , address=77.90.31.14:4686 , piece_id=4ARR5GY74KXEVSKHBQRAS2WADAACDNCGO2LPNSWO5LRUZQ7GEQIA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:40.548306308 (396h29m15.318947683s) [noise=true , address=79.127.201.215:4352 , piece_id=K4L367RZGAISJISAJZRPNAPGRSNAO2QL6YRLVCTRQGDXMT23DKBA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:40:16.301179498 (396h29m51.071820823s) [noise=true , address=64.49.13.169:4706 , piece_id=4CU34X6K2PKGF6YFOLFVGF62G4RPZ7BYBOVMVPMLTPYVZ2OAJ4LQ, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:40:31.659193440 (396h30m6.429834776s) [noise=true , address=104.36.82.23:4452 , piece_id=G4XFNFBBTJBOXSPS7YOTCSGFC7NY3VH5WUQ5BRFWWJ3IWXGPT3SA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:1sAh1mLmzbtmc4o2FAZssW8Y7iDDNmNTCXdCS8xrH8PBVQqxG5, address:104.36.82.23:4452): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:40:31.659249508 (396h30m6.429890833s) [noise=true , address=104.36.82.21:4473 , piece_id=65CNRWVKLCWLW6IVVIMV2O3U3ONRWJWTPZ5WA6AB7W7CSKHJJGHQ, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:104.36.82.21:4473): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:40:31.659428720 (396h30m6.430070055s) [noise=true , address=45.91.95.4:4036 , piece_id=GSSBZKTAVOFGTBOD2SQ3C5Y6P3KYWZD7UJFJ4L2ZA2EYZ7DQAZKA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:12iVWBkBV5xpsCNemxmJasZtAyWa7L2QxMDyvKK85DcHvnsZ4YS, address:45.91.95.4:4036): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:40:31.659540674 (396h30m6.430182009s) [noise=true , address=45.91.95.3:4004 , piece_id=V3MVFLFQMXW3KO4PSEL3BQUEOWL4OR5HPPWRONMEVFPHOORC5CYQ, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:12vyhWVEevgJh7gAqrpfHeHyXeboegjmnKZAZpGKC8nvXdu6fnr, address:45.91.95.3:4004): protocol: write tcp 188.166.225.170:54188->45.91.95.3:4004: use of closed network connection; write tcp 188.166.225.170:54188->45.91.95.3:4004: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54188->45.91.95.3:4004: use of closed network connection Jul 9 21:40:31.659695461 (396h30m6.430336796s) [noise=true , address=104.36.82.19:4425 , piece_id=GMMOB7UIYDEB37LG5WCBGSGIH6IT652DHXBTH66MLWHBFF3IQTBA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:13sAcGFjAXuesAkUn7NuLaaV8TAah3Mr1kqd33wwh9MTELqmpD, address:104.36.82.19:4425): protocol: write tcp 188.166.225.170:47080->104.36.82.19:4425: use of closed network connection; write tcp 188.166.225.170:47080->104.36.82.19:4425: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47080->104.36.82.19:4425: use of closed network connection Jul 9 21:40:31.659717543 (396h30m6.430358868s) [noise=true , address=45.91.95.2:4012 , piece_id=ZNJPEWCNHTPOFCWXHCCTTOCFNDJSZE2NRXJK2SWE5DX2YSSIT2UQ, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:12T59HTbN21RvB3a7g4eniRrCRvFiqe9Jpsncf6yUPzWTmJG9Vu, address:45.91.95.2:4012): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:40:31.659748442 (396h30m6.430389777s) [seg_pos={0 0}, upload=317212]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 21:40:31.902160282 (396h30m6.672801617s) [upload=317212]: Done uploading segments... Jul 9 21:40:31.902184678 (396h30m6.672826013s) [upload=317212]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 14.688682164s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [X.........X......X....X..X.........X...............X..X....X] duration: 1m0.000713526s FAIL (2025-07-09 21:40:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000727322s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [........X...X................X....X...............X..X.X...X] duration: 1m0.245045127s FAIL (2025-07-09 21:40:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 727635s) [upload=317212]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:39:32.743915214 (396h29m7.514556539s) [seg_pos={0 0}, upload=317212]: Begin upload segment... Jul 9 21:39:32.743955091 (396h29m7.514596416s) [seg_pos={0 0}, upload=317212]: Done begin upload segment. Jul 9 21:39:32.743976182 (396h29m7.514617517s) [noise=true , address=77.90.31.10:4535 , piece_id=KJER6ZFIVPUXCK6WEI5TX5UX3LU3BAO25MYTQW3A7TIHXHLK4ECA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.744312375 (396h29m7.51495371s) [noise=true , address=104.36.82.19:4425 , piece_id=GMMOB7UIYDEB37LG5WCBGSGIH6IT652DHXBTH66MLWHBFF3IQTBA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.744344205 (396h29m7.51498554s) [noise=true , address=77.90.31.12:4616 , piece_id=WYOLXV2SNY3PF3VWQS76KBGNRTOPKDR2DDSMZGMAQCVWKM3JRG7A, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.744776322 (396h29m7.515417667s) [noise=true , address=104.36.82.21:4473 , piece_id=65CNRWVKLCWLW6IVVIMV2O3U3ONRWJWTPZ5WA6AB7W7CSKHJJGHQ, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.744879529 (396h29m7.515520864s) [noise=true , address=77.90.31.18:4831 , piece_id=4CSJQKEAD6XPOQMIIOG6SCTBEBNUPCCED3DL6R367YFKHTKXFAVA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.745527850 (396h29m7.516169185s) [noise=true , address=64.49.13.137:4502 , piece_id=T4BAEHJ2GEW7YVSNSXTCFGX2ANUGGUYMAU7A2AIILIUBHS2AQMJQ, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.745755746 (396h29m7.516397081s) [noise=true , address=45.91.95.3:4004 , piece_id=V3MVFLFQMXW3KO4PSEL3BQUEOWL4OR5HPPWRONMEVFPHOORC5CYQ, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.746225264 (396h29m7.516866599s) [noise=true , address=45.91.95.2:4012 , piece_id=ZNJPEWCNHTPOFCWXHCCTTOCFNDJSZE2NRXJK2SWE5DX2YSSIT2UQ, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.746255782 (396h29m7.516897117s) [noise=true , address=77.90.31.17:4787 , piece_id=SIKBA3Z4FKNVGMLHVE3ODZP2IJU42LJDA26UQQZRGF45GYOICLSA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.746836593 (396h29m7.517477928s) [noise=true , address=45.91.95.4:4036 , piece_id=GSSBZKTAVOFGTBOD2SQ3C5Y6P3KYWZD7UJFJ4L2ZA2EYZ7DQAZKA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.747161365 (396h29m7.5178027s) [noise=true , address=77.90.31.16:4767 , piece_id=B3XFM5FUZQHU3CJ5OM3DJBACYYCZNIVON74C4UUSSK5K6GE24JVA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.747581919 (396h29m7.518223254s) [noise=true , address=104.36.82.23:4452 , piece_id=G4XFNFBBTJBOXSPS7YOTCSGFC7NY3VH5WUQ5BRFWWJ3IWXGPT3SA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.748234087 (396h29m7.518875412s) [noise=true , address=77.90.31.21:4881 , piece_id=DQRBEHZFEHIMGIKCQIJQQXR6LVG67YOF24ESXW3JLWHBVLWPHEHA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.748773640 (396h29m7.519414995s) [noise=true , address=77.90.31.11:4575 , piece_id=3EPIKI2CXDS4JWHP5YOT4HWZBBZBTWSJZ2BCYULSZTQPFULAVKBA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.749532161 (396h29m7.520173496s) [noise=true , address=64.49.13.169:4706 , piece_id=4CU34X6K2PKGF6YFOLFVGF62G4RPZ7BYBOVMVPMLTPYVZ2OAJ4LQ, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.750049140 (396h29m7.520690475s) [noise=true , address=79.127.219.38:4006 , piece_id=6S2FX7PM4HMCULTOQ4WW6TDN52W43CRPCZC5A3DYFI5ON3QJ3WIA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.750308366 (396h29m7.520949701s) [noise=true , address=79.127.201.215:4352 , piece_id=K4L367RZGAISJISAJZRPNAPGRSNAO2QL6YRLVCTRQGDXMT23DKBA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.764751689 (396h29m7.535393024s) [noise=true , address=77.90.31.14:4686 , piece_id=4ARR5GY74KXEVSKHBQRAS2WADAACDNCGO2LPNSWO5LRUZQ7GEQIA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:33.880679709 (396h29m8.651321044s) [upload=317212]: Next returned nil segment Jul 9 21:39:33.880697924 (396h29m8.65133926s) [upload=317212]: Waiting for error group managing segments... Jul 9 21:39:34.264937553 (396h29m9.035578888s) [noise=true , address=64.49.13.137:4502 , piece_id=T4BAEHJ2GEW7YVSNSXTCFGX2ANUGGUYMAU7A2AIILIUBHS2AQMJQ, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.740039134 (396h29m9.510680469s) [noise=true , address=77.90.31.18:4831 , piece_id=4CSJQKEAD6XPOQMIIOG6SCTBEBNUPCCED3DL6R367YFKHTKXFAVA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.742667242 (396h29m9.513308567s) [noise=true , address=77.90.31.16:4767 , piece_id=B3XFM5FUZQHU3CJ5OM3DJBACYYCZNIVON74C4UUSSK5K6GE24JVA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.745658134 (396h29m9.516299469s) [noise=true , address=77.90.31.12:4616 , piece_id=WYOLXV2SNY3PF3VWQS76KBGNRTOPKDR2DDSMZGMAQCVWKM3JRG7A, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.746157450 (396h29m9.516798785s) [noise=true , address=79.127.219.38:4006 , piece_id=6S2FX7PM4HMCULTOQ4WW6TDN52W43CRPCZC5A3DYFI5ON3QJ3WIA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.751901208 (396h29m9.522542533s) [noise=true , address=77.90.31.11:4575 , piece_id=3EPIKI2CXDS4JWHP5YOT4HWZBBZBTWSJZ2BCYULSZTQPFULAVKBA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.757250002 (396h29m9.527891337s) [noise=true , address=77.90.31.10:4535 , piece_id=KJER6ZFIVPUXCK6WEI5TX5UX3LU3BAO25MYTQW3A7TIHXHLK4ECA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.757587377 (396h29m9.528228712s) [noise=true , address=77.90.31.17:4787 , piece_id=SIKBA3Z4FKNVGMLHVE3ODZP2IJU42LJDA26UQQZRGF45GYOICLSA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.761661611 (396h29m9.532302946s) [noise=true , address=77.90.31.21:4881 , piece_id=DQRBEHZFEHIMGIKCQIJQQXR6LVG67YOF24ESXW3JLWHBVLWPHEHA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.765694217 (396h29m9.536335552s) [noise=true , address=77.90.31.14:4686 , piece_id=4ARR5GY74KXEVSKHBQRAS2WADAACDNCGO2LPNSWO5LRUZQ7GEQIA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:40.548306308 (396h29m15.318947683s) [noise=true , address=79.127.201.215:4352 , piece_id=K4L367RZGAISJISAJZRPNAPGRSNAO2QL6YRLVCTRQGDXMT23DKBA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:40:16.301179498 (396h29m51.071820823s) [noise=true , address=64.49.13.169:4706 , piece_id=4CU34X6K2PKGF6YFOLFVGF62G4RPZ7BYBOVMVPMLTPYVZ2OAJ4LQ, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:40:31.659193440 (396h30m6.429834776s) [noise=true , address=104.36.82.23:4452 , piece_id=G4XFNFBBTJBOXSPS7YOTCSGFC7NY3VH5WUQ5BRFWWJ3IWXGPT3SA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:1sAh1mLmzbtmc4o2FAZssW8Y7iDDNmNTCXdCS8xrH8PBVQqxG5, address:104.36.82.23:4452): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:40:31.659249508 (396h30m6.429890833s) [noise=true , address=104.36.82.21:4473 , piece_id=65CNRWVKLCWLW6IVVIMV2O3U3ONRWJWTPZ5WA6AB7W7CSKHJJGHQ, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:104.36.82.21:4473): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:40:31.659428720 (396h30m6.430070055s) [noise=true , address=45.91.95.4:4036 , piece_id=GSSBZKTAVOFGTBOD2SQ3C5Y6P3KYWZD7UJFJ4L2ZA2EYZ7DQAZKA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:12iVWBkBV5xpsCNemxmJasZtAyWa7L2QxMDyvKK85DcHvnsZ4YS, address:45.91.95.4:4036): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:40:31.659540674 (396h30m6.430182009s) [noise=true , address=45.91.95.3:4004 , piece_id=V3MVFLFQMXW3KO4PSEL3BQUEOWL4OR5HPPWRONMEVFPHOORC5CYQ, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:12vyhWVEevgJh7gAqrpfHeHyXeboegjmnKZAZpGKC8nvXdu6fnr, address:45.91.95.3:4004): protocol: write tcp 188.166.225.170:54188->45.91.95.3:4004: use of closed network connection; write tcp 188.166.225.170:54188->45.91.95.3:4004: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54188->45.91.95.3:4004: use of closed network connection Jul 9 21:40:31.659695461 (396h30m6.430336796s) [noise=true , address=104.36.82.19:4425 , piece_id=GMMOB7UIYDEB37LG5WCBGSGIH6IT652DHXBTH66MLWHBFF3IQTBA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:13sAcGFjAXuesAkUn7NuLaaV8TAah3Mr1kqd33wwh9MTELqmpD, address:104.36.82.19:4425): protocol: write tcp 188.166.225.170:47080->104.36.82.19:4425: use of closed network connection; write tcp 188.166.225.170:47080->104.36.82.19:4425: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47080->104.36.82.19:4425: use of closed network connection Jul 9 21:40:31.659717543 (396h30m6.430358868s) [noise=true , address=45.91.95.2:4012 , piece_id=ZNJPEWCNHTPOFCWXHCCTTOCFNDJSZE2NRXJK2SWE5DX2YSSIT2UQ, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:12T59HTbN21RvB3a7g4eniRrCRvFiqe9Jpsncf6yUPzWTmJG9Vu, address:45.91.95.2:4012): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:40:31.659748442 (396h30m6.430389777s) [seg_pos={0 0}, upload=317212]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 21:40:31.902160282 (396h30m6.672801617s) [upload=317212]: Done uploading segments... Jul 9 21:40:31.902184678 (396h30m6.672826013s) [upload=317212]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 1m0.245073821s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.946339036s FAIL (2025-07-09 21:39:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.946354226s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [XX.........X......X....X..X.........X...............X..X....] duration: 43.822420082s FAIL (2025-07-09 21:39:18 UTC) orbiter warning: context deadline exceeded Duration: 43.822431954s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.14407276s FAIL (2025-07-09 21:39:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.144093048s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.........X...X................X....X...............X..X.X...] duration: 20.013343214s FAIL (2025-07-09 21:39:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== one starting upload Jul 9 21:36:20.318624655 (396h25m55.08926599s) [upload=317168]: Uploading segments... Jul 9 21:36:20.318743822 (396h25m55.089385158s) [upload=317168]: Got next segment. Inline: false Jul 9 21:36:20.318780774 (396h25m55.089422099s) [upload=317168]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:36:20.318785232 (396h25m55.089426567s) [upload=317168]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:36:20.763719228 (396h25m55.534360563s) [seg_pos={0 0}, upload=317168]: Begin upload segment... Jul 9 21:36:20.763769214 (396h25m55.534410549s) [seg_pos={0 0}, upload=317168]: Done begin upload segment. Jul 9 21:36:20.763790274 (396h25m55.534431599s) [noise=true , address=104.36.82.44:4888 , piece_id=2F3ZGCXCJNMM7X3ITAHDUY5FOV6XBMY7O57POLYSMJGFL7XANRJQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.764193555 (396h25m55.53483489s) [noise=true , address=45.91.95.2:4032 , piece_id=NYJR7Y2CBC3IXJT6QBVOZQOUEUT4XS573ROG27LQQWBDVDOMHBUA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.764504661 (396h25m55.535145996s) [noise=true , address=104.36.82.5:4858 , piece_id=OC55PCL57AGUANZHVBSSQ4GCMZ7PKZ3AAZN5SSKWASDCH2OQ6PVA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.764789636 (396h25m55.535430971s) [noise=true , address=77.90.31.17:4785 , piece_id=RDX7V2ES2RK3QUKI7OBE6F3T7DNK5TIIS5JGJG2TENONKS26MOFQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.764857276 (396h25m55.535498611s) [noise=true , address=104.36.82.49:4996 , piece_id=DSDQOSI7R7BEK4ZS72OLCC2HLJ6WZSI652YOHY4X66FOCDKX2O4Q, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.765360789 (396h25m55.536002124s) [noise=true , address=77.90.31.19:4865 , piece_id=2KYAW2GTL7CF5BQ4AOJRWFOT7ZO6SC3AZOGG45JGIZW2LPGGCNDQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.765905511 (396h25m55.536546836s) [noise=true , address=77.90.31.14:4679 , piece_id=ESHQHAOWFYIVNRLXZ64INNNK6UYLBPAIFH4CCX6A6X3VCUZVZUEA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.766140190 (396h25m55.536781515s) [noise=true , address=104.36.82.38:4692 , piece_id=UBJ3M3HEK65QOWK7YRDJC4W4ZUVMIYDRRBV6TA2IF3ZP747L2N7Q, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.766384127 (396h25m55.537025462s) [noise=true , address=104.36.82.41:4770 , piece_id=SYPUVS5NMG4JKWTFLN5V22POSYMZOCBZAU52W6TSKR24U3PLHTLQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.766633814 (396h25m55.537275149s) [noise=true , address=77.90.31.15:4719 , piece_id=6UMEYHDUVSO2OH4K4FMUJCS75EGNCAJZEUKLX2SJLUFX56GNCYHQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.766850239 (396h25m55.537491574s) [noise=true , address=77.90.31.11:4564 , piece_id=HP7Z4KKX7O47BKY6NUNWAZF6WGD7EPQKF2DRTJ7FBGKGMPWMQE3A, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.767051463 (396h25m55.537692798s) [noise=true , address=104.36.82.22:4543 , piece_id=KJ5YMX5Z7H3XEGKXSAVXBUTDEGK2YN2BXNJHD4CBWHD7UDFMEBLA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.767262577 (396h25m55.537903913s) [noise=true , address=77.90.31.18:4820 , piece_id=RJVQ47F34SPY5X5LLGI425LKGRNNAU72VBOYZCAJCRB67OHWCQHA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.767485283 (396h25m55.538126609s) [noise=true , address=45.91.95.3:4002 , piece_id=NI64XOPSDEKRWTTRRAJI2LZZEFLXG5HYWGH7BTSYFF272BDG3TFQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.767685717 (396h25m55.538327052s) [noise=true , address=45.91.95.5:4022 , piece_id=GBTZSDCXLEURLOXZB7K2XTZPIS7QLDCU5YN2ZBF5TPEG2R6MSBTA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.767898654 (396h25m55.538539989s) [noise=true , address=77.90.31.13:4645 , piece_id=S37SRDGED55EZJUCLN26VCHJT2HFQE4PWZ4EMAIAGNE7WB4VFRTA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.768113456 (396h25m55.538754791s) [noise=true , address=77.90.31.10:4551 , piece_id=6G46ASL3W34QTB2IJMSL3HGB22YOVBTKD777IRJXTS4XEAEJIKEA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.768331041 (396h25m55.538972376s) [noise=true , address=45.91.95.4:4003 , piece_id=KP7IPIU663SIWR7UMAZWZNIGWNJSAE4BZOKD3WKLWEBGVBIPY2RA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:22.356647592 (396h25m57.127288927s) [upload=317168]: Next returned nil segment Jul 9 21:36:22.356668483 (396h25m57.127309828s) [upload=317168]: Waiting for error group managing segments... Jul 9 21:36:22.764717222 (396h25m57.535358557s) [noise=true , address=77.90.31.11:4564 , piece_id=HP7Z4KKX7O47BKY6NUNWAZF6WGD7EPQKF2DRTJ7FBGKGMPWMQE3A, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.765409827 (396h25m57.536051162s) [noise=true , address=77.90.31.14:4679 , piece_id=ESHQHAOWFYIVNRLXZ64INNNK6UYLBPAIFH4CCX6A6X3VCUZVZUEA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.766268049 (396h25m57.536909384s) [noise=true , address=77.90.31.18:4820 , piece_id=RJVQ47F34SPY5X5LLGI425LKGRNNAU72VBOYZCAJCRB67OHWCQHA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.767995543 (396h25m57.538636878s) [noise=true , address=77.90.31.17:4785 , piece_id=RDX7V2ES2RK3QUKI7OBE6F3T7DNK5TIIS5JGJG2TENONKS26MOFQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.772020253 (396h25m57.542661589s) [noise=true , address=77.90.31.19:4865 , piece_id=2KYAW2GTL7CF5BQ4AOJRWFOT7ZO6SC3AZOGG45JGIZW2LPGGCNDQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.780841681 (396h25m57.551483016s) [noise=true , address=77.90.31.15:4719 , piece_id=6UMEYHDUVSO2OH4K4FMUJCS75EGNCAJZEUKLX2SJLUFX56GNCYHQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.789850365 (396h25m57.5604917s) [noise=true , address=77.90.31.13:4645 , piece_id=S37SRDGED55EZJUCLN26VCHJT2HFQE4PWZ4EMAIAGNE7WB4VFRTA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.792072737 (396h25m57.562714072s) [noise=true , address=77.90.31.10:4551 , piece_id=6G46ASL3W34QTB2IJMSL3HGB22YOVBTKD777IRJXTS4XEAEJIKEA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:41.743417799 (396h26m16.514059134s) [noise=true , address=104.36.82.38:4692 , piece_id=UBJ3M3HEK65QOWK7YRDJC4W4ZUVMIYDRRBV6TA2IF3ZP747L2N7Q, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:46.945197674 (396h26m21.715839009s) [noise=true , address=104.36.82.5:4858 , piece_id=OC55PCL57AGUANZHVBSSQ4GCMZ7PKZ3AAZN5SSKWASDCH2OQ6PVA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:47.392071762 (396h26m22.162713087s) [noise=true , address=104.36.82.49:4996 , piece_id=DSDQOSI7R7BEK4ZS72OLCC2HLJ6WZSI652YOHY4X66FOCDKX2O4Q, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:37:04.301861761 (396h26m39.072503086s) [noise=true , address=104.36.82.41:4770 , piece_id=SYPUVS5NMG4JKWTFLN5V22POSYMZOCBZAU52W6TSKR24U3PLHTLQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:37:04.463654747 (396h26m39.234296082s) [noise=true , address=104.36.82.22:4543 , piece_id=KJ5YMX5Z7H3XEGKXSAVXBUTDEGK2YN2BXNJHD4CBWHD7UDFMEBLA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:37:19.671277798 (396h26m54.441919133s) [noise=true , address=45.91.95.2:4032 , piece_id=NYJR7Y2CBC3IXJT6QBVOZQOUEUT4XS573ROG27LQQWBDVDOMHBUA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: ecclient: upload failed (node:12BtBTji4bazMXHJXUPMZAmdFxouybziyrzrP8U6oFR8P3obi6f, address:45.91.95.2:4032): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:37:19.671365025 (396h26m54.44200636s) [noise=true , address=104.36.82.44:4888 , piece_id=2F3ZGCXCJNMM7X3ITAHDUY5FOV6XBMY7O57POLYSMJGFL7XANRJQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: ecclient: upload failed (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME, address:104.36.82.44:4888): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:37:19.671428787 (396h26m54.442070122s) [noise=true , address=45.91.95.5:4022 , piece_id=GBTZSDCXLEURLOXZB7K2XTZPIS7QLDCU5YN2ZBF5TPEG2R6MSBTA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: ecclient: upload failed (node:12X8Et5FNyAsE7ifEckfQemZkfnvYUuZ2Rn2xdxNxhmCmuDST2M, address:45.91.95.5:4022): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:37:19.671580928 (396h26m54.442222263s) [noise=true , address=45.91.95.4:4003 , piece_id=KP7IPIU663SIWR7UMAZWZNIGWNJSAE4BZOKD3WKLWEBGVBIPY2RA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: ecclient: upload failed (node:12iLBF47YcG1hvb6vSVNe41MSF6JXgEPYFwLfQcrU8Nt31GDmCg, address:45.91.95.4:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:37:19.671662775 (396h26m54.44230411s) [noise=true , address=45.91.95.3:4002 , piece_id=NI64XOPSDEKRWTTRRAJI2LZZEFLXG5HYWGH7BTSYFF272BDG3TFQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: ecclient: upload failed (node:1dpBPqn6zMpCUkCr8fEZZ17aV6Ag1ytp8Q3pvawZiHQMxGzAUw, address:45.91.95.3:4002): protocol: write tcp 188.166.225.170:42226->45.91.95.3:4002: use of closed network connection; write tcp 188.166.225.170:42226->45.91.95.3:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42226->45.91.95.3:4002: use of closed network connection Jul 9 21:37:19.671678956 (396h26m54.442320291s) [seg_pos={0 0}, upload=317168]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:37:19.903470848 (396h26m54.674112173s) [upload=317168]: Done uploading segments... Jul 9 21:37:19.903494724 (396h26m54.674136049s) [upload=317168]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 20.01336193s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X..X........................................................] duration: 6.126803755s FAIL (2025-07-09 21:39:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.126829355s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [..........X...X................X....X...............X..X.X..] duration: 31.212175479s FAIL (2025-07-09 21:38:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== one starting upload Jul 9 21:36:20.318624655 (396h25m55.08926599s) [upload=317168]: Uploading segments... Jul 9 21:36:20.318743822 (396h25m55.089385158s) [upload=317168]: Got next segment. Inline: false Jul 9 21:36:20.318780774 (396h25m55.089422099s) [upload=317168]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:36:20.318785232 (396h25m55.089426567s) [upload=317168]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:36:20.763719228 (396h25m55.534360563s) [seg_pos={0 0}, upload=317168]: Begin upload segment... Jul 9 21:36:20.763769214 (396h25m55.534410549s) [seg_pos={0 0}, upload=317168]: Done begin upload segment. Jul 9 21:36:20.763790274 (396h25m55.534431599s) [noise=true , address=104.36.82.44:4888 , piece_id=2F3ZGCXCJNMM7X3ITAHDUY5FOV6XBMY7O57POLYSMJGFL7XANRJQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.764193555 (396h25m55.53483489s) [noise=true , address=45.91.95.2:4032 , piece_id=NYJR7Y2CBC3IXJT6QBVOZQOUEUT4XS573ROG27LQQWBDVDOMHBUA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.764504661 (396h25m55.535145996s) [noise=true , address=104.36.82.5:4858 , piece_id=OC55PCL57AGUANZHVBSSQ4GCMZ7PKZ3AAZN5SSKWASDCH2OQ6PVA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.764789636 (396h25m55.535430971s) [noise=true , address=77.90.31.17:4785 , piece_id=RDX7V2ES2RK3QUKI7OBE6F3T7DNK5TIIS5JGJG2TENONKS26MOFQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.764857276 (396h25m55.535498611s) [noise=true , address=104.36.82.49:4996 , piece_id=DSDQOSI7R7BEK4ZS72OLCC2HLJ6WZSI652YOHY4X66FOCDKX2O4Q, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.765360789 (396h25m55.536002124s) [noise=true , address=77.90.31.19:4865 , piece_id=2KYAW2GTL7CF5BQ4AOJRWFOT7ZO6SC3AZOGG45JGIZW2LPGGCNDQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.765905511 (396h25m55.536546836s) [noise=true , address=77.90.31.14:4679 , piece_id=ESHQHAOWFYIVNRLXZ64INNNK6UYLBPAIFH4CCX6A6X3VCUZVZUEA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.766140190 (396h25m55.536781515s) [noise=true , address=104.36.82.38:4692 , piece_id=UBJ3M3HEK65QOWK7YRDJC4W4ZUVMIYDRRBV6TA2IF3ZP747L2N7Q, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.766384127 (396h25m55.537025462s) [noise=true , address=104.36.82.41:4770 , piece_id=SYPUVS5NMG4JKWTFLN5V22POSYMZOCBZAU52W6TSKR24U3PLHTLQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.766633814 (396h25m55.537275149s) [noise=true , address=77.90.31.15:4719 , piece_id=6UMEYHDUVSO2OH4K4FMUJCS75EGNCAJZEUKLX2SJLUFX56GNCYHQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.766850239 (396h25m55.537491574s) [noise=true , address=77.90.31.11:4564 , piece_id=HP7Z4KKX7O47BKY6NUNWAZF6WGD7EPQKF2DRTJ7FBGKGMPWMQE3A, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.767051463 (396h25m55.537692798s) [noise=true , address=104.36.82.22:4543 , piece_id=KJ5YMX5Z7H3XEGKXSAVXBUTDEGK2YN2BXNJHD4CBWHD7UDFMEBLA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.767262577 (396h25m55.537903913s) [noise=true , address=77.90.31.18:4820 , piece_id=RJVQ47F34SPY5X5LLGI425LKGRNNAU72VBOYZCAJCRB67OHWCQHA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.767485283 (396h25m55.538126609s) [noise=true , address=45.91.95.3:4002 , piece_id=NI64XOPSDEKRWTTRRAJI2LZZEFLXG5HYWGH7BTSYFF272BDG3TFQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.767685717 (396h25m55.538327052s) [noise=true , address=45.91.95.5:4022 , piece_id=GBTZSDCXLEURLOXZB7K2XTZPIS7QLDCU5YN2ZBF5TPEG2R6MSBTA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.767898654 (396h25m55.538539989s) [noise=true , address=77.90.31.13:4645 , piece_id=S37SRDGED55EZJUCLN26VCHJT2HFQE4PWZ4EMAIAGNE7WB4VFRTA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.768113456 (396h25m55.538754791s) [noise=true , address=77.90.31.10:4551 , piece_id=6G46ASL3W34QTB2IJMSL3HGB22YOVBTKD777IRJXTS4XEAEJIKEA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.768331041 (396h25m55.538972376s) [noise=true , address=45.91.95.4:4003 , piece_id=KP7IPIU663SIWR7UMAZWZNIGWNJSAE4BZOKD3WKLWEBGVBIPY2RA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:22.356647592 (396h25m57.127288927s) [upload=317168]: Next returned nil segment Jul 9 21:36:22.356668483 (396h25m57.127309828s) [upload=317168]: Waiting for error group managing segments... Jul 9 21:36:22.764717222 (396h25m57.535358557s) [noise=true , address=77.90.31.11:4564 , piece_id=HP7Z4KKX7O47BKY6NUNWAZF6WGD7EPQKF2DRTJ7FBGKGMPWMQE3A, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.765409827 (396h25m57.536051162s) [noise=true , address=77.90.31.14:4679 , piece_id=ESHQHAOWFYIVNRLXZ64INNNK6UYLBPAIFH4CCX6A6X3VCUZVZUEA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.766268049 (396h25m57.536909384s) [noise=true , address=77.90.31.18:4820 , piece_id=RJVQ47F34SPY5X5LLGI425LKGRNNAU72VBOYZCAJCRB67OHWCQHA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.767995543 (396h25m57.538636878s) [noise=true , address=77.90.31.17:4785 , piece_id=RDX7V2ES2RK3QUKI7OBE6F3T7DNK5TIIS5JGJG2TENONKS26MOFQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.772020253 (396h25m57.542661589s) [noise=true , address=77.90.31.19:4865 , piece_id=2KYAW2GTL7CF5BQ4AOJRWFOT7ZO6SC3AZOGG45JGIZW2LPGGCNDQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.780841681 (396h25m57.551483016s) [noise=true , address=77.90.31.15:4719 , piece_id=6UMEYHDUVSO2OH4K4FMUJCS75EGNCAJZEUKLX2SJLUFX56GNCYHQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.789850365 (396h25m57.5604917s) [noise=true , address=77.90.31.13:4645 , piece_id=S37SRDGED55EZJUCLN26VCHJT2HFQE4PWZ4EMAIAGNE7WB4VFRTA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.792072737 (396h25m57.562714072s) [noise=true , address=77.90.31.10:4551 , piece_id=6G46ASL3W34QTB2IJMSL3HGB22YOVBTKD777IRJXTS4XEAEJIKEA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:41.743417799 (396h26m16.514059134s) [noise=true , address=104.36.82.38:4692 , piece_id=UBJ3M3HEK65QOWK7YRDJC4W4ZUVMIYDRRBV6TA2IF3ZP747L2N7Q, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:46.945197674 (396h26m21.715839009s) [noise=true , address=104.36.82.5:4858 , piece_id=OC55PCL57AGUANZHVBSSQ4GCMZ7PKZ3AAZN5SSKWASDCH2OQ6PVA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:47.392071762 (396h26m22.162713087s) [noise=true , address=104.36.82.49:4996 , piece_id=DSDQOSI7R7BEK4ZS72OLCC2HLJ6WZSI652YOHY4X66FOCDKX2O4Q, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:37:04.301861761 (396h26m39.072503086s) [noise=true , address=104.36.82.41:4770 , piece_id=SYPUVS5NMG4JKWTFLN5V22POSYMZOCBZAU52W6TSKR24U3PLHTLQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:37:04.463654747 (396h26m39.234296082s) [noise=true , address=104.36.82.22:4543 , piece_id=KJ5YMX5Z7H3XEGKXSAVXBUTDEGK2YN2BXNJHD4CBWHD7UDFMEBLA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:37:19.671277798 (396h26m54.441919133s) [noise=true , address=45.91.95.2:4032 , piece_id=NYJR7Y2CBC3IXJT6QBVOZQOUEUT4XS573ROG27LQQWBDVDOMHBUA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: ecclient: upload failed (node:12BtBTji4bazMXHJXUPMZAmdFxouybziyrzrP8U6oFR8P3obi6f, address:45.91.95.2:4032): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:37:19.671365025 (396h26m54.44200636s) [noise=true , address=104.36.82.44:4888 , piece_id=2F3ZGCXCJNMM7X3ITAHDUY5FOV6XBMY7O57POLYSMJGFL7XANRJQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: ecclient: upload failed (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME, address:104.36.82.44:4888): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:37:19.671428787 (396h26m54.442070122s) [noise=true , address=45.91.95.5:4022 , piece_id=GBTZSDCXLEURLOXZB7K2XTZPIS7QLDCU5YN2ZBF5TPEG2R6MSBTA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: ecclient: upload failed (node:12X8Et5FNyAsE7ifEckfQemZkfnvYUuZ2Rn2xdxNxhmCmuDST2M, address:45.91.95.5:4022): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:37:19.671580928 (396h26m54.442222263s) [noise=true , address=45.91.95.4:4003 , piece_id=KP7IPIU663SIWR7UMAZWZNIGWNJSAE4BZOKD3WKLWEBGVBIPY2RA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: ecclient: upload failed (node:12iLBF47YcG1hvb6vSVNe41MSF6JXgEPYFwLfQcrU8Nt31GDmCg, address:45.91.95.4:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:37:19.671662775 (396h26m54.44230411s) [noise=true , address=45.91.95.3:4002 , piece_id=NI64XOPSDEKRWTTRRAJI2LZZEFLXG5HYWGH7BTSYFF272BDG3TFQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: ecclient: upload failed (node:1dpBPqn6zMpCUkCr8fEZZ17aV6Ag1ytp8Q3pvawZiHQMxGzAUw, address:45.91.95.3:4002): protocol: write tcp 188.166.225.170:42226->45.91.95.3:4002: use of closed network connection; write tcp 188.166.225.170:42226->45.91.95.3:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42226->45.91.95.3:4002: use of closed network connection Jul 9 21:37:19.671678956 (396h26m54.442320291s) [seg_pos={0 0}, upload=317168]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:37:19.903470848 (396h26m54.674112173s) [upload=317168]: Done uploading segments... Jul 9 21:37:19.903494724 (396h26m54.674136049s) [upload=317168]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 31.212197592s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [XXX.........X......X....X..X.........X...............X..X...] duration: 33.580156118s FAIL (2025-07-09 21:38:34 UTC) orbiter warning: context deadline exceeded Duration: 33.580179711s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X..X.......................................................] duration: 6.205240934s FAIL (2025-07-09 21:37:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.205264579s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [.XXX.........X......X....X..X.........X...............X..X..] duration: 39.304850086s FAIL (2025-07-09 21:37:45 UTC) orbiter warning: context deadline exceeded Duration: 39.304869842s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [X..........X...X................X....X...............X..X.X.] duration: 19.880698959s FAIL (2025-07-09 21:37:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== one starting upload Jul 9 21:36:20.318624655 (396h25m55.08926599s) [upload=317168]: Uploading segments... Jul 9 21:36:20.318743822 (396h25m55.089385158s) [upload=317168]: Got next segment. Inline: false Jul 9 21:36:20.318780774 (396h25m55.089422099s) [upload=317168]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:36:20.318785232 (396h25m55.089426567s) [upload=317168]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:36:20.763719228 (396h25m55.534360563s) [seg_pos={0 0}, upload=317168]: Begin upload segment... Jul 9 21:36:20.763769214 (396h25m55.534410549s) [seg_pos={0 0}, upload=317168]: Done begin upload segment. Jul 9 21:36:20.763790274 (396h25m55.534431599s) [noise=true , address=104.36.82.44:4888 , piece_id=2F3ZGCXCJNMM7X3ITAHDUY5FOV6XBMY7O57POLYSMJGFL7XANRJQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.764193555 (396h25m55.53483489s) [noise=true , address=45.91.95.2:4032 , piece_id=NYJR7Y2CBC3IXJT6QBVOZQOUEUT4XS573ROG27LQQWBDVDOMHBUA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.764504661 (396h25m55.535145996s) [noise=true , address=104.36.82.5:4858 , piece_id=OC55PCL57AGUANZHVBSSQ4GCMZ7PKZ3AAZN5SSKWASDCH2OQ6PVA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.764789636 (396h25m55.535430971s) [noise=true , address=77.90.31.17:4785 , piece_id=RDX7V2ES2RK3QUKI7OBE6F3T7DNK5TIIS5JGJG2TENONKS26MOFQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.764857276 (396h25m55.535498611s) [noise=true , address=104.36.82.49:4996 , piece_id=DSDQOSI7R7BEK4ZS72OLCC2HLJ6WZSI652YOHY4X66FOCDKX2O4Q, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.765360789 (396h25m55.536002124s) [noise=true , address=77.90.31.19:4865 , piece_id=2KYAW2GTL7CF5BQ4AOJRWFOT7ZO6SC3AZOGG45JGIZW2LPGGCNDQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.765905511 (396h25m55.536546836s) [noise=true , address=77.90.31.14:4679 , piece_id=ESHQHAOWFYIVNRLXZ64INNNK6UYLBPAIFH4CCX6A6X3VCUZVZUEA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.766140190 (396h25m55.536781515s) [noise=true , address=104.36.82.38:4692 , piece_id=UBJ3M3HEK65QOWK7YRDJC4W4ZUVMIYDRRBV6TA2IF3ZP747L2N7Q, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.766384127 (396h25m55.537025462s) [noise=true , address=104.36.82.41:4770 , piece_id=SYPUVS5NMG4JKWTFLN5V22POSYMZOCBZAU52W6TSKR24U3PLHTLQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.766633814 (396h25m55.537275149s) [noise=true , address=77.90.31.15:4719 , piece_id=6UMEYHDUVSO2OH4K4FMUJCS75EGNCAJZEUKLX2SJLUFX56GNCYHQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.766850239 (396h25m55.537491574s) [noise=true , address=77.90.31.11:4564 , piece_id=HP7Z4KKX7O47BKY6NUNWAZF6WGD7EPQKF2DRTJ7FBGKGMPWMQE3A, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.767051463 (396h25m55.537692798s) [noise=true , address=104.36.82.22:4543 , piece_id=KJ5YMX5Z7H3XEGKXSAVXBUTDEGK2YN2BXNJHD4CBWHD7UDFMEBLA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.767262577 (396h25m55.537903913s) [noise=true , address=77.90.31.18:4820 , piece_id=RJVQ47F34SPY5X5LLGI425LKGRNNAU72VBOYZCAJCRB67OHWCQHA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.767485283 (396h25m55.538126609s) [noise=true , address=45.91.95.3:4002 , piece_id=NI64XOPSDEKRWTTRRAJI2LZZEFLXG5HYWGH7BTSYFF272BDG3TFQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.767685717 (396h25m55.538327052s) [noise=true , address=45.91.95.5:4022 , piece_id=GBTZSDCXLEURLOXZB7K2XTZPIS7QLDCU5YN2ZBF5TPEG2R6MSBTA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.767898654 (396h25m55.538539989s) [noise=true , address=77.90.31.13:4645 , piece_id=S37SRDGED55EZJUCLN26VCHJT2HFQE4PWZ4EMAIAGNE7WB4VFRTA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.768113456 (396h25m55.538754791s) [noise=true , address=77.90.31.10:4551 , piece_id=6G46ASL3W34QTB2IJMSL3HGB22YOVBTKD777IRJXTS4XEAEJIKEA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.768331041 (396h25m55.538972376s) [noise=true , address=45.91.95.4:4003 , piece_id=KP7IPIU663SIWR7UMAZWZNIGWNJSAE4BZOKD3WKLWEBGVBIPY2RA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:22.356647592 (396h25m57.127288927s) [upload=317168]: Next returned nil segment Jul 9 21:36:22.356668483 (396h25m57.127309828s) [upload=317168]: Waiting for error group managing segments... Jul 9 21:36:22.764717222 (396h25m57.535358557s) [noise=true , address=77.90.31.11:4564 , piece_id=HP7Z4KKX7O47BKY6NUNWAZF6WGD7EPQKF2DRTJ7FBGKGMPWMQE3A, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.765409827 (396h25m57.536051162s) [noise=true , address=77.90.31.14:4679 , piece_id=ESHQHAOWFYIVNRLXZ64INNNK6UYLBPAIFH4CCX6A6X3VCUZVZUEA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.766268049 (396h25m57.536909384s) [noise=true , address=77.90.31.18:4820 , piece_id=RJVQ47F34SPY5X5LLGI425LKGRNNAU72VBOYZCAJCRB67OHWCQHA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.767995543 (396h25m57.538636878s) [noise=true , address=77.90.31.17:4785 , piece_id=RDX7V2ES2RK3QUKI7OBE6F3T7DNK5TIIS5JGJG2TENONKS26MOFQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.772020253 (396h25m57.542661589s) [noise=true , address=77.90.31.19:4865 , piece_id=2KYAW2GTL7CF5BQ4AOJRWFOT7ZO6SC3AZOGG45JGIZW2LPGGCNDQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.780841681 (396h25m57.551483016s) [noise=true , address=77.90.31.15:4719 , piece_id=6UMEYHDUVSO2OH4K4FMUJCS75EGNCAJZEUKLX2SJLUFX56GNCYHQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.789850365 (396h25m57.5604917s) [noise=true , address=77.90.31.13:4645 , piece_id=S37SRDGED55EZJUCLN26VCHJT2HFQE4PWZ4EMAIAGNE7WB4VFRTA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.792072737 (396h25m57.562714072s) [noise=true , address=77.90.31.10:4551 , piece_id=6G46ASL3W34QTB2IJMSL3HGB22YOVBTKD777IRJXTS4XEAEJIKEA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:41.743417799 (396h26m16.514059134s) [noise=true , address=104.36.82.38:4692 , piece_id=UBJ3M3HEK65QOWK7YRDJC4W4ZUVMIYDRRBV6TA2IF3ZP747L2N7Q, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:46.945197674 (396h26m21.715839009s) [noise=true , address=104.36.82.5:4858 , piece_id=OC55PCL57AGUANZHVBSSQ4GCMZ7PKZ3AAZN5SSKWASDCH2OQ6PVA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:47.392071762 (396h26m22.162713087s) [noise=true , address=104.36.82.49:4996 , piece_id=DSDQOSI7R7BEK4ZS72OLCC2HLJ6WZSI652YOHY4X66FOCDKX2O4Q, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:37:04.301861761 (396h26m39.072503086s) [noise=true , address=104.36.82.41:4770 , piece_id=SYPUVS5NMG4JKWTFLN5V22POSYMZOCBZAU52W6TSKR24U3PLHTLQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:37:04.463654747 (396h26m39.234296082s) [noise=true , address=104.36.82.22:4543 , piece_id=KJ5YMX5Z7H3XEGKXSAVXBUTDEGK2YN2BXNJHD4CBWHD7UDFMEBLA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:37:19.671277798 (396h26m54.441919133s) [noise=true , address=45.91.95.2:4032 , piece_id=NYJR7Y2CBC3IXJT6QBVOZQOUEUT4XS573ROG27LQQWBDVDOMHBUA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: ecclient: upload failed (node:12BtBTji4bazMXHJXUPMZAmdFxouybziyrzrP8U6oFR8P3obi6f, address:45.91.95.2:4032): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:37:19.671365025 (396h26m54.44200636s) [noise=true , address=104.36.82.44:4888 , piece_id=2F3ZGCXCJNMM7X3ITAHDUY5FOV6XBMY7O57POLYSMJGFL7XANRJQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: ecclient: upload failed (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME, address:104.36.82.44:4888): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:37:19.671428787 (396h26m54.442070122s) [noise=true , address=45.91.95.5:4022 , piece_id=GBTZSDCXLEURLOXZB7K2XTZPIS7QLDCU5YN2ZBF5TPEG2R6MSBTA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: ecclient: upload failed (node:12X8Et5FNyAsE7ifEckfQemZkfnvYUuZ2Rn2xdxNxhmCmuDST2M, address:45.91.95.5:4022): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:37:19.671580928 (396h26m54.442222263s) [noise=true , address=45.91.95.4:4003 , piece_id=KP7IPIU663SIWR7UMAZWZNIGWNJSAE4BZOKD3WKLWEBGVBIPY2RA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: ecclient: upload failed (node:12iLBF47YcG1hvb6vSVNe41MSF6JXgEPYFwLfQcrU8Nt31GDmCg, address:45.91.95.4:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:37:19.671662775 (396h26m54.44230411s) [noise=true , address=45.91.95.3:4002 , piece_id=NI64XOPSDEKRWTTRRAJI2LZZEFLXG5HYWGH7BTSYFF272BDG3TFQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: ecclient: upload failed (node:1dpBPqn6zMpCUkCr8fEZZ17aV6Ag1ytp8Q3pvawZiHQMxGzAUw, address:45.91.95.3:4002): protocol: write tcp 188.166.225.170:42226->45.91.95.3:4002: use of closed network connection; write tcp 188.166.225.170:42226->45.91.95.3:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42226->45.91.95.3:4002: use of closed network connection Jul 9 21:37:19.671678956 (396h26m54.442320291s) [seg_pos={0 0}, upload=317168]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:37:19.903470848 (396h26m54.674112173s) [upload=317168]: Done uploading segments... Jul 9 21:37:19.903494724 (396h26m54.674136049s) [upload=317168]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 19.880714168s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.X..........X...X................X....X...............X..X.X] duration: 1m0.233461344s FAIL (2025-07-09 21:37:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== one starting upload Jul 9 21:36:20.318624655 (396h25m55.08926599s) [upload=317168]: Uploading segments... Jul 9 21:36:20.318743822 (396h25m55.089385158s) [upload=317168]: Got next segment. Inline: false Jul 9 21:36:20.318780774 (396h25m55.089422099s) [upload=317168]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:36:20.318785232 (396h25m55.089426567s) [upload=317168]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:36:20.763719228 (396h25m55.534360563s) [seg_pos={0 0}, upload=317168]: Begin upload segment... Jul 9 21:36:20.763769214 (396h25m55.534410549s) [seg_pos={0 0}, upload=317168]: Done begin upload segment. Jul 9 21:36:20.763790274 (396h25m55.534431599s) [noise=true , address=104.36.82.44:4888 , piece_id=2F3ZGCXCJNMM7X3ITAHDUY5FOV6XBMY7O57POLYSMJGFL7XANRJQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.764193555 (396h25m55.53483489s) [noise=true , address=45.91.95.2:4032 , piece_id=NYJR7Y2CBC3IXJT6QBVOZQOUEUT4XS573ROG27LQQWBDVDOMHBUA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.764504661 (396h25m55.535145996s) [noise=true , address=104.36.82.5:4858 , piece_id=OC55PCL57AGUANZHVBSSQ4GCMZ7PKZ3AAZN5SSKWASDCH2OQ6PVA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.764789636 (396h25m55.535430971s) [noise=true , address=77.90.31.17:4785 , piece_id=RDX7V2ES2RK3QUKI7OBE6F3T7DNK5TIIS5JGJG2TENONKS26MOFQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.764857276 (396h25m55.535498611s) [noise=true , address=104.36.82.49:4996 , piece_id=DSDQOSI7R7BEK4ZS72OLCC2HLJ6WZSI652YOHY4X66FOCDKX2O4Q, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.765360789 (396h25m55.536002124s) [noise=true , address=77.90.31.19:4865 , piece_id=2KYAW2GTL7CF5BQ4AOJRWFOT7ZO6SC3AZOGG45JGIZW2LPGGCNDQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.765905511 (396h25m55.536546836s) [noise=true , address=77.90.31.14:4679 , piece_id=ESHQHAOWFYIVNRLXZ64INNNK6UYLBPAIFH4CCX6A6X3VCUZVZUEA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.766140190 (396h25m55.536781515s) [noise=true , address=104.36.82.38:4692 , piece_id=UBJ3M3HEK65QOWK7YRDJC4W4ZUVMIYDRRBV6TA2IF3ZP747L2N7Q, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.766384127 (396h25m55.537025462s) [noise=true , address=104.36.82.41:4770 , piece_id=SYPUVS5NMG4JKWTFLN5V22POSYMZOCBZAU52W6TSKR24U3PLHTLQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.766633814 (396h25m55.537275149s) [noise=true , address=77.90.31.15:4719 , piece_id=6UMEYHDUVSO2OH4K4FMUJCS75EGNCAJZEUKLX2SJLUFX56GNCYHQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.766850239 (396h25m55.537491574s) [noise=true , address=77.90.31.11:4564 , piece_id=HP7Z4KKX7O47BKY6NUNWAZF6WGD7EPQKF2DRTJ7FBGKGMPWMQE3A, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.767051463 (396h25m55.537692798s) [noise=true , address=104.36.82.22:4543 , piece_id=KJ5YMX5Z7H3XEGKXSAVXBUTDEGK2YN2BXNJHD4CBWHD7UDFMEBLA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.767262577 (396h25m55.537903913s) [noise=true , address=77.90.31.18:4820 , piece_id=RJVQ47F34SPY5X5LLGI425LKGRNNAU72VBOYZCAJCRB67OHWCQHA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.767485283 (396h25m55.538126609s) [noise=true , address=45.91.95.3:4002 , piece_id=NI64XOPSDEKRWTTRRAJI2LZZEFLXG5HYWGH7BTSYFF272BDG3TFQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.767685717 (396h25m55.538327052s) [noise=true , address=45.91.95.5:4022 , piece_id=GBTZSDCXLEURLOXZB7K2XTZPIS7QLDCU5YN2ZBF5TPEG2R6MSBTA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.767898654 (396h25m55.538539989s) [noise=true , address=77.90.31.13:4645 , piece_id=S37SRDGED55EZJUCLN26VCHJT2HFQE4PWZ4EMAIAGNE7WB4VFRTA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.768113456 (396h25m55.538754791s) [noise=true , address=77.90.31.10:4551 , piece_id=6G46ASL3W34QTB2IJMSL3HGB22YOVBTKD777IRJXTS4XEAEJIKEA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.768331041 (396h25m55.538972376s) [noise=true , address=45.91.95.4:4003 , piece_id=KP7IPIU663SIWR7UMAZWZNIGWNJSAE4BZOKD3WKLWEBGVBIPY2RA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:22.356647592 (396h25m57.127288927s) [upload=317168]: Next returned nil segment Jul 9 21:36:22.356668483 (396h25m57.127309828s) [upload=317168]: Waiting for error group managing segments... Jul 9 21:36:22.764717222 (396h25m57.535358557s) [noise=true , address=77.90.31.11:4564 , piece_id=HP7Z4KKX7O47BKY6NUNWAZF6WGD7EPQKF2DRTJ7FBGKGMPWMQE3A, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.765409827 (396h25m57.536051162s) [noise=true , address=77.90.31.14:4679 , piece_id=ESHQHAOWFYIVNRLXZ64INNNK6UYLBPAIFH4CCX6A6X3VCUZVZUEA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.766268049 (396h25m57.536909384s) [noise=true , address=77.90.31.18:4820 , piece_id=RJVQ47F34SPY5X5LLGI425LKGRNNAU72VBOYZCAJCRB67OHWCQHA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.767995543 (396h25m57.538636878s) [noise=true , address=77.90.31.17:4785 , piece_id=RDX7V2ES2RK3QUKI7OBE6F3T7DNK5TIIS5JGJG2TENONKS26MOFQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.772020253 (396h25m57.542661589s) [noise=true , address=77.90.31.19:4865 , piece_id=2KYAW2GTL7CF5BQ4AOJRWFOT7ZO6SC3AZOGG45JGIZW2LPGGCNDQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.780841681 (396h25m57.551483016s) [noise=true , address=77.90.31.15:4719 , piece_id=6UMEYHDUVSO2OH4K4FMUJCS75EGNCAJZEUKLX2SJLUFX56GNCYHQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.789850365 (396h25m57.5604917s) [noise=true , address=77.90.31.13:4645 , piece_id=S37SRDGED55EZJUCLN26VCHJT2HFQE4PWZ4EMAIAGNE7WB4VFRTA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.792072737 (396h25m57.562714072s) [noise=true , address=77.90.31.10:4551 , piece_id=6G46ASL3W34QTB2IJMSL3HGB22YOVBTKD777IRJXTS4XEAEJIKEA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:41.743417799 (396h26m16.514059134s) [noise=true , address=104.36.82.38:4692 , piece_id=UBJ3M3HEK65QOWK7YRDJC4W4ZUVMIYDRRBV6TA2IF3ZP747L2N7Q, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:46.945197674 (396h26m21.715839009s) [noise=true , address=104.36.82.5:4858 , piece_id=OC55PCL57AGUANZHVBSSQ4GCMZ7PKZ3AAZN5SSKWASDCH2OQ6PVA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:47.392071762 (396h26m22.162713087s) [noise=true , address=104.36.82.49:4996 , piece_id=DSDQOSI7R7BEK4ZS72OLCC2HLJ6WZSI652YOHY4X66FOCDKX2O4Q, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:37:04.301861761 (396h26m39.072503086s) [noise=true , address=104.36.82.41:4770 , piece_id=SYPUVS5NMG4JKWTFLN5V22POSYMZOCBZAU52W6TSKR24U3PLHTLQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:37:04.463654747 (396h26m39.234296082s) [noise=true , address=104.36.82.22:4543 , piece_id=KJ5YMX5Z7H3XEGKXSAVXBUTDEGK2YN2BXNJHD4CBWHD7UDFMEBLA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:37:19.671277798 (396h26m54.441919133s) [noise=true , address=45.91.95.2:4032 , piece_id=NYJR7Y2CBC3IXJT6QBVOZQOUEUT4XS573ROG27LQQWBDVDOMHBUA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: ecclient: upload failed (node:12BtBTji4bazMXHJXUPMZAmdFxouybziyrzrP8U6oFR8P3obi6f, address:45.91.95.2:4032): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:37:19.671365025 (396h26m54.44200636s) [noise=true , address=104.36.82.44:4888 , piece_id=2F3ZGCXCJNMM7X3ITAHDUY5FOV6XBMY7O57POLYSMJGFL7XANRJQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: ecclient: upload failed (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME, address:104.36.82.44:4888): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:37:19.671428787 (396h26m54.442070122s) [noise=true , address=45.91.95.5:4022 , piece_id=GBTZSDCXLEURLOXZB7K2XTZPIS7QLDCU5YN2ZBF5TPEG2R6MSBTA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: ecclient: upload failed (node:12X8Et5FNyAsE7ifEckfQemZkfnvYUuZ2Rn2xdxNxhmCmuDST2M, address:45.91.95.5:4022): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:37:19.671580928 (396h26m54.442222263s) [noise=true , address=45.91.95.4:4003 , piece_id=KP7IPIU663SIWR7UMAZWZNIGWNJSAE4BZOKD3WKLWEBGVBIPY2RA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: ecclient: upload failed (node:12iLBF47YcG1hvb6vSVNe41MSF6JXgEPYFwLfQcrU8Nt31GDmCg, address:45.91.95.4:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:37:19.671662775 (396h26m54.44230411s) [noise=true , address=45.91.95.3:4002 , piece_id=NI64XOPSDEKRWTTRRAJI2LZZEFLXG5HYWGH7BTSYFF272BDG3TFQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: ecclient: upload failed (node:1dpBPqn6zMpCUkCr8fEZZ17aV6Ag1ytp8Q3pvawZiHQMxGzAUw, address:45.91.95.3:4002): protocol: write tcp 188.166.225.170:42226->45.91.95.3:4002: use of closed network connection; write tcp 188.166.225.170:42226->45.91.95.3:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42226->45.91.95.3:4002: use of closed network connection Jul 9 21:37:19.671678956 (396h26m54.442320291s) [seg_pos={0 0}, upload=317168]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:37:19.903470848 (396h26m54.674112173s) [upload=317168]: Done uploading segments... Jul 9 21:37:19.903494724 (396h26m54.674136049s) [upload=317168]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.233481853s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [..XXX.........X......X....X..X.........X...............X..X.] duration: 50.370490149s FAIL (2025-07-09 21:37:01 UTC) orbiter warning: context deadline exceeded Duration: 50.370502583s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)