Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................................X...............] duration: 3.524108659s trace: https://jaeger.dev.storj.io/trace/469b06ffc0b434d4 FAIL (2025-11-22 12:02:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.524139378s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.7323819s trace: https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e FAIL (2025-11-22 12:01:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded ======LOGS====== nts... Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment... Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment. Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments... Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces. Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments... Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil> Duration: 5.732411456s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................................X..............] duration: 3.516155598s trace: https://jaeger.dev.storj.io/trace/469b06ffc0b434d4 FAIL (2025-11-22 12:01:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.516184863s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.828488573s trace: https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e FAIL (2025-11-22 12:00:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded ======LOGS====== nts... Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment... Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment. Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments... Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces. Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments... Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil> Duration: 5.828555952s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................................X.............] duration: 3.825644299s trace: https://jaeger.dev.storj.io/trace/469b06ffc0b434d4 FAIL (2025-11-22 12:00:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.825917109s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.711378263s trace: https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e FAIL (2025-11-22 11:59:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded ======LOGS====== nts... Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment... Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment. Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments... Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces. Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments... Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil> Duration: 5.711410956s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................................X............] duration: 3.517731327s trace: https://jaeger.dev.storj.io/trace/469b06ffc0b434d4 FAIL (2025-11-22 11:59:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.517832309s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.63944648s trace: https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e FAIL (2025-11-22 11:59:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded ======LOGS====== nts... Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment... Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment. Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments... Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces. Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments... Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil> Duration: 5.639481046s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................................X...........] duration: 3.486490938s trace: https://jaeger.dev.storj.io/trace/469b06ffc0b434d4 FAIL (2025-11-22 11:58:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.486526315s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.932563892s trace: https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e FAIL (2025-11-22 11:58:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded ======LOGS====== nts... Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment... Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment. Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments... Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces. Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments... Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil> Duration: 5.932589942s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 3.457657894s trace: https://jaeger.dev.storj.io/trace/469b06ffc0b434d4 FAIL (2025-11-22 11:57:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.45768787s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 3.503986572s trace: https://jaeger.dev.storj.io/trace/469b06ffc0b434d4 FAIL (2025-11-22 11:57:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.504019164s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.828611577s trace: https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e FAIL (2025-11-22 11:57:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded ======LOGS====== nts... Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment... Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment. Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments... Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces. Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments... Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil> Duration: 5.828639561s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 3.327779663s trace: https://jaeger.dev.storj.io/trace/469b06ffc0b434d4 FAIL (2025-11-22 11:56:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.32781521s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.797784997s trace: https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e FAIL (2025-11-22 11:56:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded ======LOGS====== nts... Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment... Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment. Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments... Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces. Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments... Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil> Duration: 5.797811969s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 3.342375435s trace: https://jaeger.dev.storj.io/trace/469b06ffc0b434d4 FAIL (2025-11-22 11:56:23 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.342406786s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.940730391s trace: https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e FAIL (2025-11-22 11:56:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded ======LOGS====== nts... Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment... Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment. Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments... Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces. Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments... Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil> Duration: 5.940759507s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 3.726387605s trace: https://jaeger.dev.storj.io/trace/469b06ffc0b434d4 FAIL (2025-11-22 11:55:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.726416049s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.697384899s trace: https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e FAIL (2025-11-22 11:55:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded ======LOGS====== nts... Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment... Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment. Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments... Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces. Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments... Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil> Duration: 5.697418192s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 7.067302669s trace: https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e FAIL (2025-11-22 11:54:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded ======LOGS====== nts... Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment... Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment. Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments... Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces. Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments... Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil> Duration: 7.067380166s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 3.542757108s trace: https://jaeger.dev.storj.io/trace/469b06ffc0b434d4 FAIL (2025-11-22 11:54:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.542784781s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.002792947s trace: https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e FAIL (2025-11-22 11:54:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded ======LOGS====== nts... Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment... Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment. Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments... Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces. Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments... Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil> Duration: 30.002808137s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 3.551642717s trace: https://jaeger.dev.storj.io/trace/469b06ffc0b434d4 FAIL (2025-11-22 11:53:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.551683023s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 3.6613077s trace: https://jaeger.dev.storj.io/trace/469b06ffc0b434d4 FAIL (2025-11-22 11:52:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.661333719s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 3.616059688s trace: https://jaeger.dev.storj.io/trace/469b06ffc0b434d4 FAIL (2025-11-22 11:52:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.616085246s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 3.808733456s trace: https://jaeger.dev.storj.io/trace/469b06ffc0b434d4 FAIL (2025-11-22 11:50:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.808759786s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.001291512s trace: https://jaeger.dev.storj.io/trace/469b06ffc0b434d4 FAIL (2025-11-22 11:50:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001315127s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.745272323s trace: https://jaeger.dev.storj.io/trace/3888219c34a2aa24 FAIL (2025-11-22 06:50:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.745303403s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 3.518738283s trace: https://jaeger.dev.storj.io/trace/450296347d077d5 FAIL (2025-11-22 06:49:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.518760525s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.631437772s trace: https://jaeger.dev.storj.io/trace/3888219c34a2aa24 FAIL (2025-11-22 06:49:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.631471146s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 3.446015246s trace: https://jaeger.dev.storj.io/trace/450296347d077d5 FAIL (2025-11-22 06:48:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.446061063s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.906046409s trace: https://jaeger.dev.storj.io/trace/3888219c34a2aa24 FAIL (2025-11-22 06:48:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.906075404s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 3.437506115s trace: https://jaeger.dev.storj.io/trace/450296347d077d5 FAIL (2025-11-22 06:47:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.437538167s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.301596572s trace: https://jaeger.dev.storj.io/trace/3888219c34a2aa24 FAIL (2025-11-22 06:47:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.301625658s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 3.380856794s trace: https://jaeger.dev.storj.io/trace/450296347d077d5 FAIL (2025-11-22 06:47:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.380887593s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.670043789s trace: https://jaeger.dev.storj.io/trace/3888219c34a2aa24 FAIL (2025-11-22 06:46:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.670069558s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.256007682s trace: https://jaeger.dev.storj.io/trace/3888219c34a2aa24 FAIL (2025-11-22 06:46:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.256039442s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 3.379249264s trace: https://jaeger.dev.storj.io/trace/450296347d077d5 FAIL (2025-11-22 06:46:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.379280903s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 3.375901468s trace: https://jaeger.dev.storj.io/trace/450296347d077d5 FAIL (2025-11-22 06:45:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.37592894s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 6.440299514s trace: https://jaeger.dev.storj.io/trace/3888219c34a2aa24 FAIL (2025-11-22 06:45:09 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.440332076s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 3.801804407s trace: https://jaeger.dev.storj.io/trace/450296347d077d5 FAIL (2025-11-22 06:44:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.801828323s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.96399799s trace: https://jaeger.dev.storj.io/trace/3888219c34a2aa24 FAIL (2025-11-22 06:43:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.964024369s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 3.464292319s trace: https://jaeger.dev.storj.io/trace/450296347d077d5 FAIL (2025-11-22 06:43:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.464312938s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 3.474482534s trace: https://jaeger.dev.storj.io/trace/450296347d077d5 FAIL (2025-11-22 06:43:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.474510968s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.91335916s trace: https://jaeger.dev.storj.io/trace/3888219c34a2aa24 FAIL (2025-11-22 06:43:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.913386803s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 3.472521579s trace: https://jaeger.dev.storj.io/trace/450296347d077d5 FAIL (2025-11-22 06:42:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.472540054s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.742013796s trace: https://jaeger.dev.storj.io/trace/3888219c34a2aa24 FAIL (2025-11-22 06:42:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.742045315s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 3.5131229s trace: https://jaeger.dev.storj.io/trace/450296347d077d5 FAIL (2025-11-22 06:41:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.513150814s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.80400329s trace: https://jaeger.dev.storj.io/trace/3888219c34a2aa24 FAIL (2025-11-22 06:41:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.804032646s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 3.7336462s trace: https://jaeger.dev.storj.io/trace/450296347d077d5 FAIL (2025-11-22 06:41:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.733694062s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)