Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X......X..........................] duration: 7.539462841s trace: https://jaeger.dev.storj.io/trace/406600cd16680b82 FAIL (2025-11-11 17:03:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 952]: Done starting upload Nov 11 16:45:40.082304163 (56h41m38.633774624s) [upload=49952]: Uploading segments... Nov 11 16:45:40.082345682 (56h41m38.633816133s) [upload=49952]: Got next segment. Inline: false Nov 11 16:45:40.082372463 (56h41m38.633842924s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 16:45:40.082377623 (56h41m38.633848074s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 16:45:40.343320114 (56h41m38.894790575s) [seg_pos={0 0}, upload=49952]: Begin upload segment... Nov 11 16:45:40.343375529 (56h41m38.89484599s) [seg_pos={0 0}, upload=49952]: Done begin upload segment. Nov 11 16:45:40.343423430 (56h41m38.894893881s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.343448578 (56h41m38.894919029s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344383769 (56h41m38.89585423s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344997518 (56h41m38.896467979s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345095395 (56h41m38.896565856s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345501417 (56h41m38.896971868s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346057606 (56h41m38.897528067s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346422612 (56h41m38.897893063s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346447920 (56h41m38.897918381s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346801393 (56h41m38.898271854s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347099851 (56h41m38.898570302s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347169985 (56h41m38.898640446s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347569877 (56h41m38.899040328s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347577421 (56h41m38.899047872s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347916837 (56h41m38.899387298s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.348105827 (56h41m38.899576288s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:42.155239479 (56h41m40.70670994s) [upload=49952]: Next returned nil segment Nov 11 16:45:42.155253296 (56h41m40.706723757s) [upload=49952]: Waiting for error group managing segments... Nov 11 16:45:42.509736187 (56h41m41.061206668s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.523781983 (56h41m41.075252444s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.524977681 (56h41m41.076448162s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.544084193 (56h41m41.095554654s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.545880774 (56h41m41.097351236s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.617572702 (56h41m41.169043163s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:49.538637614 (56h41m48.090108075s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.209224227 (56h41m53.760694688s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.376479046 (56h41m53.927949507s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.381095176 (56h41m53.932565637s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:02.516152664 (56h42m1.067623125s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:07.807035175 (56h42m6.358505636s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165478509 (56h42m6.71694897s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165504899 (56h42m6.71697536s) [seg_pos={0 0}, upload=49952]: Segment reached optimal threshold of 13 pieces. Nov 11 16:46:08.165731340 (56h42m6.717201801s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection Nov 11 16:46:08.165742050 (56h42m6.717212521s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.165752790 (56h42m6.717223252s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection Nov 11 16:46:08.165760676 (56h42m6.717231137s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166125290 (56h42m6.717595751s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Nov 11 16:46:08.166135960 (56h42m6.717606421s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166155798 (56h42m6.717626259s) [seg_pos={0 0}, upload=49952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 16:46:08.166239006 (56h42m6.717709467s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 16:46:08.166245079 (56h42m6.71771553s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 16:46:08.467714732 (56h42m7.019185193s) [upload=49952]: Done uploading segments... Nov 11 16:46:08.467737616 (56h42m7.019208077s) [upload=49952]: Upload finished. err: <nil> Duration: 7.539492176s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [....X..........] duration: 294.885329ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/553c0de2a794f216 FAIL (2025-11-11 17:03:34 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:57188->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 294.913614ms core website
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 1.442674249s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 17:03:33 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.442708063s core uplink eu1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..........X....] duration: 90.679234ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/21cc4106a0209460 FAIL (2025-11-11 17:03:25 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:51104->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 90.712116ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [...........X...] duration: 64.410912ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/21cc4106a0209460 FAIL (2025-11-11 17:03:09 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:51104->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 64.436883ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.....X.........] duration: 347.034883ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/553c0de2a794f216 FAIL (2025-11-11 17:03:05 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:57188->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 347.070702ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X......X.........................] duration: 11.126974367s trace: https://jaeger.dev.storj.io/trace/406600cd16680b82 FAIL (2025-11-11 17:02:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 952]: Done starting upload Nov 11 16:45:40.082304163 (56h41m38.633774624s) [upload=49952]: Uploading segments... Nov 11 16:45:40.082345682 (56h41m38.633816133s) [upload=49952]: Got next segment. Inline: false Nov 11 16:45:40.082372463 (56h41m38.633842924s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 16:45:40.082377623 (56h41m38.633848074s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 16:45:40.343320114 (56h41m38.894790575s) [seg_pos={0 0}, upload=49952]: Begin upload segment... Nov 11 16:45:40.343375529 (56h41m38.89484599s) [seg_pos={0 0}, upload=49952]: Done begin upload segment. Nov 11 16:45:40.343423430 (56h41m38.894893881s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.343448578 (56h41m38.894919029s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344383769 (56h41m38.89585423s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344997518 (56h41m38.896467979s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345095395 (56h41m38.896565856s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345501417 (56h41m38.896971868s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346057606 (56h41m38.897528067s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346422612 (56h41m38.897893063s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346447920 (56h41m38.897918381s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346801393 (56h41m38.898271854s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347099851 (56h41m38.898570302s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347169985 (56h41m38.898640446s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347569877 (56h41m38.899040328s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347577421 (56h41m38.899047872s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347916837 (56h41m38.899387298s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.348105827 (56h41m38.899576288s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:42.155239479 (56h41m40.70670994s) [upload=49952]: Next returned nil segment Nov 11 16:45:42.155253296 (56h41m40.706723757s) [upload=49952]: Waiting for error group managing segments... Nov 11 16:45:42.509736187 (56h41m41.061206668s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.523781983 (56h41m41.075252444s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.524977681 (56h41m41.076448162s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.544084193 (56h41m41.095554654s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.545880774 (56h41m41.097351236s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.617572702 (56h41m41.169043163s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:49.538637614 (56h41m48.090108075s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.209224227 (56h41m53.760694688s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.376479046 (56h41m53.927949507s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.381095176 (56h41m53.932565637s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:02.516152664 (56h42m1.067623125s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:07.807035175 (56h42m6.358505636s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165478509 (56h42m6.71694897s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165504899 (56h42m6.71697536s) [seg_pos={0 0}, upload=49952]: Segment reached optimal threshold of 13 pieces. Nov 11 16:46:08.165731340 (56h42m6.717201801s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection Nov 11 16:46:08.165742050 (56h42m6.717212521s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.165752790 (56h42m6.717223252s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection Nov 11 16:46:08.165760676 (56h42m6.717231137s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166125290 (56h42m6.717595751s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Nov 11 16:46:08.166135960 (56h42m6.717606421s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166155798 (56h42m6.717626259s) [seg_pos={0 0}, upload=49952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 16:46:08.166239006 (56h42m6.717709467s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 16:46:08.166245079 (56h42m6.71771553s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 16:46:08.467714732 (56h42m7.019185193s) [upload=49952]: Done uploading segments... Nov 11 16:46:08.467737616 (56h42m7.019208077s) [upload=49952]: Upload finished. err: <nil> Duration: 11.127029853s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 7.231265565s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 17:02:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 7.231295432s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [............X..] duration: 68.042068ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/21cc4106a0209460 FAIL (2025-11-11 17:02:31 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:51104->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 68.136067ms core website
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 1.408938619s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 17:02:18 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.408973004s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X......X........................] duration: 7.259683387s trace: https://jaeger.dev.storj.io/trace/406600cd16680b82 FAIL (2025-11-11 17:02:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 952]: Done starting upload Nov 11 16:45:40.082304163 (56h41m38.633774624s) [upload=49952]: Uploading segments... Nov 11 16:45:40.082345682 (56h41m38.633816133s) [upload=49952]: Got next segment. Inline: false Nov 11 16:45:40.082372463 (56h41m38.633842924s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 16:45:40.082377623 (56h41m38.633848074s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 16:45:40.343320114 (56h41m38.894790575s) [seg_pos={0 0}, upload=49952]: Begin upload segment... Nov 11 16:45:40.343375529 (56h41m38.89484599s) [seg_pos={0 0}, upload=49952]: Done begin upload segment. Nov 11 16:45:40.343423430 (56h41m38.894893881s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.343448578 (56h41m38.894919029s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344383769 (56h41m38.89585423s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344997518 (56h41m38.896467979s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345095395 (56h41m38.896565856s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345501417 (56h41m38.896971868s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346057606 (56h41m38.897528067s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346422612 (56h41m38.897893063s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346447920 (56h41m38.897918381s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346801393 (56h41m38.898271854s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347099851 (56h41m38.898570302s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347169985 (56h41m38.898640446s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347569877 (56h41m38.899040328s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347577421 (56h41m38.899047872s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347916837 (56h41m38.899387298s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.348105827 (56h41m38.899576288s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:42.155239479 (56h41m40.70670994s) [upload=49952]: Next returned nil segment Nov 11 16:45:42.155253296 (56h41m40.706723757s) [upload=49952]: Waiting for error group managing segments... Nov 11 16:45:42.509736187 (56h41m41.061206668s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.523781983 (56h41m41.075252444s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.524977681 (56h41m41.076448162s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.544084193 (56h41m41.095554654s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.545880774 (56h41m41.097351236s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.617572702 (56h41m41.169043163s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:49.538637614 (56h41m48.090108075s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.209224227 (56h41m53.760694688s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.376479046 (56h41m53.927949507s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.381095176 (56h41m53.932565637s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:02.516152664 (56h42m1.067623125s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:07.807035175 (56h42m6.358505636s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165478509 (56h42m6.71694897s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165504899 (56h42m6.71697536s) [seg_pos={0 0}, upload=49952]: Segment reached optimal threshold of 13 pieces. Nov 11 16:46:08.165731340 (56h42m6.717201801s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection Nov 11 16:46:08.165742050 (56h42m6.717212521s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.165752790 (56h42m6.717223252s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection Nov 11 16:46:08.165760676 (56h42m6.717231137s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166125290 (56h42m6.717595751s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Nov 11 16:46:08.166135960 (56h42m6.717606421s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166155798 (56h42m6.717626259s) [seg_pos={0 0}, upload=49952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 16:46:08.166239006 (56h42m6.717709467s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 16:46:08.166245079 (56h42m6.71771553s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 16:46:08.467714732 (56h42m7.019185193s) [upload=49952]: Done uploading segments... Nov 11 16:46:08.467737616 (56h42m7.019208077s) [upload=49952]: Upload finished. err: <nil> Duration: 7.259715398s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 7.156154496s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 17:02:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 7.156192348s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [......X........] duration: 413.211789ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/553c0de2a794f216 FAIL (2025-11-11 17:02:05 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:57188->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 413.24888ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.............X.] duration: 95.623359ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/21cc4106a0209460 FAIL (2025-11-11 17:02:03 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:51104->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 95.653075ms core website
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 1.457143642s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 17:01:29 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.457176083s core uplink eu1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.......X.......] duration: 320.578684ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/553c0de2a794f216 FAIL (2025-11-11 17:01:20 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:57188->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 320.608863ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.662680471s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 17:01:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.662720026s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..............X] duration: 683.664734ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/21cc4106a0209460 FAIL (2025-11-11 17:01:08 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:51104->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 683.696213ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X......X.......................] duration: 8.20299231s trace: https://jaeger.dev.storj.io/trace/406600cd16680b82 FAIL (2025-11-11 17:00:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 952]: Done starting upload Nov 11 16:45:40.082304163 (56h41m38.633774624s) [upload=49952]: Uploading segments... Nov 11 16:45:40.082345682 (56h41m38.633816133s) [upload=49952]: Got next segment. Inline: false Nov 11 16:45:40.082372463 (56h41m38.633842924s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 16:45:40.082377623 (56h41m38.633848074s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 16:45:40.343320114 (56h41m38.894790575s) [seg_pos={0 0}, upload=49952]: Begin upload segment... Nov 11 16:45:40.343375529 (56h41m38.89484599s) [seg_pos={0 0}, upload=49952]: Done begin upload segment. Nov 11 16:45:40.343423430 (56h41m38.894893881s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.343448578 (56h41m38.894919029s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344383769 (56h41m38.89585423s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344997518 (56h41m38.896467979s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345095395 (56h41m38.896565856s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345501417 (56h41m38.896971868s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346057606 (56h41m38.897528067s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346422612 (56h41m38.897893063s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346447920 (56h41m38.897918381s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346801393 (56h41m38.898271854s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347099851 (56h41m38.898570302s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347169985 (56h41m38.898640446s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347569877 (56h41m38.899040328s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347577421 (56h41m38.899047872s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347916837 (56h41m38.899387298s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.348105827 (56h41m38.899576288s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:42.155239479 (56h41m40.70670994s) [upload=49952]: Next returned nil segment Nov 11 16:45:42.155253296 (56h41m40.706723757s) [upload=49952]: Waiting for error group managing segments... Nov 11 16:45:42.509736187 (56h41m41.061206668s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.523781983 (56h41m41.075252444s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.524977681 (56h41m41.076448162s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.544084193 (56h41m41.095554654s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.545880774 (56h41m41.097351236s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.617572702 (56h41m41.169043163s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:49.538637614 (56h41m48.090108075s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.209224227 (56h41m53.760694688s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.376479046 (56h41m53.927949507s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.381095176 (56h41m53.932565637s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:02.516152664 (56h42m1.067623125s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:07.807035175 (56h42m6.358505636s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165478509 (56h42m6.71694897s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165504899 (56h42m6.71697536s) [seg_pos={0 0}, upload=49952]: Segment reached optimal threshold of 13 pieces. Nov 11 16:46:08.165731340 (56h42m6.717201801s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection Nov 11 16:46:08.165742050 (56h42m6.717212521s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.165752790 (56h42m6.717223252s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection Nov 11 16:46:08.165760676 (56h42m6.717231137s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166125290 (56h42m6.717595751s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Nov 11 16:46:08.166135960 (56h42m6.717606421s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166155798 (56h42m6.717626259s) [seg_pos={0 0}, upload=49952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 16:46:08.166239006 (56h42m6.717709467s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 16:46:08.166245079 (56h42m6.71771553s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 16:46:08.467714732 (56h42m7.019185193s) [upload=49952]: Done uploading segments... Nov 11 16:46:08.467737616 (56h42m7.019208077s) [upload=49952]: Upload finished. err: <nil> Duration: 8.203026855s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [........X......] duration: 322.211504ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/553c0de2a794f216 FAIL (2025-11-11 17:00:51 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:57188->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 322.276287ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.901666979s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 17:00:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.901695724s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 1.486741834s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 17:00:39 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.486763014s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X......X......................] duration: 8.861432311s trace: https://jaeger.dev.storj.io/trace/406600cd16680b82 FAIL (2025-11-11 17:00:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 952]: Done starting upload Nov 11 16:45:40.082304163 (56h41m38.633774624s) [upload=49952]: Uploading segments... Nov 11 16:45:40.082345682 (56h41m38.633816133s) [upload=49952]: Got next segment. Inline: false Nov 11 16:45:40.082372463 (56h41m38.633842924s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 16:45:40.082377623 (56h41m38.633848074s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 16:45:40.343320114 (56h41m38.894790575s) [seg_pos={0 0}, upload=49952]: Begin upload segment... Nov 11 16:45:40.343375529 (56h41m38.89484599s) [seg_pos={0 0}, upload=49952]: Done begin upload segment. Nov 11 16:45:40.343423430 (56h41m38.894893881s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.343448578 (56h41m38.894919029s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344383769 (56h41m38.89585423s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344997518 (56h41m38.896467979s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345095395 (56h41m38.896565856s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345501417 (56h41m38.896971868s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346057606 (56h41m38.897528067s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346422612 (56h41m38.897893063s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346447920 (56h41m38.897918381s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346801393 (56h41m38.898271854s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347099851 (56h41m38.898570302s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347169985 (56h41m38.898640446s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347569877 (56h41m38.899040328s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347577421 (56h41m38.899047872s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347916837 (56h41m38.899387298s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.348105827 (56h41m38.899576288s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:42.155239479 (56h41m40.70670994s) [upload=49952]: Next returned nil segment Nov 11 16:45:42.155253296 (56h41m40.706723757s) [upload=49952]: Waiting for error group managing segments... Nov 11 16:45:42.509736187 (56h41m41.061206668s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.523781983 (56h41m41.075252444s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.524977681 (56h41m41.076448162s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.544084193 (56h41m41.095554654s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.545880774 (56h41m41.097351236s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.617572702 (56h41m41.169043163s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:49.538637614 (56h41m48.090108075s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.209224227 (56h41m53.760694688s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.376479046 (56h41m53.927949507s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.381095176 (56h41m53.932565637s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:02.516152664 (56h42m1.067623125s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:07.807035175 (56h42m6.358505636s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165478509 (56h42m6.71694897s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165504899 (56h42m6.71697536s) [seg_pos={0 0}, upload=49952]: Segment reached optimal threshold of 13 pieces. Nov 11 16:46:08.165731340 (56h42m6.717201801s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection Nov 11 16:46:08.165742050 (56h42m6.717212521s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.165752790 (56h42m6.717223252s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection Nov 11 16:46:08.165760676 (56h42m6.717231137s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166125290 (56h42m6.717595751s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Nov 11 16:46:08.166135960 (56h42m6.717606421s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166155798 (56h42m6.717626259s) [seg_pos={0 0}, upload=49952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 16:46:08.166239006 (56h42m6.717709467s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 16:46:08.166245079 (56h42m6.71771553s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 16:46:08.467714732 (56h42m7.019185193s) [upload=49952]: Done uploading segments... Nov 11 16:46:08.467737616 (56h42m7.019208077s) [upload=49952]: Upload finished. err: <nil> Duration: 8.861466064s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.........X.....] duration: 341.925673ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/553c0de2a794f216 FAIL (2025-11-11 17:00:13 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:57188->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 341.956753ms core website
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 1.450216342s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:59:58 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.450244176s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X......X.....................] duration: 7.696223908s trace: https://jaeger.dev.storj.io/trace/406600cd16680b82 FAIL (2025-11-11 16:59:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 952]: Done starting upload Nov 11 16:45:40.082304163 (56h41m38.633774624s) [upload=49952]: Uploading segments... Nov 11 16:45:40.082345682 (56h41m38.633816133s) [upload=49952]: Got next segment. Inline: false Nov 11 16:45:40.082372463 (56h41m38.633842924s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 16:45:40.082377623 (56h41m38.633848074s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 16:45:40.343320114 (56h41m38.894790575s) [seg_pos={0 0}, upload=49952]: Begin upload segment... Nov 11 16:45:40.343375529 (56h41m38.89484599s) [seg_pos={0 0}, upload=49952]: Done begin upload segment. Nov 11 16:45:40.343423430 (56h41m38.894893881s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.343448578 (56h41m38.894919029s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344383769 (56h41m38.89585423s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344997518 (56h41m38.896467979s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345095395 (56h41m38.896565856s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345501417 (56h41m38.896971868s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346057606 (56h41m38.897528067s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346422612 (56h41m38.897893063s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346447920 (56h41m38.897918381s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346801393 (56h41m38.898271854s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347099851 (56h41m38.898570302s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347169985 (56h41m38.898640446s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347569877 (56h41m38.899040328s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347577421 (56h41m38.899047872s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347916837 (56h41m38.899387298s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.348105827 (56h41m38.899576288s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:42.155239479 (56h41m40.70670994s) [upload=49952]: Next returned nil segment Nov 11 16:45:42.155253296 (56h41m40.706723757s) [upload=49952]: Waiting for error group managing segments... Nov 11 16:45:42.509736187 (56h41m41.061206668s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.523781983 (56h41m41.075252444s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.524977681 (56h41m41.076448162s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.544084193 (56h41m41.095554654s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.545880774 (56h41m41.097351236s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.617572702 (56h41m41.169043163s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:49.538637614 (56h41m48.090108075s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.209224227 (56h41m53.760694688s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.376479046 (56h41m53.927949507s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.381095176 (56h41m53.932565637s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:02.516152664 (56h42m1.067623125s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:07.807035175 (56h42m6.358505636s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165478509 (56h42m6.71694897s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165504899 (56h42m6.71697536s) [seg_pos={0 0}, upload=49952]: Segment reached optimal threshold of 13 pieces. Nov 11 16:46:08.165731340 (56h42m6.717201801s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection Nov 11 16:46:08.165742050 (56h42m6.717212521s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.165752790 (56h42m6.717223252s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection Nov 11 16:46:08.165760676 (56h42m6.717231137s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166125290 (56h42m6.717595751s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Nov 11 16:46:08.166135960 (56h42m6.717606421s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166155798 (56h42m6.717626259s) [seg_pos={0 0}, upload=49952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 16:46:08.166239006 (56h42m6.717709467s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 16:46:08.166245079 (56h42m6.71771553s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 16:46:08.467714732 (56h42m7.019185193s) [upload=49952]: Done uploading segments... Nov 11 16:46:08.467737616 (56h42m7.019208077s) [upload=49952]: Upload finished. err: <nil> Duration: 7.696276407s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 8.714450027s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 16:59:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 8.714480015s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [..........X....] duration: 311.579038ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/553c0de2a794f216 FAIL (2025-11-11 16:59:35 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:57188->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 311.602542ms core website
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 1.421325535s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:59:14 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.421355392s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X...............................X......X....................] duration: 6.932150683s trace: https://jaeger.dev.storj.io/trace/406600cd16680b82 FAIL (2025-11-11 16:59:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 952]: Done starting upload Nov 11 16:45:40.082304163 (56h41m38.633774624s) [upload=49952]: Uploading segments... Nov 11 16:45:40.082345682 (56h41m38.633816133s) [upload=49952]: Got next segment. Inline: false Nov 11 16:45:40.082372463 (56h41m38.633842924s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 16:45:40.082377623 (56h41m38.633848074s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 16:45:40.343320114 (56h41m38.894790575s) [seg_pos={0 0}, upload=49952]: Begin upload segment... Nov 11 16:45:40.343375529 (56h41m38.89484599s) [seg_pos={0 0}, upload=49952]: Done begin upload segment. Nov 11 16:45:40.343423430 (56h41m38.894893881s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.343448578 (56h41m38.894919029s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344383769 (56h41m38.89585423s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344997518 (56h41m38.896467979s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345095395 (56h41m38.896565856s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345501417 (56h41m38.896971868s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346057606 (56h41m38.897528067s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346422612 (56h41m38.897893063s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346447920 (56h41m38.897918381s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346801393 (56h41m38.898271854s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347099851 (56h41m38.898570302s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347169985 (56h41m38.898640446s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347569877 (56h41m38.899040328s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347577421 (56h41m38.899047872s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347916837 (56h41m38.899387298s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.348105827 (56h41m38.899576288s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:42.155239479 (56h41m40.70670994s) [upload=49952]: Next returned nil segment Nov 11 16:45:42.155253296 (56h41m40.706723757s) [upload=49952]: Waiting for error group managing segments... Nov 11 16:45:42.509736187 (56h41m41.061206668s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.523781983 (56h41m41.075252444s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.524977681 (56h41m41.076448162s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.544084193 (56h41m41.095554654s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.545880774 (56h41m41.097351236s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.617572702 (56h41m41.169043163s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:49.538637614 (56h41m48.090108075s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.209224227 (56h41m53.760694688s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.376479046 (56h41m53.927949507s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.381095176 (56h41m53.932565637s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:02.516152664 (56h42m1.067623125s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:07.807035175 (56h42m6.358505636s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165478509 (56h42m6.71694897s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165504899 (56h42m6.71697536s) [seg_pos={0 0}, upload=49952]: Segment reached optimal threshold of 13 pieces. Nov 11 16:46:08.165731340 (56h42m6.717201801s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection Nov 11 16:46:08.165742050 (56h42m6.717212521s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.165752790 (56h42m6.717223252s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection Nov 11 16:46:08.165760676 (56h42m6.717231137s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166125290 (56h42m6.717595751s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Nov 11 16:46:08.166135960 (56h42m6.717606421s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166155798 (56h42m6.717626259s) [seg_pos={0 0}, upload=49952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 16:46:08.166239006 (56h42m6.717709467s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 16:46:08.166245079 (56h42m6.71771553s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 16:46:08.467714732 (56h42m7.019185193s) [upload=49952]: Done uploading segments... Nov 11 16:46:08.467737616 (56h42m7.019208077s) [upload=49952]: Upload finished. err: <nil> Duration: 6.932182673s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 7.275666147s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 16:58:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 7.275723945s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [...........X...] duration: 617.754334ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/553c0de2a794f216 FAIL (2025-11-11 16:58:33 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:57188->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 617.775355ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X...............................X......X...................] duration: 8.278766731s trace: https://jaeger.dev.storj.io/trace/406600cd16680b82 FAIL (2025-11-11 16:58:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 952]: Done starting upload Nov 11 16:45:40.082304163 (56h41m38.633774624s) [upload=49952]: Uploading segments... Nov 11 16:45:40.082345682 (56h41m38.633816133s) [upload=49952]: Got next segment. Inline: false Nov 11 16:45:40.082372463 (56h41m38.633842924s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 16:45:40.082377623 (56h41m38.633848074s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 16:45:40.343320114 (56h41m38.894790575s) [seg_pos={0 0}, upload=49952]: Begin upload segment... Nov 11 16:45:40.343375529 (56h41m38.89484599s) [seg_pos={0 0}, upload=49952]: Done begin upload segment. Nov 11 16:45:40.343423430 (56h41m38.894893881s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.343448578 (56h41m38.894919029s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344383769 (56h41m38.89585423s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344997518 (56h41m38.896467979s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345095395 (56h41m38.896565856s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345501417 (56h41m38.896971868s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346057606 (56h41m38.897528067s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346422612 (56h41m38.897893063s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346447920 (56h41m38.897918381s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346801393 (56h41m38.898271854s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347099851 (56h41m38.898570302s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347169985 (56h41m38.898640446s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347569877 (56h41m38.899040328s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347577421 (56h41m38.899047872s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347916837 (56h41m38.899387298s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.348105827 (56h41m38.899576288s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:42.155239479 (56h41m40.70670994s) [upload=49952]: Next returned nil segment Nov 11 16:45:42.155253296 (56h41m40.706723757s) [upload=49952]: Waiting for error group managing segments... Nov 11 16:45:42.509736187 (56h41m41.061206668s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.523781983 (56h41m41.075252444s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.524977681 (56h41m41.076448162s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.544084193 (56h41m41.095554654s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.545880774 (56h41m41.097351236s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.617572702 (56h41m41.169043163s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:49.538637614 (56h41m48.090108075s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.209224227 (56h41m53.760694688s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.376479046 (56h41m53.927949507s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.381095176 (56h41m53.932565637s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:02.516152664 (56h42m1.067623125s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:07.807035175 (56h42m6.358505636s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165478509 (56h42m6.71694897s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165504899 (56h42m6.71697536s) [seg_pos={0 0}, upload=49952]: Segment reached optimal threshold of 13 pieces. Nov 11 16:46:08.165731340 (56h42m6.717201801s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection Nov 11 16:46:08.165742050 (56h42m6.717212521s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.165752790 (56h42m6.717223252s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection Nov 11 16:46:08.165760676 (56h42m6.717231137s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166125290 (56h42m6.717595751s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Nov 11 16:46:08.166135960 (56h42m6.717606421s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166155798 (56h42m6.717626259s) [seg_pos={0 0}, upload=49952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 16:46:08.166239006 (56h42m6.717709467s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 16:46:08.166245079 (56h42m6.71771553s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 16:46:08.467714732 (56h42m7.019185193s) [upload=49952]: Done uploading segments... Nov 11 16:46:08.467737616 (56h42m7.019208077s) [upload=49952]: Upload finished. err: <nil> Duration: 8.278794023s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 1.434940781s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:58:09 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.434966161s core uplink eu1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [............X..] duration: 348.469354ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/553c0de2a794f216 FAIL (2025-11-11 16:58:05 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:57188->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 348.48703ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.30668998s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 16:57:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.306721901s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 7.852577477s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 16:57:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 7.852598838s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 1.425815696s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:57:25 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.425845282s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X...............................X......X..................] duration: 8.544389473s trace: https://jaeger.dev.storj.io/trace/406600cd16680b82 FAIL (2025-11-11 16:57:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 952]: Done starting upload Nov 11 16:45:40.082304163 (56h41m38.633774624s) [upload=49952]: Uploading segments... Nov 11 16:45:40.082345682 (56h41m38.633816133s) [upload=49952]: Got next segment. Inline: false Nov 11 16:45:40.082372463 (56h41m38.633842924s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 16:45:40.082377623 (56h41m38.633848074s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 16:45:40.343320114 (56h41m38.894790575s) [seg_pos={0 0}, upload=49952]: Begin upload segment... Nov 11 16:45:40.343375529 (56h41m38.89484599s) [seg_pos={0 0}, upload=49952]: Done begin upload segment. Nov 11 16:45:40.343423430 (56h41m38.894893881s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.343448578 (56h41m38.894919029s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344383769 (56h41m38.89585423s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344997518 (56h41m38.896467979s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345095395 (56h41m38.896565856s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345501417 (56h41m38.896971868s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346057606 (56h41m38.897528067s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346422612 (56h41m38.897893063s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346447920 (56h41m38.897918381s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346801393 (56h41m38.898271854s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347099851 (56h41m38.898570302s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347169985 (56h41m38.898640446s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347569877 (56h41m38.899040328s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347577421 (56h41m38.899047872s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347916837 (56h41m38.899387298s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.348105827 (56h41m38.899576288s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:42.155239479 (56h41m40.70670994s) [upload=49952]: Next returned nil segment Nov 11 16:45:42.155253296 (56h41m40.706723757s) [upload=49952]: Waiting for error group managing segments... Nov 11 16:45:42.509736187 (56h41m41.061206668s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.523781983 (56h41m41.075252444s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.524977681 (56h41m41.076448162s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.544084193 (56h41m41.095554654s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.545880774 (56h41m41.097351236s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.617572702 (56h41m41.169043163s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:49.538637614 (56h41m48.090108075s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.209224227 (56h41m53.760694688s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.376479046 (56h41m53.927949507s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.381095176 (56h41m53.932565637s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:02.516152664 (56h42m1.067623125s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:07.807035175 (56h42m6.358505636s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165478509 (56h42m6.71694897s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165504899 (56h42m6.71697536s) [seg_pos={0 0}, upload=49952]: Segment reached optimal threshold of 13 pieces. Nov 11 16:46:08.165731340 (56h42m6.717201801s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection Nov 11 16:46:08.165742050 (56h42m6.717212521s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.165752790 (56h42m6.717223252s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection Nov 11 16:46:08.165760676 (56h42m6.717231137s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166125290 (56h42m6.717595751s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Nov 11 16:46:08.166135960 (56h42m6.717606421s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166155798 (56h42m6.717626259s) [seg_pos={0 0}, upload=49952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 16:46:08.166239006 (56h42m6.717709467s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 16:46:08.166245079 (56h42m6.71771553s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 16:46:08.467714732 (56h42m7.019185193s) [upload=49952]: Done uploading segments... Nov 11 16:46:08.467737616 (56h42m7.019208077s) [upload=49952]: Upload finished. err: <nil> Duration: 8.544452993s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [X..............] duration: 80.06407ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/7a59bb67253c8438 FAIL (2025-11-11 16:57:00 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59902->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 80.114156ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.............X.] duration: 313.346944ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/553c0de2a794f216 FAIL (2025-11-11 16:56:57 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:57188->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 313.434931ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 6.262833452s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 16:56:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.262861075s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [..............X] duration: 665.773855ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/553c0de2a794f216 FAIL (2025-11-11 16:56:28 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:57188->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 665.799715ms core website
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 1.506505198s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:56:19 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.507635721s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X...............................X......X.................] duration: 7.067510775s trace: https://jaeger.dev.storj.io/trace/406600cd16680b82 FAIL (2025-11-11 16:56:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 952]: Done starting upload Nov 11 16:45:40.082304163 (56h41m38.633774624s) [upload=49952]: Uploading segments... Nov 11 16:45:40.082345682 (56h41m38.633816133s) [upload=49952]: Got next segment. Inline: false Nov 11 16:45:40.082372463 (56h41m38.633842924s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 16:45:40.082377623 (56h41m38.633848074s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 16:45:40.343320114 (56h41m38.894790575s) [seg_pos={0 0}, upload=49952]: Begin upload segment... Nov 11 16:45:40.343375529 (56h41m38.89484599s) [seg_pos={0 0}, upload=49952]: Done begin upload segment. Nov 11 16:45:40.343423430 (56h41m38.894893881s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.343448578 (56h41m38.894919029s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344383769 (56h41m38.89585423s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344997518 (56h41m38.896467979s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345095395 (56h41m38.896565856s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345501417 (56h41m38.896971868s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346057606 (56h41m38.897528067s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346422612 (56h41m38.897893063s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346447920 (56h41m38.897918381s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346801393 (56h41m38.898271854s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347099851 (56h41m38.898570302s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347169985 (56h41m38.898640446s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347569877 (56h41m38.899040328s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347577421 (56h41m38.899047872s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347916837 (56h41m38.899387298s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.348105827 (56h41m38.899576288s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:42.155239479 (56h41m40.70670994s) [upload=49952]: Next returned nil segment Nov 11 16:45:42.155253296 (56h41m40.706723757s) [upload=49952]: Waiting for error group managing segments... Nov 11 16:45:42.509736187 (56h41m41.061206668s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.523781983 (56h41m41.075252444s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.524977681 (56h41m41.076448162s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.544084193 (56h41m41.095554654s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.545880774 (56h41m41.097351236s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.617572702 (56h41m41.169043163s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:49.538637614 (56h41m48.090108075s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.209224227 (56h41m53.760694688s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.376479046 (56h41m53.927949507s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.381095176 (56h41m53.932565637s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:02.516152664 (56h42m1.067623125s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:07.807035175 (56h42m6.358505636s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165478509 (56h42m6.71694897s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165504899 (56h42m6.71697536s) [seg_pos={0 0}, upload=49952]: Segment reached optimal threshold of 13 pieces. Nov 11 16:46:08.165731340 (56h42m6.717201801s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection Nov 11 16:46:08.165742050 (56h42m6.717212521s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.165752790 (56h42m6.717223252s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection Nov 11 16:46:08.165760676 (56h42m6.717231137s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166125290 (56h42m6.717595751s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Nov 11 16:46:08.166135960 (56h42m6.717606421s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166155798 (56h42m6.717626259s) [seg_pos={0 0}, upload=49952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 16:46:08.166239006 (56h42m6.717709467s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 16:46:08.166245079 (56h42m6.71771553s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 16:46:08.467714732 (56h42m7.019185193s) [upload=49952]: Done uploading segments... Nov 11 16:46:08.467737616 (56h42m7.019208077s) [upload=49952]: Upload finished. err: <nil> Duration: 7.067574377s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.X.............] duration: 58.839293ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/7a59bb67253c8438 FAIL (2025-11-11 16:56:12 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59902->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 58.863189ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.169778591s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 16:55:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.169801735s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..X............] duration: 68.561407ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/7a59bb67253c8438 FAIL (2025-11-11 16:55:55 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59902->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 68.590783ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X...............................X......X................] duration: 8.475983546s trace: https://jaeger.dev.storj.io/trace/406600cd16680b82 FAIL (2025-11-11 16:55:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 952]: Done starting upload Nov 11 16:45:40.082304163 (56h41m38.633774624s) [upload=49952]: Uploading segments... Nov 11 16:45:40.082345682 (56h41m38.633816133s) [upload=49952]: Got next segment. Inline: false Nov 11 16:45:40.082372463 (56h41m38.633842924s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 16:45:40.082377623 (56h41m38.633848074s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 16:45:40.343320114 (56h41m38.894790575s) [seg_pos={0 0}, upload=49952]: Begin upload segment... Nov 11 16:45:40.343375529 (56h41m38.89484599s) [seg_pos={0 0}, upload=49952]: Done begin upload segment. Nov 11 16:45:40.343423430 (56h41m38.894893881s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.343448578 (56h41m38.894919029s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344383769 (56h41m38.89585423s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344997518 (56h41m38.896467979s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345095395 (56h41m38.896565856s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345501417 (56h41m38.896971868s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346057606 (56h41m38.897528067s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346422612 (56h41m38.897893063s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346447920 (56h41m38.897918381s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346801393 (56h41m38.898271854s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347099851 (56h41m38.898570302s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347169985 (56h41m38.898640446s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347569877 (56h41m38.899040328s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347577421 (56h41m38.899047872s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347916837 (56h41m38.899387298s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.348105827 (56h41m38.899576288s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:42.155239479 (56h41m40.70670994s) [upload=49952]: Next returned nil segment Nov 11 16:45:42.155253296 (56h41m40.706723757s) [upload=49952]: Waiting for error group managing segments... Nov 11 16:45:42.509736187 (56h41m41.061206668s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.523781983 (56h41m41.075252444s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.524977681 (56h41m41.076448162s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.544084193 (56h41m41.095554654s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.545880774 (56h41m41.097351236s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.617572702 (56h41m41.169043163s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:49.538637614 (56h41m48.090108075s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.209224227 (56h41m53.760694688s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.376479046 (56h41m53.927949507s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.381095176 (56h41m53.932565637s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:02.516152664 (56h42m1.067623125s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:07.807035175 (56h42m6.358505636s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165478509 (56h42m6.71694897s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165504899 (56h42m6.71697536s) [seg_pos={0 0}, upload=49952]: Segment reached optimal threshold of 13 pieces. Nov 11 16:46:08.165731340 (56h42m6.717201801s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection Nov 11 16:46:08.165742050 (56h42m6.717212521s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.165752790 (56h42m6.717223252s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection Nov 11 16:46:08.165760676 (56h42m6.717231137s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166125290 (56h42m6.717595751s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Nov 11 16:46:08.166135960 (56h42m6.717606421s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166155798 (56h42m6.717626259s) [seg_pos={0 0}, upload=49952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 16:46:08.166239006 (56h42m6.717709467s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 16:46:08.166245079 (56h42m6.71771553s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 16:46:08.467714732 (56h42m7.019185193s) [upload=49952]: Done uploading segments... Nov 11 16:46:08.467737616 (56h42m7.019208077s) [upload=49952]: Upload finished. err: <nil> Duration: 8.476016116s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 7.01442549s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 16:55:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 7.014460675s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)