Orbiter History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 1.442644847s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:46:39 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.442674063s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [............XX.....X...............................X......X.] duration: 7.521748183s trace: https://jaeger.dev.storj.io/trace/406600cd16680b82 FAIL (2025-11-11 16:46:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 952]: Done starting upload Nov 11 16:45:40.082304163 (56h41m38.633774624s) [upload=49952]: Uploading segments... Nov 11 16:45:40.082345682 (56h41m38.633816133s) [upload=49952]: Got next segment. Inline: false Nov 11 16:45:40.082372463 (56h41m38.633842924s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 16:45:40.082377623 (56h41m38.633848074s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 16:45:40.343320114 (56h41m38.894790575s) [seg_pos={0 0}, upload=49952]: Begin upload segment... Nov 11 16:45:40.343375529 (56h41m38.89484599s) [seg_pos={0 0}, upload=49952]: Done begin upload segment. Nov 11 16:45:40.343423430 (56h41m38.894893881s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.343448578 (56h41m38.894919029s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344383769 (56h41m38.89585423s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344997518 (56h41m38.896467979s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345095395 (56h41m38.896565856s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345501417 (56h41m38.896971868s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346057606 (56h41m38.897528067s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346422612 (56h41m38.897893063s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346447920 (56h41m38.897918381s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346801393 (56h41m38.898271854s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347099851 (56h41m38.898570302s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347169985 (56h41m38.898640446s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347569877 (56h41m38.899040328s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347577421 (56h41m38.899047872s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347916837 (56h41m38.899387298s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.348105827 (56h41m38.899576288s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:42.155239479 (56h41m40.70670994s) [upload=49952]: Next returned nil segment Nov 11 16:45:42.155253296 (56h41m40.706723757s) [upload=49952]: Waiting for error group managing segments... Nov 11 16:45:42.509736187 (56h41m41.061206668s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.523781983 (56h41m41.075252444s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.524977681 (56h41m41.076448162s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.544084193 (56h41m41.095554654s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.545880774 (56h41m41.097351236s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.617572702 (56h41m41.169043163s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:49.538637614 (56h41m48.090108075s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.209224227 (56h41m53.760694688s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.376479046 (56h41m53.927949507s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.381095176 (56h41m53.932565637s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:02.516152664 (56h42m1.067623125s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:07.807035175 (56h42m6.358505636s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165478509 (56h42m6.71694897s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165504899 (56h42m6.71697536s) [seg_pos={0 0}, upload=49952]: Segment reached optimal threshold of 13 pieces. Nov 11 16:46:08.165731340 (56h42m6.717201801s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection Nov 11 16:46:08.165742050 (56h42m6.717212521s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.165752790 (56h42m6.717223252s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection Nov 11 16:46:08.165760676 (56h42m6.717231137s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166125290 (56h42m6.717595751s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Nov 11 16:46:08.166135960 (56h42m6.717606421s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166155798 (56h42m6.717626259s) [seg_pos={0 0}, upload=49952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 16:46:08.166239006 (56h42m6.717709467s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 16:46:08.166245079 (56h42m6.71771553s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 16:46:08.467714732 (56h42m7.019185193s) [upload=49952]: Done uploading segments... Nov 11 16:46:08.467737616 (56h42m7.019208077s) [upload=49952]: Upload finished. err: <nil> Duration: 7.521776398s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............XX.....X...............................X......X] duration: 30.003098215s trace: https://jaeger.dev.storj.io/trace/406600cd16680b82 FAIL (2025-11-11 16:46:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 952]: Done starting upload Nov 11 16:45:40.082304163 (56h41m38.633774624s) [upload=49952]: Uploading segments... Nov 11 16:45:40.082345682 (56h41m38.633816133s) [upload=49952]: Got next segment. Inline: false Nov 11 16:45:40.082372463 (56h41m38.633842924s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 16:45:40.082377623 (56h41m38.633848074s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 16:45:40.343320114 (56h41m38.894790575s) [seg_pos={0 0}, upload=49952]: Begin upload segment... Nov 11 16:45:40.343375529 (56h41m38.89484599s) [seg_pos={0 0}, upload=49952]: Done begin upload segment. Nov 11 16:45:40.343423430 (56h41m38.894893881s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.343448578 (56h41m38.894919029s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344383769 (56h41m38.89585423s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344997518 (56h41m38.896467979s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345095395 (56h41m38.896565856s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345501417 (56h41m38.896971868s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346057606 (56h41m38.897528067s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346422612 (56h41m38.897893063s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346447920 (56h41m38.897918381s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346801393 (56h41m38.898271854s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347099851 (56h41m38.898570302s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347169985 (56h41m38.898640446s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347569877 (56h41m38.899040328s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347577421 (56h41m38.899047872s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347916837 (56h41m38.899387298s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.348105827 (56h41m38.899576288s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:42.155239479 (56h41m40.70670994s) [upload=49952]: Next returned nil segment Nov 11 16:45:42.155253296 (56h41m40.706723757s) [upload=49952]: Waiting for error group managing segments... Nov 11 16:45:42.509736187 (56h41m41.061206668s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.523781983 (56h41m41.075252444s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.524977681 (56h41m41.076448162s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.544084193 (56h41m41.095554654s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.545880774 (56h41m41.097351236s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.617572702 (56h41m41.169043163s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:49.538637614 (56h41m48.090108075s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.209224227 (56h41m53.760694688s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.376479046 (56h41m53.927949507s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.381095176 (56h41m53.932565637s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:02.516152664 (56h42m1.067623125s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:07.807035175 (56h42m6.358505636s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165478509 (56h42m6.71694897s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165504899 (56h42m6.71697536s) [seg_pos={0 0}, upload=49952]: Segment reached optimal threshold of 13 pieces. Nov 11 16:46:08.165731340 (56h42m6.717201801s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection Nov 11 16:46:08.165742050 (56h42m6.717212521s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.165752790 (56h42m6.717223252s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection Nov 11 16:46:08.165760676 (56h42m6.717231137s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166125290 (56h42m6.717595751s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Nov 11 16:46:08.166135960 (56h42m6.717606421s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166155798 (56h42m6.717626259s) [seg_pos={0 0}, upload=49952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 16:46:08.166239006 (56h42m6.717709467s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 16:46:08.166245079 (56h42m6.71771553s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 16:46:08.467714732 (56h42m7.019185193s) [upload=49952]: Done uploading segments... Nov 11 16:46:08.467737616 (56h42m7.019208077s) [upload=49952]: Upload finished. err: <nil> Duration: 30.003119025s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 2 out of 15 is failed (threshold=6) [......X.......X] duration: 279.163959ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/7a59bb67253c8438 FAIL (2025-11-11 16:46:07 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59902->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 279.194506ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X.......................................X.........] duration: 17.095559367s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 16:45:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 17.095586218s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 1.405485048s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:45:49 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.405511107s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X.......................................X........] duration: 20.076966224s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 16:45:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 20.077077304s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.......X.......] duration: 52.276967ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/644568d221d3a147 FAIL (2025-11-11 16:45:13 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:45680->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 52.313797ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X.......................................X.......] duration: 6.313278176s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 16:45:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.313305958s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 1.445496871s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:44:55 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.445524894s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............XX.....X...............................X......] duration: 6.423967136s trace: https://jaeger.dev.storj.io/trace/139e8e33ab430945 FAIL (2025-11-11 16:44:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 38:40.294055811 (56h34m38.845526263s) [seg_pos={0 0}, upload=49859]: Done begin upload segment. Nov 11 16:38:40.294082692 (56h34m38.845553153s) [noise=true , address=145.79.227.14:4837 , piece_id=3C7EFHTCO474N3WGMIFNP24PZSVKVOFPEDNSXEPE4P26QWG3AB7A, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.294130494 (56h34m38.845600955s) [noise=true , address=45.91.116.14:4190 , piece_id=3FHYEMK6OKJACPWH7DL4T5HLWUXPRRTIAIGX4KRDVKHAYMDW326Q, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.294790049 (56h34m38.846260511s) [noise=true , address=145.79.227.13:4804 , piece_id=TTSF3ZHSWO6LYGIWBXRTPKG27KGW7XQBPL23Y5WPNENGO7MTV66Q, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.295175093 (56h34m38.846645544s) [noise=true , address=45.91.116.5:4297 , piece_id=EYKUYVGHZQ6OQTCB2GDZEOLYBULRSQN76HRLKXEULVNTARTYHS5A, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.295530890 (56h34m38.847001351s) [noise=true , address=45.91.116.4:4937 , piece_id=GRAKOOGQ4HXGVV32NO2YET2WMPATCMZRMYMUMV3NX2EBLXZJFC6A, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.295828958 (56h34m38.847299419s) [noise=true , address=145.79.227.25:4618 , piece_id=FFQAWUFMKPXTC3SFYX7EWLBAYZZ5SZIUEIFEVU7B2MXKX57QOR6Q, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.296176931 (56h34m38.847647382s) [noise=true , address=77.90.31.12:4589 , piece_id=ZE6O4CU7Y2TLCXAI6HBNITBTQFFBQC77E6YUWUXX3TWMBUUGOOKA, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.296649390 (56h34m38.848119841s) [noise=true , address=45.91.116.6:4975 , piece_id=23535HUWVMJEOXR5J5UYAA3LG2E4GOWF2X4FHPXYIH7GCOY3KPIA, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.298961653 (56h34m38.850432114s) [noise=true , address=145.79.227.30:4763 , piece_id=FOA6BMKCTINQEG7UZPOWTMVL7YOVMMKBUUMECQPZL67ZKSVYDYRA, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.300626273 (56h34m38.852096734s) [noise=true , address=77.90.31.17:4770 , piece_id=ATVM4RBHXP67K5RSXIPWDLPQSDHQHF3VLZSO7EZ2S7GASJVU4ITQ, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.302012112 (56h34m38.853482563s) [noise=true , address=145.79.227.17:4970 , piece_id=SXDKTR3E2YKUIWQVHPCJK753AAP7GD7IKVJRODNFO4KF2UXKBL4Q, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.303244468 (56h34m38.854714929s) [noise=true , address=145.79.227.22:4511 , piece_id=QET7Y6RMU3QZUJGQHPE5F567O6K6Q2MELKZK4H6BQTHPUW5EPNKA, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.304670594 (56h34m38.856141055s) [noise=true , address=77.90.31.15:4698 , piece_id=D4CYCESDDCKFSDZ354LB5Y3Z3H6CNQ5GWUXXYDZHXFXCVA46E25Q, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.305090844 (56h34m38.856561305s) [noise=true , address=45.91.116.8:4048 , piece_id=M7DH5VIABLYKUKZR57AIW7LJGISJINDHSQ3CWXISUDP3XPDIY5MA, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.305470417 (56h34m38.856940868s) [noise=true , address=45.91.116.13:4153 , piece_id=IUSH3457LMG3PR5IEMA4DMPYIJTLKQRXJ47VAEFSSVKKO3THKYOA, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.309258671 (56h34m38.860729132s) [noise=true , address=145.79.227.4:4186 , piece_id=DRV7Y4IEHMRXPZFRZU7DZEZ56NGROQI526OGKRXSMPFDGF3KDP5Q, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:42.050032931 (56h34m40.601503392s) [upload=49859]: Next returned nil segment Nov 11 16:38:42.050053050 (56h34m40.601523511s) [upload=49859]: Waiting for error group managing segments... Nov 11 16:38:42.379656265 (56h34m40.931126726s) [noise=true , address=45.91.116.4:4937 , piece_id=GRAKOOGQ4HXGVV32NO2YET2WMPATCMZRMYMUMV3NX2EBLXZJFC6A, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.394803549 (56h34m40.94627401s) [noise=true , address=45.91.116.14:4190 , piece_id=3FHYEMK6OKJACPWH7DL4T5HLWUXPRRTIAIGX4KRDVKHAYMDW326Q, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.400715316 (56h34m40.952185777s) [noise=true , address=77.90.31.17:4770 , piece_id=ATVM4RBHXP67K5RSXIPWDLPQSDHQHF3VLZSO7EZ2S7GASJVU4ITQ, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.409998765 (56h34m40.961469236s) [noise=true , address=77.90.31.15:4698 , piece_id=D4CYCESDDCKFSDZ354LB5Y3Z3H6CNQ5GWUXXYDZHXFXCVA46E25Q, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.418606595 (56h34m40.970077056s) [noise=true , address=77.90.31.12:4589 , piece_id=ZE6O4CU7Y2TLCXAI6HBNITBTQFFBQC77E6YUWUXX3TWMBUUGOOKA, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.443626699 (56h34m40.99509715s) [noise=true , address=45.91.116.6:4975 , piece_id=23535HUWVMJEOXR5J5UYAA3LG2E4GOWF2X4FHPXYIH7GCOY3KPIA, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.445695820 (56h34m40.997166271s) [noise=true , address=45.91.116.13:4153 , piece_id=IUSH3457LMG3PR5IEMA4DMPYIJTLKQRXJ47VAEFSSVKKO3THKYOA, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.473525152 (56h34m41.024995613s) [noise=true , address=45.91.116.5:4297 , piece_id=EYKUYVGHZQ6OQTCB2GDZEOLYBULRSQN76HRLKXEULVNTARTYHS5A, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.636670936 (56h34m41.188141397s) [noise=true , address=45.91.116.8:4048 , piece_id=M7DH5VIABLYKUKZR57AIW7LJGISJINDHSQ3CWXISUDP3XPDIY5MA, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:39:09.324245238 (56h35m7.875715699s) [noise=true , address=145.79.227.25:4618 , piece_id=FFQAWUFMKPXTC3SFYX7EWLBAYZZ5SZIUEIFEVU7B2MXKX57QOR6Q, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:57092->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:57092->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57092->145.79.227.25:4618: use of closed network connection Nov 11 16:39:09.324393590 (56h35m7.875864061s) [noise=true , address=145.79.227.13:4804 , piece_id=TTSF3ZHSWO6LYGIWBXRTPKG27KGW7XQBPL23Y5WPNENGO7MTV66Q, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:53130->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:53130->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53130->145.79.227.13:4804: use of closed network connection Nov 11 16:39:09.324491988 (56h35m7.875962449s) [noise=true , address=145.79.227.17:4970 , piece_id=SXDKTR3E2YKUIWQVHPCJK753AAP7GD7IKVJRODNFO4KF2UXKBL4Q, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:35738->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:35738->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35738->145.79.227.17:4970: use of closed network connection Nov 11 16:39:09.324535490 (56h35m7.876005951s) [noise=true , address=145.79.227.22:4511 , piece_id=QET7Y6RMU3QZUJGQHPE5F567O6K6Q2MELKZK4H6BQTHPUW5EPNKA, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:58766->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:58766->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58766->145.79.227.22:4511: use of closed network connection Nov 11 16:39:09.324644678 (56h35m7.876115139s) [noise=true , address=145.79.227.4:4186 , piece_id=DRV7Y4IEHMRXPZFRZU7DZEZ56NGROQI526OGKRXSMPFDGF3KDP5Q, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:57480->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:57480->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57480->145.79.227.4:4186: use of closed network connection Nov 11 16:39:09.324807358 (56h35m7.876277819s) [noise=true , address=145.79.227.14:4837 , piece_id=3C7EFHTCO474N3WGMIFNP24PZSVKVOFPEDNSXEPE4P26QWG3AB7A, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:60668->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 11 16:39:09.324949739 (56h35m7.8764202s) [noise=true , address=145.79.227.30:4763 , piece_id=FOA6BMKCTINQEG7UZPOWTMVL7YOVMMKBUUMECQPZL67ZKSVYDYRA, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:45392->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 11 16:39:09.324975198 (56h35m7.876445659s) [seg_pos={0 0}, upload=49859]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 11 16:39:09.578246041 (56h35m8.129716492s) [upload=49859]: Done uploading segments... Nov 11 16:39:09.578291948 (56h35m8.129762409s) [upload=49859]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.423997575s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 1.482382897s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:44:26 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.48241078s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X.......................................X......] duration: 6.243041822s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 16:44:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.243061118s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [........X......] duration: 56.574198ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/644568d221d3a147 FAIL (2025-11-11 16:44:10 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:45680->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 56.609346ms core website
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 1.441636267s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:43:56 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.44166994s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X.......................................X.....] duration: 6.735424882s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 16:43:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.735446975s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.........X.....] duration: 57.996458ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/644568d221d3a147 FAIL (2025-11-11 16:43:37 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:45680->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 58.024431ms core website
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 1.518344546s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:43:36 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.518375766s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............XX.....X...............................X.....] duration: 10.681480897s trace: https://jaeger.dev.storj.io/trace/139e8e33ab430945 FAIL (2025-11-11 16:43:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 38:40.294055811 (56h34m38.845526263s) [seg_pos={0 0}, upload=49859]: Done begin upload segment. Nov 11 16:38:40.294082692 (56h34m38.845553153s) [noise=true , address=145.79.227.14:4837 , piece_id=3C7EFHTCO474N3WGMIFNP24PZSVKVOFPEDNSXEPE4P26QWG3AB7A, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.294130494 (56h34m38.845600955s) [noise=true , address=45.91.116.14:4190 , piece_id=3FHYEMK6OKJACPWH7DL4T5HLWUXPRRTIAIGX4KRDVKHAYMDW326Q, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.294790049 (56h34m38.846260511s) [noise=true , address=145.79.227.13:4804 , piece_id=TTSF3ZHSWO6LYGIWBXRTPKG27KGW7XQBPL23Y5WPNENGO7MTV66Q, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.295175093 (56h34m38.846645544s) [noise=true , address=45.91.116.5:4297 , piece_id=EYKUYVGHZQ6OQTCB2GDZEOLYBULRSQN76HRLKXEULVNTARTYHS5A, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.295530890 (56h34m38.847001351s) [noise=true , address=45.91.116.4:4937 , piece_id=GRAKOOGQ4HXGVV32NO2YET2WMPATCMZRMYMUMV3NX2EBLXZJFC6A, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.295828958 (56h34m38.847299419s) [noise=true , address=145.79.227.25:4618 , piece_id=FFQAWUFMKPXTC3SFYX7EWLBAYZZ5SZIUEIFEVU7B2MXKX57QOR6Q, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.296176931 (56h34m38.847647382s) [noise=true , address=77.90.31.12:4589 , piece_id=ZE6O4CU7Y2TLCXAI6HBNITBTQFFBQC77E6YUWUXX3TWMBUUGOOKA, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.296649390 (56h34m38.848119841s) [noise=true , address=45.91.116.6:4975 , piece_id=23535HUWVMJEOXR5J5UYAA3LG2E4GOWF2X4FHPXYIH7GCOY3KPIA, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.298961653 (56h34m38.850432114s) [noise=true , address=145.79.227.30:4763 , piece_id=FOA6BMKCTINQEG7UZPOWTMVL7YOVMMKBUUMECQPZL67ZKSVYDYRA, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.300626273 (56h34m38.852096734s) [noise=true , address=77.90.31.17:4770 , piece_id=ATVM4RBHXP67K5RSXIPWDLPQSDHQHF3VLZSO7EZ2S7GASJVU4ITQ, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.302012112 (56h34m38.853482563s) [noise=true , address=145.79.227.17:4970 , piece_id=SXDKTR3E2YKUIWQVHPCJK753AAP7GD7IKVJRODNFO4KF2UXKBL4Q, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.303244468 (56h34m38.854714929s) [noise=true , address=145.79.227.22:4511 , piece_id=QET7Y6RMU3QZUJGQHPE5F567O6K6Q2MELKZK4H6BQTHPUW5EPNKA, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.304670594 (56h34m38.856141055s) [noise=true , address=77.90.31.15:4698 , piece_id=D4CYCESDDCKFSDZ354LB5Y3Z3H6CNQ5GWUXXYDZHXFXCVA46E25Q, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.305090844 (56h34m38.856561305s) [noise=true , address=45.91.116.8:4048 , piece_id=M7DH5VIABLYKUKZR57AIW7LJGISJINDHSQ3CWXISUDP3XPDIY5MA, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.305470417 (56h34m38.856940868s) [noise=true , address=45.91.116.13:4153 , piece_id=IUSH3457LMG3PR5IEMA4DMPYIJTLKQRXJ47VAEFSSVKKO3THKYOA, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.309258671 (56h34m38.860729132s) [noise=true , address=145.79.227.4:4186 , piece_id=DRV7Y4IEHMRXPZFRZU7DZEZ56NGROQI526OGKRXSMPFDGF3KDP5Q, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:42.050032931 (56h34m40.601503392s) [upload=49859]: Next returned nil segment Nov 11 16:38:42.050053050 (56h34m40.601523511s) [upload=49859]: Waiting for error group managing segments... Nov 11 16:38:42.379656265 (56h34m40.931126726s) [noise=true , address=45.91.116.4:4937 , piece_id=GRAKOOGQ4HXGVV32NO2YET2WMPATCMZRMYMUMV3NX2EBLXZJFC6A, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.394803549 (56h34m40.94627401s) [noise=true , address=45.91.116.14:4190 , piece_id=3FHYEMK6OKJACPWH7DL4T5HLWUXPRRTIAIGX4KRDVKHAYMDW326Q, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.400715316 (56h34m40.952185777s) [noise=true , address=77.90.31.17:4770 , piece_id=ATVM4RBHXP67K5RSXIPWDLPQSDHQHF3VLZSO7EZ2S7GASJVU4ITQ, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.409998765 (56h34m40.961469236s) [noise=true , address=77.90.31.15:4698 , piece_id=D4CYCESDDCKFSDZ354LB5Y3Z3H6CNQ5GWUXXYDZHXFXCVA46E25Q, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.418606595 (56h34m40.970077056s) [noise=true , address=77.90.31.12:4589 , piece_id=ZE6O4CU7Y2TLCXAI6HBNITBTQFFBQC77E6YUWUXX3TWMBUUGOOKA, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.443626699 (56h34m40.99509715s) [noise=true , address=45.91.116.6:4975 , piece_id=23535HUWVMJEOXR5J5UYAA3LG2E4GOWF2X4FHPXYIH7GCOY3KPIA, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.445695820 (56h34m40.997166271s) [noise=true , address=45.91.116.13:4153 , piece_id=IUSH3457LMG3PR5IEMA4DMPYIJTLKQRXJ47VAEFSSVKKO3THKYOA, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.473525152 (56h34m41.024995613s) [noise=true , address=45.91.116.5:4297 , piece_id=EYKUYVGHZQ6OQTCB2GDZEOLYBULRSQN76HRLKXEULVNTARTYHS5A, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.636670936 (56h34m41.188141397s) [noise=true , address=45.91.116.8:4048 , piece_id=M7DH5VIABLYKUKZR57AIW7LJGISJINDHSQ3CWXISUDP3XPDIY5MA, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:39:09.324245238 (56h35m7.875715699s) [noise=true , address=145.79.227.25:4618 , piece_id=FFQAWUFMKPXTC3SFYX7EWLBAYZZ5SZIUEIFEVU7B2MXKX57QOR6Q, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:57092->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:57092->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57092->145.79.227.25:4618: use of closed network connection Nov 11 16:39:09.324393590 (56h35m7.875864061s) [noise=true , address=145.79.227.13:4804 , piece_id=TTSF3ZHSWO6LYGIWBXRTPKG27KGW7XQBPL23Y5WPNENGO7MTV66Q, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:53130->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:53130->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53130->145.79.227.13:4804: use of closed network connection Nov 11 16:39:09.324491988 (56h35m7.875962449s) [noise=true , address=145.79.227.17:4970 , piece_id=SXDKTR3E2YKUIWQVHPCJK753AAP7GD7IKVJRODNFO4KF2UXKBL4Q, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:35738->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:35738->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35738->145.79.227.17:4970: use of closed network connection Nov 11 16:39:09.324535490 (56h35m7.876005951s) [noise=true , address=145.79.227.22:4511 , piece_id=QET7Y6RMU3QZUJGQHPE5F567O6K6Q2MELKZK4H6BQTHPUW5EPNKA, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:58766->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:58766->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58766->145.79.227.22:4511: use of closed network connection Nov 11 16:39:09.324644678 (56h35m7.876115139s) [noise=true , address=145.79.227.4:4186 , piece_id=DRV7Y4IEHMRXPZFRZU7DZEZ56NGROQI526OGKRXSMPFDGF3KDP5Q, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:57480->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:57480->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57480->145.79.227.4:4186: use of closed network connection Nov 11 16:39:09.324807358 (56h35m7.876277819s) [noise=true , address=145.79.227.14:4837 , piece_id=3C7EFHTCO474N3WGMIFNP24PZSVKVOFPEDNSXEPE4P26QWG3AB7A, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:60668->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 11 16:39:09.324949739 (56h35m7.8764202s) [noise=true , address=145.79.227.30:4763 , piece_id=FOA6BMKCTINQEG7UZPOWTMVL7YOVMMKBUUMECQPZL67ZKSVYDYRA, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:45392->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 11 16:39:09.324975198 (56h35m7.876445659s) [seg_pos={0 0}, upload=49859]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 11 16:39:09.578246041 (56h35m8.129716492s) [upload=49859]: Done uploading segments... Nov 11 16:39:09.578291948 (56h35m8.129762409s) [upload=49859]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 10.681510454s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................XX.....X...............................X....] duration: 8.316762322s trace: https://jaeger.dev.storj.io/trace/139e8e33ab430945 FAIL (2025-11-11 16:43:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 38:40.294055811 (56h34m38.845526263s) [seg_pos={0 0}, upload=49859]: Done begin upload segment. Nov 11 16:38:40.294082692 (56h34m38.845553153s) [noise=true , address=145.79.227.14:4837 , piece_id=3C7EFHTCO474N3WGMIFNP24PZSVKVOFPEDNSXEPE4P26QWG3AB7A, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.294130494 (56h34m38.845600955s) [noise=true , address=45.91.116.14:4190 , piece_id=3FHYEMK6OKJACPWH7DL4T5HLWUXPRRTIAIGX4KRDVKHAYMDW326Q, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.294790049 (56h34m38.846260511s) [noise=true , address=145.79.227.13:4804 , piece_id=TTSF3ZHSWO6LYGIWBXRTPKG27KGW7XQBPL23Y5WPNENGO7MTV66Q, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.295175093 (56h34m38.846645544s) [noise=true , address=45.91.116.5:4297 , piece_id=EYKUYVGHZQ6OQTCB2GDZEOLYBULRSQN76HRLKXEULVNTARTYHS5A, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.295530890 (56h34m38.847001351s) [noise=true , address=45.91.116.4:4937 , piece_id=GRAKOOGQ4HXGVV32NO2YET2WMPATCMZRMYMUMV3NX2EBLXZJFC6A, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.295828958 (56h34m38.847299419s) [noise=true , address=145.79.227.25:4618 , piece_id=FFQAWUFMKPXTC3SFYX7EWLBAYZZ5SZIUEIFEVU7B2MXKX57QOR6Q, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.296176931 (56h34m38.847647382s) [noise=true , address=77.90.31.12:4589 , piece_id=ZE6O4CU7Y2TLCXAI6HBNITBTQFFBQC77E6YUWUXX3TWMBUUGOOKA, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.296649390 (56h34m38.848119841s) [noise=true , address=45.91.116.6:4975 , piece_id=23535HUWVMJEOXR5J5UYAA3LG2E4GOWF2X4FHPXYIH7GCOY3KPIA, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.298961653 (56h34m38.850432114s) [noise=true , address=145.79.227.30:4763 , piece_id=FOA6BMKCTINQEG7UZPOWTMVL7YOVMMKBUUMECQPZL67ZKSVYDYRA, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.300626273 (56h34m38.852096734s) [noise=true , address=77.90.31.17:4770 , piece_id=ATVM4RBHXP67K5RSXIPWDLPQSDHQHF3VLZSO7EZ2S7GASJVU4ITQ, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.302012112 (56h34m38.853482563s) [noise=true , address=145.79.227.17:4970 , piece_id=SXDKTR3E2YKUIWQVHPCJK753AAP7GD7IKVJRODNFO4KF2UXKBL4Q, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.303244468 (56h34m38.854714929s) [noise=true , address=145.79.227.22:4511 , piece_id=QET7Y6RMU3QZUJGQHPE5F567O6K6Q2MELKZK4H6BQTHPUW5EPNKA, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.304670594 (56h34m38.856141055s) [noise=true , address=77.90.31.15:4698 , piece_id=D4CYCESDDCKFSDZ354LB5Y3Z3H6CNQ5GWUXXYDZHXFXCVA46E25Q, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.305090844 (56h34m38.856561305s) [noise=true , address=45.91.116.8:4048 , piece_id=M7DH5VIABLYKUKZR57AIW7LJGISJINDHSQ3CWXISUDP3XPDIY5MA, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.305470417 (56h34m38.856940868s) [noise=true , address=45.91.116.13:4153 , piece_id=IUSH3457LMG3PR5IEMA4DMPYIJTLKQRXJ47VAEFSSVKKO3THKYOA, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.309258671 (56h34m38.860729132s) [noise=true , address=145.79.227.4:4186 , piece_id=DRV7Y4IEHMRXPZFRZU7DZEZ56NGROQI526OGKRXSMPFDGF3KDP5Q, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:42.050032931 (56h34m40.601503392s) [upload=49859]: Next returned nil segment Nov 11 16:38:42.050053050 (56h34m40.601523511s) [upload=49859]: Waiting for error group managing segments... Nov 11 16:38:42.379656265 (56h34m40.931126726s) [noise=true , address=45.91.116.4:4937 , piece_id=GRAKOOGQ4HXGVV32NO2YET2WMPATCMZRMYMUMV3NX2EBLXZJFC6A, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.394803549 (56h34m40.94627401s) [noise=true , address=45.91.116.14:4190 , piece_id=3FHYEMK6OKJACPWH7DL4T5HLWUXPRRTIAIGX4KRDVKHAYMDW326Q, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.400715316 (56h34m40.952185777s) [noise=true , address=77.90.31.17:4770 , piece_id=ATVM4RBHXP67K5RSXIPWDLPQSDHQHF3VLZSO7EZ2S7GASJVU4ITQ, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.409998765 (56h34m40.961469236s) [noise=true , address=77.90.31.15:4698 , piece_id=D4CYCESDDCKFSDZ354LB5Y3Z3H6CNQ5GWUXXYDZHXFXCVA46E25Q, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.418606595 (56h34m40.970077056s) [noise=true , address=77.90.31.12:4589 , piece_id=ZE6O4CU7Y2TLCXAI6HBNITBTQFFBQC77E6YUWUXX3TWMBUUGOOKA, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.443626699 (56h34m40.99509715s) [noise=true , address=45.91.116.6:4975 , piece_id=23535HUWVMJEOXR5J5UYAA3LG2E4GOWF2X4FHPXYIH7GCOY3KPIA, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.445695820 (56h34m40.997166271s) [noise=true , address=45.91.116.13:4153 , piece_id=IUSH3457LMG3PR5IEMA4DMPYIJTLKQRXJ47VAEFSSVKKO3THKYOA, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.473525152 (56h34m41.024995613s) [noise=true , address=45.91.116.5:4297 , piece_id=EYKUYVGHZQ6OQTCB2GDZEOLYBULRSQN76HRLKXEULVNTARTYHS5A, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.636670936 (56h34m41.188141397s) [noise=true , address=45.91.116.8:4048 , piece_id=M7DH5VIABLYKUKZR57AIW7LJGISJINDHSQ3CWXISUDP3XPDIY5MA, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:39:09.324245238 (56h35m7.875715699s) [noise=true , address=145.79.227.25:4618 , piece_id=FFQAWUFMKPXTC3SFYX7EWLBAYZZ5SZIUEIFEVU7B2MXKX57QOR6Q, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:57092->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:57092->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57092->145.79.227.25:4618: use of closed network connection Nov 11 16:39:09.324393590 (56h35m7.875864061s) [noise=true , address=145.79.227.13:4804 , piece_id=TTSF3ZHSWO6LYGIWBXRTPKG27KGW7XQBPL23Y5WPNENGO7MTV66Q, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:53130->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:53130->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53130->145.79.227.13:4804: use of closed network connection Nov 11 16:39:09.324491988 (56h35m7.875962449s) [noise=true , address=145.79.227.17:4970 , piece_id=SXDKTR3E2YKUIWQVHPCJK753AAP7GD7IKVJRODNFO4KF2UXKBL4Q, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:35738->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:35738->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35738->145.79.227.17:4970: use of closed network connection Nov 11 16:39:09.324535490 (56h35m7.876005951s) [noise=true , address=145.79.227.22:4511 , piece_id=QET7Y6RMU3QZUJGQHPE5F567O6K6Q2MELKZK4H6BQTHPUW5EPNKA, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:58766->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:58766->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58766->145.79.227.22:4511: use of closed network connection Nov 11 16:39:09.324644678 (56h35m7.876115139s) [noise=true , address=145.79.227.4:4186 , piece_id=DRV7Y4IEHMRXPZFRZU7DZEZ56NGROQI526OGKRXSMPFDGF3KDP5Q, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:57480->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:57480->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57480->145.79.227.4:4186: use of closed network connection Nov 11 16:39:09.324807358 (56h35m7.876277819s) [noise=true , address=145.79.227.14:4837 , piece_id=3C7EFHTCO474N3WGMIFNP24PZSVKVOFPEDNSXEPE4P26QWG3AB7A, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:60668->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 11 16:39:09.324949739 (56h35m7.8764202s) [noise=true , address=145.79.227.30:4763 , piece_id=FOA6BMKCTINQEG7UZPOWTMVL7YOVMMKBUUMECQPZL67ZKSVYDYRA, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:45392->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 11 16:39:09.324975198 (56h35m7.876445659s) [seg_pos={0 0}, upload=49859]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 11 16:39:09.578246041 (56h35m8.129716492s) [upload=49859]: Done uploading segments... Nov 11 16:39:09.578291948 (56h35m8.129762409s) [upload=49859]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 8.316795636s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..........X....] duration: 78.01605ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/644568d221d3a147 FAIL (2025-11-11 16:42:58 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:45680->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 78.047409ms core website
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 1.4181431s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:42:55 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.418173959s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X.......................................X....] duration: 6.744108363s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 16:42:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.744140453s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................XX.....X...............................X...] duration: 7.668422225s trace: https://jaeger.dev.storj.io/trace/139e8e33ab430945 FAIL (2025-11-11 16:42:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 38:40.294055811 (56h34m38.845526263s) [seg_pos={0 0}, upload=49859]: Done begin upload segment. Nov 11 16:38:40.294082692 (56h34m38.845553153s) [noise=true , address=145.79.227.14:4837 , piece_id=3C7EFHTCO474N3WGMIFNP24PZSVKVOFPEDNSXEPE4P26QWG3AB7A, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.294130494 (56h34m38.845600955s) [noise=true , address=45.91.116.14:4190 , piece_id=3FHYEMK6OKJACPWH7DL4T5HLWUXPRRTIAIGX4KRDVKHAYMDW326Q, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.294790049 (56h34m38.846260511s) [noise=true , address=145.79.227.13:4804 , piece_id=TTSF3ZHSWO6LYGIWBXRTPKG27KGW7XQBPL23Y5WPNENGO7MTV66Q, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.295175093 (56h34m38.846645544s) [noise=true , address=45.91.116.5:4297 , piece_id=EYKUYVGHZQ6OQTCB2GDZEOLYBULRSQN76HRLKXEULVNTARTYHS5A, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.295530890 (56h34m38.847001351s) [noise=true , address=45.91.116.4:4937 , piece_id=GRAKOOGQ4HXGVV32NO2YET2WMPATCMZRMYMUMV3NX2EBLXZJFC6A, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.295828958 (56h34m38.847299419s) [noise=true , address=145.79.227.25:4618 , piece_id=FFQAWUFMKPXTC3SFYX7EWLBAYZZ5SZIUEIFEVU7B2MXKX57QOR6Q, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.296176931 (56h34m38.847647382s) [noise=true , address=77.90.31.12:4589 , piece_id=ZE6O4CU7Y2TLCXAI6HBNITBTQFFBQC77E6YUWUXX3TWMBUUGOOKA, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.296649390 (56h34m38.848119841s) [noise=true , address=45.91.116.6:4975 , piece_id=23535HUWVMJEOXR5J5UYAA3LG2E4GOWF2X4FHPXYIH7GCOY3KPIA, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.298961653 (56h34m38.850432114s) [noise=true , address=145.79.227.30:4763 , piece_id=FOA6BMKCTINQEG7UZPOWTMVL7YOVMMKBUUMECQPZL67ZKSVYDYRA, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.300626273 (56h34m38.852096734s) [noise=true , address=77.90.31.17:4770 , piece_id=ATVM4RBHXP67K5RSXIPWDLPQSDHQHF3VLZSO7EZ2S7GASJVU4ITQ, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.302012112 (56h34m38.853482563s) [noise=true , address=145.79.227.17:4970 , piece_id=SXDKTR3E2YKUIWQVHPCJK753AAP7GD7IKVJRODNFO4KF2UXKBL4Q, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.303244468 (56h34m38.854714929s) [noise=true , address=145.79.227.22:4511 , piece_id=QET7Y6RMU3QZUJGQHPE5F567O6K6Q2MELKZK4H6BQTHPUW5EPNKA, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.304670594 (56h34m38.856141055s) [noise=true , address=77.90.31.15:4698 , piece_id=D4CYCESDDCKFSDZ354LB5Y3Z3H6CNQ5GWUXXYDZHXFXCVA46E25Q, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.305090844 (56h34m38.856561305s) [noise=true , address=45.91.116.8:4048 , piece_id=M7DH5VIABLYKUKZR57AIW7LJGISJINDHSQ3CWXISUDP3XPDIY5MA, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.305470417 (56h34m38.856940868s) [noise=true , address=45.91.116.13:4153 , piece_id=IUSH3457LMG3PR5IEMA4DMPYIJTLKQRXJ47VAEFSSVKKO3THKYOA, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.309258671 (56h34m38.860729132s) [noise=true , address=145.79.227.4:4186 , piece_id=DRV7Y4IEHMRXPZFRZU7DZEZ56NGROQI526OGKRXSMPFDGF3KDP5Q, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:42.050032931 (56h34m40.601503392s) [upload=49859]: Next returned nil segment Nov 11 16:38:42.050053050 (56h34m40.601523511s) [upload=49859]: Waiting for error group managing segments... Nov 11 16:38:42.379656265 (56h34m40.931126726s) [noise=true , address=45.91.116.4:4937 , piece_id=GRAKOOGQ4HXGVV32NO2YET2WMPATCMZRMYMUMV3NX2EBLXZJFC6A, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.394803549 (56h34m40.94627401s) [noise=true , address=45.91.116.14:4190 , piece_id=3FHYEMK6OKJACPWH7DL4T5HLWUXPRRTIAIGX4KRDVKHAYMDW326Q, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.400715316 (56h34m40.952185777s) [noise=true , address=77.90.31.17:4770 , piece_id=ATVM4RBHXP67K5RSXIPWDLPQSDHQHF3VLZSO7EZ2S7GASJVU4ITQ, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.409998765 (56h34m40.961469236s) [noise=true , address=77.90.31.15:4698 , piece_id=D4CYCESDDCKFSDZ354LB5Y3Z3H6CNQ5GWUXXYDZHXFXCVA46E25Q, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.418606595 (56h34m40.970077056s) [noise=true , address=77.90.31.12:4589 , piece_id=ZE6O4CU7Y2TLCXAI6HBNITBTQFFBQC77E6YUWUXX3TWMBUUGOOKA, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.443626699 (56h34m40.99509715s) [noise=true , address=45.91.116.6:4975 , piece_id=23535HUWVMJEOXR5J5UYAA3LG2E4GOWF2X4FHPXYIH7GCOY3KPIA, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.445695820 (56h34m40.997166271s) [noise=true , address=45.91.116.13:4153 , piece_id=IUSH3457LMG3PR5IEMA4DMPYIJTLKQRXJ47VAEFSSVKKO3THKYOA, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.473525152 (56h34m41.024995613s) [noise=true , address=45.91.116.5:4297 , piece_id=EYKUYVGHZQ6OQTCB2GDZEOLYBULRSQN76HRLKXEULVNTARTYHS5A, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.636670936 (56h34m41.188141397s) [noise=true , address=45.91.116.8:4048 , piece_id=M7DH5VIABLYKUKZR57AIW7LJGISJINDHSQ3CWXISUDP3XPDIY5MA, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:39:09.324245238 (56h35m7.875715699s) [noise=true , address=145.79.227.25:4618 , piece_id=FFQAWUFMKPXTC3SFYX7EWLBAYZZ5SZIUEIFEVU7B2MXKX57QOR6Q, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:57092->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:57092->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57092->145.79.227.25:4618: use of closed network connection Nov 11 16:39:09.324393590 (56h35m7.875864061s) [noise=true , address=145.79.227.13:4804 , piece_id=TTSF3ZHSWO6LYGIWBXRTPKG27KGW7XQBPL23Y5WPNENGO7MTV66Q, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:53130->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:53130->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53130->145.79.227.13:4804: use of closed network connection Nov 11 16:39:09.324491988 (56h35m7.875962449s) [noise=true , address=145.79.227.17:4970 , piece_id=SXDKTR3E2YKUIWQVHPCJK753AAP7GD7IKVJRODNFO4KF2UXKBL4Q, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:35738->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:35738->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35738->145.79.227.17:4970: use of closed network connection Nov 11 16:39:09.324535490 (56h35m7.876005951s) [noise=true , address=145.79.227.22:4511 , piece_id=QET7Y6RMU3QZUJGQHPE5F567O6K6Q2MELKZK4H6BQTHPUW5EPNKA, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:58766->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:58766->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58766->145.79.227.22:4511: use of closed network connection Nov 11 16:39:09.324644678 (56h35m7.876115139s) [noise=true , address=145.79.227.4:4186 , piece_id=DRV7Y4IEHMRXPZFRZU7DZEZ56NGROQI526OGKRXSMPFDGF3KDP5Q, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:57480->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:57480->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57480->145.79.227.4:4186: use of closed network connection Nov 11 16:39:09.324807358 (56h35m7.876277819s) [noise=true , address=145.79.227.14:4837 , piece_id=3C7EFHTCO474N3WGMIFNP24PZSVKVOFPEDNSXEPE4P26QWG3AB7A, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:60668->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 11 16:39:09.324949739 (56h35m7.8764202s) [noise=true , address=145.79.227.30:4763 , piece_id=FOA6BMKCTINQEG7UZPOWTMVL7YOVMMKBUUMECQPZL67ZKSVYDYRA, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:45392->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 11 16:39:09.324975198 (56h35m7.876445659s) [seg_pos={0 0}, upload=49859]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 11 16:39:09.578246041 (56h35m8.129716492s) [upload=49859]: Done uploading segments... Nov 11 16:39:09.578291948 (56h35m8.129762409s) [upload=49859]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 7.668452755s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 1.445231422s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:41:53 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.4452612s core uplink eu1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [...........X...] duration: 50.51397ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/644568d221d3a147 FAIL (2025-11-11 16:41:52 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:45680->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 50.544627ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X.......................................X...] duration: 7.299985759s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 16:41:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 7.300011308s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [............X..] duration: 59.542381ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/644568d221d3a147 FAIL (2025-11-11 16:41:27 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:45680->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 59.568822ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X.......................................X..] duration: 6.64688727s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 16:41:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.64691878s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.............X.] duration: 82.549201ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/644568d221d3a147 FAIL (2025-11-11 16:41:09 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:45680->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 82.579649ms core website
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 1.432894651s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:41:06 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.433043375s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................XX.....X...............................X..] duration: 7.208363689s trace: https://jaeger.dev.storj.io/trace/139e8e33ab430945 FAIL (2025-11-11 16:41:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 38:40.294055811 (56h34m38.845526263s) [seg_pos={0 0}, upload=49859]: Done begin upload segment. Nov 11 16:38:40.294082692 (56h34m38.845553153s) [noise=true , address=145.79.227.14:4837 , piece_id=3C7EFHTCO474N3WGMIFNP24PZSVKVOFPEDNSXEPE4P26QWG3AB7A, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.294130494 (56h34m38.845600955s) [noise=true , address=45.91.116.14:4190 , piece_id=3FHYEMK6OKJACPWH7DL4T5HLWUXPRRTIAIGX4KRDVKHAYMDW326Q, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.294790049 (56h34m38.846260511s) [noise=true , address=145.79.227.13:4804 , piece_id=TTSF3ZHSWO6LYGIWBXRTPKG27KGW7XQBPL23Y5WPNENGO7MTV66Q, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.295175093 (56h34m38.846645544s) [noise=true , address=45.91.116.5:4297 , piece_id=EYKUYVGHZQ6OQTCB2GDZEOLYBULRSQN76HRLKXEULVNTARTYHS5A, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.295530890 (56h34m38.847001351s) [noise=true , address=45.91.116.4:4937 , piece_id=GRAKOOGQ4HXGVV32NO2YET2WMPATCMZRMYMUMV3NX2EBLXZJFC6A, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.295828958 (56h34m38.847299419s) [noise=true , address=145.79.227.25:4618 , piece_id=FFQAWUFMKPXTC3SFYX7EWLBAYZZ5SZIUEIFEVU7B2MXKX57QOR6Q, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.296176931 (56h34m38.847647382s) [noise=true , address=77.90.31.12:4589 , piece_id=ZE6O4CU7Y2TLCXAI6HBNITBTQFFBQC77E6YUWUXX3TWMBUUGOOKA, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.296649390 (56h34m38.848119841s) [noise=true , address=45.91.116.6:4975 , piece_id=23535HUWVMJEOXR5J5UYAA3LG2E4GOWF2X4FHPXYIH7GCOY3KPIA, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.298961653 (56h34m38.850432114s) [noise=true , address=145.79.227.30:4763 , piece_id=FOA6BMKCTINQEG7UZPOWTMVL7YOVMMKBUUMECQPZL67ZKSVYDYRA, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.300626273 (56h34m38.852096734s) [noise=true , address=77.90.31.17:4770 , piece_id=ATVM4RBHXP67K5RSXIPWDLPQSDHQHF3VLZSO7EZ2S7GASJVU4ITQ, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.302012112 (56h34m38.853482563s) [noise=true , address=145.79.227.17:4970 , piece_id=SXDKTR3E2YKUIWQVHPCJK753AAP7GD7IKVJRODNFO4KF2UXKBL4Q, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.303244468 (56h34m38.854714929s) [noise=true , address=145.79.227.22:4511 , piece_id=QET7Y6RMU3QZUJGQHPE5F567O6K6Q2MELKZK4H6BQTHPUW5EPNKA, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.304670594 (56h34m38.856141055s) [noise=true , address=77.90.31.15:4698 , piece_id=D4CYCESDDCKFSDZ354LB5Y3Z3H6CNQ5GWUXXYDZHXFXCVA46E25Q, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.305090844 (56h34m38.856561305s) [noise=true , address=45.91.116.8:4048 , piece_id=M7DH5VIABLYKUKZR57AIW7LJGISJINDHSQ3CWXISUDP3XPDIY5MA, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.305470417 (56h34m38.856940868s) [noise=true , address=45.91.116.13:4153 , piece_id=IUSH3457LMG3PR5IEMA4DMPYIJTLKQRXJ47VAEFSSVKKO3THKYOA, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.309258671 (56h34m38.860729132s) [noise=true , address=145.79.227.4:4186 , piece_id=DRV7Y4IEHMRXPZFRZU7DZEZ56NGROQI526OGKRXSMPFDGF3KDP5Q, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:42.050032931 (56h34m40.601503392s) [upload=49859]: Next returned nil segment Nov 11 16:38:42.050053050 (56h34m40.601523511s) [upload=49859]: Waiting for error group managing segments... Nov 11 16:38:42.379656265 (56h34m40.931126726s) [noise=true , address=45.91.116.4:4937 , piece_id=GRAKOOGQ4HXGVV32NO2YET2WMPATCMZRMYMUMV3NX2EBLXZJFC6A, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.394803549 (56h34m40.94627401s) [noise=true , address=45.91.116.14:4190 , piece_id=3FHYEMK6OKJACPWH7DL4T5HLWUXPRRTIAIGX4KRDVKHAYMDW326Q, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.400715316 (56h34m40.952185777s) [noise=true , address=77.90.31.17:4770 , piece_id=ATVM4RBHXP67K5RSXIPWDLPQSDHQHF3VLZSO7EZ2S7GASJVU4ITQ, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.409998765 (56h34m40.961469236s) [noise=true , address=77.90.31.15:4698 , piece_id=D4CYCESDDCKFSDZ354LB5Y3Z3H6CNQ5GWUXXYDZHXFXCVA46E25Q, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.418606595 (56h34m40.970077056s) [noise=true , address=77.90.31.12:4589 , piece_id=ZE6O4CU7Y2TLCXAI6HBNITBTQFFBQC77E6YUWUXX3TWMBUUGOOKA, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.443626699 (56h34m40.99509715s) [noise=true , address=45.91.116.6:4975 , piece_id=23535HUWVMJEOXR5J5UYAA3LG2E4GOWF2X4FHPXYIH7GCOY3KPIA, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.445695820 (56h34m40.997166271s) [noise=true , address=45.91.116.13:4153 , piece_id=IUSH3457LMG3PR5IEMA4DMPYIJTLKQRXJ47VAEFSSVKKO3THKYOA, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.473525152 (56h34m41.024995613s) [noise=true , address=45.91.116.5:4297 , piece_id=EYKUYVGHZQ6OQTCB2GDZEOLYBULRSQN76HRLKXEULVNTARTYHS5A, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.636670936 (56h34m41.188141397s) [noise=true , address=45.91.116.8:4048 , piece_id=M7DH5VIABLYKUKZR57AIW7LJGISJINDHSQ3CWXISUDP3XPDIY5MA, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:39:09.324245238 (56h35m7.875715699s) [noise=true , address=145.79.227.25:4618 , piece_id=FFQAWUFMKPXTC3SFYX7EWLBAYZZ5SZIUEIFEVU7B2MXKX57QOR6Q, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:57092->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:57092->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57092->145.79.227.25:4618: use of closed network connection Nov 11 16:39:09.324393590 (56h35m7.875864061s) [noise=true , address=145.79.227.13:4804 , piece_id=TTSF3ZHSWO6LYGIWBXRTPKG27KGW7XQBPL23Y5WPNENGO7MTV66Q, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:53130->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:53130->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53130->145.79.227.13:4804: use of closed network connection Nov 11 16:39:09.324491988 (56h35m7.875962449s) [noise=true , address=145.79.227.17:4970 , piece_id=SXDKTR3E2YKUIWQVHPCJK753AAP7GD7IKVJRODNFO4KF2UXKBL4Q, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:35738->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:35738->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35738->145.79.227.17:4970: use of closed network connection Nov 11 16:39:09.324535490 (56h35m7.876005951s) [noise=true , address=145.79.227.22:4511 , piece_id=QET7Y6RMU3QZUJGQHPE5F567O6K6Q2MELKZK4H6BQTHPUW5EPNKA, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:58766->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:58766->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58766->145.79.227.22:4511: use of closed network connection Nov 11 16:39:09.324644678 (56h35m7.876115139s) [noise=true , address=145.79.227.4:4186 , piece_id=DRV7Y4IEHMRXPZFRZU7DZEZ56NGROQI526OGKRXSMPFDGF3KDP5Q, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:57480->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:57480->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57480->145.79.227.4:4186: use of closed network connection Nov 11 16:39:09.324807358 (56h35m7.876277819s) [noise=true , address=145.79.227.14:4837 , piece_id=3C7EFHTCO474N3WGMIFNP24PZSVKVOFPEDNSXEPE4P26QWG3AB7A, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:60668->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 11 16:39:09.324949739 (56h35m7.8764202s) [noise=true , address=145.79.227.30:4763 , piece_id=FOA6BMKCTINQEG7UZPOWTMVL7YOVMMKBUUMECQPZL67ZKSVYDYRA, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:45392->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 11 16:39:09.324975198 (56h35m7.876445659s) [seg_pos={0 0}, upload=49859]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 11 16:39:09.578246041 (56h35m8.129716492s) [upload=49859]: Done uploading segments... Nov 11 16:39:09.578291948 (56h35m8.129762409s) [upload=49859]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 7.208396902s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 1.48423352s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:40:22 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.484260713s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X.......................................X.] duration: 6.294810688s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 16:39:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.294826318s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 2 out of 15 is failed (threshold=6) [X.............X] duration: 226.068652ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/644568d221d3a147 FAIL (2025-11-11 16:39:57 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:45680->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 226.095103ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................XX.....X...............................X.] duration: 9.870704735s trace: https://jaeger.dev.storj.io/trace/139e8e33ab430945 FAIL (2025-11-11 16:39:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 38:40.294055811 (56h34m38.845526263s) [seg_pos={0 0}, upload=49859]: Done begin upload segment. Nov 11 16:38:40.294082692 (56h34m38.845553153s) [noise=true , address=145.79.227.14:4837 , piece_id=3C7EFHTCO474N3WGMIFNP24PZSVKVOFPEDNSXEPE4P26QWG3AB7A, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.294130494 (56h34m38.845600955s) [noise=true , address=45.91.116.14:4190 , piece_id=3FHYEMK6OKJACPWH7DL4T5HLWUXPRRTIAIGX4KRDVKHAYMDW326Q, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.294790049 (56h34m38.846260511s) [noise=true , address=145.79.227.13:4804 , piece_id=TTSF3ZHSWO6LYGIWBXRTPKG27KGW7XQBPL23Y5WPNENGO7MTV66Q, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.295175093 (56h34m38.846645544s) [noise=true , address=45.91.116.5:4297 , piece_id=EYKUYVGHZQ6OQTCB2GDZEOLYBULRSQN76HRLKXEULVNTARTYHS5A, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.295530890 (56h34m38.847001351s) [noise=true , address=45.91.116.4:4937 , piece_id=GRAKOOGQ4HXGVV32NO2YET2WMPATCMZRMYMUMV3NX2EBLXZJFC6A, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.295828958 (56h34m38.847299419s) [noise=true , address=145.79.227.25:4618 , piece_id=FFQAWUFMKPXTC3SFYX7EWLBAYZZ5SZIUEIFEVU7B2MXKX57QOR6Q, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.296176931 (56h34m38.847647382s) [noise=true , address=77.90.31.12:4589 , piece_id=ZE6O4CU7Y2TLCXAI6HBNITBTQFFBQC77E6YUWUXX3TWMBUUGOOKA, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.296649390 (56h34m38.848119841s) [noise=true , address=45.91.116.6:4975 , piece_id=23535HUWVMJEOXR5J5UYAA3LG2E4GOWF2X4FHPXYIH7GCOY3KPIA, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.298961653 (56h34m38.850432114s) [noise=true , address=145.79.227.30:4763 , piece_id=FOA6BMKCTINQEG7UZPOWTMVL7YOVMMKBUUMECQPZL67ZKSVYDYRA, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.300626273 (56h34m38.852096734s) [noise=true , address=77.90.31.17:4770 , piece_id=ATVM4RBHXP67K5RSXIPWDLPQSDHQHF3VLZSO7EZ2S7GASJVU4ITQ, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.302012112 (56h34m38.853482563s) [noise=true , address=145.79.227.17:4970 , piece_id=SXDKTR3E2YKUIWQVHPCJK753AAP7GD7IKVJRODNFO4KF2UXKBL4Q, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.303244468 (56h34m38.854714929s) [noise=true , address=145.79.227.22:4511 , piece_id=QET7Y6RMU3QZUJGQHPE5F567O6K6Q2MELKZK4H6BQTHPUW5EPNKA, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.304670594 (56h34m38.856141055s) [noise=true , address=77.90.31.15:4698 , piece_id=D4CYCESDDCKFSDZ354LB5Y3Z3H6CNQ5GWUXXYDZHXFXCVA46E25Q, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.305090844 (56h34m38.856561305s) [noise=true , address=45.91.116.8:4048 , piece_id=M7DH5VIABLYKUKZR57AIW7LJGISJINDHSQ3CWXISUDP3XPDIY5MA, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.305470417 (56h34m38.856940868s) [noise=true , address=45.91.116.13:4153 , piece_id=IUSH3457LMG3PR5IEMA4DMPYIJTLKQRXJ47VAEFSSVKKO3THKYOA, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.309258671 (56h34m38.860729132s) [noise=true , address=145.79.227.4:4186 , piece_id=DRV7Y4IEHMRXPZFRZU7DZEZ56NGROQI526OGKRXSMPFDGF3KDP5Q, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:42.050032931 (56h34m40.601503392s) [upload=49859]: Next returned nil segment Nov 11 16:38:42.050053050 (56h34m40.601523511s) [upload=49859]: Waiting for error group managing segments... Nov 11 16:38:42.379656265 (56h34m40.931126726s) [noise=true , address=45.91.116.4:4937 , piece_id=GRAKOOGQ4HXGVV32NO2YET2WMPATCMZRMYMUMV3NX2EBLXZJFC6A, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.394803549 (56h34m40.94627401s) [noise=true , address=45.91.116.14:4190 , piece_id=3FHYEMK6OKJACPWH7DL4T5HLWUXPRRTIAIGX4KRDVKHAYMDW326Q, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.400715316 (56h34m40.952185777s) [noise=true , address=77.90.31.17:4770 , piece_id=ATVM4RBHXP67K5RSXIPWDLPQSDHQHF3VLZSO7EZ2S7GASJVU4ITQ, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.409998765 (56h34m40.961469236s) [noise=true , address=77.90.31.15:4698 , piece_id=D4CYCESDDCKFSDZ354LB5Y3Z3H6CNQ5GWUXXYDZHXFXCVA46E25Q, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.418606595 (56h34m40.970077056s) [noise=true , address=77.90.31.12:4589 , piece_id=ZE6O4CU7Y2TLCXAI6HBNITBTQFFBQC77E6YUWUXX3TWMBUUGOOKA, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.443626699 (56h34m40.99509715s) [noise=true , address=45.91.116.6:4975 , piece_id=23535HUWVMJEOXR5J5UYAA3LG2E4GOWF2X4FHPXYIH7GCOY3KPIA, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.445695820 (56h34m40.997166271s) [noise=true , address=45.91.116.13:4153 , piece_id=IUSH3457LMG3PR5IEMA4DMPYIJTLKQRXJ47VAEFSSVKKO3THKYOA, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.473525152 (56h34m41.024995613s) [noise=true , address=45.91.116.5:4297 , piece_id=EYKUYVGHZQ6OQTCB2GDZEOLYBULRSQN76HRLKXEULVNTARTYHS5A, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.636670936 (56h34m41.188141397s) [noise=true , address=45.91.116.8:4048 , piece_id=M7DH5VIABLYKUKZR57AIW7LJGISJINDHSQ3CWXISUDP3XPDIY5MA, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:39:09.324245238 (56h35m7.875715699s) [noise=true , address=145.79.227.25:4618 , piece_id=FFQAWUFMKPXTC3SFYX7EWLBAYZZ5SZIUEIFEVU7B2MXKX57QOR6Q, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:57092->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:57092->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57092->145.79.227.25:4618: use of closed network connection Nov 11 16:39:09.324393590 (56h35m7.875864061s) [noise=true , address=145.79.227.13:4804 , piece_id=TTSF3ZHSWO6LYGIWBXRTPKG27KGW7XQBPL23Y5WPNENGO7MTV66Q, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:53130->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:53130->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53130->145.79.227.13:4804: use of closed network connection Nov 11 16:39:09.324491988 (56h35m7.875962449s) [noise=true , address=145.79.227.17:4970 , piece_id=SXDKTR3E2YKUIWQVHPCJK753AAP7GD7IKVJRODNFO4KF2UXKBL4Q, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:35738->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:35738->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35738->145.79.227.17:4970: use of closed network connection Nov 11 16:39:09.324535490 (56h35m7.876005951s) [noise=true , address=145.79.227.22:4511 , piece_id=QET7Y6RMU3QZUJGQHPE5F567O6K6Q2MELKZK4H6BQTHPUW5EPNKA, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:58766->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:58766->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58766->145.79.227.22:4511: use of closed network connection Nov 11 16:39:09.324644678 (56h35m7.876115139s) [noise=true , address=145.79.227.4:4186 , piece_id=DRV7Y4IEHMRXPZFRZU7DZEZ56NGROQI526OGKRXSMPFDGF3KDP5Q, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:57480->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:57480->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57480->145.79.227.4:4186: use of closed network connection Nov 11 16:39:09.324807358 (56h35m7.876277819s) [noise=true , address=145.79.227.14:4837 , piece_id=3C7EFHTCO474N3WGMIFNP24PZSVKVOFPEDNSXEPE4P26QWG3AB7A, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:60668->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 11 16:39:09.324949739 (56h35m7.8764202s) [noise=true , address=145.79.227.30:4763 , piece_id=FOA6BMKCTINQEG7UZPOWTMVL7YOVMMKBUUMECQPZL67ZKSVYDYRA, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:45392->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 11 16:39:09.324975198 (56h35m7.876445659s) [seg_pos={0 0}, upload=49859]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 11 16:39:09.578246041 (56h35m8.129716492s) [upload=49859]: Done uploading segments... Nov 11 16:39:09.578291948 (56h35m8.129762409s) [upload=49859]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 9.870745061s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X.......................................X] duration: 1m0.291689331s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 16:39:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.291715641s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 1.488215855s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:39:40 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.488245651s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 1.429687051s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:39:16 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.429711068s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................XX.....X...............................X] duration: 30.255609211s trace: https://jaeger.dev.storj.io/trace/139e8e33ab430945 FAIL (2025-11-11 16:39:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 38:40.294055811 (56h34m38.845526263s) [seg_pos={0 0}, upload=49859]: Done begin upload segment. Nov 11 16:38:40.294082692 (56h34m38.845553153s) [noise=true , address=145.79.227.14:4837 , piece_id=3C7EFHTCO474N3WGMIFNP24PZSVKVOFPEDNSXEPE4P26QWG3AB7A, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.294130494 (56h34m38.845600955s) [noise=true , address=45.91.116.14:4190 , piece_id=3FHYEMK6OKJACPWH7DL4T5HLWUXPRRTIAIGX4KRDVKHAYMDW326Q, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.294790049 (56h34m38.846260511s) [noise=true , address=145.79.227.13:4804 , piece_id=TTSF3ZHSWO6LYGIWBXRTPKG27KGW7XQBPL23Y5WPNENGO7MTV66Q, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.295175093 (56h34m38.846645544s) [noise=true , address=45.91.116.5:4297 , piece_id=EYKUYVGHZQ6OQTCB2GDZEOLYBULRSQN76HRLKXEULVNTARTYHS5A, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.295530890 (56h34m38.847001351s) [noise=true , address=45.91.116.4:4937 , piece_id=GRAKOOGQ4HXGVV32NO2YET2WMPATCMZRMYMUMV3NX2EBLXZJFC6A, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.295828958 (56h34m38.847299419s) [noise=true , address=145.79.227.25:4618 , piece_id=FFQAWUFMKPXTC3SFYX7EWLBAYZZ5SZIUEIFEVU7B2MXKX57QOR6Q, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.296176931 (56h34m38.847647382s) [noise=true , address=77.90.31.12:4589 , piece_id=ZE6O4CU7Y2TLCXAI6HBNITBTQFFBQC77E6YUWUXX3TWMBUUGOOKA, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.296649390 (56h34m38.848119841s) [noise=true , address=45.91.116.6:4975 , piece_id=23535HUWVMJEOXR5J5UYAA3LG2E4GOWF2X4FHPXYIH7GCOY3KPIA, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.298961653 (56h34m38.850432114s) [noise=true , address=145.79.227.30:4763 , piece_id=FOA6BMKCTINQEG7UZPOWTMVL7YOVMMKBUUMECQPZL67ZKSVYDYRA, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.300626273 (56h34m38.852096734s) [noise=true , address=77.90.31.17:4770 , piece_id=ATVM4RBHXP67K5RSXIPWDLPQSDHQHF3VLZSO7EZ2S7GASJVU4ITQ, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.302012112 (56h34m38.853482563s) [noise=true , address=145.79.227.17:4970 , piece_id=SXDKTR3E2YKUIWQVHPCJK753AAP7GD7IKVJRODNFO4KF2UXKBL4Q, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.303244468 (56h34m38.854714929s) [noise=true , address=145.79.227.22:4511 , piece_id=QET7Y6RMU3QZUJGQHPE5F567O6K6Q2MELKZK4H6BQTHPUW5EPNKA, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.304670594 (56h34m38.856141055s) [noise=true , address=77.90.31.15:4698 , piece_id=D4CYCESDDCKFSDZ354LB5Y3Z3H6CNQ5GWUXXYDZHXFXCVA46E25Q, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.305090844 (56h34m38.856561305s) [noise=true , address=45.91.116.8:4048 , piece_id=M7DH5VIABLYKUKZR57AIW7LJGISJINDHSQ3CWXISUDP3XPDIY5MA, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.305470417 (56h34m38.856940868s) [noise=true , address=45.91.116.13:4153 , piece_id=IUSH3457LMG3PR5IEMA4DMPYIJTLKQRXJ47VAEFSSVKKO3THKYOA, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:40.309258671 (56h34m38.860729132s) [noise=true , address=145.79.227.4:4186 , piece_id=DRV7Y4IEHMRXPZFRZU7DZEZ56NGROQI526OGKRXSMPFDGF3KDP5Q, seg_pos={0 0}, upload=49859]: Uploading piece... Nov 11 16:38:42.050032931 (56h34m40.601503392s) [upload=49859]: Next returned nil segment Nov 11 16:38:42.050053050 (56h34m40.601523511s) [upload=49859]: Waiting for error group managing segments... Nov 11 16:38:42.379656265 (56h34m40.931126726s) [noise=true , address=45.91.116.4:4937 , piece_id=GRAKOOGQ4HXGVV32NO2YET2WMPATCMZRMYMUMV3NX2EBLXZJFC6A, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.394803549 (56h34m40.94627401s) [noise=true , address=45.91.116.14:4190 , piece_id=3FHYEMK6OKJACPWH7DL4T5HLWUXPRRTIAIGX4KRDVKHAYMDW326Q, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.400715316 (56h34m40.952185777s) [noise=true , address=77.90.31.17:4770 , piece_id=ATVM4RBHXP67K5RSXIPWDLPQSDHQHF3VLZSO7EZ2S7GASJVU4ITQ, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.409998765 (56h34m40.961469236s) [noise=true , address=77.90.31.15:4698 , piece_id=D4CYCESDDCKFSDZ354LB5Y3Z3H6CNQ5GWUXXYDZHXFXCVA46E25Q, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.418606595 (56h34m40.970077056s) [noise=true , address=77.90.31.12:4589 , piece_id=ZE6O4CU7Y2TLCXAI6HBNITBTQFFBQC77E6YUWUXX3TWMBUUGOOKA, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.443626699 (56h34m40.99509715s) [noise=true , address=45.91.116.6:4975 , piece_id=23535HUWVMJEOXR5J5UYAA3LG2E4GOWF2X4FHPXYIH7GCOY3KPIA, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.445695820 (56h34m40.997166271s) [noise=true , address=45.91.116.13:4153 , piece_id=IUSH3457LMG3PR5IEMA4DMPYIJTLKQRXJ47VAEFSSVKKO3THKYOA, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.473525152 (56h34m41.024995613s) [noise=true , address=45.91.116.5:4297 , piece_id=EYKUYVGHZQ6OQTCB2GDZEOLYBULRSQN76HRLKXEULVNTARTYHS5A, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:38:42.636670936 (56h34m41.188141397s) [noise=true , address=45.91.116.8:4048 , piece_id=M7DH5VIABLYKUKZR57AIW7LJGISJINDHSQ3CWXISUDP3XPDIY5MA, seg_pos={0 0}, upload=49859]: Done uploading piece. err: <nil> Nov 11 16:39:09.324245238 (56h35m7.875715699s) [noise=true , address=145.79.227.25:4618 , piece_id=FFQAWUFMKPXTC3SFYX7EWLBAYZZ5SZIUEIFEVU7B2MXKX57QOR6Q, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:57092->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:57092->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57092->145.79.227.25:4618: use of closed network connection Nov 11 16:39:09.324393590 (56h35m7.875864061s) [noise=true , address=145.79.227.13:4804 , piece_id=TTSF3ZHSWO6LYGIWBXRTPKG27KGW7XQBPL23Y5WPNENGO7MTV66Q, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:53130->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:53130->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53130->145.79.227.13:4804: use of closed network connection Nov 11 16:39:09.324491988 (56h35m7.875962449s) [noise=true , address=145.79.227.17:4970 , piece_id=SXDKTR3E2YKUIWQVHPCJK753AAP7GD7IKVJRODNFO4KF2UXKBL4Q, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:35738->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:35738->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35738->145.79.227.17:4970: use of closed network connection Nov 11 16:39:09.324535490 (56h35m7.876005951s) [noise=true , address=145.79.227.22:4511 , piece_id=QET7Y6RMU3QZUJGQHPE5F567O6K6Q2MELKZK4H6BQTHPUW5EPNKA, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:58766->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:58766->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58766->145.79.227.22:4511: use of closed network connection Nov 11 16:39:09.324644678 (56h35m7.876115139s) [noise=true , address=145.79.227.4:4186 , piece_id=DRV7Y4IEHMRXPZFRZU7DZEZ56NGROQI526OGKRXSMPFDGF3KDP5Q, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:57480->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:57480->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57480->145.79.227.4:4186: use of closed network connection Nov 11 16:39:09.324807358 (56h35m7.876277819s) [noise=true , address=145.79.227.14:4837 , piece_id=3C7EFHTCO474N3WGMIFNP24PZSVKVOFPEDNSXEPE4P26QWG3AB7A, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:60668->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 11 16:39:09.324949739 (56h35m7.8764202s) [noise=true , address=145.79.227.30:4763 , piece_id=FOA6BMKCTINQEG7UZPOWTMVL7YOVMMKBUUMECQPZL67ZKSVYDYRA, seg_pos={0 0}, upload=49859]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:45392->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 11 16:39:09.324975198 (56h35m7.876445659s) [seg_pos={0 0}, upload=49859]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 11 16:39:09.578246041 (56h35m8.129716492s) [upload=49859]: Done uploading segments... Nov 11 16:39:09.578291948 (56h35m8.129762409s) [upload=49859]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.255633275s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.X.............] duration: 58.436436ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/54bf027531dae78c FAIL (2025-11-11 16:38:42 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:45794->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 58.464208ms core website
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 1.430671006s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:38:21 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.430698538s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.297039935s trace: https://jaeger.dev.storj.io/trace/44d8b784072367e4 FAIL (2025-11-11 16:38:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== iece... Nov 11 16:08:48.413266786 (56h4m46.964737247s) [noise=true , address=45.91.116.9:4289 , piece_id=Z5IN2YPT34E7OOXG5DUULXXJBNSFTBLKHUECCQ7B3V43BYNYYU3Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.413968803 (56h4m46.965439264s) [noise=true , address=145.79.227.37:4597 , piece_id=2RZEBVXKMEMFLOCYYCBHWYUFULN6AV57ID3YNPODCDRL6QTULKBQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.414381078 (56h4m46.965851539s) [noise=true , address=77.90.31.14:4677 , piece_id=AT7AAAXAFYRVEWXPJLBTN6FHOLXRHSIP5UZVIBUJGR7ZKJWCCV3Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.414796339 (56h4m46.9662668s) [noise=true , address=145.79.227.33:4881 , piece_id=UM4J3RWOYYEZDPDQAEWPW2LAQ2NZ732HS4VHYJV6GIVNIIS7ZCKA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.415222971 (56h4m46.966693432s) [noise=true , address=77.90.31.19:4871 , piece_id=P7NSRE4J5GU56IKHETU65J32VUJZ6GCML3TLCFUUBZOLTOT7FNNQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.415704538 (56h4m46.967174989s) [noise=true , address=145.79.227.44:4228 , piece_id=MF3DSBMBYAPI3A4VNYZOTMCUPQOFG7ALITVFTLVXSI67QZNGGH5Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416081355 (56h4m46.967551816s) [noise=true , address=77.90.31.11:4580 , piece_id=SUUPKQKDEP7FD5ZBPWCWMDDX4O5HNXYTDQCH3SF52PBRONBJX3BA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416488901 (56h4m46.967959362s) [noise=true , address=64.49.13.169:4708 , piece_id=BGOKG2A22ONOW3C42ZDGJVLCFYWBDZCNDNWEJFH5CI7IWNCEEJHQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416811835 (56h4m46.968282287s) [noise=true , address=77.90.31.18:4808 , piece_id=WG7FSO6WEY2MZVTNZ4UPRZOT57ZZSBWADNHRHCO7S7CWIFIHA4OA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.417126284 (56h4m46.968596735s) [noise=true , address=77.90.31.17:4782 , piece_id=6ZE4O6KNM7BAPNEQGEUJ5AIPQT2MUS7B4HSPX6O4CHRZZ7RDVLTA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.417504084 (56h4m46.968974535s) [noise=true , address=5.231.73.2:4019 , piece_id=RZ6A6LCAFTR5QEU4HRDH6VPZHJLLIYEEIKPYYSHPIYLKYWJTSGIQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.418850267 (56h4m46.970320719s) [noise=true , address=45.91.116.8:4073 , piece_id=NKCUJYJCSSWAUYG7DG7ZHZK7MTD7NI5QWAEDP5OSUQ56NVE65DAA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.419254357 (56h4m46.970724818s) [noise=true , address=5.231.73.5:4029 , piece_id=BFGNXLGWWONZVAU2YN5M72C4GRVMLEY4UPMYONODCJU4SBOMPKCA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.421629780 (56h4m46.973100241s) [noise=true , address=5.231.73.3:4014 , piece_id=I6JOMX6V6RHVI56BNK5QEP7I66DXLMFPXBVLV7MDEHGI4DQYVPGA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.423823017 (56h4m46.975293478s) [noise=true , address=64.49.13.154:4930 , piece_id=QJDOKUCE66TGMZBIH2H3X6VKIA6TEUY3FLWWHHOGKDFAG73E5APA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.426626956 (56h4m46.978097417s) [noise=true , address=77.90.31.13:4650 , piece_id=BBV45CNGTXMXMWYS4OMFLKOT5KOPH2GBB6NA57S47JW75AOQXVYQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.428526082 (56h4m46.979996543s) [noise=true , address=145.79.227.27:4707 , piece_id=4GYBLM647IHWXQB63TINHQXJQLKZDGEYWMDSTKFM5ZHD5RMSVSYQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:49.743855894 (56h4m48.295326345s) [upload=49413]: Next returned nil segment Nov 11 16:08:49.743869990 (56h4m48.295340451s) [upload=49413]: Waiting for error group managing segments... Nov 11 16:08:50.083885022 (56h4m48.635355483s) [noise=true , address=64.49.13.169:4708 , piece_id=BGOKG2A22ONOW3C42ZDGJVLCFYWBDZCNDNWEJFH5CI7IWNCEEJHQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.099619755 (56h4m48.651090216s) [noise=true , address=64.49.13.154:4930 , piece_id=QJDOKUCE66TGMZBIH2H3X6VKIA6TEUY3FLWWHHOGKDFAG73E5APA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.185889601 (56h4m48.737360062s) [noise=true , address=5.231.73.3:4014 , piece_id=I6JOMX6V6RHVI56BNK5QEP7I66DXLMFPXBVLV7MDEHGI4DQYVPGA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.210908903 (56h4m48.762379354s) [noise=true , address=5.231.73.2:4019 , piece_id=RZ6A6LCAFTR5QEU4HRDH6VPZHJLLIYEEIKPYYSHPIYLKYWJTSGIQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.233314608 (56h4m48.784785069s) [noise=true , address=5.231.73.5:4029 , piece_id=BFGNXLGWWONZVAU2YN5M72C4GRVMLEY4UPMYONODCJU4SBOMPKCA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.385166156 (56h4m48.936636617s) [noise=true , address=45.91.116.8:4073 , piece_id=NKCUJYJCSSWAUYG7DG7ZHZK7MTD7NI5QWAEDP5OSUQ56NVE65DAA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.412131434 (56h4m48.963601895s) [noise=true , address=45.91.116.9:4289 , piece_id=Z5IN2YPT34E7OOXG5DUULXXJBNSFTBLKHUECCQ7B3V43BYNYYU3Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.421729551 (56h4m48.973200012s) [noise=true , address=77.90.31.18:4808 , piece_id=WG7FSO6WEY2MZVTNZ4UPRZOT57ZZSBWADNHRHCO7S7CWIFIHA4OA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.430205912 (56h4m48.981676373s) [noise=true , address=77.90.31.14:4677 , piece_id=AT7AAAXAFYRVEWXPJLBTN6FHOLXRHSIP5UZVIBUJGR7ZKJWCCV3Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.455736998 (56h4m49.007207459s) [noise=true , address=77.90.31.11:4580 , piece_id=SUUPKQKDEP7FD5ZBPWCWMDDX4O5HNXYTDQCH3SF52PBRONBJX3BA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.497672944 (56h4m49.049143395s) [noise=true , address=77.90.31.17:4782 , piece_id=6ZE4O6KNM7BAPNEQGEUJ5AIPQT2MUS7B4HSPX6O4CHRZZ7RDVLTA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.511232905 (56h4m49.062703366s) [noise=true , address=77.90.31.13:4650 , piece_id=BBV45CNGTXMXMWYS4OMFLKOT5KOPH2GBB6NA57S47JW75AOQXVYQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.569351094 (56h4m49.120821555s) [noise=true , address=77.90.31.19:4871 , piece_id=P7NSRE4J5GU56IKHETU65J32VUJZ6GCML3TLCFUUBZOLTOT7FNNQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:09:47.409293414 (56h5m45.960763875s) [noise=true , address=145.79.227.17:4994 , piece_id=ALC7PHRXP3EPN37WJZ5K4CF7SENX2T2T4GZZVEGCKO6EUXHL4GYA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:13nneaPpsRT7mVEc6cDt7GiZSMTU68Nnk1rr9Lbn5ycZkmQGrB, address:145.79.227.17:4994): protocol: write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection; write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection Nov 11 16:09:47.409343289 (56h5m45.96081375s) [noise=true , address=145.79.227.33:4881 , piece_id=UM4J3RWOYYEZDPDQAEWPW2LAQ2NZ732HS4VHYJV6GIVNIIS7ZCKA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, address:145.79.227.33:4881): protocol: write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection; write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection Nov 11 16:09:47.409602432 (56h5m45.961072903s) [noise=true , address=145.79.227.27:4707 , piece_id=4GYBLM647IHWXQB63TINHQXJQLKZDGEYWMDSTKFM5ZHD5RMSVSYQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1N7XaDABpimefP5xEz5Swk53fiZk4DTv5L8Hg5cNDgeG6gHP1T, address:145.79.227.27:4707): protocol: write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection; write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection Nov 11 16:09:47.409644141 (56h5m45.961114602s) [noise=true , address=145.79.227.37:4597 , piece_id=2RZEBVXKMEMFLOCYYCBHWYUFULN6AV57ID3YNPODCDRL6QTULKBQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1q6zcreD1FCjZXuYowJz9amfcQtWySL2iyrFLU8BcPzbD8PhPb, address:145.79.227.37:4597): protocol: write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection; write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection Nov 11 16:09:47.409737079 (56h5m45.96120753s) [noise=true , address=145.79.227.44:4228 , piece_id=MF3DSBMBYAPI3A4VNYZOTMCUPQOFG7ALITVFTLVXSI67QZNGGH5Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:145.79.227.44:4228): protocol: write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection; write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection Nov 11 16:09:47.409766335 (56h5m45.961236786s) [seg_pos={0 0}, upload=49413]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 11 16:09:47.658593534 (56h5m46.210063996s) [upload=49413]: Done uploading segments... Nov 11 16:09:47.658685831 (56h5m46.210156282s) [upload=49413]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.29705897s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..X............] duration: 50.873894ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/54bf027531dae78c FAIL (2025-11-11 16:38:05 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:45794->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 50.901969ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................XX.....X...............................] duration: 7.414957322s trace: https://jaeger.dev.storj.io/trace/6ba329aed526607a FAIL (2025-11-11 16:37:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== J6QQAXZF22QSZDLYNOYZZEJHZOZ7ECQ2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.204239805 (56h9m48.755710266s) [noise=true , address=145.79.227.27:4689 , piece_id=7KGGGQOLEZXVMSB24VPACO4CT74JIL4X5M5WJMVVMBZWTFMBLGQA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.205024630 (56h9m48.756495091s) [noise=true , address=145.79.227.15:4874 , piece_id=GDFIHSA62E6AZ5UIBDPPHAVCIXK2LSGXVSILV377HS6FUJD2JYEQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.205617369 (56h9m48.75708783s) [noise=true , address=45.91.116.7:4009 , piece_id=BU2TOI6TLQGSIDUZRIF447VXGMGWJUQPSS32VMGVGG6JRDZCEEQQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.206216129 (56h9m48.75768659s) [noise=true , address=145.79.227.16:4911 , piece_id=UZVJTOTXNOV5ZD3XRBJQJZTCZJXVDNKGPFKS5VIL5QJAWRKSAGYA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.207869137 (56h9m48.759339598s) [noise=true , address=145.79.227.14:4837 , piece_id=BAJBO6ABMXBQSCGJTON4RVNYIWCJW2JMZUE6VQ5KIXXUA5SM7HPA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.209417975 (56h9m48.760888436s) [noise=true , address=145.79.227.25:4617 , piece_id=F6OMLV36XBOM467JQC4M3EQWKNHZMFEDTJEOGKX3XBXYC7USULTQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.210891231 (56h9m48.762361693s) [noise=true , address=45.91.116.5:4297 , piece_id=J5RQAJ4REJXDUKZAN6J47FW6H6M63GV4XHEQAQMRJKSW4LUWUXYQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.212264837 (56h9m48.763735288s) [noise=true , address=145.79.227.28:4727 , piece_id=ROIVR57UPY33YJRUVW2MH5PPMBLKP3ZBUR5QEW76X6BM5AVWSSDA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.213197944 (56h9m48.764668405s) [noise=true , address=45.91.116.8:4048 , piece_id=Q2CU5DS5XZAGAVNFIY4ZVGIE446ALQF3MLY53ZP7ZPFPOVAWOTIA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.217083202 (56h9m48.768553663s) [noise=true , address=45.91.116.9:4261 , piece_id=BYO7FKVEW3T4QEBT3BRHU3PAF6L6Y5WRGTOQHDWWAVQ7MNRMKG2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.218920661 (56h9m48.770391122s) [noise=true , address=145.79.227.32:4834 , piece_id=IDANNLR65LHL5JF5ARCKDE7CDAZKTOU7Q6ESIH2JZWG7J36N5WUQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.220461635 (56h9m48.771932086s) [noise=true , address=145.79.227.11:4731 , piece_id=25Z2HAC7D5QBWTMMZF5FVYR7VPHNU5YPVSXIANUO4OWFSEYHLH2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.221453104 (56h9m48.772923565s) [noise=true , address=45.91.116.13:4155 , piece_id=LHQPEZZOD2ZFPTLEDP7GXEZDSXJNN7NXXGXJEZ3P65MDPDQBE25A, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.222919085 (56h9m48.774389576s) [noise=true , address=77.90.31.16:4735 , piece_id=6TZBUPJFXC6LARP6A5V5U4RFSYK6DDNRULNPDBI55A7KCJQO43FQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.224462674 (56h9m48.775933135s) [noise=true , address=77.90.31.19:4843 , piece_id=TZGTZDKKMKAGX2HLFPZJPMI5JPKZ3NIW2MMBIHBDPV4A54T4IWTA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:51.977601348 (56h9m50.52907181s) [upload=49489]: Next returned nil segment Nov 11 16:13:51.977618881 (56h9m50.529089342s) [upload=49489]: Waiting for error group managing segments... Nov 11 16:13:52.318041740 (56h9m50.869512201s) [noise=true , address=77.90.31.19:4843 , piece_id=TZGTZDKKMKAGX2HLFPZJPMI5JPKZ3NIW2MMBIHBDPV4A54T4IWTA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.319228510 (56h9m50.870699001s) [noise=true , address=45.91.116.8:4048 , piece_id=Q2CU5DS5XZAGAVNFIY4ZVGIE446ALQF3MLY53ZP7ZPFPOVAWOTIA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.330974096 (56h9m50.882444547s) [noise=true , address=45.91.116.5:4297 , piece_id=J5RQAJ4REJXDUKZAN6J47FW6H6M63GV4XHEQAQMRJKSW4LUWUXYQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.366779437 (56h9m50.918249898s) [noise=true , address=77.90.31.16:4735 , piece_id=6TZBUPJFXC6LARP6A5V5U4RFSYK6DDNRULNPDBI55A7KCJQO43FQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.403498277 (56h9m50.954968738s) [noise=true , address=45.91.116.13:4155 , piece_id=LHQPEZZOD2ZFPTLEDP7GXEZDSXJNN7NXXGXJEZ3P65MDPDQBE25A, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.410852602 (56h9m50.962323093s) [noise=true , address=45.91.116.7:4009 , piece_id=BU2TOI6TLQGSIDUZRIF447VXGMGWJUQPSS32VMGVGG6JRDZCEEQQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.441652722 (56h9m50.993123183s) [noise=true , address=77.90.31.15:4698 , piece_id=KJZ2GOHKXXWREHUAIMSJ6QQAXZF22QSZDLYNOYZZEJHZOZ7ECQ2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.943383035 (56h9m51.494853496s) [noise=true , address=45.91.116.9:4261 , piece_id=BYO7FKVEW3T4QEBT3BRHU3PAF6L6Y5WRGTOQHDWWAVQ7MNRMKG2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:14:19.226465189 (56h10m17.77793565s) [noise=true , address=145.79.227.16:4911 , piece_id=UZVJTOTXNOV5ZD3XRBJQJZTCZJXVDNKGPFKS5VIL5QJAWRKSAGYA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection; write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection Nov 11 16:14:19.226675380 (56h10m17.778145851s) [noise=true , address=145.79.227.25:4617 , piece_id=F6OMLV36XBOM467JQC4M3EQWKNHZMFEDTJEOGKX3XBXYC7USULTQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection Nov 11 16:14:19.226719734 (56h10m17.778190195s) [noise=true , address=145.79.227.15:4874 , piece_id=GDFIHSA62E6AZ5UIBDPPHAVCIXK2LSGXVSILV377HS6FUJD2JYEQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection Nov 11 16:14:19.226939854 (56h10m17.778410325s) [noise=true , address=145.79.227.27:4689 , piece_id=7KGGGQOLEZXVMSB24VPACO4CT74JIL4X5M5WJMVVMBZWTFMBLGQA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:46848->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 11 16:14:19.226954371 (56h10m17.778424832s) [noise=true , address=145.79.227.14:4837 , piece_id=BAJBO6ABMXBQSCGJTON4RVNYIWCJW2JMZUE6VQ5KIXXUA5SM7HPA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection Nov 11 16:14:19.227194228 (56h10m17.778664699s) [noise=true , address=145.79.227.32:4834 , piece_id=IDANNLR65LHL5JF5ARCKDE7CDAZKTOU7Q6ESIH2JZWG7J36N5WUQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:47846->145.79.227.32:4834: use of closed network connection; context deadline exceeded Nov 11 16:14:19.227398437 (56h10m17.778868898s) [noise=true , address=145.79.227.11:4731 , piece_id=25Z2HAC7D5QBWTMMZF5FVYR7VPHNU5YPVSXIANUO4OWFSEYHLH2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:46890->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 11 16:14:19.228143486 (56h10m17.779614047s) [noise=true , address=145.79.227.28:4727 , piece_id=ROIVR57UPY33YJRUVW2MH5PPMBLKP3ZBUR5QEW76X6BM5AVWSSDA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection Nov 11 16:14:19.228201325 (56h10m17.779671797s) [seg_pos={0 0}, upload=49489]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 16:14:19.496444177 (56h10m18.047914638s) [upload=49489]: Done uploading segments... Nov 11 16:14:19.496495415 (56h10m18.047965877s) [upload=49489]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 7.414998291s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.240386022s trace: https://jaeger.dev.storj.io/trace/44d8b784072367e4 FAIL (2025-11-11 16:37:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== iece... Nov 11 16:08:48.413266786 (56h4m46.964737247s) [noise=true , address=45.91.116.9:4289 , piece_id=Z5IN2YPT34E7OOXG5DUULXXJBNSFTBLKHUECCQ7B3V43BYNYYU3Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.413968803 (56h4m46.965439264s) [noise=true , address=145.79.227.37:4597 , piece_id=2RZEBVXKMEMFLOCYYCBHWYUFULN6AV57ID3YNPODCDRL6QTULKBQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.414381078 (56h4m46.965851539s) [noise=true , address=77.90.31.14:4677 , piece_id=AT7AAAXAFYRVEWXPJLBTN6FHOLXRHSIP5UZVIBUJGR7ZKJWCCV3Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.414796339 (56h4m46.9662668s) [noise=true , address=145.79.227.33:4881 , piece_id=UM4J3RWOYYEZDPDQAEWPW2LAQ2NZ732HS4VHYJV6GIVNIIS7ZCKA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.415222971 (56h4m46.966693432s) [noise=true , address=77.90.31.19:4871 , piece_id=P7NSRE4J5GU56IKHETU65J32VUJZ6GCML3TLCFUUBZOLTOT7FNNQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.415704538 (56h4m46.967174989s) [noise=true , address=145.79.227.44:4228 , piece_id=MF3DSBMBYAPI3A4VNYZOTMCUPQOFG7ALITVFTLVXSI67QZNGGH5Q, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416081355 (56h4m46.967551816s) [noise=true , address=77.90.31.11:4580 , piece_id=SUUPKQKDEP7FD5ZBPWCWMDDX4O5HNXYTDQCH3SF52PBRONBJX3BA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416488901 (56h4m46.967959362s) [noise=true , address=64.49.13.169:4708 , piece_id=BGOKG2A22ONOW3C42ZDGJVLCFYWBDZCNDNWEJFH5CI7IWNCEEJHQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.416811835 (56h4m46.968282287s) [noise=true , address=77.90.31.18:4808 , piece_id=WG7FSO6WEY2MZVTNZ4UPRZOT57ZZSBWADNHRHCO7S7CWIFIHA4OA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.417126284 (56h4m46.968596735s) [noise=true , address=77.90.31.17:4782 , piece_id=6ZE4O6KNM7BAPNEQGEUJ5AIPQT2MUS7B4HSPX6O4CHRZZ7RDVLTA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.417504084 (56h4m46.968974535s) [noise=true , address=5.231.73.2:4019 , piece_id=RZ6A6LCAFTR5QEU4HRDH6VPZHJLLIYEEIKPYYSHPIYLKYWJTSGIQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.418850267 (56h4m46.970320719s) [noise=true , address=45.91.116.8:4073 , piece_id=NKCUJYJCSSWAUYG7DG7ZHZK7MTD7NI5QWAEDP5OSUQ56NVE65DAA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.419254357 (56h4m46.970724818s) [noise=true , address=5.231.73.5:4029 , piece_id=BFGNXLGWWONZVAU2YN5M72C4GRVMLEY4UPMYONODCJU4SBOMPKCA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.421629780 (56h4m46.973100241s) [noise=true , address=5.231.73.3:4014 , piece_id=I6JOMX6V6RHVI56BNK5QEP7I66DXLMFPXBVLV7MDEHGI4DQYVPGA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.423823017 (56h4m46.975293478s) [noise=true , address=64.49.13.154:4930 , piece_id=QJDOKUCE66TGMZBIH2H3X6VKIA6TEUY3FLWWHHOGKDFAG73E5APA, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.426626956 (56h4m46.978097417s) [noise=true , address=77.90.31.13:4650 , piece_id=BBV45CNGTXMXMWYS4OMFLKOT5KOPH2GBB6NA57S47JW75AOQXVYQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:48.428526082 (56h4m46.979996543s) [noise=true , address=145.79.227.27:4707 , piece_id=4GYBLM647IHWXQB63TINHQXJQLKZDGEYWMDSTKFM5ZHD5RMSVSYQ, seg_pos={0 0}, upload=49413]: Uploading piece... Nov 11 16:08:49.743855894 (56h4m48.295326345s) [upload=49413]: Next returned nil segment Nov 11 16:08:49.743869990 (56h4m48.295340451s) [upload=49413]: Waiting for error group managing segments... Nov 11 16:08:50.083885022 (56h4m48.635355483s) [noise=true , address=64.49.13.169:4708 , piece_id=BGOKG2A22ONOW3C42ZDGJVLCFYWBDZCNDNWEJFH5CI7IWNCEEJHQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.099619755 (56h4m48.651090216s) [noise=true , address=64.49.13.154:4930 , piece_id=QJDOKUCE66TGMZBIH2H3X6VKIA6TEUY3FLWWHHOGKDFAG73E5APA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.185889601 (56h4m48.737360062s) [noise=true , address=5.231.73.3:4014 , piece_id=I6JOMX6V6RHVI56BNK5QEP7I66DXLMFPXBVLV7MDEHGI4DQYVPGA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.210908903 (56h4m48.762379354s) [noise=true , address=5.231.73.2:4019 , piece_id=RZ6A6LCAFTR5QEU4HRDH6VPZHJLLIYEEIKPYYSHPIYLKYWJTSGIQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.233314608 (56h4m48.784785069s) [noise=true , address=5.231.73.5:4029 , piece_id=BFGNXLGWWONZVAU2YN5M72C4GRVMLEY4UPMYONODCJU4SBOMPKCA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.385166156 (56h4m48.936636617s) [noise=true , address=45.91.116.8:4073 , piece_id=NKCUJYJCSSWAUYG7DG7ZHZK7MTD7NI5QWAEDP5OSUQ56NVE65DAA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.412131434 (56h4m48.963601895s) [noise=true , address=45.91.116.9:4289 , piece_id=Z5IN2YPT34E7OOXG5DUULXXJBNSFTBLKHUECCQ7B3V43BYNYYU3Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.421729551 (56h4m48.973200012s) [noise=true , address=77.90.31.18:4808 , piece_id=WG7FSO6WEY2MZVTNZ4UPRZOT57ZZSBWADNHRHCO7S7CWIFIHA4OA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.430205912 (56h4m48.981676373s) [noise=true , address=77.90.31.14:4677 , piece_id=AT7AAAXAFYRVEWXPJLBTN6FHOLXRHSIP5UZVIBUJGR7ZKJWCCV3Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.455736998 (56h4m49.007207459s) [noise=true , address=77.90.31.11:4580 , piece_id=SUUPKQKDEP7FD5ZBPWCWMDDX4O5HNXYTDQCH3SF52PBRONBJX3BA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.497672944 (56h4m49.049143395s) [noise=true , address=77.90.31.17:4782 , piece_id=6ZE4O6KNM7BAPNEQGEUJ5AIPQT2MUS7B4HSPX6O4CHRZZ7RDVLTA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.511232905 (56h4m49.062703366s) [noise=true , address=77.90.31.13:4650 , piece_id=BBV45CNGTXMXMWYS4OMFLKOT5KOPH2GBB6NA57S47JW75AOQXVYQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:08:50.569351094 (56h4m49.120821555s) [noise=true , address=77.90.31.19:4871 , piece_id=P7NSRE4J5GU56IKHETU65J32VUJZ6GCML3TLCFUUBZOLTOT7FNNQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: <nil> Nov 11 16:09:47.409293414 (56h5m45.960763875s) [noise=true , address=145.79.227.17:4994 , piece_id=ALC7PHRXP3EPN37WJZ5K4CF7SENX2T2T4GZZVEGCKO6EUXHL4GYA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:13nneaPpsRT7mVEc6cDt7GiZSMTU68Nnk1rr9Lbn5ycZkmQGrB, address:145.79.227.17:4994): protocol: write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection; write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36002->145.79.227.17:4994: use of closed network connection Nov 11 16:09:47.409343289 (56h5m45.96081375s) [noise=true , address=145.79.227.33:4881 , piece_id=UM4J3RWOYYEZDPDQAEWPW2LAQ2NZ732HS4VHYJV6GIVNIIS7ZCKA, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, address:145.79.227.33:4881): protocol: write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection; write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->145.79.227.33:4881: use of closed network connection Nov 11 16:09:47.409602432 (56h5m45.961072903s) [noise=true , address=145.79.227.27:4707 , piece_id=4GYBLM647IHWXQB63TINHQXJQLKZDGEYWMDSTKFM5ZHD5RMSVSYQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1N7XaDABpimefP5xEz5Swk53fiZk4DTv5L8Hg5cNDgeG6gHP1T, address:145.79.227.27:4707): protocol: write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection; write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35222->145.79.227.27:4707: use of closed network connection Nov 11 16:09:47.409644141 (56h5m45.961114602s) [noise=true , address=145.79.227.37:4597 , piece_id=2RZEBVXKMEMFLOCYYCBHWYUFULN6AV57ID3YNPODCDRL6QTULKBQ, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:1q6zcreD1FCjZXuYowJz9amfcQtWySL2iyrFLU8BcPzbD8PhPb, address:145.79.227.37:4597): protocol: write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection; write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34602->145.79.227.37:4597: use of closed network connection Nov 11 16:09:47.409737079 (56h5m45.96120753s) [noise=true , address=145.79.227.44:4228 , piece_id=MF3DSBMBYAPI3A4VNYZOTMCUPQOFG7ALITVFTLVXSI67QZNGGH5Q, seg_pos={0 0}, upload=49413]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:145.79.227.44:4228): protocol: write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection; write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46990->145.79.227.44:4228: use of closed network connection Nov 11 16:09:47.409766335 (56h5m45.961236786s) [seg_pos={0 0}, upload=49413]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 11 16:09:47.658593534 (56h5m46.210063996s) [upload=49413]: Done uploading segments... Nov 11 16:09:47.658685831 (56h5m46.210156282s) [upload=49413]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.240414745s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................XX.....X..............................] duration: 7.271434015s trace: https://jaeger.dev.storj.io/trace/6ba329aed526607a FAIL (2025-11-11 16:37:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== J6QQAXZF22QSZDLYNOYZZEJHZOZ7ECQ2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.204239805 (56h9m48.755710266s) [noise=true , address=145.79.227.27:4689 , piece_id=7KGGGQOLEZXVMSB24VPACO4CT74JIL4X5M5WJMVVMBZWTFMBLGQA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.205024630 (56h9m48.756495091s) [noise=true , address=145.79.227.15:4874 , piece_id=GDFIHSA62E6AZ5UIBDPPHAVCIXK2LSGXVSILV377HS6FUJD2JYEQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.205617369 (56h9m48.75708783s) [noise=true , address=45.91.116.7:4009 , piece_id=BU2TOI6TLQGSIDUZRIF447VXGMGWJUQPSS32VMGVGG6JRDZCEEQQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.206216129 (56h9m48.75768659s) [noise=true , address=145.79.227.16:4911 , piece_id=UZVJTOTXNOV5ZD3XRBJQJZTCZJXVDNKGPFKS5VIL5QJAWRKSAGYA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.207869137 (56h9m48.759339598s) [noise=true , address=145.79.227.14:4837 , piece_id=BAJBO6ABMXBQSCGJTON4RVNYIWCJW2JMZUE6VQ5KIXXUA5SM7HPA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.209417975 (56h9m48.760888436s) [noise=true , address=145.79.227.25:4617 , piece_id=F6OMLV36XBOM467JQC4M3EQWKNHZMFEDTJEOGKX3XBXYC7USULTQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.210891231 (56h9m48.762361693s) [noise=true , address=45.91.116.5:4297 , piece_id=J5RQAJ4REJXDUKZAN6J47FW6H6M63GV4XHEQAQMRJKSW4LUWUXYQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.212264837 (56h9m48.763735288s) [noise=true , address=145.79.227.28:4727 , piece_id=ROIVR57UPY33YJRUVW2MH5PPMBLKP3ZBUR5QEW76X6BM5AVWSSDA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.213197944 (56h9m48.764668405s) [noise=true , address=45.91.116.8:4048 , piece_id=Q2CU5DS5XZAGAVNFIY4ZVGIE446ALQF3MLY53ZP7ZPFPOVAWOTIA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.217083202 (56h9m48.768553663s) [noise=true , address=45.91.116.9:4261 , piece_id=BYO7FKVEW3T4QEBT3BRHU3PAF6L6Y5WRGTOQHDWWAVQ7MNRMKG2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.218920661 (56h9m48.770391122s) [noise=true , address=145.79.227.32:4834 , piece_id=IDANNLR65LHL5JF5ARCKDE7CDAZKTOU7Q6ESIH2JZWG7J36N5WUQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.220461635 (56h9m48.771932086s) [noise=true , address=145.79.227.11:4731 , piece_id=25Z2HAC7D5QBWTMMZF5FVYR7VPHNU5YPVSXIANUO4OWFSEYHLH2Q, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.221453104 (56h9m48.772923565s) [noise=true , address=45.91.116.13:4155 , piece_id=LHQPEZZOD2ZFPTLEDP7GXEZDSXJNN7NXXGXJEZ3P65MDPDQBE25A, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.222919085 (56h9m48.774389576s) [noise=true , address=77.90.31.16:4735 , piece_id=6TZBUPJFXC6LARP6A5V5U4RFSYK6DDNRULNPDBI55A7KCJQO43FQ, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:50.224462674 (56h9m48.775933135s) [noise=true , address=77.90.31.19:4843 , piece_id=TZGTZDKKMKAGX2HLFPZJPMI5JPKZ3NIW2MMBIHBDPV4A54T4IWTA, seg_pos={0 0}, upload=49489]: Uploading piece... Nov 11 16:13:51.977601348 (56h9m50.52907181s) [upload=49489]: Next returned nil segment Nov 11 16:13:51.977618881 (56h9m50.529089342s) [upload=49489]: Waiting for error group managing segments... Nov 11 16:13:52.318041740 (56h9m50.869512201s) [noise=true , address=77.90.31.19:4843 , piece_id=TZGTZDKKMKAGX2HLFPZJPMI5JPKZ3NIW2MMBIHBDPV4A54T4IWTA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.319228510 (56h9m50.870699001s) [noise=true , address=45.91.116.8:4048 , piece_id=Q2CU5DS5XZAGAVNFIY4ZVGIE446ALQF3MLY53ZP7ZPFPOVAWOTIA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.330974096 (56h9m50.882444547s) [noise=true , address=45.91.116.5:4297 , piece_id=J5RQAJ4REJXDUKZAN6J47FW6H6M63GV4XHEQAQMRJKSW4LUWUXYQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.366779437 (56h9m50.918249898s) [noise=true , address=77.90.31.16:4735 , piece_id=6TZBUPJFXC6LARP6A5V5U4RFSYK6DDNRULNPDBI55A7KCJQO43FQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.403498277 (56h9m50.954968738s) [noise=true , address=45.91.116.13:4155 , piece_id=LHQPEZZOD2ZFPTLEDP7GXEZDSXJNN7NXXGXJEZ3P65MDPDQBE25A, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.410852602 (56h9m50.962323093s) [noise=true , address=45.91.116.7:4009 , piece_id=BU2TOI6TLQGSIDUZRIF447VXGMGWJUQPSS32VMGVGG6JRDZCEEQQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.441652722 (56h9m50.993123183s) [noise=true , address=77.90.31.15:4698 , piece_id=KJZ2GOHKXXWREHUAIMSJ6QQAXZF22QSZDLYNOYZZEJHZOZ7ECQ2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:13:52.943383035 (56h9m51.494853496s) [noise=true , address=45.91.116.9:4261 , piece_id=BYO7FKVEW3T4QEBT3BRHU3PAF6L6Y5WRGTOQHDWWAVQ7MNRMKG2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: <nil> Nov 11 16:14:19.226465189 (56h10m17.77793565s) [noise=true , address=145.79.227.16:4911 , piece_id=UZVJTOTXNOV5ZD3XRBJQJZTCZJXVDNKGPFKS5VIL5QJAWRKSAGYA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection; write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54034->145.79.227.16:4911: use of closed network connection Nov 11 16:14:19.226675380 (56h10m17.778145851s) [noise=true , address=145.79.227.25:4617 , piece_id=F6OMLV36XBOM467JQC4M3EQWKNHZMFEDTJEOGKX3XBXYC7USULTQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36394->145.79.227.25:4617: use of closed network connection Nov 11 16:14:19.226719734 (56h10m17.778190195s) [noise=true , address=145.79.227.15:4874 , piece_id=GDFIHSA62E6AZ5UIBDPPHAVCIXK2LSGXVSILV377HS6FUJD2JYEQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36768->145.79.227.15:4874: use of closed network connection Nov 11 16:14:19.226939854 (56h10m17.778410325s) [noise=true , address=145.79.227.27:4689 , piece_id=7KGGGQOLEZXVMSB24VPACO4CT74JIL4X5M5WJMVVMBZWTFMBLGQA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:46848->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 11 16:14:19.226954371 (56h10m17.778424832s) [noise=true , address=145.79.227.14:4837 , piece_id=BAJBO6ABMXBQSCGJTON4RVNYIWCJW2JMZUE6VQ5KIXXUA5SM7HPA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56454->145.79.227.14:4837: use of closed network connection Nov 11 16:14:19.227194228 (56h10m17.778664699s) [noise=true , address=145.79.227.32:4834 , piece_id=IDANNLR65LHL5JF5ARCKDE7CDAZKTOU7Q6ESIH2JZWG7J36N5WUQ, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:47846->145.79.227.32:4834: use of closed network connection; context deadline exceeded Nov 11 16:14:19.227398437 (56h10m17.778868898s) [noise=true , address=145.79.227.11:4731 , piece_id=25Z2HAC7D5QBWTMMZF5FVYR7VPHNU5YPVSXIANUO4OWFSEYHLH2Q, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:46890->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 11 16:14:19.228143486 (56h10m17.779614047s) [noise=true , address=145.79.227.28:4727 , piece_id=ROIVR57UPY33YJRUVW2MH5PPMBLKP3ZBUR5QEW76X6BM5AVWSSDA, seg_pos={0 0}, upload=49489]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35828->145.79.227.28:4727: use of closed network connection Nov 11 16:14:19.228201325 (56h10m17.779671797s) [seg_pos={0 0}, upload=49489]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 16:14:19.496444177 (56h10m18.047914638s) [upload=49489]: Done uploading segments... Nov 11 16:14:19.496495415 (56h10m18.047965877s) [upload=49489]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 7.271465044s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 1.463361668s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:37:18 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.463391936s core uplink eu1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [...X...........] duration: 64.364344ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/54bf027531dae78c FAIL (2025-11-11 16:36:57 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:45794->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 64.402927ms core website
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 1.467038591s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:36:53 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.46706962s core uplink eu1

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)