Orbiter History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 1.452127211s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:55:31 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.45215796s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.047271552s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 16:55:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.047338269s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X...............................X......X...............] duration: 7.787707235s trace: https://jaeger.dev.storj.io/trace/406600cd16680b82 FAIL (2025-11-11 16:55:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 952]: Done starting upload Nov 11 16:45:40.082304163 (56h41m38.633774624s) [upload=49952]: Uploading segments... Nov 11 16:45:40.082345682 (56h41m38.633816133s) [upload=49952]: Got next segment. Inline: false Nov 11 16:45:40.082372463 (56h41m38.633842924s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 16:45:40.082377623 (56h41m38.633848074s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 16:45:40.343320114 (56h41m38.894790575s) [seg_pos={0 0}, upload=49952]: Begin upload segment... Nov 11 16:45:40.343375529 (56h41m38.89484599s) [seg_pos={0 0}, upload=49952]: Done begin upload segment. Nov 11 16:45:40.343423430 (56h41m38.894893881s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.343448578 (56h41m38.894919029s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344383769 (56h41m38.89585423s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344997518 (56h41m38.896467979s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345095395 (56h41m38.896565856s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345501417 (56h41m38.896971868s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346057606 (56h41m38.897528067s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346422612 (56h41m38.897893063s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346447920 (56h41m38.897918381s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346801393 (56h41m38.898271854s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347099851 (56h41m38.898570302s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347169985 (56h41m38.898640446s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347569877 (56h41m38.899040328s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347577421 (56h41m38.899047872s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347916837 (56h41m38.899387298s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.348105827 (56h41m38.899576288s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:42.155239479 (56h41m40.70670994s) [upload=49952]: Next returned nil segment Nov 11 16:45:42.155253296 (56h41m40.706723757s) [upload=49952]: Waiting for error group managing segments... Nov 11 16:45:42.509736187 (56h41m41.061206668s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.523781983 (56h41m41.075252444s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.524977681 (56h41m41.076448162s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.544084193 (56h41m41.095554654s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.545880774 (56h41m41.097351236s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.617572702 (56h41m41.169043163s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:49.538637614 (56h41m48.090108075s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.209224227 (56h41m53.760694688s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.376479046 (56h41m53.927949507s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.381095176 (56h41m53.932565637s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:02.516152664 (56h42m1.067623125s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:07.807035175 (56h42m6.358505636s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165478509 (56h42m6.71694897s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165504899 (56h42m6.71697536s) [seg_pos={0 0}, upload=49952]: Segment reached optimal threshold of 13 pieces. Nov 11 16:46:08.165731340 (56h42m6.717201801s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection Nov 11 16:46:08.165742050 (56h42m6.717212521s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.165752790 (56h42m6.717223252s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection Nov 11 16:46:08.165760676 (56h42m6.717231137s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166125290 (56h42m6.717595751s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Nov 11 16:46:08.166135960 (56h42m6.717606421s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166155798 (56h42m6.717626259s) [seg_pos={0 0}, upload=49952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 16:46:08.166239006 (56h42m6.717709467s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 16:46:08.166245079 (56h42m6.71771553s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 16:46:08.467714732 (56h42m7.019185193s) [upload=49952]: Done uploading segments... Nov 11 16:46:08.467737616 (56h42m7.019208077s) [upload=49952]: Upload finished. err: <nil> Duration: 7.78775772s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [...X...........] duration: 62.13082ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/7a59bb67253c8438 FAIL (2025-11-11 16:54:49 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59902->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 62.157141ms core website
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 1.447494722s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:54:47 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.447533677s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X.....X...............................X......X..............] duration: 7.513171139s trace: https://jaeger.dev.storj.io/trace/406600cd16680b82 FAIL (2025-11-11 16:54:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 952]: Done starting upload Nov 11 16:45:40.082304163 (56h41m38.633774624s) [upload=49952]: Uploading segments... Nov 11 16:45:40.082345682 (56h41m38.633816133s) [upload=49952]: Got next segment. Inline: false Nov 11 16:45:40.082372463 (56h41m38.633842924s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 16:45:40.082377623 (56h41m38.633848074s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 16:45:40.343320114 (56h41m38.894790575s) [seg_pos={0 0}, upload=49952]: Begin upload segment... Nov 11 16:45:40.343375529 (56h41m38.89484599s) [seg_pos={0 0}, upload=49952]: Done begin upload segment. Nov 11 16:45:40.343423430 (56h41m38.894893881s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.343448578 (56h41m38.894919029s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344383769 (56h41m38.89585423s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344997518 (56h41m38.896467979s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345095395 (56h41m38.896565856s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345501417 (56h41m38.896971868s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346057606 (56h41m38.897528067s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346422612 (56h41m38.897893063s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346447920 (56h41m38.897918381s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346801393 (56h41m38.898271854s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347099851 (56h41m38.898570302s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347169985 (56h41m38.898640446s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347569877 (56h41m38.899040328s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347577421 (56h41m38.899047872s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347916837 (56h41m38.899387298s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.348105827 (56h41m38.899576288s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:42.155239479 (56h41m40.70670994s) [upload=49952]: Next returned nil segment Nov 11 16:45:42.155253296 (56h41m40.706723757s) [upload=49952]: Waiting for error group managing segments... Nov 11 16:45:42.509736187 (56h41m41.061206668s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.523781983 (56h41m41.075252444s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.524977681 (56h41m41.076448162s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.544084193 (56h41m41.095554654s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.545880774 (56h41m41.097351236s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.617572702 (56h41m41.169043163s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:49.538637614 (56h41m48.090108075s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.209224227 (56h41m53.760694688s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.376479046 (56h41m53.927949507s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.381095176 (56h41m53.932565637s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:02.516152664 (56h42m1.067623125s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:07.807035175 (56h42m6.358505636s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165478509 (56h42m6.71694897s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165504899 (56h42m6.71697536s) [seg_pos={0 0}, upload=49952]: Segment reached optimal threshold of 13 pieces. Nov 11 16:46:08.165731340 (56h42m6.717201801s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection Nov 11 16:46:08.165742050 (56h42m6.717212521s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.165752790 (56h42m6.717223252s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection Nov 11 16:46:08.165760676 (56h42m6.717231137s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166125290 (56h42m6.717595751s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Nov 11 16:46:08.166135960 (56h42m6.717606421s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166155798 (56h42m6.717626259s) [seg_pos={0 0}, upload=49952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 16:46:08.166239006 (56h42m6.717709467s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 16:46:08.166245079 (56h42m6.71771553s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 16:46:08.467714732 (56h42m7.019185193s) [upload=49952]: Done uploading segments... Nov 11 16:46:08.467737616 (56h42m7.019208077s) [upload=49952]: Upload finished. err: <nil> Duration: 7.513237227s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.......................................X...................] duration: 6.014652548s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 16:54:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.014679349s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 1.417724885s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:53:59 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.417756405s core uplink eu1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [....X..........] duration: 68.753803ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/7a59bb67253c8438 FAIL (2025-11-11 16:53:42 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59902->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 68.783069ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [XX.....X...............................X......X.............] duration: 7.910876198s trace: https://jaeger.dev.storj.io/trace/406600cd16680b82 FAIL (2025-11-11 16:53:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 952]: Done starting upload Nov 11 16:45:40.082304163 (56h41m38.633774624s) [upload=49952]: Uploading segments... Nov 11 16:45:40.082345682 (56h41m38.633816133s) [upload=49952]: Got next segment. Inline: false Nov 11 16:45:40.082372463 (56h41m38.633842924s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 16:45:40.082377623 (56h41m38.633848074s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 16:45:40.343320114 (56h41m38.894790575s) [seg_pos={0 0}, upload=49952]: Begin upload segment... Nov 11 16:45:40.343375529 (56h41m38.89484599s) [seg_pos={0 0}, upload=49952]: Done begin upload segment. Nov 11 16:45:40.343423430 (56h41m38.894893881s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.343448578 (56h41m38.894919029s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344383769 (56h41m38.89585423s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344997518 (56h41m38.896467979s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345095395 (56h41m38.896565856s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345501417 (56h41m38.896971868s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346057606 (56h41m38.897528067s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346422612 (56h41m38.897893063s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346447920 (56h41m38.897918381s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346801393 (56h41m38.898271854s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347099851 (56h41m38.898570302s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347169985 (56h41m38.898640446s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347569877 (56h41m38.899040328s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347577421 (56h41m38.899047872s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347916837 (56h41m38.899387298s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.348105827 (56h41m38.899576288s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:42.155239479 (56h41m40.70670994s) [upload=49952]: Next returned nil segment Nov 11 16:45:42.155253296 (56h41m40.706723757s) [upload=49952]: Waiting for error group managing segments... Nov 11 16:45:42.509736187 (56h41m41.061206668s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.523781983 (56h41m41.075252444s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.524977681 (56h41m41.076448162s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.544084193 (56h41m41.095554654s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.545880774 (56h41m41.097351236s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.617572702 (56h41m41.169043163s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:49.538637614 (56h41m48.090108075s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.209224227 (56h41m53.760694688s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.376479046 (56h41m53.927949507s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.381095176 (56h41m53.932565637s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:02.516152664 (56h42m1.067623125s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:07.807035175 (56h42m6.358505636s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165478509 (56h42m6.71694897s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165504899 (56h42m6.71697536s) [seg_pos={0 0}, upload=49952]: Segment reached optimal threshold of 13 pieces. Nov 11 16:46:08.165731340 (56h42m6.717201801s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection Nov 11 16:46:08.165742050 (56h42m6.717212521s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.165752790 (56h42m6.717223252s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection Nov 11 16:46:08.165760676 (56h42m6.717231137s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166125290 (56h42m6.717595751s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Nov 11 16:46:08.166135960 (56h42m6.717606421s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166155798 (56h42m6.717626259s) [seg_pos={0 0}, upload=49952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 16:46:08.166239006 (56h42m6.717709467s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 16:46:08.166245079 (56h42m6.71771553s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 16:46:08.467714732 (56h42m7.019185193s) [upload=49952]: Done uploading segments... Nov 11 16:46:08.467737616 (56h42m7.019208077s) [upload=49952]: Upload finished. err: <nil> Duration: 7.91090847s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.....X.........] duration: 56.976185ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/7a59bb67253c8438 FAIL (2025-11-11 16:53:18 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59902->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 57.247772ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.XX.....X...............................X......X............] duration: 7.499015497s trace: https://jaeger.dev.storj.io/trace/406600cd16680b82 FAIL (2025-11-11 16:53:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 952]: Done starting upload Nov 11 16:45:40.082304163 (56h41m38.633774624s) [upload=49952]: Uploading segments... Nov 11 16:45:40.082345682 (56h41m38.633816133s) [upload=49952]: Got next segment. Inline: false Nov 11 16:45:40.082372463 (56h41m38.633842924s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 16:45:40.082377623 (56h41m38.633848074s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 16:45:40.343320114 (56h41m38.894790575s) [seg_pos={0 0}, upload=49952]: Begin upload segment... Nov 11 16:45:40.343375529 (56h41m38.89484599s) [seg_pos={0 0}, upload=49952]: Done begin upload segment. Nov 11 16:45:40.343423430 (56h41m38.894893881s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.343448578 (56h41m38.894919029s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344383769 (56h41m38.89585423s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344997518 (56h41m38.896467979s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345095395 (56h41m38.896565856s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345501417 (56h41m38.896971868s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346057606 (56h41m38.897528067s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346422612 (56h41m38.897893063s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346447920 (56h41m38.897918381s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346801393 (56h41m38.898271854s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347099851 (56h41m38.898570302s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347169985 (56h41m38.898640446s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347569877 (56h41m38.899040328s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347577421 (56h41m38.899047872s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347916837 (56h41m38.899387298s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.348105827 (56h41m38.899576288s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:42.155239479 (56h41m40.70670994s) [upload=49952]: Next returned nil segment Nov 11 16:45:42.155253296 (56h41m40.706723757s) [upload=49952]: Waiting for error group managing segments... Nov 11 16:45:42.509736187 (56h41m41.061206668s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.523781983 (56h41m41.075252444s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.524977681 (56h41m41.076448162s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.544084193 (56h41m41.095554654s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.545880774 (56h41m41.097351236s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.617572702 (56h41m41.169043163s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:49.538637614 (56h41m48.090108075s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.209224227 (56h41m53.760694688s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.376479046 (56h41m53.927949507s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.381095176 (56h41m53.932565637s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:02.516152664 (56h42m1.067623125s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:07.807035175 (56h42m6.358505636s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165478509 (56h42m6.71694897s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165504899 (56h42m6.71697536s) [seg_pos={0 0}, upload=49952]: Segment reached optimal threshold of 13 pieces. Nov 11 16:46:08.165731340 (56h42m6.717201801s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection Nov 11 16:46:08.165742050 (56h42m6.717212521s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.165752790 (56h42m6.717223252s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection Nov 11 16:46:08.165760676 (56h42m6.717231137s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166125290 (56h42m6.717595751s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Nov 11 16:46:08.166135960 (56h42m6.717606421s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166155798 (56h42m6.717626259s) [seg_pos={0 0}, upload=49952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 16:46:08.166239006 (56h42m6.717709467s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 16:46:08.166245079 (56h42m6.71771553s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 16:46:08.467714732 (56h42m7.019185193s) [upload=49952]: Done uploading segments... Nov 11 16:46:08.467737616 (56h42m7.019208077s) [upload=49952]: Upload finished. err: <nil> Duration: 7.499047588s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 1.455357205s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:53:04 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.455384906s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.......................................X..................] duration: 6.02395387s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 16:53:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.023985661s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 1.436783316s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:52:35 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.436814314s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X.......................................X.................] duration: 6.759329873s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 16:52:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.75935973s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 1.457900396s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:52:17 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.457931535s core uplink eu1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [......X........] duration: 54.572409ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/7a59bb67253c8438 FAIL (2025-11-11 16:52:13 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59902->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 54.603226ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..XX.....X...............................X......X...........] duration: 6.821696463s trace: https://jaeger.dev.storj.io/trace/406600cd16680b82 FAIL (2025-11-11 16:51:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 952]: Done starting upload Nov 11 16:45:40.082304163 (56h41m38.633774624s) [upload=49952]: Uploading segments... Nov 11 16:45:40.082345682 (56h41m38.633816133s) [upload=49952]: Got next segment. Inline: false Nov 11 16:45:40.082372463 (56h41m38.633842924s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 16:45:40.082377623 (56h41m38.633848074s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 16:45:40.343320114 (56h41m38.894790575s) [seg_pos={0 0}, upload=49952]: Begin upload segment... Nov 11 16:45:40.343375529 (56h41m38.89484599s) [seg_pos={0 0}, upload=49952]: Done begin upload segment. Nov 11 16:45:40.343423430 (56h41m38.894893881s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.343448578 (56h41m38.894919029s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344383769 (56h41m38.89585423s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344997518 (56h41m38.896467979s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345095395 (56h41m38.896565856s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345501417 (56h41m38.896971868s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346057606 (56h41m38.897528067s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346422612 (56h41m38.897893063s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346447920 (56h41m38.897918381s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346801393 (56h41m38.898271854s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347099851 (56h41m38.898570302s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347169985 (56h41m38.898640446s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347569877 (56h41m38.899040328s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347577421 (56h41m38.899047872s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347916837 (56h41m38.899387298s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.348105827 (56h41m38.899576288s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:42.155239479 (56h41m40.70670994s) [upload=49952]: Next returned nil segment Nov 11 16:45:42.155253296 (56h41m40.706723757s) [upload=49952]: Waiting for error group managing segments... Nov 11 16:45:42.509736187 (56h41m41.061206668s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.523781983 (56h41m41.075252444s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.524977681 (56h41m41.076448162s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.544084193 (56h41m41.095554654s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.545880774 (56h41m41.097351236s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.617572702 (56h41m41.169043163s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:49.538637614 (56h41m48.090108075s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.209224227 (56h41m53.760694688s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.376479046 (56h41m53.927949507s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.381095176 (56h41m53.932565637s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:02.516152664 (56h42m1.067623125s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:07.807035175 (56h42m6.358505636s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165478509 (56h42m6.71694897s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165504899 (56h42m6.71697536s) [seg_pos={0 0}, upload=49952]: Segment reached optimal threshold of 13 pieces. Nov 11 16:46:08.165731340 (56h42m6.717201801s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection Nov 11 16:46:08.165742050 (56h42m6.717212521s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.165752790 (56h42m6.717223252s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection Nov 11 16:46:08.165760676 (56h42m6.717231137s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166125290 (56h42m6.717595751s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Nov 11 16:46:08.166135960 (56h42m6.717606421s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166155798 (56h42m6.717626259s) [seg_pos={0 0}, upload=49952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 16:46:08.166239006 (56h42m6.717709467s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 16:46:08.166245079 (56h42m6.71771553s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 16:46:08.467714732 (56h42m7.019185193s) [upload=49952]: Done uploading segments... Nov 11 16:46:08.467737616 (56h42m7.019208077s) [upload=49952]: Upload finished. err: <nil> Duration: 6.821725508s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 1.426813109s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:51:54 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.426841874s core uplink eu1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.......X.......] duration: 87.571055ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/7a59bb67253c8438 FAIL (2025-11-11 16:51:37 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59902->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 87.661968ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...XX.....X...............................X......X..........] duration: 7.354700764s trace: https://jaeger.dev.storj.io/trace/406600cd16680b82 FAIL (2025-11-11 16:51:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 952]: Done starting upload Nov 11 16:45:40.082304163 (56h41m38.633774624s) [upload=49952]: Uploading segments... Nov 11 16:45:40.082345682 (56h41m38.633816133s) [upload=49952]: Got next segment. Inline: false Nov 11 16:45:40.082372463 (56h41m38.633842924s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 16:45:40.082377623 (56h41m38.633848074s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 16:45:40.343320114 (56h41m38.894790575s) [seg_pos={0 0}, upload=49952]: Begin upload segment... Nov 11 16:45:40.343375529 (56h41m38.89484599s) [seg_pos={0 0}, upload=49952]: Done begin upload segment. Nov 11 16:45:40.343423430 (56h41m38.894893881s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.343448578 (56h41m38.894919029s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344383769 (56h41m38.89585423s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344997518 (56h41m38.896467979s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345095395 (56h41m38.896565856s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345501417 (56h41m38.896971868s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346057606 (56h41m38.897528067s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346422612 (56h41m38.897893063s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346447920 (56h41m38.897918381s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346801393 (56h41m38.898271854s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347099851 (56h41m38.898570302s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347169985 (56h41m38.898640446s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347569877 (56h41m38.899040328s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347577421 (56h41m38.899047872s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347916837 (56h41m38.899387298s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.348105827 (56h41m38.899576288s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:42.155239479 (56h41m40.70670994s) [upload=49952]: Next returned nil segment Nov 11 16:45:42.155253296 (56h41m40.706723757s) [upload=49952]: Waiting for error group managing segments... Nov 11 16:45:42.509736187 (56h41m41.061206668s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.523781983 (56h41m41.075252444s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.524977681 (56h41m41.076448162s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.544084193 (56h41m41.095554654s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.545880774 (56h41m41.097351236s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.617572702 (56h41m41.169043163s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:49.538637614 (56h41m48.090108075s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.209224227 (56h41m53.760694688s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.376479046 (56h41m53.927949507s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.381095176 (56h41m53.932565637s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:02.516152664 (56h42m1.067623125s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:07.807035175 (56h42m6.358505636s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165478509 (56h42m6.71694897s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165504899 (56h42m6.71697536s) [seg_pos={0 0}, upload=49952]: Segment reached optimal threshold of 13 pieces. Nov 11 16:46:08.165731340 (56h42m6.717201801s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection Nov 11 16:46:08.165742050 (56h42m6.717212521s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.165752790 (56h42m6.717223252s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection Nov 11 16:46:08.165760676 (56h42m6.717231137s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166125290 (56h42m6.717595751s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Nov 11 16:46:08.166135960 (56h42m6.717606421s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166155798 (56h42m6.717626259s) [seg_pos={0 0}, upload=49952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 16:46:08.166239006 (56h42m6.717709467s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 16:46:08.166245079 (56h42m6.71771553s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 16:46:08.467714732 (56h42m7.019185193s) [upload=49952]: Done uploading segments... Nov 11 16:46:08.467737616 (56h42m7.019208077s) [upload=49952]: Upload finished. err: <nil> Duration: 7.354774054s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X.......................................X................] duration: 6.474002687s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 16:51:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.474034898s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 1.476515585s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:51:07 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.476549249s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....XX.....X...............................X......X.........] duration: 6.916655001s trace: https://jaeger.dev.storj.io/trace/406600cd16680b82 FAIL (2025-11-11 16:50:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 952]: Done starting upload Nov 11 16:45:40.082304163 (56h41m38.633774624s) [upload=49952]: Uploading segments... Nov 11 16:45:40.082345682 (56h41m38.633816133s) [upload=49952]: Got next segment. Inline: false Nov 11 16:45:40.082372463 (56h41m38.633842924s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 16:45:40.082377623 (56h41m38.633848074s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 16:45:40.343320114 (56h41m38.894790575s) [seg_pos={0 0}, upload=49952]: Begin upload segment... Nov 11 16:45:40.343375529 (56h41m38.89484599s) [seg_pos={0 0}, upload=49952]: Done begin upload segment. Nov 11 16:45:40.343423430 (56h41m38.894893881s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.343448578 (56h41m38.894919029s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344383769 (56h41m38.89585423s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344997518 (56h41m38.896467979s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345095395 (56h41m38.896565856s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345501417 (56h41m38.896971868s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346057606 (56h41m38.897528067s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346422612 (56h41m38.897893063s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346447920 (56h41m38.897918381s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346801393 (56h41m38.898271854s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347099851 (56h41m38.898570302s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347169985 (56h41m38.898640446s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347569877 (56h41m38.899040328s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347577421 (56h41m38.899047872s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347916837 (56h41m38.899387298s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.348105827 (56h41m38.899576288s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:42.155239479 (56h41m40.70670994s) [upload=49952]: Next returned nil segment Nov 11 16:45:42.155253296 (56h41m40.706723757s) [upload=49952]: Waiting for error group managing segments... Nov 11 16:45:42.509736187 (56h41m41.061206668s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.523781983 (56h41m41.075252444s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.524977681 (56h41m41.076448162s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.544084193 (56h41m41.095554654s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.545880774 (56h41m41.097351236s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.617572702 (56h41m41.169043163s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:49.538637614 (56h41m48.090108075s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.209224227 (56h41m53.760694688s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.376479046 (56h41m53.927949507s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.381095176 (56h41m53.932565637s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:02.516152664 (56h42m1.067623125s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:07.807035175 (56h42m6.358505636s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165478509 (56h42m6.71694897s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165504899 (56h42m6.71697536s) [seg_pos={0 0}, upload=49952]: Segment reached optimal threshold of 13 pieces. Nov 11 16:46:08.165731340 (56h42m6.717201801s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection Nov 11 16:46:08.165742050 (56h42m6.717212521s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.165752790 (56h42m6.717223252s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection Nov 11 16:46:08.165760676 (56h42m6.717231137s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166125290 (56h42m6.717595751s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Nov 11 16:46:08.166135960 (56h42m6.717606421s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166155798 (56h42m6.717626259s) [seg_pos={0 0}, upload=49952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 16:46:08.166239006 (56h42m6.717709467s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 16:46:08.166245079 (56h42m6.71771553s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 16:46:08.467714732 (56h42m7.019185193s) [upload=49952]: Done uploading segments... Nov 11 16:46:08.467737616 (56h42m7.019208077s) [upload=49952]: Upload finished. err: <nil> Duration: 6.916683605s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 2 out of 15 is failed (threshold=6) [X.......X......] duration: 56.847089ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/7a59bb67253c8438 FAIL (2025-11-11 16:50:36 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59902->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 56.860986ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....XX.....X...............................X......X........] duration: 6.838930841s trace: https://jaeger.dev.storj.io/trace/406600cd16680b82 FAIL (2025-11-11 16:50:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 952]: Done starting upload Nov 11 16:45:40.082304163 (56h41m38.633774624s) [upload=49952]: Uploading segments... Nov 11 16:45:40.082345682 (56h41m38.633816133s) [upload=49952]: Got next segment. Inline: false Nov 11 16:45:40.082372463 (56h41m38.633842924s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 16:45:40.082377623 (56h41m38.633848074s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 16:45:40.343320114 (56h41m38.894790575s) [seg_pos={0 0}, upload=49952]: Begin upload segment... Nov 11 16:45:40.343375529 (56h41m38.89484599s) [seg_pos={0 0}, upload=49952]: Done begin upload segment. Nov 11 16:45:40.343423430 (56h41m38.894893881s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.343448578 (56h41m38.894919029s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344383769 (56h41m38.89585423s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344997518 (56h41m38.896467979s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345095395 (56h41m38.896565856s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345501417 (56h41m38.896971868s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346057606 (56h41m38.897528067s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346422612 (56h41m38.897893063s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346447920 (56h41m38.897918381s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346801393 (56h41m38.898271854s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347099851 (56h41m38.898570302s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347169985 (56h41m38.898640446s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347569877 (56h41m38.899040328s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347577421 (56h41m38.899047872s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347916837 (56h41m38.899387298s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.348105827 (56h41m38.899576288s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:42.155239479 (56h41m40.70670994s) [upload=49952]: Next returned nil segment Nov 11 16:45:42.155253296 (56h41m40.706723757s) [upload=49952]: Waiting for error group managing segments... Nov 11 16:45:42.509736187 (56h41m41.061206668s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.523781983 (56h41m41.075252444s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.524977681 (56h41m41.076448162s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.544084193 (56h41m41.095554654s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.545880774 (56h41m41.097351236s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.617572702 (56h41m41.169043163s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:49.538637614 (56h41m48.090108075s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.209224227 (56h41m53.760694688s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.376479046 (56h41m53.927949507s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.381095176 (56h41m53.932565637s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:02.516152664 (56h42m1.067623125s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:07.807035175 (56h42m6.358505636s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165478509 (56h42m6.71694897s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165504899 (56h42m6.71697536s) [seg_pos={0 0}, upload=49952]: Segment reached optimal threshold of 13 pieces. Nov 11 16:46:08.165731340 (56h42m6.717201801s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection Nov 11 16:46:08.165742050 (56h42m6.717212521s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.165752790 (56h42m6.717223252s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection Nov 11 16:46:08.165760676 (56h42m6.717231137s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166125290 (56h42m6.717595751s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Nov 11 16:46:08.166135960 (56h42m6.717606421s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166155798 (56h42m6.717626259s) [seg_pos={0 0}, upload=49952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 16:46:08.166239006 (56h42m6.717709467s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 16:46:08.166245079 (56h42m6.71771553s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 16:46:08.467714732 (56h42m7.019185193s) [upload=49952]: Done uploading segments... Nov 11 16:46:08.467737616 (56h42m7.019208077s) [upload=49952]: Upload finished. err: <nil> Duration: 6.838960608s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 1.43763035s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:50:16 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.437660877s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X.......................................X...............] duration: 6.289557761s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 16:50:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.289595273s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X.......................................X..............] duration: 9.819192524s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 16:49:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 9.819355986s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......XX.....X...............................X......X.......] duration: 6.845036178s trace: https://jaeger.dev.storj.io/trace/406600cd16680b82 FAIL (2025-11-11 16:49:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 952]: Done starting upload Nov 11 16:45:40.082304163 (56h41m38.633774624s) [upload=49952]: Uploading segments... Nov 11 16:45:40.082345682 (56h41m38.633816133s) [upload=49952]: Got next segment. Inline: false Nov 11 16:45:40.082372463 (56h41m38.633842924s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 16:45:40.082377623 (56h41m38.633848074s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 16:45:40.343320114 (56h41m38.894790575s) [seg_pos={0 0}, upload=49952]: Begin upload segment... Nov 11 16:45:40.343375529 (56h41m38.89484599s) [seg_pos={0 0}, upload=49952]: Done begin upload segment. Nov 11 16:45:40.343423430 (56h41m38.894893881s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.343448578 (56h41m38.894919029s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344383769 (56h41m38.89585423s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344997518 (56h41m38.896467979s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345095395 (56h41m38.896565856s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345501417 (56h41m38.896971868s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346057606 (56h41m38.897528067s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346422612 (56h41m38.897893063s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346447920 (56h41m38.897918381s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346801393 (56h41m38.898271854s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347099851 (56h41m38.898570302s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347169985 (56h41m38.898640446s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347569877 (56h41m38.899040328s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347577421 (56h41m38.899047872s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347916837 (56h41m38.899387298s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.348105827 (56h41m38.899576288s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:42.155239479 (56h41m40.70670994s) [upload=49952]: Next returned nil segment Nov 11 16:45:42.155253296 (56h41m40.706723757s) [upload=49952]: Waiting for error group managing segments... Nov 11 16:45:42.509736187 (56h41m41.061206668s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.523781983 (56h41m41.075252444s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.524977681 (56h41m41.076448162s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.544084193 (56h41m41.095554654s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.545880774 (56h41m41.097351236s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.617572702 (56h41m41.169043163s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:49.538637614 (56h41m48.090108075s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.209224227 (56h41m53.760694688s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.376479046 (56h41m53.927949507s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.381095176 (56h41m53.932565637s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:02.516152664 (56h42m1.067623125s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:07.807035175 (56h42m6.358505636s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165478509 (56h42m6.71694897s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165504899 (56h42m6.71697536s) [seg_pos={0 0}, upload=49952]: Segment reached optimal threshold of 13 pieces. Nov 11 16:46:08.165731340 (56h42m6.717201801s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection Nov 11 16:46:08.165742050 (56h42m6.717212521s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.165752790 (56h42m6.717223252s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection Nov 11 16:46:08.165760676 (56h42m6.717231137s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166125290 (56h42m6.717595751s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Nov 11 16:46:08.166135960 (56h42m6.717606421s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166155798 (56h42m6.717626259s) [seg_pos={0 0}, upload=49952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 16:46:08.166239006 (56h42m6.717709467s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 16:46:08.166245079 (56h42m6.71771553s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 16:46:08.467714732 (56h42m7.019185193s) [upload=49952]: Done uploading segments... Nov 11 16:46:08.467737616 (56h42m7.019208077s) [upload=49952]: Upload finished. err: <nil> Duration: 6.845080692s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 2 out of 15 is failed (threshold=6) [.X.......X.....] duration: 73.305169ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/7a59bb67253c8438 FAIL (2025-11-11 16:49:41 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59902->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 73.324004ms core website
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 1.422410192s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:49:34 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.422453394s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X.......................................X.............] duration: 6.610245173s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 16:49:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.610281352s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......XX.....X...............................X......X......] duration: 6.656549286s trace: https://jaeger.dev.storj.io/trace/406600cd16680b82 FAIL (2025-11-11 16:48:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 952]: Done starting upload Nov 11 16:45:40.082304163 (56h41m38.633774624s) [upload=49952]: Uploading segments... Nov 11 16:45:40.082345682 (56h41m38.633816133s) [upload=49952]: Got next segment. Inline: false Nov 11 16:45:40.082372463 (56h41m38.633842924s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 16:45:40.082377623 (56h41m38.633848074s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 16:45:40.343320114 (56h41m38.894790575s) [seg_pos={0 0}, upload=49952]: Begin upload segment... Nov 11 16:45:40.343375529 (56h41m38.89484599s) [seg_pos={0 0}, upload=49952]: Done begin upload segment. Nov 11 16:45:40.343423430 (56h41m38.894893881s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.343448578 (56h41m38.894919029s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344383769 (56h41m38.89585423s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344997518 (56h41m38.896467979s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345095395 (56h41m38.896565856s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345501417 (56h41m38.896971868s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346057606 (56h41m38.897528067s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346422612 (56h41m38.897893063s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346447920 (56h41m38.897918381s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346801393 (56h41m38.898271854s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347099851 (56h41m38.898570302s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347169985 (56h41m38.898640446s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347569877 (56h41m38.899040328s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347577421 (56h41m38.899047872s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347916837 (56h41m38.899387298s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.348105827 (56h41m38.899576288s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:42.155239479 (56h41m40.70670994s) [upload=49952]: Next returned nil segment Nov 11 16:45:42.155253296 (56h41m40.706723757s) [upload=49952]: Waiting for error group managing segments... Nov 11 16:45:42.509736187 (56h41m41.061206668s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.523781983 (56h41m41.075252444s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.524977681 (56h41m41.076448162s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.544084193 (56h41m41.095554654s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.545880774 (56h41m41.097351236s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.617572702 (56h41m41.169043163s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:49.538637614 (56h41m48.090108075s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.209224227 (56h41m53.760694688s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.376479046 (56h41m53.927949507s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.381095176 (56h41m53.932565637s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:02.516152664 (56h42m1.067623125s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:07.807035175 (56h42m6.358505636s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165478509 (56h42m6.71694897s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165504899 (56h42m6.71697536s) [seg_pos={0 0}, upload=49952]: Segment reached optimal threshold of 13 pieces. Nov 11 16:46:08.165731340 (56h42m6.717201801s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection Nov 11 16:46:08.165742050 (56h42m6.717212521s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.165752790 (56h42m6.717223252s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection Nov 11 16:46:08.165760676 (56h42m6.717231137s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166125290 (56h42m6.717595751s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Nov 11 16:46:08.166135960 (56h42m6.717606421s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166155798 (56h42m6.717626259s) [seg_pos={0 0}, upload=49952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 16:46:08.166239006 (56h42m6.717709467s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 16:46:08.166245079 (56h42m6.71771553s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 16:46:08.467714732 (56h42m7.019185193s) [upload=49952]: Done uploading segments... Nov 11 16:46:08.467737616 (56h42m7.019208077s) [upload=49952]: Upload finished. err: <nil> Duration: 6.656577239s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 2 out of 15 is failed (threshold=6) [..X.......X....] duration: 85.772659ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/7a59bb67253c8438 FAIL (2025-11-11 16:48:50 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59902->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 85.802517ms core website
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 1.420673038s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:48:48 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.420701853s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X.......................................X............] duration: 5.965851068s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 16:48:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.965878491s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........XX.....X...............................X......X.....] duration: 6.904625928s trace: https://jaeger.dev.storj.io/trace/406600cd16680b82 FAIL (2025-11-11 16:48:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 952]: Done starting upload Nov 11 16:45:40.082304163 (56h41m38.633774624s) [upload=49952]: Uploading segments... Nov 11 16:45:40.082345682 (56h41m38.633816133s) [upload=49952]: Got next segment. Inline: false Nov 11 16:45:40.082372463 (56h41m38.633842924s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 16:45:40.082377623 (56h41m38.633848074s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 16:45:40.343320114 (56h41m38.894790575s) [seg_pos={0 0}, upload=49952]: Begin upload segment... Nov 11 16:45:40.343375529 (56h41m38.89484599s) [seg_pos={0 0}, upload=49952]: Done begin upload segment. Nov 11 16:45:40.343423430 (56h41m38.894893881s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.343448578 (56h41m38.894919029s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344383769 (56h41m38.89585423s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344997518 (56h41m38.896467979s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345095395 (56h41m38.896565856s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345501417 (56h41m38.896971868s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346057606 (56h41m38.897528067s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346422612 (56h41m38.897893063s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346447920 (56h41m38.897918381s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346801393 (56h41m38.898271854s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347099851 (56h41m38.898570302s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347169985 (56h41m38.898640446s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347569877 (56h41m38.899040328s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347577421 (56h41m38.899047872s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347916837 (56h41m38.899387298s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.348105827 (56h41m38.899576288s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:42.155239479 (56h41m40.70670994s) [upload=49952]: Next returned nil segment Nov 11 16:45:42.155253296 (56h41m40.706723757s) [upload=49952]: Waiting for error group managing segments... Nov 11 16:45:42.509736187 (56h41m41.061206668s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.523781983 (56h41m41.075252444s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.524977681 (56h41m41.076448162s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.544084193 (56h41m41.095554654s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.545880774 (56h41m41.097351236s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.617572702 (56h41m41.169043163s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:49.538637614 (56h41m48.090108075s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.209224227 (56h41m53.760694688s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.376479046 (56h41m53.927949507s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.381095176 (56h41m53.932565637s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:02.516152664 (56h42m1.067623125s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:07.807035175 (56h42m6.358505636s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165478509 (56h42m6.71694897s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165504899 (56h42m6.71697536s) [seg_pos={0 0}, upload=49952]: Segment reached optimal threshold of 13 pieces. Nov 11 16:46:08.165731340 (56h42m6.717201801s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection Nov 11 16:46:08.165742050 (56h42m6.717212521s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.165752790 (56h42m6.717223252s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection Nov 11 16:46:08.165760676 (56h42m6.717231137s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166125290 (56h42m6.717595751s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Nov 11 16:46:08.166135960 (56h42m6.717606421s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166155798 (56h42m6.717626259s) [seg_pos={0 0}, upload=49952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 16:46:08.166239006 (56h42m6.717709467s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 16:46:08.166245079 (56h42m6.71771553s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 16:46:08.467714732 (56h42m7.019185193s) [upload=49952]: Done uploading segments... Nov 11 16:46:08.467737616 (56h42m7.019208077s) [upload=49952]: Upload finished. err: <nil> Duration: 6.90465369s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 2 out of 15 is failed (threshold=6) [...X.......X...] duration: 63.087732ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/7a59bb67253c8438 FAIL (2025-11-11 16:48:18 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59902->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 63.114253ms core website
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 1.424330984s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:48:14 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.424378936s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........XX.....X...............................X......X....] duration: 6.71717864s trace: https://jaeger.dev.storj.io/trace/406600cd16680b82 FAIL (2025-11-11 16:48:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 952]: Done starting upload Nov 11 16:45:40.082304163 (56h41m38.633774624s) [upload=49952]: Uploading segments... Nov 11 16:45:40.082345682 (56h41m38.633816133s) [upload=49952]: Got next segment. Inline: false Nov 11 16:45:40.082372463 (56h41m38.633842924s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 16:45:40.082377623 (56h41m38.633848074s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 16:45:40.343320114 (56h41m38.894790575s) [seg_pos={0 0}, upload=49952]: Begin upload segment... Nov 11 16:45:40.343375529 (56h41m38.89484599s) [seg_pos={0 0}, upload=49952]: Done begin upload segment. Nov 11 16:45:40.343423430 (56h41m38.894893881s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.343448578 (56h41m38.894919029s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344383769 (56h41m38.89585423s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344997518 (56h41m38.896467979s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345095395 (56h41m38.896565856s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345501417 (56h41m38.896971868s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346057606 (56h41m38.897528067s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346422612 (56h41m38.897893063s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346447920 (56h41m38.897918381s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346801393 (56h41m38.898271854s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347099851 (56h41m38.898570302s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347169985 (56h41m38.898640446s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347569877 (56h41m38.899040328s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347577421 (56h41m38.899047872s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347916837 (56h41m38.899387298s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.348105827 (56h41m38.899576288s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:42.155239479 (56h41m40.70670994s) [upload=49952]: Next returned nil segment Nov 11 16:45:42.155253296 (56h41m40.706723757s) [upload=49952]: Waiting for error group managing segments... Nov 11 16:45:42.509736187 (56h41m41.061206668s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.523781983 (56h41m41.075252444s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.524977681 (56h41m41.076448162s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.544084193 (56h41m41.095554654s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.545880774 (56h41m41.097351236s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.617572702 (56h41m41.169043163s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:49.538637614 (56h41m48.090108075s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.209224227 (56h41m53.760694688s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.376479046 (56h41m53.927949507s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.381095176 (56h41m53.932565637s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:02.516152664 (56h42m1.067623125s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:07.807035175 (56h42m6.358505636s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165478509 (56h42m6.71694897s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165504899 (56h42m6.71697536s) [seg_pos={0 0}, upload=49952]: Segment reached optimal threshold of 13 pieces. Nov 11 16:46:08.165731340 (56h42m6.717201801s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection Nov 11 16:46:08.165742050 (56h42m6.717212521s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.165752790 (56h42m6.717223252s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection Nov 11 16:46:08.165760676 (56h42m6.717231137s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166125290 (56h42m6.717595751s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Nov 11 16:46:08.166135960 (56h42m6.717606421s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166155798 (56h42m6.717626259s) [seg_pos={0 0}, upload=49952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 16:46:08.166239006 (56h42m6.717709467s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 16:46:08.166245079 (56h42m6.71771553s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 16:46:08.467714732 (56h42m7.019185193s) [upload=49952]: Done uploading segments... Nov 11 16:46:08.467737616 (56h42m7.019208077s) [upload=49952]: Upload finished. err: <nil> Duration: 6.717206002s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 2 out of 15 is failed (threshold=6) [....X.......X..] duration: 77.01289ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/7a59bb67253c8438 FAIL (2025-11-11 16:48:00 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59902->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 77.042365ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X.......................................X...........] duration: 5.851863627s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 16:47:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.851891161s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........XX.....X...............................X......X...] duration: 6.817162884s trace: https://jaeger.dev.storj.io/trace/406600cd16680b82 FAIL (2025-11-11 16:47:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 952]: Done starting upload Nov 11 16:45:40.082304163 (56h41m38.633774624s) [upload=49952]: Uploading segments... Nov 11 16:45:40.082345682 (56h41m38.633816133s) [upload=49952]: Got next segment. Inline: false Nov 11 16:45:40.082372463 (56h41m38.633842924s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 16:45:40.082377623 (56h41m38.633848074s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 16:45:40.343320114 (56h41m38.894790575s) [seg_pos={0 0}, upload=49952]: Begin upload segment... Nov 11 16:45:40.343375529 (56h41m38.89484599s) [seg_pos={0 0}, upload=49952]: Done begin upload segment. Nov 11 16:45:40.343423430 (56h41m38.894893881s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.343448578 (56h41m38.894919029s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344383769 (56h41m38.89585423s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344997518 (56h41m38.896467979s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345095395 (56h41m38.896565856s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345501417 (56h41m38.896971868s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346057606 (56h41m38.897528067s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346422612 (56h41m38.897893063s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346447920 (56h41m38.897918381s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346801393 (56h41m38.898271854s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347099851 (56h41m38.898570302s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347169985 (56h41m38.898640446s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347569877 (56h41m38.899040328s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347577421 (56h41m38.899047872s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347916837 (56h41m38.899387298s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.348105827 (56h41m38.899576288s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:42.155239479 (56h41m40.70670994s) [upload=49952]: Next returned nil segment Nov 11 16:45:42.155253296 (56h41m40.706723757s) [upload=49952]: Waiting for error group managing segments... Nov 11 16:45:42.509736187 (56h41m41.061206668s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.523781983 (56h41m41.075252444s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.524977681 (56h41m41.076448162s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.544084193 (56h41m41.095554654s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.545880774 (56h41m41.097351236s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.617572702 (56h41m41.169043163s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:49.538637614 (56h41m48.090108075s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.209224227 (56h41m53.760694688s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.376479046 (56h41m53.927949507s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.381095176 (56h41m53.932565637s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:02.516152664 (56h42m1.067623125s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:07.807035175 (56h42m6.358505636s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165478509 (56h42m6.71694897s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165504899 (56h42m6.71697536s) [seg_pos={0 0}, upload=49952]: Segment reached optimal threshold of 13 pieces. Nov 11 16:46:08.165731340 (56h42m6.717201801s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection Nov 11 16:46:08.165742050 (56h42m6.717212521s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.165752790 (56h42m6.717223252s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection Nov 11 16:46:08.165760676 (56h42m6.717231137s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166125290 (56h42m6.717595751s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Nov 11 16:46:08.166135960 (56h42m6.717606421s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166155798 (56h42m6.717626259s) [seg_pos={0 0}, upload=49952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 16:46:08.166239006 (56h42m6.717709467s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 16:46:08.166245079 (56h42m6.71771553s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 16:46:08.467714732 (56h42m7.019185193s) [upload=49952]: Done uploading segments... Nov 11 16:46:08.467737616 (56h42m7.019208077s) [upload=49952]: Upload finished. err: <nil> Duration: 6.817189784s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 1.411356566s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:47:38 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.411515941s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........XX.....X...............................X......X..] duration: 7.871034126s trace: https://jaeger.dev.storj.io/trace/406600cd16680b82 FAIL (2025-11-11 16:47:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 952]: Done starting upload Nov 11 16:45:40.082304163 (56h41m38.633774624s) [upload=49952]: Uploading segments... Nov 11 16:45:40.082345682 (56h41m38.633816133s) [upload=49952]: Got next segment. Inline: false Nov 11 16:45:40.082372463 (56h41m38.633842924s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 16:45:40.082377623 (56h41m38.633848074s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 16:45:40.343320114 (56h41m38.894790575s) [seg_pos={0 0}, upload=49952]: Begin upload segment... Nov 11 16:45:40.343375529 (56h41m38.89484599s) [seg_pos={0 0}, upload=49952]: Done begin upload segment. Nov 11 16:45:40.343423430 (56h41m38.894893881s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.343448578 (56h41m38.894919029s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344383769 (56h41m38.89585423s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.344997518 (56h41m38.896467979s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345095395 (56h41m38.896565856s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.345501417 (56h41m38.896971868s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346057606 (56h41m38.897528067s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346422612 (56h41m38.897893063s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346447920 (56h41m38.897918381s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.346801393 (56h41m38.898271854s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347099851 (56h41m38.898570302s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347169985 (56h41m38.898640446s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347569877 (56h41m38.899040328s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347577421 (56h41m38.899047872s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.347916837 (56h41m38.899387298s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:40.348105827 (56h41m38.899576288s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Uploading piece... Nov 11 16:45:42.155239479 (56h41m40.70670994s) [upload=49952]: Next returned nil segment Nov 11 16:45:42.155253296 (56h41m40.706723757s) [upload=49952]: Waiting for error group managing segments... Nov 11 16:45:42.509736187 (56h41m41.061206668s) [noise=true , address=45.91.116.6:4975 , piece_id=Q37CZ57OB6MIUANG3NJQTLV5KIXY5UOG6CZFPWOQD6SRFAFZK3KA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.523781983 (56h41m41.075252444s) [noise=true , address=77.90.31.10:4520 , piece_id=2NNCTW2JVRNENX5Y7ZGSYF2QOFOGQDZIZZ3L5VJWE7WCRV5BZ4MQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.524977681 (56h41m41.076448162s) [noise=true , address=45.91.116.13:4153 , piece_id=2D36QFGUZ7TYLXRIH6FL2YJ2ZIPJVWSNETFHIXRIT643TFDMGLPQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.544084193 (56h41m41.095554654s) [noise=true , address=45.91.116.9:4263 , piece_id=DAMXH4PWAPWJLCFVYJVWZZIPDS3FXQYEYJGC67OVLJXUF6OTCHDA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.545880774 (56h41m41.097351236s) [noise=true , address=77.90.31.19:4842 , piece_id=CS6FMJQGYXDECHFBL25MW5LUQUSXHKEMZTZ2AJEBP7BSGCRQTHLQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:42.617572702 (56h41m41.169043163s) [noise=true , address=77.90.31.11:4555 , piece_id=DZWNBS3R25M3A4HRP5JOD76TCAH34KHKYDANCPSD64OQ6FGUWXZQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:49.538637614 (56h41m48.090108075s) [noise=true , address=145.79.227.24:4583 , piece_id=RH22P33ZXH7ZTL5S77JBPCOPJLA6BDIW2ESX7765BY33EAP2LJHA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.209224227 (56h41m53.760694688s) [noise=true , address=145.79.227.10:4402 , piece_id=5JPRHG7JSK6PTXAC7COEIDEFVXPSKM57A52LHABZTJ463LGROAXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.376479046 (56h41m53.927949507s) [noise=true , address=145.79.227.9:4365 , piece_id=25HX2WWDEFGQT6M4JXSMYNV5WAP4KE762ZXTSHZVJ5OCELINQUIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:45:55.381095176 (56h41m53.932565637s) [noise=true , address=145.79.227.27:4691 , piece_id=PJABFTXINBQXEOQ4HBNPDLAM4CEO3KMODHZAKEL7AYMWIWNVEYDQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:02.516152664 (56h42m1.067623125s) [noise=true , address=145.79.227.4:4186 , piece_id=HISMIKNW7CZGLPLZY3BSX4IZB6J3ACI5G2ELTHQBDC5WDXXR4XOQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:07.807035175 (56h42m6.358505636s) [noise=true , address=145.79.227.20:4439 , piece_id=H3TLTU4JATFAVDTACCGOMGLLKQ34DTC75KNG4V6TN6TGM5N6JDNA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165478509 (56h42m6.71694897s) [noise=true , address=145.79.227.11:4731 , piece_id=RKX62HPG3EULP65OMAUB63VFUQHL5XRGVMI4YJ5GHSVDMK7IJUZA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: <nil> Nov 11 16:46:08.165504899 (56h42m6.71697536s) [seg_pos={0 0}, upload=49952]: Segment reached optimal threshold of 13 pieces. Nov 11 16:46:08.165731340 (56h42m6.717201801s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41440->145.79.227.16:4910: use of closed network connection Nov 11 16:46:08.165742050 (56h42m6.717212521s) [noise=true , address=145.79.227.16:4910 , piece_id=DMFDFVYB2JVJYFJZG3BVZNO53XOEF7LZGOYGSP5TPBARCAQEUXTA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.165752790 (56h42m6.717223252s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44508->145.79.227.14:4837: use of closed network connection Nov 11 16:46:08.165760676 (56h42m6.717231137s) [noise=true , address=145.79.227.14:4837 , piece_id=ZUCSN3BWJKGDFUEGTEGQDPTXYJV3BDQ22LEWTDOWDUNPQTDPJXIA, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166125290 (56h42m6.717595751s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: expected piece hash; context canceled; EOF Nov 11 16:46:08.166135960 (56h42m6.717606421s) [noise=true , address=145.79.227.9:4367 , piece_id=BT7A7ACXZSYMMZVLJMZ7AFYMUDPVKAJ4T4GHTTTGW5PQDACTYWXQ, seg_pos={0 0}, upload=49952]: Long tail cancellation due to optimal threshold Nov 11 16:46:08.166155798 (56h42m6.717626259s) [seg_pos={0 0}, upload=49952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 16:46:08.166239006 (56h42m6.717709467s) [upload=49952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 16:46:08.166245079 (56h42m6.71771553s) [upload=49952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 16:46:08.467714732 (56h42m7.019185193s) [upload=49952]: Done uploading segments... Nov 11 16:46:08.467737616 (56h42m7.019208077s) [upload=49952]: Upload finished. err: <nil> Duration: 7.871066078s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 2 out of 15 is failed (threshold=6) [.....X.......X.] duration: 76.228736ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/7a59bb67253c8438 FAIL (2025-11-11 16:46:59 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59902->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 76.289181ms core website
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 1.436201457s trace: https://jaeger.dev.storj.io/trace/39de2fe32d12e1ed FAIL (2025-11-11 16:46:56 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.436249559s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X.......................................X..........] duration: 5.926831556s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 16:46:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.926854059s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)