Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 33.336574334s trace: https://jaeger.dev.storj.io/trace/26257de5bcb7c920 FAIL (2025-12-31 09:16:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== , upload=540795]: Uploading piece... Dec 31 08:35:48.315166374 (617h6m48.186249409s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.315823664 (617h6m48.186906699s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316244806 (617h6m48.187327841s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316251710 (617h6m48.187334755s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316290584 (617h6m48.187373619s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317050111 (617h6m48.188133146s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317140853 (617h6m48.188223888s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318123574 (617h6m48.189206609s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318885204 (617h6m48.189968249s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.319835805 (617h6m48.19091885s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.320309847 (617h6m48.191392882s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321148273 (617h6m48.192231308s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321502088 (617h6m48.192585133s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322658430 (617h6m48.193741495s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322775803 (617h6m48.193858848s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322882035 (617h6m48.19396507s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.323403308 (617h6m48.194486343s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:49.768189393 (617h6m49.639272428s) [upload=540795]: Next returned nil segment Dec 31 08:35:49.768201797 (617h6m49.639284832s) [upload=540795]: Waiting for error group managing segments... Dec 31 08:35:50.197657834 (617h6m50.068740869s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.209847924 (617h6m50.08093096s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.217381621 (617h6m50.088464666s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.270225653 (617h6m50.141308698s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.411342630 (617h6m50.282425665s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473471693 (617h6m50.344554738s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473607101 (617h6m50.344690136s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.479010038 (617h6m50.350093074s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.482534590 (617h6m50.353617625s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.485954982 (617h6m50.357038017s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.058742323 (617h6m50.929825358s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.071853740 (617h6m50.942936775s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.177856875 (617h6m51.04893991s) [noise=true , address=145.79.227.33:4873 , piece_id=KWLS4DUDNAEADPUA3CTATKK4MEQSUPEIPYJS3SSOGVPDOJPJHDIQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179660939 (617h6m51.050743984s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179686989 (617h6m51.050770024s) [seg_pos={0 0}, upload=540795]: Segment reached optimal threshold of 14 pieces. Dec 31 08:35:51.179944619 (617h6m51.051027684s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.179956683 (617h6m51.051039728s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180323110 (617h6m51.051406155s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180346325 (617h6m51.05142937s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180485580 (617h6m51.051568615s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180495899 (617h6m51.051578934s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180529313 (617h6m51.051612358s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180536927 (617h6m51.051619972s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180560653 (617h6m51.051643698s) [seg_pos={0 0}, upload=540795]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 08:35:51.180641827 (617h6m51.051724872s) [upload=540795]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 08:35:51.180647598 (617h6m51.051730643s) [upload=540795]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 08:35:51.479648850 (617h6m51.350731896s) [upload=540795]: Done uploading segments... Dec 31 08:35:51.479871193 (617h6m51.350954228s) [upload=540795]: Upload finished. err: <nil> Duration: 33.336593421s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [....X.X....X...XXX..X.X.....X........XX....X..XX............] duration: 7.559335212s trace: https://jaeger.dev.storj.io/trace/354579860c475801 FAIL (2025-12-31 09:15:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA) context deadline exceeded ======LOGS====== Dec 31 09:06:05.623627963 (617h37m5.494710998s) [upload=541230]: Starting upload Dec 31 09:06:05.623687576 (617h37m5.494770611s) [upload=541230]: Done starting upload Dec 31 09:06:05.623797315 (617h37m5.49488035s) [upload=541230]: Uploading segments... Dec 31 09:06:05.623963341 (617h37m5.495046377s) [upload=541230]: Got next segment. Inline: false Dec 31 09:06:05.624003378 (617h37m5.495086423s) [upload=541230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 09:06:05.624008136 (617h37m5.495091172s) [upload=541230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 09:06:05.867160403 (617h37m5.738243438s) [seg_pos={0 0}, upload=541230]: Begin upload segment... Dec 31 09:06:05.867217431 (617h37m5.738300466s) [seg_pos={0 0}, upload=541230]: Done begin upload segment. Dec 31 09:06:05.867253921 (617h37m5.738336956s) [noise=true , address=145.79.227.5:4222 , piece_id=MWRH3O4FG5ICW7JONZ4FYMKKIIXSYJZ5WPAWDH5R3OPRXNLDCCVA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.867292885 (617h37m5.73837592s) [noise=true , address=145.79.227.16:4911 , piece_id=4VJTSRLYNOFOSVS3QOUROMUTEUMFW5DJ6ORRUSFSIIXORITHIWEQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.867729256 (617h37m5.738812301s) [noise=true , address=145.79.227.39:4014 , piece_id=FXY36EU2IXZXBX2SPU3GP6N3O25EFKV6PDJTYSIIIW77G6UAEKFA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.868133556 (617h37m5.739216591s) [noise=true , address=145.79.227.47:4266 , piece_id=PPOTI2TRBUR3Y2VN23BEEHE5XHLXUOZLJZ6KWDEQAB3RTKNSNZ5A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.868533548 (617h37m5.739616593s) [noise=true , address=77.90.31.13:4627 , piece_id=QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869018351 (617h37m5.740101386s) [noise=true , address=45.91.116.6:4975 , piece_id=BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869464290 (617h37m5.740547325s) [noise=true , address=77.90.31.19:4842 , piece_id=LTE74CWDYFDNZIMZT4T5X4SSMSG27HYTV6ECSXM3E4MDLTY3R5CQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869870603 (617h37m5.740953638s) [noise=true , address=77.90.31.18:4805 , piece_id=5KOIVSWCCKOZCNXE5XLPHXBKXKATG3CWX52UM7YVYR4F5ONMH67A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.870347171 (617h37m5.741430216s) [noise=true , address=145.79.227.38:4978 , piece_id=C343XQRJMHQ3O2F3UN7H26HYJ3ZA7FSE2REO6KJD734PD5YFPCAQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.870799663 (617h37m5.741882698s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.871259889 (617h37m5.742342924s) [noise=true , address=145.79.227.41:4086 , piece_id=RMA25UH4U6DVQLJWQLSVUVEJXO6NCU43PTIXMZTUAQAURWER47GA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.871817150 (617h37m5.742900195s) [noise=true , address=145.79.227.32:4834 , piece_id=ALGY4CXOJUYNK6BTWSW4O7JPC3RSUOF62M2DEWSXYUD5YX6ND7HA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.872641620 (617h37m5.743724665s) [noise=true , address=145.79.227.36:4519 , piece_id=K4MKJ3TFQE33LJ6VMTUJJTHR725C4CUX4KW3PCD73XKPTRCKMNAA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.872695763 (617h37m5.743778808s) [noise=true , address=77.90.31.11:4555 , piece_id=DUR2NOB43B572QCLAN5KLD53SNDX4XCSQJ5UJMYXHFGV7D3Y72RA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.873688093 (617h37m5.744771128s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.873947867 (617h37m5.745030912s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:07.557273878 (617h37m7.428356913s) [upload=541230]: Next returned nil segment Dec 31 09:06:07.557296371 (617h37m7.428379416s) [upload=541230]: Waiting for error group managing segments... Dec 31 09:06:07.869989232 (617h37m7.741072267s) [noise=true , address=145.79.227.36:4519 , piece_id=K4MKJ3TFQE33LJ6VMTUJJTHR725C4CUX4KW3PCD73XKPTRCKMNAA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.877879937 (617h37m7.748962972s) [noise=true , address=145.79.227.47:4266 , piece_id=PPOTI2TRBUR3Y2VN23BEEHE5XHLXUOZLJZ6KWDEQAB3RTKNSNZ5A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.887580157 (617h37m7.758663203s) [noise=true , address=145.79.227.41:4086 , piece_id=RMA25UH4U6DVQLJWQLSVUVEJXO6NCU43PTIXMZTUAQAURWER47GA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894258915 (617h37m7.76534196s) [noise=true , address=145.79.227.38:4978 , piece_id=C343XQRJMHQ3O2F3UN7H26HYJ3ZA7FSE2REO6KJD734PD5YFPCAQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894331724 (617h37m7.765414759s) [noise=true , address=145.79.227.5:4222 , piece_id=MWRH3O4FG5ICW7JONZ4FYMKKIIXSYJZ5WPAWDH5R3OPRXNLDCCVA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894725193 (617h37m7.765808238s) [noise=true , address=145.79.227.39:4014 , piece_id=FXY36EU2IXZXBX2SPU3GP6N3O25EFKV6PDJTYSIIIW77G6UAEKFA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.933008700 (617h37m7.804091735s) [noise=true , address=145.79.227.16:4911 , piece_id=4VJTSRLYNOFOSVS3QOUROMUTEUMFW5DJ6ORRUSFSIIXORITHIWEQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.945354127 (617h37m7.816437172s) [noise=true , address=145.79.227.32:4834 , piece_id=ALGY4CXOJUYNK6BTWSW4O7JPC3RSUOF62M2DEWSXYUD5YX6ND7HA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.037141884 (617h37m7.908224929s) [noise=true , address=77.90.31.19:4842 , piece_id=LTE74CWDYFDNZIMZT4T5X4SSMSG27HYTV6ECSXM3E4MDLTY3R5CQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.105167495 (617h37m7.97625054s) [noise=true , address=77.90.31.11:4555 , piece_id=DUR2NOB43B572QCLAN5KLD53SNDX4XCSQJ5UJMYXHFGV7D3Y72RA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.156289979 (617h37m8.027373024s) [noise=true , address=77.90.31.13:4627 , piece_id=QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.174675076 (617h37m8.045758111s) [noise=true , address=77.90.31.18:4805 , piece_id=5KOIVSWCCKOZCNXE5XLPHXBKXKATG3CWX52UM7YVYR4F5ONMH67A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.188351619 (617h37m8.059434664s) [noise=true , address=45.91.116.6:4975 , piece_id=BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.188388489 (617h37m8.059471534s) [seg_pos={0 0}, upload=541230]: Segment reached optimal threshold of 13 pieces. Dec 31 09:06:08.188621363 (617h37m8.059704398s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188628386 (617h37m8.059711431s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188762321 (617h37m8.059845356s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188778482 (617h37m8.059861517s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188804071 (617h37m8.059887106s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188811945 (617h37m8.05989498s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188858845 (617h37m8.05994188s) [seg_pos={0 0}, upload=541230]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 09:06:08.189175598 (617h37m8.060258633s) [upload=541230]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:06:08.189182111 (617h37m8.060265146s) [upload=541230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:06:08.483598230 (617h37m8.354681275s) [upload=541230]: Done uploading segments... Dec 31 09:06:08.483622396 (617h37m8.354705431s) [upload=541230]: Upload finished. err: <nil> Duration: 7.559363346s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [..........X...........X....X....XXX.XX.............XX.......] duration: 3.390218076s trace: https://jaeger.dev.storj.io/trace/b3fe148e36f5660 FAIL (2025-12-31 09:15:26 UTC) orbiter warning: context deadline exceeded Duration: 3.390287429s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 15.377891347s trace: https://jaeger.dev.storj.io/trace/26257de5bcb7c920 FAIL (2025-12-31 09:15:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== , upload=540795]: Uploading piece... Dec 31 08:35:48.315166374 (617h6m48.186249409s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.315823664 (617h6m48.186906699s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316244806 (617h6m48.187327841s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316251710 (617h6m48.187334755s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316290584 (617h6m48.187373619s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317050111 (617h6m48.188133146s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317140853 (617h6m48.188223888s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318123574 (617h6m48.189206609s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318885204 (617h6m48.189968249s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.319835805 (617h6m48.19091885s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.320309847 (617h6m48.191392882s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321148273 (617h6m48.192231308s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321502088 (617h6m48.192585133s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322658430 (617h6m48.193741495s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322775803 (617h6m48.193858848s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322882035 (617h6m48.19396507s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.323403308 (617h6m48.194486343s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:49.768189393 (617h6m49.639272428s) [upload=540795]: Next returned nil segment Dec 31 08:35:49.768201797 (617h6m49.639284832s) [upload=540795]: Waiting for error group managing segments... Dec 31 08:35:50.197657834 (617h6m50.068740869s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.209847924 (617h6m50.08093096s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.217381621 (617h6m50.088464666s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.270225653 (617h6m50.141308698s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.411342630 (617h6m50.282425665s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473471693 (617h6m50.344554738s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473607101 (617h6m50.344690136s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.479010038 (617h6m50.350093074s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.482534590 (617h6m50.353617625s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.485954982 (617h6m50.357038017s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.058742323 (617h6m50.929825358s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.071853740 (617h6m50.942936775s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.177856875 (617h6m51.04893991s) [noise=true , address=145.79.227.33:4873 , piece_id=KWLS4DUDNAEADPUA3CTATKK4MEQSUPEIPYJS3SSOGVPDOJPJHDIQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179660939 (617h6m51.050743984s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179686989 (617h6m51.050770024s) [seg_pos={0 0}, upload=540795]: Segment reached optimal threshold of 14 pieces. Dec 31 08:35:51.179944619 (617h6m51.051027684s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.179956683 (617h6m51.051039728s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180323110 (617h6m51.051406155s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180346325 (617h6m51.05142937s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180485580 (617h6m51.051568615s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180495899 (617h6m51.051578934s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180529313 (617h6m51.051612358s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180536927 (617h6m51.051619972s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180560653 (617h6m51.051643698s) [seg_pos={0 0}, upload=540795]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 08:35:51.180641827 (617h6m51.051724872s) [upload=540795]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 08:35:51.180647598 (617h6m51.051730643s) [upload=540795]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 08:35:51.479648850 (617h6m51.350731896s) [upload=540795]: Done uploading segments... Dec 31 08:35:51.479871193 (617h6m51.350954228s) [upload=540795]: Upload finished. err: <nil> Duration: 15.377911094s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 40.343117342s trace: https://jaeger.dev.storj.io/trace/26257de5bcb7c920 FAIL (2025-12-31 09:14:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== , upload=540795]: Uploading piece... Dec 31 08:35:48.315166374 (617h6m48.186249409s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.315823664 (617h6m48.186906699s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316244806 (617h6m48.187327841s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316251710 (617h6m48.187334755s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316290584 (617h6m48.187373619s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317050111 (617h6m48.188133146s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317140853 (617h6m48.188223888s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318123574 (617h6m48.189206609s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318885204 (617h6m48.189968249s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.319835805 (617h6m48.19091885s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.320309847 (617h6m48.191392882s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321148273 (617h6m48.192231308s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321502088 (617h6m48.192585133s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322658430 (617h6m48.193741495s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322775803 (617h6m48.193858848s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322882035 (617h6m48.19396507s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.323403308 (617h6m48.194486343s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:49.768189393 (617h6m49.639272428s) [upload=540795]: Next returned nil segment Dec 31 08:35:49.768201797 (617h6m49.639284832s) [upload=540795]: Waiting for error group managing segments... Dec 31 08:35:50.197657834 (617h6m50.068740869s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.209847924 (617h6m50.08093096s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.217381621 (617h6m50.088464666s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.270225653 (617h6m50.141308698s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.411342630 (617h6m50.282425665s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473471693 (617h6m50.344554738s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473607101 (617h6m50.344690136s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.479010038 (617h6m50.350093074s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.482534590 (617h6m50.353617625s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.485954982 (617h6m50.357038017s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.058742323 (617h6m50.929825358s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.071853740 (617h6m50.942936775s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.177856875 (617h6m51.04893991s) [noise=true , address=145.79.227.33:4873 , piece_id=KWLS4DUDNAEADPUA3CTATKK4MEQSUPEIPYJS3SSOGVPDOJPJHDIQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179660939 (617h6m51.050743984s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179686989 (617h6m51.050770024s) [seg_pos={0 0}, upload=540795]: Segment reached optimal threshold of 14 pieces. Dec 31 08:35:51.179944619 (617h6m51.051027684s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.179956683 (617h6m51.051039728s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180323110 (617h6m51.051406155s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180346325 (617h6m51.05142937s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180485580 (617h6m51.051568615s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180495899 (617h6m51.051578934s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180529313 (617h6m51.051612358s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180536927 (617h6m51.051619972s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180560653 (617h6m51.051643698s) [seg_pos={0 0}, upload=540795]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 08:35:51.180641827 (617h6m51.051724872s) [upload=540795]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 08:35:51.180647598 (617h6m51.051730643s) [upload=540795]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 08:35:51.479648850 (617h6m51.350731896s) [upload=540795]: Done uploading segments... Dec 31 08:35:51.479871193 (617h6m51.350954228s) [upload=540795]: Upload finished. err: <nil> Duration: 40.343137601s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X....X.X....X...XXX..X.X.....X........XX....X..XX...........] duration: 5.953274565s trace: https://jaeger.dev.storj.io/trace/354579860c475801 FAIL (2025-12-31 09:14:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA) context deadline exceeded ======LOGS====== Dec 31 09:06:05.623627963 (617h37m5.494710998s) [upload=541230]: Starting upload Dec 31 09:06:05.623687576 (617h37m5.494770611s) [upload=541230]: Done starting upload Dec 31 09:06:05.623797315 (617h37m5.49488035s) [upload=541230]: Uploading segments... Dec 31 09:06:05.623963341 (617h37m5.495046377s) [upload=541230]: Got next segment. Inline: false Dec 31 09:06:05.624003378 (617h37m5.495086423s) [upload=541230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 09:06:05.624008136 (617h37m5.495091172s) [upload=541230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 09:06:05.867160403 (617h37m5.738243438s) [seg_pos={0 0}, upload=541230]: Begin upload segment... Dec 31 09:06:05.867217431 (617h37m5.738300466s) [seg_pos={0 0}, upload=541230]: Done begin upload segment. Dec 31 09:06:05.867253921 (617h37m5.738336956s) [noise=true , address=145.79.227.5:4222 , piece_id=MWRH3O4FG5ICW7JONZ4FYMKKIIXSYJZ5WPAWDH5R3OPRXNLDCCVA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.867292885 (617h37m5.73837592s) [noise=true , address=145.79.227.16:4911 , piece_id=4VJTSRLYNOFOSVS3QOUROMUTEUMFW5DJ6ORRUSFSIIXORITHIWEQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.867729256 (617h37m5.738812301s) [noise=true , address=145.79.227.39:4014 , piece_id=FXY36EU2IXZXBX2SPU3GP6N3O25EFKV6PDJTYSIIIW77G6UAEKFA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.868133556 (617h37m5.739216591s) [noise=true , address=145.79.227.47:4266 , piece_id=PPOTI2TRBUR3Y2VN23BEEHE5XHLXUOZLJZ6KWDEQAB3RTKNSNZ5A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.868533548 (617h37m5.739616593s) [noise=true , address=77.90.31.13:4627 , piece_id=QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869018351 (617h37m5.740101386s) [noise=true , address=45.91.116.6:4975 , piece_id=BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869464290 (617h37m5.740547325s) [noise=true , address=77.90.31.19:4842 , piece_id=LTE74CWDYFDNZIMZT4T5X4SSMSG27HYTV6ECSXM3E4MDLTY3R5CQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869870603 (617h37m5.740953638s) [noise=true , address=77.90.31.18:4805 , piece_id=5KOIVSWCCKOZCNXE5XLPHXBKXKATG3CWX52UM7YVYR4F5ONMH67A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.870347171 (617h37m5.741430216s) [noise=true , address=145.79.227.38:4978 , piece_id=C343XQRJMHQ3O2F3UN7H26HYJ3ZA7FSE2REO6KJD734PD5YFPCAQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.870799663 (617h37m5.741882698s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.871259889 (617h37m5.742342924s) [noise=true , address=145.79.227.41:4086 , piece_id=RMA25UH4U6DVQLJWQLSVUVEJXO6NCU43PTIXMZTUAQAURWER47GA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.871817150 (617h37m5.742900195s) [noise=true , address=145.79.227.32:4834 , piece_id=ALGY4CXOJUYNK6BTWSW4O7JPC3RSUOF62M2DEWSXYUD5YX6ND7HA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.872641620 (617h37m5.743724665s) [noise=true , address=145.79.227.36:4519 , piece_id=K4MKJ3TFQE33LJ6VMTUJJTHR725C4CUX4KW3PCD73XKPTRCKMNAA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.872695763 (617h37m5.743778808s) [noise=true , address=77.90.31.11:4555 , piece_id=DUR2NOB43B572QCLAN5KLD53SNDX4XCSQJ5UJMYXHFGV7D3Y72RA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.873688093 (617h37m5.744771128s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.873947867 (617h37m5.745030912s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:07.557273878 (617h37m7.428356913s) [upload=541230]: Next returned nil segment Dec 31 09:06:07.557296371 (617h37m7.428379416s) [upload=541230]: Waiting for error group managing segments... Dec 31 09:06:07.869989232 (617h37m7.741072267s) [noise=true , address=145.79.227.36:4519 , piece_id=K4MKJ3TFQE33LJ6VMTUJJTHR725C4CUX4KW3PCD73XKPTRCKMNAA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.877879937 (617h37m7.748962972s) [noise=true , address=145.79.227.47:4266 , piece_id=PPOTI2TRBUR3Y2VN23BEEHE5XHLXUOZLJZ6KWDEQAB3RTKNSNZ5A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.887580157 (617h37m7.758663203s) [noise=true , address=145.79.227.41:4086 , piece_id=RMA25UH4U6DVQLJWQLSVUVEJXO6NCU43PTIXMZTUAQAURWER47GA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894258915 (617h37m7.76534196s) [noise=true , address=145.79.227.38:4978 , piece_id=C343XQRJMHQ3O2F3UN7H26HYJ3ZA7FSE2REO6KJD734PD5YFPCAQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894331724 (617h37m7.765414759s) [noise=true , address=145.79.227.5:4222 , piece_id=MWRH3O4FG5ICW7JONZ4FYMKKIIXSYJZ5WPAWDH5R3OPRXNLDCCVA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894725193 (617h37m7.765808238s) [noise=true , address=145.79.227.39:4014 , piece_id=FXY36EU2IXZXBX2SPU3GP6N3O25EFKV6PDJTYSIIIW77G6UAEKFA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.933008700 (617h37m7.804091735s) [noise=true , address=145.79.227.16:4911 , piece_id=4VJTSRLYNOFOSVS3QOUROMUTEUMFW5DJ6ORRUSFSIIXORITHIWEQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.945354127 (617h37m7.816437172s) [noise=true , address=145.79.227.32:4834 , piece_id=ALGY4CXOJUYNK6BTWSW4O7JPC3RSUOF62M2DEWSXYUD5YX6ND7HA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.037141884 (617h37m7.908224929s) [noise=true , address=77.90.31.19:4842 , piece_id=LTE74CWDYFDNZIMZT4T5X4SSMSG27HYTV6ECSXM3E4MDLTY3R5CQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.105167495 (617h37m7.97625054s) [noise=true , address=77.90.31.11:4555 , piece_id=DUR2NOB43B572QCLAN5KLD53SNDX4XCSQJ5UJMYXHFGV7D3Y72RA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.156289979 (617h37m8.027373024s) [noise=true , address=77.90.31.13:4627 , piece_id=QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.174675076 (617h37m8.045758111s) [noise=true , address=77.90.31.18:4805 , piece_id=5KOIVSWCCKOZCNXE5XLPHXBKXKATG3CWX52UM7YVYR4F5ONMH67A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.188351619 (617h37m8.059434664s) [noise=true , address=45.91.116.6:4975 , piece_id=BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.188388489 (617h37m8.059471534s) [seg_pos={0 0}, upload=541230]: Segment reached optimal threshold of 13 pieces. Dec 31 09:06:08.188621363 (617h37m8.059704398s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188628386 (617h37m8.059711431s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188762321 (617h37m8.059845356s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188778482 (617h37m8.059861517s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188804071 (617h37m8.059887106s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188811945 (617h37m8.05989498s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188858845 (617h37m8.05994188s) [seg_pos={0 0}, upload=541230]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 09:06:08.189175598 (617h37m8.060258633s) [upload=541230]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:06:08.189182111 (617h37m8.060265146s) [upload=541230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:06:08.483598230 (617h37m8.354681275s) [upload=541230]: Done uploading segments... Dec 31 09:06:08.483622396 (617h37m8.354705431s) [upload=541230]: Upload finished. err: <nil> Duration: 5.953292599s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [...........X...........X....X....XXX.XX.............XX......] duration: 3.535484995s trace: https://jaeger.dev.storj.io/trace/b3fe148e36f5660 FAIL (2025-12-31 09:14:20 UTC) orbiter warning: context deadline exceeded Duration: 3.535514472s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 31.844900721s trace: https://jaeger.dev.storj.io/trace/26257de5bcb7c920 FAIL (2025-12-31 09:14:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== , upload=540795]: Uploading piece... Dec 31 08:35:48.315166374 (617h6m48.186249409s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.315823664 (617h6m48.186906699s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316244806 (617h6m48.187327841s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316251710 (617h6m48.187334755s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316290584 (617h6m48.187373619s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317050111 (617h6m48.188133146s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317140853 (617h6m48.188223888s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318123574 (617h6m48.189206609s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318885204 (617h6m48.189968249s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.319835805 (617h6m48.19091885s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.320309847 (617h6m48.191392882s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321148273 (617h6m48.192231308s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321502088 (617h6m48.192585133s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322658430 (617h6m48.193741495s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322775803 (617h6m48.193858848s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322882035 (617h6m48.19396507s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.323403308 (617h6m48.194486343s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:49.768189393 (617h6m49.639272428s) [upload=540795]: Next returned nil segment Dec 31 08:35:49.768201797 (617h6m49.639284832s) [upload=540795]: Waiting for error group managing segments... Dec 31 08:35:50.197657834 (617h6m50.068740869s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.209847924 (617h6m50.08093096s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.217381621 (617h6m50.088464666s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.270225653 (617h6m50.141308698s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.411342630 (617h6m50.282425665s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473471693 (617h6m50.344554738s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473607101 (617h6m50.344690136s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.479010038 (617h6m50.350093074s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.482534590 (617h6m50.353617625s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.485954982 (617h6m50.357038017s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.058742323 (617h6m50.929825358s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.071853740 (617h6m50.942936775s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.177856875 (617h6m51.04893991s) [noise=true , address=145.79.227.33:4873 , piece_id=KWLS4DUDNAEADPUA3CTATKK4MEQSUPEIPYJS3SSOGVPDOJPJHDIQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179660939 (617h6m51.050743984s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179686989 (617h6m51.050770024s) [seg_pos={0 0}, upload=540795]: Segment reached optimal threshold of 14 pieces. Dec 31 08:35:51.179944619 (617h6m51.051027684s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.179956683 (617h6m51.051039728s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180323110 (617h6m51.051406155s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180346325 (617h6m51.05142937s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180485580 (617h6m51.051568615s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180495899 (617h6m51.051578934s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180529313 (617h6m51.051612358s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180536927 (617h6m51.051619972s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180560653 (617h6m51.051643698s) [seg_pos={0 0}, upload=540795]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 08:35:51.180641827 (617h6m51.051724872s) [upload=540795]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 08:35:51.180647598 (617h6m51.051730643s) [upload=540795]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 08:35:51.479648850 (617h6m51.350731896s) [upload=540795]: Done uploading segments... Dec 31 08:35:51.479871193 (617h6m51.350954228s) [upload=540795]: Upload finished. err: <nil> Duration: 31.844921801s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X....X.X....X...XXX..X.X.....X........XX....X..XX..........] duration: 7.74085258s trace: https://jaeger.dev.storj.io/trace/354579860c475801 FAIL (2025-12-31 09:14:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA) context deadline exceeded ======LOGS====== Dec 31 09:06:05.623627963 (617h37m5.494710998s) [upload=541230]: Starting upload Dec 31 09:06:05.623687576 (617h37m5.494770611s) [upload=541230]: Done starting upload Dec 31 09:06:05.623797315 (617h37m5.49488035s) [upload=541230]: Uploading segments... Dec 31 09:06:05.623963341 (617h37m5.495046377s) [upload=541230]: Got next segment. Inline: false Dec 31 09:06:05.624003378 (617h37m5.495086423s) [upload=541230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 09:06:05.624008136 (617h37m5.495091172s) [upload=541230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 09:06:05.867160403 (617h37m5.738243438s) [seg_pos={0 0}, upload=541230]: Begin upload segment... Dec 31 09:06:05.867217431 (617h37m5.738300466s) [seg_pos={0 0}, upload=541230]: Done begin upload segment. Dec 31 09:06:05.867253921 (617h37m5.738336956s) [noise=true , address=145.79.227.5:4222 , piece_id=MWRH3O4FG5ICW7JONZ4FYMKKIIXSYJZ5WPAWDH5R3OPRXNLDCCVA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.867292885 (617h37m5.73837592s) [noise=true , address=145.79.227.16:4911 , piece_id=4VJTSRLYNOFOSVS3QOUROMUTEUMFW5DJ6ORRUSFSIIXORITHIWEQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.867729256 (617h37m5.738812301s) [noise=true , address=145.79.227.39:4014 , piece_id=FXY36EU2IXZXBX2SPU3GP6N3O25EFKV6PDJTYSIIIW77G6UAEKFA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.868133556 (617h37m5.739216591s) [noise=true , address=145.79.227.47:4266 , piece_id=PPOTI2TRBUR3Y2VN23BEEHE5XHLXUOZLJZ6KWDEQAB3RTKNSNZ5A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.868533548 (617h37m5.739616593s) [noise=true , address=77.90.31.13:4627 , piece_id=QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869018351 (617h37m5.740101386s) [noise=true , address=45.91.116.6:4975 , piece_id=BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869464290 (617h37m5.740547325s) [noise=true , address=77.90.31.19:4842 , piece_id=LTE74CWDYFDNZIMZT4T5X4SSMSG27HYTV6ECSXM3E4MDLTY3R5CQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869870603 (617h37m5.740953638s) [noise=true , address=77.90.31.18:4805 , piece_id=5KOIVSWCCKOZCNXE5XLPHXBKXKATG3CWX52UM7YVYR4F5ONMH67A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.870347171 (617h37m5.741430216s) [noise=true , address=145.79.227.38:4978 , piece_id=C343XQRJMHQ3O2F3UN7H26HYJ3ZA7FSE2REO6KJD734PD5YFPCAQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.870799663 (617h37m5.741882698s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.871259889 (617h37m5.742342924s) [noise=true , address=145.79.227.41:4086 , piece_id=RMA25UH4U6DVQLJWQLSVUVEJXO6NCU43PTIXMZTUAQAURWER47GA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.871817150 (617h37m5.742900195s) [noise=true , address=145.79.227.32:4834 , piece_id=ALGY4CXOJUYNK6BTWSW4O7JPC3RSUOF62M2DEWSXYUD5YX6ND7HA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.872641620 (617h37m5.743724665s) [noise=true , address=145.79.227.36:4519 , piece_id=K4MKJ3TFQE33LJ6VMTUJJTHR725C4CUX4KW3PCD73XKPTRCKMNAA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.872695763 (617h37m5.743778808s) [noise=true , address=77.90.31.11:4555 , piece_id=DUR2NOB43B572QCLAN5KLD53SNDX4XCSQJ5UJMYXHFGV7D3Y72RA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.873688093 (617h37m5.744771128s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.873947867 (617h37m5.745030912s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:07.557273878 (617h37m7.428356913s) [upload=541230]: Next returned nil segment Dec 31 09:06:07.557296371 (617h37m7.428379416s) [upload=541230]: Waiting for error group managing segments... Dec 31 09:06:07.869989232 (617h37m7.741072267s) [noise=true , address=145.79.227.36:4519 , piece_id=K4MKJ3TFQE33LJ6VMTUJJTHR725C4CUX4KW3PCD73XKPTRCKMNAA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.877879937 (617h37m7.748962972s) [noise=true , address=145.79.227.47:4266 , piece_id=PPOTI2TRBUR3Y2VN23BEEHE5XHLXUOZLJZ6KWDEQAB3RTKNSNZ5A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.887580157 (617h37m7.758663203s) [noise=true , address=145.79.227.41:4086 , piece_id=RMA25UH4U6DVQLJWQLSVUVEJXO6NCU43PTIXMZTUAQAURWER47GA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894258915 (617h37m7.76534196s) [noise=true , address=145.79.227.38:4978 , piece_id=C343XQRJMHQ3O2F3UN7H26HYJ3ZA7FSE2REO6KJD734PD5YFPCAQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894331724 (617h37m7.765414759s) [noise=true , address=145.79.227.5:4222 , piece_id=MWRH3O4FG5ICW7JONZ4FYMKKIIXSYJZ5WPAWDH5R3OPRXNLDCCVA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894725193 (617h37m7.765808238s) [noise=true , address=145.79.227.39:4014 , piece_id=FXY36EU2IXZXBX2SPU3GP6N3O25EFKV6PDJTYSIIIW77G6UAEKFA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.933008700 (617h37m7.804091735s) [noise=true , address=145.79.227.16:4911 , piece_id=4VJTSRLYNOFOSVS3QOUROMUTEUMFW5DJ6ORRUSFSIIXORITHIWEQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.945354127 (617h37m7.816437172s) [noise=true , address=145.79.227.32:4834 , piece_id=ALGY4CXOJUYNK6BTWSW4O7JPC3RSUOF62M2DEWSXYUD5YX6ND7HA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.037141884 (617h37m7.908224929s) [noise=true , address=77.90.31.19:4842 , piece_id=LTE74CWDYFDNZIMZT4T5X4SSMSG27HYTV6ECSXM3E4MDLTY3R5CQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.105167495 (617h37m7.97625054s) [noise=true , address=77.90.31.11:4555 , piece_id=DUR2NOB43B572QCLAN5KLD53SNDX4XCSQJ5UJMYXHFGV7D3Y72RA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.156289979 (617h37m8.027373024s) [noise=true , address=77.90.31.13:4627 , piece_id=QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.174675076 (617h37m8.045758111s) [noise=true , address=77.90.31.18:4805 , piece_id=5KOIVSWCCKOZCNXE5XLPHXBKXKATG3CWX52UM7YVYR4F5ONMH67A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.188351619 (617h37m8.059434664s) [noise=true , address=45.91.116.6:4975 , piece_id=BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.188388489 (617h37m8.059471534s) [seg_pos={0 0}, upload=541230]: Segment reached optimal threshold of 13 pieces. Dec 31 09:06:08.188621363 (617h37m8.059704398s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188628386 (617h37m8.059711431s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188762321 (617h37m8.059845356s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188778482 (617h37m8.059861517s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188804071 (617h37m8.059887106s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188811945 (617h37m8.05989498s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188858845 (617h37m8.05994188s) [seg_pos={0 0}, upload=541230]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 09:06:08.189175598 (617h37m8.060258633s) [upload=541230]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:06:08.189182111 (617h37m8.060265146s) [upload=541230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:06:08.483598230 (617h37m8.354681275s) [upload=541230]: Done uploading segments... Dec 31 09:06:08.483622396 (617h37m8.354705431s) [upload=541230]: Upload finished. err: <nil> Duration: 7.740883288s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [............X...........X....X....XXX.XX.............XX.....] duration: 3.473391719s trace: https://jaeger.dev.storj.io/trace/b3fe148e36f5660 FAIL (2025-12-31 09:13:55 UTC) orbiter warning: context deadline exceeded Duration: 3.473415887s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 26.187789814s trace: https://jaeger.dev.storj.io/trace/26257de5bcb7c920 FAIL (2025-12-31 09:13:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== , upload=540795]: Uploading piece... Dec 31 08:35:48.315166374 (617h6m48.186249409s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.315823664 (617h6m48.186906699s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316244806 (617h6m48.187327841s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316251710 (617h6m48.187334755s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316290584 (617h6m48.187373619s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317050111 (617h6m48.188133146s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317140853 (617h6m48.188223888s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318123574 (617h6m48.189206609s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318885204 (617h6m48.189968249s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.319835805 (617h6m48.19091885s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.320309847 (617h6m48.191392882s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321148273 (617h6m48.192231308s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321502088 (617h6m48.192585133s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322658430 (617h6m48.193741495s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322775803 (617h6m48.193858848s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322882035 (617h6m48.19396507s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.323403308 (617h6m48.194486343s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:49.768189393 (617h6m49.639272428s) [upload=540795]: Next returned nil segment Dec 31 08:35:49.768201797 (617h6m49.639284832s) [upload=540795]: Waiting for error group managing segments... Dec 31 08:35:50.197657834 (617h6m50.068740869s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.209847924 (617h6m50.08093096s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.217381621 (617h6m50.088464666s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.270225653 (617h6m50.141308698s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.411342630 (617h6m50.282425665s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473471693 (617h6m50.344554738s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473607101 (617h6m50.344690136s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.479010038 (617h6m50.350093074s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.482534590 (617h6m50.353617625s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.485954982 (617h6m50.357038017s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.058742323 (617h6m50.929825358s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.071853740 (617h6m50.942936775s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.177856875 (617h6m51.04893991s) [noise=true , address=145.79.227.33:4873 , piece_id=KWLS4DUDNAEADPUA3CTATKK4MEQSUPEIPYJS3SSOGVPDOJPJHDIQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179660939 (617h6m51.050743984s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179686989 (617h6m51.050770024s) [seg_pos={0 0}, upload=540795]: Segment reached optimal threshold of 14 pieces. Dec 31 08:35:51.179944619 (617h6m51.051027684s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.179956683 (617h6m51.051039728s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180323110 (617h6m51.051406155s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180346325 (617h6m51.05142937s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180485580 (617h6m51.051568615s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180495899 (617h6m51.051578934s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180529313 (617h6m51.051612358s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180536927 (617h6m51.051619972s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180560653 (617h6m51.051643698s) [seg_pos={0 0}, upload=540795]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 08:35:51.180641827 (617h6m51.051724872s) [upload=540795]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 08:35:51.180647598 (617h6m51.051730643s) [upload=540795]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 08:35:51.479648850 (617h6m51.350731896s) [upload=540795]: Done uploading segments... Dec 31 08:35:51.479871193 (617h6m51.350954228s) [upload=540795]: Upload finished. err: <nil> Duration: 26.187818328s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..X....X.X....X...XXX..X.X.....X........XX....X..XX.........] duration: 5.807822145s trace: https://jaeger.dev.storj.io/trace/354579860c475801 FAIL (2025-12-31 09:13:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA) context deadline exceeded ======LOGS====== Dec 31 09:06:05.623627963 (617h37m5.494710998s) [upload=541230]: Starting upload Dec 31 09:06:05.623687576 (617h37m5.494770611s) [upload=541230]: Done starting upload Dec 31 09:06:05.623797315 (617h37m5.49488035s) [upload=541230]: Uploading segments... Dec 31 09:06:05.623963341 (617h37m5.495046377s) [upload=541230]: Got next segment. Inline: false Dec 31 09:06:05.624003378 (617h37m5.495086423s) [upload=541230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 09:06:05.624008136 (617h37m5.495091172s) [upload=541230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 09:06:05.867160403 (617h37m5.738243438s) [seg_pos={0 0}, upload=541230]: Begin upload segment... Dec 31 09:06:05.867217431 (617h37m5.738300466s) [seg_pos={0 0}, upload=541230]: Done begin upload segment. Dec 31 09:06:05.867253921 (617h37m5.738336956s) [noise=true , address=145.79.227.5:4222 , piece_id=MWRH3O4FG5ICW7JONZ4FYMKKIIXSYJZ5WPAWDH5R3OPRXNLDCCVA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.867292885 (617h37m5.73837592s) [noise=true , address=145.79.227.16:4911 , piece_id=4VJTSRLYNOFOSVS3QOUROMUTEUMFW5DJ6ORRUSFSIIXORITHIWEQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.867729256 (617h37m5.738812301s) [noise=true , address=145.79.227.39:4014 , piece_id=FXY36EU2IXZXBX2SPU3GP6N3O25EFKV6PDJTYSIIIW77G6UAEKFA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.868133556 (617h37m5.739216591s) [noise=true , address=145.79.227.47:4266 , piece_id=PPOTI2TRBUR3Y2VN23BEEHE5XHLXUOZLJZ6KWDEQAB3RTKNSNZ5A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.868533548 (617h37m5.739616593s) [noise=true , address=77.90.31.13:4627 , piece_id=QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869018351 (617h37m5.740101386s) [noise=true , address=45.91.116.6:4975 , piece_id=BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869464290 (617h37m5.740547325s) [noise=true , address=77.90.31.19:4842 , piece_id=LTE74CWDYFDNZIMZT4T5X4SSMSG27HYTV6ECSXM3E4MDLTY3R5CQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869870603 (617h37m5.740953638s) [noise=true , address=77.90.31.18:4805 , piece_id=5KOIVSWCCKOZCNXE5XLPHXBKXKATG3CWX52UM7YVYR4F5ONMH67A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.870347171 (617h37m5.741430216s) [noise=true , address=145.79.227.38:4978 , piece_id=C343XQRJMHQ3O2F3UN7H26HYJ3ZA7FSE2REO6KJD734PD5YFPCAQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.870799663 (617h37m5.741882698s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.871259889 (617h37m5.742342924s) [noise=true , address=145.79.227.41:4086 , piece_id=RMA25UH4U6DVQLJWQLSVUVEJXO6NCU43PTIXMZTUAQAURWER47GA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.871817150 (617h37m5.742900195s) [noise=true , address=145.79.227.32:4834 , piece_id=ALGY4CXOJUYNK6BTWSW4O7JPC3RSUOF62M2DEWSXYUD5YX6ND7HA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.872641620 (617h37m5.743724665s) [noise=true , address=145.79.227.36:4519 , piece_id=K4MKJ3TFQE33LJ6VMTUJJTHR725C4CUX4KW3PCD73XKPTRCKMNAA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.872695763 (617h37m5.743778808s) [noise=true , address=77.90.31.11:4555 , piece_id=DUR2NOB43B572QCLAN5KLD53SNDX4XCSQJ5UJMYXHFGV7D3Y72RA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.873688093 (617h37m5.744771128s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.873947867 (617h37m5.745030912s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:07.557273878 (617h37m7.428356913s) [upload=541230]: Next returned nil segment Dec 31 09:06:07.557296371 (617h37m7.428379416s) [upload=541230]: Waiting for error group managing segments... Dec 31 09:06:07.869989232 (617h37m7.741072267s) [noise=true , address=145.79.227.36:4519 , piece_id=K4MKJ3TFQE33LJ6VMTUJJTHR725C4CUX4KW3PCD73XKPTRCKMNAA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.877879937 (617h37m7.748962972s) [noise=true , address=145.79.227.47:4266 , piece_id=PPOTI2TRBUR3Y2VN23BEEHE5XHLXUOZLJZ6KWDEQAB3RTKNSNZ5A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.887580157 (617h37m7.758663203s) [noise=true , address=145.79.227.41:4086 , piece_id=RMA25UH4U6DVQLJWQLSVUVEJXO6NCU43PTIXMZTUAQAURWER47GA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894258915 (617h37m7.76534196s) [noise=true , address=145.79.227.38:4978 , piece_id=C343XQRJMHQ3O2F3UN7H26HYJ3ZA7FSE2REO6KJD734PD5YFPCAQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894331724 (617h37m7.765414759s) [noise=true , address=145.79.227.5:4222 , piece_id=MWRH3O4FG5ICW7JONZ4FYMKKIIXSYJZ5WPAWDH5R3OPRXNLDCCVA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894725193 (617h37m7.765808238s) [noise=true , address=145.79.227.39:4014 , piece_id=FXY36EU2IXZXBX2SPU3GP6N3O25EFKV6PDJTYSIIIW77G6UAEKFA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.933008700 (617h37m7.804091735s) [noise=true , address=145.79.227.16:4911 , piece_id=4VJTSRLYNOFOSVS3QOUROMUTEUMFW5DJ6ORRUSFSIIXORITHIWEQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.945354127 (617h37m7.816437172s) [noise=true , address=145.79.227.32:4834 , piece_id=ALGY4CXOJUYNK6BTWSW4O7JPC3RSUOF62M2DEWSXYUD5YX6ND7HA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.037141884 (617h37m7.908224929s) [noise=true , address=77.90.31.19:4842 , piece_id=LTE74CWDYFDNZIMZT4T5X4SSMSG27HYTV6ECSXM3E4MDLTY3R5CQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.105167495 (617h37m7.97625054s) [noise=true , address=77.90.31.11:4555 , piece_id=DUR2NOB43B572QCLAN5KLD53SNDX4XCSQJ5UJMYXHFGV7D3Y72RA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.156289979 (617h37m8.027373024s) [noise=true , address=77.90.31.13:4627 , piece_id=QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.174675076 (617h37m8.045758111s) [noise=true , address=77.90.31.18:4805 , piece_id=5KOIVSWCCKOZCNXE5XLPHXBKXKATG3CWX52UM7YVYR4F5ONMH67A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.188351619 (617h37m8.059434664s) [noise=true , address=45.91.116.6:4975 , piece_id=BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.188388489 (617h37m8.059471534s) [seg_pos={0 0}, upload=541230]: Segment reached optimal threshold of 13 pieces. Dec 31 09:06:08.188621363 (617h37m8.059704398s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188628386 (617h37m8.059711431s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188762321 (617h37m8.059845356s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188778482 (617h37m8.059861517s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188804071 (617h37m8.059887106s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188811945 (617h37m8.05989498s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188858845 (617h37m8.05994188s) [seg_pos={0 0}, upload=541230]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 09:06:08.189175598 (617h37m8.060258633s) [upload=541230]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:06:08.189182111 (617h37m8.060265146s) [upload=541230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:06:08.483598230 (617h37m8.354681275s) [upload=541230]: Done uploading segments... Dec 31 09:06:08.483622396 (617h37m8.354705431s) [upload=541230]: Upload finished. err: <nil> Duration: 5.80784129s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 18.927945932s trace: https://jaeger.dev.storj.io/trace/26257de5bcb7c920 FAIL (2025-12-31 09:13:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== , upload=540795]: Uploading piece... Dec 31 08:35:48.315166374 (617h6m48.186249409s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.315823664 (617h6m48.186906699s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316244806 (617h6m48.187327841s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316251710 (617h6m48.187334755s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316290584 (617h6m48.187373619s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317050111 (617h6m48.188133146s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317140853 (617h6m48.188223888s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318123574 (617h6m48.189206609s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318885204 (617h6m48.189968249s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.319835805 (617h6m48.19091885s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.320309847 (617h6m48.191392882s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321148273 (617h6m48.192231308s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321502088 (617h6m48.192585133s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322658430 (617h6m48.193741495s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322775803 (617h6m48.193858848s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322882035 (617h6m48.19396507s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.323403308 (617h6m48.194486343s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:49.768189393 (617h6m49.639272428s) [upload=540795]: Next returned nil segment Dec 31 08:35:49.768201797 (617h6m49.639284832s) [upload=540795]: Waiting for error group managing segments... Dec 31 08:35:50.197657834 (617h6m50.068740869s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.209847924 (617h6m50.08093096s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.217381621 (617h6m50.088464666s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.270225653 (617h6m50.141308698s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.411342630 (617h6m50.282425665s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473471693 (617h6m50.344554738s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473607101 (617h6m50.344690136s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.479010038 (617h6m50.350093074s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.482534590 (617h6m50.353617625s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.485954982 (617h6m50.357038017s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.058742323 (617h6m50.929825358s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.071853740 (617h6m50.942936775s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.177856875 (617h6m51.04893991s) [noise=true , address=145.79.227.33:4873 , piece_id=KWLS4DUDNAEADPUA3CTATKK4MEQSUPEIPYJS3SSOGVPDOJPJHDIQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179660939 (617h6m51.050743984s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179686989 (617h6m51.050770024s) [seg_pos={0 0}, upload=540795]: Segment reached optimal threshold of 14 pieces. Dec 31 08:35:51.179944619 (617h6m51.051027684s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.179956683 (617h6m51.051039728s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180323110 (617h6m51.051406155s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180346325 (617h6m51.05142937s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180485580 (617h6m51.051568615s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180495899 (617h6m51.051578934s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180529313 (617h6m51.051612358s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180536927 (617h6m51.051619972s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180560653 (617h6m51.051643698s) [seg_pos={0 0}, upload=540795]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 08:35:51.180641827 (617h6m51.051724872s) [upload=540795]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 08:35:51.180647598 (617h6m51.051730643s) [upload=540795]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 08:35:51.479648850 (617h6m51.350731896s) [upload=540795]: Done uploading segments... Dec 31 08:35:51.479871193 (617h6m51.350954228s) [upload=540795]: Upload finished. err: <nil> Duration: 18.927986218s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.............X...........X....X....XXX.XX.............XX....] duration: 4.117603648s trace: https://jaeger.dev.storj.io/trace/b3fe148e36f5660 FAIL (2025-12-31 09:12:54 UTC) orbiter warning: context deadline exceeded Duration: 4.117628256s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...X....X.X....X...XXX..X.X.....X........XX....X..XX........] duration: 5.888257363s trace: https://jaeger.dev.storj.io/trace/354579860c475801 FAIL (2025-12-31 09:12:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA) context deadline exceeded ======LOGS====== Dec 31 09:06:05.623627963 (617h37m5.494710998s) [upload=541230]: Starting upload Dec 31 09:06:05.623687576 (617h37m5.494770611s) [upload=541230]: Done starting upload Dec 31 09:06:05.623797315 (617h37m5.49488035s) [upload=541230]: Uploading segments... Dec 31 09:06:05.623963341 (617h37m5.495046377s) [upload=541230]: Got next segment. Inline: false Dec 31 09:06:05.624003378 (617h37m5.495086423s) [upload=541230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 09:06:05.624008136 (617h37m5.495091172s) [upload=541230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 09:06:05.867160403 (617h37m5.738243438s) [seg_pos={0 0}, upload=541230]: Begin upload segment... Dec 31 09:06:05.867217431 (617h37m5.738300466s) [seg_pos={0 0}, upload=541230]: Done begin upload segment. Dec 31 09:06:05.867253921 (617h37m5.738336956s) [noise=true , address=145.79.227.5:4222 , piece_id=MWRH3O4FG5ICW7JONZ4FYMKKIIXSYJZ5WPAWDH5R3OPRXNLDCCVA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.867292885 (617h37m5.73837592s) [noise=true , address=145.79.227.16:4911 , piece_id=4VJTSRLYNOFOSVS3QOUROMUTEUMFW5DJ6ORRUSFSIIXORITHIWEQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.867729256 (617h37m5.738812301s) [noise=true , address=145.79.227.39:4014 , piece_id=FXY36EU2IXZXBX2SPU3GP6N3O25EFKV6PDJTYSIIIW77G6UAEKFA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.868133556 (617h37m5.739216591s) [noise=true , address=145.79.227.47:4266 , piece_id=PPOTI2TRBUR3Y2VN23BEEHE5XHLXUOZLJZ6KWDEQAB3RTKNSNZ5A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.868533548 (617h37m5.739616593s) [noise=true , address=77.90.31.13:4627 , piece_id=QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869018351 (617h37m5.740101386s) [noise=true , address=45.91.116.6:4975 , piece_id=BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869464290 (617h37m5.740547325s) [noise=true , address=77.90.31.19:4842 , piece_id=LTE74CWDYFDNZIMZT4T5X4SSMSG27HYTV6ECSXM3E4MDLTY3R5CQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869870603 (617h37m5.740953638s) [noise=true , address=77.90.31.18:4805 , piece_id=5KOIVSWCCKOZCNXE5XLPHXBKXKATG3CWX52UM7YVYR4F5ONMH67A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.870347171 (617h37m5.741430216s) [noise=true , address=145.79.227.38:4978 , piece_id=C343XQRJMHQ3O2F3UN7H26HYJ3ZA7FSE2REO6KJD734PD5YFPCAQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.870799663 (617h37m5.741882698s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.871259889 (617h37m5.742342924s) [noise=true , address=145.79.227.41:4086 , piece_id=RMA25UH4U6DVQLJWQLSVUVEJXO6NCU43PTIXMZTUAQAURWER47GA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.871817150 (617h37m5.742900195s) [noise=true , address=145.79.227.32:4834 , piece_id=ALGY4CXOJUYNK6BTWSW4O7JPC3RSUOF62M2DEWSXYUD5YX6ND7HA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.872641620 (617h37m5.743724665s) [noise=true , address=145.79.227.36:4519 , piece_id=K4MKJ3TFQE33LJ6VMTUJJTHR725C4CUX4KW3PCD73XKPTRCKMNAA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.872695763 (617h37m5.743778808s) [noise=true , address=77.90.31.11:4555 , piece_id=DUR2NOB43B572QCLAN5KLD53SNDX4XCSQJ5UJMYXHFGV7D3Y72RA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.873688093 (617h37m5.744771128s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.873947867 (617h37m5.745030912s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:07.557273878 (617h37m7.428356913s) [upload=541230]: Next returned nil segment Dec 31 09:06:07.557296371 (617h37m7.428379416s) [upload=541230]: Waiting for error group managing segments... Dec 31 09:06:07.869989232 (617h37m7.741072267s) [noise=true , address=145.79.227.36:4519 , piece_id=K4MKJ3TFQE33LJ6VMTUJJTHR725C4CUX4KW3PCD73XKPTRCKMNAA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.877879937 (617h37m7.748962972s) [noise=true , address=145.79.227.47:4266 , piece_id=PPOTI2TRBUR3Y2VN23BEEHE5XHLXUOZLJZ6KWDEQAB3RTKNSNZ5A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.887580157 (617h37m7.758663203s) [noise=true , address=145.79.227.41:4086 , piece_id=RMA25UH4U6DVQLJWQLSVUVEJXO6NCU43PTIXMZTUAQAURWER47GA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894258915 (617h37m7.76534196s) [noise=true , address=145.79.227.38:4978 , piece_id=C343XQRJMHQ3O2F3UN7H26HYJ3ZA7FSE2REO6KJD734PD5YFPCAQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894331724 (617h37m7.765414759s) [noise=true , address=145.79.227.5:4222 , piece_id=MWRH3O4FG5ICW7JONZ4FYMKKIIXSYJZ5WPAWDH5R3OPRXNLDCCVA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894725193 (617h37m7.765808238s) [noise=true , address=145.79.227.39:4014 , piece_id=FXY36EU2IXZXBX2SPU3GP6N3O25EFKV6PDJTYSIIIW77G6UAEKFA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.933008700 (617h37m7.804091735s) [noise=true , address=145.79.227.16:4911 , piece_id=4VJTSRLYNOFOSVS3QOUROMUTEUMFW5DJ6ORRUSFSIIXORITHIWEQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.945354127 (617h37m7.816437172s) [noise=true , address=145.79.227.32:4834 , piece_id=ALGY4CXOJUYNK6BTWSW4O7JPC3RSUOF62M2DEWSXYUD5YX6ND7HA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.037141884 (617h37m7.908224929s) [noise=true , address=77.90.31.19:4842 , piece_id=LTE74CWDYFDNZIMZT4T5X4SSMSG27HYTV6ECSXM3E4MDLTY3R5CQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.105167495 (617h37m7.97625054s) [noise=true , address=77.90.31.11:4555 , piece_id=DUR2NOB43B572QCLAN5KLD53SNDX4XCSQJ5UJMYXHFGV7D3Y72RA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.156289979 (617h37m8.027373024s) [noise=true , address=77.90.31.13:4627 , piece_id=QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.174675076 (617h37m8.045758111s) [noise=true , address=77.90.31.18:4805 , piece_id=5KOIVSWCCKOZCNXE5XLPHXBKXKATG3CWX52UM7YVYR4F5ONMH67A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.188351619 (617h37m8.059434664s) [noise=true , address=45.91.116.6:4975 , piece_id=BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.188388489 (617h37m8.059471534s) [seg_pos={0 0}, upload=541230]: Segment reached optimal threshold of 13 pieces. Dec 31 09:06:08.188621363 (617h37m8.059704398s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188628386 (617h37m8.059711431s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188762321 (617h37m8.059845356s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188778482 (617h37m8.059861517s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188804071 (617h37m8.059887106s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188811945 (617h37m8.05989498s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188858845 (617h37m8.05994188s) [seg_pos={0 0}, upload=541230]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 09:06:08.189175598 (617h37m8.060258633s) [upload=541230]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:06:08.189182111 (617h37m8.060265146s) [upload=541230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:06:08.483598230 (617h37m8.354681275s) [upload=541230]: Done uploading segments... Dec 31 09:06:08.483622396 (617h37m8.354705431s) [upload=541230]: Upload finished. err: <nil> Duration: 5.888287461s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 35.569902748s trace: https://jaeger.dev.storj.io/trace/26257de5bcb7c920 FAIL (2025-12-31 09:12:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== , upload=540795]: Uploading piece... Dec 31 08:35:48.315166374 (617h6m48.186249409s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.315823664 (617h6m48.186906699s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316244806 (617h6m48.187327841s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316251710 (617h6m48.187334755s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316290584 (617h6m48.187373619s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317050111 (617h6m48.188133146s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317140853 (617h6m48.188223888s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318123574 (617h6m48.189206609s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318885204 (617h6m48.189968249s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.319835805 (617h6m48.19091885s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.320309847 (617h6m48.191392882s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321148273 (617h6m48.192231308s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321502088 (617h6m48.192585133s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322658430 (617h6m48.193741495s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322775803 (617h6m48.193858848s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322882035 (617h6m48.19396507s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.323403308 (617h6m48.194486343s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:49.768189393 (617h6m49.639272428s) [upload=540795]: Next returned nil segment Dec 31 08:35:49.768201797 (617h6m49.639284832s) [upload=540795]: Waiting for error group managing segments... Dec 31 08:35:50.197657834 (617h6m50.068740869s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.209847924 (617h6m50.08093096s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.217381621 (617h6m50.088464666s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.270225653 (617h6m50.141308698s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.411342630 (617h6m50.282425665s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473471693 (617h6m50.344554738s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473607101 (617h6m50.344690136s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.479010038 (617h6m50.350093074s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.482534590 (617h6m50.353617625s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.485954982 (617h6m50.357038017s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.058742323 (617h6m50.929825358s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.071853740 (617h6m50.942936775s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.177856875 (617h6m51.04893991s) [noise=true , address=145.79.227.33:4873 , piece_id=KWLS4DUDNAEADPUA3CTATKK4MEQSUPEIPYJS3SSOGVPDOJPJHDIQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179660939 (617h6m51.050743984s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179686989 (617h6m51.050770024s) [seg_pos={0 0}, upload=540795]: Segment reached optimal threshold of 14 pieces. Dec 31 08:35:51.179944619 (617h6m51.051027684s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.179956683 (617h6m51.051039728s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180323110 (617h6m51.051406155s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180346325 (617h6m51.05142937s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180485580 (617h6m51.051568615s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180495899 (617h6m51.051578934s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180529313 (617h6m51.051612358s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180536927 (617h6m51.051619972s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180560653 (617h6m51.051643698s) [seg_pos={0 0}, upload=540795]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 08:35:51.180641827 (617h6m51.051724872s) [upload=540795]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 08:35:51.180647598 (617h6m51.051730643s) [upload=540795]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 08:35:51.479648850 (617h6m51.350731896s) [upload=540795]: Done uploading segments... Dec 31 08:35:51.479871193 (617h6m51.350954228s) [upload=540795]: Upload finished. err: <nil> Duration: 35.56992007s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [..............X...........X....X....XXX.XX.............XX...] duration: 3.742654472s trace: https://jaeger.dev.storj.io/trace/b3fe148e36f5660 FAIL (2025-12-31 09:11:52 UTC) orbiter warning: context deadline exceeded Duration: 3.742678458s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [....X....X.X....X...XXX..X.X.....X........XX....X..XX.......] duration: 5.991809644s trace: https://jaeger.dev.storj.io/trace/354579860c475801 FAIL (2025-12-31 09:11:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA) context deadline exceeded ======LOGS====== Dec 31 09:06:05.623627963 (617h37m5.494710998s) [upload=541230]: Starting upload Dec 31 09:06:05.623687576 (617h37m5.494770611s) [upload=541230]: Done starting upload Dec 31 09:06:05.623797315 (617h37m5.49488035s) [upload=541230]: Uploading segments... Dec 31 09:06:05.623963341 (617h37m5.495046377s) [upload=541230]: Got next segment. Inline: false Dec 31 09:06:05.624003378 (617h37m5.495086423s) [upload=541230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 09:06:05.624008136 (617h37m5.495091172s) [upload=541230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 09:06:05.867160403 (617h37m5.738243438s) [seg_pos={0 0}, upload=541230]: Begin upload segment... Dec 31 09:06:05.867217431 (617h37m5.738300466s) [seg_pos={0 0}, upload=541230]: Done begin upload segment. Dec 31 09:06:05.867253921 (617h37m5.738336956s) [noise=true , address=145.79.227.5:4222 , piece_id=MWRH3O4FG5ICW7JONZ4FYMKKIIXSYJZ5WPAWDH5R3OPRXNLDCCVA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.867292885 (617h37m5.73837592s) [noise=true , address=145.79.227.16:4911 , piece_id=4VJTSRLYNOFOSVS3QOUROMUTEUMFW5DJ6ORRUSFSIIXORITHIWEQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.867729256 (617h37m5.738812301s) [noise=true , address=145.79.227.39:4014 , piece_id=FXY36EU2IXZXBX2SPU3GP6N3O25EFKV6PDJTYSIIIW77G6UAEKFA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.868133556 (617h37m5.739216591s) [noise=true , address=145.79.227.47:4266 , piece_id=PPOTI2TRBUR3Y2VN23BEEHE5XHLXUOZLJZ6KWDEQAB3RTKNSNZ5A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.868533548 (617h37m5.739616593s) [noise=true , address=77.90.31.13:4627 , piece_id=QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869018351 (617h37m5.740101386s) [noise=true , address=45.91.116.6:4975 , piece_id=BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869464290 (617h37m5.740547325s) [noise=true , address=77.90.31.19:4842 , piece_id=LTE74CWDYFDNZIMZT4T5X4SSMSG27HYTV6ECSXM3E4MDLTY3R5CQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869870603 (617h37m5.740953638s) [noise=true , address=77.90.31.18:4805 , piece_id=5KOIVSWCCKOZCNXE5XLPHXBKXKATG3CWX52UM7YVYR4F5ONMH67A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.870347171 (617h37m5.741430216s) [noise=true , address=145.79.227.38:4978 , piece_id=C343XQRJMHQ3O2F3UN7H26HYJ3ZA7FSE2REO6KJD734PD5YFPCAQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.870799663 (617h37m5.741882698s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.871259889 (617h37m5.742342924s) [noise=true , address=145.79.227.41:4086 , piece_id=RMA25UH4U6DVQLJWQLSVUVEJXO6NCU43PTIXMZTUAQAURWER47GA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.871817150 (617h37m5.742900195s) [noise=true , address=145.79.227.32:4834 , piece_id=ALGY4CXOJUYNK6BTWSW4O7JPC3RSUOF62M2DEWSXYUD5YX6ND7HA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.872641620 (617h37m5.743724665s) [noise=true , address=145.79.227.36:4519 , piece_id=K4MKJ3TFQE33LJ6VMTUJJTHR725C4CUX4KW3PCD73XKPTRCKMNAA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.872695763 (617h37m5.743778808s) [noise=true , address=77.90.31.11:4555 , piece_id=DUR2NOB43B572QCLAN5KLD53SNDX4XCSQJ5UJMYXHFGV7D3Y72RA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.873688093 (617h37m5.744771128s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.873947867 (617h37m5.745030912s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:07.557273878 (617h37m7.428356913s) [upload=541230]: Next returned nil segment Dec 31 09:06:07.557296371 (617h37m7.428379416s) [upload=541230]: Waiting for error group managing segments... Dec 31 09:06:07.869989232 (617h37m7.741072267s) [noise=true , address=145.79.227.36:4519 , piece_id=K4MKJ3TFQE33LJ6VMTUJJTHR725C4CUX4KW3PCD73XKPTRCKMNAA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.877879937 (617h37m7.748962972s) [noise=true , address=145.79.227.47:4266 , piece_id=PPOTI2TRBUR3Y2VN23BEEHE5XHLXUOZLJZ6KWDEQAB3RTKNSNZ5A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.887580157 (617h37m7.758663203s) [noise=true , address=145.79.227.41:4086 , piece_id=RMA25UH4U6DVQLJWQLSVUVEJXO6NCU43PTIXMZTUAQAURWER47GA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894258915 (617h37m7.76534196s) [noise=true , address=145.79.227.38:4978 , piece_id=C343XQRJMHQ3O2F3UN7H26HYJ3ZA7FSE2REO6KJD734PD5YFPCAQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894331724 (617h37m7.765414759s) [noise=true , address=145.79.227.5:4222 , piece_id=MWRH3O4FG5ICW7JONZ4FYMKKIIXSYJZ5WPAWDH5R3OPRXNLDCCVA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894725193 (617h37m7.765808238s) [noise=true , address=145.79.227.39:4014 , piece_id=FXY36EU2IXZXBX2SPU3GP6N3O25EFKV6PDJTYSIIIW77G6UAEKFA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.933008700 (617h37m7.804091735s) [noise=true , address=145.79.227.16:4911 , piece_id=4VJTSRLYNOFOSVS3QOUROMUTEUMFW5DJ6ORRUSFSIIXORITHIWEQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.945354127 (617h37m7.816437172s) [noise=true , address=145.79.227.32:4834 , piece_id=ALGY4CXOJUYNK6BTWSW4O7JPC3RSUOF62M2DEWSXYUD5YX6ND7HA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.037141884 (617h37m7.908224929s) [noise=true , address=77.90.31.19:4842 , piece_id=LTE74CWDYFDNZIMZT4T5X4SSMSG27HYTV6ECSXM3E4MDLTY3R5CQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.105167495 (617h37m7.97625054s) [noise=true , address=77.90.31.11:4555 , piece_id=DUR2NOB43B572QCLAN5KLD53SNDX4XCSQJ5UJMYXHFGV7D3Y72RA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.156289979 (617h37m8.027373024s) [noise=true , address=77.90.31.13:4627 , piece_id=QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.174675076 (617h37m8.045758111s) [noise=true , address=77.90.31.18:4805 , piece_id=5KOIVSWCCKOZCNXE5XLPHXBKXKATG3CWX52UM7YVYR4F5ONMH67A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.188351619 (617h37m8.059434664s) [noise=true , address=45.91.116.6:4975 , piece_id=BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.188388489 (617h37m8.059471534s) [seg_pos={0 0}, upload=541230]: Segment reached optimal threshold of 13 pieces. Dec 31 09:06:08.188621363 (617h37m8.059704398s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188628386 (617h37m8.059711431s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188762321 (617h37m8.059845356s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188778482 (617h37m8.059861517s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188804071 (617h37m8.059887106s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188811945 (617h37m8.05989498s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188858845 (617h37m8.05994188s) [seg_pos={0 0}, upload=541230]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 09:06:08.189175598 (617h37m8.060258633s) [upload=541230]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:06:08.189182111 (617h37m8.060265146s) [upload=541230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:06:08.483598230 (617h37m8.354681275s) [upload=541230]: Done uploading segments... Dec 31 09:06:08.483622396 (617h37m8.354705431s) [upload=541230]: Upload finished. err: <nil> Duration: 5.991894354s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 15.656168388s trace: https://jaeger.dev.storj.io/trace/26257de5bcb7c920 FAIL (2025-12-31 09:10:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== , upload=540795]: Uploading piece... Dec 31 08:35:48.315166374 (617h6m48.186249409s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.315823664 (617h6m48.186906699s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316244806 (617h6m48.187327841s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316251710 (617h6m48.187334755s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316290584 (617h6m48.187373619s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317050111 (617h6m48.188133146s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317140853 (617h6m48.188223888s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318123574 (617h6m48.189206609s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318885204 (617h6m48.189968249s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.319835805 (617h6m48.19091885s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.320309847 (617h6m48.191392882s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321148273 (617h6m48.192231308s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321502088 (617h6m48.192585133s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322658430 (617h6m48.193741495s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322775803 (617h6m48.193858848s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322882035 (617h6m48.19396507s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.323403308 (617h6m48.194486343s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:49.768189393 (617h6m49.639272428s) [upload=540795]: Next returned nil segment Dec 31 08:35:49.768201797 (617h6m49.639284832s) [upload=540795]: Waiting for error group managing segments... Dec 31 08:35:50.197657834 (617h6m50.068740869s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.209847924 (617h6m50.08093096s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.217381621 (617h6m50.088464666s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.270225653 (617h6m50.141308698s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.411342630 (617h6m50.282425665s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473471693 (617h6m50.344554738s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473607101 (617h6m50.344690136s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.479010038 (617h6m50.350093074s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.482534590 (617h6m50.353617625s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.485954982 (617h6m50.357038017s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.058742323 (617h6m50.929825358s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.071853740 (617h6m50.942936775s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.177856875 (617h6m51.04893991s) [noise=true , address=145.79.227.33:4873 , piece_id=KWLS4DUDNAEADPUA3CTATKK4MEQSUPEIPYJS3SSOGVPDOJPJHDIQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179660939 (617h6m51.050743984s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179686989 (617h6m51.050770024s) [seg_pos={0 0}, upload=540795]: Segment reached optimal threshold of 14 pieces. Dec 31 08:35:51.179944619 (617h6m51.051027684s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.179956683 (617h6m51.051039728s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180323110 (617h6m51.051406155s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180346325 (617h6m51.05142937s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180485580 (617h6m51.051568615s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180495899 (617h6m51.051578934s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180529313 (617h6m51.051612358s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180536927 (617h6m51.051619972s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180560653 (617h6m51.051643698s) [seg_pos={0 0}, upload=540795]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 08:35:51.180641827 (617h6m51.051724872s) [upload=540795]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 08:35:51.180647598 (617h6m51.051730643s) [upload=540795]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 08:35:51.479648850 (617h6m51.350731896s) [upload=540795]: Done uploading segments... Dec 31 08:35:51.479871193 (617h6m51.350954228s) [upload=540795]: Upload finished. err: <nil> Duration: 15.656194557s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.....X....X.X....X...XXX..X.X.....X........XX....X..XX......] duration: 16.163933781s trace: https://jaeger.dev.storj.io/trace/354579860c475801 FAIL (2025-12-31 09:10:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA) context deadline exceeded ======LOGS====== Dec 31 09:06:05.623627963 (617h37m5.494710998s) [upload=541230]: Starting upload Dec 31 09:06:05.623687576 (617h37m5.494770611s) [upload=541230]: Done starting upload Dec 31 09:06:05.623797315 (617h37m5.49488035s) [upload=541230]: Uploading segments... Dec 31 09:06:05.623963341 (617h37m5.495046377s) [upload=541230]: Got next segment. Inline: false Dec 31 09:06:05.624003378 (617h37m5.495086423s) [upload=541230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 09:06:05.624008136 (617h37m5.495091172s) [upload=541230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 09:06:05.867160403 (617h37m5.738243438s) [seg_pos={0 0}, upload=541230]: Begin upload segment... Dec 31 09:06:05.867217431 (617h37m5.738300466s) [seg_pos={0 0}, upload=541230]: Done begin upload segment. Dec 31 09:06:05.867253921 (617h37m5.738336956s) [noise=true , address=145.79.227.5:4222 , piece_id=MWRH3O4FG5ICW7JONZ4FYMKKIIXSYJZ5WPAWDH5R3OPRXNLDCCVA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.867292885 (617h37m5.73837592s) [noise=true , address=145.79.227.16:4911 , piece_id=4VJTSRLYNOFOSVS3QOUROMUTEUMFW5DJ6ORRUSFSIIXORITHIWEQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.867729256 (617h37m5.738812301s) [noise=true , address=145.79.227.39:4014 , piece_id=FXY36EU2IXZXBX2SPU3GP6N3O25EFKV6PDJTYSIIIW77G6UAEKFA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.868133556 (617h37m5.739216591s) [noise=true , address=145.79.227.47:4266 , piece_id=PPOTI2TRBUR3Y2VN23BEEHE5XHLXUOZLJZ6KWDEQAB3RTKNSNZ5A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.868533548 (617h37m5.739616593s) [noise=true , address=77.90.31.13:4627 , piece_id=QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869018351 (617h37m5.740101386s) [noise=true , address=45.91.116.6:4975 , piece_id=BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869464290 (617h37m5.740547325s) [noise=true , address=77.90.31.19:4842 , piece_id=LTE74CWDYFDNZIMZT4T5X4SSMSG27HYTV6ECSXM3E4MDLTY3R5CQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869870603 (617h37m5.740953638s) [noise=true , address=77.90.31.18:4805 , piece_id=5KOIVSWCCKOZCNXE5XLPHXBKXKATG3CWX52UM7YVYR4F5ONMH67A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.870347171 (617h37m5.741430216s) [noise=true , address=145.79.227.38:4978 , piece_id=C343XQRJMHQ3O2F3UN7H26HYJ3ZA7FSE2REO6KJD734PD5YFPCAQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.870799663 (617h37m5.741882698s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.871259889 (617h37m5.742342924s) [noise=true , address=145.79.227.41:4086 , piece_id=RMA25UH4U6DVQLJWQLSVUVEJXO6NCU43PTIXMZTUAQAURWER47GA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.871817150 (617h37m5.742900195s) [noise=true , address=145.79.227.32:4834 , piece_id=ALGY4CXOJUYNK6BTWSW4O7JPC3RSUOF62M2DEWSXYUD5YX6ND7HA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.872641620 (617h37m5.743724665s) [noise=true , address=145.79.227.36:4519 , piece_id=K4MKJ3TFQE33LJ6VMTUJJTHR725C4CUX4KW3PCD73XKPTRCKMNAA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.872695763 (617h37m5.743778808s) [noise=true , address=77.90.31.11:4555 , piece_id=DUR2NOB43B572QCLAN5KLD53SNDX4XCSQJ5UJMYXHFGV7D3Y72RA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.873688093 (617h37m5.744771128s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.873947867 (617h37m5.745030912s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:07.557273878 (617h37m7.428356913s) [upload=541230]: Next returned nil segment Dec 31 09:06:07.557296371 (617h37m7.428379416s) [upload=541230]: Waiting for error group managing segments... Dec 31 09:06:07.869989232 (617h37m7.741072267s) [noise=true , address=145.79.227.36:4519 , piece_id=K4MKJ3TFQE33LJ6VMTUJJTHR725C4CUX4KW3PCD73XKPTRCKMNAA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.877879937 (617h37m7.748962972s) [noise=true , address=145.79.227.47:4266 , piece_id=PPOTI2TRBUR3Y2VN23BEEHE5XHLXUOZLJZ6KWDEQAB3RTKNSNZ5A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.887580157 (617h37m7.758663203s) [noise=true , address=145.79.227.41:4086 , piece_id=RMA25UH4U6DVQLJWQLSVUVEJXO6NCU43PTIXMZTUAQAURWER47GA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894258915 (617h37m7.76534196s) [noise=true , address=145.79.227.38:4978 , piece_id=C343XQRJMHQ3O2F3UN7H26HYJ3ZA7FSE2REO6KJD734PD5YFPCAQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894331724 (617h37m7.765414759s) [noise=true , address=145.79.227.5:4222 , piece_id=MWRH3O4FG5ICW7JONZ4FYMKKIIXSYJZ5WPAWDH5R3OPRXNLDCCVA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894725193 (617h37m7.765808238s) [noise=true , address=145.79.227.39:4014 , piece_id=FXY36EU2IXZXBX2SPU3GP6N3O25EFKV6PDJTYSIIIW77G6UAEKFA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.933008700 (617h37m7.804091735s) [noise=true , address=145.79.227.16:4911 , piece_id=4VJTSRLYNOFOSVS3QOUROMUTEUMFW5DJ6ORRUSFSIIXORITHIWEQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.945354127 (617h37m7.816437172s) [noise=true , address=145.79.227.32:4834 , piece_id=ALGY4CXOJUYNK6BTWSW4O7JPC3RSUOF62M2DEWSXYUD5YX6ND7HA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.037141884 (617h37m7.908224929s) [noise=true , address=77.90.31.19:4842 , piece_id=LTE74CWDYFDNZIMZT4T5X4SSMSG27HYTV6ECSXM3E4MDLTY3R5CQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.105167495 (617h37m7.97625054s) [noise=true , address=77.90.31.11:4555 , piece_id=DUR2NOB43B572QCLAN5KLD53SNDX4XCSQJ5UJMYXHFGV7D3Y72RA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.156289979 (617h37m8.027373024s) [noise=true , address=77.90.31.13:4627 , piece_id=QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.174675076 (617h37m8.045758111s) [noise=true , address=77.90.31.18:4805 , piece_id=5KOIVSWCCKOZCNXE5XLPHXBKXKATG3CWX52UM7YVYR4F5ONMH67A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.188351619 (617h37m8.059434664s) [noise=true , address=45.91.116.6:4975 , piece_id=BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.188388489 (617h37m8.059471534s) [seg_pos={0 0}, upload=541230]: Segment reached optimal threshold of 13 pieces. Dec 31 09:06:08.188621363 (617h37m8.059704398s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188628386 (617h37m8.059711431s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188762321 (617h37m8.059845356s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188778482 (617h37m8.059861517s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188804071 (617h37m8.059887106s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188811945 (617h37m8.05989498s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188858845 (617h37m8.05994188s) [seg_pos={0 0}, upload=541230]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 09:06:08.189175598 (617h37m8.060258633s) [upload=541230]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:06:08.189182111 (617h37m8.060265146s) [upload=541230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:06:08.483598230 (617h37m8.354681275s) [upload=541230]: Done uploading segments... Dec 31 09:06:08.483622396 (617h37m8.354705431s) [upload=541230]: Upload finished. err: <nil> Duration: 16.163961565s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [...............X...........X....X....XXX.XX.............XX..] duration: 3.498766235s trace: https://jaeger.dev.storj.io/trace/b3fe148e36f5660 FAIL (2025-12-31 09:10:46 UTC) orbiter warning: context deadline exceeded Duration: 3.498792966s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [................X...........X....X....XXX.XX.............XX.] duration: 3.406575502s trace: https://jaeger.dev.storj.io/trace/b3fe148e36f5660 FAIL (2025-12-31 09:10:31 UTC) orbiter warning: context deadline exceeded Duration: 3.406602754s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 28.020504599s trace: https://jaeger.dev.storj.io/trace/26257de5bcb7c920 FAIL (2025-12-31 09:10:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== , upload=540795]: Uploading piece... Dec 31 08:35:48.315166374 (617h6m48.186249409s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.315823664 (617h6m48.186906699s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316244806 (617h6m48.187327841s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316251710 (617h6m48.187334755s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316290584 (617h6m48.187373619s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317050111 (617h6m48.188133146s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317140853 (617h6m48.188223888s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318123574 (617h6m48.189206609s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318885204 (617h6m48.189968249s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.319835805 (617h6m48.19091885s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.320309847 (617h6m48.191392882s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321148273 (617h6m48.192231308s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321502088 (617h6m48.192585133s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322658430 (617h6m48.193741495s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322775803 (617h6m48.193858848s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322882035 (617h6m48.19396507s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.323403308 (617h6m48.194486343s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:49.768189393 (617h6m49.639272428s) [upload=540795]: Next returned nil segment Dec 31 08:35:49.768201797 (617h6m49.639284832s) [upload=540795]: Waiting for error group managing segments... Dec 31 08:35:50.197657834 (617h6m50.068740869s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.209847924 (617h6m50.08093096s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.217381621 (617h6m50.088464666s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.270225653 (617h6m50.141308698s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.411342630 (617h6m50.282425665s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473471693 (617h6m50.344554738s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473607101 (617h6m50.344690136s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.479010038 (617h6m50.350093074s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.482534590 (617h6m50.353617625s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.485954982 (617h6m50.357038017s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.058742323 (617h6m50.929825358s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.071853740 (617h6m50.942936775s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.177856875 (617h6m51.04893991s) [noise=true , address=145.79.227.33:4873 , piece_id=KWLS4DUDNAEADPUA3CTATKK4MEQSUPEIPYJS3SSOGVPDOJPJHDIQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179660939 (617h6m51.050743984s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179686989 (617h6m51.050770024s) [seg_pos={0 0}, upload=540795]: Segment reached optimal threshold of 14 pieces. Dec 31 08:35:51.179944619 (617h6m51.051027684s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.179956683 (617h6m51.051039728s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180323110 (617h6m51.051406155s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180346325 (617h6m51.05142937s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180485580 (617h6m51.051568615s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180495899 (617h6m51.051578934s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180529313 (617h6m51.051612358s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180536927 (617h6m51.051619972s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180560653 (617h6m51.051643698s) [seg_pos={0 0}, upload=540795]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 08:35:51.180641827 (617h6m51.051724872s) [upload=540795]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 08:35:51.180647598 (617h6m51.051730643s) [upload=540795]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 08:35:51.479648850 (617h6m51.350731896s) [upload=540795]: Done uploading segments... Dec 31 08:35:51.479871193 (617h6m51.350954228s) [upload=540795]: Upload finished. err: <nil> Duration: 28.020532894s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.................X...........X....X....XXX.XX.............XX] duration: 5.001887996s trace: https://jaeger.dev.storj.io/trace/b3fe148e36f5660 FAIL (2025-12-31 09:10:05 UTC) orbiter warning: context deadline exceeded Duration: 5.001914317s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 41.786093739s trace: https://jaeger.dev.storj.io/trace/26257de5bcb7c920 FAIL (2025-12-31 09:09:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== , upload=540795]: Uploading piece... Dec 31 08:35:48.315166374 (617h6m48.186249409s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.315823664 (617h6m48.186906699s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316244806 (617h6m48.187327841s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316251710 (617h6m48.187334755s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316290584 (617h6m48.187373619s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317050111 (617h6m48.188133146s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317140853 (617h6m48.188223888s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318123574 (617h6m48.189206609s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318885204 (617h6m48.189968249s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.319835805 (617h6m48.19091885s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.320309847 (617h6m48.191392882s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321148273 (617h6m48.192231308s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321502088 (617h6m48.192585133s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322658430 (617h6m48.193741495s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322775803 (617h6m48.193858848s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322882035 (617h6m48.19396507s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.323403308 (617h6m48.194486343s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:49.768189393 (617h6m49.639272428s) [upload=540795]: Next returned nil segment Dec 31 08:35:49.768201797 (617h6m49.639284832s) [upload=540795]: Waiting for error group managing segments... Dec 31 08:35:50.197657834 (617h6m50.068740869s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.209847924 (617h6m50.08093096s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.217381621 (617h6m50.088464666s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.270225653 (617h6m50.141308698s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.411342630 (617h6m50.282425665s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473471693 (617h6m50.344554738s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473607101 (617h6m50.344690136s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.479010038 (617h6m50.350093074s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.482534590 (617h6m50.353617625s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.485954982 (617h6m50.357038017s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.058742323 (617h6m50.929825358s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.071853740 (617h6m50.942936775s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.177856875 (617h6m51.04893991s) [noise=true , address=145.79.227.33:4873 , piece_id=KWLS4DUDNAEADPUA3CTATKK4MEQSUPEIPYJS3SSOGVPDOJPJHDIQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179660939 (617h6m51.050743984s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179686989 (617h6m51.050770024s) [seg_pos={0 0}, upload=540795]: Segment reached optimal threshold of 14 pieces. Dec 31 08:35:51.179944619 (617h6m51.051027684s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.179956683 (617h6m51.051039728s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180323110 (617h6m51.051406155s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180346325 (617h6m51.05142937s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180485580 (617h6m51.051568615s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180495899 (617h6m51.051578934s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180529313 (617h6m51.051612358s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180536927 (617h6m51.051619972s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180560653 (617h6m51.051643698s) [seg_pos={0 0}, upload=540795]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 08:35:51.180641827 (617h6m51.051724872s) [upload=540795]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 08:35:51.180647598 (617h6m51.051730643s) [upload=540795]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 08:35:51.479648850 (617h6m51.350731896s) [upload=540795]: Done uploading segments... Dec 31 08:35:51.479871193 (617h6m51.350954228s) [upload=540795]: Upload finished. err: <nil> Duration: 41.78612039s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [..................X...........X....X....XXX.XX.............X] duration: 5.001301236s trace: https://jaeger.dev.storj.io/trace/6ea5a3b91b74d76e FAIL (2025-12-31 09:09:40 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001321916s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [......X....X.X....X...XXX..X.X.....X........XX....X..XX.....] duration: 5.972286661s trace: https://jaeger.dev.storj.io/trace/354579860c475801 FAIL (2025-12-31 09:09:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA) context deadline exceeded ======LOGS====== Dec 31 09:06:05.623627963 (617h37m5.494710998s) [upload=541230]: Starting upload Dec 31 09:06:05.623687576 (617h37m5.494770611s) [upload=541230]: Done starting upload Dec 31 09:06:05.623797315 (617h37m5.49488035s) [upload=541230]: Uploading segments... Dec 31 09:06:05.623963341 (617h37m5.495046377s) [upload=541230]: Got next segment. Inline: false Dec 31 09:06:05.624003378 (617h37m5.495086423s) [upload=541230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 09:06:05.624008136 (617h37m5.495091172s) [upload=541230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 09:06:05.867160403 (617h37m5.738243438s) [seg_pos={0 0}, upload=541230]: Begin upload segment... Dec 31 09:06:05.867217431 (617h37m5.738300466s) [seg_pos={0 0}, upload=541230]: Done begin upload segment. Dec 31 09:06:05.867253921 (617h37m5.738336956s) [noise=true , address=145.79.227.5:4222 , piece_id=MWRH3O4FG5ICW7JONZ4FYMKKIIXSYJZ5WPAWDH5R3OPRXNLDCCVA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.867292885 (617h37m5.73837592s) [noise=true , address=145.79.227.16:4911 , piece_id=4VJTSRLYNOFOSVS3QOUROMUTEUMFW5DJ6ORRUSFSIIXORITHIWEQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.867729256 (617h37m5.738812301s) [noise=true , address=145.79.227.39:4014 , piece_id=FXY36EU2IXZXBX2SPU3GP6N3O25EFKV6PDJTYSIIIW77G6UAEKFA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.868133556 (617h37m5.739216591s) [noise=true , address=145.79.227.47:4266 , piece_id=PPOTI2TRBUR3Y2VN23BEEHE5XHLXUOZLJZ6KWDEQAB3RTKNSNZ5A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.868533548 (617h37m5.739616593s) [noise=true , address=77.90.31.13:4627 , piece_id=QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869018351 (617h37m5.740101386s) [noise=true , address=45.91.116.6:4975 , piece_id=BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869464290 (617h37m5.740547325s) [noise=true , address=77.90.31.19:4842 , piece_id=LTE74CWDYFDNZIMZT4T5X4SSMSG27HYTV6ECSXM3E4MDLTY3R5CQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869870603 (617h37m5.740953638s) [noise=true , address=77.90.31.18:4805 , piece_id=5KOIVSWCCKOZCNXE5XLPHXBKXKATG3CWX52UM7YVYR4F5ONMH67A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.870347171 (617h37m5.741430216s) [noise=true , address=145.79.227.38:4978 , piece_id=C343XQRJMHQ3O2F3UN7H26HYJ3ZA7FSE2REO6KJD734PD5YFPCAQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.870799663 (617h37m5.741882698s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.871259889 (617h37m5.742342924s) [noise=true , address=145.79.227.41:4086 , piece_id=RMA25UH4U6DVQLJWQLSVUVEJXO6NCU43PTIXMZTUAQAURWER47GA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.871817150 (617h37m5.742900195s) [noise=true , address=145.79.227.32:4834 , piece_id=ALGY4CXOJUYNK6BTWSW4O7JPC3RSUOF62M2DEWSXYUD5YX6ND7HA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.872641620 (617h37m5.743724665s) [noise=true , address=145.79.227.36:4519 , piece_id=K4MKJ3TFQE33LJ6VMTUJJTHR725C4CUX4KW3PCD73XKPTRCKMNAA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.872695763 (617h37m5.743778808s) [noise=true , address=77.90.31.11:4555 , piece_id=DUR2NOB43B572QCLAN5KLD53SNDX4XCSQJ5UJMYXHFGV7D3Y72RA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.873688093 (617h37m5.744771128s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.873947867 (617h37m5.745030912s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:07.557273878 (617h37m7.428356913s) [upload=541230]: Next returned nil segment Dec 31 09:06:07.557296371 (617h37m7.428379416s) [upload=541230]: Waiting for error group managing segments... Dec 31 09:06:07.869989232 (617h37m7.741072267s) [noise=true , address=145.79.227.36:4519 , piece_id=K4MKJ3TFQE33LJ6VMTUJJTHR725C4CUX4KW3PCD73XKPTRCKMNAA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.877879937 (617h37m7.748962972s) [noise=true , address=145.79.227.47:4266 , piece_id=PPOTI2TRBUR3Y2VN23BEEHE5XHLXUOZLJZ6KWDEQAB3RTKNSNZ5A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.887580157 (617h37m7.758663203s) [noise=true , address=145.79.227.41:4086 , piece_id=RMA25UH4U6DVQLJWQLSVUVEJXO6NCU43PTIXMZTUAQAURWER47GA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894258915 (617h37m7.76534196s) [noise=true , address=145.79.227.38:4978 , piece_id=C343XQRJMHQ3O2F3UN7H26HYJ3ZA7FSE2REO6KJD734PD5YFPCAQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894331724 (617h37m7.765414759s) [noise=true , address=145.79.227.5:4222 , piece_id=MWRH3O4FG5ICW7JONZ4FYMKKIIXSYJZ5WPAWDH5R3OPRXNLDCCVA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894725193 (617h37m7.765808238s) [noise=true , address=145.79.227.39:4014 , piece_id=FXY36EU2IXZXBX2SPU3GP6N3O25EFKV6PDJTYSIIIW77G6UAEKFA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.933008700 (617h37m7.804091735s) [noise=true , address=145.79.227.16:4911 , piece_id=4VJTSRLYNOFOSVS3QOUROMUTEUMFW5DJ6ORRUSFSIIXORITHIWEQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.945354127 (617h37m7.816437172s) [noise=true , address=145.79.227.32:4834 , piece_id=ALGY4CXOJUYNK6BTWSW4O7JPC3RSUOF62M2DEWSXYUD5YX6ND7HA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.037141884 (617h37m7.908224929s) [noise=true , address=77.90.31.19:4842 , piece_id=LTE74CWDYFDNZIMZT4T5X4SSMSG27HYTV6ECSXM3E4MDLTY3R5CQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.105167495 (617h37m7.97625054s) [noise=true , address=77.90.31.11:4555 , piece_id=DUR2NOB43B572QCLAN5KLD53SNDX4XCSQJ5UJMYXHFGV7D3Y72RA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.156289979 (617h37m8.027373024s) [noise=true , address=77.90.31.13:4627 , piece_id=QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.174675076 (617h37m8.045758111s) [noise=true , address=77.90.31.18:4805 , piece_id=5KOIVSWCCKOZCNXE5XLPHXBKXKATG3CWX52UM7YVYR4F5ONMH67A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.188351619 (617h37m8.059434664s) [noise=true , address=45.91.116.6:4975 , piece_id=BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.188388489 (617h37m8.059471534s) [seg_pos={0 0}, upload=541230]: Segment reached optimal threshold of 13 pieces. Dec 31 09:06:08.188621363 (617h37m8.059704398s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188628386 (617h37m8.059711431s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188762321 (617h37m8.059845356s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188778482 (617h37m8.059861517s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188804071 (617h37m8.059887106s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188811945 (617h37m8.05989498s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188858845 (617h37m8.05994188s) [seg_pos={0 0}, upload=541230]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 09:06:08.189175598 (617h37m8.060258633s) [upload=541230]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:06:08.189182111 (617h37m8.060265146s) [upload=541230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:06:08.483598230 (617h37m8.354681275s) [upload=541230]: Done uploading segments... Dec 31 09:06:08.483622396 (617h37m8.354705431s) [upload=541230]: Upload finished. err: <nil> Duration: 5.972315607s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [...................X...........X....X....XXX.XX.............] duration: 3.339762072s trace: https://jaeger.dev.storj.io/trace/3c18fb1b63204f7a FAIL (2025-12-31 09:09:10 UTC) orbiter warning: context deadline exceeded Duration: 3.339788752s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 28.322669729s trace: https://jaeger.dev.storj.io/trace/26257de5bcb7c920 FAIL (2025-12-31 09:09:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== , upload=540795]: Uploading piece... Dec 31 08:35:48.315166374 (617h6m48.186249409s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.315823664 (617h6m48.186906699s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316244806 (617h6m48.187327841s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316251710 (617h6m48.187334755s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316290584 (617h6m48.187373619s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317050111 (617h6m48.188133146s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317140853 (617h6m48.188223888s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318123574 (617h6m48.189206609s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318885204 (617h6m48.189968249s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.319835805 (617h6m48.19091885s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.320309847 (617h6m48.191392882s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321148273 (617h6m48.192231308s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321502088 (617h6m48.192585133s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322658430 (617h6m48.193741495s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322775803 (617h6m48.193858848s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322882035 (617h6m48.19396507s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.323403308 (617h6m48.194486343s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:49.768189393 (617h6m49.639272428s) [upload=540795]: Next returned nil segment Dec 31 08:35:49.768201797 (617h6m49.639284832s) [upload=540795]: Waiting for error group managing segments... Dec 31 08:35:50.197657834 (617h6m50.068740869s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.209847924 (617h6m50.08093096s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.217381621 (617h6m50.088464666s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.270225653 (617h6m50.141308698s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.411342630 (617h6m50.282425665s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473471693 (617h6m50.344554738s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473607101 (617h6m50.344690136s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.479010038 (617h6m50.350093074s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.482534590 (617h6m50.353617625s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.485954982 (617h6m50.357038017s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.058742323 (617h6m50.929825358s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.071853740 (617h6m50.942936775s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.177856875 (617h6m51.04893991s) [noise=true , address=145.79.227.33:4873 , piece_id=KWLS4DUDNAEADPUA3CTATKK4MEQSUPEIPYJS3SSOGVPDOJPJHDIQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179660939 (617h6m51.050743984s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179686989 (617h6m51.050770024s) [seg_pos={0 0}, upload=540795]: Segment reached optimal threshold of 14 pieces. Dec 31 08:35:51.179944619 (617h6m51.051027684s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.179956683 (617h6m51.051039728s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180323110 (617h6m51.051406155s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180346325 (617h6m51.05142937s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180485580 (617h6m51.051568615s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180495899 (617h6m51.051578934s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180529313 (617h6m51.051612358s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180536927 (617h6m51.051619972s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180560653 (617h6m51.051643698s) [seg_pos={0 0}, upload=540795]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 08:35:51.180641827 (617h6m51.051724872s) [upload=540795]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 08:35:51.180647598 (617h6m51.051730643s) [upload=540795]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 08:35:51.479648850 (617h6m51.350731896s) [upload=540795]: Done uploading segments... Dec 31 08:35:51.479871193 (617h6m51.350954228s) [upload=540795]: Upload finished. err: <nil> Duration: 28.32269135s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.......X....X.X....X...XXX..X.X.....X........XX....X..XX....] duration: 29.233986786s trace: https://jaeger.dev.storj.io/trace/354579860c475801 FAIL (2025-12-31 09:09:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA) context deadline exceeded ======LOGS====== Dec 31 09:06:05.623627963 (617h37m5.494710998s) [upload=541230]: Starting upload Dec 31 09:06:05.623687576 (617h37m5.494770611s) [upload=541230]: Done starting upload Dec 31 09:06:05.623797315 (617h37m5.49488035s) [upload=541230]: Uploading segments... Dec 31 09:06:05.623963341 (617h37m5.495046377s) [upload=541230]: Got next segment. Inline: false Dec 31 09:06:05.624003378 (617h37m5.495086423s) [upload=541230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 09:06:05.624008136 (617h37m5.495091172s) [upload=541230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 09:06:05.867160403 (617h37m5.738243438s) [seg_pos={0 0}, upload=541230]: Begin upload segment... Dec 31 09:06:05.867217431 (617h37m5.738300466s) [seg_pos={0 0}, upload=541230]: Done begin upload segment. Dec 31 09:06:05.867253921 (617h37m5.738336956s) [noise=true , address=145.79.227.5:4222 , piece_id=MWRH3O4FG5ICW7JONZ4FYMKKIIXSYJZ5WPAWDH5R3OPRXNLDCCVA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.867292885 (617h37m5.73837592s) [noise=true , address=145.79.227.16:4911 , piece_id=4VJTSRLYNOFOSVS3QOUROMUTEUMFW5DJ6ORRUSFSIIXORITHIWEQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.867729256 (617h37m5.738812301s) [noise=true , address=145.79.227.39:4014 , piece_id=FXY36EU2IXZXBX2SPU3GP6N3O25EFKV6PDJTYSIIIW77G6UAEKFA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.868133556 (617h37m5.739216591s) [noise=true , address=145.79.227.47:4266 , piece_id=PPOTI2TRBUR3Y2VN23BEEHE5XHLXUOZLJZ6KWDEQAB3RTKNSNZ5A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.868533548 (617h37m5.739616593s) [noise=true , address=77.90.31.13:4627 , piece_id=QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869018351 (617h37m5.740101386s) [noise=true , address=45.91.116.6:4975 , piece_id=BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869464290 (617h37m5.740547325s) [noise=true , address=77.90.31.19:4842 , piece_id=LTE74CWDYFDNZIMZT4T5X4SSMSG27HYTV6ECSXM3E4MDLTY3R5CQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869870603 (617h37m5.740953638s) [noise=true , address=77.90.31.18:4805 , piece_id=5KOIVSWCCKOZCNXE5XLPHXBKXKATG3CWX52UM7YVYR4F5ONMH67A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.870347171 (617h37m5.741430216s) [noise=true , address=145.79.227.38:4978 , piece_id=C343XQRJMHQ3O2F3UN7H26HYJ3ZA7FSE2REO6KJD734PD5YFPCAQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.870799663 (617h37m5.741882698s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.871259889 (617h37m5.742342924s) [noise=true , address=145.79.227.41:4086 , piece_id=RMA25UH4U6DVQLJWQLSVUVEJXO6NCU43PTIXMZTUAQAURWER47GA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.871817150 (617h37m5.742900195s) [noise=true , address=145.79.227.32:4834 , piece_id=ALGY4CXOJUYNK6BTWSW4O7JPC3RSUOF62M2DEWSXYUD5YX6ND7HA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.872641620 (617h37m5.743724665s) [noise=true , address=145.79.227.36:4519 , piece_id=K4MKJ3TFQE33LJ6VMTUJJTHR725C4CUX4KW3PCD73XKPTRCKMNAA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.872695763 (617h37m5.743778808s) [noise=true , address=77.90.31.11:4555 , piece_id=DUR2NOB43B572QCLAN5KLD53SNDX4XCSQJ5UJMYXHFGV7D3Y72RA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.873688093 (617h37m5.744771128s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.873947867 (617h37m5.745030912s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:07.557273878 (617h37m7.428356913s) [upload=541230]: Next returned nil segment Dec 31 09:06:07.557296371 (617h37m7.428379416s) [upload=541230]: Waiting for error group managing segments... Dec 31 09:06:07.869989232 (617h37m7.741072267s) [noise=true , address=145.79.227.36:4519 , piece_id=K4MKJ3TFQE33LJ6VMTUJJTHR725C4CUX4KW3PCD73XKPTRCKMNAA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.877879937 (617h37m7.748962972s) [noise=true , address=145.79.227.47:4266 , piece_id=PPOTI2TRBUR3Y2VN23BEEHE5XHLXUOZLJZ6KWDEQAB3RTKNSNZ5A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.887580157 (617h37m7.758663203s) [noise=true , address=145.79.227.41:4086 , piece_id=RMA25UH4U6DVQLJWQLSVUVEJXO6NCU43PTIXMZTUAQAURWER47GA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894258915 (617h37m7.76534196s) [noise=true , address=145.79.227.38:4978 , piece_id=C343XQRJMHQ3O2F3UN7H26HYJ3ZA7FSE2REO6KJD734PD5YFPCAQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894331724 (617h37m7.765414759s) [noise=true , address=145.79.227.5:4222 , piece_id=MWRH3O4FG5ICW7JONZ4FYMKKIIXSYJZ5WPAWDH5R3OPRXNLDCCVA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894725193 (617h37m7.765808238s) [noise=true , address=145.79.227.39:4014 , piece_id=FXY36EU2IXZXBX2SPU3GP6N3O25EFKV6PDJTYSIIIW77G6UAEKFA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.933008700 (617h37m7.804091735s) [noise=true , address=145.79.227.16:4911 , piece_id=4VJTSRLYNOFOSVS3QOUROMUTEUMFW5DJ6ORRUSFSIIXORITHIWEQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.945354127 (617h37m7.816437172s) [noise=true , address=145.79.227.32:4834 , piece_id=ALGY4CXOJUYNK6BTWSW4O7JPC3RSUOF62M2DEWSXYUD5YX6ND7HA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.037141884 (617h37m7.908224929s) [noise=true , address=77.90.31.19:4842 , piece_id=LTE74CWDYFDNZIMZT4T5X4SSMSG27HYTV6ECSXM3E4MDLTY3R5CQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.105167495 (617h37m7.97625054s) [noise=true , address=77.90.31.11:4555 , piece_id=DUR2NOB43B572QCLAN5KLD53SNDX4XCSQJ5UJMYXHFGV7D3Y72RA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.156289979 (617h37m8.027373024s) [noise=true , address=77.90.31.13:4627 , piece_id=QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.174675076 (617h37m8.045758111s) [noise=true , address=77.90.31.18:4805 , piece_id=5KOIVSWCCKOZCNXE5XLPHXBKXKATG3CWX52UM7YVYR4F5ONMH67A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.188351619 (617h37m8.059434664s) [noise=true , address=45.91.116.6:4975 , piece_id=BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.188388489 (617h37m8.059471534s) [seg_pos={0 0}, upload=541230]: Segment reached optimal threshold of 13 pieces. Dec 31 09:06:08.188621363 (617h37m8.059704398s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188628386 (617h37m8.059711431s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188762321 (617h37m8.059845356s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188778482 (617h37m8.059861517s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188804071 (617h37m8.059887106s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188811945 (617h37m8.05989498s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188858845 (617h37m8.05994188s) [seg_pos={0 0}, upload=541230]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 09:06:08.189175598 (617h37m8.060258633s) [upload=541230]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:06:08.189182111 (617h37m8.060265146s) [upload=541230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:06:08.483598230 (617h37m8.354681275s) [upload=541230]: Done uploading segments... Dec 31 09:06:08.483622396 (617h37m8.354705431s) [upload=541230]: Upload finished. err: <nil> Duration: 29.234018185s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 35.300357831s trace: https://jaeger.dev.storj.io/trace/26257de5bcb7c920 FAIL (2025-12-31 09:08:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== , upload=540795]: Uploading piece... Dec 31 08:35:48.315166374 (617h6m48.186249409s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.315823664 (617h6m48.186906699s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316244806 (617h6m48.187327841s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316251710 (617h6m48.187334755s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316290584 (617h6m48.187373619s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317050111 (617h6m48.188133146s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317140853 (617h6m48.188223888s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318123574 (617h6m48.189206609s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318885204 (617h6m48.189968249s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.319835805 (617h6m48.19091885s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.320309847 (617h6m48.191392882s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321148273 (617h6m48.192231308s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321502088 (617h6m48.192585133s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322658430 (617h6m48.193741495s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322775803 (617h6m48.193858848s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322882035 (617h6m48.19396507s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.323403308 (617h6m48.194486343s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:49.768189393 (617h6m49.639272428s) [upload=540795]: Next returned nil segment Dec 31 08:35:49.768201797 (617h6m49.639284832s) [upload=540795]: Waiting for error group managing segments... Dec 31 08:35:50.197657834 (617h6m50.068740869s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.209847924 (617h6m50.08093096s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.217381621 (617h6m50.088464666s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.270225653 (617h6m50.141308698s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.411342630 (617h6m50.282425665s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473471693 (617h6m50.344554738s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473607101 (617h6m50.344690136s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.479010038 (617h6m50.350093074s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.482534590 (617h6m50.353617625s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.485954982 (617h6m50.357038017s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.058742323 (617h6m50.929825358s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.071853740 (617h6m50.942936775s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.177856875 (617h6m51.04893991s) [noise=true , address=145.79.227.33:4873 , piece_id=KWLS4DUDNAEADPUA3CTATKK4MEQSUPEIPYJS3SSOGVPDOJPJHDIQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179660939 (617h6m51.050743984s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179686989 (617h6m51.050770024s) [seg_pos={0 0}, upload=540795]: Segment reached optimal threshold of 14 pieces. Dec 31 08:35:51.179944619 (617h6m51.051027684s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.179956683 (617h6m51.051039728s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180323110 (617h6m51.051406155s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180346325 (617h6m51.05142937s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180485580 (617h6m51.051568615s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180495899 (617h6m51.051578934s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180529313 (617h6m51.051612358s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180536927 (617h6m51.051619972s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180560653 (617h6m51.051643698s) [seg_pos={0 0}, upload=540795]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 08:35:51.180641827 (617h6m51.051724872s) [upload=540795]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 08:35:51.180647598 (617h6m51.051730643s) [upload=540795]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 08:35:51.479648850 (617h6m51.350731896s) [upload=540795]: Done uploading segments... Dec 31 08:35:51.479871193 (617h6m51.350954228s) [upload=540795]: Upload finished. err: <nil> Duration: 35.300454934s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [....................X...........X....X....XXX.XX............] duration: 4.067676281s trace: https://jaeger.dev.storj.io/trace/3c18fb1b63204f7a FAIL (2025-12-31 09:08:23 UTC) orbiter warning: context deadline exceeded Duration: 4.067708643s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 28.215384183s trace: https://jaeger.dev.storj.io/trace/26257de5bcb7c920 FAIL (2025-12-31 09:07:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== , upload=540795]: Uploading piece... Dec 31 08:35:48.315166374 (617h6m48.186249409s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.315823664 (617h6m48.186906699s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316244806 (617h6m48.187327841s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316251710 (617h6m48.187334755s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316290584 (617h6m48.187373619s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317050111 (617h6m48.188133146s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317140853 (617h6m48.188223888s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318123574 (617h6m48.189206609s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318885204 (617h6m48.189968249s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.319835805 (617h6m48.19091885s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.320309847 (617h6m48.191392882s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321148273 (617h6m48.192231308s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321502088 (617h6m48.192585133s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322658430 (617h6m48.193741495s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322775803 (617h6m48.193858848s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322882035 (617h6m48.19396507s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.323403308 (617h6m48.194486343s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:49.768189393 (617h6m49.639272428s) [upload=540795]: Next returned nil segment Dec 31 08:35:49.768201797 (617h6m49.639284832s) [upload=540795]: Waiting for error group managing segments... Dec 31 08:35:50.197657834 (617h6m50.068740869s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.209847924 (617h6m50.08093096s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.217381621 (617h6m50.088464666s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.270225653 (617h6m50.141308698s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.411342630 (617h6m50.282425665s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473471693 (617h6m50.344554738s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473607101 (617h6m50.344690136s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.479010038 (617h6m50.350093074s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.482534590 (617h6m50.353617625s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.485954982 (617h6m50.357038017s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.058742323 (617h6m50.929825358s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.071853740 (617h6m50.942936775s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.177856875 (617h6m51.04893991s) [noise=true , address=145.79.227.33:4873 , piece_id=KWLS4DUDNAEADPUA3CTATKK4MEQSUPEIPYJS3SSOGVPDOJPJHDIQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179660939 (617h6m51.050743984s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179686989 (617h6m51.050770024s) [seg_pos={0 0}, upload=540795]: Segment reached optimal threshold of 14 pieces. Dec 31 08:35:51.179944619 (617h6m51.051027684s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.179956683 (617h6m51.051039728s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180323110 (617h6m51.051406155s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180346325 (617h6m51.05142937s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180485580 (617h6m51.051568615s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180495899 (617h6m51.051578934s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180529313 (617h6m51.051612358s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180536927 (617h6m51.051619972s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180560653 (617h6m51.051643698s) [seg_pos={0 0}, upload=540795]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 08:35:51.180641827 (617h6m51.051724872s) [upload=540795]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 08:35:51.180647598 (617h6m51.051730643s) [upload=540795]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 08:35:51.479648850 (617h6m51.350731896s) [upload=540795]: Done uploading segments... Dec 31 08:35:51.479871193 (617h6m51.350954228s) [upload=540795]: Upload finished. err: <nil> Duration: 28.215410533s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [........X....X.X....X...XXX..X.X.....X........XX....X..XX...] duration: 5.883063369s trace: https://jaeger.dev.storj.io/trace/354579860c475801 FAIL (2025-12-31 09:07:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA) context deadline exceeded ======LOGS====== Dec 31 09:06:05.623627963 (617h37m5.494710998s) [upload=541230]: Starting upload Dec 31 09:06:05.623687576 (617h37m5.494770611s) [upload=541230]: Done starting upload Dec 31 09:06:05.623797315 (617h37m5.49488035s) [upload=541230]: Uploading segments... Dec 31 09:06:05.623963341 (617h37m5.495046377s) [upload=541230]: Got next segment. Inline: false Dec 31 09:06:05.624003378 (617h37m5.495086423s) [upload=541230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 09:06:05.624008136 (617h37m5.495091172s) [upload=541230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 09:06:05.867160403 (617h37m5.738243438s) [seg_pos={0 0}, upload=541230]: Begin upload segment... Dec 31 09:06:05.867217431 (617h37m5.738300466s) [seg_pos={0 0}, upload=541230]: Done begin upload segment. Dec 31 09:06:05.867253921 (617h37m5.738336956s) [noise=true , address=145.79.227.5:4222 , piece_id=MWRH3O4FG5ICW7JONZ4FYMKKIIXSYJZ5WPAWDH5R3OPRXNLDCCVA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.867292885 (617h37m5.73837592s) [noise=true , address=145.79.227.16:4911 , piece_id=4VJTSRLYNOFOSVS3QOUROMUTEUMFW5DJ6ORRUSFSIIXORITHIWEQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.867729256 (617h37m5.738812301s) [noise=true , address=145.79.227.39:4014 , piece_id=FXY36EU2IXZXBX2SPU3GP6N3O25EFKV6PDJTYSIIIW77G6UAEKFA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.868133556 (617h37m5.739216591s) [noise=true , address=145.79.227.47:4266 , piece_id=PPOTI2TRBUR3Y2VN23BEEHE5XHLXUOZLJZ6KWDEQAB3RTKNSNZ5A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.868533548 (617h37m5.739616593s) [noise=true , address=77.90.31.13:4627 , piece_id=QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869018351 (617h37m5.740101386s) [noise=true , address=45.91.116.6:4975 , piece_id=BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869464290 (617h37m5.740547325s) [noise=true , address=77.90.31.19:4842 , piece_id=LTE74CWDYFDNZIMZT4T5X4SSMSG27HYTV6ECSXM3E4MDLTY3R5CQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869870603 (617h37m5.740953638s) [noise=true , address=77.90.31.18:4805 , piece_id=5KOIVSWCCKOZCNXE5XLPHXBKXKATG3CWX52UM7YVYR4F5ONMH67A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.870347171 (617h37m5.741430216s) [noise=true , address=145.79.227.38:4978 , piece_id=C343XQRJMHQ3O2F3UN7H26HYJ3ZA7FSE2REO6KJD734PD5YFPCAQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.870799663 (617h37m5.741882698s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.871259889 (617h37m5.742342924s) [noise=true , address=145.79.227.41:4086 , piece_id=RMA25UH4U6DVQLJWQLSVUVEJXO6NCU43PTIXMZTUAQAURWER47GA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.871817150 (617h37m5.742900195s) [noise=true , address=145.79.227.32:4834 , piece_id=ALGY4CXOJUYNK6BTWSW4O7JPC3RSUOF62M2DEWSXYUD5YX6ND7HA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.872641620 (617h37m5.743724665s) [noise=true , address=145.79.227.36:4519 , piece_id=K4MKJ3TFQE33LJ6VMTUJJTHR725C4CUX4KW3PCD73XKPTRCKMNAA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.872695763 (617h37m5.743778808s) [noise=true , address=77.90.31.11:4555 , piece_id=DUR2NOB43B572QCLAN5KLD53SNDX4XCSQJ5UJMYXHFGV7D3Y72RA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.873688093 (617h37m5.744771128s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.873947867 (617h37m5.745030912s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:07.557273878 (617h37m7.428356913s) [upload=541230]: Next returned nil segment Dec 31 09:06:07.557296371 (617h37m7.428379416s) [upload=541230]: Waiting for error group managing segments... Dec 31 09:06:07.869989232 (617h37m7.741072267s) [noise=true , address=145.79.227.36:4519 , piece_id=K4MKJ3TFQE33LJ6VMTUJJTHR725C4CUX4KW3PCD73XKPTRCKMNAA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.877879937 (617h37m7.748962972s) [noise=true , address=145.79.227.47:4266 , piece_id=PPOTI2TRBUR3Y2VN23BEEHE5XHLXUOZLJZ6KWDEQAB3RTKNSNZ5A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.887580157 (617h37m7.758663203s) [noise=true , address=145.79.227.41:4086 , piece_id=RMA25UH4U6DVQLJWQLSVUVEJXO6NCU43PTIXMZTUAQAURWER47GA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894258915 (617h37m7.76534196s) [noise=true , address=145.79.227.38:4978 , piece_id=C343XQRJMHQ3O2F3UN7H26HYJ3ZA7FSE2REO6KJD734PD5YFPCAQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894331724 (617h37m7.765414759s) [noise=true , address=145.79.227.5:4222 , piece_id=MWRH3O4FG5ICW7JONZ4FYMKKIIXSYJZ5WPAWDH5R3OPRXNLDCCVA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894725193 (617h37m7.765808238s) [noise=true , address=145.79.227.39:4014 , piece_id=FXY36EU2IXZXBX2SPU3GP6N3O25EFKV6PDJTYSIIIW77G6UAEKFA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.933008700 (617h37m7.804091735s) [noise=true , address=145.79.227.16:4911 , piece_id=4VJTSRLYNOFOSVS3QOUROMUTEUMFW5DJ6ORRUSFSIIXORITHIWEQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.945354127 (617h37m7.816437172s) [noise=true , address=145.79.227.32:4834 , piece_id=ALGY4CXOJUYNK6BTWSW4O7JPC3RSUOF62M2DEWSXYUD5YX6ND7HA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.037141884 (617h37m7.908224929s) [noise=true , address=77.90.31.19:4842 , piece_id=LTE74CWDYFDNZIMZT4T5X4SSMSG27HYTV6ECSXM3E4MDLTY3R5CQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.105167495 (617h37m7.97625054s) [noise=true , address=77.90.31.11:4555 , piece_id=DUR2NOB43B572QCLAN5KLD53SNDX4XCSQJ5UJMYXHFGV7D3Y72RA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.156289979 (617h37m8.027373024s) [noise=true , address=77.90.31.13:4627 , piece_id=QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.174675076 (617h37m8.045758111s) [noise=true , address=77.90.31.18:4805 , piece_id=5KOIVSWCCKOZCNXE5XLPHXBKXKATG3CWX52UM7YVYR4F5ONMH67A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.188351619 (617h37m8.059434664s) [noise=true , address=45.91.116.6:4975 , piece_id=BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.188388489 (617h37m8.059471534s) [seg_pos={0 0}, upload=541230]: Segment reached optimal threshold of 13 pieces. Dec 31 09:06:08.188621363 (617h37m8.059704398s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188628386 (617h37m8.059711431s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188762321 (617h37m8.059845356s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188778482 (617h37m8.059861517s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188804071 (617h37m8.059887106s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188811945 (617h37m8.05989498s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188858845 (617h37m8.05994188s) [seg_pos={0 0}, upload=541230]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 09:06:08.189175598 (617h37m8.060258633s) [upload=541230]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:06:08.189182111 (617h37m8.060265146s) [upload=541230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:06:08.483598230 (617h37m8.354681275s) [upload=541230]: Done uploading segments... Dec 31 09:06:08.483622396 (617h37m8.354705431s) [upload=541230]: Upload finished. err: <nil> Duration: 5.883091021s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.........X....X.X....X...XXX..X.X.....X........XX....X..XX..] duration: 28.213241327s trace: https://jaeger.dev.storj.io/trace/354579860c475801 FAIL (2025-12-31 09:07:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA) context deadline exceeded ======LOGS====== Dec 31 09:06:05.623627963 (617h37m5.494710998s) [upload=541230]: Starting upload Dec 31 09:06:05.623687576 (617h37m5.494770611s) [upload=541230]: Done starting upload Dec 31 09:06:05.623797315 (617h37m5.49488035s) [upload=541230]: Uploading segments... Dec 31 09:06:05.623963341 (617h37m5.495046377s) [upload=541230]: Got next segment. Inline: false Dec 31 09:06:05.624003378 (617h37m5.495086423s) [upload=541230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 09:06:05.624008136 (617h37m5.495091172s) [upload=541230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 09:06:05.867160403 (617h37m5.738243438s) [seg_pos={0 0}, upload=541230]: Begin upload segment... Dec 31 09:06:05.867217431 (617h37m5.738300466s) [seg_pos={0 0}, upload=541230]: Done begin upload segment. Dec 31 09:06:05.867253921 (617h37m5.738336956s) [noise=true , address=145.79.227.5:4222 , piece_id=MWRH3O4FG5ICW7JONZ4FYMKKIIXSYJZ5WPAWDH5R3OPRXNLDCCVA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.867292885 (617h37m5.73837592s) [noise=true , address=145.79.227.16:4911 , piece_id=4VJTSRLYNOFOSVS3QOUROMUTEUMFW5DJ6ORRUSFSIIXORITHIWEQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.867729256 (617h37m5.738812301s) [noise=true , address=145.79.227.39:4014 , piece_id=FXY36EU2IXZXBX2SPU3GP6N3O25EFKV6PDJTYSIIIW77G6UAEKFA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.868133556 (617h37m5.739216591s) [noise=true , address=145.79.227.47:4266 , piece_id=PPOTI2TRBUR3Y2VN23BEEHE5XHLXUOZLJZ6KWDEQAB3RTKNSNZ5A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.868533548 (617h37m5.739616593s) [noise=true , address=77.90.31.13:4627 , piece_id=QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869018351 (617h37m5.740101386s) [noise=true , address=45.91.116.6:4975 , piece_id=BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869464290 (617h37m5.740547325s) [noise=true , address=77.90.31.19:4842 , piece_id=LTE74CWDYFDNZIMZT4T5X4SSMSG27HYTV6ECSXM3E4MDLTY3R5CQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869870603 (617h37m5.740953638s) [noise=true , address=77.90.31.18:4805 , piece_id=5KOIVSWCCKOZCNXE5XLPHXBKXKATG3CWX52UM7YVYR4F5ONMH67A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.870347171 (617h37m5.741430216s) [noise=true , address=145.79.227.38:4978 , piece_id=C343XQRJMHQ3O2F3UN7H26HYJ3ZA7FSE2REO6KJD734PD5YFPCAQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.870799663 (617h37m5.741882698s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.871259889 (617h37m5.742342924s) [noise=true , address=145.79.227.41:4086 , piece_id=RMA25UH4U6DVQLJWQLSVUVEJXO6NCU43PTIXMZTUAQAURWER47GA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.871817150 (617h37m5.742900195s) [noise=true , address=145.79.227.32:4834 , piece_id=ALGY4CXOJUYNK6BTWSW4O7JPC3RSUOF62M2DEWSXYUD5YX6ND7HA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.872641620 (617h37m5.743724665s) [noise=true , address=145.79.227.36:4519 , piece_id=K4MKJ3TFQE33LJ6VMTUJJTHR725C4CUX4KW3PCD73XKPTRCKMNAA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.872695763 (617h37m5.743778808s) [noise=true , address=77.90.31.11:4555 , piece_id=DUR2NOB43B572QCLAN5KLD53SNDX4XCSQJ5UJMYXHFGV7D3Y72RA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.873688093 (617h37m5.744771128s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.873947867 (617h37m5.745030912s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:07.557273878 (617h37m7.428356913s) [upload=541230]: Next returned nil segment Dec 31 09:06:07.557296371 (617h37m7.428379416s) [upload=541230]: Waiting for error group managing segments... Dec 31 09:06:07.869989232 (617h37m7.741072267s) [noise=true , address=145.79.227.36:4519 , piece_id=K4MKJ3TFQE33LJ6VMTUJJTHR725C4CUX4KW3PCD73XKPTRCKMNAA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.877879937 (617h37m7.748962972s) [noise=true , address=145.79.227.47:4266 , piece_id=PPOTI2TRBUR3Y2VN23BEEHE5XHLXUOZLJZ6KWDEQAB3RTKNSNZ5A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.887580157 (617h37m7.758663203s) [noise=true , address=145.79.227.41:4086 , piece_id=RMA25UH4U6DVQLJWQLSVUVEJXO6NCU43PTIXMZTUAQAURWER47GA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894258915 (617h37m7.76534196s) [noise=true , address=145.79.227.38:4978 , piece_id=C343XQRJMHQ3O2F3UN7H26HYJ3ZA7FSE2REO6KJD734PD5YFPCAQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894331724 (617h37m7.765414759s) [noise=true , address=145.79.227.5:4222 , piece_id=MWRH3O4FG5ICW7JONZ4FYMKKIIXSYJZ5WPAWDH5R3OPRXNLDCCVA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894725193 (617h37m7.765808238s) [noise=true , address=145.79.227.39:4014 , piece_id=FXY36EU2IXZXBX2SPU3GP6N3O25EFKV6PDJTYSIIIW77G6UAEKFA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.933008700 (617h37m7.804091735s) [noise=true , address=145.79.227.16:4911 , piece_id=4VJTSRLYNOFOSVS3QOUROMUTEUMFW5DJ6ORRUSFSIIXORITHIWEQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.945354127 (617h37m7.816437172s) [noise=true , address=145.79.227.32:4834 , piece_id=ALGY4CXOJUYNK6BTWSW4O7JPC3RSUOF62M2DEWSXYUD5YX6ND7HA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.037141884 (617h37m7.908224929s) [noise=true , address=77.90.31.19:4842 , piece_id=LTE74CWDYFDNZIMZT4T5X4SSMSG27HYTV6ECSXM3E4MDLTY3R5CQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.105167495 (617h37m7.97625054s) [noise=true , address=77.90.31.11:4555 , piece_id=DUR2NOB43B572QCLAN5KLD53SNDX4XCSQJ5UJMYXHFGV7D3Y72RA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.156289979 (617h37m8.027373024s) [noise=true , address=77.90.31.13:4627 , piece_id=QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.174675076 (617h37m8.045758111s) [noise=true , address=77.90.31.18:4805 , piece_id=5KOIVSWCCKOZCNXE5XLPHXBKXKATG3CWX52UM7YVYR4F5ONMH67A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.188351619 (617h37m8.059434664s) [noise=true , address=45.91.116.6:4975 , piece_id=BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.188388489 (617h37m8.059471534s) [seg_pos={0 0}, upload=541230]: Segment reached optimal threshold of 13 pieces. Dec 31 09:06:08.188621363 (617h37m8.059704398s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188628386 (617h37m8.059711431s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188762321 (617h37m8.059845356s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188778482 (617h37m8.059861517s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188804071 (617h37m8.059887106s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188811945 (617h37m8.05989498s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188858845 (617h37m8.05994188s) [seg_pos={0 0}, upload=541230]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 09:06:08.189175598 (617h37m8.060258633s) [upload=541230]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:06:08.189182111 (617h37m8.060265146s) [upload=541230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:06:08.483598230 (617h37m8.354681275s) [upload=541230]: Done uploading segments... Dec 31 09:06:08.483622396 (617h37m8.354705431s) [upload=541230]: Upload finished. err: <nil> Duration: 28.213270402s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.....................X...........X....X....XXX.XX...........] duration: 3.393447444s trace: https://jaeger.dev.storj.io/trace/3c18fb1b63204f7a FAIL (2025-12-31 09:07:13 UTC) orbiter warning: context deadline exceeded Duration: 3.393471299s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 31.649501922s trace: https://jaeger.dev.storj.io/trace/26257de5bcb7c920 FAIL (2025-12-31 09:07:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== , upload=540795]: Uploading piece... Dec 31 08:35:48.315166374 (617h6m48.186249409s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.315823664 (617h6m48.186906699s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316244806 (617h6m48.187327841s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316251710 (617h6m48.187334755s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316290584 (617h6m48.187373619s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317050111 (617h6m48.188133146s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317140853 (617h6m48.188223888s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318123574 (617h6m48.189206609s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318885204 (617h6m48.189968249s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.319835805 (617h6m48.19091885s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.320309847 (617h6m48.191392882s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321148273 (617h6m48.192231308s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321502088 (617h6m48.192585133s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322658430 (617h6m48.193741495s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322775803 (617h6m48.193858848s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322882035 (617h6m48.19396507s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.323403308 (617h6m48.194486343s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:49.768189393 (617h6m49.639272428s) [upload=540795]: Next returned nil segment Dec 31 08:35:49.768201797 (617h6m49.639284832s) [upload=540795]: Waiting for error group managing segments... Dec 31 08:35:50.197657834 (617h6m50.068740869s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.209847924 (617h6m50.08093096s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.217381621 (617h6m50.088464666s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.270225653 (617h6m50.141308698s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.411342630 (617h6m50.282425665s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473471693 (617h6m50.344554738s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473607101 (617h6m50.344690136s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.479010038 (617h6m50.350093074s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.482534590 (617h6m50.353617625s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.485954982 (617h6m50.357038017s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.058742323 (617h6m50.929825358s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.071853740 (617h6m50.942936775s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.177856875 (617h6m51.04893991s) [noise=true , address=145.79.227.33:4873 , piece_id=KWLS4DUDNAEADPUA3CTATKK4MEQSUPEIPYJS3SSOGVPDOJPJHDIQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179660939 (617h6m51.050743984s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179686989 (617h6m51.050770024s) [seg_pos={0 0}, upload=540795]: Segment reached optimal threshold of 14 pieces. Dec 31 08:35:51.179944619 (617h6m51.051027684s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.179956683 (617h6m51.051039728s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180323110 (617h6m51.051406155s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180346325 (617h6m51.05142937s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180485580 (617h6m51.051568615s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180495899 (617h6m51.051578934s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180529313 (617h6m51.051612358s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180536927 (617h6m51.051619972s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180560653 (617h6m51.051643698s) [seg_pos={0 0}, upload=540795]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 08:35:51.180641827 (617h6m51.051724872s) [upload=540795]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 08:35:51.180647598 (617h6m51.051730643s) [upload=540795]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 08:35:51.479648850 (617h6m51.350731896s) [upload=540795]: Done uploading segments... Dec 31 08:35:51.479871193 (617h6m51.350954228s) [upload=540795]: Upload finished. err: <nil> Duration: 31.649531819s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..........X....X.X....X...XXX..X.X.....X........XX....X..XX.] duration: 5.920308446s trace: https://jaeger.dev.storj.io/trace/354579860c475801 FAIL (2025-12-31 09:06:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA) context deadline exceeded ======LOGS====== Dec 31 09:06:05.623627963 (617h37m5.494710998s) [upload=541230]: Starting upload Dec 31 09:06:05.623687576 (617h37m5.494770611s) [upload=541230]: Done starting upload Dec 31 09:06:05.623797315 (617h37m5.49488035s) [upload=541230]: Uploading segments... Dec 31 09:06:05.623963341 (617h37m5.495046377s) [upload=541230]: Got next segment. Inline: false Dec 31 09:06:05.624003378 (617h37m5.495086423s) [upload=541230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 09:06:05.624008136 (617h37m5.495091172s) [upload=541230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 09:06:05.867160403 (617h37m5.738243438s) [seg_pos={0 0}, upload=541230]: Begin upload segment... Dec 31 09:06:05.867217431 (617h37m5.738300466s) [seg_pos={0 0}, upload=541230]: Done begin upload segment. Dec 31 09:06:05.867253921 (617h37m5.738336956s) [noise=true , address=145.79.227.5:4222 , piece_id=MWRH3O4FG5ICW7JONZ4FYMKKIIXSYJZ5WPAWDH5R3OPRXNLDCCVA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.867292885 (617h37m5.73837592s) [noise=true , address=145.79.227.16:4911 , piece_id=4VJTSRLYNOFOSVS3QOUROMUTEUMFW5DJ6ORRUSFSIIXORITHIWEQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.867729256 (617h37m5.738812301s) [noise=true , address=145.79.227.39:4014 , piece_id=FXY36EU2IXZXBX2SPU3GP6N3O25EFKV6PDJTYSIIIW77G6UAEKFA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.868133556 (617h37m5.739216591s) [noise=true , address=145.79.227.47:4266 , piece_id=PPOTI2TRBUR3Y2VN23BEEHE5XHLXUOZLJZ6KWDEQAB3RTKNSNZ5A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.868533548 (617h37m5.739616593s) [noise=true , address=77.90.31.13:4627 , piece_id=QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869018351 (617h37m5.740101386s) [noise=true , address=45.91.116.6:4975 , piece_id=BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869464290 (617h37m5.740547325s) [noise=true , address=77.90.31.19:4842 , piece_id=LTE74CWDYFDNZIMZT4T5X4SSMSG27HYTV6ECSXM3E4MDLTY3R5CQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869870603 (617h37m5.740953638s) [noise=true , address=77.90.31.18:4805 , piece_id=5KOIVSWCCKOZCNXE5XLPHXBKXKATG3CWX52UM7YVYR4F5ONMH67A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.870347171 (617h37m5.741430216s) [noise=true , address=145.79.227.38:4978 , piece_id=C343XQRJMHQ3O2F3UN7H26HYJ3ZA7FSE2REO6KJD734PD5YFPCAQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.870799663 (617h37m5.741882698s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.871259889 (617h37m5.742342924s) [noise=true , address=145.79.227.41:4086 , piece_id=RMA25UH4U6DVQLJWQLSVUVEJXO6NCU43PTIXMZTUAQAURWER47GA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.871817150 (617h37m5.742900195s) [noise=true , address=145.79.227.32:4834 , piece_id=ALGY4CXOJUYNK6BTWSW4O7JPC3RSUOF62M2DEWSXYUD5YX6ND7HA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.872641620 (617h37m5.743724665s) [noise=true , address=145.79.227.36:4519 , piece_id=K4MKJ3TFQE33LJ6VMTUJJTHR725C4CUX4KW3PCD73XKPTRCKMNAA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.872695763 (617h37m5.743778808s) [noise=true , address=77.90.31.11:4555 , piece_id=DUR2NOB43B572QCLAN5KLD53SNDX4XCSQJ5UJMYXHFGV7D3Y72RA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.873688093 (617h37m5.744771128s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.873947867 (617h37m5.745030912s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:07.557273878 (617h37m7.428356913s) [upload=541230]: Next returned nil segment Dec 31 09:06:07.557296371 (617h37m7.428379416s) [upload=541230]: Waiting for error group managing segments... Dec 31 09:06:07.869989232 (617h37m7.741072267s) [noise=true , address=145.79.227.36:4519 , piece_id=K4MKJ3TFQE33LJ6VMTUJJTHR725C4CUX4KW3PCD73XKPTRCKMNAA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.877879937 (617h37m7.748962972s) [noise=true , address=145.79.227.47:4266 , piece_id=PPOTI2TRBUR3Y2VN23BEEHE5XHLXUOZLJZ6KWDEQAB3RTKNSNZ5A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.887580157 (617h37m7.758663203s) [noise=true , address=145.79.227.41:4086 , piece_id=RMA25UH4U6DVQLJWQLSVUVEJXO6NCU43PTIXMZTUAQAURWER47GA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894258915 (617h37m7.76534196s) [noise=true , address=145.79.227.38:4978 , piece_id=C343XQRJMHQ3O2F3UN7H26HYJ3ZA7FSE2REO6KJD734PD5YFPCAQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894331724 (617h37m7.765414759s) [noise=true , address=145.79.227.5:4222 , piece_id=MWRH3O4FG5ICW7JONZ4FYMKKIIXSYJZ5WPAWDH5R3OPRXNLDCCVA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894725193 (617h37m7.765808238s) [noise=true , address=145.79.227.39:4014 , piece_id=FXY36EU2IXZXBX2SPU3GP6N3O25EFKV6PDJTYSIIIW77G6UAEKFA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.933008700 (617h37m7.804091735s) [noise=true , address=145.79.227.16:4911 , piece_id=4VJTSRLYNOFOSVS3QOUROMUTEUMFW5DJ6ORRUSFSIIXORITHIWEQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.945354127 (617h37m7.816437172s) [noise=true , address=145.79.227.32:4834 , piece_id=ALGY4CXOJUYNK6BTWSW4O7JPC3RSUOF62M2DEWSXYUD5YX6ND7HA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.037141884 (617h37m7.908224929s) [noise=true , address=77.90.31.19:4842 , piece_id=LTE74CWDYFDNZIMZT4T5X4SSMSG27HYTV6ECSXM3E4MDLTY3R5CQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.105167495 (617h37m7.97625054s) [noise=true , address=77.90.31.11:4555 , piece_id=DUR2NOB43B572QCLAN5KLD53SNDX4XCSQJ5UJMYXHFGV7D3Y72RA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.156289979 (617h37m8.027373024s) [noise=true , address=77.90.31.13:4627 , piece_id=QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.174675076 (617h37m8.045758111s) [noise=true , address=77.90.31.18:4805 , piece_id=5KOIVSWCCKOZCNXE5XLPHXBKXKATG3CWX52UM7YVYR4F5ONMH67A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.188351619 (617h37m8.059434664s) [noise=true , address=45.91.116.6:4975 , piece_id=BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.188388489 (617h37m8.059471534s) [seg_pos={0 0}, upload=541230]: Segment reached optimal threshold of 13 pieces. Dec 31 09:06:08.188621363 (617h37m8.059704398s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188628386 (617h37m8.059711431s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188762321 (617h37m8.059845356s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188778482 (617h37m8.059861517s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188804071 (617h37m8.059887106s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188811945 (617h37m8.05989498s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188858845 (617h37m8.05994188s) [seg_pos={0 0}, upload=541230]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 09:06:08.189175598 (617h37m8.060258633s) [upload=541230]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:06:08.189182111 (617h37m8.060265146s) [upload=541230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:06:08.483598230 (617h37m8.354681275s) [upload=541230]: Done uploading segments... Dec 31 09:06:08.483622396 (617h37m8.354705431s) [upload=541230]: Upload finished. err: <nil> Duration: 5.920325097s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X..........X....X.X....X...XXX..X.X.....X........XX....X..XX] duration: 30.003539422s trace: https://jaeger.dev.storj.io/trace/354579860c475801 FAIL (2025-12-31 09:06:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA) context deadline exceeded ======LOGS====== Dec 31 09:06:05.623627963 (617h37m5.494710998s) [upload=541230]: Starting upload Dec 31 09:06:05.623687576 (617h37m5.494770611s) [upload=541230]: Done starting upload Dec 31 09:06:05.623797315 (617h37m5.49488035s) [upload=541230]: Uploading segments... Dec 31 09:06:05.623963341 (617h37m5.495046377s) [upload=541230]: Got next segment. Inline: false Dec 31 09:06:05.624003378 (617h37m5.495086423s) [upload=541230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 09:06:05.624008136 (617h37m5.495091172s) [upload=541230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 09:06:05.867160403 (617h37m5.738243438s) [seg_pos={0 0}, upload=541230]: Begin upload segment... Dec 31 09:06:05.867217431 (617h37m5.738300466s) [seg_pos={0 0}, upload=541230]: Done begin upload segment. Dec 31 09:06:05.867253921 (617h37m5.738336956s) [noise=true , address=145.79.227.5:4222 , piece_id=MWRH3O4FG5ICW7JONZ4FYMKKIIXSYJZ5WPAWDH5R3OPRXNLDCCVA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.867292885 (617h37m5.73837592s) [noise=true , address=145.79.227.16:4911 , piece_id=4VJTSRLYNOFOSVS3QOUROMUTEUMFW5DJ6ORRUSFSIIXORITHIWEQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.867729256 (617h37m5.738812301s) [noise=true , address=145.79.227.39:4014 , piece_id=FXY36EU2IXZXBX2SPU3GP6N3O25EFKV6PDJTYSIIIW77G6UAEKFA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.868133556 (617h37m5.739216591s) [noise=true , address=145.79.227.47:4266 , piece_id=PPOTI2TRBUR3Y2VN23BEEHE5XHLXUOZLJZ6KWDEQAB3RTKNSNZ5A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.868533548 (617h37m5.739616593s) [noise=true , address=77.90.31.13:4627 , piece_id=QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869018351 (617h37m5.740101386s) [noise=true , address=45.91.116.6:4975 , piece_id=BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869464290 (617h37m5.740547325s) [noise=true , address=77.90.31.19:4842 , piece_id=LTE74CWDYFDNZIMZT4T5X4SSMSG27HYTV6ECSXM3E4MDLTY3R5CQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869870603 (617h37m5.740953638s) [noise=true , address=77.90.31.18:4805 , piece_id=5KOIVSWCCKOZCNXE5XLPHXBKXKATG3CWX52UM7YVYR4F5ONMH67A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.870347171 (617h37m5.741430216s) [noise=true , address=145.79.227.38:4978 , piece_id=C343XQRJMHQ3O2F3UN7H26HYJ3ZA7FSE2REO6KJD734PD5YFPCAQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.870799663 (617h37m5.741882698s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.871259889 (617h37m5.742342924s) [noise=true , address=145.79.227.41:4086 , piece_id=RMA25UH4U6DVQLJWQLSVUVEJXO6NCU43PTIXMZTUAQAURWER47GA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.871817150 (617h37m5.742900195s) [noise=true , address=145.79.227.32:4834 , piece_id=ALGY4CXOJUYNK6BTWSW4O7JPC3RSUOF62M2DEWSXYUD5YX6ND7HA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.872641620 (617h37m5.743724665s) [noise=true , address=145.79.227.36:4519 , piece_id=K4MKJ3TFQE33LJ6VMTUJJTHR725C4CUX4KW3PCD73XKPTRCKMNAA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.872695763 (617h37m5.743778808s) [noise=true , address=77.90.31.11:4555 , piece_id=DUR2NOB43B572QCLAN5KLD53SNDX4XCSQJ5UJMYXHFGV7D3Y72RA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.873688093 (617h37m5.744771128s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.873947867 (617h37m5.745030912s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:07.557273878 (617h37m7.428356913s) [upload=541230]: Next returned nil segment Dec 31 09:06:07.557296371 (617h37m7.428379416s) [upload=541230]: Waiting for error group managing segments... Dec 31 09:06:07.869989232 (617h37m7.741072267s) [noise=true , address=145.79.227.36:4519 , piece_id=K4MKJ3TFQE33LJ6VMTUJJTHR725C4CUX4KW3PCD73XKPTRCKMNAA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.877879937 (617h37m7.748962972s) [noise=true , address=145.79.227.47:4266 , piece_id=PPOTI2TRBUR3Y2VN23BEEHE5XHLXUOZLJZ6KWDEQAB3RTKNSNZ5A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.887580157 (617h37m7.758663203s) [noise=true , address=145.79.227.41:4086 , piece_id=RMA25UH4U6DVQLJWQLSVUVEJXO6NCU43PTIXMZTUAQAURWER47GA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894258915 (617h37m7.76534196s) [noise=true , address=145.79.227.38:4978 , piece_id=C343XQRJMHQ3O2F3UN7H26HYJ3ZA7FSE2REO6KJD734PD5YFPCAQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894331724 (617h37m7.765414759s) [noise=true , address=145.79.227.5:4222 , piece_id=MWRH3O4FG5ICW7JONZ4FYMKKIIXSYJZ5WPAWDH5R3OPRXNLDCCVA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894725193 (617h37m7.765808238s) [noise=true , address=145.79.227.39:4014 , piece_id=FXY36EU2IXZXBX2SPU3GP6N3O25EFKV6PDJTYSIIIW77G6UAEKFA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.933008700 (617h37m7.804091735s) [noise=true , address=145.79.227.16:4911 , piece_id=4VJTSRLYNOFOSVS3QOUROMUTEUMFW5DJ6ORRUSFSIIXORITHIWEQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.945354127 (617h37m7.816437172s) [noise=true , address=145.79.227.32:4834 , piece_id=ALGY4CXOJUYNK6BTWSW4O7JPC3RSUOF62M2DEWSXYUD5YX6ND7HA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.037141884 (617h37m7.908224929s) [noise=true , address=77.90.31.19:4842 , piece_id=LTE74CWDYFDNZIMZT4T5X4SSMSG27HYTV6ECSXM3E4MDLTY3R5CQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.105167495 (617h37m7.97625054s) [noise=true , address=77.90.31.11:4555 , piece_id=DUR2NOB43B572QCLAN5KLD53SNDX4XCSQJ5UJMYXHFGV7D3Y72RA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.156289979 (617h37m8.027373024s) [noise=true , address=77.90.31.13:4627 , piece_id=QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.174675076 (617h37m8.045758111s) [noise=true , address=77.90.31.18:4805 , piece_id=5KOIVSWCCKOZCNXE5XLPHXBKXKATG3CWX52UM7YVYR4F5ONMH67A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.188351619 (617h37m8.059434664s) [noise=true , address=45.91.116.6:4975 , piece_id=BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.188388489 (617h37m8.059471534s) [seg_pos={0 0}, upload=541230]: Segment reached optimal threshold of 13 pieces. Dec 31 09:06:08.188621363 (617h37m8.059704398s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188628386 (617h37m8.059711431s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188762321 (617h37m8.059845356s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188778482 (617h37m8.059861517s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188804071 (617h37m8.059887106s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188811945 (617h37m8.05989498s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188858845 (617h37m8.05994188s) [seg_pos={0 0}, upload=541230]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 09:06:08.189175598 (617h37m8.060258633s) [upload=541230]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:06:08.189182111 (617h37m8.060265146s) [upload=541230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:06:08.483598230 (617h37m8.354681275s) [upload=541230]: Done uploading segments... Dec 31 09:06:08.483622396 (617h37m8.354705431s) [upload=541230]: Upload finished. err: <nil> Duration: 30.003549451s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [......................X...........X....X....XXX.XX..........] duration: 3.436221664s trace: https://jaeger.dev.storj.io/trace/3c18fb1b63204f7a FAIL (2025-12-31 09:06:23 UTC) orbiter warning: context deadline exceeded Duration: 3.436246851s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 24.013332743s trace: https://jaeger.dev.storj.io/trace/26257de5bcb7c920 FAIL (2025-12-31 09:06:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== , upload=540795]: Uploading piece... Dec 31 08:35:48.315166374 (617h6m48.186249409s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.315823664 (617h6m48.186906699s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316244806 (617h6m48.187327841s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316251710 (617h6m48.187334755s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316290584 (617h6m48.187373619s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317050111 (617h6m48.188133146s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317140853 (617h6m48.188223888s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318123574 (617h6m48.189206609s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318885204 (617h6m48.189968249s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.319835805 (617h6m48.19091885s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.320309847 (617h6m48.191392882s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321148273 (617h6m48.192231308s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321502088 (617h6m48.192585133s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322658430 (617h6m48.193741495s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322775803 (617h6m48.193858848s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322882035 (617h6m48.19396507s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.323403308 (617h6m48.194486343s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:49.768189393 (617h6m49.639272428s) [upload=540795]: Next returned nil segment Dec 31 08:35:49.768201797 (617h6m49.639284832s) [upload=540795]: Waiting for error group managing segments... Dec 31 08:35:50.197657834 (617h6m50.068740869s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.209847924 (617h6m50.08093096s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.217381621 (617h6m50.088464666s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.270225653 (617h6m50.141308698s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.411342630 (617h6m50.282425665s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473471693 (617h6m50.344554738s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473607101 (617h6m50.344690136s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.479010038 (617h6m50.350093074s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.482534590 (617h6m50.353617625s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.485954982 (617h6m50.357038017s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.058742323 (617h6m50.929825358s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.071853740 (617h6m50.942936775s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.177856875 (617h6m51.04893991s) [noise=true , address=145.79.227.33:4873 , piece_id=KWLS4DUDNAEADPUA3CTATKK4MEQSUPEIPYJS3SSOGVPDOJPJHDIQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179660939 (617h6m51.050743984s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179686989 (617h6m51.050770024s) [seg_pos={0 0}, upload=540795]: Segment reached optimal threshold of 14 pieces. Dec 31 08:35:51.179944619 (617h6m51.051027684s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.179956683 (617h6m51.051039728s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180323110 (617h6m51.051406155s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180346325 (617h6m51.05142937s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180485580 (617h6m51.051568615s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180495899 (617h6m51.051578934s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180529313 (617h6m51.051612358s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180536927 (617h6m51.051619972s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180560653 (617h6m51.051643698s) [seg_pos={0 0}, upload=540795]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 08:35:51.180641827 (617h6m51.051724872s) [upload=540795]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 08:35:51.180647598 (617h6m51.051730643s) [upload=540795]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 08:35:51.479648850 (617h6m51.350731896s) [upload=540795]: Done uploading segments... Dec 31 08:35:51.479871193 (617h6m51.350954228s) [upload=540795]: Upload finished. err: <nil> Duration: 24.013350978s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X..........X....X.X....X...XXX..X.X.....X........XX....X..X] duration: 30.003524159s trace: https://jaeger.dev.storj.io/trace/3b6bbf20f29283cc FAIL (2025-12-31 09:06:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 31 09:05:35.380163092 (617h36m35.251246138s) [upload=541223]: Starting upload Dec 31 09:05:35.380237524 (617h36m35.251320559s) [upload=541223]: Done starting upload Dec 31 09:05:35.380365408 (617h36m35.251448463s) [upload=541223]: Uploading segments... Dec 31 09:05:35.380471279 (617h36m35.251554325s) [upload=541223]: Got next segment. Inline: false Dec 31 09:05:35.380512598 (617h36m35.251595643s) [upload=541223]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 09:05:35.380518830 (617h36m35.251601875s) [upload=541223]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 09:05:35.628302075 (617h36m35.49938513s) [seg_pos={0 0}, upload=541223]: Begin upload segment... Dec 31 09:05:35.628408237 (617h36m35.499491272s) [seg_pos={0 0}, upload=541223]: Done begin upload segment. Dec 31 09:05:35.628552491 (617h36m35.499635526s) [noise=true , address=145.79.227.4:4187 , piece_id=56JQCKQSTSYPLBIIOJCTKA4NLY5WF7FW6QGHB2K7XSJOVC4YS5MQ, seg_pos={0 0}, upload=541223]: Uploading piece... Dec 31 09:05:35.628616383 (617h36m35.499699428s) [noise=true , address=145.79.227.32:4836 , piece_id=ZEFIQGT6EVEZLDHIQVUFFUENNMS6K2AH3AHDTP5MYQGTS63YWI4A, seg_pos={0 0}, upload=541223]: Uploading piece... Dec 31 09:05:35.631201946 (617h36m35.502285011s) [noise=true , address=145.79.227.3:4151 , piece_id=4YTFSXQITHDBPFV5LO4G5Q3XXMEEIGSXXT5QRR62PZOHLZY4HXEQ, seg_pos={0 0}, upload=541223]: Uploading piece... Dec 31 09:05:35.631346141 (617h36m35.502429186s) [noise=true , address=45.91.116.10:4081 , piece_id=U7PNMGD5XYK6XEAUNMITNBILQSIABDEMRGGOPGJTE5BMM27PBYJA, seg_pos={0 0}, upload=541223]: Uploading piece... Dec 31 09:05:35.631879246 (617h36m35.502962291s) [noise=true , address=77.90.31.16:4736 , piece_id=RZMSII2SDNPHQWLLA2RZYONFUI64MRQHZLFIBVNVEMU6QPKI6UAQ, seg_pos={0 0}, upload=541223]: Uploading piece... Dec 31 09:05:35.632102551 (617h36m35.503185586s) [noise=true , address=145.79.227.8:4330 , piece_id=YYY6WMV3RFK5LECIHQ4LAPZKSD6F5RCTTMQAYMGYD25CS464KH5A, seg_pos={0 0}, upload=541223]: Uploading piece... Dec 31 09:05:35.632480139 (617h36m35.503563175s) [noise=true , address=145.79.227.11:4730 , piece_id=LAFLAG2KIWT6DOQKTVS2HNXEMMO3JZ5R267V4AHDIWMTJOWTCJ3Q, seg_pos={0 0}, upload=541223]: Uploading piece... Dec 31 09:05:35.632768159 (617h36m35.503851204s) [noise=true , address=45.91.116.11:4120 , piece_id=3QP553L5EQUDK2W2E4CG4LGB2OVTI6BGU7IFKBCBUCT2ZNRWTH2A, seg_pos={0 0}, upload=541223]: Uploading piece... Dec 31 09:05:35.633229006 (617h36m35.504312041s) [noise=true , address=77.90.31.18:4808 , piece_id=3BNB4GZ2QQEFTGXGPSX3S76MFNJ6GNEZUI5MGSE3LSDH7VG66DNQ, seg_pos={0 0}, upload=541223]: Uploading piece... Dec 31 09:05:35.633652292 (617h36m35.504735327s) [noise=true , address=45.91.116.9:4263 , piece_id=3LCLG2HBVK7ADYGX5DPQG6TNMP3SULYNQJHN4UI5OP5VIVPD24FA, seg_pos={0 0}, upload=541223]: Uploading piece... Dec 31 09:05:35.634264708 (617h36m35.505347743s) [noise=true , address=145.79.227.6:4258 , piece_id=GZYVIR2UDHBCINMEJMXUJLPWIW3BNB3P23EOY64C3QBHFG7RLRAQ, seg_pos={0 0}, upload=541223]: Uploading piece... Dec 31 09:05:35.634473877 (617h36m35.505556912s) [noise=true , address=145.79.227.13:4804 , piece_id=YJDLFHBEWWY425TLSWPU766FTWCE6URDASY3WUYHG6IHWSPD63PA, seg_pos={0 0}, upload=541223]: Uploading piece... Dec 31 09:05:35.634842678 (617h36m35.505925723s) [noise=true , address=145.79.227.39:4013 , piece_id=NWVVZ4VSYJJKTVARMBXXBN3QZJEFT3NW7JRSLWXPHIO3BNATMBOA, seg_pos={0 0}, upload=541223]: Uploading piece... Dec 31 09:05:35.635374591 (617h36m35.506457626s) [noise=true , address=77.90.31.13:4626 , piece_id=BDAZSEK4V4T2YNHJQZHQ63YMDHPHI57BH4IKIPOQI5Q34MKFBOSQ, seg_pos={0 0}, upload=541223]: Uploading piece... Dec 31 09:05:35.635798048 (617h36m35.506881083s) [noise=true , address=77.90.31.14:4661 , piece_id=X4ZAWVUEIK2RVDMGFG65GTJMPYP577VJI344243FNM7XLPZSU6MA, seg_pos={0 0}, upload=541223]: Uploading piece... Dec 31 09:05:35.636373183 (617h36m35.507456248s) [noise=true , address=45.91.116.5:4297 , piece_id=ULBKM5DJQWTHHT3LWP5DG6VHDTD3MWQMMNTWK6ATDHHF2P6XQIEQ, seg_pos={0 0}, upload=541223]: Uploading piece... Dec 31 09:05:37.327415524 (617h36m37.198498569s) [upload=541223]: Next returned nil segment Dec 31 09:05:37.327478984 (617h36m37.198562019s) [upload=541223]: Waiting for error group managing segments... Dec 31 09:05:37.622408421 (617h36m37.493491456s) [noise=true , address=145.79.227.4:4187 , piece_id=56JQCKQSTSYPLBIIOJCTKA4NLY5WF7FW6QGHB2K7XSJOVC4YS5MQ, seg_pos={0 0}, upload=541223]: Done uploading piece. err: <nil> Dec 31 09:05:37.628793820 (617h36m37.499876855s) [noise=true , address=145.79.227.11:4730 , piece_id=LAFLAG2KIWT6DOQKTVS2HNXEMMO3JZ5R267V4AHDIWMTJOWTCJ3Q, seg_pos={0 0}, upload=541223]: Done uploading piece. err: <nil> Dec 31 09:05:37.642506781 (617h36m37.513589826s) [noise=true , address=145.79.227.13:4804 , piece_id=YJDLFHBEWWY425TLSWPU766FTWCE6URDASY3WUYHG6IHWSPD63PA, seg_pos={0 0}, upload=541223]: Done uploading piece. err: <nil> Dec 31 09:05:37.644188523 (617h36m37.515271558s) [noise=true , address=145.79.227.8:4330 , piece_id=YYY6WMV3RFK5LECIHQ4LAPZKSD6F5RCTTMQAYMGYD25CS464KH5A, seg_pos={0 0}, upload=541223]: Done uploading piece. err: <nil> Dec 31 09:05:37.647699048 (617h36m37.518782083s) [noise=true , address=145.79.227.6:4258 , piece_id=GZYVIR2UDHBCINMEJMXUJLPWIW3BNB3P23EOY64C3QBHFG7RLRAQ, seg_pos={0 0}, upload=541223]: Done uploading piece. err: <nil> Dec 31 09:05:37.699077630 (617h36m37.570160665s) [noise=true , address=145.79.227.32:4836 , piece_id=ZEFIQGT6EVEZLDHIQVUFFUENNMS6K2AH3AHDTP5MYQGTS63YWI4A, seg_pos={0 0}, upload=541223]: Done uploading piece. err: <nil> Dec 31 09:05:37.701073881 (617h36m37.572156916s) [noise=true , address=145.79.227.3:4151 , piece_id=4YTFSXQITHDBPFV5LO4G5Q3XXMEEIGSXXT5QRR62PZOHLZY4HXEQ, seg_pos={0 0}, upload=541223]: Done uploading piece. err: <nil> Dec 31 09:05:37.705516690 (617h36m37.576599725s) [noise=true , address=145.79.227.39:4013 , piece_id=NWVVZ4VSYJJKTVARMBXXBN3QZJEFT3NW7JRSLWXPHIO3BNATMBOA, seg_pos={0 0}, upload=541223]: Done uploading piece. err: <nil> Dec 31 09:05:37.785426209 (617h36m37.656509244s) [noise=true , address=77.90.31.13:4626 , piece_id=BDAZSEK4V4T2YNHJQZHQ63YMDHPHI57BH4IKIPOQI5Q34MKFBOSQ, seg_pos={0 0}, upload=541223]: Done uploading piece. err: <nil> Dec 31 09:05:37.897329309 (617h36m37.768412344s) [noise=true , address=77.90.31.18:4808 , piece_id=3BNB4GZ2QQEFTGXGPSX3S76MFNJ6GNEZUI5MGSE3LSDH7VG66DNQ, seg_pos={0 0}, upload=541223]: Done uploading piece. err: <nil> Dec 31 09:05:37.921838056 (617h36m37.792921102s) [noise=true , address=77.90.31.16:4736 , piece_id=RZMSII2SDNPHQWLLA2RZYONFUI64MRQHZLFIBVNVEMU6QPKI6UAQ, seg_pos={0 0}, upload=541223]: Done uploading piece. err: <nil> Dec 31 09:05:37.964828126 (617h36m37.835911171s) [noise=true , address=45.91.116.9:4263 , piece_id=3LCLG2HBVK7ADYGX5DPQG6TNMP3SULYNQJHN4UI5OP5VIVPD24FA, seg_pos={0 0}, upload=541223]: Done uploading piece. err: <nil> Dec 31 09:05:38.003531834 (617h36m37.874614869s) [noise=true , address=45.91.116.10:4081 , piece_id=U7PNMGD5XYK6XEAUNMITNBILQSIABDEMRGGOPGJTE5BMM27PBYJA, seg_pos={0 0}, upload=541223]: Done uploading piece. err: <nil> Dec 31 09:05:38.003554377 (617h36m37.874637412s) [seg_pos={0 0}, upload=541223]: Segment reached optimal threshold of 13 pieces. Dec 31 09:05:38.003720723 (617h36m37.874803768s) [noise=true , address=45.91.116.5:4297 , piece_id=ULBKM5DJQWTHHT3LWP5DG6VHDTD3MWQMMNTWK6ATDHHF2P6XQIEQ, seg_pos={0 0}, upload=541223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 31 09:05:38.003727176 (617h36m37.874810211s) [noise=true , address=45.91.116.5:4297 , piece_id=ULBKM5DJQWTHHT3LWP5DG6VHDTD3MWQMMNTWK6ATDHHF2P6XQIEQ, seg_pos={0 0}, upload=541223]: Long tail cancellation due to optimal threshold Dec 31 09:05:38.003974868 (617h36m37.875057903s) [noise=true , address=77.90.31.14:4661 , piece_id=X4ZAWVUEIK2RVDMGFG65GTJMPYP577VJI344243FNM7XLPZSU6MA, seg_pos={0 0}, upload=541223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:41692->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:41692->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41692->77.90.31.14:4661: use of closed network connection Dec 31 09:05:38.003981139 (617h36m37.875064174s) [noise=true , address=77.90.31.14:4661 , piece_id=X4ZAWVUEIK2RVDMGFG65GTJMPYP577VJI344243FNM7XLPZSU6MA, seg_pos={0 0}, upload=541223]: Long tail cancellation due to optimal threshold Dec 31 09:05:38.004094465 (617h36m37.8751775s) [noise=true , address=45.91.116.11:4120 , piece_id=3QP553L5EQUDK2W2E4CG4LGB2OVTI6BGU7IFKBCBUCT2ZNRWTH2A, seg_pos={0 0}, upload=541223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 31 09:05:38.004102440 (617h36m37.875185475s) [noise=true , address=45.91.116.11:4120 , piece_id=3QP553L5EQUDK2W2E4CG4LGB2OVTI6BGU7IFKBCBUCT2ZNRWTH2A, seg_pos={0 0}, upload=541223]: Long tail cancellation due to optimal threshold Dec 31 09:05:38.004119493 (617h36m37.875202528s) [seg_pos={0 0}, upload=541223]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 09:05:38.004248409 (617h36m37.875331454s) [upload=541223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:05:38.004255452 (617h36m37.875338487s) [upload=541223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:05:38.292796395 (617h36m38.16387946s) [upload=541223]: Done uploading segments... Dec 31 09:05:38.292847051 (617h36m38.163930106s) [upload=541223]: Upload finished. err: <nil> Duration: 30.003550079s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [X...................X.......................................] duration: 31.542853528s trace: https://jaeger.dev.storj.io/trace/26257de5bcb7c920 FAIL (2025-12-31 09:05:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== , upload=540795]: Uploading piece... Dec 31 08:35:48.315166374 (617h6m48.186249409s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.315823664 (617h6m48.186906699s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316244806 (617h6m48.187327841s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316251710 (617h6m48.187334755s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316290584 (617h6m48.187373619s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317050111 (617h6m48.188133146s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317140853 (617h6m48.188223888s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318123574 (617h6m48.189206609s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318885204 (617h6m48.189968249s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.319835805 (617h6m48.19091885s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.320309847 (617h6m48.191392882s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321148273 (617h6m48.192231308s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321502088 (617h6m48.192585133s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322658430 (617h6m48.193741495s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322775803 (617h6m48.193858848s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322882035 (617h6m48.19396507s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.323403308 (617h6m48.194486343s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:49.768189393 (617h6m49.639272428s) [upload=540795]: Next returned nil segment Dec 31 08:35:49.768201797 (617h6m49.639284832s) [upload=540795]: Waiting for error group managing segments... Dec 31 08:35:50.197657834 (617h6m50.068740869s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.209847924 (617h6m50.08093096s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.217381621 (617h6m50.088464666s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.270225653 (617h6m50.141308698s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.411342630 (617h6m50.282425665s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473471693 (617h6m50.344554738s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473607101 (617h6m50.344690136s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.479010038 (617h6m50.350093074s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.482534590 (617h6m50.353617625s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.485954982 (617h6m50.357038017s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.058742323 (617h6m50.929825358s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.071853740 (617h6m50.942936775s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.177856875 (617h6m51.04893991s) [noise=true , address=145.79.227.33:4873 , piece_id=KWLS4DUDNAEADPUA3CTATKK4MEQSUPEIPYJS3SSOGVPDOJPJHDIQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179660939 (617h6m51.050743984s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179686989 (617h6m51.050770024s) [seg_pos={0 0}, upload=540795]: Segment reached optimal threshold of 14 pieces. Dec 31 08:35:51.179944619 (617h6m51.051027684s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.179956683 (617h6m51.051039728s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180323110 (617h6m51.051406155s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180346325 (617h6m51.05142937s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180485580 (617h6m51.051568615s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180495899 (617h6m51.051578934s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180529313 (617h6m51.051612358s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180536927 (617h6m51.051619972s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180560653 (617h6m51.051643698s) [seg_pos={0 0}, upload=540795]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 08:35:51.180641827 (617h6m51.051724872s) [upload=540795]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 08:35:51.180647598 (617h6m51.051730643s) [upload=540795]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 08:35:51.479648850 (617h6m51.350731896s) [upload=540795]: Done uploading segments... Dec 31 08:35:51.479871193 (617h6m51.350954228s) [upload=540795]: Upload finished. err: <nil> Duration: 31.542883254s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.......................X...........X....X....XXX.XX.........] duration: 3.556695393s trace: https://jaeger.dev.storj.io/trace/3c18fb1b63204f7a FAIL (2025-12-31 09:05:18 UTC) orbiter warning: context deadline exceeded Duration: 3.556723657s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..X..........X....X.X....X...XXX..X.X.....X........XX....X..] duration: 5.984576743s trace: https://jaeger.dev.storj.io/trace/74f20ce86fac7c8b FAIL (2025-12-31 09:04:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: DHUDYJTLZ6CBWDZ4LE5O2JZRC5MDMCTQS3YRP7CANTALXYTFMOXA) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: YWHKPVJZVB66ETI5FECWHQ25INWXM2UTDJXHHQWKQK7HRMA3HEXQ) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: IMNPWFVW462ACRWXMA6PXXPNTGXOSFMT7UOE2K55354J7WJVSEDQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: UNWZ35J2LAFCDJAHI7ERXRBSWHKPPUEKYCK3FFVZJ7MMR6W42SWA) context deadline exceeded ======LOGS====== Dec 31 09:02:53.561273706 (617h33m53.432356751s) [upload=541180]: Starting upload Dec 31 09:02:53.561367975 (617h33m53.43245102s) [upload=541180]: Done starting upload Dec 31 09:02:53.561458178 (617h33m53.432541223s) [upload=541180]: Uploading segments... Dec 31 09:02:53.561571132 (617h33m53.432654177s) [upload=541180]: Got next segment. Inline: false Dec 31 09:02:53.561613363 (617h33m53.432696408s) [upload=541180]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 09:02:53.561618472 (617h33m53.432701507s) [upload=541180]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 09:02:53.807298131 (617h33m53.678381166s) [seg_pos={0 0}, upload=541180]: Begin upload segment... Dec 31 09:02:53.807356573 (617h33m53.678439608s) [seg_pos={0 0}, upload=541180]: Done begin upload segment. Dec 31 09:02:53.807390377 (617h33m53.678473422s) [noise=true , address=77.90.31.16:4735 , piece_id=MUAMAUXQ6PZWIP56Y3QCA6B6QCLAASXJ7KBLOVZWF7R272VY44JQ, seg_pos={0 0}, upload=541180]: Uploading piece... Dec 31 09:02:53.807420443 (617h33m53.678503489s) [noise=true , address=145.79.227.37:4566 , piece_id=WG4KH2GGG6PNJAMAXP326F6TTTOJ7KXIILFKEMXPD42UMD5GY5AA, seg_pos={0 0}, upload=541180]: Uploading piece... Dec 31 09:02:53.807537617 (617h33m53.678620662s) [noise=true , address=145.79.227.20:4437 , piece_id=TISY3WR4VMK66BH7NTYQYR676TUOR276THO4SXONNJ4MEGTLUG5A, seg_pos={0 0}, upload=541180]: Uploading piece... Dec 31 09:02:53.807874658 (617h33m53.678957703s) [noise=true , address=77.90.31.21:4880 , piece_id=UNWZ35J2LAFCDJAHI7ERXRBSWHKPPUEKYCK3FFVZJ7MMR6W42SWA, seg_pos={0 0}, upload=541180]: Uploading piece... Dec 31 09:02:53.808167767 (617h33m53.679250812s) [noise=true , address=77.90.31.11:4556 , piece_id=5J7AGFAGS2JZKSPYFXOIVAT4UAKVS4JEL55W5X36UT3N66TOYXLA, seg_pos={0 0}, upload=541180]: Uploading piece... Dec 31 09:02:53.808522232 (617h33m53.679605267s) [noise=true , address=145.79.227.15:4874 , piece_id=NTP2XYGONMVYBWRBPHBXBSXADZNLZ5JD7R62OLL2XCRK2FMBT6QA, seg_pos={0 0}, upload=541180]: Uploading piece... Dec 31 09:02:53.808806333 (617h33m53.679889368s) [noise=true , address=45.91.116.11:4120 , piece_id=DHUDYJTLZ6CBWDZ4LE5O2JZRC5MDMCTQS3YRP7CANTALXYTFMOXA, seg_pos={0 0}, upload=541180]: Uploading piece... Dec 31 09:02:53.809274284 (617h33m53.680357319s) [noise=true , address=45.91.116.4:4937 , piece_id=KVI533MMNG4XWKYWFL5GUYYNW7GDNKTNV7XHS3G4SED263AREBOQ, seg_pos={0 0}, upload=541180]: Uploading piece... Dec 31 09:02:53.809642665 (617h33m53.6807257s) [noise=true , address=145.79.227.38:4979 , piece_id=JADBBBSZ4GXW75ACP4E5KJDESKS6QDRS5XRHQ364QCZ4DFXU36UQ, seg_pos={0 0}, upload=541180]: Uploading piece... Dec 31 09:02:53.810058627 (617h33m53.681141662s) [noise=true , address=145.79.227.44:4193 , piece_id=SRQA6TYQKOZRRF6JSNZIC5XAGCW6B56EUMFGJ3OSJRXUQAXRSVJQ, seg_pos={0 0}, upload=541180]: Uploading piece... Dec 31 09:02:53.811622696 (617h33m53.682705731s) [noise=true , address=77.90.31.15:4700 , piece_id=YWHKPVJZVB66ETI5FECWHQ25INWXM2UTDJXHHQWKQK7HRMA3HEXQ, seg_pos={0 0}, upload=541180]: Uploading piece... Dec 31 09:02:53.812838250 (617h33m53.683921305s) [noise=true , address=45.91.116.12:4225 , piece_id=IMNPWFVW462ACRWXMA6PXXPNTGXOSFMT7UOE2K55354J7WJVSEDQ, seg_pos={0 0}, upload=541180]: Uploading piece... Dec 31 09:02:53.814330341 (617h33m53.685413376s) [noise=true , address=145.79.227.52:4445 , piece_id=52BJTIOOJ6PEFUQSZRSY6EFCT4ATHXPUVR7LXN74JLDIV35PITDA, seg_pos={0 0}, upload=541180]: Uploading piece... Dec 31 09:02:53.815651687 (617h33m53.686734722s) [noise=true , address=145.79.227.47:4265 , piece_id=JT7WQCY5OVACMEG6FKTZ6N4IA6FK3WQGHJ3NCHMS7WJO6ZTH346Q, seg_pos={0 0}, upload=541180]: Uploading piece... Dec 31 09:02:53.817304625 (617h33m53.68838767s) [noise=true , address=145.79.227.7:4295 , piece_id=YNCEMIJXTG3RHB4JN477YSGZMV422M2QS76TXOZ3Q5VOWWO7MSPQ, seg_pos={0 0}, upload=541180]: Uploading piece... Dec 31 09:02:53.818937003 (617h33m53.690020068s) [noise=true , address=145.79.227.40:4050 , piece_id=OMMSJWVDDMY3LC3527EXCDXHDQPASCVV7BPDPQI362VYMSCZ6MPA, seg_pos={0 0}, upload=541180]: Uploading piece... Dec 31 09:02:55.513418016 (617h33m55.384501051s) [upload=541180]: Next returned nil segment Dec 31 09:02:55.513469343 (617h33m55.384552378s) [upload=541180]: Waiting for error group managing segments... Dec 31 09:02:55.798713358 (617h33m55.669796403s) [noise=true , address=145.79.227.37:4566 , piece_id=WG4KH2GGG6PNJAMAXP326F6TTTOJ7KXIILFKEMXPD42UMD5GY5AA, seg_pos={0 0}, upload=541180]: Done uploading piece. err: <nil> Dec 31 09:02:55.801325341 (617h33m55.672408386s) [noise=true , address=145.79.227.7:4295 , piece_id=YNCEMIJXTG3RHB4JN477YSGZMV422M2QS76TXOZ3Q5VOWWO7MSPQ, seg_pos={0 0}, upload=541180]: Done uploading piece. err: <nil> Dec 31 09:02:55.802561185 (617h33m55.67364423s) [noise=true , address=145.79.227.15:4874 , piece_id=NTP2XYGONMVYBWRBPHBXBSXADZNLZ5JD7R62OLL2XCRK2FMBT6QA, seg_pos={0 0}, upload=541180]: Done uploading piece. err: <nil> Dec 31 09:02:55.808591708 (617h33m55.679674743s) [noise=true , address=145.79.227.52:4445 , piece_id=52BJTIOOJ6PEFUQSZRSY6EFCT4ATHXPUVR7LXN74JLDIV35PITDA, seg_pos={0 0}, upload=541180]: Done uploading piece. err: <nil> Dec 31 09:02:55.815086364 (617h33m55.686169399s) [noise=true , address=145.79.227.47:4265 , piece_id=JT7WQCY5OVACMEG6FKTZ6N4IA6FK3WQGHJ3NCHMS7WJO6ZTH346Q, seg_pos={0 0}, upload=541180]: Done uploading piece. err: <nil> Dec 31 09:02:55.832005872 (617h33m55.703088917s) [noise=true , address=145.79.227.20:4437 , piece_id=TISY3WR4VMK66BH7NTYQYR676TUOR276THO4SXONNJ4MEGTLUG5A, seg_pos={0 0}, upload=541180]: Done uploading piece. err: <nil> Dec 31 09:02:55.878536272 (617h33m55.749619317s) [noise=true , address=145.79.227.38:4979 , piece_id=JADBBBSZ4GXW75ACP4E5KJDESKS6QDRS5XRHQ364QCZ4DFXU36UQ, seg_pos={0 0}, upload=541180]: Done uploading piece. err: <nil> Dec 31 09:02:55.959711070 (617h33m55.830794105s) [noise=true , address=77.90.31.11:4556 , piece_id=5J7AGFAGS2JZKSPYFXOIVAT4UAKVS4JEL55W5X36UT3N66TOYXLA, seg_pos={0 0}, upload=541180]: Done uploading piece. err: <nil> Dec 31 09:02:56.036089085 (617h33m55.90717213s) [noise=true , address=77.90.31.21:4880 , piece_id=UNWZ35J2LAFCDJAHI7ERXRBSWHKPPUEKYCK3FFVZJ7MMR6W42SWA, seg_pos={0 0}, upload=541180]: Done uploading piece. err: <nil> Dec 31 09:02:56.103323819 (617h33m55.974406854s) [noise=true , address=77.90.31.15:4700 , piece_id=YWHKPVJZVB66ETI5FECWHQ25INWXM2UTDJXHHQWKQK7HRMA3HEXQ, seg_pos={0 0}, upload=541180]: Done uploading piece. err: <nil> Dec 31 09:02:56.386490739 (617h33m56.257573774s) [noise=true , address=45.91.116.11:4120 , piece_id=DHUDYJTLZ6CBWDZ4LE5O2JZRC5MDMCTQS3YRP7CANTALXYTFMOXA, seg_pos={0 0}, upload=541180]: Done uploading piece. err: <nil> Dec 31 09:02:56.433793821 (617h33m56.304876856s) [noise=true , address=45.91.116.12:4225 , piece_id=IMNPWFVW462ACRWXMA6PXXPNTGXOSFMT7UOE2K55354J7WJVSEDQ, seg_pos={0 0}, upload=541180]: Done uploading piece. err: <nil> Dec 31 09:02:56.713331479 (617h33m56.584414515s) [noise=true , address=77.90.31.16:4735 , piece_id=MUAMAUXQ6PZWIP56Y3QCA6B6QCLAASXJ7KBLOVZWF7R272VY44JQ, seg_pos={0 0}, upload=541180]: Done uploading piece. err: <nil> Dec 31 09:02:56.713354283 (617h33m56.584437318s) [seg_pos={0 0}, upload=541180]: Segment reached optimal threshold of 13 pieces. Dec 31 09:02:56.713578029 (617h33m56.584661084s) [noise=true , address=145.79.227.44:4193 , piece_id=SRQA6TYQKOZRRF6JSNZIC5XAGCW6B56EUMFGJ3OSJRXUQAXRSVJQ, seg_pos={0 0}, upload=541180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 31 09:02:56.713586726 (617h33m56.584669781s) [noise=true , address=145.79.227.44:4193 , piece_id=SRQA6TYQKOZRRF6JSNZIC5XAGCW6B56EUMFGJ3OSJRXUQAXRSVJQ, seg_pos={0 0}, upload=541180]: Long tail cancellation due to optimal threshold Dec 31 09:02:56.713852932 (617h33m56.584935968s) [noise=true , address=45.91.116.4:4937 , piece_id=KVI533MMNG4XWKYWFL5GUYYNW7GDNKTNV7XHS3G4SED263AREBOQ, seg_pos={0 0}, upload=541180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 31 09:02:56.713863532 (617h33m56.584946567s) [noise=true , address=45.91.116.4:4937 , piece_id=KVI533MMNG4XWKYWFL5GUYYNW7GDNKTNV7XHS3G4SED263AREBOQ, seg_pos={0 0}, upload=541180]: Long tail cancellation due to optimal threshold Dec 31 09:02:56.713903299 (617h33m56.584986374s) [noise=true , address=145.79.227.40:4050 , piece_id=OMMSJWVDDMY3LC3527EXCDXHDQPASCVV7BPDPQI362VYMSCZ6MPA, seg_pos={0 0}, upload=541180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 31 09:02:56.713926202 (617h33m56.585009267s) [noise=true , address=145.79.227.40:4050 , piece_id=OMMSJWVDDMY3LC3527EXCDXHDQPASCVV7BPDPQI362VYMSCZ6MPA, seg_pos={0 0}, upload=541180]: Long tail cancellation due to optimal threshold Dec 31 09:02:56.713964996 (617h33m56.585048041s) [seg_pos={0 0}, upload=541180]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 09:02:56.714059526 (617h33m56.585142561s) [upload=541180]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:02:56.714066619 (617h33m56.585149654s) [upload=541180]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:02:56.997894066 (617h33m56.868977101s) [upload=541180]: Done uploading segments... Dec 31 09:02:56.997912231 (617h33m56.868995276s) [upload=541180]: Upload finished. err: <nil> Duration: 5.984603313s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [........................X...........X....X....XXX.XX........] duration: 3.517924216s trace: https://jaeger.dev.storj.io/trace/3c18fb1b63204f7a FAIL (2025-12-31 09:04:22 UTC) orbiter warning: context deadline exceeded Duration: 3.517950476s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X...................X......................................] duration: 22.104055879s trace: https://jaeger.dev.storj.io/trace/26257de5bcb7c920 FAIL (2025-12-31 09:04:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== , upload=540795]: Uploading piece... Dec 31 08:35:48.315166374 (617h6m48.186249409s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.315823664 (617h6m48.186906699s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316244806 (617h6m48.187327841s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316251710 (617h6m48.187334755s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316290584 (617h6m48.187373619s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317050111 (617h6m48.188133146s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317140853 (617h6m48.188223888s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318123574 (617h6m48.189206609s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318885204 (617h6m48.189968249s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.319835805 (617h6m48.19091885s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.320309847 (617h6m48.191392882s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321148273 (617h6m48.192231308s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321502088 (617h6m48.192585133s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322658430 (617h6m48.193741495s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322775803 (617h6m48.193858848s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322882035 (617h6m48.19396507s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.323403308 (617h6m48.194486343s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:49.768189393 (617h6m49.639272428s) [upload=540795]: Next returned nil segment Dec 31 08:35:49.768201797 (617h6m49.639284832s) [upload=540795]: Waiting for error group managing segments... Dec 31 08:35:50.197657834 (617h6m50.068740869s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.209847924 (617h6m50.08093096s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.217381621 (617h6m50.088464666s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.270225653 (617h6m50.141308698s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.411342630 (617h6m50.282425665s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473471693 (617h6m50.344554738s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473607101 (617h6m50.344690136s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.479010038 (617h6m50.350093074s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.482534590 (617h6m50.353617625s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.485954982 (617h6m50.357038017s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.058742323 (617h6m50.929825358s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.071853740 (617h6m50.942936775s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.177856875 (617h6m51.04893991s) [noise=true , address=145.79.227.33:4873 , piece_id=KWLS4DUDNAEADPUA3CTATKK4MEQSUPEIPYJS3SSOGVPDOJPJHDIQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179660939 (617h6m51.050743984s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179686989 (617h6m51.050770024s) [seg_pos={0 0}, upload=540795]: Segment reached optimal threshold of 14 pieces. Dec 31 08:35:51.179944619 (617h6m51.051027684s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.179956683 (617h6m51.051039728s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180323110 (617h6m51.051406155s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180346325 (617h6m51.05142937s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180485580 (617h6m51.051568615s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180495899 (617h6m51.051578934s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180529313 (617h6m51.051612358s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180536927 (617h6m51.051619972s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180560653 (617h6m51.051643698s) [seg_pos={0 0}, upload=540795]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 08:35:51.180641827 (617h6m51.051724872s) [upload=540795]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 08:35:51.180647598 (617h6m51.051730643s) [upload=540795]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 08:35:51.479648850 (617h6m51.350731896s) [upload=540795]: Done uploading segments... Dec 31 08:35:51.479871193 (617h6m51.350954228s) [upload=540795]: Upload finished. err: <nil> Duration: 22.104084304s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [...X..........X....X.X....X...XXX..X.X.....X........XX....X.] duration: 5.746010777s trace: https://jaeger.dev.storj.io/trace/74f20ce86fac7c8b FAIL (2025-12-31 09:04:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: DHUDYJTLZ6CBWDZ4LE5O2JZRC5MDMCTQS3YRP7CANTALXYTFMOXA) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: YWHKPVJZVB66ETI5FECWHQ25INWXM2UTDJXHHQWKQK7HRMA3HEXQ) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: IMNPWFVW462ACRWXMA6PXXPNTGXOSFMT7UOE2K55354J7WJVSEDQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: UNWZ35J2LAFCDJAHI7ERXRBSWHKPPUEKYCK3FFVZJ7MMR6W42SWA) context deadline exceeded ======LOGS====== Dec 31 09:02:53.561273706 (617h33m53.432356751s) [upload=541180]: Starting upload Dec 31 09:02:53.561367975 (617h33m53.43245102s) [upload=541180]: Done starting upload Dec 31 09:02:53.561458178 (617h33m53.432541223s) [upload=541180]: Uploading segments... Dec 31 09:02:53.561571132 (617h33m53.432654177s) [upload=541180]: Got next segment. Inline: false Dec 31 09:02:53.561613363 (617h33m53.432696408s) [upload=541180]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 09:02:53.561618472 (617h33m53.432701507s) [upload=541180]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 09:02:53.807298131 (617h33m53.678381166s) [seg_pos={0 0}, upload=541180]: Begin upload segment... Dec 31 09:02:53.807356573 (617h33m53.678439608s) [seg_pos={0 0}, upload=541180]: Done begin upload segment. Dec 31 09:02:53.807390377 (617h33m53.678473422s) [noise=true , address=77.90.31.16:4735 , piece_id=MUAMAUXQ6PZWIP56Y3QCA6B6QCLAASXJ7KBLOVZWF7R272VY44JQ, seg_pos={0 0}, upload=541180]: Uploading piece... Dec 31 09:02:53.807420443 (617h33m53.678503489s) [noise=true , address=145.79.227.37:4566 , piece_id=WG4KH2GGG6PNJAMAXP326F6TTTOJ7KXIILFKEMXPD42UMD5GY5AA, seg_pos={0 0}, upload=541180]: Uploading piece... Dec 31 09:02:53.807537617 (617h33m53.678620662s) [noise=true , address=145.79.227.20:4437 , piece_id=TISY3WR4VMK66BH7NTYQYR676TUOR276THO4SXONNJ4MEGTLUG5A, seg_pos={0 0}, upload=541180]: Uploading piece... Dec 31 09:02:53.807874658 (617h33m53.678957703s) [noise=true , address=77.90.31.21:4880 , piece_id=UNWZ35J2LAFCDJAHI7ERXRBSWHKPPUEKYCK3FFVZJ7MMR6W42SWA, seg_pos={0 0}, upload=541180]: Uploading piece... Dec 31 09:02:53.808167767 (617h33m53.679250812s) [noise=true , address=77.90.31.11:4556 , piece_id=5J7AGFAGS2JZKSPYFXOIVAT4UAKVS4JEL55W5X36UT3N66TOYXLA, seg_pos={0 0}, upload=541180]: Uploading piece... Dec 31 09:02:53.808522232 (617h33m53.679605267s) [noise=true , address=145.79.227.15:4874 , piece_id=NTP2XYGONMVYBWRBPHBXBSXADZNLZ5JD7R62OLL2XCRK2FMBT6QA, seg_pos={0 0}, upload=541180]: Uploading piece... Dec 31 09:02:53.808806333 (617h33m53.679889368s) [noise=true , address=45.91.116.11:4120 , piece_id=DHUDYJTLZ6CBWDZ4LE5O2JZRC5MDMCTQS3YRP7CANTALXYTFMOXA, seg_pos={0 0}, upload=541180]: Uploading piece... Dec 31 09:02:53.809274284 (617h33m53.680357319s) [noise=true , address=45.91.116.4:4937 , piece_id=KVI533MMNG4XWKYWFL5GUYYNW7GDNKTNV7XHS3G4SED263AREBOQ, seg_pos={0 0}, upload=541180]: Uploading piece... Dec 31 09:02:53.809642665 (617h33m53.6807257s) [noise=true , address=145.79.227.38:4979 , piece_id=JADBBBSZ4GXW75ACP4E5KJDESKS6QDRS5XRHQ364QCZ4DFXU36UQ, seg_pos={0 0}, upload=541180]: Uploading piece... Dec 31 09:02:53.810058627 (617h33m53.681141662s) [noise=true , address=145.79.227.44:4193 , piece_id=SRQA6TYQKOZRRF6JSNZIC5XAGCW6B56EUMFGJ3OSJRXUQAXRSVJQ, seg_pos={0 0}, upload=541180]: Uploading piece... Dec 31 09:02:53.811622696 (617h33m53.682705731s) [noise=true , address=77.90.31.15:4700 , piece_id=YWHKPVJZVB66ETI5FECWHQ25INWXM2UTDJXHHQWKQK7HRMA3HEXQ, seg_pos={0 0}, upload=541180]: Uploading piece... Dec 31 09:02:53.812838250 (617h33m53.683921305s) [noise=true , address=45.91.116.12:4225 , piece_id=IMNPWFVW462ACRWXMA6PXXPNTGXOSFMT7UOE2K55354J7WJVSEDQ, seg_pos={0 0}, upload=541180]: Uploading piece... Dec 31 09:02:53.814330341 (617h33m53.685413376s) [noise=true , address=145.79.227.52:4445 , piece_id=52BJTIOOJ6PEFUQSZRSY6EFCT4ATHXPUVR7LXN74JLDIV35PITDA, seg_pos={0 0}, upload=541180]: Uploading piece... Dec 31 09:02:53.815651687 (617h33m53.686734722s) [noise=true , address=145.79.227.47:4265 , piece_id=JT7WQCY5OVACMEG6FKTZ6N4IA6FK3WQGHJ3NCHMS7WJO6ZTH346Q, seg_pos={0 0}, upload=541180]: Uploading piece... Dec 31 09:02:53.817304625 (617h33m53.68838767s) [noise=true , address=145.79.227.7:4295 , piece_id=YNCEMIJXTG3RHB4JN477YSGZMV422M2QS76TXOZ3Q5VOWWO7MSPQ, seg_pos={0 0}, upload=541180]: Uploading piece... Dec 31 09:02:53.818937003 (617h33m53.690020068s) [noise=true , address=145.79.227.40:4050 , piece_id=OMMSJWVDDMY3LC3527EXCDXHDQPASCVV7BPDPQI362VYMSCZ6MPA, seg_pos={0 0}, upload=541180]: Uploading piece... Dec 31 09:02:55.513418016 (617h33m55.384501051s) [upload=541180]: Next returned nil segment Dec 31 09:02:55.513469343 (617h33m55.384552378s) [upload=541180]: Waiting for error group managing segments... Dec 31 09:02:55.798713358 (617h33m55.669796403s) [noise=true , address=145.79.227.37:4566 , piece_id=WG4KH2GGG6PNJAMAXP326F6TTTOJ7KXIILFKEMXPD42UMD5GY5AA, seg_pos={0 0}, upload=541180]: Done uploading piece. err: <nil> Dec 31 09:02:55.801325341 (617h33m55.672408386s) [noise=true , address=145.79.227.7:4295 , piece_id=YNCEMIJXTG3RHB4JN477YSGZMV422M2QS76TXOZ3Q5VOWWO7MSPQ, seg_pos={0 0}, upload=541180]: Done uploading piece. err: <nil> Dec 31 09:02:55.802561185 (617h33m55.67364423s) [noise=true , address=145.79.227.15:4874 , piece_id=NTP2XYGONMVYBWRBPHBXBSXADZNLZ5JD7R62OLL2XCRK2FMBT6QA, seg_pos={0 0}, upload=541180]: Done uploading piece. err: <nil> Dec 31 09:02:55.808591708 (617h33m55.679674743s) [noise=true , address=145.79.227.52:4445 , piece_id=52BJTIOOJ6PEFUQSZRSY6EFCT4ATHXPUVR7LXN74JLDIV35PITDA, seg_pos={0 0}, upload=541180]: Done uploading piece. err: <nil> Dec 31 09:02:55.815086364 (617h33m55.686169399s) [noise=true , address=145.79.227.47:4265 , piece_id=JT7WQCY5OVACMEG6FKTZ6N4IA6FK3WQGHJ3NCHMS7WJO6ZTH346Q, seg_pos={0 0}, upload=541180]: Done uploading piece. err: <nil> Dec 31 09:02:55.832005872 (617h33m55.703088917s) [noise=true , address=145.79.227.20:4437 , piece_id=TISY3WR4VMK66BH7NTYQYR676TUOR276THO4SXONNJ4MEGTLUG5A, seg_pos={0 0}, upload=541180]: Done uploading piece. err: <nil> Dec 31 09:02:55.878536272 (617h33m55.749619317s) [noise=true , address=145.79.227.38:4979 , piece_id=JADBBBSZ4GXW75ACP4E5KJDESKS6QDRS5XRHQ364QCZ4DFXU36UQ, seg_pos={0 0}, upload=541180]: Done uploading piece. err: <nil> Dec 31 09:02:55.959711070 (617h33m55.830794105s) [noise=true , address=77.90.31.11:4556 , piece_id=5J7AGFAGS2JZKSPYFXOIVAT4UAKVS4JEL55W5X36UT3N66TOYXLA, seg_pos={0 0}, upload=541180]: Done uploading piece. err: <nil> Dec 31 09:02:56.036089085 (617h33m55.90717213s) [noise=true , address=77.90.31.21:4880 , piece_id=UNWZ35J2LAFCDJAHI7ERXRBSWHKPPUEKYCK3FFVZJ7MMR6W42SWA, seg_pos={0 0}, upload=541180]: Done uploading piece. err: <nil> Dec 31 09:02:56.103323819 (617h33m55.974406854s) [noise=true , address=77.90.31.15:4700 , piece_id=YWHKPVJZVB66ETI5FECWHQ25INWXM2UTDJXHHQWKQK7HRMA3HEXQ, seg_pos={0 0}, upload=541180]: Done uploading piece. err: <nil> Dec 31 09:02:56.386490739 (617h33m56.257573774s) [noise=true , address=45.91.116.11:4120 , piece_id=DHUDYJTLZ6CBWDZ4LE5O2JZRC5MDMCTQS3YRP7CANTALXYTFMOXA, seg_pos={0 0}, upload=541180]: Done uploading piece. err: <nil> Dec 31 09:02:56.433793821 (617h33m56.304876856s) [noise=true , address=45.91.116.12:4225 , piece_id=IMNPWFVW462ACRWXMA6PXXPNTGXOSFMT7UOE2K55354J7WJVSEDQ, seg_pos={0 0}, upload=541180]: Done uploading piece. err: <nil> Dec 31 09:02:56.713331479 (617h33m56.584414515s) [noise=true , address=77.90.31.16:4735 , piece_id=MUAMAUXQ6PZWIP56Y3QCA6B6QCLAASXJ7KBLOVZWF7R272VY44JQ, seg_pos={0 0}, upload=541180]: Done uploading piece. err: <nil> Dec 31 09:02:56.713354283 (617h33m56.584437318s) [seg_pos={0 0}, upload=541180]: Segment reached optimal threshold of 13 pieces. Dec 31 09:02:56.713578029 (617h33m56.584661084s) [noise=true , address=145.79.227.44:4193 , piece_id=SRQA6TYQKOZRRF6JSNZIC5XAGCW6B56EUMFGJ3OSJRXUQAXRSVJQ, seg_pos={0 0}, upload=541180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 31 09:02:56.713586726 (617h33m56.584669781s) [noise=true , address=145.79.227.44:4193 , piece_id=SRQA6TYQKOZRRF6JSNZIC5XAGCW6B56EUMFGJ3OSJRXUQAXRSVJQ, seg_pos={0 0}, upload=541180]: Long tail cancellation due to optimal threshold Dec 31 09:02:56.713852932 (617h33m56.584935968s) [noise=true , address=45.91.116.4:4937 , piece_id=KVI533MMNG4XWKYWFL5GUYYNW7GDNKTNV7XHS3G4SED263AREBOQ, seg_pos={0 0}, upload=541180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 31 09:02:56.713863532 (617h33m56.584946567s) [noise=true , address=45.91.116.4:4937 , piece_id=KVI533MMNG4XWKYWFL5GUYYNW7GDNKTNV7XHS3G4SED263AREBOQ, seg_pos={0 0}, upload=541180]: Long tail cancellation due to optimal threshold Dec 31 09:02:56.713903299 (617h33m56.584986374s) [noise=true , address=145.79.227.40:4050 , piece_id=OMMSJWVDDMY3LC3527EXCDXHDQPASCVV7BPDPQI362VYMSCZ6MPA, seg_pos={0 0}, upload=541180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 31 09:02:56.713926202 (617h33m56.585009267s) [noise=true , address=145.79.227.40:4050 , piece_id=OMMSJWVDDMY3LC3527EXCDXHDQPASCVV7BPDPQI362VYMSCZ6MPA, seg_pos={0 0}, upload=541180]: Long tail cancellation due to optimal threshold Dec 31 09:02:56.713964996 (617h33m56.585048041s) [seg_pos={0 0}, upload=541180]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 09:02:56.714059526 (617h33m56.585142561s) [upload=541180]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:02:56.714066619 (617h33m56.585149654s) [upload=541180]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:02:56.997894066 (617h33m56.868977101s) [upload=541180]: Done uploading segments... Dec 31 09:02:56.997912231 (617h33m56.868995276s) [upload=541180]: Upload finished. err: <nil> Duration: 5.7460385s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X...................X.....................................] duration: 36.085083095s trace: https://jaeger.dev.storj.io/trace/26257de5bcb7c920 FAIL (2025-12-31 09:03:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== , upload=540795]: Uploading piece... Dec 31 08:35:48.315166374 (617h6m48.186249409s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.315823664 (617h6m48.186906699s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316244806 (617h6m48.187327841s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316251710 (617h6m48.187334755s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316290584 (617h6m48.187373619s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317050111 (617h6m48.188133146s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317140853 (617h6m48.188223888s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318123574 (617h6m48.189206609s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318885204 (617h6m48.189968249s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.319835805 (617h6m48.19091885s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.320309847 (617h6m48.191392882s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321148273 (617h6m48.192231308s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321502088 (617h6m48.192585133s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322658430 (617h6m48.193741495s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322775803 (617h6m48.193858848s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322882035 (617h6m48.19396507s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.323403308 (617h6m48.194486343s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:49.768189393 (617h6m49.639272428s) [upload=540795]: Next returned nil segment Dec 31 08:35:49.768201797 (617h6m49.639284832s) [upload=540795]: Waiting for error group managing segments... Dec 31 08:35:50.197657834 (617h6m50.068740869s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.209847924 (617h6m50.08093096s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.217381621 (617h6m50.088464666s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.270225653 (617h6m50.141308698s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.411342630 (617h6m50.282425665s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473471693 (617h6m50.344554738s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473607101 (617h6m50.344690136s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.479010038 (617h6m50.350093074s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.482534590 (617h6m50.353617625s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.485954982 (617h6m50.357038017s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.058742323 (617h6m50.929825358s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.071853740 (617h6m50.942936775s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.177856875 (617h6m51.04893991s) [noise=true , address=145.79.227.33:4873 , piece_id=KWLS4DUDNAEADPUA3CTATKK4MEQSUPEIPYJS3SSOGVPDOJPJHDIQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179660939 (617h6m51.050743984s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179686989 (617h6m51.050770024s) [seg_pos={0 0}, upload=540795]: Segment reached optimal threshold of 14 pieces. Dec 31 08:35:51.179944619 (617h6m51.051027684s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.179956683 (617h6m51.051039728s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180323110 (617h6m51.051406155s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180346325 (617h6m51.05142937s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180485580 (617h6m51.051568615s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180495899 (617h6m51.051578934s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180529313 (617h6m51.051612358s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180536927 (617h6m51.051619972s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180560653 (617h6m51.051643698s) [seg_pos={0 0}, upload=540795]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 08:35:51.180641827 (617h6m51.051724872s) [upload=540795]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 08:35:51.180647598 (617h6m51.051730643s) [upload=540795]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 08:35:51.479648850 (617h6m51.350731896s) [upload=540795]: Done uploading segments... Dec 31 08:35:51.479871193 (617h6m51.350954228s) [upload=540795]: Upload finished. err: <nil> Duration: 36.085098786s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.........................X...........X....X....XXX.XX.......] duration: 3.535511131s trace: https://jaeger.dev.storj.io/trace/3c18fb1b63204f7a FAIL (2025-12-31 09:03:38 UTC) orbiter warning: context deadline exceeded Duration: 3.535544133s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)