Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXX] duration: 5.001213928s trace: https://jaeger.dev.storj.io/trace/2c549373858e819 FAIL (2025-12-29 12:45:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001234458s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X.......X..........X....................] duration: 42.834542716s trace: https://jaeger.dev.storj.io/trace/68764c6cb4286314 FAIL (2025-12-29 12:45:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528204662 (572h59m20.399287707s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528497489 (572h59m20.399580524s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528793153 (572h59m20.399876198s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529083555 (572h59m20.40016659s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529391081 (572h59m20.400474126s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529735096 (572h59m20.400818131s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530109970 (572h59m20.401193015s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530449556 (572h59m20.401532591s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530831554 (572h59m20.401914599s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531229512 (572h59m20.402312557s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531698945 (572h59m20.402781991s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532204238 (572h59m20.403287283s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532744366 (572h59m20.403827401s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533231203 (572h59m20.404314248s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533740113 (572h59m20.404823148s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:21.876539279 (572h59m21.747622324s) [upload=502322]: Next returned nil segment Dec 29 12:28:21.876614272 (572h59m21.747697307s) [upload=502322]: Waiting for error group managing segments... Dec 29 12:28:22.383674429 (572h59m22.254757464s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389083709 (572h59m22.260166754s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389450968 (572h59m22.260534003s) [noise=true , address=145.79.227.46:4625 , piece_id=6NEND2CKGIZLK5Y6IMM4XB6SXPSXUYSZIVZ7CZ25IKAHFFW33C5Q, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.418373151 (572h59m22.289456186s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.619432477 (572h59m22.490515552s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.741876388 (572h59m22.612959423s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.133731187 (572h59m23.004814222s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.146041929 (572h59m23.017124964s) [noise=true , address=193.149.164.143:4697 , piece_id=HZZY3SUOEC7AIHSW6MKZRNZVSOGXSYBYSUKN7STBFTXWAOMDRYBA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.261354601 (572h59m23.132437636s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.291387197 (572h59m23.162470243s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.318370928 (572h59m23.189453973s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.321236023 (572h59m23.192319058s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.408408673 (572h59m23.279491718s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419570137 (572h59m23.290653172s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419592470 (572h59m23.290675515s) [seg_pos={0 0}, upload=502322]: Segment reached optimal threshold of 14 pieces. Dec 29 12:28:23.419837406 (572h59m23.290920441s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123jUFZTVjq6adDuvt3cco7dTvwNTQkszZX7CeajMfbeCUmEmuJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419845241 (572h59m23.290928276s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.419873174 (572h59m23.290956209s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419882021 (572h59m23.290965076s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420073716 (572h59m23.291156751s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection Dec 29 12:28:23.420082262 (572h59m23.291165307s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420257506 (572h59m23.291340551s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: read tcp 188.166.225.170:41270->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:28:23.420264649 (572h59m23.291347684s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420281822 (572h59m23.291364857s) [seg_pos={0 0}, upload=502322]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:28:23.420364349 (572h59m23.291447394s) [upload=502322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:28:23.420370270 (572h59m23.291453315s) [upload=502322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:28:23.725225770 (572h59m23.596308815s) [upload=502322]: Done uploading segments... Dec 29 12:28:23.725255426 (572h59m23.596338481s) [upload=502322]: Upload finished. err: <nil> Duration: 42.834565771s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.....X.XX...X.......X.X..XXX....X...X..X....XXXX..XX..X...XX] duration: 30.003983978s trace: https://jaeger.dev.storj.io/trace/32e6b698c7ec7c2c FAIL (2025-12-29 12:45:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: MWON7CM55F5AUTWA2T3YJBST3EC3MG75FZOEZABKFNDHOZ2VUHLA) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: XQUGMK3DS2LKQCMU6ODUIIOAQKPSVCR3TMRE5VOO7XENMJK6YJRQ) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: KZJLFLO4RFQ7R2J3M6BWQ4KX5EBF2GVFI6PGXBUGD3OQBXLZVIMQ) context deadline exceeded ======LOGS====== Dec 29 12:45:00.387531028 (573h16m0.258614073s) [upload=502577]: Starting upload Dec 29 12:45:00.387618485 (573h16m0.25870153s) [upload=502577]: Done starting upload Dec 29 12:45:00.387689780 (573h16m0.258772825s) [upload=502577]: Uploading segments... Dec 29 12:45:00.387805200 (573h16m0.258888235s) [upload=502577]: Got next segment. Inline: false Dec 29 12:45:00.387872829 (573h16m0.258955874s) [upload=502577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:45:00.387880974 (573h16m0.258964009s) [upload=502577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:45:00.645740537 (573h16m0.516823572s) [seg_pos={0 0}, upload=502577]: Begin upload segment... Dec 29 12:45:00.645871266 (573h16m0.516954311s) [noise=true , address=45.91.116.10:4082 , piece_id=6ESYFNWFWEW5HARBQHA5GD7B4VB3LSGVBRLGEUWH7ZGK6YD44XQA, seg_pos={0 0}, upload=502577]: Uploading piece... Dec 29 12:45:00.645902095 (573h16m0.51698514s) [seg_pos={0 0}, upload=502577]: Done begin upload segment. Dec 29 12:45:00.645946078 (573h16m0.517029143s) [noise=true , address=145.79.227.19:4078 , piece_id=3C5T2MEHAP4GOZEEP37D7GZCP7V3URIS7ZP5SELBRETTQ5OTA7WA, seg_pos={0 0}, upload=502577]: Uploading piece... Dec 29 12:45:00.646530812 (573h16m0.517613857s) [noise=true , address=145.79.227.2:4115 , piece_id=BL2L5ZRBXJ54JAEVZ326UUCPLC7TIH6IV56VVSMFFSRNWL7NJJPA, seg_pos={0 0}, upload=502577]: Uploading piece... Dec 29 12:45:00.646900725 (573h16m0.51798377s) [noise=true , address=45.91.116.9:4263 , piece_id=XQUGMK3DS2LKQCMU6ODUIIOAQKPSVCR3TMRE5VOO7XENMJK6YJRQ, seg_pos={0 0}, upload=502577]: Uploading piece... Dec 29 12:45:00.647425164 (573h16m0.518508209s) [noise=true , address=45.91.116.7:4009 , piece_id=XNFFGVZUD5TNFWMGDTEQ7ZA5H3SPBHFS6U54ACPIMKYJHJYA3RYA, seg_pos={0 0}, upload=502577]: Uploading piece... Dec 29 12:45:00.647906090 (573h16m0.518989125s) [noise=true , address=77.90.31.19:4842 , piece_id=KZJLFLO4RFQ7R2J3M6BWQ4KX5EBF2GVFI6PGXBUGD3OQBXLZVIMQ, seg_pos={0 0}, upload=502577]: Uploading piece... Dec 29 12:45:00.648341258 (573h16m0.519424303s) [noise=true , address=145.79.227.41:4085 , piece_id=U6AQN5I2YGTCJZEZBI3JQQZVIPWMFASADQKHMG5XF5D4BNKUSBPA, seg_pos={0 0}, upload=502577]: Uploading piece... Dec 29 12:45:00.648827905 (573h16m0.51991094s) [noise=true , address=45.91.116.12:4225 , piece_id=TY7GYO4QJ3S5NJFILSLIQDCOXBHEQJ4NDE7IOT57UV5PUYTI26VQ, seg_pos={0 0}, upload=502577]: Uploading piece... Dec 29 12:45:00.649778195 (573h16m0.52086124s) [noise=true , address=145.79.227.26:4655 , piece_id=K5C7R5SV5YF3ATV2VQB5PD76346OL6D7UYOY5AU3R2XNAMC7NZSQ, seg_pos={0 0}, upload=502577]: Uploading piece... Dec 29 12:45:00.650448030 (573h16m0.521531075s) [noise=true , address=145.79.227.47:4266 , piece_id=I5FAQBFLLXXQ5TROVUW3E3C6SKPEHHDIPPE4AKF33PB55AY6UBKA, seg_pos={0 0}, upload=502577]: Uploading piece... Dec 29 12:45:00.651198859 (573h16m0.522281904s) [noise=true , address=145.79.227.42:4121 , piece_id=TJPQE52LECY6YQJ3A334RHLALU7HJYNYSFDX5C3TQBTV5NEEDY6A, seg_pos={0 0}, upload=502577]: Uploading piece... Dec 29 12:45:00.652174267 (573h16m0.523257312s) [noise=true , address=145.79.227.34:4907 , piece_id=O7E5P6NG4HQN3IUUNBC753GOSTEYD6K5FHPLGWMOFSNGO2XX47IQ, seg_pos={0 0}, upload=502577]: Uploading piece... Dec 29 12:45:00.654216295 (573h16m0.52529934s) [noise=true , address=45.91.116.6:4973 , piece_id=MWON7CM55F5AUTWA2T3YJBST3EC3MG75FZOEZABKFNDHOZ2VUHLA, seg_pos={0 0}, upload=502577]: Uploading piece... Dec 29 12:45:00.656113468 (573h16m0.527196503s) [noise=true , address=45.91.116.14:4189 , piece_id=ZBDGDEU46HPMBN5BL4SMJ46FY3FNCT6K7MWJZLSARQXKYSCCOE3Q, seg_pos={0 0}, upload=502577]: Uploading piece... Dec 29 12:45:00.657988127 (573h16m0.529071192s) [noise=true , address=145.79.227.11:4731 , piece_id=EACPQJCXKO57FHHMBW3GL4WHRGFPGXC7YXP7W64VDA4SGS7CST4A, seg_pos={0 0}, upload=502577]: Uploading piece... Dec 29 12:45:00.659802573 (573h16m0.530885618s) [noise=true , address=45.91.116.4:4937 , piece_id=XSQ2JSQT2QC3A57K6HADD3COSKWFYLNMABLT5ALUXJNLDNOIV44Q, seg_pos={0 0}, upload=502577]: Uploading piece... Dec 29 12:45:02.295689719 (573h16m2.166772764s) [upload=502577]: Next returned nil segment Dec 29 12:45:02.295709025 (573h16m2.16679209s) [upload=502577]: Waiting for error group managing segments... Dec 29 12:45:02.727188155 (573h16m2.59827121s) [noise=true , address=145.79.227.2:4115 , piece_id=BL2L5ZRBXJ54JAEVZ326UUCPLC7TIH6IV56VVSMFFSRNWL7NJJPA, seg_pos={0 0}, upload=502577]: Done uploading piece. err: <nil> Dec 29 12:45:02.727228001 (573h16m2.598311046s) [noise=true , address=145.79.227.19:4078 , piece_id=3C5T2MEHAP4GOZEEP37D7GZCP7V3URIS7ZP5SELBRETTQ5OTA7WA, seg_pos={0 0}, upload=502577]: Done uploading piece. err: <nil> Dec 29 12:45:02.727955777 (573h16m2.599038822s) [noise=true , address=145.79.227.26:4655 , piece_id=K5C7R5SV5YF3ATV2VQB5PD76346OL6D7UYOY5AU3R2XNAMC7NZSQ, seg_pos={0 0}, upload=502577]: Done uploading piece. err: <nil> Dec 29 12:45:02.728158231 (573h16m2.599241316s) [noise=true , address=145.79.227.42:4121 , piece_id=TJPQE52LECY6YQJ3A334RHLALU7HJYNYSFDX5C3TQBTV5NEEDY6A, seg_pos={0 0}, upload=502577]: Done uploading piece. err: <nil> Dec 29 12:45:02.728698210 (573h16m2.599781255s) [noise=true , address=145.79.227.11:4731 , piece_id=EACPQJCXKO57FHHMBW3GL4WHRGFPGXC7YXP7W64VDA4SGS7CST4A, seg_pos={0 0}, upload=502577]: Done uploading piece. err: <nil> Dec 29 12:45:02.733262190 (573h16m2.604345236s) [noise=true , address=145.79.227.47:4266 , piece_id=I5FAQBFLLXXQ5TROVUW3E3C6SKPEHHDIPPE4AKF33PB55AY6UBKA, seg_pos={0 0}, upload=502577]: Done uploading piece. err: <nil> Dec 29 12:45:02.752347342 (573h16m2.623430387s) [noise=true , address=145.79.227.41:4085 , piece_id=U6AQN5I2YGTCJZEZBI3JQQZVIPWMFASADQKHMG5XF5D4BNKUSBPA, seg_pos={0 0}, upload=502577]: Done uploading piece. err: <nil> Dec 29 12:45:03.026622441 (573h16m2.897705486s) [noise=true , address=45.91.116.14:4189 , piece_id=ZBDGDEU46HPMBN5BL4SMJ46FY3FNCT6K7MWJZLSARQXKYSCCOE3Q, seg_pos={0 0}, upload=502577]: Done uploading piece. err: <nil> Dec 29 12:45:03.067697874 (573h16m2.938780929s) [noise=true , address=45.91.116.7:4009 , piece_id=XNFFGVZUD5TNFWMGDTEQ7ZA5H3SPBHFS6U54ACPIMKYJHJYA3RYA, seg_pos={0 0}, upload=502577]: Done uploading piece. err: <nil> Dec 29 12:45:03.090050296 (573h16m2.961133341s) [noise=true , address=45.91.116.9:4263 , piece_id=XQUGMK3DS2LKQCMU6ODUIIOAQKPSVCR3TMRE5VOO7XENMJK6YJRQ, seg_pos={0 0}, upload=502577]: Done uploading piece. err: <nil> Dec 29 12:45:03.125518713 (573h16m2.996601768s) [noise=true , address=45.91.116.4:4937 , piece_id=XSQ2JSQT2QC3A57K6HADD3COSKWFYLNMABLT5ALUXJNLDNOIV44Q, seg_pos={0 0}, upload=502577]: Done uploading piece. err: <nil> Dec 29 12:45:03.141167661 (573h16m3.012250706s) [noise=true , address=77.90.31.19:4842 , piece_id=KZJLFLO4RFQ7R2J3M6BWQ4KX5EBF2GVFI6PGXBUGD3OQBXLZVIMQ, seg_pos={0 0}, upload=502577]: Done uploading piece. err: <nil> Dec 29 12:45:03.324711035 (573h16m3.19579409s) [noise=true , address=45.91.116.6:4973 , piece_id=MWON7CM55F5AUTWA2T3YJBST3EC3MG75FZOEZABKFNDHOZ2VUHLA, seg_pos={0 0}, upload=502577]: Done uploading piece. err: <nil> Dec 29 12:45:03.324744919 (573h16m3.195827964s) [seg_pos={0 0}, upload=502577]: Segment reached optimal threshold of 13 pieces. Dec 29 12:45:03.325754822 (573h16m3.196837867s) [noise=true , address=45.91.116.10:4082 , piece_id=6ESYFNWFWEW5HARBQHA5GD7B4VB3LSGVBRLGEUWH7ZGK6YD44XQA, seg_pos={0 0}, upload=502577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:45280->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:45280->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45280->45.91.116.10:4082: use of closed network connection Dec 29 12:45:03.325773649 (573h16m3.196856694s) [noise=true , address=45.91.116.10:4082 , piece_id=6ESYFNWFWEW5HARBQHA5GD7B4VB3LSGVBRLGEUWH7ZGK6YD44XQA, seg_pos={0 0}, upload=502577]: Long tail cancellation due to optimal threshold Dec 29 12:45:03.326325359 (573h16m3.197408414s) [noise=true , address=145.79.227.34:4907 , piece_id=O7E5P6NG4HQN3IUUNBC753GOSTEYD6K5FHPLGWMOFSNGO2XX47IQ, seg_pos={0 0}, upload=502577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 29 12:45:03.326337111 (573h16m3.197420156s) [noise=true , address=145.79.227.34:4907 , piece_id=O7E5P6NG4HQN3IUUNBC753GOSTEYD6K5FHPLGWMOFSNGO2XX47IQ, seg_pos={0 0}, upload=502577]: Long tail cancellation due to optimal threshold Dec 29 12:45:03.326899041 (573h16m3.197982116s) [noise=true , address=45.91.116.12:4225 , piece_id=TY7GYO4QJ3S5NJFILSLIQDCOXBHEQJ4NDE7IOT57UV5PUYTI26VQ, seg_pos={0 0}, upload=502577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 29 12:45:03.326912136 (573h16m3.197995181s) [noise=true , address=45.91.116.12:4225 , piece_id=TY7GYO4QJ3S5NJFILSLIQDCOXBHEQJ4NDE7IOT57UV5PUYTI26VQ, seg_pos={0 0}, upload=502577]: Long tail cancellation due to optimal threshold Dec 29 12:45:03.327620325 (573h16m3.19870337s) [seg_pos={0 0}, upload=502577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:45:03.328059380 (573h16m3.199142436s) [upload=502577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:45:03.328070652 (573h16m3.199153697s) [upload=502577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:45:03.683882468 (573h16m3.554965513s) [upload=502577]: Done uploading segments... Dec 29 12:45:03.683913788 (573h16m3.554996833s) [upload=502577]: Upload finished. err: <nil> Duration: 30.003995249s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [......X.XX...X.......X.X..XXX....X...X..X....XXXX..XX..X...X] duration: 30.003119266s trace: https://jaeger.dev.storj.io/trace/7d33bf1e518a8dce FAIL (2025-12-29 12:44:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: WEDDDOUUEKNLPF4JCQOIDTPUJ5ALWCZW5JRNYKTKD2H3LGSDZIHA) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: XI5GQP3JZM54NQSFBOJG6U5P545TFGVKJEERGW7322QYE3M33JLA) context deadline exceeded ======LOGS====== Dec 29 12:44:30.170448975 (573h15m30.04153201s) [upload=502569]: Starting upload Dec 29 12:44:30.170495053 (573h15m30.041578088s) [upload=502569]: Done starting upload Dec 29 12:44:30.170674335 (573h15m30.04175737s) [upload=502569]: Uploading segments... Dec 29 12:44:30.170708059 (573h15m30.041791104s) [upload=502569]: Got next segment. Inline: false Dec 29 12:44:30.170731603 (573h15m30.041814648s) [upload=502569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:44:30.170735320 (573h15m30.041818355s) [upload=502569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:44:30.418468552 (573h15m30.289551587s) [seg_pos={0 0}, upload=502569]: Begin upload segment... Dec 29 12:44:30.418521682 (573h15m30.289604727s) [seg_pos={0 0}, upload=502569]: Done begin upload segment. Dec 29 12:44:30.418561628 (573h15m30.289644663s) [noise=true , address=45.91.116.8:4045 , piece_id=5I3NHMQ64UNRDCYF6ICJVHUURIHWLNYEQHWGY6DK5ZJOA7KG3IDQ, seg_pos={0 0}, upload=502569]: Uploading piece... Dec 29 12:44:30.418609881 (573h15m30.289692916s) [noise=true , address=145.79.227.24:4582 , piece_id=4DIP3BOFQ7Z7VBRUTIRXML22F25OEYTDQ2ULNHQL53OY4ULVYVFQ, seg_pos={0 0}, upload=502569]: Uploading piece... Dec 29 12:44:30.419141573 (573h15m30.290224608s) [noise=true , address=145.79.227.15:4875 , piece_id=VUA7VAYAYH5M7GASRTHXOTA57E7T4BL7W4EGODN2D3RPKDWVP3UQ, seg_pos={0 0}, upload=502569]: Uploading piece... Dec 29 12:44:30.419668817 (573h15m30.290751852s) [noise=true , address=145.79.227.12:4767 , piece_id=QPON3W37QMW744OLIGUZHZITIEW3CFIMGJ443SJWCISJ76WV7WVQ, seg_pos={0 0}, upload=502569]: Uploading piece... Dec 29 12:44:30.420010347 (573h15m30.291093392s) [noise=true , address=145.79.227.48:4303 , piece_id=PAJL5DQIKSAFMIK3D4RDNESQO7XIPFAD7BZJCB2V7KCJIIRQ3KBA, seg_pos={0 0}, upload=502569]: Uploading piece... Dec 29 12:44:30.420347149 (573h15m30.291430184s) [noise=true , address=145.79.227.19:4079 , piece_id=FGHZ3WG3TEMNEQ4BKOLI7VAXCD5APDIYYZS33A2QYGS2RFMVDEAA, seg_pos={0 0}, upload=502569]: Uploading piece... Dec 29 12:44:30.420664293 (573h15m30.291747328s) [noise=true , address=45.91.116.10:4084 , piece_id=7CYWGJY2LPDMPIRU6NGYQ7VHSY5R7C6SO3KU7CHJPOQ5P5RTLKZA, seg_pos={0 0}, upload=502569]: Uploading piece... Dec 29 12:44:30.421022454 (573h15m30.292105489s) [noise=true , address=77.90.31.16:4735 , piece_id=TNG3U6GLIPKU6F7V3IK2MSCSKNJBWMXQC4W73QDTLOHUE3HMEIJQ, seg_pos={0 0}, upload=502569]: Uploading piece... Dec 29 12:44:30.421670599 (573h15m30.292753634s) [noise=true , address=77.90.31.15:4697 , piece_id=GKAT5NO4BB4N374SKUCNYFEJ53SM6NMGPNM4YC3RMCVW6JCCRTNA, seg_pos={0 0}, upload=502569]: Uploading piece... Dec 29 12:44:30.422124283 (573h15m30.293207318s) [noise=true , address=145.79.227.2:4115 , piece_id=DMYT2AMD4JAFNREFVRMU4HW43MH62DLDZ2675DFRDF6REHB6RTAA, seg_pos={0 0}, upload=502569]: Uploading piece... Dec 29 12:44:30.422551897 (573h15m30.293634932s) [noise=true , address=145.79.227.6:4257 , piece_id=4UPVYALSMJZIN7KYWZXRX4WCSVBWWDIHRQI4DHHROM7RCEQ5RKYQ, seg_pos={0 0}, upload=502569]: Uploading piece... Dec 29 12:44:30.422989109 (573h15m30.294072144s) [noise=true , address=45.91.116.12:4228 , piece_id=6CUV7RJO23WM6TEW63ZLGL55CJWDBJCFPYIOUC57PKVFQUE4MLYQ, seg_pos={0 0}, upload=502569]: Uploading piece... Dec 29 12:44:30.423320670 (573h15m30.294403716s) [noise=true , address=145.79.227.45:4230 , piece_id=BW477JLRQQLOA5FE5BKTXGH6QRA5WKBBLGHQEIF7RHBELDFZCBHQ, seg_pos={0 0}, upload=502569]: Uploading piece... Dec 29 12:44:30.423670657 (573h15m30.294753692s) [noise=true , address=45.91.116.14:4191 , piece_id=WEDDDOUUEKNLPF4JCQOIDTPUJ5ALWCZW5JRNYKTKD2H3LGSDZIHA, seg_pos={0 0}, upload=502569]: Uploading piece... Dec 29 12:44:30.424077662 (573h15m30.295160697s) [noise=true , address=77.90.31.14:4662 , piece_id=74H2E2QH3VXWAGC3IF2QMMSBRMR4BBPYCASBB5FGSVVBG26FBOIA, seg_pos={0 0}, upload=502569]: Uploading piece... Dec 29 12:44:30.424415124 (573h15m30.295498159s) [noise=true , address=45.91.116.13:4154 , piece_id=XI5GQP3JZM54NQSFBOJG6U5P545TFGVKJEERGW7322QYE3M33JLA, seg_pos={0 0}, upload=502569]: Uploading piece... Dec 29 12:44:32.111526769 (573h15m31.982609804s) [upload=502569]: Next returned nil segment Dec 29 12:44:32.111538982 (573h15m31.982622017s) [upload=502569]: Waiting for error group managing segments... Dec 29 12:44:32.393421016 (573h15m32.264504051s) [noise=true , address=145.79.227.6:4257 , piece_id=4UPVYALSMJZIN7KYWZXRX4WCSVBWWDIHRQI4DHHROM7RCEQ5RKYQ, seg_pos={0 0}, upload=502569]: Done uploading piece. err: <nil> Dec 29 12:44:32.398534814 (573h15m32.269617859s) [noise=true , address=145.79.227.19:4079 , piece_id=FGHZ3WG3TEMNEQ4BKOLI7VAXCD5APDIYYZS33A2QYGS2RFMVDEAA, seg_pos={0 0}, upload=502569]: Done uploading piece. err: <nil> Dec 29 12:44:32.405646114 (573h15m32.276729149s) [noise=true , address=145.79.227.2:4115 , piece_id=DMYT2AMD4JAFNREFVRMU4HW43MH62DLDZ2675DFRDF6REHB6RTAA, seg_pos={0 0}, upload=502569]: Done uploading piece. err: <nil> Dec 29 12:44:32.405938070 (573h15m32.277021105s) [noise=true , address=145.79.227.15:4875 , piece_id=VUA7VAYAYH5M7GASRTHXOTA57E7T4BL7W4EGODN2D3RPKDWVP3UQ, seg_pos={0 0}, upload=502569]: Done uploading piece. err: <nil> Dec 29 12:44:32.412752396 (573h15m32.283835431s) [noise=true , address=145.79.227.12:4767 , piece_id=QPON3W37QMW744OLIGUZHZITIEW3CFIMGJ443SJWCISJ76WV7WVQ, seg_pos={0 0}, upload=502569]: Done uploading piece. err: <nil> Dec 29 12:44:32.427804057 (573h15m32.298887092s) [noise=true , address=145.79.227.24:4582 , piece_id=4DIP3BOFQ7Z7VBRUTIRXML22F25OEYTDQ2ULNHQL53OY4ULVYVFQ, seg_pos={0 0}, upload=502569]: Done uploading piece. err: <nil> Dec 29 12:44:32.428449064 (573h15m32.299532109s) [noise=true , address=145.79.227.48:4303 , piece_id=PAJL5DQIKSAFMIK3D4RDNESQO7XIPFAD7BZJCB2V7KCJIIRQ3KBA, seg_pos={0 0}, upload=502569]: Done uploading piece. err: <nil> Dec 29 12:44:32.428513207 (573h15m32.299596242s) [noise=true , address=145.79.227.45:4230 , piece_id=BW477JLRQQLOA5FE5BKTXGH6QRA5WKBBLGHQEIF7RHBELDFZCBHQ, seg_pos={0 0}, upload=502569]: Done uploading piece. err: <nil> Dec 29 12:44:32.615989919 (573h15m32.487072964s) [noise=true , address=77.90.31.15:4697 , piece_id=GKAT5NO4BB4N374SKUCNYFEJ53SM6NMGPNM4YC3RMCVW6JCCRTNA, seg_pos={0 0}, upload=502569]: Done uploading piece. err: <nil> Dec 29 12:44:32.689840130 (573h15m32.560923165s) [noise=true , address=45.91.116.10:4084 , piece_id=7CYWGJY2LPDMPIRU6NGYQ7VHSY5R7C6SO3KU7CHJPOQ5P5RTLKZA, seg_pos={0 0}, upload=502569]: Done uploading piece. err: <nil> Dec 29 12:44:32.700610689 (573h15m32.571693724s) [noise=true , address=45.91.116.12:4228 , piece_id=6CUV7RJO23WM6TEW63ZLGL55CJWDBJCFPYIOUC57PKVFQUE4MLYQ, seg_pos={0 0}, upload=502569]: Done uploading piece. err: <nil> Dec 29 12:44:32.793200052 (573h15m32.664283097s) [noise=true , address=45.91.116.14:4191 , piece_id=WEDDDOUUEKNLPF4JCQOIDTPUJ5ALWCZW5JRNYKTKD2H3LGSDZIHA, seg_pos={0 0}, upload=502569]: Done uploading piece. err: <nil> Dec 29 12:44:33.513877607 (573h15m33.384960642s) [noise=true , address=45.91.116.13:4154 , piece_id=XI5GQP3JZM54NQSFBOJG6U5P545TFGVKJEERGW7322QYE3M33JLA, seg_pos={0 0}, upload=502569]: Done uploading piece. err: <nil> Dec 29 12:44:33.513903385 (573h15m33.38498646s) [seg_pos={0 0}, upload=502569]: Segment reached optimal threshold of 13 pieces. Dec 29 12:44:33.514107203 (573h15m33.385190238s) [noise=true , address=45.91.116.8:4045 , piece_id=5I3NHMQ64UNRDCYF6ICJVHUURIHWLNYEQHWGY6DK5ZJOA7KG3IDQ, seg_pos={0 0}, upload=502569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 29 12:44:33.514115980 (573h15m33.385199015s) [noise=true , address=45.91.116.8:4045 , piece_id=5I3NHMQ64UNRDCYF6ICJVHUURIHWLNYEQHWGY6DK5ZJOA7KG3IDQ, seg_pos={0 0}, upload=502569]: Long tail cancellation due to optimal threshold Dec 29 12:44:33.514344626 (573h15m33.385427661s) [noise=true , address=77.90.31.16:4735 , piece_id=TNG3U6GLIPKU6F7V3IK2MSCSKNJBWMXQC4W73QDTLOHUE3HMEIJQ, seg_pos={0 0}, upload=502569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 29 12:44:33.514351649 (573h15m33.385434684s) [noise=true , address=77.90.31.16:4735 , piece_id=TNG3U6GLIPKU6F7V3IK2MSCSKNJBWMXQC4W73QDTLOHUE3HMEIJQ, seg_pos={0 0}, upload=502569]: Long tail cancellation due to optimal threshold Dec 29 12:44:33.514430399 (573h15m33.385513434s) [noise=true , address=77.90.31.14:4662 , piece_id=74H2E2QH3VXWAGC3IF2QMMSBRMR4BBPYCASBB5FGSVVBG26FBOIA, seg_pos={0 0}, upload=502569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Dec 29 12:44:33.514452170 (573h15m33.385535205s) [noise=true , address=77.90.31.14:4662 , piece_id=74H2E2QH3VXWAGC3IF2QMMSBRMR4BBPYCASBB5FGSVVBG26FBOIA, seg_pos={0 0}, upload=502569]: Long tail cancellation due to optimal threshold Dec 29 12:44:33.514478361 (573h15m33.385561416s) [seg_pos={0 0}, upload=502569]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:44:33.514658624 (573h15m33.385741669s) [upload=502569]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:44:33.514665687 (573h15m33.385748732s) [upload=502569]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:44:33.814093811 (573h15m33.685176856s) [upload=502569]: Done uploading segments... Dec 29 12:44:33.814116394 (573h15m33.68519943s) [upload=502569]: Upload finished. err: <nil> Duration: 30.003143733s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [.X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXXXXXXXXX.XXXXXXXXXXXXX] duration: 5.001471752s trace: https://jaeger.dev.storj.io/trace/6550d8a1c1069b74 FAIL (2025-12-29 12:44:55 UTC) orbiter warning: context deadline exceeded Duration: 5.001492764s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X.......X..........X...................] duration: 28.561645934s trace: https://jaeger.dev.storj.io/trace/68764c6cb4286314 FAIL (2025-12-29 12:44:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528204662 (572h59m20.399287707s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528497489 (572h59m20.399580524s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528793153 (572h59m20.399876198s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529083555 (572h59m20.40016659s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529391081 (572h59m20.400474126s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529735096 (572h59m20.400818131s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530109970 (572h59m20.401193015s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530449556 (572h59m20.401532591s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530831554 (572h59m20.401914599s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531229512 (572h59m20.402312557s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531698945 (572h59m20.402781991s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532204238 (572h59m20.403287283s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532744366 (572h59m20.403827401s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533231203 (572h59m20.404314248s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533740113 (572h59m20.404823148s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:21.876539279 (572h59m21.747622324s) [upload=502322]: Next returned nil segment Dec 29 12:28:21.876614272 (572h59m21.747697307s) [upload=502322]: Waiting for error group managing segments... Dec 29 12:28:22.383674429 (572h59m22.254757464s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389083709 (572h59m22.260166754s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389450968 (572h59m22.260534003s) [noise=true , address=145.79.227.46:4625 , piece_id=6NEND2CKGIZLK5Y6IMM4XB6SXPSXUYSZIVZ7CZ25IKAHFFW33C5Q, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.418373151 (572h59m22.289456186s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.619432477 (572h59m22.490515552s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.741876388 (572h59m22.612959423s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.133731187 (572h59m23.004814222s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.146041929 (572h59m23.017124964s) [noise=true , address=193.149.164.143:4697 , piece_id=HZZY3SUOEC7AIHSW6MKZRNZVSOGXSYBYSUKN7STBFTXWAOMDRYBA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.261354601 (572h59m23.132437636s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.291387197 (572h59m23.162470243s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.318370928 (572h59m23.189453973s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.321236023 (572h59m23.192319058s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.408408673 (572h59m23.279491718s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419570137 (572h59m23.290653172s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419592470 (572h59m23.290675515s) [seg_pos={0 0}, upload=502322]: Segment reached optimal threshold of 14 pieces. Dec 29 12:28:23.419837406 (572h59m23.290920441s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123jUFZTVjq6adDuvt3cco7dTvwNTQkszZX7CeajMfbeCUmEmuJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419845241 (572h59m23.290928276s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.419873174 (572h59m23.290956209s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419882021 (572h59m23.290965076s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420073716 (572h59m23.291156751s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection Dec 29 12:28:23.420082262 (572h59m23.291165307s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420257506 (572h59m23.291340551s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: read tcp 188.166.225.170:41270->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:28:23.420264649 (572h59m23.291347684s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420281822 (572h59m23.291364857s) [seg_pos={0 0}, upload=502322]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:28:23.420364349 (572h59m23.291447394s) [upload=502322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:28:23.420370270 (572h59m23.291453315s) [upload=502322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:28:23.725225770 (572h59m23.596308815s) [upload=502322]: Done uploading segments... Dec 29 12:28:23.725255426 (572h59m23.596338481s) [upload=502322]: Upload finished. err: <nil> Duration: 28.561674158s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXXXXXXXXX.XXXXXXXXXXXX] duration: 5.00142877s trace: https://jaeger.dev.storj.io/trace/4dc881abca9fb68d FAIL (2025-12-29 12:44:30 UTC) orbiter warning: context deadline exceeded Duration: 5.001451224s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.......X.XX...X.......X.X..XXX....X...X..X....XXXX..XX..X...] duration: 5.741960949s trace: https://jaeger.dev.storj.io/trace/5e948f8d1bad1dad FAIL (2025-12-29 12:44:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: VHMM7VF34FH4TDJAEQU2ZDF3BSAVB2VVT3G5XHI2AK6DXEZ63GQQ) context deadline exceeded; piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: HQQ2SABSZEN45NLAPEUSDZHD72SGMVZ6DVHJ3SXZD7MZQGKJNKNA) context deadline exceeded ======LOGS====== Dec 29 12:42:07.403179501 (573h13m7.274262546s) [upload=502531]: Starting upload Dec 29 12:42:07.403288959 (573h13m7.274372015s) [upload=502531]: Done starting upload Dec 29 12:42:07.403374883 (573h13m7.274457918s) [upload=502531]: Uploading segments... Dec 29 12:42:07.403718888 (573h13m7.274801933s) [upload=502531]: Got next segment. Inline: false Dec 29 12:42:07.403773041 (573h13m7.274856086s) [upload=502531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:42:07.403779213 (573h13m7.274862258s) [upload=502531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:42:07.654568402 (573h13m7.525651438s) [seg_pos={0 0}, upload=502531]: Begin upload segment... Dec 29 12:42:07.654616113 (573h13m7.525699148s) [seg_pos={0 0}, upload=502531]: Done begin upload segment. Dec 29 12:42:07.654643696 (573h13m7.525726731s) [noise=true , address=145.79.227.38:4979 , piece_id=VVCSK4FS7Z4ZM3EQUY7BCS46DLRRF5IA4S3JB2W6P7OU5W37XOYQ, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.654710523 (573h13m7.525793558s) [noise=true , address=45.91.116.11:4117 , piece_id=KHWWWALS6R2XTC6M7VXP4VTUSPLKL5FHMQSFXULFRTFVZMKA6GMQ, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.655311487 (573h13m7.526394532s) [noise=true , address=145.79.227.6:4259 , piece_id=WQ5YYDNHEHROWCH3RVHVWLKV2EJLL55665KMXIHFLPRHZ3WA4AXQ, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.655907151 (573h13m7.526990196s) [noise=true , address=145.79.227.13:4803 , piece_id=YTGV7HUKCZXIEYEQDD3BJGA4FKALPI7NMWYEWZDOH2BSLY6KT4QQ, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.656386645 (573h13m7.52746969s) [noise=true , address=77.90.31.12:4590 , piece_id=S7Y2FSJ37MTG6ESMQWQAFQ2Z44V3H274ZO3B2BBASSKR3N75CW7Q, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.656841170 (573h13m7.527924215s) [noise=true , address=145.79.227.7:4295 , piece_id=RXOGRJYRYQEOMDZYVNQP5O4XXOF5C7BHQXKAKGXQPB5GYDMGVV7Q, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.657114901 (573h13m7.528197936s) [noise=true , address=145.79.227.39:4015 , piece_id=2SKE644NT6VXQDG6ZCW3UYGE6UIZFEHEEUYJXDYRXT7KSV7HCPIA, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.657319521 (573h13m7.528402556s) [noise=true , address=45.91.116.14:4190 , piece_id=NBU5XGV4X76V6AE6H5NSHOL64KEH7PVC4W7AYPETQBX2PBT2R2PQ, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.657587001 (573h13m7.528670046s) [noise=true , address=145.79.227.15:4874 , piece_id=VYVBEK7GV4HBKKZEINUIE6NM6FOT3R3XMGK2VR7B3EMSBRRRGKWA, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.657913863 (573h13m7.528996918s) [noise=true , address=77.90.31.13:4629 , piece_id=H4DBWJU5NLM5VWNPBB6Y27YGYVXFJJ4G25I7OSBQAIV3FDMY2PLQ, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.658408795 (573h13m7.52949184s) [noise=true , address=145.79.227.50:4374 , piece_id=R2U4VVZIEA2LQUZXXLP2HYJ7RIMWKUHBIAXS2KSGJEDCKIZ3XQKQ, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.658878420 (573h13m7.529961455s) [noise=true , address=45.91.116.5:4297 , piece_id=VHMM7VF34FH4TDJAEQU2ZDF3BSAVB2VVT3G5XHI2AK6DXEZ63GQQ, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.659504312 (573h13m7.530587357s) [noise=true , address=77.90.31.11:4553 , piece_id=HQQ2SABSZEN45NLAPEUSDZHD72SGMVZ6DVHJ3SXZD7MZQGKJNKNA, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.660046533 (573h13m7.531129568s) [noise=true , address=145.79.227.41:4085 , piece_id=4Z2BHPEUTOMZINUDWOC2KQEQRUUYBZ3LGL4IFR7KXLKKE3O4L3GA, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.660529844 (573h13m7.531612879s) [noise=true , address=145.79.227.46:4602 , piece_id=MWDWPSJMBRVMYL2KF4FSI4IPCLY3HGTLAQFK6L7PE3ZN5AENSSZQ, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.661272357 (573h13m7.532355402s) [noise=true , address=45.91.116.4:4940 , piece_id=KVC6HS6DV6LFFST7IMW7H7XADCVL3SG2L662XBVDD3YURAQJ32MA, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:09.264170401 (573h13m9.135253436s) [upload=502531]: Next returned nil segment Dec 29 12:42:09.264191772 (573h13m9.135274817s) [upload=502531]: Waiting for error group managing segments... Dec 29 12:42:09.647093953 (573h13m9.518176998s) [noise=true , address=145.79.227.13:4803 , piece_id=YTGV7HUKCZXIEYEQDD3BJGA4FKALPI7NMWYEWZDOH2BSLY6KT4QQ, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:09.655301803 (573h13m9.526384838s) [noise=true , address=145.79.227.39:4015 , piece_id=2SKE644NT6VXQDG6ZCW3UYGE6UIZFEHEEUYJXDYRXT7KSV7HCPIA, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:09.668888253 (573h13m9.539971299s) [noise=true , address=145.79.227.15:4874 , piece_id=VYVBEK7GV4HBKKZEINUIE6NM6FOT3R3XMGK2VR7B3EMSBRRRGKWA, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:09.669110677 (573h13m9.540193712s) [noise=true , address=145.79.227.38:4979 , piece_id=VVCSK4FS7Z4ZM3EQUY7BCS46DLRRF5IA4S3JB2W6P7OU5W37XOYQ, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:09.674622341 (573h13m9.545705386s) [noise=true , address=145.79.227.50:4374 , piece_id=R2U4VVZIEA2LQUZXXLP2HYJ7RIMWKUHBIAXS2KSGJEDCKIZ3XQKQ, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:09.682159644 (573h13m9.553242689s) [noise=true , address=145.79.227.41:4085 , piece_id=4Z2BHPEUTOMZINUDWOC2KQEQRUUYBZ3LGL4IFR7KXLKKE3O4L3GA, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:09.704823409 (573h13m9.575906454s) [noise=true , address=145.79.227.46:4602 , piece_id=MWDWPSJMBRVMYL2KF4FSI4IPCLY3HGTLAQFK6L7PE3ZN5AENSSZQ, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:09.707920647 (573h13m9.579003692s) [noise=true , address=145.79.227.6:4259 , piece_id=WQ5YYDNHEHROWCH3RVHVWLKV2EJLL55665KMXIHFLPRHZ3WA4AXQ, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:09.726170056 (573h13m9.597253091s) [noise=true , address=145.79.227.7:4295 , piece_id=RXOGRJYRYQEOMDZYVNQP5O4XXOF5C7BHQXKAKGXQPB5GYDMGVV7Q, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:09.927444964 (573h13m9.798528009s) [noise=true , address=45.91.116.5:4297 , piece_id=VHMM7VF34FH4TDJAEQU2ZDF3BSAVB2VVT3G5XHI2AK6DXEZ63GQQ, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:09.963733091 (573h13m9.834816136s) [noise=true , address=77.90.31.11:4553 , piece_id=HQQ2SABSZEN45NLAPEUSDZHD72SGMVZ6DVHJ3SXZD7MZQGKJNKNA, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:09.964969365 (573h13m9.83605241s) [noise=true , address=77.90.31.12:4590 , piece_id=S7Y2FSJ37MTG6ESMQWQAFQ2Z44V3H274ZO3B2BBASSKR3N75CW7Q, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:10.045617821 (573h13m9.916700866s) [noise=true , address=45.91.116.4:4940 , piece_id=KVC6HS6DV6LFFST7IMW7H7XADCVL3SG2L662XBVDD3YURAQJ32MA, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:10.045642789 (573h13m9.916725824s) [seg_pos={0 0}, upload=502531]: Segment reached optimal threshold of 13 pieces. Dec 29 12:42:10.045944103 (573h13m9.917027148s) [noise=true , address=77.90.31.13:4629 , piece_id=H4DBWJU5NLM5VWNPBB6Y27YGYVXFJJ4G25I7OSBQAIV3FDMY2PLQ, seg_pos={0 0}, upload=502531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:51256->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:51256->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51256->77.90.31.13:4629: use of closed network connection Dec 29 12:42:10.045952739 (573h13m9.917035785s) [noise=true , address=77.90.31.13:4629 , piece_id=H4DBWJU5NLM5VWNPBB6Y27YGYVXFJJ4G25I7OSBQAIV3FDMY2PLQ, seg_pos={0 0}, upload=502531]: Long tail cancellation due to optimal threshold Dec 29 12:42:10.046307395 (573h13m9.91739043s) [noise=true , address=45.91.116.14:4190 , piece_id=NBU5XGV4X76V6AE6H5NSHOL64KEH7PVC4W7AYPETQBX2PBT2R2PQ, seg_pos={0 0}, upload=502531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 29 12:42:10.046316001 (573h13m9.917399046s) [noise=true , address=45.91.116.14:4190 , piece_id=NBU5XGV4X76V6AE6H5NSHOL64KEH7PVC4W7AYPETQBX2PBT2R2PQ, seg_pos={0 0}, upload=502531]: Long tail cancellation due to optimal threshold Dec 29 12:42:10.046492246 (573h13m9.917575291s) [noise=true , address=45.91.116.11:4117 , piece_id=KHWWWALS6R2XTC6M7VXP4VTUSPLKL5FHMQSFXULFRTFVZMKA6GMQ, seg_pos={0 0}, upload=502531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 29 12:42:10.046499421 (573h13m9.917582466s) [noise=true , address=45.91.116.11:4117 , piece_id=KHWWWALS6R2XTC6M7VXP4VTUSPLKL5FHMQSFXULFRTFVZMKA6GMQ, seg_pos={0 0}, upload=502531]: Long tail cancellation due to optimal threshold Dec 29 12:42:10.046520360 (573h13m9.917603405s) [seg_pos={0 0}, upload=502531]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:42:10.046606515 (573h13m9.91768956s) [upload=502531]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:42:10.046612135 (573h13m9.91769518s) [upload=502531]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:42:10.363374347 (573h13m10.234457402s) [upload=502531]: Done uploading segments... Dec 29 12:42:10.363399485 (573h13m10.23448252s) [upload=502531]: Upload finished. err: <nil> Duration: 5.741993682s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X.......X..........X..................] duration: 26.926093188s trace: https://jaeger.dev.storj.io/trace/68764c6cb4286314 FAIL (2025-12-29 12:44:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528204662 (572h59m20.399287707s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528497489 (572h59m20.399580524s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528793153 (572h59m20.399876198s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529083555 (572h59m20.40016659s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529391081 (572h59m20.400474126s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529735096 (572h59m20.400818131s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530109970 (572h59m20.401193015s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530449556 (572h59m20.401532591s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530831554 (572h59m20.401914599s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531229512 (572h59m20.402312557s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531698945 (572h59m20.402781991s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532204238 (572h59m20.403287283s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532744366 (572h59m20.403827401s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533231203 (572h59m20.404314248s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533740113 (572h59m20.404823148s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:21.876539279 (572h59m21.747622324s) [upload=502322]: Next returned nil segment Dec 29 12:28:21.876614272 (572h59m21.747697307s) [upload=502322]: Waiting for error group managing segments... Dec 29 12:28:22.383674429 (572h59m22.254757464s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389083709 (572h59m22.260166754s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389450968 (572h59m22.260534003s) [noise=true , address=145.79.227.46:4625 , piece_id=6NEND2CKGIZLK5Y6IMM4XB6SXPSXUYSZIVZ7CZ25IKAHFFW33C5Q, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.418373151 (572h59m22.289456186s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.619432477 (572h59m22.490515552s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.741876388 (572h59m22.612959423s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.133731187 (572h59m23.004814222s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.146041929 (572h59m23.017124964s) [noise=true , address=193.149.164.143:4697 , piece_id=HZZY3SUOEC7AIHSW6MKZRNZVSOGXSYBYSUKN7STBFTXWAOMDRYBA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.261354601 (572h59m23.132437636s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.291387197 (572h59m23.162470243s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.318370928 (572h59m23.189453973s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.321236023 (572h59m23.192319058s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.408408673 (572h59m23.279491718s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419570137 (572h59m23.290653172s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419592470 (572h59m23.290675515s) [seg_pos={0 0}, upload=502322]: Segment reached optimal threshold of 14 pieces. Dec 29 12:28:23.419837406 (572h59m23.290920441s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123jUFZTVjq6adDuvt3cco7dTvwNTQkszZX7CeajMfbeCUmEmuJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419845241 (572h59m23.290928276s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.419873174 (572h59m23.290956209s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419882021 (572h59m23.290965076s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420073716 (572h59m23.291156751s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection Dec 29 12:28:23.420082262 (572h59m23.291165307s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420257506 (572h59m23.291340551s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: read tcp 188.166.225.170:41270->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:28:23.420264649 (572h59m23.291347684s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420281822 (572h59m23.291364857s) [seg_pos={0 0}, upload=502322]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:28:23.420364349 (572h59m23.291447394s) [upload=502322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:28:23.420370270 (572h59m23.291453315s) [upload=502322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:28:23.725225770 (572h59m23.596308815s) [upload=502322]: Done uploading segments... Dec 29 12:28:23.725255426 (572h59m23.596338481s) [upload=502322]: Upload finished. err: <nil> Duration: 26.926123808s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXXXXXXXXX.XXXXXXXXXXX] duration: 5.001549663s trace: https://jaeger.dev.storj.io/trace/68eda27ed450a943 FAIL (2025-12-29 12:43:57 UTC) orbiter warning: context deadline exceeded Duration: 5.00157483s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X.......X.XX...X.......X.X..XXX....X...X..X....XXXX..XX..X..] duration: 6.094747127s trace: https://jaeger.dev.storj.io/trace/5e948f8d1bad1dad FAIL (2025-12-29 12:43:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: VHMM7VF34FH4TDJAEQU2ZDF3BSAVB2VVT3G5XHI2AK6DXEZ63GQQ) context deadline exceeded; piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: HQQ2SABSZEN45NLAPEUSDZHD72SGMVZ6DVHJ3SXZD7MZQGKJNKNA) context deadline exceeded ======LOGS====== Dec 29 12:42:07.403179501 (573h13m7.274262546s) [upload=502531]: Starting upload Dec 29 12:42:07.403288959 (573h13m7.274372015s) [upload=502531]: Done starting upload Dec 29 12:42:07.403374883 (573h13m7.274457918s) [upload=502531]: Uploading segments... Dec 29 12:42:07.403718888 (573h13m7.274801933s) [upload=502531]: Got next segment. Inline: false Dec 29 12:42:07.403773041 (573h13m7.274856086s) [upload=502531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:42:07.403779213 (573h13m7.274862258s) [upload=502531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:42:07.654568402 (573h13m7.525651438s) [seg_pos={0 0}, upload=502531]: Begin upload segment... Dec 29 12:42:07.654616113 (573h13m7.525699148s) [seg_pos={0 0}, upload=502531]: Done begin upload segment. Dec 29 12:42:07.654643696 (573h13m7.525726731s) [noise=true , address=145.79.227.38:4979 , piece_id=VVCSK4FS7Z4ZM3EQUY7BCS46DLRRF5IA4S3JB2W6P7OU5W37XOYQ, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.654710523 (573h13m7.525793558s) [noise=true , address=45.91.116.11:4117 , piece_id=KHWWWALS6R2XTC6M7VXP4VTUSPLKL5FHMQSFXULFRTFVZMKA6GMQ, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.655311487 (573h13m7.526394532s) [noise=true , address=145.79.227.6:4259 , piece_id=WQ5YYDNHEHROWCH3RVHVWLKV2EJLL55665KMXIHFLPRHZ3WA4AXQ, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.655907151 (573h13m7.526990196s) [noise=true , address=145.79.227.13:4803 , piece_id=YTGV7HUKCZXIEYEQDD3BJGA4FKALPI7NMWYEWZDOH2BSLY6KT4QQ, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.656386645 (573h13m7.52746969s) [noise=true , address=77.90.31.12:4590 , piece_id=S7Y2FSJ37MTG6ESMQWQAFQ2Z44V3H274ZO3B2BBASSKR3N75CW7Q, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.656841170 (573h13m7.527924215s) [noise=true , address=145.79.227.7:4295 , piece_id=RXOGRJYRYQEOMDZYVNQP5O4XXOF5C7BHQXKAKGXQPB5GYDMGVV7Q, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.657114901 (573h13m7.528197936s) [noise=true , address=145.79.227.39:4015 , piece_id=2SKE644NT6VXQDG6ZCW3UYGE6UIZFEHEEUYJXDYRXT7KSV7HCPIA, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.657319521 (573h13m7.528402556s) [noise=true , address=45.91.116.14:4190 , piece_id=NBU5XGV4X76V6AE6H5NSHOL64KEH7PVC4W7AYPETQBX2PBT2R2PQ, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.657587001 (573h13m7.528670046s) [noise=true , address=145.79.227.15:4874 , piece_id=VYVBEK7GV4HBKKZEINUIE6NM6FOT3R3XMGK2VR7B3EMSBRRRGKWA, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.657913863 (573h13m7.528996918s) [noise=true , address=77.90.31.13:4629 , piece_id=H4DBWJU5NLM5VWNPBB6Y27YGYVXFJJ4G25I7OSBQAIV3FDMY2PLQ, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.658408795 (573h13m7.52949184s) [noise=true , address=145.79.227.50:4374 , piece_id=R2U4VVZIEA2LQUZXXLP2HYJ7RIMWKUHBIAXS2KSGJEDCKIZ3XQKQ, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.658878420 (573h13m7.529961455s) [noise=true , address=45.91.116.5:4297 , piece_id=VHMM7VF34FH4TDJAEQU2ZDF3BSAVB2VVT3G5XHI2AK6DXEZ63GQQ, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.659504312 (573h13m7.530587357s) [noise=true , address=77.90.31.11:4553 , piece_id=HQQ2SABSZEN45NLAPEUSDZHD72SGMVZ6DVHJ3SXZD7MZQGKJNKNA, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.660046533 (573h13m7.531129568s) [noise=true , address=145.79.227.41:4085 , piece_id=4Z2BHPEUTOMZINUDWOC2KQEQRUUYBZ3LGL4IFR7KXLKKE3O4L3GA, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.660529844 (573h13m7.531612879s) [noise=true , address=145.79.227.46:4602 , piece_id=MWDWPSJMBRVMYL2KF4FSI4IPCLY3HGTLAQFK6L7PE3ZN5AENSSZQ, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.661272357 (573h13m7.532355402s) [noise=true , address=45.91.116.4:4940 , piece_id=KVC6HS6DV6LFFST7IMW7H7XADCVL3SG2L662XBVDD3YURAQJ32MA, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:09.264170401 (573h13m9.135253436s) [upload=502531]: Next returned nil segment Dec 29 12:42:09.264191772 (573h13m9.135274817s) [upload=502531]: Waiting for error group managing segments... Dec 29 12:42:09.647093953 (573h13m9.518176998s) [noise=true , address=145.79.227.13:4803 , piece_id=YTGV7HUKCZXIEYEQDD3BJGA4FKALPI7NMWYEWZDOH2BSLY6KT4QQ, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:09.655301803 (573h13m9.526384838s) [noise=true , address=145.79.227.39:4015 , piece_id=2SKE644NT6VXQDG6ZCW3UYGE6UIZFEHEEUYJXDYRXT7KSV7HCPIA, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:09.668888253 (573h13m9.539971299s) [noise=true , address=145.79.227.15:4874 , piece_id=VYVBEK7GV4HBKKZEINUIE6NM6FOT3R3XMGK2VR7B3EMSBRRRGKWA, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:09.669110677 (573h13m9.540193712s) [noise=true , address=145.79.227.38:4979 , piece_id=VVCSK4FS7Z4ZM3EQUY7BCS46DLRRF5IA4S3JB2W6P7OU5W37XOYQ, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:09.674622341 (573h13m9.545705386s) [noise=true , address=145.79.227.50:4374 , piece_id=R2U4VVZIEA2LQUZXXLP2HYJ7RIMWKUHBIAXS2KSGJEDCKIZ3XQKQ, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:09.682159644 (573h13m9.553242689s) [noise=true , address=145.79.227.41:4085 , piece_id=4Z2BHPEUTOMZINUDWOC2KQEQRUUYBZ3LGL4IFR7KXLKKE3O4L3GA, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:09.704823409 (573h13m9.575906454s) [noise=true , address=145.79.227.46:4602 , piece_id=MWDWPSJMBRVMYL2KF4FSI4IPCLY3HGTLAQFK6L7PE3ZN5AENSSZQ, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:09.707920647 (573h13m9.579003692s) [noise=true , address=145.79.227.6:4259 , piece_id=WQ5YYDNHEHROWCH3RVHVWLKV2EJLL55665KMXIHFLPRHZ3WA4AXQ, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:09.726170056 (573h13m9.597253091s) [noise=true , address=145.79.227.7:4295 , piece_id=RXOGRJYRYQEOMDZYVNQP5O4XXOF5C7BHQXKAKGXQPB5GYDMGVV7Q, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:09.927444964 (573h13m9.798528009s) [noise=true , address=45.91.116.5:4297 , piece_id=VHMM7VF34FH4TDJAEQU2ZDF3BSAVB2VVT3G5XHI2AK6DXEZ63GQQ, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:09.963733091 (573h13m9.834816136s) [noise=true , address=77.90.31.11:4553 , piece_id=HQQ2SABSZEN45NLAPEUSDZHD72SGMVZ6DVHJ3SXZD7MZQGKJNKNA, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:09.964969365 (573h13m9.83605241s) [noise=true , address=77.90.31.12:4590 , piece_id=S7Y2FSJ37MTG6ESMQWQAFQ2Z44V3H274ZO3B2BBASSKR3N75CW7Q, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:10.045617821 (573h13m9.916700866s) [noise=true , address=45.91.116.4:4940 , piece_id=KVC6HS6DV6LFFST7IMW7H7XADCVL3SG2L662XBVDD3YURAQJ32MA, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:10.045642789 (573h13m9.916725824s) [seg_pos={0 0}, upload=502531]: Segment reached optimal threshold of 13 pieces. Dec 29 12:42:10.045944103 (573h13m9.917027148s) [noise=true , address=77.90.31.13:4629 , piece_id=H4DBWJU5NLM5VWNPBB6Y27YGYVXFJJ4G25I7OSBQAIV3FDMY2PLQ, seg_pos={0 0}, upload=502531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:51256->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:51256->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51256->77.90.31.13:4629: use of closed network connection Dec 29 12:42:10.045952739 (573h13m9.917035785s) [noise=true , address=77.90.31.13:4629 , piece_id=H4DBWJU5NLM5VWNPBB6Y27YGYVXFJJ4G25I7OSBQAIV3FDMY2PLQ, seg_pos={0 0}, upload=502531]: Long tail cancellation due to optimal threshold Dec 29 12:42:10.046307395 (573h13m9.91739043s) [noise=true , address=45.91.116.14:4190 , piece_id=NBU5XGV4X76V6AE6H5NSHOL64KEH7PVC4W7AYPETQBX2PBT2R2PQ, seg_pos={0 0}, upload=502531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 29 12:42:10.046316001 (573h13m9.917399046s) [noise=true , address=45.91.116.14:4190 , piece_id=NBU5XGV4X76V6AE6H5NSHOL64KEH7PVC4W7AYPETQBX2PBT2R2PQ, seg_pos={0 0}, upload=502531]: Long tail cancellation due to optimal threshold Dec 29 12:42:10.046492246 (573h13m9.917575291s) [noise=true , address=45.91.116.11:4117 , piece_id=KHWWWALS6R2XTC6M7VXP4VTUSPLKL5FHMQSFXULFRTFVZMKA6GMQ, seg_pos={0 0}, upload=502531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 29 12:42:10.046499421 (573h13m9.917582466s) [noise=true , address=45.91.116.11:4117 , piece_id=KHWWWALS6R2XTC6M7VXP4VTUSPLKL5FHMQSFXULFRTFVZMKA6GMQ, seg_pos={0 0}, upload=502531]: Long tail cancellation due to optimal threshold Dec 29 12:42:10.046520360 (573h13m9.917603405s) [seg_pos={0 0}, upload=502531]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:42:10.046606515 (573h13m9.91768956s) [upload=502531]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:42:10.046612135 (573h13m9.91769518s) [upload=502531]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:42:10.363374347 (573h13m10.234457402s) [upload=502531]: Done uploading segments... Dec 29 12:42:10.363399485 (573h13m10.23448252s) [upload=502531]: Upload finished. err: <nil> Duration: 6.094773789s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X.......X..........X.................] duration: 49.746623552s trace: https://jaeger.dev.storj.io/trace/68764c6cb4286314 FAIL (2025-12-29 12:43:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528204662 (572h59m20.399287707s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528497489 (572h59m20.399580524s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528793153 (572h59m20.399876198s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529083555 (572h59m20.40016659s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529391081 (572h59m20.400474126s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529735096 (572h59m20.400818131s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530109970 (572h59m20.401193015s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530449556 (572h59m20.401532591s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530831554 (572h59m20.401914599s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531229512 (572h59m20.402312557s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531698945 (572h59m20.402781991s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532204238 (572h59m20.403287283s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532744366 (572h59m20.403827401s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533231203 (572h59m20.404314248s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533740113 (572h59m20.404823148s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:21.876539279 (572h59m21.747622324s) [upload=502322]: Next returned nil segment Dec 29 12:28:21.876614272 (572h59m21.747697307s) [upload=502322]: Waiting for error group managing segments... Dec 29 12:28:22.383674429 (572h59m22.254757464s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389083709 (572h59m22.260166754s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389450968 (572h59m22.260534003s) [noise=true , address=145.79.227.46:4625 , piece_id=6NEND2CKGIZLK5Y6IMM4XB6SXPSXUYSZIVZ7CZ25IKAHFFW33C5Q, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.418373151 (572h59m22.289456186s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.619432477 (572h59m22.490515552s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.741876388 (572h59m22.612959423s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.133731187 (572h59m23.004814222s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.146041929 (572h59m23.017124964s) [noise=true , address=193.149.164.143:4697 , piece_id=HZZY3SUOEC7AIHSW6MKZRNZVSOGXSYBYSUKN7STBFTXWAOMDRYBA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.261354601 (572h59m23.132437636s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.291387197 (572h59m23.162470243s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.318370928 (572h59m23.189453973s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.321236023 (572h59m23.192319058s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.408408673 (572h59m23.279491718s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419570137 (572h59m23.290653172s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419592470 (572h59m23.290675515s) [seg_pos={0 0}, upload=502322]: Segment reached optimal threshold of 14 pieces. Dec 29 12:28:23.419837406 (572h59m23.290920441s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123jUFZTVjq6adDuvt3cco7dTvwNTQkszZX7CeajMfbeCUmEmuJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419845241 (572h59m23.290928276s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.419873174 (572h59m23.290956209s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419882021 (572h59m23.290965076s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420073716 (572h59m23.291156751s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection Dec 29 12:28:23.420082262 (572h59m23.291165307s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420257506 (572h59m23.291340551s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: read tcp 188.166.225.170:41270->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:28:23.420264649 (572h59m23.291347684s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420281822 (572h59m23.291364857s) [seg_pos={0 0}, upload=502322]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:28:23.420364349 (572h59m23.291447394s) [upload=502322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:28:23.420370270 (572h59m23.291453315s) [upload=502322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:28:23.725225770 (572h59m23.596308815s) [upload=502322]: Done uploading segments... Dec 29 12:28:23.725255426 (572h59m23.596338481s) [upload=502322]: Upload finished. err: <nil> Duration: 49.746640956s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X.......X.XX...X.......X.X..XXX....X...X..X....XXXX..XX..X.] duration: 5.73297411s trace: https://jaeger.dev.storj.io/trace/5e948f8d1bad1dad FAIL (2025-12-29 12:42:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: VHMM7VF34FH4TDJAEQU2ZDF3BSAVB2VVT3G5XHI2AK6DXEZ63GQQ) context deadline exceeded; piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: HQQ2SABSZEN45NLAPEUSDZHD72SGMVZ6DVHJ3SXZD7MZQGKJNKNA) context deadline exceeded ======LOGS====== Dec 29 12:42:07.403179501 (573h13m7.274262546s) [upload=502531]: Starting upload Dec 29 12:42:07.403288959 (573h13m7.274372015s) [upload=502531]: Done starting upload Dec 29 12:42:07.403374883 (573h13m7.274457918s) [upload=502531]: Uploading segments... Dec 29 12:42:07.403718888 (573h13m7.274801933s) [upload=502531]: Got next segment. Inline: false Dec 29 12:42:07.403773041 (573h13m7.274856086s) [upload=502531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:42:07.403779213 (573h13m7.274862258s) [upload=502531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:42:07.654568402 (573h13m7.525651438s) [seg_pos={0 0}, upload=502531]: Begin upload segment... Dec 29 12:42:07.654616113 (573h13m7.525699148s) [seg_pos={0 0}, upload=502531]: Done begin upload segment. Dec 29 12:42:07.654643696 (573h13m7.525726731s) [noise=true , address=145.79.227.38:4979 , piece_id=VVCSK4FS7Z4ZM3EQUY7BCS46DLRRF5IA4S3JB2W6P7OU5W37XOYQ, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.654710523 (573h13m7.525793558s) [noise=true , address=45.91.116.11:4117 , piece_id=KHWWWALS6R2XTC6M7VXP4VTUSPLKL5FHMQSFXULFRTFVZMKA6GMQ, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.655311487 (573h13m7.526394532s) [noise=true , address=145.79.227.6:4259 , piece_id=WQ5YYDNHEHROWCH3RVHVWLKV2EJLL55665KMXIHFLPRHZ3WA4AXQ, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.655907151 (573h13m7.526990196s) [noise=true , address=145.79.227.13:4803 , piece_id=YTGV7HUKCZXIEYEQDD3BJGA4FKALPI7NMWYEWZDOH2BSLY6KT4QQ, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.656386645 (573h13m7.52746969s) [noise=true , address=77.90.31.12:4590 , piece_id=S7Y2FSJ37MTG6ESMQWQAFQ2Z44V3H274ZO3B2BBASSKR3N75CW7Q, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.656841170 (573h13m7.527924215s) [noise=true , address=145.79.227.7:4295 , piece_id=RXOGRJYRYQEOMDZYVNQP5O4XXOF5C7BHQXKAKGXQPB5GYDMGVV7Q, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.657114901 (573h13m7.528197936s) [noise=true , address=145.79.227.39:4015 , piece_id=2SKE644NT6VXQDG6ZCW3UYGE6UIZFEHEEUYJXDYRXT7KSV7HCPIA, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.657319521 (573h13m7.528402556s) [noise=true , address=45.91.116.14:4190 , piece_id=NBU5XGV4X76V6AE6H5NSHOL64KEH7PVC4W7AYPETQBX2PBT2R2PQ, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.657587001 (573h13m7.528670046s) [noise=true , address=145.79.227.15:4874 , piece_id=VYVBEK7GV4HBKKZEINUIE6NM6FOT3R3XMGK2VR7B3EMSBRRRGKWA, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.657913863 (573h13m7.528996918s) [noise=true , address=77.90.31.13:4629 , piece_id=H4DBWJU5NLM5VWNPBB6Y27YGYVXFJJ4G25I7OSBQAIV3FDMY2PLQ, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.658408795 (573h13m7.52949184s) [noise=true , address=145.79.227.50:4374 , piece_id=R2U4VVZIEA2LQUZXXLP2HYJ7RIMWKUHBIAXS2KSGJEDCKIZ3XQKQ, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.658878420 (573h13m7.529961455s) [noise=true , address=45.91.116.5:4297 , piece_id=VHMM7VF34FH4TDJAEQU2ZDF3BSAVB2VVT3G5XHI2AK6DXEZ63GQQ, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.659504312 (573h13m7.530587357s) [noise=true , address=77.90.31.11:4553 , piece_id=HQQ2SABSZEN45NLAPEUSDZHD72SGMVZ6DVHJ3SXZD7MZQGKJNKNA, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.660046533 (573h13m7.531129568s) [noise=true , address=145.79.227.41:4085 , piece_id=4Z2BHPEUTOMZINUDWOC2KQEQRUUYBZ3LGL4IFR7KXLKKE3O4L3GA, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.660529844 (573h13m7.531612879s) [noise=true , address=145.79.227.46:4602 , piece_id=MWDWPSJMBRVMYL2KF4FSI4IPCLY3HGTLAQFK6L7PE3ZN5AENSSZQ, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.661272357 (573h13m7.532355402s) [noise=true , address=45.91.116.4:4940 , piece_id=KVC6HS6DV6LFFST7IMW7H7XADCVL3SG2L662XBVDD3YURAQJ32MA, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:09.264170401 (573h13m9.135253436s) [upload=502531]: Next returned nil segment Dec 29 12:42:09.264191772 (573h13m9.135274817s) [upload=502531]: Waiting for error group managing segments... Dec 29 12:42:09.647093953 (573h13m9.518176998s) [noise=true , address=145.79.227.13:4803 , piece_id=YTGV7HUKCZXIEYEQDD3BJGA4FKALPI7NMWYEWZDOH2BSLY6KT4QQ, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:09.655301803 (573h13m9.526384838s) [noise=true , address=145.79.227.39:4015 , piece_id=2SKE644NT6VXQDG6ZCW3UYGE6UIZFEHEEUYJXDYRXT7KSV7HCPIA, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:09.668888253 (573h13m9.539971299s) [noise=true , address=145.79.227.15:4874 , piece_id=VYVBEK7GV4HBKKZEINUIE6NM6FOT3R3XMGK2VR7B3EMSBRRRGKWA, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:09.669110677 (573h13m9.540193712s) [noise=true , address=145.79.227.38:4979 , piece_id=VVCSK4FS7Z4ZM3EQUY7BCS46DLRRF5IA4S3JB2W6P7OU5W37XOYQ, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:09.674622341 (573h13m9.545705386s) [noise=true , address=145.79.227.50:4374 , piece_id=R2U4VVZIEA2LQUZXXLP2HYJ7RIMWKUHBIAXS2KSGJEDCKIZ3XQKQ, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:09.682159644 (573h13m9.553242689s) [noise=true , address=145.79.227.41:4085 , piece_id=4Z2BHPEUTOMZINUDWOC2KQEQRUUYBZ3LGL4IFR7KXLKKE3O4L3GA, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:09.704823409 (573h13m9.575906454s) [noise=true , address=145.79.227.46:4602 , piece_id=MWDWPSJMBRVMYL2KF4FSI4IPCLY3HGTLAQFK6L7PE3ZN5AENSSZQ, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:09.707920647 (573h13m9.579003692s) [noise=true , address=145.79.227.6:4259 , piece_id=WQ5YYDNHEHROWCH3RVHVWLKV2EJLL55665KMXIHFLPRHZ3WA4AXQ, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:09.726170056 (573h13m9.597253091s) [noise=true , address=145.79.227.7:4295 , piece_id=RXOGRJYRYQEOMDZYVNQP5O4XXOF5C7BHQXKAKGXQPB5GYDMGVV7Q, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:09.927444964 (573h13m9.798528009s) [noise=true , address=45.91.116.5:4297 , piece_id=VHMM7VF34FH4TDJAEQU2ZDF3BSAVB2VVT3G5XHI2AK6DXEZ63GQQ, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:09.963733091 (573h13m9.834816136s) [noise=true , address=77.90.31.11:4553 , piece_id=HQQ2SABSZEN45NLAPEUSDZHD72SGMVZ6DVHJ3SXZD7MZQGKJNKNA, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:09.964969365 (573h13m9.83605241s) [noise=true , address=77.90.31.12:4590 , piece_id=S7Y2FSJ37MTG6ESMQWQAFQ2Z44V3H274ZO3B2BBASSKR3N75CW7Q, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:10.045617821 (573h13m9.916700866s) [noise=true , address=45.91.116.4:4940 , piece_id=KVC6HS6DV6LFFST7IMW7H7XADCVL3SG2L662XBVDD3YURAQJ32MA, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:10.045642789 (573h13m9.916725824s) [seg_pos={0 0}, upload=502531]: Segment reached optimal threshold of 13 pieces. Dec 29 12:42:10.045944103 (573h13m9.917027148s) [noise=true , address=77.90.31.13:4629 , piece_id=H4DBWJU5NLM5VWNPBB6Y27YGYVXFJJ4G25I7OSBQAIV3FDMY2PLQ, seg_pos={0 0}, upload=502531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:51256->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:51256->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51256->77.90.31.13:4629: use of closed network connection Dec 29 12:42:10.045952739 (573h13m9.917035785s) [noise=true , address=77.90.31.13:4629 , piece_id=H4DBWJU5NLM5VWNPBB6Y27YGYVXFJJ4G25I7OSBQAIV3FDMY2PLQ, seg_pos={0 0}, upload=502531]: Long tail cancellation due to optimal threshold Dec 29 12:42:10.046307395 (573h13m9.91739043s) [noise=true , address=45.91.116.14:4190 , piece_id=NBU5XGV4X76V6AE6H5NSHOL64KEH7PVC4W7AYPETQBX2PBT2R2PQ, seg_pos={0 0}, upload=502531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 29 12:42:10.046316001 (573h13m9.917399046s) [noise=true , address=45.91.116.14:4190 , piece_id=NBU5XGV4X76V6AE6H5NSHOL64KEH7PVC4W7AYPETQBX2PBT2R2PQ, seg_pos={0 0}, upload=502531]: Long tail cancellation due to optimal threshold Dec 29 12:42:10.046492246 (573h13m9.917575291s) [noise=true , address=45.91.116.11:4117 , piece_id=KHWWWALS6R2XTC6M7VXP4VTUSPLKL5FHMQSFXULFRTFVZMKA6GMQ, seg_pos={0 0}, upload=502531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 29 12:42:10.046499421 (573h13m9.917582466s) [noise=true , address=45.91.116.11:4117 , piece_id=KHWWWALS6R2XTC6M7VXP4VTUSPLKL5FHMQSFXULFRTFVZMKA6GMQ, seg_pos={0 0}, upload=502531]: Long tail cancellation due to optimal threshold Dec 29 12:42:10.046520360 (573h13m9.917603405s) [seg_pos={0 0}, upload=502531]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:42:10.046606515 (573h13m9.91768956s) [upload=502531]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:42:10.046612135 (573h13m9.91769518s) [upload=502531]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:42:10.363374347 (573h13m10.234457402s) [upload=502531]: Done uploading segments... Dec 29 12:42:10.363399485 (573h13m10.23448252s) [upload=502531]: Upload finished. err: <nil> Duration: 5.732999509s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXXXXXXXXX.XXXXXXXXXX] duration: 5.001687457s trace: https://jaeger.dev.storj.io/trace/39a741eedd60a643 FAIL (2025-12-29 12:42:45 UTC) orbiter warning: context deadline exceeded Duration: 5.001711504s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X.X.......X.XX...X.......X.X..XXX....X...X..X....XXXX..XX..X] duration: 30.003140798s trace: https://jaeger.dev.storj.io/trace/5e948f8d1bad1dad FAIL (2025-12-29 12:42:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: VHMM7VF34FH4TDJAEQU2ZDF3BSAVB2VVT3G5XHI2AK6DXEZ63GQQ) context deadline exceeded; piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: HQQ2SABSZEN45NLAPEUSDZHD72SGMVZ6DVHJ3SXZD7MZQGKJNKNA) context deadline exceeded ======LOGS====== Dec 29 12:42:07.403179501 (573h13m7.274262546s) [upload=502531]: Starting upload Dec 29 12:42:07.403288959 (573h13m7.274372015s) [upload=502531]: Done starting upload Dec 29 12:42:07.403374883 (573h13m7.274457918s) [upload=502531]: Uploading segments... Dec 29 12:42:07.403718888 (573h13m7.274801933s) [upload=502531]: Got next segment. Inline: false Dec 29 12:42:07.403773041 (573h13m7.274856086s) [upload=502531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:42:07.403779213 (573h13m7.274862258s) [upload=502531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:42:07.654568402 (573h13m7.525651438s) [seg_pos={0 0}, upload=502531]: Begin upload segment... Dec 29 12:42:07.654616113 (573h13m7.525699148s) [seg_pos={0 0}, upload=502531]: Done begin upload segment. Dec 29 12:42:07.654643696 (573h13m7.525726731s) [noise=true , address=145.79.227.38:4979 , piece_id=VVCSK4FS7Z4ZM3EQUY7BCS46DLRRF5IA4S3JB2W6P7OU5W37XOYQ, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.654710523 (573h13m7.525793558s) [noise=true , address=45.91.116.11:4117 , piece_id=KHWWWALS6R2XTC6M7VXP4VTUSPLKL5FHMQSFXULFRTFVZMKA6GMQ, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.655311487 (573h13m7.526394532s) [noise=true , address=145.79.227.6:4259 , piece_id=WQ5YYDNHEHROWCH3RVHVWLKV2EJLL55665KMXIHFLPRHZ3WA4AXQ, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.655907151 (573h13m7.526990196s) [noise=true , address=145.79.227.13:4803 , piece_id=YTGV7HUKCZXIEYEQDD3BJGA4FKALPI7NMWYEWZDOH2BSLY6KT4QQ, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.656386645 (573h13m7.52746969s) [noise=true , address=77.90.31.12:4590 , piece_id=S7Y2FSJ37MTG6ESMQWQAFQ2Z44V3H274ZO3B2BBASSKR3N75CW7Q, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.656841170 (573h13m7.527924215s) [noise=true , address=145.79.227.7:4295 , piece_id=RXOGRJYRYQEOMDZYVNQP5O4XXOF5C7BHQXKAKGXQPB5GYDMGVV7Q, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.657114901 (573h13m7.528197936s) [noise=true , address=145.79.227.39:4015 , piece_id=2SKE644NT6VXQDG6ZCW3UYGE6UIZFEHEEUYJXDYRXT7KSV7HCPIA, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.657319521 (573h13m7.528402556s) [noise=true , address=45.91.116.14:4190 , piece_id=NBU5XGV4X76V6AE6H5NSHOL64KEH7PVC4W7AYPETQBX2PBT2R2PQ, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.657587001 (573h13m7.528670046s) [noise=true , address=145.79.227.15:4874 , piece_id=VYVBEK7GV4HBKKZEINUIE6NM6FOT3R3XMGK2VR7B3EMSBRRRGKWA, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.657913863 (573h13m7.528996918s) [noise=true , address=77.90.31.13:4629 , piece_id=H4DBWJU5NLM5VWNPBB6Y27YGYVXFJJ4G25I7OSBQAIV3FDMY2PLQ, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.658408795 (573h13m7.52949184s) [noise=true , address=145.79.227.50:4374 , piece_id=R2U4VVZIEA2LQUZXXLP2HYJ7RIMWKUHBIAXS2KSGJEDCKIZ3XQKQ, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.658878420 (573h13m7.529961455s) [noise=true , address=45.91.116.5:4297 , piece_id=VHMM7VF34FH4TDJAEQU2ZDF3BSAVB2VVT3G5XHI2AK6DXEZ63GQQ, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.659504312 (573h13m7.530587357s) [noise=true , address=77.90.31.11:4553 , piece_id=HQQ2SABSZEN45NLAPEUSDZHD72SGMVZ6DVHJ3SXZD7MZQGKJNKNA, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.660046533 (573h13m7.531129568s) [noise=true , address=145.79.227.41:4085 , piece_id=4Z2BHPEUTOMZINUDWOC2KQEQRUUYBZ3LGL4IFR7KXLKKE3O4L3GA, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.660529844 (573h13m7.531612879s) [noise=true , address=145.79.227.46:4602 , piece_id=MWDWPSJMBRVMYL2KF4FSI4IPCLY3HGTLAQFK6L7PE3ZN5AENSSZQ, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:07.661272357 (573h13m7.532355402s) [noise=true , address=45.91.116.4:4940 , piece_id=KVC6HS6DV6LFFST7IMW7H7XADCVL3SG2L662XBVDD3YURAQJ32MA, seg_pos={0 0}, upload=502531]: Uploading piece... Dec 29 12:42:09.264170401 (573h13m9.135253436s) [upload=502531]: Next returned nil segment Dec 29 12:42:09.264191772 (573h13m9.135274817s) [upload=502531]: Waiting for error group managing segments... Dec 29 12:42:09.647093953 (573h13m9.518176998s) [noise=true , address=145.79.227.13:4803 , piece_id=YTGV7HUKCZXIEYEQDD3BJGA4FKALPI7NMWYEWZDOH2BSLY6KT4QQ, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:09.655301803 (573h13m9.526384838s) [noise=true , address=145.79.227.39:4015 , piece_id=2SKE644NT6VXQDG6ZCW3UYGE6UIZFEHEEUYJXDYRXT7KSV7HCPIA, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:09.668888253 (573h13m9.539971299s) [noise=true , address=145.79.227.15:4874 , piece_id=VYVBEK7GV4HBKKZEINUIE6NM6FOT3R3XMGK2VR7B3EMSBRRRGKWA, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:09.669110677 (573h13m9.540193712s) [noise=true , address=145.79.227.38:4979 , piece_id=VVCSK4FS7Z4ZM3EQUY7BCS46DLRRF5IA4S3JB2W6P7OU5W37XOYQ, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:09.674622341 (573h13m9.545705386s) [noise=true , address=145.79.227.50:4374 , piece_id=R2U4VVZIEA2LQUZXXLP2HYJ7RIMWKUHBIAXS2KSGJEDCKIZ3XQKQ, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:09.682159644 (573h13m9.553242689s) [noise=true , address=145.79.227.41:4085 , piece_id=4Z2BHPEUTOMZINUDWOC2KQEQRUUYBZ3LGL4IFR7KXLKKE3O4L3GA, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:09.704823409 (573h13m9.575906454s) [noise=true , address=145.79.227.46:4602 , piece_id=MWDWPSJMBRVMYL2KF4FSI4IPCLY3HGTLAQFK6L7PE3ZN5AENSSZQ, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:09.707920647 (573h13m9.579003692s) [noise=true , address=145.79.227.6:4259 , piece_id=WQ5YYDNHEHROWCH3RVHVWLKV2EJLL55665KMXIHFLPRHZ3WA4AXQ, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:09.726170056 (573h13m9.597253091s) [noise=true , address=145.79.227.7:4295 , piece_id=RXOGRJYRYQEOMDZYVNQP5O4XXOF5C7BHQXKAKGXQPB5GYDMGVV7Q, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:09.927444964 (573h13m9.798528009s) [noise=true , address=45.91.116.5:4297 , piece_id=VHMM7VF34FH4TDJAEQU2ZDF3BSAVB2VVT3G5XHI2AK6DXEZ63GQQ, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:09.963733091 (573h13m9.834816136s) [noise=true , address=77.90.31.11:4553 , piece_id=HQQ2SABSZEN45NLAPEUSDZHD72SGMVZ6DVHJ3SXZD7MZQGKJNKNA, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:09.964969365 (573h13m9.83605241s) [noise=true , address=77.90.31.12:4590 , piece_id=S7Y2FSJ37MTG6ESMQWQAFQ2Z44V3H274ZO3B2BBASSKR3N75CW7Q, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:10.045617821 (573h13m9.916700866s) [noise=true , address=45.91.116.4:4940 , piece_id=KVC6HS6DV6LFFST7IMW7H7XADCVL3SG2L662XBVDD3YURAQJ32MA, seg_pos={0 0}, upload=502531]: Done uploading piece. err: <nil> Dec 29 12:42:10.045642789 (573h13m9.916725824s) [seg_pos={0 0}, upload=502531]: Segment reached optimal threshold of 13 pieces. Dec 29 12:42:10.045944103 (573h13m9.917027148s) [noise=true , address=77.90.31.13:4629 , piece_id=H4DBWJU5NLM5VWNPBB6Y27YGYVXFJJ4G25I7OSBQAIV3FDMY2PLQ, seg_pos={0 0}, upload=502531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:51256->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:51256->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51256->77.90.31.13:4629: use of closed network connection Dec 29 12:42:10.045952739 (573h13m9.917035785s) [noise=true , address=77.90.31.13:4629 , piece_id=H4DBWJU5NLM5VWNPBB6Y27YGYVXFJJ4G25I7OSBQAIV3FDMY2PLQ, seg_pos={0 0}, upload=502531]: Long tail cancellation due to optimal threshold Dec 29 12:42:10.046307395 (573h13m9.91739043s) [noise=true , address=45.91.116.14:4190 , piece_id=NBU5XGV4X76V6AE6H5NSHOL64KEH7PVC4W7AYPETQBX2PBT2R2PQ, seg_pos={0 0}, upload=502531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 29 12:42:10.046316001 (573h13m9.917399046s) [noise=true , address=45.91.116.14:4190 , piece_id=NBU5XGV4X76V6AE6H5NSHOL64KEH7PVC4W7AYPETQBX2PBT2R2PQ, seg_pos={0 0}, upload=502531]: Long tail cancellation due to optimal threshold Dec 29 12:42:10.046492246 (573h13m9.917575291s) [noise=true , address=45.91.116.11:4117 , piece_id=KHWWWALS6R2XTC6M7VXP4VTUSPLKL5FHMQSFXULFRTFVZMKA6GMQ, seg_pos={0 0}, upload=502531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 29 12:42:10.046499421 (573h13m9.917582466s) [noise=true , address=45.91.116.11:4117 , piece_id=KHWWWALS6R2XTC6M7VXP4VTUSPLKL5FHMQSFXULFRTFVZMKA6GMQ, seg_pos={0 0}, upload=502531]: Long tail cancellation due to optimal threshold Dec 29 12:42:10.046520360 (573h13m9.917603405s) [seg_pos={0 0}, upload=502531]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:42:10.046606515 (573h13m9.91768956s) [upload=502531]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:42:10.046612135 (573h13m9.91769518s) [upload=502531]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:42:10.363374347 (573h13m10.234457402s) [upload=502531]: Done uploading segments... Dec 29 12:42:10.363399485 (573h13m10.23448252s) [upload=502531]: Upload finished. err: <nil> Duration: 30.003164573s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X.......X..........X................] duration: 46.433487041s trace: https://jaeger.dev.storj.io/trace/68764c6cb4286314 FAIL (2025-12-29 12:42:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528204662 (572h59m20.399287707s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528497489 (572h59m20.399580524s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528793153 (572h59m20.399876198s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529083555 (572h59m20.40016659s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529391081 (572h59m20.400474126s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529735096 (572h59m20.400818131s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530109970 (572h59m20.401193015s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530449556 (572h59m20.401532591s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530831554 (572h59m20.401914599s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531229512 (572h59m20.402312557s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531698945 (572h59m20.402781991s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532204238 (572h59m20.403287283s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532744366 (572h59m20.403827401s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533231203 (572h59m20.404314248s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533740113 (572h59m20.404823148s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:21.876539279 (572h59m21.747622324s) [upload=502322]: Next returned nil segment Dec 29 12:28:21.876614272 (572h59m21.747697307s) [upload=502322]: Waiting for error group managing segments... Dec 29 12:28:22.383674429 (572h59m22.254757464s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389083709 (572h59m22.260166754s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389450968 (572h59m22.260534003s) [noise=true , address=145.79.227.46:4625 , piece_id=6NEND2CKGIZLK5Y6IMM4XB6SXPSXUYSZIVZ7CZ25IKAHFFW33C5Q, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.418373151 (572h59m22.289456186s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.619432477 (572h59m22.490515552s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.741876388 (572h59m22.612959423s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.133731187 (572h59m23.004814222s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.146041929 (572h59m23.017124964s) [noise=true , address=193.149.164.143:4697 , piece_id=HZZY3SUOEC7AIHSW6MKZRNZVSOGXSYBYSUKN7STBFTXWAOMDRYBA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.261354601 (572h59m23.132437636s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.291387197 (572h59m23.162470243s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.318370928 (572h59m23.189453973s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.321236023 (572h59m23.192319058s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.408408673 (572h59m23.279491718s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419570137 (572h59m23.290653172s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419592470 (572h59m23.290675515s) [seg_pos={0 0}, upload=502322]: Segment reached optimal threshold of 14 pieces. Dec 29 12:28:23.419837406 (572h59m23.290920441s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123jUFZTVjq6adDuvt3cco7dTvwNTQkszZX7CeajMfbeCUmEmuJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419845241 (572h59m23.290928276s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.419873174 (572h59m23.290956209s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419882021 (572h59m23.290965076s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420073716 (572h59m23.291156751s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection Dec 29 12:28:23.420082262 (572h59m23.291165307s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420257506 (572h59m23.291340551s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: read tcp 188.166.225.170:41270->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:28:23.420264649 (572h59m23.291347684s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420281822 (572h59m23.291364857s) [seg_pos={0 0}, upload=502322]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:28:23.420364349 (572h59m23.291447394s) [upload=502322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:28:23.420370270 (572h59m23.291453315s) [upload=502322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:28:23.725225770 (572h59m23.596308815s) [upload=502322]: Done uploading segments... Dec 29 12:28:23.725255426 (572h59m23.596338481s) [upload=502322]: Upload finished. err: <nil> Duration: 46.433502058s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [X.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXXXXXXXXX.XXXXXXXXX] duration: 5.000631695s trace: https://jaeger.dev.storj.io/trace/4677e9fb1a6f48f6 FAIL (2025-12-29 12:41:52 UTC) orbiter warning: context deadline exceeded Duration: 5.000657585s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X.X.......X.XX...X.......X.X..XXX....X...X..X....XXXX..XX..] duration: 5.867085942s trace: https://jaeger.dev.storj.io/trace/21fde371d273b7b9 FAIL (2025-12-29 12:41:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: CU5DA6PX4YVIFWK4J3425WNBCV62DHWO47JY6V5K6DALA54OGRKA) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: 6W5E24DBPHFKZUQQ6VH7IXF65ZGWVKQFXP5ITBSADFQA4DWW4WFA) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: DV5IJ67OMZW6C4GHHYVF7KRJN76MTT4LCWCNPAXBTGB7VVX2ZTHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: XIP3WOQLA4XRM472AJHOAVMICJLMAUD7OVNOLP3RG7HARFOU2DVQ) context deadline exceeded ======LOGS====== Dec 29 12:39:58.079293040 (573h10m57.950376086s) [upload=502495]: Starting upload Dec 29 12:39:58.079401347 (573h10m57.950484382s) [upload=502495]: Done starting upload Dec 29 12:39:58.079483573 (573h10m57.950566618s) [upload=502495]: Uploading segments... Dec 29 12:39:58.079697892 (573h10m57.950780927s) [upload=502495]: Got next segment. Inline: false Dec 29 12:39:58.079745191 (573h10m57.950828236s) [upload=502495]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:39:58.079751894 (573h10m57.950834939s) [upload=502495]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:39:58.324037700 (573h10m58.195120745s) [seg_pos={0 0}, upload=502495]: Begin upload segment... Dec 29 12:39:58.324121059 (573h10m58.195204104s) [seg_pos={0 0}, upload=502495]: Done begin upload segment. Dec 29 12:39:58.324508537 (573h10m58.195591572s) [noise=true , address=45.91.116.6:4974 , piece_id=WIPNRFHYXBU4LUV2M53RC3F3UEPSSGHSV52U44PH5CBNU36O2CGQ, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.325309292 (573h10m58.196392337s) [noise=true , address=145.79.227.6:4259 , piece_id=IN3S5E3FZ7DSAF4PP6JSBNMSFZPIQL2LYH4VK2IYRV6MO6BXTFBA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.327037432 (573h10m58.198120467s) [noise=true , address=145.79.227.13:4802 , piece_id=TDV4WFGSDNSGHS4OLWKFVK4GK3YD62RA3Q6CXH27DFRQ2WHARNGA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.327819862 (573h10m58.198902897s) [noise=true , address=77.90.31.12:4589 , piece_id=6QWZ4J4JKVG6AZF67TJCTDFZ3BTSYNGYDR5JWUWY6CJTPDLSSMYA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.328759832 (573h10m58.199842867s) [noise=true , address=45.91.116.5:4298 , piece_id=DV5IJ67OMZW6C4GHHYVF7KRJN76MTT4LCWCNPAXBTGB7VVX2ZTHA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.329778361 (573h10m58.200861396s) [noise=true , address=145.79.227.16:4911 , piece_id=HIKXTI3QWEB36ZRDN3JKY3EXMTPIEURY3NI3HBDEBTFNPLXEPTLA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.330143697 (573h10m58.201226732s) [noise=true , address=145.79.227.11:4732 , piece_id=XH3VKJFMRUICQAOL55CCVRIKSOZNFKOELTXXPGVBWV73UBFJKPYQ, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.330486359 (573h10m58.201569394s) [noise=true , address=77.90.31.19:4844 , piece_id=XIP3WOQLA4XRM472AJHOAVMICJLMAUD7OVNOLP3RG7HARFOU2DVQ, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.330888655 (573h10m58.20197169s) [noise=true , address=77.90.31.15:4697 , piece_id=CR3FVBY3IABGTT7VTJ7RZ3VAERQFFXBG2C6RNDQRZHECK6QTKOXA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.331282535 (573h10m58.20236557s) [noise=true , address=45.91.116.10:4082 , piece_id=CU5DA6PX4YVIFWK4J3425WNBCV62DHWO47JY6V5K6DALA54OGRKA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.331646378 (573h10m58.202729414s) [noise=true , address=145.79.227.38:4977 , piece_id=IIXS6JWAIRDLRHH24T7POQILIEB2DR3F3WEOWGLJK3ZDVALPOYHA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.331878901 (573h10m58.202961936s) [noise=true , address=145.79.227.3:4149 , piece_id=ZPPSSXJH5QPBC5UREMQQRML2TLW6L6RSK7WJ5TDIBG5N5VBHGTBQ, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.332011724 (573h10m58.203094759s) [noise=true , address=145.79.227.42:4122 , piece_id=YUIQXL7EORDFJ5ALQ5AK6EIB7W2BVK25K5CYVTEBRA23YV43VWTQ, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.332324780 (573h10m58.203407815s) [noise=true , address=145.79.227.52:4446 , piece_id=SMWRXV7DXGUUDRGZ4UZS2NT5CPRTK4R2JDZZ7ID4B5OI65YWWJOA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.332627797 (573h10m58.203710832s) [noise=true , address=45.91.116.7:4012 , piece_id=6W5E24DBPHFKZUQQ6VH7IXF65ZGWVKQFXP5ITBSADFQA4DWW4WFA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.338701111 (573h10m58.209784146s) [noise=true , address=45.91.116.12:4226 , piece_id=BTT33OZFMS22BD2AVBNWYXE34OVCREDUQPAB5YWYTCMFINBUGWYA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:40:00.040558968 (573h10m59.911642013s) [upload=502495]: Next returned nil segment Dec 29 12:40:00.040622679 (573h10m59.911705724s) [upload=502495]: Waiting for error group managing segments... Dec 29 12:40:00.321872750 (573h11m0.192955785s) [noise=true , address=145.79.227.6:4259 , piece_id=IN3S5E3FZ7DSAF4PP6JSBNMSFZPIQL2LYH4VK2IYRV6MO6BXTFBA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.325245653 (573h11m0.196328688s) [noise=true , address=145.79.227.11:4732 , piece_id=XH3VKJFMRUICQAOL55CCVRIKSOZNFKOELTXXPGVBWV73UBFJKPYQ, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.327096597 (573h11m0.198179643s) [noise=true , address=145.79.227.42:4122 , piece_id=YUIQXL7EORDFJ5ALQ5AK6EIB7W2BVK25K5CYVTEBRA23YV43VWTQ, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.335103083 (573h11m0.206186128s) [noise=true , address=145.79.227.38:4977 , piece_id=IIXS6JWAIRDLRHH24T7POQILIEB2DR3F3WEOWGLJK3ZDVALPOYHA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.335909939 (573h11m0.206992984s) [noise=true , address=145.79.227.16:4911 , piece_id=HIKXTI3QWEB36ZRDN3JKY3EXMTPIEURY3NI3HBDEBTFNPLXEPTLA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.336842656 (573h11m0.207925691s) [noise=true , address=145.79.227.52:4446 , piece_id=SMWRXV7DXGUUDRGZ4UZS2NT5CPRTK4R2JDZZ7ID4B5OI65YWWJOA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.348942666 (573h11m0.220025711s) [noise=true , address=145.79.227.3:4149 , piece_id=ZPPSSXJH5QPBC5UREMQQRML2TLW6L6RSK7WJ5TDIBG5N5VBHGTBQ, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.356133288 (573h11m0.227216333s) [noise=true , address=145.79.227.13:4802 , piece_id=TDV4WFGSDNSGHS4OLWKFVK4GK3YD62RA3Q6CXH27DFRQ2WHARNGA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.596712026 (573h11m0.467795071s) [noise=true , address=45.91.116.5:4298 , piece_id=DV5IJ67OMZW6C4GHHYVF7KRJN76MTT4LCWCNPAXBTGB7VVX2ZTHA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.621897211 (573h11m0.492980256s) [noise=true , address=45.91.116.12:4226 , piece_id=BTT33OZFMS22BD2AVBNWYXE34OVCREDUQPAB5YWYTCMFINBUGWYA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.634502033 (573h11m0.505585078s) [noise=true , address=77.90.31.19:4844 , piece_id=XIP3WOQLA4XRM472AJHOAVMICJLMAUD7OVNOLP3RG7HARFOU2DVQ, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.697141127 (573h11m0.568224162s) [noise=true , address=45.91.116.10:4082 , piece_id=CU5DA6PX4YVIFWK4J3425WNBCV62DHWO47JY6V5K6DALA54OGRKA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.908852586 (573h11m0.779935631s) [noise=true , address=45.91.116.7:4012 , piece_id=6W5E24DBPHFKZUQQ6VH7IXF65ZGWVKQFXP5ITBSADFQA4DWW4WFA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.908873036 (573h11m0.779956071s) [seg_pos={0 0}, upload=502495]: Segment reached optimal threshold of 13 pieces. Dec 29 12:40:00.909170142 (573h11m0.780253177s) [noise=true , address=77.90.31.15:4697 , piece_id=CR3FVBY3IABGTT7VTJ7RZ3VAERQFFXBG2C6RNDQRZHECK6QTKOXA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 29 12:40:00.909178027 (573h11m0.780261072s) [noise=true , address=77.90.31.15:4697 , piece_id=CR3FVBY3IABGTT7VTJ7RZ3VAERQFFXBG2C6RNDQRZHECK6QTKOXA, seg_pos={0 0}, upload=502495]: Long tail cancellation due to optimal threshold Dec 29 12:40:00.909232610 (573h11m0.780315645s) [noise=true , address=77.90.31.12:4589 , piece_id=6QWZ4J4JKVG6AZF67TJCTDFZ3BTSYNGYDR5JWUWY6CJTPDLSSMYA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:41878->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:41878->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41878->77.90.31.12:4589: use of closed network connection Dec 29 12:40:00.909248421 (573h11m0.780331466s) [noise=true , address=77.90.31.12:4589 , piece_id=6QWZ4J4JKVG6AZF67TJCTDFZ3BTSYNGYDR5JWUWY6CJTPDLSSMYA, seg_pos={0 0}, upload=502495]: Long tail cancellation due to optimal threshold Dec 29 12:40:00.909337450 (573h11m0.780420485s) [noise=true , address=45.91.116.6:4974 , piece_id=WIPNRFHYXBU4LUV2M53RC3F3UEPSSGHSV52U44PH5CBNU36O2CGQ, seg_pos={0 0}, upload=502495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:40:00.909352629 (573h11m0.780435704s) [noise=true , address=45.91.116.6:4974 , piece_id=WIPNRFHYXBU4LUV2M53RC3F3UEPSSGHSV52U44PH5CBNU36O2CGQ, seg_pos={0 0}, upload=502495]: Long tail cancellation due to optimal threshold Dec 29 12:40:00.909382035 (573h11m0.78046508s) [seg_pos={0 0}, upload=502495]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:40:00.909502023 (573h11m0.780585068s) [upload=502495]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:40:00.909508005 (573h11m0.78059104s) [upload=502495]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:40:01.205188494 (573h11m1.076286077s) [upload=502495]: Done uploading segments... Dec 29 12:40:01.205233319 (573h11m1.076316364s) [upload=502495]: Upload finished. err: <nil> Duration: 5.867112963s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X.......X..........X...............] duration: 42.727467379s trace: https://jaeger.dev.storj.io/trace/68764c6cb4286314 FAIL (2025-12-29 12:41:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528204662 (572h59m20.399287707s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528497489 (572h59m20.399580524s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528793153 (572h59m20.399876198s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529083555 (572h59m20.40016659s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529391081 (572h59m20.400474126s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529735096 (572h59m20.400818131s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530109970 (572h59m20.401193015s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530449556 (572h59m20.401532591s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530831554 (572h59m20.401914599s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531229512 (572h59m20.402312557s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531698945 (572h59m20.402781991s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532204238 (572h59m20.403287283s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532744366 (572h59m20.403827401s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533231203 (572h59m20.404314248s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533740113 (572h59m20.404823148s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:21.876539279 (572h59m21.747622324s) [upload=502322]: Next returned nil segment Dec 29 12:28:21.876614272 (572h59m21.747697307s) [upload=502322]: Waiting for error group managing segments... Dec 29 12:28:22.383674429 (572h59m22.254757464s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389083709 (572h59m22.260166754s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389450968 (572h59m22.260534003s) [noise=true , address=145.79.227.46:4625 , piece_id=6NEND2CKGIZLK5Y6IMM4XB6SXPSXUYSZIVZ7CZ25IKAHFFW33C5Q, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.418373151 (572h59m22.289456186s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.619432477 (572h59m22.490515552s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.741876388 (572h59m22.612959423s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.133731187 (572h59m23.004814222s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.146041929 (572h59m23.017124964s) [noise=true , address=193.149.164.143:4697 , piece_id=HZZY3SUOEC7AIHSW6MKZRNZVSOGXSYBYSUKN7STBFTXWAOMDRYBA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.261354601 (572h59m23.132437636s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.291387197 (572h59m23.162470243s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.318370928 (572h59m23.189453973s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.321236023 (572h59m23.192319058s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.408408673 (572h59m23.279491718s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419570137 (572h59m23.290653172s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419592470 (572h59m23.290675515s) [seg_pos={0 0}, upload=502322]: Segment reached optimal threshold of 14 pieces. Dec 29 12:28:23.419837406 (572h59m23.290920441s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123jUFZTVjq6adDuvt3cco7dTvwNTQkszZX7CeajMfbeCUmEmuJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419845241 (572h59m23.290928276s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.419873174 (572h59m23.290956209s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419882021 (572h59m23.290965076s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420073716 (572h59m23.291156751s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection Dec 29 12:28:23.420082262 (572h59m23.291165307s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420257506 (572h59m23.291340551s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: read tcp 188.166.225.170:41270->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:28:23.420264649 (572h59m23.291347684s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420281822 (572h59m23.291364857s) [seg_pos={0 0}, upload=502322]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:28:23.420364349 (572h59m23.291447394s) [upload=502322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:28:23.420370270 (572h59m23.291453315s) [upload=502322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:28:23.725225770 (572h59m23.596308815s) [upload=502322]: Done uploading segments... Dec 29 12:28:23.725255426 (572h59m23.596338481s) [upload=502322]: Upload finished. err: <nil> Duration: 42.727488819s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..X.X.......X.XX...X.......X.X..XXX....X...X..X....XXXX..XX.] duration: 5.705774881s trace: https://jaeger.dev.storj.io/trace/21fde371d273b7b9 FAIL (2025-12-29 12:41:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: CU5DA6PX4YVIFWK4J3425WNBCV62DHWO47JY6V5K6DALA54OGRKA) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: 6W5E24DBPHFKZUQQ6VH7IXF65ZGWVKQFXP5ITBSADFQA4DWW4WFA) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: DV5IJ67OMZW6C4GHHYVF7KRJN76MTT4LCWCNPAXBTGB7VVX2ZTHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: XIP3WOQLA4XRM472AJHOAVMICJLMAUD7OVNOLP3RG7HARFOU2DVQ) context deadline exceeded ======LOGS====== Dec 29 12:39:58.079293040 (573h10m57.950376086s) [upload=502495]: Starting upload Dec 29 12:39:58.079401347 (573h10m57.950484382s) [upload=502495]: Done starting upload Dec 29 12:39:58.079483573 (573h10m57.950566618s) [upload=502495]: Uploading segments... Dec 29 12:39:58.079697892 (573h10m57.950780927s) [upload=502495]: Got next segment. Inline: false Dec 29 12:39:58.079745191 (573h10m57.950828236s) [upload=502495]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:39:58.079751894 (573h10m57.950834939s) [upload=502495]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:39:58.324037700 (573h10m58.195120745s) [seg_pos={0 0}, upload=502495]: Begin upload segment... Dec 29 12:39:58.324121059 (573h10m58.195204104s) [seg_pos={0 0}, upload=502495]: Done begin upload segment. Dec 29 12:39:58.324508537 (573h10m58.195591572s) [noise=true , address=45.91.116.6:4974 , piece_id=WIPNRFHYXBU4LUV2M53RC3F3UEPSSGHSV52U44PH5CBNU36O2CGQ, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.325309292 (573h10m58.196392337s) [noise=true , address=145.79.227.6:4259 , piece_id=IN3S5E3FZ7DSAF4PP6JSBNMSFZPIQL2LYH4VK2IYRV6MO6BXTFBA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.327037432 (573h10m58.198120467s) [noise=true , address=145.79.227.13:4802 , piece_id=TDV4WFGSDNSGHS4OLWKFVK4GK3YD62RA3Q6CXH27DFRQ2WHARNGA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.327819862 (573h10m58.198902897s) [noise=true , address=77.90.31.12:4589 , piece_id=6QWZ4J4JKVG6AZF67TJCTDFZ3BTSYNGYDR5JWUWY6CJTPDLSSMYA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.328759832 (573h10m58.199842867s) [noise=true , address=45.91.116.5:4298 , piece_id=DV5IJ67OMZW6C4GHHYVF7KRJN76MTT4LCWCNPAXBTGB7VVX2ZTHA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.329778361 (573h10m58.200861396s) [noise=true , address=145.79.227.16:4911 , piece_id=HIKXTI3QWEB36ZRDN3JKY3EXMTPIEURY3NI3HBDEBTFNPLXEPTLA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.330143697 (573h10m58.201226732s) [noise=true , address=145.79.227.11:4732 , piece_id=XH3VKJFMRUICQAOL55CCVRIKSOZNFKOELTXXPGVBWV73UBFJKPYQ, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.330486359 (573h10m58.201569394s) [noise=true , address=77.90.31.19:4844 , piece_id=XIP3WOQLA4XRM472AJHOAVMICJLMAUD7OVNOLP3RG7HARFOU2DVQ, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.330888655 (573h10m58.20197169s) [noise=true , address=77.90.31.15:4697 , piece_id=CR3FVBY3IABGTT7VTJ7RZ3VAERQFFXBG2C6RNDQRZHECK6QTKOXA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.331282535 (573h10m58.20236557s) [noise=true , address=45.91.116.10:4082 , piece_id=CU5DA6PX4YVIFWK4J3425WNBCV62DHWO47JY6V5K6DALA54OGRKA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.331646378 (573h10m58.202729414s) [noise=true , address=145.79.227.38:4977 , piece_id=IIXS6JWAIRDLRHH24T7POQILIEB2DR3F3WEOWGLJK3ZDVALPOYHA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.331878901 (573h10m58.202961936s) [noise=true , address=145.79.227.3:4149 , piece_id=ZPPSSXJH5QPBC5UREMQQRML2TLW6L6RSK7WJ5TDIBG5N5VBHGTBQ, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.332011724 (573h10m58.203094759s) [noise=true , address=145.79.227.42:4122 , piece_id=YUIQXL7EORDFJ5ALQ5AK6EIB7W2BVK25K5CYVTEBRA23YV43VWTQ, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.332324780 (573h10m58.203407815s) [noise=true , address=145.79.227.52:4446 , piece_id=SMWRXV7DXGUUDRGZ4UZS2NT5CPRTK4R2JDZZ7ID4B5OI65YWWJOA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.332627797 (573h10m58.203710832s) [noise=true , address=45.91.116.7:4012 , piece_id=6W5E24DBPHFKZUQQ6VH7IXF65ZGWVKQFXP5ITBSADFQA4DWW4WFA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.338701111 (573h10m58.209784146s) [noise=true , address=45.91.116.12:4226 , piece_id=BTT33OZFMS22BD2AVBNWYXE34OVCREDUQPAB5YWYTCMFINBUGWYA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:40:00.040558968 (573h10m59.911642013s) [upload=502495]: Next returned nil segment Dec 29 12:40:00.040622679 (573h10m59.911705724s) [upload=502495]: Waiting for error group managing segments... Dec 29 12:40:00.321872750 (573h11m0.192955785s) [noise=true , address=145.79.227.6:4259 , piece_id=IN3S5E3FZ7DSAF4PP6JSBNMSFZPIQL2LYH4VK2IYRV6MO6BXTFBA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.325245653 (573h11m0.196328688s) [noise=true , address=145.79.227.11:4732 , piece_id=XH3VKJFMRUICQAOL55CCVRIKSOZNFKOELTXXPGVBWV73UBFJKPYQ, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.327096597 (573h11m0.198179643s) [noise=true , address=145.79.227.42:4122 , piece_id=YUIQXL7EORDFJ5ALQ5AK6EIB7W2BVK25K5CYVTEBRA23YV43VWTQ, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.335103083 (573h11m0.206186128s) [noise=true , address=145.79.227.38:4977 , piece_id=IIXS6JWAIRDLRHH24T7POQILIEB2DR3F3WEOWGLJK3ZDVALPOYHA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.335909939 (573h11m0.206992984s) [noise=true , address=145.79.227.16:4911 , piece_id=HIKXTI3QWEB36ZRDN3JKY3EXMTPIEURY3NI3HBDEBTFNPLXEPTLA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.336842656 (573h11m0.207925691s) [noise=true , address=145.79.227.52:4446 , piece_id=SMWRXV7DXGUUDRGZ4UZS2NT5CPRTK4R2JDZZ7ID4B5OI65YWWJOA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.348942666 (573h11m0.220025711s) [noise=true , address=145.79.227.3:4149 , piece_id=ZPPSSXJH5QPBC5UREMQQRML2TLW6L6RSK7WJ5TDIBG5N5VBHGTBQ, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.356133288 (573h11m0.227216333s) [noise=true , address=145.79.227.13:4802 , piece_id=TDV4WFGSDNSGHS4OLWKFVK4GK3YD62RA3Q6CXH27DFRQ2WHARNGA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.596712026 (573h11m0.467795071s) [noise=true , address=45.91.116.5:4298 , piece_id=DV5IJ67OMZW6C4GHHYVF7KRJN76MTT4LCWCNPAXBTGB7VVX2ZTHA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.621897211 (573h11m0.492980256s) [noise=true , address=45.91.116.12:4226 , piece_id=BTT33OZFMS22BD2AVBNWYXE34OVCREDUQPAB5YWYTCMFINBUGWYA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.634502033 (573h11m0.505585078s) [noise=true , address=77.90.31.19:4844 , piece_id=XIP3WOQLA4XRM472AJHOAVMICJLMAUD7OVNOLP3RG7HARFOU2DVQ, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.697141127 (573h11m0.568224162s) [noise=true , address=45.91.116.10:4082 , piece_id=CU5DA6PX4YVIFWK4J3425WNBCV62DHWO47JY6V5K6DALA54OGRKA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.908852586 (573h11m0.779935631s) [noise=true , address=45.91.116.7:4012 , piece_id=6W5E24DBPHFKZUQQ6VH7IXF65ZGWVKQFXP5ITBSADFQA4DWW4WFA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.908873036 (573h11m0.779956071s) [seg_pos={0 0}, upload=502495]: Segment reached optimal threshold of 13 pieces. Dec 29 12:40:00.909170142 (573h11m0.780253177s) [noise=true , address=77.90.31.15:4697 , piece_id=CR3FVBY3IABGTT7VTJ7RZ3VAERQFFXBG2C6RNDQRZHECK6QTKOXA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 29 12:40:00.909178027 (573h11m0.780261072s) [noise=true , address=77.90.31.15:4697 , piece_id=CR3FVBY3IABGTT7VTJ7RZ3VAERQFFXBG2C6RNDQRZHECK6QTKOXA, seg_pos={0 0}, upload=502495]: Long tail cancellation due to optimal threshold Dec 29 12:40:00.909232610 (573h11m0.780315645s) [noise=true , address=77.90.31.12:4589 , piece_id=6QWZ4J4JKVG6AZF67TJCTDFZ3BTSYNGYDR5JWUWY6CJTPDLSSMYA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:41878->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:41878->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41878->77.90.31.12:4589: use of closed network connection Dec 29 12:40:00.909248421 (573h11m0.780331466s) [noise=true , address=77.90.31.12:4589 , piece_id=6QWZ4J4JKVG6AZF67TJCTDFZ3BTSYNGYDR5JWUWY6CJTPDLSSMYA, seg_pos={0 0}, upload=502495]: Long tail cancellation due to optimal threshold Dec 29 12:40:00.909337450 (573h11m0.780420485s) [noise=true , address=45.91.116.6:4974 , piece_id=WIPNRFHYXBU4LUV2M53RC3F3UEPSSGHSV52U44PH5CBNU36O2CGQ, seg_pos={0 0}, upload=502495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:40:00.909352629 (573h11m0.780435704s) [noise=true , address=45.91.116.6:4974 , piece_id=WIPNRFHYXBU4LUV2M53RC3F3UEPSSGHSV52U44PH5CBNU36O2CGQ, seg_pos={0 0}, upload=502495]: Long tail cancellation due to optimal threshold Dec 29 12:40:00.909382035 (573h11m0.78046508s) [seg_pos={0 0}, upload=502495]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:40:00.909502023 (573h11m0.780585068s) [upload=502495]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:40:00.909508005 (573h11m0.78059104s) [upload=502495]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:40:01.205188494 (573h11m1.076286077s) [upload=502495]: Done uploading segments... Dec 29 12:40:01.205233319 (573h11m1.076316364s) [upload=502495]: Upload finished. err: <nil> Duration: 5.705791793s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXXXXXXXXX.XXXXXXXX] duration: 5.001147681s trace: https://jaeger.dev.storj.io/trace/5c453fcaae0bc3bf FAIL (2025-12-29 12:40:55 UTC) orbiter warning: context deadline exceeded Duration: 5.001166898s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................X.......X..........X..............] duration: 52.972212034s trace: https://jaeger.dev.storj.io/trace/68764c6cb4286314 FAIL (2025-12-29 12:40:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528204662 (572h59m20.399287707s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528497489 (572h59m20.399580524s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528793153 (572h59m20.399876198s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529083555 (572h59m20.40016659s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529391081 (572h59m20.400474126s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529735096 (572h59m20.400818131s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530109970 (572h59m20.401193015s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530449556 (572h59m20.401532591s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530831554 (572h59m20.401914599s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531229512 (572h59m20.402312557s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531698945 (572h59m20.402781991s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532204238 (572h59m20.403287283s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532744366 (572h59m20.403827401s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533231203 (572h59m20.404314248s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533740113 (572h59m20.404823148s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:21.876539279 (572h59m21.747622324s) [upload=502322]: Next returned nil segment Dec 29 12:28:21.876614272 (572h59m21.747697307s) [upload=502322]: Waiting for error group managing segments... Dec 29 12:28:22.383674429 (572h59m22.254757464s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389083709 (572h59m22.260166754s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389450968 (572h59m22.260534003s) [noise=true , address=145.79.227.46:4625 , piece_id=6NEND2CKGIZLK5Y6IMM4XB6SXPSXUYSZIVZ7CZ25IKAHFFW33C5Q, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.418373151 (572h59m22.289456186s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.619432477 (572h59m22.490515552s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.741876388 (572h59m22.612959423s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.133731187 (572h59m23.004814222s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.146041929 (572h59m23.017124964s) [noise=true , address=193.149.164.143:4697 , piece_id=HZZY3SUOEC7AIHSW6MKZRNZVSOGXSYBYSUKN7STBFTXWAOMDRYBA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.261354601 (572h59m23.132437636s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.291387197 (572h59m23.162470243s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.318370928 (572h59m23.189453973s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.321236023 (572h59m23.192319058s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.408408673 (572h59m23.279491718s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419570137 (572h59m23.290653172s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419592470 (572h59m23.290675515s) [seg_pos={0 0}, upload=502322]: Segment reached optimal threshold of 14 pieces. Dec 29 12:28:23.419837406 (572h59m23.290920441s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123jUFZTVjq6adDuvt3cco7dTvwNTQkszZX7CeajMfbeCUmEmuJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419845241 (572h59m23.290928276s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.419873174 (572h59m23.290956209s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419882021 (572h59m23.290965076s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420073716 (572h59m23.291156751s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection Dec 29 12:28:23.420082262 (572h59m23.291165307s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420257506 (572h59m23.291340551s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: read tcp 188.166.225.170:41270->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:28:23.420264649 (572h59m23.291347684s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420281822 (572h59m23.291364857s) [seg_pos={0 0}, upload=502322]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:28:23.420364349 (572h59m23.291447394s) [upload=502322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:28:23.420370270 (572h59m23.291453315s) [upload=502322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:28:23.725225770 (572h59m23.596308815s) [upload=502322]: Done uploading segments... Dec 29 12:28:23.725255426 (572h59m23.596338481s) [upload=502322]: Upload finished. err: <nil> Duration: 52.972231381s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 42 out of 60 is failed (threshold=6) [.XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXXXXXXXXX.XXXXXXX] duration: 5.001685805s trace: https://jaeger.dev.storj.io/trace/624b21da5f99a626 FAIL (2025-12-29 12:40:38 UTC) orbiter warning: context deadline exceeded Duration: 5.001710763s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X..X.X.......X.XX...X.......X.X..XXX....X...X..X....XXXX..XX] duration: 30.004040939s trace: https://jaeger.dev.storj.io/trace/21fde371d273b7b9 FAIL (2025-12-29 12:40:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: CU5DA6PX4YVIFWK4J3425WNBCV62DHWO47JY6V5K6DALA54OGRKA) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: 6W5E24DBPHFKZUQQ6VH7IXF65ZGWVKQFXP5ITBSADFQA4DWW4WFA) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: DV5IJ67OMZW6C4GHHYVF7KRJN76MTT4LCWCNPAXBTGB7VVX2ZTHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: XIP3WOQLA4XRM472AJHOAVMICJLMAUD7OVNOLP3RG7HARFOU2DVQ) context deadline exceeded ======LOGS====== Dec 29 12:39:58.079293040 (573h10m57.950376086s) [upload=502495]: Starting upload Dec 29 12:39:58.079401347 (573h10m57.950484382s) [upload=502495]: Done starting upload Dec 29 12:39:58.079483573 (573h10m57.950566618s) [upload=502495]: Uploading segments... Dec 29 12:39:58.079697892 (573h10m57.950780927s) [upload=502495]: Got next segment. Inline: false Dec 29 12:39:58.079745191 (573h10m57.950828236s) [upload=502495]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:39:58.079751894 (573h10m57.950834939s) [upload=502495]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:39:58.324037700 (573h10m58.195120745s) [seg_pos={0 0}, upload=502495]: Begin upload segment... Dec 29 12:39:58.324121059 (573h10m58.195204104s) [seg_pos={0 0}, upload=502495]: Done begin upload segment. Dec 29 12:39:58.324508537 (573h10m58.195591572s) [noise=true , address=45.91.116.6:4974 , piece_id=WIPNRFHYXBU4LUV2M53RC3F3UEPSSGHSV52U44PH5CBNU36O2CGQ, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.325309292 (573h10m58.196392337s) [noise=true , address=145.79.227.6:4259 , piece_id=IN3S5E3FZ7DSAF4PP6JSBNMSFZPIQL2LYH4VK2IYRV6MO6BXTFBA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.327037432 (573h10m58.198120467s) [noise=true , address=145.79.227.13:4802 , piece_id=TDV4WFGSDNSGHS4OLWKFVK4GK3YD62RA3Q6CXH27DFRQ2WHARNGA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.327819862 (573h10m58.198902897s) [noise=true , address=77.90.31.12:4589 , piece_id=6QWZ4J4JKVG6AZF67TJCTDFZ3BTSYNGYDR5JWUWY6CJTPDLSSMYA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.328759832 (573h10m58.199842867s) [noise=true , address=45.91.116.5:4298 , piece_id=DV5IJ67OMZW6C4GHHYVF7KRJN76MTT4LCWCNPAXBTGB7VVX2ZTHA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.329778361 (573h10m58.200861396s) [noise=true , address=145.79.227.16:4911 , piece_id=HIKXTI3QWEB36ZRDN3JKY3EXMTPIEURY3NI3HBDEBTFNPLXEPTLA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.330143697 (573h10m58.201226732s) [noise=true , address=145.79.227.11:4732 , piece_id=XH3VKJFMRUICQAOL55CCVRIKSOZNFKOELTXXPGVBWV73UBFJKPYQ, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.330486359 (573h10m58.201569394s) [noise=true , address=77.90.31.19:4844 , piece_id=XIP3WOQLA4XRM472AJHOAVMICJLMAUD7OVNOLP3RG7HARFOU2DVQ, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.330888655 (573h10m58.20197169s) [noise=true , address=77.90.31.15:4697 , piece_id=CR3FVBY3IABGTT7VTJ7RZ3VAERQFFXBG2C6RNDQRZHECK6QTKOXA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.331282535 (573h10m58.20236557s) [noise=true , address=45.91.116.10:4082 , piece_id=CU5DA6PX4YVIFWK4J3425WNBCV62DHWO47JY6V5K6DALA54OGRKA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.331646378 (573h10m58.202729414s) [noise=true , address=145.79.227.38:4977 , piece_id=IIXS6JWAIRDLRHH24T7POQILIEB2DR3F3WEOWGLJK3ZDVALPOYHA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.331878901 (573h10m58.202961936s) [noise=true , address=145.79.227.3:4149 , piece_id=ZPPSSXJH5QPBC5UREMQQRML2TLW6L6RSK7WJ5TDIBG5N5VBHGTBQ, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.332011724 (573h10m58.203094759s) [noise=true , address=145.79.227.42:4122 , piece_id=YUIQXL7EORDFJ5ALQ5AK6EIB7W2BVK25K5CYVTEBRA23YV43VWTQ, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.332324780 (573h10m58.203407815s) [noise=true , address=145.79.227.52:4446 , piece_id=SMWRXV7DXGUUDRGZ4UZS2NT5CPRTK4R2JDZZ7ID4B5OI65YWWJOA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.332627797 (573h10m58.203710832s) [noise=true , address=45.91.116.7:4012 , piece_id=6W5E24DBPHFKZUQQ6VH7IXF65ZGWVKQFXP5ITBSADFQA4DWW4WFA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.338701111 (573h10m58.209784146s) [noise=true , address=45.91.116.12:4226 , piece_id=BTT33OZFMS22BD2AVBNWYXE34OVCREDUQPAB5YWYTCMFINBUGWYA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:40:00.040558968 (573h10m59.911642013s) [upload=502495]: Next returned nil segment Dec 29 12:40:00.040622679 (573h10m59.911705724s) [upload=502495]: Waiting for error group managing segments... Dec 29 12:40:00.321872750 (573h11m0.192955785s) [noise=true , address=145.79.227.6:4259 , piece_id=IN3S5E3FZ7DSAF4PP6JSBNMSFZPIQL2LYH4VK2IYRV6MO6BXTFBA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.325245653 (573h11m0.196328688s) [noise=true , address=145.79.227.11:4732 , piece_id=XH3VKJFMRUICQAOL55CCVRIKSOZNFKOELTXXPGVBWV73UBFJKPYQ, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.327096597 (573h11m0.198179643s) [noise=true , address=145.79.227.42:4122 , piece_id=YUIQXL7EORDFJ5ALQ5AK6EIB7W2BVK25K5CYVTEBRA23YV43VWTQ, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.335103083 (573h11m0.206186128s) [noise=true , address=145.79.227.38:4977 , piece_id=IIXS6JWAIRDLRHH24T7POQILIEB2DR3F3WEOWGLJK3ZDVALPOYHA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.335909939 (573h11m0.206992984s) [noise=true , address=145.79.227.16:4911 , piece_id=HIKXTI3QWEB36ZRDN3JKY3EXMTPIEURY3NI3HBDEBTFNPLXEPTLA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.336842656 (573h11m0.207925691s) [noise=true , address=145.79.227.52:4446 , piece_id=SMWRXV7DXGUUDRGZ4UZS2NT5CPRTK4R2JDZZ7ID4B5OI65YWWJOA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.348942666 (573h11m0.220025711s) [noise=true , address=145.79.227.3:4149 , piece_id=ZPPSSXJH5QPBC5UREMQQRML2TLW6L6RSK7WJ5TDIBG5N5VBHGTBQ, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.356133288 (573h11m0.227216333s) [noise=true , address=145.79.227.13:4802 , piece_id=TDV4WFGSDNSGHS4OLWKFVK4GK3YD62RA3Q6CXH27DFRQ2WHARNGA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.596712026 (573h11m0.467795071s) [noise=true , address=45.91.116.5:4298 , piece_id=DV5IJ67OMZW6C4GHHYVF7KRJN76MTT4LCWCNPAXBTGB7VVX2ZTHA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.621897211 (573h11m0.492980256s) [noise=true , address=45.91.116.12:4226 , piece_id=BTT33OZFMS22BD2AVBNWYXE34OVCREDUQPAB5YWYTCMFINBUGWYA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.634502033 (573h11m0.505585078s) [noise=true , address=77.90.31.19:4844 , piece_id=XIP3WOQLA4XRM472AJHOAVMICJLMAUD7OVNOLP3RG7HARFOU2DVQ, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.697141127 (573h11m0.568224162s) [noise=true , address=45.91.116.10:4082 , piece_id=CU5DA6PX4YVIFWK4J3425WNBCV62DHWO47JY6V5K6DALA54OGRKA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.908852586 (573h11m0.779935631s) [noise=true , address=45.91.116.7:4012 , piece_id=6W5E24DBPHFKZUQQ6VH7IXF65ZGWVKQFXP5ITBSADFQA4DWW4WFA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.908873036 (573h11m0.779956071s) [seg_pos={0 0}, upload=502495]: Segment reached optimal threshold of 13 pieces. Dec 29 12:40:00.909170142 (573h11m0.780253177s) [noise=true , address=77.90.31.15:4697 , piece_id=CR3FVBY3IABGTT7VTJ7RZ3VAERQFFXBG2C6RNDQRZHECK6QTKOXA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 29 12:40:00.909178027 (573h11m0.780261072s) [noise=true , address=77.90.31.15:4697 , piece_id=CR3FVBY3IABGTT7VTJ7RZ3VAERQFFXBG2C6RNDQRZHECK6QTKOXA, seg_pos={0 0}, upload=502495]: Long tail cancellation due to optimal threshold Dec 29 12:40:00.909232610 (573h11m0.780315645s) [noise=true , address=77.90.31.12:4589 , piece_id=6QWZ4J4JKVG6AZF67TJCTDFZ3BTSYNGYDR5JWUWY6CJTPDLSSMYA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:41878->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:41878->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41878->77.90.31.12:4589: use of closed network connection Dec 29 12:40:00.909248421 (573h11m0.780331466s) [noise=true , address=77.90.31.12:4589 , piece_id=6QWZ4J4JKVG6AZF67TJCTDFZ3BTSYNGYDR5JWUWY6CJTPDLSSMYA, seg_pos={0 0}, upload=502495]: Long tail cancellation due to optimal threshold Dec 29 12:40:00.909337450 (573h11m0.780420485s) [noise=true , address=45.91.116.6:4974 , piece_id=WIPNRFHYXBU4LUV2M53RC3F3UEPSSGHSV52U44PH5CBNU36O2CGQ, seg_pos={0 0}, upload=502495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:40:00.909352629 (573h11m0.780435704s) [noise=true , address=45.91.116.6:4974 , piece_id=WIPNRFHYXBU4LUV2M53RC3F3UEPSSGHSV52U44PH5CBNU36O2CGQ, seg_pos={0 0}, upload=502495]: Long tail cancellation due to optimal threshold Dec 29 12:40:00.909382035 (573h11m0.78046508s) [seg_pos={0 0}, upload=502495]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:40:00.909502023 (573h11m0.780585068s) [upload=502495]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:40:00.909508005 (573h11m0.78059104s) [upload=502495]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:40:01.205188494 (573h11m1.076286077s) [upload=502495]: Done uploading segments... Dec 29 12:40:01.205233319 (573h11m1.076316364s) [upload=502495]: Upload finished. err: <nil> Duration: 30.004067879s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [..XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXXXXXXXXX.XXXXXX] duration: 5.001459464s trace: https://jaeger.dev.storj.io/trace/7092646b240658b4 FAIL (2025-12-29 12:40:21 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001487318s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................X.......X..........X.............] duration: 48.080279293s trace: https://jaeger.dev.storj.io/trace/68764c6cb4286314 FAIL (2025-12-29 12:39:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528204662 (572h59m20.399287707s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528497489 (572h59m20.399580524s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528793153 (572h59m20.399876198s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529083555 (572h59m20.40016659s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529391081 (572h59m20.400474126s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529735096 (572h59m20.400818131s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530109970 (572h59m20.401193015s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530449556 (572h59m20.401532591s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530831554 (572h59m20.401914599s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531229512 (572h59m20.402312557s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531698945 (572h59m20.402781991s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532204238 (572h59m20.403287283s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532744366 (572h59m20.403827401s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533231203 (572h59m20.404314248s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533740113 (572h59m20.404823148s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:21.876539279 (572h59m21.747622324s) [upload=502322]: Next returned nil segment Dec 29 12:28:21.876614272 (572h59m21.747697307s) [upload=502322]: Waiting for error group managing segments... Dec 29 12:28:22.383674429 (572h59m22.254757464s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389083709 (572h59m22.260166754s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389450968 (572h59m22.260534003s) [noise=true , address=145.79.227.46:4625 , piece_id=6NEND2CKGIZLK5Y6IMM4XB6SXPSXUYSZIVZ7CZ25IKAHFFW33C5Q, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.418373151 (572h59m22.289456186s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.619432477 (572h59m22.490515552s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.741876388 (572h59m22.612959423s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.133731187 (572h59m23.004814222s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.146041929 (572h59m23.017124964s) [noise=true , address=193.149.164.143:4697 , piece_id=HZZY3SUOEC7AIHSW6MKZRNZVSOGXSYBYSUKN7STBFTXWAOMDRYBA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.261354601 (572h59m23.132437636s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.291387197 (572h59m23.162470243s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.318370928 (572h59m23.189453973s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.321236023 (572h59m23.192319058s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.408408673 (572h59m23.279491718s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419570137 (572h59m23.290653172s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419592470 (572h59m23.290675515s) [seg_pos={0 0}, upload=502322]: Segment reached optimal threshold of 14 pieces. Dec 29 12:28:23.419837406 (572h59m23.290920441s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123jUFZTVjq6adDuvt3cco7dTvwNTQkszZX7CeajMfbeCUmEmuJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419845241 (572h59m23.290928276s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.419873174 (572h59m23.290956209s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419882021 (572h59m23.290965076s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420073716 (572h59m23.291156751s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection Dec 29 12:28:23.420082262 (572h59m23.291165307s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420257506 (572h59m23.291340551s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: read tcp 188.166.225.170:41270->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:28:23.420264649 (572h59m23.291347684s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420281822 (572h59m23.291364857s) [seg_pos={0 0}, upload=502322]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:28:23.420364349 (572h59m23.291447394s) [upload=502322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:28:23.420370270 (572h59m23.291453315s) [upload=502322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:28:23.725225770 (572h59m23.596308815s) [upload=502322]: Done uploading segments... Dec 29 12:28:23.725255426 (572h59m23.596338481s) [upload=502322]: Upload finished. err: <nil> Duration: 48.080307627s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XX..X.X.......X.XX...X.......X.X..XXX....X...X..X....XXXX..X] duration: 30.00293109s trace: https://jaeger.dev.storj.io/trace/21c017eafb222c2f FAIL (2025-12-29 12:39:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: HKXTBGBIYUHEIVRYZAWZE74TAZ3KJNN5LFZWHRKMG3BXECXR6JCA) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: UNNLHTGL5GCEKYNGRGB7LDFQDEXNECNDNVKHAZ3VSLVDENFJFRXA) context deadline exceeded ======LOGS====== Dec 29 12:39:13.395539621 (573h10m13.266622656s) [upload=502486]: Starting upload Dec 29 12:39:13.395617038 (573h10m13.266700073s) [upload=502486]: Done starting upload Dec 29 12:39:13.395762635 (573h10m13.26684567s) [upload=502486]: Uploading segments... Dec 29 12:39:13.396294368 (573h10m13.267377433s) [upload=502486]: Got next segment. Inline: false Dec 29 12:39:13.396364732 (573h10m13.267447767s) [upload=502486]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:39:13.396378027 (573h10m13.267461062s) [upload=502486]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:39:13.643573624 (573h10m13.514656659s) [seg_pos={0 0}, upload=502486]: Begin upload segment... Dec 29 12:39:13.643624971 (573h10m13.514708006s) [seg_pos={0 0}, upload=502486]: Done begin upload segment. Dec 29 12:39:13.643653726 (573h10m13.514736771s) [noise=true , address=77.90.31.16:4735 , piece_id=GVVYVKXSABYKTB2LJ4P3DR6WUGT43TWN64KUX5XAFA6XVFLLO77Q, seg_pos={0 0}, upload=502486]: Uploading piece... Dec 29 12:39:13.643730041 (573h10m13.514813086s) [noise=true , address=145.79.227.47:4265 , piece_id=RXKFTRDQRUGFFLBH5F3CK3UDRDAAVSLRFHSXGRRIEPFUPUCFHJ4Q, seg_pos={0 0}, upload=502486]: Uploading piece... Dec 29 12:39:13.644341977 (573h10m13.515425012s) [noise=true , address=145.79.227.42:4122 , piece_id=APA6XQNVMGXNHTILOJNAQIMVP6C55ZK6DYDLF6O355SQAMUJMVKQ, seg_pos={0 0}, upload=502486]: Uploading piece... Dec 29 12:39:13.644565973 (573h10m13.515649018s) [noise=true , address=145.79.227.36:4519 , piece_id=6WJHDDBMWJA5JZPRBKQ7HSWYW5AYNS4DCOBDRDLKST22CXRRWIDA, seg_pos={0 0}, upload=502486]: Uploading piece... Dec 29 12:39:13.644836388 (573h10m13.515919423s) [noise=true , address=77.90.31.11:4555 , piece_id=HKXTBGBIYUHEIVRYZAWZE74TAZ3KJNN5LFZWHRKMG3BXECXR6JCA, seg_pos={0 0}, upload=502486]: Uploading piece... Dec 29 12:39:13.645054754 (573h10m13.516137789s) [noise=true , address=145.79.227.3:4150 , piece_id=CY5SJKOKFGMA36SFMLLVMRSSAE2IYNB6R6XDQGXJALKUU2AHQK4Q, seg_pos={0 0}, upload=502486]: Uploading piece... Dec 29 12:39:13.645249375 (573h10m13.51633241s) [noise=true , address=45.91.116.12:4226 , piece_id=UNNLHTGL5GCEKYNGRGB7LDFQDEXNECNDNVKHAZ3VSLVDENFJFRXA, seg_pos={0 0}, upload=502486]: Uploading piece... Dec 29 12:39:13.645743045 (573h10m13.51682608s) [noise=true , address=145.79.227.51:4409 , piece_id=CKNQ5QHUM6UKBUDKYO3GXTI2YSVKQ6HD5G3VVA2YWFJH5GYDAKRA, seg_pos={0 0}, upload=502486]: Uploading piece... Dec 29 12:39:13.646205405 (573h10m13.51728844s) [noise=true , address=145.79.227.6:4258 , piece_id=A3H7P5ZSFWO5R2HQZPB3BQK7MN5QJQHBN6L2P5LRWV42T6OZLCJA, seg_pos={0 0}, upload=502486]: Uploading piece... Dec 29 12:39:13.646634112 (573h10m13.517717147s) [noise=true , address=45.91.116.4:4940 , piece_id=FWN5EZMVXNH7636CCKRWSSVSCZQIYUK6YLS7JJXLY4QKZOTCAR3Q, seg_pos={0 0}, upload=502486]: Uploading piece... Dec 29 12:39:13.647025296 (573h10m13.518108341s) [noise=true , address=77.90.31.21:4877 , piece_id=W56E4BPS7NVKJ2UH2HEHVFEI3TVAZZT27FKWWEBCXCHDNGQPDHLQ, seg_pos={0 0}, upload=502486]: Uploading piece... Dec 29 12:39:13.647364513 (573h10m13.518447548s) [noise=true , address=45.91.116.13:4155 , piece_id=CGPCS7FFKTLNQBAAWEKVFUTVPEP47LVPYYSWOWUQMOWWVR2KBDRA, seg_pos={0 0}, upload=502486]: Uploading piece... Dec 29 12:39:13.647730359 (573h10m13.518813394s) [noise=true , address=145.79.227.26:4654 , piece_id=UHPHLT5TGTH7M2IOC6HNSIXN4XBNYQ5TYRXX3P4BM454B367AMGQ, seg_pos={0 0}, upload=502486]: Uploading piece... Dec 29 12:39:13.647882588 (573h10m13.518965623s) [noise=true , address=145.79.227.21:4473 , piece_id=OGWM4YESCYKHYQPSLNG45G53EXAALTPOX3INIRJJPMG5PEYCBVFA, seg_pos={0 0}, upload=502486]: Uploading piece... Dec 29 12:39:13.648511095 (573h10m13.51959413s) [noise=true , address=145.79.227.32:4834 , piece_id=4E3WBRVNVZ5VM4SINUMTSFWX34WKJOJBY7CF5OCXTS6LE4EHSRWA, seg_pos={0 0}, upload=502486]: Uploading piece... Dec 29 12:39:13.648872373 (573h10m13.519955418s) [noise=true , address=145.79.227.4:4187 , piece_id=X2LMTFKJETIIEJVFILFPM6ANDDE4NBEL7QBRB4KNAQG7TT4FFVNA, seg_pos={0 0}, upload=502486]: Uploading piece... Dec 29 12:39:15.338610873 (573h10m15.209693908s) [upload=502486]: Next returned nil segment Dec 29 12:39:15.338624509 (573h10m15.209707544s) [upload=502486]: Waiting for error group managing segments... Dec 29 12:39:15.630861170 (573h10m15.501944205s) [noise=true , address=145.79.227.4:4187 , piece_id=X2LMTFKJETIIEJVFILFPM6ANDDE4NBEL7QBRB4KNAQG7TT4FFVNA, seg_pos={0 0}, upload=502486]: Done uploading piece. err: <nil> Dec 29 12:39:15.632015588 (573h10m15.503098623s) [noise=true , address=145.79.227.21:4473 , piece_id=OGWM4YESCYKHYQPSLNG45G53EXAALTPOX3INIRJJPMG5PEYCBVFA, seg_pos={0 0}, upload=502486]: Done uploading piece. err: <nil> Dec 29 12:39:15.633438318 (573h10m15.504521353s) [noise=true , address=145.79.227.36:4519 , piece_id=6WJHDDBMWJA5JZPRBKQ7HSWYW5AYNS4DCOBDRDLKST22CXRRWIDA, seg_pos={0 0}, upload=502486]: Done uploading piece. err: <nil> Dec 29 12:39:15.633740383 (573h10m15.504823418s) [noise=true , address=145.79.227.51:4409 , piece_id=CKNQ5QHUM6UKBUDKYO3GXTI2YSVKQ6HD5G3VVA2YWFJH5GYDAKRA, seg_pos={0 0}, upload=502486]: Done uploading piece. err: <nil> Dec 29 12:39:15.637589111 (573h10m15.508672146s) [noise=true , address=145.79.227.26:4654 , piece_id=UHPHLT5TGTH7M2IOC6HNSIXN4XBNYQ5TYRXX3P4BM454B367AMGQ, seg_pos={0 0}, upload=502486]: Done uploading piece. err: <nil> Dec 29 12:39:15.645457494 (573h10m15.516540539s) [noise=true , address=145.79.227.6:4258 , piece_id=A3H7P5ZSFWO5R2HQZPB3BQK7MN5QJQHBN6L2P5LRWV42T6OZLCJA, seg_pos={0 0}, upload=502486]: Done uploading piece. err: <nil> Dec 29 12:39:15.650268955 (573h10m15.521351991s) [noise=true , address=145.79.227.32:4834 , piece_id=4E3WBRVNVZ5VM4SINUMTSFWX34WKJOJBY7CF5OCXTS6LE4EHSRWA, seg_pos={0 0}, upload=502486]: Done uploading piece. err: <nil> Dec 29 12:39:15.654572921 (573h10m15.525655956s) [noise=true , address=145.79.227.3:4150 , piece_id=CY5SJKOKFGMA36SFMLLVMRSSAE2IYNB6R6XDQGXJALKUU2AHQK4Q, seg_pos={0 0}, upload=502486]: Done uploading piece. err: <nil> Dec 29 12:39:15.659751952 (573h10m15.530834987s) [noise=true , address=145.79.227.42:4122 , piece_id=APA6XQNVMGXNHTILOJNAQIMVP6C55ZK6DYDLF6O355SQAMUJMVKQ, seg_pos={0 0}, upload=502486]: Done uploading piece. err: <nil> Dec 29 12:39:15.667170017 (573h10m15.538253063s) [noise=true , address=145.79.227.47:4265 , piece_id=RXKFTRDQRUGFFLBH5F3CK3UDRDAAVSLRFHSXGRRIEPFUPUCFHJ4Q, seg_pos={0 0}, upload=502486]: Done uploading piece. err: <nil> Dec 29 12:39:15.905858154 (573h10m15.776941199s) [noise=true , address=77.90.31.16:4735 , piece_id=GVVYVKXSABYKTB2LJ4P3DR6WUGT43TWN64KUX5XAFA6XVFLLO77Q, seg_pos={0 0}, upload=502486]: Done uploading piece. err: <nil> Dec 29 12:39:15.935432518 (573h10m15.806515553s) [noise=true , address=45.91.116.12:4226 , piece_id=UNNLHTGL5GCEKYNGRGB7LDFQDEXNECNDNVKHAZ3VSLVDENFJFRXA, seg_pos={0 0}, upload=502486]: Done uploading piece. err: <nil> Dec 29 12:39:15.969983028 (573h10m15.841066063s) [noise=true , address=77.90.31.11:4555 , piece_id=HKXTBGBIYUHEIVRYZAWZE74TAZ3KJNN5LFZWHRKMG3BXECXR6JCA, seg_pos={0 0}, upload=502486]: Done uploading piece. err: <nil> Dec 29 12:39:15.970001262 (573h10m15.841084307s) [seg_pos={0 0}, upload=502486]: Segment reached optimal threshold of 13 pieces. Dec 29 12:39:15.970202896 (573h10m15.841285961s) [noise=true , address=45.91.116.13:4155 , piece_id=CGPCS7FFKTLNQBAAWEKVFUTVPEP47LVPYYSWOWUQMOWWVR2KBDRA, seg_pos={0 0}, upload=502486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 12:39:15.970214057 (573h10m15.841297092s) [noise=true , address=45.91.116.13:4155 , piece_id=CGPCS7FFKTLNQBAAWEKVFUTVPEP47LVPYYSWOWUQMOWWVR2KBDRA, seg_pos={0 0}, upload=502486]: Long tail cancellation due to optimal threshold Dec 29 12:39:15.970440639 (573h10m15.841523694s) [noise=true , address=77.90.31.21:4877 , piece_id=W56E4BPS7NVKJ2UH2HEHVFEI3TVAZZT27FKWWEBCXCHDNGQPDHLQ, seg_pos={0 0}, upload=502486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 29 12:39:15.970450317 (573h10m15.841533352s) [noise=true , address=77.90.31.21:4877 , piece_id=W56E4BPS7NVKJ2UH2HEHVFEI3TVAZZT27FKWWEBCXCHDNGQPDHLQ, seg_pos={0 0}, upload=502486]: Long tail cancellation due to optimal threshold Dec 29 12:39:15.970454796 (573h10m15.841537831s) [noise=true , address=45.91.116.4:4940 , piece_id=FWN5EZMVXNH7636CCKRWSSVSCZQIYUK6YLS7JJXLY4QKZOTCAR3Q, seg_pos={0 0}, upload=502486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 29 12:39:15.970465727 (573h10m15.841548782s) [noise=true , address=45.91.116.4:4940 , piece_id=FWN5EZMVXNH7636CCKRWSSVSCZQIYUK6YLS7JJXLY4QKZOTCAR3Q, seg_pos={0 0}, upload=502486]: Long tail cancellation due to optimal threshold Dec 29 12:39:15.970509801 (573h10m15.841592846s) [seg_pos={0 0}, upload=502486]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:39:15.970640600 (573h10m15.841723635s) [upload=502486]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:39:15.970648505 (573h10m15.84173154s) [upload=502486]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:39:16.267526194 (573h10m16.138609229s) [upload=502486]: Done uploading segments... Dec 29 12:39:16.267543817 (573h10m16.138626852s) [upload=502486]: Upload finished. err: <nil> Duration: 30.002956658s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [X..XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXXXXXXXXX.XXXXX] duration: 5.001239775s trace: https://jaeger.dev.storj.io/trace/3732de33f4a19986 FAIL (2025-12-29 12:39:08 UTC) orbiter warning: context deadline exceeded Duration: 5.001268901s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XXX..X.X.......X.XX...X.......X.X..XXX....X...X..X....XXXX..] duration: 6.080351511s trace: https://jaeger.dev.storj.io/trace/500a2d0c59c5b5ec FAIL (2025-12-29 12:38:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: MSHLBIQYVLQ4FWFLBKPZTXU7O7VQD6YIPBED5WNSLW5ORD3UZ55A) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: 75M2ZOIGQPDO7EQ24OBOOB2FGTDPTT74BQIERHPSCQU2SNFFWXNQ) context deadline exceeded ======LOGS====== Dec 29 12:36:54.758223142 (573h7m54.629306188s) [upload=502450]: Starting upload Dec 29 12:36:54.758313345 (573h7m54.62939639s) [upload=502450]: Done starting upload Dec 29 12:36:54.758408967 (573h7m54.629492002s) [upload=502450]: Uploading segments... Dec 29 12:36:54.758586685 (573h7m54.62966973s) [upload=502450]: Got next segment. Inline: false Dec 29 12:36:54.758621131 (573h7m54.629704176s) [upload=502450]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:36:54.758626170 (573h7m54.629709215s) [upload=502450]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:36:55.009502065 (573h7m54.8805851s) [seg_pos={0 0}, upload=502450]: Begin upload segment... Dec 29 12:36:55.009547962 (573h7m54.880630997s) [seg_pos={0 0}, upload=502450]: Done begin upload segment. Dec 29 12:36:55.009574803 (573h7m54.880657848s) [noise=true , address=145.79.227.38:4977 , piece_id=INA7GEZGKAEDYLE6LDUKFJR6QM7STG5AVVKBXQFGOTGXD5AQOEUA, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.009702536 (573h7m54.880785592s) [noise=true , address=145.79.227.36:4519 , piece_id=KNHDIQFXPS6QE6NHRFUEP6P53NHG6U3DND34FAO3MH2WRI45ODGQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.010481970 (573h7m54.881565015s) [noise=true , address=145.79.227.32:4836 , piece_id=CVHL5TIGJO6BITRNSJ5ZGZDQZPO3D7X7L7TCWFKOKK4BDURAXUCQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.010949090 (573h7m54.882032125s) [noise=true , address=145.79.227.24:4582 , piece_id=A53AHESJPWXLPZGBS7IQNUQPEAXUPHCGLPUD3H7XHFWH4Y6GB6TQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.011378528 (573h7m54.882461573s) [noise=true , address=145.79.227.18:4006 , piece_id=WP3HU6PJ7W7DAYRHQREEL42JTN6Q6ZR5XHQUDPC2N3HIS64CABTA, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.011806102 (573h7m54.882889137s) [noise=true , address=145.79.227.14:4840 , piece_id=GFLHTJYOYZLSYO7Y4NASF53P6WZJKERKUDIG3IN7NHGL7Y5TIBSQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.012261579 (573h7m54.883344624s) [noise=true , address=145.79.227.6:4257 , piece_id=IY7AK5N5AH26KTLPPWPNCVYIHO2KGQPX5V334EI6AQ4ZLXLVTSBQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.013384478 (573h7m54.884467523s) [noise=true , address=145.79.227.30:4761 , piece_id=LHCAVI2MWXDWEIESJ3U7JOVJJIGYZM7RNS7TMQ2IUM4V67PKX6VA, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.013711470 (573h7m54.884794515s) [noise=true , address=145.79.227.37:4566 , piece_id=WMAXPAYJQ6D57FTNPJ7UQDOCNV4RQAZIZ7XLL4XTOQBFHFX7MP6Q, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.014701555 (573h7m54.88578459s) [noise=true , address=77.90.31.12:4589 , piece_id=MSHLBIQYVLQ4FWFLBKPZTXU7O7VQD6YIPBED5WNSLW5ORD3UZ55A, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.015806499 (573h7m54.886889544s) [noise=true , address=145.79.227.3:4151 , piece_id=A2EENCSAR6PH22OL7MI7LF35MWUVHYITJX7Y4UZP4EMN6HFZXVJA, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.017025931 (573h7m54.888108966s) [noise=true , address=45.91.116.13:4155 , piece_id=75M2ZOIGQPDO7EQ24OBOOB2FGTDPTT74BQIERHPSCQU2SNFFWXNQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.018296120 (573h7m54.889379165s) [noise=true , address=145.79.227.50:4374 , piece_id=ZWI2LXLPGKFOFDROSXIUK2RH42SIGKKXB4I7EE2I52MXZALFVBDQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.018681073 (573h7m54.889764118s) [noise=true , address=45.91.116.9:4264 , piece_id=IVTKOWFT6PIYIHFBAHQRGF6MAAPHTHVIOMK6BBZTCHHV5E72GMUA, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.021355326 (573h7m54.892438371s) [noise=true , address=145.79.227.17:4971 , piece_id=O5FSPZHF4KFZAYLS4GB24PG2JXYRWRA6T4KAXM5ZR7X4MIJI7IUQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.022526836 (573h7m54.893609881s) [noise=true , address=145.79.227.49:4337 , piece_id=ZZ4VXS5PJ2JUTLB35ICCJYQY6JIIUM3GREXPHLW55VUZL2BIKMIQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:56.711504397 (573h7m56.582587442s) [upload=502450]: Next returned nil segment Dec 29 12:36:56.711543431 (573h7m56.582626466s) [upload=502450]: Waiting for error group managing segments... Dec 29 12:36:56.993515769 (573h7m56.864598804s) [noise=true , address=145.79.227.37:4566 , piece_id=WMAXPAYJQ6D57FTNPJ7UQDOCNV4RQAZIZ7XLL4XTOQBFHFX7MP6Q, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:56.998999490 (573h7m56.870082525s) [noise=true , address=145.79.227.14:4840 , piece_id=GFLHTJYOYZLSYO7Y4NASF53P6WZJKERKUDIG3IN7NHGL7Y5TIBSQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.002018459 (573h7m56.873101524s) [noise=true , address=145.79.227.6:4257 , piece_id=IY7AK5N5AH26KTLPPWPNCVYIHO2KGQPX5V334EI6AQ4ZLXLVTSBQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.002541535 (573h7m56.87362459s) [noise=true , address=145.79.227.24:4582 , piece_id=A53AHESJPWXLPZGBS7IQNUQPEAXUPHCGLPUD3H7XHFWH4Y6GB6TQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.004427697 (573h7m56.875510762s) [noise=true , address=145.79.227.3:4151 , piece_id=A2EENCSAR6PH22OL7MI7LF35MWUVHYITJX7Y4UZP4EMN6HFZXVJA, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.006494572 (573h7m56.877577617s) [noise=true , address=145.79.227.32:4836 , piece_id=CVHL5TIGJO6BITRNSJ5ZGZDQZPO3D7X7L7TCWFKOKK4BDURAXUCQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.006585926 (573h7m56.877668971s) [noise=true , address=145.79.227.17:4971 , piece_id=O5FSPZHF4KFZAYLS4GB24PG2JXYRWRA6T4KAXM5ZR7X4MIJI7IUQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.023263683 (573h7m56.894346718s) [noise=true , address=145.79.227.18:4006 , piece_id=WP3HU6PJ7W7DAYRHQREEL42JTN6Q6ZR5XHQUDPC2N3HIS64CABTA, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.023978755 (573h7m56.8950618s) [noise=true , address=145.79.227.50:4374 , piece_id=ZWI2LXLPGKFOFDROSXIUK2RH42SIGKKXB4I7EE2I52MXZALFVBDQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.026445592 (573h7m56.897528627s) [noise=true , address=145.79.227.38:4977 , piece_id=INA7GEZGKAEDYLE6LDUKFJR6QM7STG5AVVKBXQFGOTGXD5AQOEUA, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.039208685 (573h7m56.91029172s) [noise=true , address=145.79.227.36:4519 , piece_id=KNHDIQFXPS6QE6NHRFUEP6P53NHG6U3DND34FAO3MH2WRI45ODGQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.200168957 (573h7m57.071251992s) [noise=true , address=77.90.31.12:4589 , piece_id=MSHLBIQYVLQ4FWFLBKPZTXU7O7VQD6YIPBED5WNSLW5ORD3UZ55A, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.264465446 (573h7m57.135548481s) [noise=true , address=45.91.116.13:4155 , piece_id=75M2ZOIGQPDO7EQ24OBOOB2FGTDPTT74BQIERHPSCQU2SNFFWXNQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.264483962 (573h7m57.135566997s) [seg_pos={0 0}, upload=502450]: Segment reached optimal threshold of 13 pieces. Dec 29 12:36:57.264656921 (573h7m57.135739957s) [noise=true , address=145.79.227.49:4337 , piece_id=ZZ4VXS5PJ2JUTLB35ICCJYQY6JIIUM3GREXPHLW55VUZL2BIKMIQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 29 12:36:57.264663303 (573h7m57.135746339s) [noise=true , address=145.79.227.49:4337 , piece_id=ZZ4VXS5PJ2JUTLB35ICCJYQY6JIIUM3GREXPHLW55VUZL2BIKMIQ, seg_pos={0 0}, upload=502450]: Long tail cancellation due to optimal threshold Dec 29 12:36:57.264799012 (573h7m57.135882067s) [noise=true , address=145.79.227.30:4761 , piece_id=LHCAVI2MWXDWEIESJ3U7JOVJJIGYZM7RNS7TMQ2IUM4V67PKX6VA, seg_pos={0 0}, upload=502450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 29 12:36:57.264807869 (573h7m57.135890914s) [noise=true , address=145.79.227.30:4761 , piece_id=LHCAVI2MWXDWEIESJ3U7JOVJJIGYZM7RNS7TMQ2IUM4V67PKX6VA, seg_pos={0 0}, upload=502450]: Long tail cancellation due to optimal threshold Dec 29 12:36:57.264981509 (573h7m57.136064554s) [noise=true , address=45.91.116.9:4264 , piece_id=IVTKOWFT6PIYIHFBAHQRGF6MAAPHTHVIOMK6BBZTCHHV5E72GMUA, seg_pos={0 0}, upload=502450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: read tcp 188.166.225.170:45708->45.91.116.9:4264: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:36:57.264992690 (573h7m57.136075745s) [noise=true , address=45.91.116.9:4264 , piece_id=IVTKOWFT6PIYIHFBAHQRGF6MAAPHTHVIOMK6BBZTCHHV5E72GMUA, seg_pos={0 0}, upload=502450]: Long tail cancellation due to optimal threshold Dec 29 12:36:57.265053376 (573h7m57.136136421s) [seg_pos={0 0}, upload=502450]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:36:57.265158436 (573h7m57.136241471s) [upload=502450]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:36:57.265163325 (573h7m57.13624636s) [upload=502450]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:36:57.562292653 (573h7m57.433375698s) [upload=502450]: Done uploading segments... Dec 29 12:36:57.562315276 (573h7m57.433398321s) [upload=502450]: Upload finished. err: <nil> Duration: 6.080380446s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................X.......X..........X............] duration: 35.346547626s trace: https://jaeger.dev.storj.io/trace/68764c6cb4286314 FAIL (2025-12-29 12:38:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528204662 (572h59m20.399287707s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528497489 (572h59m20.399580524s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528793153 (572h59m20.399876198s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529083555 (572h59m20.40016659s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529391081 (572h59m20.400474126s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529735096 (572h59m20.400818131s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530109970 (572h59m20.401193015s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530449556 (572h59m20.401532591s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530831554 (572h59m20.401914599s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531229512 (572h59m20.402312557s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531698945 (572h59m20.402781991s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532204238 (572h59m20.403287283s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532744366 (572h59m20.403827401s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533231203 (572h59m20.404314248s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533740113 (572h59m20.404823148s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:21.876539279 (572h59m21.747622324s) [upload=502322]: Next returned nil segment Dec 29 12:28:21.876614272 (572h59m21.747697307s) [upload=502322]: Waiting for error group managing segments... Dec 29 12:28:22.383674429 (572h59m22.254757464s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389083709 (572h59m22.260166754s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389450968 (572h59m22.260534003s) [noise=true , address=145.79.227.46:4625 , piece_id=6NEND2CKGIZLK5Y6IMM4XB6SXPSXUYSZIVZ7CZ25IKAHFFW33C5Q, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.418373151 (572h59m22.289456186s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.619432477 (572h59m22.490515552s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.741876388 (572h59m22.612959423s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.133731187 (572h59m23.004814222s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.146041929 (572h59m23.017124964s) [noise=true , address=193.149.164.143:4697 , piece_id=HZZY3SUOEC7AIHSW6MKZRNZVSOGXSYBYSUKN7STBFTXWAOMDRYBA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.261354601 (572h59m23.132437636s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.291387197 (572h59m23.162470243s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.318370928 (572h59m23.189453973s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.321236023 (572h59m23.192319058s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.408408673 (572h59m23.279491718s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419570137 (572h59m23.290653172s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419592470 (572h59m23.290675515s) [seg_pos={0 0}, upload=502322]: Segment reached optimal threshold of 14 pieces. Dec 29 12:28:23.419837406 (572h59m23.290920441s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123jUFZTVjq6adDuvt3cco7dTvwNTQkszZX7CeajMfbeCUmEmuJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419845241 (572h59m23.290928276s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.419873174 (572h59m23.290956209s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419882021 (572h59m23.290965076s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420073716 (572h59m23.291156751s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection Dec 29 12:28:23.420082262 (572h59m23.291165307s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420257506 (572h59m23.291340551s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: read tcp 188.166.225.170:41270->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:28:23.420264649 (572h59m23.291347684s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420281822 (572h59m23.291364857s) [seg_pos={0 0}, upload=502322]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:28:23.420364349 (572h59m23.291447394s) [upload=502322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:28:23.420370270 (572h59m23.291453315s) [upload=502322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:28:23.725225770 (572h59m23.596308815s) [upload=502322]: Done uploading segments... Dec 29 12:28:23.725255426 (572h59m23.596338481s) [upload=502322]: Upload finished. err: <nil> Duration: 35.346561091s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XX..XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXXXXXXXXX.XXXX] duration: 5.000608065s trace: https://jaeger.dev.storj.io/trace/6d7506f73ad42c99 FAIL (2025-12-29 12:38:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000634868s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXX..XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXXXXXXXXX.XXX] duration: 5.001160123s trace: https://jaeger.dev.storj.io/trace/2ccf3ee2f427d57 FAIL (2025-12-29 12:38:01 UTC) orbiter warning: context deadline exceeded Duration: 5.001187536s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................X.......X..........X...........] duration: 43.877802523s trace: https://jaeger.dev.storj.io/trace/68764c6cb4286314 FAIL (2025-12-29 12:37:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528204662 (572h59m20.399287707s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528497489 (572h59m20.399580524s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528793153 (572h59m20.399876198s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529083555 (572h59m20.40016659s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529391081 (572h59m20.400474126s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529735096 (572h59m20.400818131s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530109970 (572h59m20.401193015s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530449556 (572h59m20.401532591s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530831554 (572h59m20.401914599s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531229512 (572h59m20.402312557s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531698945 (572h59m20.402781991s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532204238 (572h59m20.403287283s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532744366 (572h59m20.403827401s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533231203 (572h59m20.404314248s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533740113 (572h59m20.404823148s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:21.876539279 (572h59m21.747622324s) [upload=502322]: Next returned nil segment Dec 29 12:28:21.876614272 (572h59m21.747697307s) [upload=502322]: Waiting for error group managing segments... Dec 29 12:28:22.383674429 (572h59m22.254757464s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389083709 (572h59m22.260166754s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389450968 (572h59m22.260534003s) [noise=true , address=145.79.227.46:4625 , piece_id=6NEND2CKGIZLK5Y6IMM4XB6SXPSXUYSZIVZ7CZ25IKAHFFW33C5Q, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.418373151 (572h59m22.289456186s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.619432477 (572h59m22.490515552s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.741876388 (572h59m22.612959423s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.133731187 (572h59m23.004814222s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.146041929 (572h59m23.017124964s) [noise=true , address=193.149.164.143:4697 , piece_id=HZZY3SUOEC7AIHSW6MKZRNZVSOGXSYBYSUKN7STBFTXWAOMDRYBA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.261354601 (572h59m23.132437636s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.291387197 (572h59m23.162470243s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.318370928 (572h59m23.189453973s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.321236023 (572h59m23.192319058s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.408408673 (572h59m23.279491718s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419570137 (572h59m23.290653172s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419592470 (572h59m23.290675515s) [seg_pos={0 0}, upload=502322]: Segment reached optimal threshold of 14 pieces. Dec 29 12:28:23.419837406 (572h59m23.290920441s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123jUFZTVjq6adDuvt3cco7dTvwNTQkszZX7CeajMfbeCUmEmuJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419845241 (572h59m23.290928276s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.419873174 (572h59m23.290956209s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419882021 (572h59m23.290965076s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420073716 (572h59m23.291156751s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection Dec 29 12:28:23.420082262 (572h59m23.291165307s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420257506 (572h59m23.291340551s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: read tcp 188.166.225.170:41270->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:28:23.420264649 (572h59m23.291347684s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420281822 (572h59m23.291364857s) [seg_pos={0 0}, upload=502322]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:28:23.420364349 (572h59m23.291447394s) [upload=502322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:28:23.420370270 (572h59m23.291453315s) [upload=502322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:28:23.725225770 (572h59m23.596308815s) [upload=502322]: Done uploading segments... Dec 29 12:28:23.725255426 (572h59m23.596338481s) [upload=502322]: Upload finished. err: <nil> Duration: 43.877817852s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XXXX..X.X.......X.XX...X.......X.X..XXX....X...X..X....XXXX.] duration: 5.756860128s trace: https://jaeger.dev.storj.io/trace/500a2d0c59c5b5ec FAIL (2025-12-29 12:37:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: MSHLBIQYVLQ4FWFLBKPZTXU7O7VQD6YIPBED5WNSLW5ORD3UZ55A) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: 75M2ZOIGQPDO7EQ24OBOOB2FGTDPTT74BQIERHPSCQU2SNFFWXNQ) context deadline exceeded ======LOGS====== Dec 29 12:36:54.758223142 (573h7m54.629306188s) [upload=502450]: Starting upload Dec 29 12:36:54.758313345 (573h7m54.62939639s) [upload=502450]: Done starting upload Dec 29 12:36:54.758408967 (573h7m54.629492002s) [upload=502450]: Uploading segments... Dec 29 12:36:54.758586685 (573h7m54.62966973s) [upload=502450]: Got next segment. Inline: false Dec 29 12:36:54.758621131 (573h7m54.629704176s) [upload=502450]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:36:54.758626170 (573h7m54.629709215s) [upload=502450]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:36:55.009502065 (573h7m54.8805851s) [seg_pos={0 0}, upload=502450]: Begin upload segment... Dec 29 12:36:55.009547962 (573h7m54.880630997s) [seg_pos={0 0}, upload=502450]: Done begin upload segment. Dec 29 12:36:55.009574803 (573h7m54.880657848s) [noise=true , address=145.79.227.38:4977 , piece_id=INA7GEZGKAEDYLE6LDUKFJR6QM7STG5AVVKBXQFGOTGXD5AQOEUA, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.009702536 (573h7m54.880785592s) [noise=true , address=145.79.227.36:4519 , piece_id=KNHDIQFXPS6QE6NHRFUEP6P53NHG6U3DND34FAO3MH2WRI45ODGQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.010481970 (573h7m54.881565015s) [noise=true , address=145.79.227.32:4836 , piece_id=CVHL5TIGJO6BITRNSJ5ZGZDQZPO3D7X7L7TCWFKOKK4BDURAXUCQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.010949090 (573h7m54.882032125s) [noise=true , address=145.79.227.24:4582 , piece_id=A53AHESJPWXLPZGBS7IQNUQPEAXUPHCGLPUD3H7XHFWH4Y6GB6TQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.011378528 (573h7m54.882461573s) [noise=true , address=145.79.227.18:4006 , piece_id=WP3HU6PJ7W7DAYRHQREEL42JTN6Q6ZR5XHQUDPC2N3HIS64CABTA, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.011806102 (573h7m54.882889137s) [noise=true , address=145.79.227.14:4840 , piece_id=GFLHTJYOYZLSYO7Y4NASF53P6WZJKERKUDIG3IN7NHGL7Y5TIBSQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.012261579 (573h7m54.883344624s) [noise=true , address=145.79.227.6:4257 , piece_id=IY7AK5N5AH26KTLPPWPNCVYIHO2KGQPX5V334EI6AQ4ZLXLVTSBQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.013384478 (573h7m54.884467523s) [noise=true , address=145.79.227.30:4761 , piece_id=LHCAVI2MWXDWEIESJ3U7JOVJJIGYZM7RNS7TMQ2IUM4V67PKX6VA, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.013711470 (573h7m54.884794515s) [noise=true , address=145.79.227.37:4566 , piece_id=WMAXPAYJQ6D57FTNPJ7UQDOCNV4RQAZIZ7XLL4XTOQBFHFX7MP6Q, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.014701555 (573h7m54.88578459s) [noise=true , address=77.90.31.12:4589 , piece_id=MSHLBIQYVLQ4FWFLBKPZTXU7O7VQD6YIPBED5WNSLW5ORD3UZ55A, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.015806499 (573h7m54.886889544s) [noise=true , address=145.79.227.3:4151 , piece_id=A2EENCSAR6PH22OL7MI7LF35MWUVHYITJX7Y4UZP4EMN6HFZXVJA, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.017025931 (573h7m54.888108966s) [noise=true , address=45.91.116.13:4155 , piece_id=75M2ZOIGQPDO7EQ24OBOOB2FGTDPTT74BQIERHPSCQU2SNFFWXNQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.018296120 (573h7m54.889379165s) [noise=true , address=145.79.227.50:4374 , piece_id=ZWI2LXLPGKFOFDROSXIUK2RH42SIGKKXB4I7EE2I52MXZALFVBDQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.018681073 (573h7m54.889764118s) [noise=true , address=45.91.116.9:4264 , piece_id=IVTKOWFT6PIYIHFBAHQRGF6MAAPHTHVIOMK6BBZTCHHV5E72GMUA, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.021355326 (573h7m54.892438371s) [noise=true , address=145.79.227.17:4971 , piece_id=O5FSPZHF4KFZAYLS4GB24PG2JXYRWRA6T4KAXM5ZR7X4MIJI7IUQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.022526836 (573h7m54.893609881s) [noise=true , address=145.79.227.49:4337 , piece_id=ZZ4VXS5PJ2JUTLB35ICCJYQY6JIIUM3GREXPHLW55VUZL2BIKMIQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:56.711504397 (573h7m56.582587442s) [upload=502450]: Next returned nil segment Dec 29 12:36:56.711543431 (573h7m56.582626466s) [upload=502450]: Waiting for error group managing segments... Dec 29 12:36:56.993515769 (573h7m56.864598804s) [noise=true , address=145.79.227.37:4566 , piece_id=WMAXPAYJQ6D57FTNPJ7UQDOCNV4RQAZIZ7XLL4XTOQBFHFX7MP6Q, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:56.998999490 (573h7m56.870082525s) [noise=true , address=145.79.227.14:4840 , piece_id=GFLHTJYOYZLSYO7Y4NASF53P6WZJKERKUDIG3IN7NHGL7Y5TIBSQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.002018459 (573h7m56.873101524s) [noise=true , address=145.79.227.6:4257 , piece_id=IY7AK5N5AH26KTLPPWPNCVYIHO2KGQPX5V334EI6AQ4ZLXLVTSBQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.002541535 (573h7m56.87362459s) [noise=true , address=145.79.227.24:4582 , piece_id=A53AHESJPWXLPZGBS7IQNUQPEAXUPHCGLPUD3H7XHFWH4Y6GB6TQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.004427697 (573h7m56.875510762s) [noise=true , address=145.79.227.3:4151 , piece_id=A2EENCSAR6PH22OL7MI7LF35MWUVHYITJX7Y4UZP4EMN6HFZXVJA, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.006494572 (573h7m56.877577617s) [noise=true , address=145.79.227.32:4836 , piece_id=CVHL5TIGJO6BITRNSJ5ZGZDQZPO3D7X7L7TCWFKOKK4BDURAXUCQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.006585926 (573h7m56.877668971s) [noise=true , address=145.79.227.17:4971 , piece_id=O5FSPZHF4KFZAYLS4GB24PG2JXYRWRA6T4KAXM5ZR7X4MIJI7IUQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.023263683 (573h7m56.894346718s) [noise=true , address=145.79.227.18:4006 , piece_id=WP3HU6PJ7W7DAYRHQREEL42JTN6Q6ZR5XHQUDPC2N3HIS64CABTA, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.023978755 (573h7m56.8950618s) [noise=true , address=145.79.227.50:4374 , piece_id=ZWI2LXLPGKFOFDROSXIUK2RH42SIGKKXB4I7EE2I52MXZALFVBDQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.026445592 (573h7m56.897528627s) [noise=true , address=145.79.227.38:4977 , piece_id=INA7GEZGKAEDYLE6LDUKFJR6QM7STG5AVVKBXQFGOTGXD5AQOEUA, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.039208685 (573h7m56.91029172s) [noise=true , address=145.79.227.36:4519 , piece_id=KNHDIQFXPS6QE6NHRFUEP6P53NHG6U3DND34FAO3MH2WRI45ODGQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.200168957 (573h7m57.071251992s) [noise=true , address=77.90.31.12:4589 , piece_id=MSHLBIQYVLQ4FWFLBKPZTXU7O7VQD6YIPBED5WNSLW5ORD3UZ55A, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.264465446 (573h7m57.135548481s) [noise=true , address=45.91.116.13:4155 , piece_id=75M2ZOIGQPDO7EQ24OBOOB2FGTDPTT74BQIERHPSCQU2SNFFWXNQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.264483962 (573h7m57.135566997s) [seg_pos={0 0}, upload=502450]: Segment reached optimal threshold of 13 pieces. Dec 29 12:36:57.264656921 (573h7m57.135739957s) [noise=true , address=145.79.227.49:4337 , piece_id=ZZ4VXS5PJ2JUTLB35ICCJYQY6JIIUM3GREXPHLW55VUZL2BIKMIQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 29 12:36:57.264663303 (573h7m57.135746339s) [noise=true , address=145.79.227.49:4337 , piece_id=ZZ4VXS5PJ2JUTLB35ICCJYQY6JIIUM3GREXPHLW55VUZL2BIKMIQ, seg_pos={0 0}, upload=502450]: Long tail cancellation due to optimal threshold Dec 29 12:36:57.264799012 (573h7m57.135882067s) [noise=true , address=145.79.227.30:4761 , piece_id=LHCAVI2MWXDWEIESJ3U7JOVJJIGYZM7RNS7TMQ2IUM4V67PKX6VA, seg_pos={0 0}, upload=502450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 29 12:36:57.264807869 (573h7m57.135890914s) [noise=true , address=145.79.227.30:4761 , piece_id=LHCAVI2MWXDWEIESJ3U7JOVJJIGYZM7RNS7TMQ2IUM4V67PKX6VA, seg_pos={0 0}, upload=502450]: Long tail cancellation due to optimal threshold Dec 29 12:36:57.264981509 (573h7m57.136064554s) [noise=true , address=45.91.116.9:4264 , piece_id=IVTKOWFT6PIYIHFBAHQRGF6MAAPHTHVIOMK6BBZTCHHV5E72GMUA, seg_pos={0 0}, upload=502450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: read tcp 188.166.225.170:45708->45.91.116.9:4264: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:36:57.264992690 (573h7m57.136075745s) [noise=true , address=45.91.116.9:4264 , piece_id=IVTKOWFT6PIYIHFBAHQRGF6MAAPHTHVIOMK6BBZTCHHV5E72GMUA, seg_pos={0 0}, upload=502450]: Long tail cancellation due to optimal threshold Dec 29 12:36:57.265053376 (573h7m57.136136421s) [seg_pos={0 0}, upload=502450]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:36:57.265158436 (573h7m57.136241471s) [upload=502450]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:36:57.265163325 (573h7m57.13624636s) [upload=502450]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:36:57.562292653 (573h7m57.433375698s) [upload=502450]: Done uploading segments... Dec 29 12:36:57.562315276 (573h7m57.433398321s) [upload=502450]: Upload finished. err: <nil> Duration: 5.756885976s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XXXXX..X.X.......X.XX...X.......X.X..XXX....X...X..X....XXXX] duration: 30.002738193s trace: https://jaeger.dev.storj.io/trace/500a2d0c59c5b5ec FAIL (2025-12-29 12:37:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: MSHLBIQYVLQ4FWFLBKPZTXU7O7VQD6YIPBED5WNSLW5ORD3UZ55A) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: 75M2ZOIGQPDO7EQ24OBOOB2FGTDPTT74BQIERHPSCQU2SNFFWXNQ) context deadline exceeded ======LOGS====== Dec 29 12:36:54.758223142 (573h7m54.629306188s) [upload=502450]: Starting upload Dec 29 12:36:54.758313345 (573h7m54.62939639s) [upload=502450]: Done starting upload Dec 29 12:36:54.758408967 (573h7m54.629492002s) [upload=502450]: Uploading segments... Dec 29 12:36:54.758586685 (573h7m54.62966973s) [upload=502450]: Got next segment. Inline: false Dec 29 12:36:54.758621131 (573h7m54.629704176s) [upload=502450]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:36:54.758626170 (573h7m54.629709215s) [upload=502450]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:36:55.009502065 (573h7m54.8805851s) [seg_pos={0 0}, upload=502450]: Begin upload segment... Dec 29 12:36:55.009547962 (573h7m54.880630997s) [seg_pos={0 0}, upload=502450]: Done begin upload segment. Dec 29 12:36:55.009574803 (573h7m54.880657848s) [noise=true , address=145.79.227.38:4977 , piece_id=INA7GEZGKAEDYLE6LDUKFJR6QM7STG5AVVKBXQFGOTGXD5AQOEUA, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.009702536 (573h7m54.880785592s) [noise=true , address=145.79.227.36:4519 , piece_id=KNHDIQFXPS6QE6NHRFUEP6P53NHG6U3DND34FAO3MH2WRI45ODGQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.010481970 (573h7m54.881565015s) [noise=true , address=145.79.227.32:4836 , piece_id=CVHL5TIGJO6BITRNSJ5ZGZDQZPO3D7X7L7TCWFKOKK4BDURAXUCQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.010949090 (573h7m54.882032125s) [noise=true , address=145.79.227.24:4582 , piece_id=A53AHESJPWXLPZGBS7IQNUQPEAXUPHCGLPUD3H7XHFWH4Y6GB6TQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.011378528 (573h7m54.882461573s) [noise=true , address=145.79.227.18:4006 , piece_id=WP3HU6PJ7W7DAYRHQREEL42JTN6Q6ZR5XHQUDPC2N3HIS64CABTA, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.011806102 (573h7m54.882889137s) [noise=true , address=145.79.227.14:4840 , piece_id=GFLHTJYOYZLSYO7Y4NASF53P6WZJKERKUDIG3IN7NHGL7Y5TIBSQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.012261579 (573h7m54.883344624s) [noise=true , address=145.79.227.6:4257 , piece_id=IY7AK5N5AH26KTLPPWPNCVYIHO2KGQPX5V334EI6AQ4ZLXLVTSBQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.013384478 (573h7m54.884467523s) [noise=true , address=145.79.227.30:4761 , piece_id=LHCAVI2MWXDWEIESJ3U7JOVJJIGYZM7RNS7TMQ2IUM4V67PKX6VA, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.013711470 (573h7m54.884794515s) [noise=true , address=145.79.227.37:4566 , piece_id=WMAXPAYJQ6D57FTNPJ7UQDOCNV4RQAZIZ7XLL4XTOQBFHFX7MP6Q, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.014701555 (573h7m54.88578459s) [noise=true , address=77.90.31.12:4589 , piece_id=MSHLBIQYVLQ4FWFLBKPZTXU7O7VQD6YIPBED5WNSLW5ORD3UZ55A, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.015806499 (573h7m54.886889544s) [noise=true , address=145.79.227.3:4151 , piece_id=A2EENCSAR6PH22OL7MI7LF35MWUVHYITJX7Y4UZP4EMN6HFZXVJA, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.017025931 (573h7m54.888108966s) [noise=true , address=45.91.116.13:4155 , piece_id=75M2ZOIGQPDO7EQ24OBOOB2FGTDPTT74BQIERHPSCQU2SNFFWXNQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.018296120 (573h7m54.889379165s) [noise=true , address=145.79.227.50:4374 , piece_id=ZWI2LXLPGKFOFDROSXIUK2RH42SIGKKXB4I7EE2I52MXZALFVBDQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.018681073 (573h7m54.889764118s) [noise=true , address=45.91.116.9:4264 , piece_id=IVTKOWFT6PIYIHFBAHQRGF6MAAPHTHVIOMK6BBZTCHHV5E72GMUA, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.021355326 (573h7m54.892438371s) [noise=true , address=145.79.227.17:4971 , piece_id=O5FSPZHF4KFZAYLS4GB24PG2JXYRWRA6T4KAXM5ZR7X4MIJI7IUQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.022526836 (573h7m54.893609881s) [noise=true , address=145.79.227.49:4337 , piece_id=ZZ4VXS5PJ2JUTLB35ICCJYQY6JIIUM3GREXPHLW55VUZL2BIKMIQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:56.711504397 (573h7m56.582587442s) [upload=502450]: Next returned nil segment Dec 29 12:36:56.711543431 (573h7m56.582626466s) [upload=502450]: Waiting for error group managing segments... Dec 29 12:36:56.993515769 (573h7m56.864598804s) [noise=true , address=145.79.227.37:4566 , piece_id=WMAXPAYJQ6D57FTNPJ7UQDOCNV4RQAZIZ7XLL4XTOQBFHFX7MP6Q, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:56.998999490 (573h7m56.870082525s) [noise=true , address=145.79.227.14:4840 , piece_id=GFLHTJYOYZLSYO7Y4NASF53P6WZJKERKUDIG3IN7NHGL7Y5TIBSQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.002018459 (573h7m56.873101524s) [noise=true , address=145.79.227.6:4257 , piece_id=IY7AK5N5AH26KTLPPWPNCVYIHO2KGQPX5V334EI6AQ4ZLXLVTSBQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.002541535 (573h7m56.87362459s) [noise=true , address=145.79.227.24:4582 , piece_id=A53AHESJPWXLPZGBS7IQNUQPEAXUPHCGLPUD3H7XHFWH4Y6GB6TQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.004427697 (573h7m56.875510762s) [noise=true , address=145.79.227.3:4151 , piece_id=A2EENCSAR6PH22OL7MI7LF35MWUVHYITJX7Y4UZP4EMN6HFZXVJA, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.006494572 (573h7m56.877577617s) [noise=true , address=145.79.227.32:4836 , piece_id=CVHL5TIGJO6BITRNSJ5ZGZDQZPO3D7X7L7TCWFKOKK4BDURAXUCQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.006585926 (573h7m56.877668971s) [noise=true , address=145.79.227.17:4971 , piece_id=O5FSPZHF4KFZAYLS4GB24PG2JXYRWRA6T4KAXM5ZR7X4MIJI7IUQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.023263683 (573h7m56.894346718s) [noise=true , address=145.79.227.18:4006 , piece_id=WP3HU6PJ7W7DAYRHQREEL42JTN6Q6ZR5XHQUDPC2N3HIS64CABTA, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.023978755 (573h7m56.8950618s) [noise=true , address=145.79.227.50:4374 , piece_id=ZWI2LXLPGKFOFDROSXIUK2RH42SIGKKXB4I7EE2I52MXZALFVBDQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.026445592 (573h7m56.897528627s) [noise=true , address=145.79.227.38:4977 , piece_id=INA7GEZGKAEDYLE6LDUKFJR6QM7STG5AVVKBXQFGOTGXD5AQOEUA, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.039208685 (573h7m56.91029172s) [noise=true , address=145.79.227.36:4519 , piece_id=KNHDIQFXPS6QE6NHRFUEP6P53NHG6U3DND34FAO3MH2WRI45ODGQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.200168957 (573h7m57.071251992s) [noise=true , address=77.90.31.12:4589 , piece_id=MSHLBIQYVLQ4FWFLBKPZTXU7O7VQD6YIPBED5WNSLW5ORD3UZ55A, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.264465446 (573h7m57.135548481s) [noise=true , address=45.91.116.13:4155 , piece_id=75M2ZOIGQPDO7EQ24OBOOB2FGTDPTT74BQIERHPSCQU2SNFFWXNQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.264483962 (573h7m57.135566997s) [seg_pos={0 0}, upload=502450]: Segment reached optimal threshold of 13 pieces. Dec 29 12:36:57.264656921 (573h7m57.135739957s) [noise=true , address=145.79.227.49:4337 , piece_id=ZZ4VXS5PJ2JUTLB35ICCJYQY6JIIUM3GREXPHLW55VUZL2BIKMIQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 29 12:36:57.264663303 (573h7m57.135746339s) [noise=true , address=145.79.227.49:4337 , piece_id=ZZ4VXS5PJ2JUTLB35ICCJYQY6JIIUM3GREXPHLW55VUZL2BIKMIQ, seg_pos={0 0}, upload=502450]: Long tail cancellation due to optimal threshold Dec 29 12:36:57.264799012 (573h7m57.135882067s) [noise=true , address=145.79.227.30:4761 , piece_id=LHCAVI2MWXDWEIESJ3U7JOVJJIGYZM7RNS7TMQ2IUM4V67PKX6VA, seg_pos={0 0}, upload=502450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 29 12:36:57.264807869 (573h7m57.135890914s) [noise=true , address=145.79.227.30:4761 , piece_id=LHCAVI2MWXDWEIESJ3U7JOVJJIGYZM7RNS7TMQ2IUM4V67PKX6VA, seg_pos={0 0}, upload=502450]: Long tail cancellation due to optimal threshold Dec 29 12:36:57.264981509 (573h7m57.136064554s) [noise=true , address=45.91.116.9:4264 , piece_id=IVTKOWFT6PIYIHFBAHQRGF6MAAPHTHVIOMK6BBZTCHHV5E72GMUA, seg_pos={0 0}, upload=502450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: read tcp 188.166.225.170:45708->45.91.116.9:4264: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:36:57.264992690 (573h7m57.136075745s) [noise=true , address=45.91.116.9:4264 , piece_id=IVTKOWFT6PIYIHFBAHQRGF6MAAPHTHVIOMK6BBZTCHHV5E72GMUA, seg_pos={0 0}, upload=502450]: Long tail cancellation due to optimal threshold Dec 29 12:36:57.265053376 (573h7m57.136136421s) [seg_pos={0 0}, upload=502450]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:36:57.265158436 (573h7m57.136241471s) [upload=502450]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:36:57.265163325 (573h7m57.13624636s) [upload=502450]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:36:57.562292653 (573h7m57.433375698s) [upload=502450]: Done uploading segments... Dec 29 12:36:57.562315276 (573h7m57.433398321s) [upload=502450]: Upload finished. err: <nil> Duration: 30.002760225s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [.XXX..XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXXXXXXXXX.XX] duration: 5.001493278s trace: https://jaeger.dev.storj.io/trace/4c872b9f771e22c8 FAIL (2025-12-29 12:37:19 UTC) orbiter warning: context deadline exceeded Duration: 5.001518305s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................X.......X..........X..........] duration: 50.342683992s trace: https://jaeger.dev.storj.io/trace/68764c6cb4286314 FAIL (2025-12-29 12:37:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528204662 (572h59m20.399287707s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528497489 (572h59m20.399580524s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528793153 (572h59m20.399876198s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529083555 (572h59m20.40016659s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529391081 (572h59m20.400474126s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529735096 (572h59m20.400818131s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530109970 (572h59m20.401193015s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530449556 (572h59m20.401532591s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530831554 (572h59m20.401914599s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531229512 (572h59m20.402312557s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531698945 (572h59m20.402781991s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532204238 (572h59m20.403287283s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532744366 (572h59m20.403827401s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533231203 (572h59m20.404314248s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533740113 (572h59m20.404823148s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:21.876539279 (572h59m21.747622324s) [upload=502322]: Next returned nil segment Dec 29 12:28:21.876614272 (572h59m21.747697307s) [upload=502322]: Waiting for error group managing segments... Dec 29 12:28:22.383674429 (572h59m22.254757464s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389083709 (572h59m22.260166754s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389450968 (572h59m22.260534003s) [noise=true , address=145.79.227.46:4625 , piece_id=6NEND2CKGIZLK5Y6IMM4XB6SXPSXUYSZIVZ7CZ25IKAHFFW33C5Q, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.418373151 (572h59m22.289456186s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.619432477 (572h59m22.490515552s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.741876388 (572h59m22.612959423s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.133731187 (572h59m23.004814222s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.146041929 (572h59m23.017124964s) [noise=true , address=193.149.164.143:4697 , piece_id=HZZY3SUOEC7AIHSW6MKZRNZVSOGXSYBYSUKN7STBFTXWAOMDRYBA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.261354601 (572h59m23.132437636s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.291387197 (572h59m23.162470243s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.318370928 (572h59m23.189453973s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.321236023 (572h59m23.192319058s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.408408673 (572h59m23.279491718s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419570137 (572h59m23.290653172s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419592470 (572h59m23.290675515s) [seg_pos={0 0}, upload=502322]: Segment reached optimal threshold of 14 pieces. Dec 29 12:28:23.419837406 (572h59m23.290920441s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123jUFZTVjq6adDuvt3cco7dTvwNTQkszZX7CeajMfbeCUmEmuJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419845241 (572h59m23.290928276s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.419873174 (572h59m23.290956209s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419882021 (572h59m23.290965076s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420073716 (572h59m23.291156751s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection Dec 29 12:28:23.420082262 (572h59m23.291165307s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420257506 (572h59m23.291340551s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: read tcp 188.166.225.170:41270->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:28:23.420264649 (572h59m23.291347684s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420281822 (572h59m23.291364857s) [seg_pos={0 0}, upload=502322]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:28:23.420364349 (572h59m23.291447394s) [upload=502322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:28:23.420370270 (572h59m23.291453315s) [upload=502322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:28:23.725225770 (572h59m23.596308815s) [upload=502322]: Done uploading segments... Dec 29 12:28:23.725255426 (572h59m23.596338481s) [upload=502322]: Upload finished. err: <nil> Duration: 50.342711144s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 39 out of 60 is failed (threshold=6) [..XXX..XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXXXXXXXXX.X] duration: 5.001100594s trace: https://jaeger.dev.storj.io/trace/1bd9f56a9e88d2ee FAIL (2025-12-29 12:36:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001113318s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XXXXXX..X.X.......X.XX...X.......X.X..XXX....X...X..X....XXX] duration: 30.003065887s trace: https://jaeger.dev.storj.io/trace/3b55af95361e57c1 FAIL (2025-12-29 12:36:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =502439]: Starting upload Dec 29 12:35:56.314641001 (573h6m56.185724036s) [upload=502439]: Done starting upload Dec 29 12:35:56.314728368 (573h6m56.185811413s) [upload=502439]: Uploading segments... Dec 29 12:35:56.314888352 (573h6m56.185971397s) [upload=502439]: Got next segment. Inline: false Dec 29 12:35:56.314932406 (573h6m56.186015441s) [upload=502439]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:35:56.314938959 (573h6m56.186021994s) [upload=502439]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:35:56.567188299 (573h6m56.438271334s) [seg_pos={0 0}, upload=502439]: Begin upload segment... Dec 29 12:35:56.567242431 (573h6m56.438325466s) [seg_pos={0 0}, upload=502439]: Done begin upload segment. Dec 29 12:35:56.567276065 (573h6m56.43835911s) [noise=true , address=145.79.227.36:4518 , piece_id=755BMSS3R3L65EKO3VGQRYJHLEFYCGXT27YVJCAHUJRIEPEPPJVA, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.567302065 (573h6m56.4383851s) [noise=true , address=145.79.227.3:4150 , piece_id=C2SKGSU7DBMKEGNW6SQ7ESNITDHVTNGHE4JVDQJRVDPDE5O5VUDQ, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.567347221 (573h6m56.438430256s) [noise=true , address=145.79.227.5:4222 , piece_id=SEYX67D3UBXQCBDT3XOHRSWGNPBU63GT3ZNSC7XUKXA7NVKVBGXA, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.567788782 (573h6m56.438871817s) [noise=true , address=145.79.227.17:4971 , piece_id=3JD35EOECXSTO3BMZ3XTCICMBIRVFZUTPQLKJMUVOCYU72IPI75A, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.568135863 (573h6m56.439218908s) [noise=true , address=145.79.227.45:4231 , piece_id=GCOXLBWZNZPQWLOTDQ4U3XWPZR2XMKBIOWZT2VYBG2QLVTRD6SFQ, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.568463307 (573h6m56.439546342s) [noise=true , address=145.79.227.52:4446 , piece_id=UHTHU6UCDVXFEMUO57CPRBN45FZ4VSCZMRF4VUHK6Q54ZFWYVGJQ, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.568793405 (573h6m56.43987644s) [noise=true , address=45.91.116.12:4225 , piece_id=4U5XB27V6GOJV2G2QZ24ZBPE5HLVAD7PZCLZCA3VCI74F6G4XEXQ, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.571479169 (573h6m56.442562214s) [noise=true , address=145.79.227.15:4873 , piece_id=HXJ7MUK5NXVDGK475QLSS2SWBBBYDU2E52TIJR5OPLPHZI73IC7A, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.572622166 (573h6m56.443705201s) [noise=true , address=77.90.31.18:4805 , piece_id=ETDGD5Z4FJVTIP5AXTB4R6ATEQ5YKU6HE4B4YRLAKWNN2CGDMAUA, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.573898686 (573h6m56.444981721s) [noise=true , address=145.79.227.4:4187 , piece_id=O7DY3KACKXB6Z3ZZFMN2W3JWL7R7XUIMZDF3O5TGMBDHA6ASP3HA, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.574956531 (573h6m56.446039566s) [noise=true , address=145.79.227.48:4302 , piece_id=FMNLGFHK77LWP3XIPQOGY4K43UFDCFO3YJQLLENXN7DR4UCWUE4A, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.575308882 (573h6m56.446391917s) [noise=true , address=45.91.116.14:4191 , piece_id=TOACUKBN4S2P5BBEUYA5RQSOLFBLZW2OUGHXTHE54X6BYRQQ2CEQ, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.576339433 (573h6m56.447422468s) [noise=true , address=145.79.227.19:4078 , piece_id=7RMN7V72KNQJ5XQOY6A55FKVBLYRESRMRQDDQQ2CW4W57BPW7EYA, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.576879522 (573h6m56.447962557s) [noise=true , address=45.91.116.4:4937 , piece_id=G5EVO4HSZMQVIYA6JB7MSFL6QN6XCCLP55VTTVZG26OLVPS2KZLQ, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.577692750 (573h6m56.448775795s) [noise=true , address=145.79.227.42:4122 , piece_id=TYWKGHSQS767HBDZLFMLBH6T7WK2VZYJJOL43NWOAYG7MKPASSUQ, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.578460442 (573h6m56.449543477s) [noise=true , address=45.91.116.13:4154 , piece_id=QYH7DSEKKM7F7P6PJ3EUZUIH5BBFTBPW7MZ75ZNVOHWADFNCKURQ, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:58.273077944 (573h6m58.144160979s) [upload=502439]: Next returned nil segment Dec 29 12:35:58.273107380 (573h6m58.144190425s) [upload=502439]: Waiting for error group managing segments... Dec 29 12:35:58.552982213 (573h6m58.424065248s) [noise=true , address=145.79.227.17:4971 , piece_id=3JD35EOECXSTO3BMZ3XTCICMBIRVFZUTPQLKJMUVOCYU72IPI75A, seg_pos={0 0}, upload=502439]: Done uploading piece. err: <nil> Dec 29 12:35:58.553037217 (573h6m58.424120252s) [noise=true , address=145.79.227.5:4222 , piece_id=SEYX67D3UBXQCBDT3XOHRSWGNPBU63GT3ZNSC7XUKXA7NVKVBGXA, seg_pos={0 0}, upload=502439]: Done uploading piece. err: <nil> Dec 29 12:35:58.558023892 (573h6m58.429106927s) [noise=true , address=145.79.227.4:4187 , piece_id=O7DY3KACKXB6Z3ZZFMN2W3JWL7R7XUIMZDF3O5TGMBDHA6ASP3HA, seg_pos={0 0}, upload=502439]: Done uploading piece. err: <nil> Dec 29 12:35:58.558358510 (573h6m58.429441545s) [noise=true , address=145.79.227.36:4518 , piece_id=755BMSS3R3L65EKO3VGQRYJHLEFYCGXT27YVJCAHUJRIEPEPPJVA, seg_pos={0 0}, upload=502439]: Done uploading piece. err: <nil> Dec 29 12:35:58.561949057 (573h6m58.433032092s) [noise=true , address=145.79.227.45:4231 , piece_id=GCOXLBWZNZPQWLOTDQ4U3XWPZR2XMKBIOWZT2VYBG2QLVTRD6SFQ, seg_pos={0 0}, upload=502439]: Done uploading piece. err: <nil> Dec 29 12:35:58.562142565 (573h6m58.4332256s) [noise=true , address=145.79.227.19:4078 , piece_id=7RMN7V72KNQJ5XQOY6A55FKVBLYRESRMRQDDQQ2CW4W57BPW7EYA, seg_pos={0 0}, upload=502439]: Done uploading piece. err: <nil> Dec 29 12:35:58.562620154 (573h6m58.433703189s) [noise=true , address=145.79.227.42:4122 , piece_id=TYWKGHSQS767HBDZLFMLBH6T7WK2VZYJJOL43NWOAYG7MKPASSUQ, seg_pos={0 0}, upload=502439]: Done uploading piece. err: <nil> Dec 29 12:35:58.573164152 (573h6m58.444247197s) [noise=true , address=145.79.227.3:4150 , piece_id=C2SKGSU7DBMKEGNW6SQ7ESNITDHVTNGHE4JVDQJRVDPDE5O5VUDQ, seg_pos={0 0}, upload=502439]: Done uploading piece. err: <nil> Dec 29 12:35:58.574742457 (573h6m58.445825502s) [noise=true , address=145.79.227.52:4446 , piece_id=UHTHU6UCDVXFEMUO57CPRBN45FZ4VSCZMRF4VUHK6Q54ZFWYVGJQ, seg_pos={0 0}, upload=502439]: Done uploading piece. err: <nil> Dec 29 12:35:58.578854627 (573h6m58.449937672s) [noise=true , address=145.79.227.15:4873 , piece_id=HXJ7MUK5NXVDGK475QLSS2SWBBBYDU2E52TIJR5OPLPHZI73IC7A, seg_pos={0 0}, upload=502439]: Done uploading piece. err: <nil> Dec 29 12:35:58.580262918 (573h6m58.451345953s) [noise=true , address=145.79.227.48:4302 , piece_id=FMNLGFHK77LWP3XIPQOGY4K43UFDCFO3YJQLLENXN7DR4UCWUE4A, seg_pos={0 0}, upload=502439]: Done uploading piece. err: <nil> Dec 29 12:35:58.876595519 (573h6m58.747678554s) [noise=true , address=77.90.31.18:4805 , piece_id=ETDGD5Z4FJVTIP5AXTB4R6ATEQ5YKU6HE4B4YRLAKWNN2CGDMAUA, seg_pos={0 0}, upload=502439]: Done uploading piece. err: <nil> Dec 29 12:35:59.876081688 (573h6m59.747164723s) [noise=true , address=45.91.116.14:4191 , piece_id=TOACUKBN4S2P5BBEUYA5RQSOLFBLZW2OUGHXTHE54X6BYRQQ2CEQ, seg_pos={0 0}, upload=502439]: Done uploading piece. err: <nil> Dec 29 12:35:59.876099331 (573h6m59.747182377s) [seg_pos={0 0}, upload=502439]: Segment reached optimal threshold of 13 pieces. Dec 29 12:35:59.876274745 (573h6m59.747357811s) [noise=true , address=45.91.116.12:4225 , piece_id=4U5XB27V6GOJV2G2QZ24ZBPE5HLVAD7PZCLZCA3VCI74F6G4XEXQ, seg_pos={0 0}, upload=502439]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:34228->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:34228->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34228->45.91.116.12:4225: use of closed network connection Dec 29 12:35:59.876285737 (573h6m59.747368782s) [noise=true , address=45.91.116.12:4225 , piece_id=4U5XB27V6GOJV2G2QZ24ZBPE5HLVAD7PZCLZCA3VCI74F6G4XEXQ, seg_pos={0 0}, upload=502439]: Long tail cancellation due to optimal threshold Dec 29 12:35:59.876428148 (573h6m59.747511183s) [noise=true , address=45.91.116.13:4154 , piece_id=QYH7DSEKKM7F7P6PJ3EUZUIH5BBFTBPW7MZ75ZNVOHWADFNCKURQ, seg_pos={0 0}, upload=502439]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 29 12:35:59.876433899 (573h6m59.747516934s) [noise=true , address=45.91.116.13:4154 , piece_id=QYH7DSEKKM7F7P6PJ3EUZUIH5BBFTBPW7MZ75ZNVOHWADFNCKURQ, seg_pos={0 0}, upload=502439]: Long tail cancellation due to optimal threshold Dec 29 12:35:59.876543237 (573h6m59.747626272s) [noise=true , address=45.91.116.4:4937 , piece_id=G5EVO4HSZMQVIYA6JB7MSFL6QN6XCCLP55VTTVZG26OLVPS2KZLQ, seg_pos={0 0}, upload=502439]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: read tcp 188.166.225.170:52370->45.91.116.4:4937: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:35:59.876552224 (573h6m59.747635269s) [noise=true , address=45.91.116.4:4937 , piece_id=G5EVO4HSZMQVIYA6JB7MSFL6QN6XCCLP55VTTVZG26OLVPS2KZLQ, seg_pos={0 0}, upload=502439]: Long tail cancellation due to optimal threshold Dec 29 12:35:59.876615244 (573h6m59.747698279s) [seg_pos={0 0}, upload=502439]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:35:59.876698392 (573h6m59.747781427s) [upload=502439]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:35:59.876703523 (573h6m59.747786558s) [upload=502439]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:36:00.186922785 (573h7m0.05800582s) [upload=502439]: Done uploading segments... Dec 29 12:36:00.186947212 (573h7m0.058030247s) [upload=502439]: Upload finished. err: <nil> Duration: 30.003094903s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................X.......X..........X.........] duration: 45.577635294s trace: https://jaeger.dev.storj.io/trace/68764c6cb4286314 FAIL (2025-12-29 12:36:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528204662 (572h59m20.399287707s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528497489 (572h59m20.399580524s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528793153 (572h59m20.399876198s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529083555 (572h59m20.40016659s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529391081 (572h59m20.400474126s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529735096 (572h59m20.400818131s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530109970 (572h59m20.401193015s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530449556 (572h59m20.401532591s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530831554 (572h59m20.401914599s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531229512 (572h59m20.402312557s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531698945 (572h59m20.402781991s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532204238 (572h59m20.403287283s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532744366 (572h59m20.403827401s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533231203 (572h59m20.404314248s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533740113 (572h59m20.404823148s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:21.876539279 (572h59m21.747622324s) [upload=502322]: Next returned nil segment Dec 29 12:28:21.876614272 (572h59m21.747697307s) [upload=502322]: Waiting for error group managing segments... Dec 29 12:28:22.383674429 (572h59m22.254757464s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389083709 (572h59m22.260166754s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389450968 (572h59m22.260534003s) [noise=true , address=145.79.227.46:4625 , piece_id=6NEND2CKGIZLK5Y6IMM4XB6SXPSXUYSZIVZ7CZ25IKAHFFW33C5Q, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.418373151 (572h59m22.289456186s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.619432477 (572h59m22.490515552s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.741876388 (572h59m22.612959423s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.133731187 (572h59m23.004814222s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.146041929 (572h59m23.017124964s) [noise=true , address=193.149.164.143:4697 , piece_id=HZZY3SUOEC7AIHSW6MKZRNZVSOGXSYBYSUKN7STBFTXWAOMDRYBA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.261354601 (572h59m23.132437636s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.291387197 (572h59m23.162470243s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.318370928 (572h59m23.189453973s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.321236023 (572h59m23.192319058s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.408408673 (572h59m23.279491718s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419570137 (572h59m23.290653172s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419592470 (572h59m23.290675515s) [seg_pos={0 0}, upload=502322]: Segment reached optimal threshold of 14 pieces. Dec 29 12:28:23.419837406 (572h59m23.290920441s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123jUFZTVjq6adDuvt3cco7dTvwNTQkszZX7CeajMfbeCUmEmuJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419845241 (572h59m23.290928276s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.419873174 (572h59m23.290956209s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419882021 (572h59m23.290965076s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420073716 (572h59m23.291156751s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection Dec 29 12:28:23.420082262 (572h59m23.291165307s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420257506 (572h59m23.291340551s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: read tcp 188.166.225.170:41270->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:28:23.420264649 (572h59m23.291347684s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420281822 (572h59m23.291364857s) [seg_pos={0 0}, upload=502322]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:28:23.420364349 (572h59m23.291447394s) [upload=502322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:28:23.420370270 (572h59m23.291453315s) [upload=502322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:28:23.725225770 (572h59m23.596308815s) [upload=502322]: Done uploading segments... Dec 29 12:28:23.725255426 (572h59m23.596338481s) [upload=502322]: Upload finished. err: <nil> Duration: 45.577651515s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 38 out of 60 is failed (threshold=6) [...XXX..XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXXXXXXXXX.] duration: 3.706965248s trace: https://jaeger.dev.storj.io/trace/4fd775f7e1cc83f9 FAIL (2025-12-29 12:35:58 UTC) orbiter warning: context deadline exceeded Duration: 3.706991269s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XXXXXX..X.X.......X.XX...X.......X.X..XXX....X...X..X....XX] duration: 30.002936319s trace: https://jaeger.dev.storj.io/trace/4e35e671575b849 FAIL (2025-12-29 12:35:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: B4BO3I4FPLZ7FTVHJFSVGT6SDG3IUJYP5QIGXZZCAFOI6DNBHPYA) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: IMCJGAJ2ODHJP2FZOSDPHMBEGIG6GJ37L6U5Q7U3IC6PDXBTUCWQ) context deadline exceeded ======LOGS====== Dec 29 12:35:24.044815582 (573h6m23.915898617s) [upload=502429]: Starting upload Dec 29 12:35:24.044886276 (573h6m23.915969311s) [upload=502429]: Done starting upload Dec 29 12:35:24.044948275 (573h6m23.91603131s) [upload=502429]: Uploading segments... Dec 29 12:35:24.045084364 (573h6m23.916167409s) [upload=502429]: Got next segment. Inline: false Dec 29 12:35:24.045121955 (573h6m23.91620499s) [upload=502429]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:35:24.045126454 (573h6m23.916209489s) [upload=502429]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:35:24.287483216 (573h6m24.158566261s) [seg_pos={0 0}, upload=502429]: Begin upload segment... Dec 29 12:35:24.287536046 (573h6m24.158619081s) [seg_pos={0 0}, upload=502429]: Done begin upload segment. Dec 29 12:35:24.287562727 (573h6m24.158645762s) [noise=true , address=77.90.31.13:4627 , piece_id=N7BDRLSYN6TTRHFMTGXUOWIZTILU6YYF5GZEAQILASDFCGB3CXDQ, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.287675902 (573h6m24.158758947s) [noise=true , address=145.79.227.32:4834 , piece_id=WCIVXR772TG6LYLFTHENKG46ZPMTW6IUOGVISZOJ4F7L6RFRRTQA, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.288122433 (573h6m24.159205468s) [noise=true , address=145.79.227.35:4482 , piece_id=FLFKAN6YIDLTZAAH6J4A4AJFDPJMJIUKRM566ZNWWRDGREDLXMMQ, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.288420340 (573h6m24.159503385s) [noise=true , address=145.79.227.49:4338 , piece_id=M3FAA24AAD5SN2WBDRKPWMQLFLRCQBPSUVCSETC3YCIB55EHYCLQ, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.288803800 (573h6m24.159886835s) [noise=true , address=45.91.116.13:4153 , piece_id=IMCJGAJ2ODHJP2FZOSDPHMBEGIG6GJ37L6U5Q7U3IC6PDXBTUCWQ, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.289091979 (573h6m24.160175024s) [noise=true , address=145.79.227.5:4222 , piece_id=DDKILONXB4HFKHCWL26DP6GBBD4A4I3ONZXUARL3RL3L3CFV2NRQ, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.289542066 (573h6m24.160625101s) [noise=true , address=145.79.227.26:4655 , piece_id=65YEWVC7SHYKE2P3PGPZYJDSN7FWAJT56IYA7IFYAGZ5EBCZJU7Q, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.290075622 (573h6m24.161158677s) [noise=true , address=45.91.116.14:4190 , piece_id=B4BO3I4FPLZ7FTVHJFSVGT6SDG3IUJYP5QIGXZZCAFOI6DNBHPYA, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.290435267 (573h6m24.161518312s) [noise=true , address=145.79.227.3:4149 , piece_id=DZIBDRTWGBOXZYMCVMNITJHFZSG27TYDYT4HQTGNBLPKHKL5GJPQ, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.291668425 (573h6m24.16275148s) [noise=true , address=145.79.227.4:4185 , piece_id=E36ZVT2O4M65WONJZRKQRSU7HBAZQNRSDHONXMVEM3L62YGTW3PA, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.292660214 (573h6m24.163743259s) [noise=true , address=145.79.227.42:4121 , piece_id=5WIFOHDUBGAARBHJ2XKG2ETNB45YQ6VEBMAMHM7EKDAAB3PNV4MQ, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.294086660 (573h6m24.165169695s) [noise=true , address=145.79.227.21:4473 , piece_id=WVW7EH3IZGQBQLVCHOHENLZRZP4QGO3ZWHDPNYNAAOYONI44J7CA, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.295434696 (573h6m24.166517731s) [noise=true , address=145.79.227.50:4374 , piece_id=QTNBBE7B3KOVMZG4CFQZWTYBLI4HFAJIZEMLKV7JSG74WNTW7IOQ, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.296744150 (573h6m24.167827195s) [noise=true , address=145.79.227.28:4725 , piece_id=OVRZCIYCFACNX4XQKLOHMTJKP6GHLTELIZCA6JF5Z7RKPXABQCDQ, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.298077228 (573h6m24.169160263s) [noise=true , address=77.90.31.11:4553 , piece_id=M5UP67N6S5A4QAZ7ESCASC3RYZYXKIUMX3CFJKKTOPMSU6ZDUQBA, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.299128450 (573h6m24.170211485s) [noise=true , address=145.79.227.38:4978 , piece_id=P3EBFDHECSKYXUPXMTQA54A7GH3EBM26SMKCU7MJR2MVDAECWVSA, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:25.918110760 (573h6m25.789193806s) [upload=502429]: Next returned nil segment Dec 29 12:35:25.918124516 (573h6m25.789207561s) [upload=502429]: Waiting for error group managing segments... Dec 29 12:35:26.274366734 (573h6m26.145449769s) [noise=true , address=145.79.227.5:4222 , piece_id=DDKILONXB4HFKHCWL26DP6GBBD4A4I3ONZXUARL3RL3L3CFV2NRQ, seg_pos={0 0}, upload=502429]: Done uploading piece. err: <nil> Dec 29 12:35:26.274616219 (573h6m26.145699264s) [noise=true , address=145.79.227.4:4185 , piece_id=E36ZVT2O4M65WONJZRKQRSU7HBAZQNRSDHONXMVEM3L62YGTW3PA, seg_pos={0 0}, upload=502429]: Done uploading piece. err: <nil> Dec 29 12:35:26.278366591 (573h6m26.149449636s) [noise=true , address=145.79.227.3:4149 , piece_id=DZIBDRTWGBOXZYMCVMNITJHFZSG27TYDYT4HQTGNBLPKHKL5GJPQ, seg_pos={0 0}, upload=502429]: Done uploading piece. err: <nil> Dec 29 12:35:26.279706392 (573h6m26.150789427s) [noise=true , address=145.79.227.50:4374 , piece_id=QTNBBE7B3KOVMZG4CFQZWTYBLI4HFAJIZEMLKV7JSG74WNTW7IOQ, seg_pos={0 0}, upload=502429]: Done uploading piece. err: <nil> Dec 29 12:35:26.284638052 (573h6m26.155721087s) [noise=true , address=145.79.227.32:4834 , piece_id=WCIVXR772TG6LYLFTHENKG46ZPMTW6IUOGVISZOJ4F7L6RFRRTQA, seg_pos={0 0}, upload=502429]: Done uploading piece. err: <nil> Dec 29 12:35:26.285181236 (573h6m26.156264291s) [noise=true , address=145.79.227.28:4725 , piece_id=OVRZCIYCFACNX4XQKLOHMTJKP6GHLTELIZCA6JF5Z7RKPXABQCDQ, seg_pos={0 0}, upload=502429]: Done uploading piece. err: <nil> Dec 29 12:35:26.301041737 (573h6m26.172124772s) [noise=true , address=145.79.227.21:4473 , piece_id=WVW7EH3IZGQBQLVCHOHENLZRZP4QGO3ZWHDPNYNAAOYONI44J7CA, seg_pos={0 0}, upload=502429]: Done uploading piece. err: <nil> Dec 29 12:35:26.301536389 (573h6m26.172619434s) [noise=true , address=145.79.227.38:4978 , piece_id=P3EBFDHECSKYXUPXMTQA54A7GH3EBM26SMKCU7MJR2MVDAECWVSA, seg_pos={0 0}, upload=502429]: Done uploading piece. err: <nil> Dec 29 12:35:26.306014897 (573h6m26.177097942s) [noise=true , address=145.79.227.42:4121 , piece_id=5WIFOHDUBGAARBHJ2XKG2ETNB45YQ6VEBMAMHM7EKDAAB3PNV4MQ, seg_pos={0 0}, upload=502429]: Done uploading piece. err: <nil> Dec 29 12:35:26.306836561 (573h6m26.177919596s) [noise=true , address=145.79.227.26:4655 , piece_id=65YEWVC7SHYKE2P3PGPZYJDSN7FWAJT56IYA7IFYAGZ5EBCZJU7Q, seg_pos={0 0}, upload=502429]: Done uploading piece. err: <nil> Dec 29 12:35:26.556961686 (573h6m26.428044721s) [noise=true , address=45.91.116.13:4153 , piece_id=IMCJGAJ2ODHJP2FZOSDPHMBEGIG6GJ37L6U5Q7U3IC6PDXBTUCWQ, seg_pos={0 0}, upload=502429]: Done uploading piece. err: <nil> Dec 29 12:35:26.565925003 (573h6m26.437008038s) [noise=true , address=45.91.116.14:4190 , piece_id=B4BO3I4FPLZ7FTVHJFSVGT6SDG3IUJYP5QIGXZZCAFOI6DNBHPYA, seg_pos={0 0}, upload=502429]: Done uploading piece. err: <nil> Dec 29 12:35:26.610410009 (573h6m26.481493044s) [noise=true , address=77.90.31.13:4627 , piece_id=N7BDRLSYN6TTRHFMTGXUOWIZTILU6YYF5GZEAQILASDFCGB3CXDQ, seg_pos={0 0}, upload=502429]: Done uploading piece. err: <nil> Dec 29 12:35:26.610427303 (573h6m26.481510338s) [seg_pos={0 0}, upload=502429]: Segment reached optimal threshold of 13 pieces. Dec 29 12:35:26.610672970 (573h6m26.481756016s) [noise=true , address=145.79.227.49:4338 , piece_id=M3FAA24AAD5SN2WBDRKPWMQLFLRCQBPSUVCSETC3YCIB55EHYCLQ, seg_pos={0 0}, upload=502429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 29 12:35:26.610683330 (573h6m26.481766365s) [noise=true , address=145.79.227.49:4338 , piece_id=M3FAA24AAD5SN2WBDRKPWMQLFLRCQBPSUVCSETC3YCIB55EHYCLQ, seg_pos={0 0}, upload=502429]: Long tail cancellation due to optimal threshold Dec 29 12:35:26.610952302 (573h6m26.482035337s) [noise=true , address=77.90.31.11:4553 , piece_id=M5UP67N6S5A4QAZ7ESCASC3RYZYXKIUMX3CFJKKTOPMSU6ZDUQBA, seg_pos={0 0}, upload=502429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:35696->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:35696->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35696->77.90.31.11:4553: use of closed network connection Dec 29 12:35:26.610962682 (573h6m26.482045737s) [noise=true , address=77.90.31.11:4553 , piece_id=M5UP67N6S5A4QAZ7ESCASC3RYZYXKIUMX3CFJKKTOPMSU6ZDUQBA, seg_pos={0 0}, upload=502429]: Long tail cancellation due to optimal threshold Dec 29 12:35:26.610982750 (573h6m26.482065785s) [noise=true , address=145.79.227.35:4482 , piece_id=FLFKAN6YIDLTZAAH6J4A4AJFDPJMJIUKRM566ZNWWRDGREDLXMMQ, seg_pos={0 0}, upload=502429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 29 12:35:26.611001165 (573h6m26.4820842s) [noise=true , address=145.79.227.35:4482 , piece_id=FLFKAN6YIDLTZAAH6J4A4AJFDPJMJIUKRM566ZNWWRDGREDLXMMQ, seg_pos={0 0}, upload=502429]: Long tail cancellation due to optimal threshold Dec 29 12:35:26.611060909 (573h6m26.482143944s) [seg_pos={0 0}, upload=502429]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:35:26.611127947 (573h6m26.482210982s) [upload=502429]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:35:26.611155329 (573h6m26.482238364s) [upload=502429]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:35:26.918139471 (573h6m26.789222516s) [upload=502429]: Done uploading segments... Dec 29 12:35:26.918164068 (573h6m26.789247103s) [upload=502429]: Upload finished. err: <nil> Duration: 30.002956148s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................X.......X..........X........] duration: 42.088652307s trace: https://jaeger.dev.storj.io/trace/68764c6cb4286314 FAIL (2025-12-29 12:35:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528204662 (572h59m20.399287707s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528497489 (572h59m20.399580524s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528793153 (572h59m20.399876198s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529083555 (572h59m20.40016659s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529391081 (572h59m20.400474126s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529735096 (572h59m20.400818131s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530109970 (572h59m20.401193015s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530449556 (572h59m20.401532591s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530831554 (572h59m20.401914599s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531229512 (572h59m20.402312557s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531698945 (572h59m20.402781991s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532204238 (572h59m20.403287283s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532744366 (572h59m20.403827401s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533231203 (572h59m20.404314248s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533740113 (572h59m20.404823148s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:21.876539279 (572h59m21.747622324s) [upload=502322]: Next returned nil segment Dec 29 12:28:21.876614272 (572h59m21.747697307s) [upload=502322]: Waiting for error group managing segments... Dec 29 12:28:22.383674429 (572h59m22.254757464s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389083709 (572h59m22.260166754s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389450968 (572h59m22.260534003s) [noise=true , address=145.79.227.46:4625 , piece_id=6NEND2CKGIZLK5Y6IMM4XB6SXPSXUYSZIVZ7CZ25IKAHFFW33C5Q, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.418373151 (572h59m22.289456186s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.619432477 (572h59m22.490515552s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.741876388 (572h59m22.612959423s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.133731187 (572h59m23.004814222s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.146041929 (572h59m23.017124964s) [noise=true , address=193.149.164.143:4697 , piece_id=HZZY3SUOEC7AIHSW6MKZRNZVSOGXSYBYSUKN7STBFTXWAOMDRYBA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.261354601 (572h59m23.132437636s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.291387197 (572h59m23.162470243s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.318370928 (572h59m23.189453973s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.321236023 (572h59m23.192319058s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.408408673 (572h59m23.279491718s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419570137 (572h59m23.290653172s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419592470 (572h59m23.290675515s) [seg_pos={0 0}, upload=502322]: Segment reached optimal threshold of 14 pieces. Dec 29 12:28:23.419837406 (572h59m23.290920441s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123jUFZTVjq6adDuvt3cco7dTvwNTQkszZX7CeajMfbeCUmEmuJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419845241 (572h59m23.290928276s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.419873174 (572h59m23.290956209s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419882021 (572h59m23.290965076s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420073716 (572h59m23.291156751s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection Dec 29 12:28:23.420082262 (572h59m23.291165307s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420257506 (572h59m23.291340551s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: read tcp 188.166.225.170:41270->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:28:23.420264649 (572h59m23.291347684s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420281822 (572h59m23.291364857s) [seg_pos={0 0}, upload=502322]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:28:23.420364349 (572h59m23.291447394s) [upload=502322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:28:23.420370270 (572h59m23.291453315s) [upload=502322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:28:23.725225770 (572h59m23.596308815s) [upload=502322]: Done uploading segments... Dec 29 12:28:23.725255426 (572h59m23.596338481s) [upload=502322]: Upload finished. err: <nil> Duration: 42.088669319s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..XXXXXX..X.X.......X.XX...X.......X.X..XXX....X...X..X....X] duration: 30.002823037s trace: https://jaeger.dev.storj.io/trace/29290dcdebfdabb FAIL (2025-12-29 12:35:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: OAXXRPE2HCSYT6ECJJN3QCTW7RX6RQZGMDAMZDYAR7PDQ4IBPYJA) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: CXJEXDSXFCZBBBF447GL6INCXX5VX6M2EXGMFW64XQBJWDTGGUHQ) context deadline exceeded ======LOGS====== m42.937291868s) [upload=502419]: Uploading segments... Dec 29 12:34:43.066323461 (573h5m42.937406496s) [upload=502419]: Got next segment. Inline: false Dec 29 12:34:43.066369318 (573h5m42.937452353s) [upload=502419]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:34:43.066373927 (573h5m42.937456962s) [upload=502419]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:34:43.325012811 (573h5m43.196095856s) [seg_pos={0 0}, upload=502419]: Begin upload segment... Dec 29 12:34:43.325067394 (573h5m43.196150429s) [seg_pos={0 0}, upload=502419]: Done begin upload segment. Dec 29 12:34:43.325084688 (573h5m43.196167723s) [noise=true , address=145.79.227.20:4437 , piece_id=P4ORQRTHDIZT3MKDLIKJJNJKNTZBOIKN4BGB46B5F2WI6GR4XIUA, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.325097221 (573h5m43.196180256s) [noise=true , address=77.90.31.14:4661 , piece_id=OAXXRPE2HCSYT6ECJJN3QCTW7RX6RQZGMDAMZDYAR7PDQ4IBPYJA, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.325255443 (573h5m43.196338478s) [noise=true , address=145.79.227.47:4267 , piece_id=IDV6KZCBTCKY2JM2U5B2GYRFSDGQDIK33JJFAMWCKUFYBPZIRTSA, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.325549322 (573h5m43.196632357s) [noise=true , address=145.79.227.21:4473 , piece_id=UYGLGKEX7MXSPSVZ5SQU42WLSHIU2FPI4HHJUK63EYTYRI2KTZ6A, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.326029567 (573h5m43.197112602s) [noise=true , address=45.91.116.5:4297 , piece_id=5FADDZAHAYDWDNYGXH6IS2VHLLBE4UVWUD4DCCAIJDMNTAIVSM7Q, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.326298168 (573h5m43.197381204s) [noise=true , address=77.90.31.21:4878 , piece_id=5NZHBG4LBZU74LGWGEAT5IOF37EMVDQAE7ZVPZSWD2E3IRMV4G7A, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.326545079 (573h5m43.197628114s) [noise=true , address=45.91.116.12:4226 , piece_id=JMU7N3SBOBGTKSP35UQF3EUC3OXGN4F4CZHUBBFLJ4HPER7BYSDQ, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.326827136 (573h5m43.197910171s) [noise=true , address=145.79.227.41:4086 , piece_id=BBE4OMYOUHHRA5LKZ4ACYELN3AUVPDIA7PKBSTESTWN3YO2GDUFQ, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.327137436 (573h5m43.198220502s) [noise=true , address=145.79.227.5:4221 , piece_id=S43XISH5LCQDSKRSRHNBX4I7WM2IRYYSPWXLKPJVAFQKOEL2776A, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.327510146 (573h5m43.198593182s) [noise=true , address=145.79.227.3:4149 , piece_id=4PCA6YHFKOTB5P5X4YWFELQ4W3OUHWCSDYDB5FBO2NKBZIIWLDEQ, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.327834744 (573h5m43.198917779s) [noise=true , address=145.79.227.38:4977 , piece_id=7KLAXLYXGYNXBMTVHAGRP2XZS7MF2UJ42367L6CHWZIAXBQANNZA, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.328725660 (573h5m43.199808695s) [noise=true , address=77.90.31.16:4735 , piece_id=CXJEXDSXFCZBBBF447GL6INCXX5VX6M2EXGMFW64XQBJWDTGGUHQ, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.328838596 (573h5m43.199921641s) [noise=true , address=145.79.227.16:4909 , piece_id=U37QA2QKVMAE4P55CVYHXL2XEM3P3IFM2X24J25PL3WDDLVRXCWA, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.329920797 (573h5m43.201003832s) [noise=true , address=145.79.227.39:4013 , piece_id=DXWQD2L57XBL2QNPN2BV3YXEAY7JQJJMFQXCLKPMB2RKZ74JJYTQ, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.331052321 (573h5m43.202135356s) [noise=true , address=45.91.116.4:4940 , piece_id=TWLYKO7OVPB56J5QBOSDFXHCVIWBD7FLILLYN5XX35BUV5F2HUQA, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.332046695 (573h5m43.20312973s) [noise=true , address=145.79.227.26:4654 , piece_id=A7BOQWYJQRIMLU7UAQ46NAWRWQ3SIYAV3YQIHK5DYAD4VGRDBY2A, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:44.936900204 (573h5m44.80798324s) [upload=502419]: Next returned nil segment Dec 29 12:34:44.936914050 (573h5m44.807997086s) [upload=502419]: Waiting for error group managing segments... Dec 29 12:34:45.313615295 (573h5m45.18469833s) [noise=true , address=145.79.227.21:4473 , piece_id=UYGLGKEX7MXSPSVZ5SQU42WLSHIU2FPI4HHJUK63EYTYRI2KTZ6A, seg_pos={0 0}, upload=502419]: Done uploading piece. err: <nil> Dec 29 12:34:45.314226138 (573h5m45.185309183s) [noise=true , address=145.79.227.20:4437 , piece_id=P4ORQRTHDIZT3MKDLIKJJNJKNTZBOIKN4BGB46B5F2WI6GR4XIUA, seg_pos={0 0}, upload=502419]: Done uploading piece. err: <nil> Dec 29 12:34:45.315203449 (573h5m45.186286484s) [noise=true , address=145.79.227.41:4086 , piece_id=BBE4OMYOUHHRA5LKZ4ACYELN3AUVPDIA7PKBSTESTWN3YO2GDUFQ, seg_pos={0 0}, upload=502419]: Done uploading piece. err: <nil> Dec 29 12:34:45.317324057 (573h5m45.188407102s) [noise=true , address=145.79.227.5:4221 , piece_id=S43XISH5LCQDSKRSRHNBX4I7WM2IRYYSPWXLKPJVAFQKOEL2776A, seg_pos={0 0}, upload=502419]: Done uploading piece. err: <nil> Dec 29 12:34:45.318854060 (573h5m45.189937105s) [noise=true , address=145.79.227.16:4909 , piece_id=U37QA2QKVMAE4P55CVYHXL2XEM3P3IFM2X24J25PL3WDDLVRXCWA, seg_pos={0 0}, upload=502419]: Done uploading piece. err: <nil> Dec 29 12:34:45.319441118 (573h5m45.190524153s) [noise=true , address=145.79.227.3:4149 , piece_id=4PCA6YHFKOTB5P5X4YWFELQ4W3OUHWCSDYDB5FBO2NKBZIIWLDEQ, seg_pos={0 0}, upload=502419]: Done uploading piece. err: <nil> Dec 29 12:34:45.320131413 (573h5m45.191214458s) [noise=true , address=145.79.227.38:4977 , piece_id=7KLAXLYXGYNXBMTVHAGRP2XZS7MF2UJ42367L6CHWZIAXBQANNZA, seg_pos={0 0}, upload=502419]: Done uploading piece. err: <nil> Dec 29 12:34:45.325024109 (573h5m45.196107144s) [noise=true , address=145.79.227.26:4654 , piece_id=A7BOQWYJQRIMLU7UAQ46NAWRWQ3SIYAV3YQIHK5DYAD4VGRDBY2A, seg_pos={0 0}, upload=502419]: Done uploading piece. err: <nil> Dec 29 12:34:45.327497859 (573h5m45.198580894s) [noise=true , address=145.79.227.47:4267 , piece_id=IDV6KZCBTCKY2JM2U5B2GYRFSDGQDIK33JJFAMWCKUFYBPZIRTSA, seg_pos={0 0}, upload=502419]: Done uploading piece. err: <nil> Dec 29 12:34:45.346052220 (573h5m45.217135255s) [noise=true , address=145.79.227.39:4013 , piece_id=DXWQD2L57XBL2QNPN2BV3YXEAY7JQJJMFQXCLKPMB2RKZ74JJYTQ, seg_pos={0 0}, upload=502419]: Done uploading piece. err: <nil> Dec 29 12:34:45.521418984 (573h5m45.392502029s) [noise=true , address=77.90.31.21:4878 , piece_id=5NZHBG4LBZU74LGWGEAT5IOF37EMVDQAE7ZVPZSWD2E3IRMV4G7A, seg_pos={0 0}, upload=502419]: Done uploading piece. err: <nil> Dec 29 12:34:45.620408424 (573h5m45.491491469s) [noise=true , address=77.90.31.16:4735 , piece_id=CXJEXDSXFCZBBBF447GL6INCXX5VX6M2EXGMFW64XQBJWDTGGUHQ, seg_pos={0 0}, upload=502419]: Done uploading piece. err: <nil> Dec 29 12:34:45.645376457 (573h5m45.516459502s) [noise=true , address=77.90.31.14:4661 , piece_id=OAXXRPE2HCSYT6ECJJN3QCTW7RX6RQZGMDAMZDYAR7PDQ4IBPYJA, seg_pos={0 0}, upload=502419]: Done uploading piece. err: <nil> Dec 29 12:34:45.645398809 (573h5m45.516481854s) [seg_pos={0 0}, upload=502419]: Segment reached optimal threshold of 13 pieces. Dec 29 12:34:45.645677029 (573h5m45.516760064s) [noise=true , address=45.91.116.12:4226 , piece_id=JMU7N3SBOBGTKSP35UQF3EUC3OXGN4F4CZHUBBFLJ4HPER7BYSDQ, seg_pos={0 0}, upload=502419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:41234->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:41234->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41234->45.91.116.12:4226: use of closed network connection Dec 29 12:34:45.645704241 (573h5m45.516787276s) [noise=true , address=45.91.116.12:4226 , piece_id=JMU7N3SBOBGTKSP35UQF3EUC3OXGN4F4CZHUBBFLJ4HPER7BYSDQ, seg_pos={0 0}, upload=502419]: Long tail cancellation due to optimal threshold Dec 29 12:34:45.645895356 (573h5m45.516978391s) [noise=true , address=45.91.116.4:4940 , piece_id=TWLYKO7OVPB56J5QBOSDFXHCVIWBD7FLILLYN5XX35BUV5F2HUQA, seg_pos={0 0}, upload=502419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:42122->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:42122->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42122->45.91.116.4:4940: use of closed network connection Dec 29 12:34:45.645924140 (573h5m45.517007185s) [noise=true , address=45.91.116.4:4940 , piece_id=TWLYKO7OVPB56J5QBOSDFXHCVIWBD7FLILLYN5XX35BUV5F2HUQA, seg_pos={0 0}, upload=502419]: Long tail cancellation due to optimal threshold Dec 29 12:34:45.645956301 (573h5m45.517039336s) [noise=true , address=45.91.116.5:4297 , piece_id=5FADDZAHAYDWDNYGXH6IS2VHLLBE4UVWUD4DCCAIJDMNTAIVSM7Q, seg_pos={0 0}, upload=502419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 29 12:34:45.645986929 (573h5m45.517069964s) [noise=true , address=45.91.116.5:4297 , piece_id=5FADDZAHAYDWDNYGXH6IS2VHLLBE4UVWUD4DCCAIJDMNTAIVSM7Q, seg_pos={0 0}, upload=502419]: Long tail cancellation due to optimal threshold Dec 29 12:34:45.646010334 (573h5m45.517093379s) [seg_pos={0 0}, upload=502419]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:34:45.646113591 (573h5m45.517196626s) [upload=502419]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:34:45.646118430 (573h5m45.517201465s) [upload=502419]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:34:45.935447816 (573h5m45.806530861s) [upload=502419]: Done uploading segments... Dec 29 12:34:45.935475378 (573h5m45.806558413s) [upload=502419]: Upload finished. err: <nil> Duration: 30.00284547s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 39 out of 60 is failed (threshold=6) [X...XXX..XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXXXXXXXXX] duration: 5.001603875s trace: https://jaeger.dev.storj.io/trace/4fd775f7e1cc83f9 FAIL (2025-12-29 12:35:02 UTC) orbiter warning: context deadline exceeded Duration: 5.001630946s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................X.......X..........X.......] duration: 44.805290832s trace: https://jaeger.dev.storj.io/trace/68764c6cb4286314 FAIL (2025-12-29 12:34:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528204662 (572h59m20.399287707s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528497489 (572h59m20.399580524s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528793153 (572h59m20.399876198s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529083555 (572h59m20.40016659s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529391081 (572h59m20.400474126s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529735096 (572h59m20.400818131s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530109970 (572h59m20.401193015s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530449556 (572h59m20.401532591s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530831554 (572h59m20.401914599s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531229512 (572h59m20.402312557s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531698945 (572h59m20.402781991s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532204238 (572h59m20.403287283s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532744366 (572h59m20.403827401s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533231203 (572h59m20.404314248s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533740113 (572h59m20.404823148s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:21.876539279 (572h59m21.747622324s) [upload=502322]: Next returned nil segment Dec 29 12:28:21.876614272 (572h59m21.747697307s) [upload=502322]: Waiting for error group managing segments... Dec 29 12:28:22.383674429 (572h59m22.254757464s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389083709 (572h59m22.260166754s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389450968 (572h59m22.260534003s) [noise=true , address=145.79.227.46:4625 , piece_id=6NEND2CKGIZLK5Y6IMM4XB6SXPSXUYSZIVZ7CZ25IKAHFFW33C5Q, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.418373151 (572h59m22.289456186s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.619432477 (572h59m22.490515552s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.741876388 (572h59m22.612959423s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.133731187 (572h59m23.004814222s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.146041929 (572h59m23.017124964s) [noise=true , address=193.149.164.143:4697 , piece_id=HZZY3SUOEC7AIHSW6MKZRNZVSOGXSYBYSUKN7STBFTXWAOMDRYBA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.261354601 (572h59m23.132437636s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.291387197 (572h59m23.162470243s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.318370928 (572h59m23.189453973s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.321236023 (572h59m23.192319058s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.408408673 (572h59m23.279491718s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419570137 (572h59m23.290653172s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419592470 (572h59m23.290675515s) [seg_pos={0 0}, upload=502322]: Segment reached optimal threshold of 14 pieces. Dec 29 12:28:23.419837406 (572h59m23.290920441s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123jUFZTVjq6adDuvt3cco7dTvwNTQkszZX7CeajMfbeCUmEmuJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419845241 (572h59m23.290928276s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.419873174 (572h59m23.290956209s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419882021 (572h59m23.290965076s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420073716 (572h59m23.291156751s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection Dec 29 12:28:23.420082262 (572h59m23.291165307s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420257506 (572h59m23.291340551s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: read tcp 188.166.225.170:41270->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:28:23.420264649 (572h59m23.291347684s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420281822 (572h59m23.291364857s) [seg_pos={0 0}, upload=502322]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:28:23.420364349 (572h59m23.291447394s) [upload=502322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:28:23.420370270 (572h59m23.291453315s) [upload=502322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:28:23.725225770 (572h59m23.596308815s) [upload=502322]: Done uploading segments... Dec 29 12:28:23.725255426 (572h59m23.596338481s) [upload=502322]: Upload finished. err: <nil> Duration: 44.805307845s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [...XXXXXX..X.X.......X.XX...X.......X.X..XXX....X...X..X....] duration: 6.012061916s trace: https://jaeger.dev.storj.io/trace/55af87306b44a0fb FAIL (2025-12-29 12:34:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 12:30:27.976557996 (573h1m27.847641031s) [upload=502356]: Starting upload Dec 29 12:30:27.976643769 (573h1m27.847726804s) [upload=502356]: Done starting upload Dec 29 12:30:27.976805467 (573h1m27.847888512s) [upload=502356]: Uploading segments... Dec 29 12:30:27.976959821 (573h1m27.848042856s) [upload=502356]: Got next segment. Inline: false Dec 29 12:30:27.977030205 (573h1m27.84811324s) [upload=502356]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:30:27.977037499 (573h1m27.848120544s) [upload=502356]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:30:28.227612220 (573h1m28.098695265s) [seg_pos={0 0}, upload=502356]: Begin upload segment... Dec 29 12:30:28.227680119 (573h1m28.098763154s) [seg_pos={0 0}, upload=502356]: Done begin upload segment. Dec 29 12:30:28.227731227 (573h1m28.098814272s) [noise=true , address=145.79.227.2:4115 , piece_id=FYK6H2CTV3YRNPU3TW5ICDUUHNJLUGYB7YVL242N7EKXA3OFMUHQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.227777565 (573h1m28.0988606s) [noise=true , address=45.91.116.14:4190 , piece_id=SVSZXQDV6WXOB5BID3HVGTAPTKSOJ2PXSGGW3OIQ5HEZRGVRG6DA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.228331690 (573h1m28.099414725s) [noise=true , address=45.91.116.12:4227 , piece_id=34PI32NCVIDSE2W6UAATADRFY2ZKVIS5DOXTHGL2TWDNUYZK5I4A, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.228647301 (573h1m28.099730336s) [noise=true , address=145.79.227.10:4401 , piece_id=TCQXBURRVQY5VJBL6CVM56AVQ23IALWXMJYCXI2FQNK5VEA7ZP6A, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.228978120 (573h1m28.100061155s) [noise=true , address=145.79.227.41:4085 , piece_id=MZYJNEXIKOQVVEZPIEZ33W6XJWKMY3LLYKVUW5PNSRHCD7Y6AZQA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.229164997 (573h1m28.100248032s) [noise=true , address=145.79.227.46:4601 , piece_id=C3VY24KTEOXUNLPAVDT45J2LRQPQ6AOR2OKTEJGLARZOUGOS64YQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.229770089 (573h1m28.100853124s) [noise=true , address=145.79.227.32:4835 , piece_id=GI4T5K7YGHDDF75UUH722QEZYSYU3DRXS4HPYWLLADM6J2EHNR5A, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.230196019 (573h1m28.101279054s) [noise=true , address=145.79.227.48:4301 , piece_id=VEJVLLQU2KDKYM3Y5XZB6TNUUXSX2MDC6Y2KXN6RUEWASB7G6TAQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.230229163 (573h1m28.101312198s) [noise=true , address=145.79.227.45:4231 , piece_id=AI4FTKKS7SWUMAQ35E4ML7TT3O2CGOZHAA5FQPIM4L2TSDOV3TSQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.230721130 (573h1m28.101804165s) [noise=true , address=77.90.31.16:4735 , piece_id=LED3MXVURDX5ITUWGZGJHXM2LXAX3ISSJKAECUJUOABOLUHEAKKA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.230975274 (573h1m28.102058309s) [noise=true , address=145.79.227.13:4802 , piece_id=MR6H5CX267HPWCETWENONX5QMETKXN3SL5LKWYNBNDEMBWR46ZAA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.231314189 (573h1m28.102397235s) [noise=true , address=45.91.116.10:4083 , piece_id=AYQXAYPZMWZN2YSVCNUENMW4KCE6BFBEBV2N57BRRDLNYL5ZCLTA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.231509651 (573h1m28.102592686s) [noise=true , address=145.79.227.26:4654 , piece_id=CJJW6NM2HL572SDZSTGZID6S6PRCFPS6DEJMQU4WN3DX2GMMLEHQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.231966481 (573h1m28.103049526s) [noise=true , address=45.91.116.6:4974 , piece_id=NCAE2UNBQ7IAI2B2EE5QKMIJQZ5NPAMBGEYODLFMERSD5IVYUUHA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.232489376 (573h1m28.103572411s) [noise=true , address=145.79.227.7:4293 , piece_id=6FYYAD6K52LDUNKOGFFAGNLPFHKA7V3WVBRPKHOZBZXMHRHIACYA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.232858590 (573h1m28.103941625s) [noise=true , address=45.91.116.5:4297 , piece_id=63ABPGW4YSCEPCPMQHFZ6NZVWNYJAELGJNXJJXWFTHVTW6YIVAQA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:29.934302048 (573h1m29.805385083s) [upload=502356]: Next returned nil segment Dec 29 12:30:29.934324049 (573h1m29.805407094s) [upload=502356]: Waiting for error group managing segments... Dec 29 12:30:30.219330104 (573h1m30.090413149s) [noise=true , address=145.79.227.2:4115 , piece_id=FYK6H2CTV3YRNPU3TW5ICDUUHNJLUGYB7YVL242N7EKXA3OFMUHQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.222585522 (573h1m30.093668557s) [noise=true , address=145.79.227.26:4654 , piece_id=CJJW6NM2HL572SDZSTGZID6S6PRCFPS6DEJMQU4WN3DX2GMMLEHQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.223473883 (573h1m30.094556928s) [noise=true , address=145.79.227.7:4293 , piece_id=6FYYAD6K52LDUNKOGFFAGNLPFHKA7V3WVBRPKHOZBZXMHRHIACYA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.225048181 (573h1m30.096131226s) [noise=true , address=145.79.227.32:4835 , piece_id=GI4T5K7YGHDDF75UUH722QEZYSYU3DRXS4HPYWLLADM6J2EHNR5A, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.227668892 (573h1m30.098751937s) [noise=true , address=145.79.227.10:4401 , piece_id=TCQXBURRVQY5VJBL6CVM56AVQ23IALWXMJYCXI2FQNK5VEA7ZP6A, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.228574516 (573h1m30.099657561s) [noise=true , address=145.79.227.13:4802 , piece_id=MR6H5CX267HPWCETWENONX5QMETKXN3SL5LKWYNBNDEMBWR46ZAA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.234122320 (573h1m30.105205355s) [noise=true , address=145.79.227.45:4231 , piece_id=AI4FTKKS7SWUMAQ35E4ML7TT3O2CGOZHAA5FQPIM4L2TSDOV3TSQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.240891529 (573h1m30.111974574s) [noise=true , address=145.79.227.48:4301 , piece_id=VEJVLLQU2KDKYM3Y5XZB6TNUUXSX2MDC6Y2KXN6RUEWASB7G6TAQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.241789950 (573h1m30.112872985s) [noise=true , address=145.79.227.46:4601 , piece_id=C3VY24KTEOXUNLPAVDT45J2LRQPQ6AOR2OKTEJGLARZOUGOS64YQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.247350498 (573h1m30.118433534s) [noise=true , address=145.79.227.41:4085 , piece_id=MZYJNEXIKOQVVEZPIEZ33W6XJWKMY3LLYKVUW5PNSRHCD7Y6AZQA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.417471593 (573h1m30.288554628s) [noise=true , address=77.90.31.16:4735 , piece_id=LED3MXVURDX5ITUWGZGJHXM2LXAX3ISSJKAECUJUOABOLUHEAKKA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.432824027 (573h1m30.303907062s) [noise=true , address=45.91.116.10:4083 , piece_id=AYQXAYPZMWZN2YSVCNUENMW4KCE6BFBEBV2N57BRRDLNYL5ZCLTA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.583724624 (573h1m30.454807659s) [noise=true , address=45.91.116.14:4190 , piece_id=SVSZXQDV6WXOB5BID3HVGTAPTKSOJ2PXSGGW3OIQ5HEZRGVRG6DA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.583743831 (573h1m30.454826876s) [seg_pos={0 0}, upload=502356]: Segment reached optimal threshold of 13 pieces. Dec 29 12:30:30.583922962 (573h1m30.455005997s) [noise=true , address=45.91.116.5:4297 , piece_id=63ABPGW4YSCEPCPMQHFZ6NZVWNYJAELGJNXJJXWFTHVTW6YIVAQA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 29 12:30:30.583929113 (573h1m30.455012149s) [noise=true , address=45.91.116.5:4297 , piece_id=63ABPGW4YSCEPCPMQHFZ6NZVWNYJAELGJNXJJXWFTHVTW6YIVAQA, seg_pos={0 0}, upload=502356]: Long tail cancellation due to optimal threshold Dec 29 12:30:30.584091974 (573h1m30.455175009s) [noise=true , address=45.91.116.6:4974 , piece_id=NCAE2UNBQ7IAI2B2EE5QKMIJQZ5NPAMBGEYODLFMERSD5IVYUUHA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:30:30.584108805 (573h1m30.455191841s) [noise=true , address=45.91.116.6:4974 , piece_id=NCAE2UNBQ7IAI2B2EE5QKMIJQZ5NPAMBGEYODLFMERSD5IVYUUHA, seg_pos={0 0}, upload=502356]: Long tail cancellation due to optimal threshold Dec 29 12:30:30.584341749 (573h1m30.455424794s) [noise=true , address=45.91.116.12:4227 , piece_id=34PI32NCVIDSE2W6UAATADRFY2ZKVIS5DOXTHGL2TWDNUYZK5I4A, seg_pos={0 0}, upload=502356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: context canceled; context canceled Dec 29 12:30:30.584349564 (573h1m30.455432609s) [noise=true , address=45.91.116.12:4227 , piece_id=34PI32NCVIDSE2W6UAATADRFY2ZKVIS5DOXTHGL2TWDNUYZK5I4A, seg_pos={0 0}, upload=502356]: Long tail cancellation due to optimal threshold Dec 29 12:30:30.584369573 (573h1m30.455452608s) [seg_pos={0 0}, upload=502356]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:30:30.584449214 (573h1m30.455532259s) [upload=502356]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:30:30.584453732 (573h1m30.455536767s) [upload=502356]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:30:30.886256078 (573h1m30.757339123s) [upload=502356]: Done uploading segments... Dec 29 12:30:30.886281006 (573h1m30.757364041s) [upload=502356]: Upload finished. err: <nil> Duration: 6.01210074s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 39 out of 60 is failed (threshold=6) [XX...XXX..XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXXXXXXXX] duration: 5.001439456s trace: https://jaeger.dev.storj.io/trace/810ca148700c57 FAIL (2025-12-29 12:34:13 UTC) orbiter warning: context deadline exceeded Duration: 5.001461187s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................X.......X..........X......] duration: 42.908403656s trace: https://jaeger.dev.storj.io/trace/68764c6cb4286314 FAIL (2025-12-29 12:34:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528204662 (572h59m20.399287707s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528497489 (572h59m20.399580524s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528793153 (572h59m20.399876198s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529083555 (572h59m20.40016659s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529391081 (572h59m20.400474126s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529735096 (572h59m20.400818131s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530109970 (572h59m20.401193015s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530449556 (572h59m20.401532591s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530831554 (572h59m20.401914599s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531229512 (572h59m20.402312557s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531698945 (572h59m20.402781991s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532204238 (572h59m20.403287283s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532744366 (572h59m20.403827401s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533231203 (572h59m20.404314248s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533740113 (572h59m20.404823148s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:21.876539279 (572h59m21.747622324s) [upload=502322]: Next returned nil segment Dec 29 12:28:21.876614272 (572h59m21.747697307s) [upload=502322]: Waiting for error group managing segments... Dec 29 12:28:22.383674429 (572h59m22.254757464s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389083709 (572h59m22.260166754s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389450968 (572h59m22.260534003s) [noise=true , address=145.79.227.46:4625 , piece_id=6NEND2CKGIZLK5Y6IMM4XB6SXPSXUYSZIVZ7CZ25IKAHFFW33C5Q, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.418373151 (572h59m22.289456186s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.619432477 (572h59m22.490515552s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.741876388 (572h59m22.612959423s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.133731187 (572h59m23.004814222s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.146041929 (572h59m23.017124964s) [noise=true , address=193.149.164.143:4697 , piece_id=HZZY3SUOEC7AIHSW6MKZRNZVSOGXSYBYSUKN7STBFTXWAOMDRYBA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.261354601 (572h59m23.132437636s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.291387197 (572h59m23.162470243s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.318370928 (572h59m23.189453973s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.321236023 (572h59m23.192319058s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.408408673 (572h59m23.279491718s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419570137 (572h59m23.290653172s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419592470 (572h59m23.290675515s) [seg_pos={0 0}, upload=502322]: Segment reached optimal threshold of 14 pieces. Dec 29 12:28:23.419837406 (572h59m23.290920441s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123jUFZTVjq6adDuvt3cco7dTvwNTQkszZX7CeajMfbeCUmEmuJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419845241 (572h59m23.290928276s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.419873174 (572h59m23.290956209s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419882021 (572h59m23.290965076s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420073716 (572h59m23.291156751s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection Dec 29 12:28:23.420082262 (572h59m23.291165307s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420257506 (572h59m23.291340551s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: read tcp 188.166.225.170:41270->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:28:23.420264649 (572h59m23.291347684s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420281822 (572h59m23.291364857s) [seg_pos={0 0}, upload=502322]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:28:23.420364349 (572h59m23.291447394s) [upload=502322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:28:23.420370270 (572h59m23.291453315s) [upload=502322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:28:23.725225770 (572h59m23.596308815s) [upload=502322]: Done uploading segments... Dec 29 12:28:23.725255426 (572h59m23.596338481s) [upload=502322]: Upload finished. err: <nil> Duration: 42.908433502s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 39 out of 60 is failed (threshold=6) [XXX...XXX..XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXXXXXXX] duration: 5.00182055s trace: https://jaeger.dev.storj.io/trace/1adb64ac4b59522 FAIL (2025-12-29 12:33:47 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001850756s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)