Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................X.......X..........X..........] duration: 50.342683992s trace: https://jaeger.dev.storj.io/trace/68764c6cb4286314 FAIL (2025-12-29 12:37:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528204662 (572h59m20.399287707s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528497489 (572h59m20.399580524s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528793153 (572h59m20.399876198s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529083555 (572h59m20.40016659s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529391081 (572h59m20.400474126s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529735096 (572h59m20.400818131s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530109970 (572h59m20.401193015s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530449556 (572h59m20.401532591s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530831554 (572h59m20.401914599s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531229512 (572h59m20.402312557s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531698945 (572h59m20.402781991s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532204238 (572h59m20.403287283s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532744366 (572h59m20.403827401s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533231203 (572h59m20.404314248s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533740113 (572h59m20.404823148s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:21.876539279 (572h59m21.747622324s) [upload=502322]: Next returned nil segment Dec 29 12:28:21.876614272 (572h59m21.747697307s) [upload=502322]: Waiting for error group managing segments... Dec 29 12:28:22.383674429 (572h59m22.254757464s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389083709 (572h59m22.260166754s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389450968 (572h59m22.260534003s) [noise=true , address=145.79.227.46:4625 , piece_id=6NEND2CKGIZLK5Y6IMM4XB6SXPSXUYSZIVZ7CZ25IKAHFFW33C5Q, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.418373151 (572h59m22.289456186s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.619432477 (572h59m22.490515552s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.741876388 (572h59m22.612959423s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.133731187 (572h59m23.004814222s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.146041929 (572h59m23.017124964s) [noise=true , address=193.149.164.143:4697 , piece_id=HZZY3SUOEC7AIHSW6MKZRNZVSOGXSYBYSUKN7STBFTXWAOMDRYBA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.261354601 (572h59m23.132437636s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.291387197 (572h59m23.162470243s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.318370928 (572h59m23.189453973s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.321236023 (572h59m23.192319058s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.408408673 (572h59m23.279491718s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419570137 (572h59m23.290653172s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419592470 (572h59m23.290675515s) [seg_pos={0 0}, upload=502322]: Segment reached optimal threshold of 14 pieces. Dec 29 12:28:23.419837406 (572h59m23.290920441s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123jUFZTVjq6adDuvt3cco7dTvwNTQkszZX7CeajMfbeCUmEmuJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419845241 (572h59m23.290928276s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.419873174 (572h59m23.290956209s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419882021 (572h59m23.290965076s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420073716 (572h59m23.291156751s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection Dec 29 12:28:23.420082262 (572h59m23.291165307s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420257506 (572h59m23.291340551s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: read tcp 188.166.225.170:41270->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:28:23.420264649 (572h59m23.291347684s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420281822 (572h59m23.291364857s) [seg_pos={0 0}, upload=502322]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:28:23.420364349 (572h59m23.291447394s) [upload=502322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:28:23.420370270 (572h59m23.291453315s) [upload=502322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:28:23.725225770 (572h59m23.596308815s) [upload=502322]: Done uploading segments... Dec 29 12:28:23.725255426 (572h59m23.596338481s) [upload=502322]: Upload finished. err: <nil> Duration: 50.342711144s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 39 out of 60 is failed (threshold=6) [..XXX..XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXXXXXXXXX.X] duration: 5.001100594s trace: https://jaeger.dev.storj.io/trace/1bd9f56a9e88d2ee FAIL (2025-12-29 12:36:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001113318s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XXXXXX..X.X.......X.XX...X.......X.X..XXX....X...X..X....XXX] duration: 30.003065887s trace: https://jaeger.dev.storj.io/trace/3b55af95361e57c1 FAIL (2025-12-29 12:36:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =502439]: Starting upload Dec 29 12:35:56.314641001 (573h6m56.185724036s) [upload=502439]: Done starting upload Dec 29 12:35:56.314728368 (573h6m56.185811413s) [upload=502439]: Uploading segments... Dec 29 12:35:56.314888352 (573h6m56.185971397s) [upload=502439]: Got next segment. Inline: false Dec 29 12:35:56.314932406 (573h6m56.186015441s) [upload=502439]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:35:56.314938959 (573h6m56.186021994s) [upload=502439]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:35:56.567188299 (573h6m56.438271334s) [seg_pos={0 0}, upload=502439]: Begin upload segment... Dec 29 12:35:56.567242431 (573h6m56.438325466s) [seg_pos={0 0}, upload=502439]: Done begin upload segment. Dec 29 12:35:56.567276065 (573h6m56.43835911s) [noise=true , address=145.79.227.36:4518 , piece_id=755BMSS3R3L65EKO3VGQRYJHLEFYCGXT27YVJCAHUJRIEPEPPJVA, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.567302065 (573h6m56.4383851s) [noise=true , address=145.79.227.3:4150 , piece_id=C2SKGSU7DBMKEGNW6SQ7ESNITDHVTNGHE4JVDQJRVDPDE5O5VUDQ, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.567347221 (573h6m56.438430256s) [noise=true , address=145.79.227.5:4222 , piece_id=SEYX67D3UBXQCBDT3XOHRSWGNPBU63GT3ZNSC7XUKXA7NVKVBGXA, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.567788782 (573h6m56.438871817s) [noise=true , address=145.79.227.17:4971 , piece_id=3JD35EOECXSTO3BMZ3XTCICMBIRVFZUTPQLKJMUVOCYU72IPI75A, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.568135863 (573h6m56.439218908s) [noise=true , address=145.79.227.45:4231 , piece_id=GCOXLBWZNZPQWLOTDQ4U3XWPZR2XMKBIOWZT2VYBG2QLVTRD6SFQ, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.568463307 (573h6m56.439546342s) [noise=true , address=145.79.227.52:4446 , piece_id=UHTHU6UCDVXFEMUO57CPRBN45FZ4VSCZMRF4VUHK6Q54ZFWYVGJQ, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.568793405 (573h6m56.43987644s) [noise=true , address=45.91.116.12:4225 , piece_id=4U5XB27V6GOJV2G2QZ24ZBPE5HLVAD7PZCLZCA3VCI74F6G4XEXQ, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.571479169 (573h6m56.442562214s) [noise=true , address=145.79.227.15:4873 , piece_id=HXJ7MUK5NXVDGK475QLSS2SWBBBYDU2E52TIJR5OPLPHZI73IC7A, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.572622166 (573h6m56.443705201s) [noise=true , address=77.90.31.18:4805 , piece_id=ETDGD5Z4FJVTIP5AXTB4R6ATEQ5YKU6HE4B4YRLAKWNN2CGDMAUA, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.573898686 (573h6m56.444981721s) [noise=true , address=145.79.227.4:4187 , piece_id=O7DY3KACKXB6Z3ZZFMN2W3JWL7R7XUIMZDF3O5TGMBDHA6ASP3HA, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.574956531 (573h6m56.446039566s) [noise=true , address=145.79.227.48:4302 , piece_id=FMNLGFHK77LWP3XIPQOGY4K43UFDCFO3YJQLLENXN7DR4UCWUE4A, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.575308882 (573h6m56.446391917s) [noise=true , address=45.91.116.14:4191 , piece_id=TOACUKBN4S2P5BBEUYA5RQSOLFBLZW2OUGHXTHE54X6BYRQQ2CEQ, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.576339433 (573h6m56.447422468s) [noise=true , address=145.79.227.19:4078 , piece_id=7RMN7V72KNQJ5XQOY6A55FKVBLYRESRMRQDDQQ2CW4W57BPW7EYA, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.576879522 (573h6m56.447962557s) [noise=true , address=45.91.116.4:4937 , piece_id=G5EVO4HSZMQVIYA6JB7MSFL6QN6XCCLP55VTTVZG26OLVPS2KZLQ, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.577692750 (573h6m56.448775795s) [noise=true , address=145.79.227.42:4122 , piece_id=TYWKGHSQS767HBDZLFMLBH6T7WK2VZYJJOL43NWOAYG7MKPASSUQ, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.578460442 (573h6m56.449543477s) [noise=true , address=45.91.116.13:4154 , piece_id=QYH7DSEKKM7F7P6PJ3EUZUIH5BBFTBPW7MZ75ZNVOHWADFNCKURQ, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:58.273077944 (573h6m58.144160979s) [upload=502439]: Next returned nil segment Dec 29 12:35:58.273107380 (573h6m58.144190425s) [upload=502439]: Waiting for error group managing segments... Dec 29 12:35:58.552982213 (573h6m58.424065248s) [noise=true , address=145.79.227.17:4971 , piece_id=3JD35EOECXSTO3BMZ3XTCICMBIRVFZUTPQLKJMUVOCYU72IPI75A, seg_pos={0 0}, upload=502439]: Done uploading piece. err: <nil> Dec 29 12:35:58.553037217 (573h6m58.424120252s) [noise=true , address=145.79.227.5:4222 , piece_id=SEYX67D3UBXQCBDT3XOHRSWGNPBU63GT3ZNSC7XUKXA7NVKVBGXA, seg_pos={0 0}, upload=502439]: Done uploading piece. err: <nil> Dec 29 12:35:58.558023892 (573h6m58.429106927s) [noise=true , address=145.79.227.4:4187 , piece_id=O7DY3KACKXB6Z3ZZFMN2W3JWL7R7XUIMZDF3O5TGMBDHA6ASP3HA, seg_pos={0 0}, upload=502439]: Done uploading piece. err: <nil> Dec 29 12:35:58.558358510 (573h6m58.429441545s) [noise=true , address=145.79.227.36:4518 , piece_id=755BMSS3R3L65EKO3VGQRYJHLEFYCGXT27YVJCAHUJRIEPEPPJVA, seg_pos={0 0}, upload=502439]: Done uploading piece. err: <nil> Dec 29 12:35:58.561949057 (573h6m58.433032092s) [noise=true , address=145.79.227.45:4231 , piece_id=GCOXLBWZNZPQWLOTDQ4U3XWPZR2XMKBIOWZT2VYBG2QLVTRD6SFQ, seg_pos={0 0}, upload=502439]: Done uploading piece. err: <nil> Dec 29 12:35:58.562142565 (573h6m58.4332256s) [noise=true , address=145.79.227.19:4078 , piece_id=7RMN7V72KNQJ5XQOY6A55FKVBLYRESRMRQDDQQ2CW4W57BPW7EYA, seg_pos={0 0}, upload=502439]: Done uploading piece. err: <nil> Dec 29 12:35:58.562620154 (573h6m58.433703189s) [noise=true , address=145.79.227.42:4122 , piece_id=TYWKGHSQS767HBDZLFMLBH6T7WK2VZYJJOL43NWOAYG7MKPASSUQ, seg_pos={0 0}, upload=502439]: Done uploading piece. err: <nil> Dec 29 12:35:58.573164152 (573h6m58.444247197s) [noise=true , address=145.79.227.3:4150 , piece_id=C2SKGSU7DBMKEGNW6SQ7ESNITDHVTNGHE4JVDQJRVDPDE5O5VUDQ, seg_pos={0 0}, upload=502439]: Done uploading piece. err: <nil> Dec 29 12:35:58.574742457 (573h6m58.445825502s) [noise=true , address=145.79.227.52:4446 , piece_id=UHTHU6UCDVXFEMUO57CPRBN45FZ4VSCZMRF4VUHK6Q54ZFWYVGJQ, seg_pos={0 0}, upload=502439]: Done uploading piece. err: <nil> Dec 29 12:35:58.578854627 (573h6m58.449937672s) [noise=true , address=145.79.227.15:4873 , piece_id=HXJ7MUK5NXVDGK475QLSS2SWBBBYDU2E52TIJR5OPLPHZI73IC7A, seg_pos={0 0}, upload=502439]: Done uploading piece. err: <nil> Dec 29 12:35:58.580262918 (573h6m58.451345953s) [noise=true , address=145.79.227.48:4302 , piece_id=FMNLGFHK77LWP3XIPQOGY4K43UFDCFO3YJQLLENXN7DR4UCWUE4A, seg_pos={0 0}, upload=502439]: Done uploading piece. err: <nil> Dec 29 12:35:58.876595519 (573h6m58.747678554s) [noise=true , address=77.90.31.18:4805 , piece_id=ETDGD5Z4FJVTIP5AXTB4R6ATEQ5YKU6HE4B4YRLAKWNN2CGDMAUA, seg_pos={0 0}, upload=502439]: Done uploading piece. err: <nil> Dec 29 12:35:59.876081688 (573h6m59.747164723s) [noise=true , address=45.91.116.14:4191 , piece_id=TOACUKBN4S2P5BBEUYA5RQSOLFBLZW2OUGHXTHE54X6BYRQQ2CEQ, seg_pos={0 0}, upload=502439]: Done uploading piece. err: <nil> Dec 29 12:35:59.876099331 (573h6m59.747182377s) [seg_pos={0 0}, upload=502439]: Segment reached optimal threshold of 13 pieces. Dec 29 12:35:59.876274745 (573h6m59.747357811s) [noise=true , address=45.91.116.12:4225 , piece_id=4U5XB27V6GOJV2G2QZ24ZBPE5HLVAD7PZCLZCA3VCI74F6G4XEXQ, seg_pos={0 0}, upload=502439]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:34228->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:34228->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34228->45.91.116.12:4225: use of closed network connection Dec 29 12:35:59.876285737 (573h6m59.747368782s) [noise=true , address=45.91.116.12:4225 , piece_id=4U5XB27V6GOJV2G2QZ24ZBPE5HLVAD7PZCLZCA3VCI74F6G4XEXQ, seg_pos={0 0}, upload=502439]: Long tail cancellation due to optimal threshold Dec 29 12:35:59.876428148 (573h6m59.747511183s) [noise=true , address=45.91.116.13:4154 , piece_id=QYH7DSEKKM7F7P6PJ3EUZUIH5BBFTBPW7MZ75ZNVOHWADFNCKURQ, seg_pos={0 0}, upload=502439]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 29 12:35:59.876433899 (573h6m59.747516934s) [noise=true , address=45.91.116.13:4154 , piece_id=QYH7DSEKKM7F7P6PJ3EUZUIH5BBFTBPW7MZ75ZNVOHWADFNCKURQ, seg_pos={0 0}, upload=502439]: Long tail cancellation due to optimal threshold Dec 29 12:35:59.876543237 (573h6m59.747626272s) [noise=true , address=45.91.116.4:4937 , piece_id=G5EVO4HSZMQVIYA6JB7MSFL6QN6XCCLP55VTTVZG26OLVPS2KZLQ, seg_pos={0 0}, upload=502439]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: read tcp 188.166.225.170:52370->45.91.116.4:4937: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:35:59.876552224 (573h6m59.747635269s) [noise=true , address=45.91.116.4:4937 , piece_id=G5EVO4HSZMQVIYA6JB7MSFL6QN6XCCLP55VTTVZG26OLVPS2KZLQ, seg_pos={0 0}, upload=502439]: Long tail cancellation due to optimal threshold Dec 29 12:35:59.876615244 (573h6m59.747698279s) [seg_pos={0 0}, upload=502439]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:35:59.876698392 (573h6m59.747781427s) [upload=502439]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:35:59.876703523 (573h6m59.747786558s) [upload=502439]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:36:00.186922785 (573h7m0.05800582s) [upload=502439]: Done uploading segments... Dec 29 12:36:00.186947212 (573h7m0.058030247s) [upload=502439]: Upload finished. err: <nil> Duration: 30.003094903s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................X.......X..........X.........] duration: 45.577635294s trace: https://jaeger.dev.storj.io/trace/68764c6cb4286314 FAIL (2025-12-29 12:36:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528204662 (572h59m20.399287707s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528497489 (572h59m20.399580524s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528793153 (572h59m20.399876198s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529083555 (572h59m20.40016659s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529391081 (572h59m20.400474126s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529735096 (572h59m20.400818131s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530109970 (572h59m20.401193015s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530449556 (572h59m20.401532591s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530831554 (572h59m20.401914599s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531229512 (572h59m20.402312557s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531698945 (572h59m20.402781991s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532204238 (572h59m20.403287283s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532744366 (572h59m20.403827401s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533231203 (572h59m20.404314248s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533740113 (572h59m20.404823148s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:21.876539279 (572h59m21.747622324s) [upload=502322]: Next returned nil segment Dec 29 12:28:21.876614272 (572h59m21.747697307s) [upload=502322]: Waiting for error group managing segments... Dec 29 12:28:22.383674429 (572h59m22.254757464s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389083709 (572h59m22.260166754s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389450968 (572h59m22.260534003s) [noise=true , address=145.79.227.46:4625 , piece_id=6NEND2CKGIZLK5Y6IMM4XB6SXPSXUYSZIVZ7CZ25IKAHFFW33C5Q, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.418373151 (572h59m22.289456186s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.619432477 (572h59m22.490515552s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.741876388 (572h59m22.612959423s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.133731187 (572h59m23.004814222s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.146041929 (572h59m23.017124964s) [noise=true , address=193.149.164.143:4697 , piece_id=HZZY3SUOEC7AIHSW6MKZRNZVSOGXSYBYSUKN7STBFTXWAOMDRYBA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.261354601 (572h59m23.132437636s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.291387197 (572h59m23.162470243s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.318370928 (572h59m23.189453973s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.321236023 (572h59m23.192319058s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.408408673 (572h59m23.279491718s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419570137 (572h59m23.290653172s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419592470 (572h59m23.290675515s) [seg_pos={0 0}, upload=502322]: Segment reached optimal threshold of 14 pieces. Dec 29 12:28:23.419837406 (572h59m23.290920441s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123jUFZTVjq6adDuvt3cco7dTvwNTQkszZX7CeajMfbeCUmEmuJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419845241 (572h59m23.290928276s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.419873174 (572h59m23.290956209s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419882021 (572h59m23.290965076s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420073716 (572h59m23.291156751s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection Dec 29 12:28:23.420082262 (572h59m23.291165307s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420257506 (572h59m23.291340551s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: read tcp 188.166.225.170:41270->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:28:23.420264649 (572h59m23.291347684s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420281822 (572h59m23.291364857s) [seg_pos={0 0}, upload=502322]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:28:23.420364349 (572h59m23.291447394s) [upload=502322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:28:23.420370270 (572h59m23.291453315s) [upload=502322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:28:23.725225770 (572h59m23.596308815s) [upload=502322]: Done uploading segments... Dec 29 12:28:23.725255426 (572h59m23.596338481s) [upload=502322]: Upload finished. err: <nil> Duration: 45.577651515s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 38 out of 60 is failed (threshold=6) [...XXX..XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXXXXXXXXX.] duration: 3.706965248s trace: https://jaeger.dev.storj.io/trace/4fd775f7e1cc83f9 FAIL (2025-12-29 12:35:58 UTC) orbiter warning: context deadline exceeded Duration: 3.706991269s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XXXXXX..X.X.......X.XX...X.......X.X..XXX....X...X..X....XX] duration: 30.002936319s trace: https://jaeger.dev.storj.io/trace/4e35e671575b849 FAIL (2025-12-29 12:35:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: B4BO3I4FPLZ7FTVHJFSVGT6SDG3IUJYP5QIGXZZCAFOI6DNBHPYA) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: IMCJGAJ2ODHJP2FZOSDPHMBEGIG6GJ37L6U5Q7U3IC6PDXBTUCWQ) context deadline exceeded ======LOGS====== Dec 29 12:35:24.044815582 (573h6m23.915898617s) [upload=502429]: Starting upload Dec 29 12:35:24.044886276 (573h6m23.915969311s) [upload=502429]: Done starting upload Dec 29 12:35:24.044948275 (573h6m23.91603131s) [upload=502429]: Uploading segments... Dec 29 12:35:24.045084364 (573h6m23.916167409s) [upload=502429]: Got next segment. Inline: false Dec 29 12:35:24.045121955 (573h6m23.91620499s) [upload=502429]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:35:24.045126454 (573h6m23.916209489s) [upload=502429]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:35:24.287483216 (573h6m24.158566261s) [seg_pos={0 0}, upload=502429]: Begin upload segment... Dec 29 12:35:24.287536046 (573h6m24.158619081s) [seg_pos={0 0}, upload=502429]: Done begin upload segment. Dec 29 12:35:24.287562727 (573h6m24.158645762s) [noise=true , address=77.90.31.13:4627 , piece_id=N7BDRLSYN6TTRHFMTGXUOWIZTILU6YYF5GZEAQILASDFCGB3CXDQ, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.287675902 (573h6m24.158758947s) [noise=true , address=145.79.227.32:4834 , piece_id=WCIVXR772TG6LYLFTHENKG46ZPMTW6IUOGVISZOJ4F7L6RFRRTQA, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.288122433 (573h6m24.159205468s) [noise=true , address=145.79.227.35:4482 , piece_id=FLFKAN6YIDLTZAAH6J4A4AJFDPJMJIUKRM566ZNWWRDGREDLXMMQ, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.288420340 (573h6m24.159503385s) [noise=true , address=145.79.227.49:4338 , piece_id=M3FAA24AAD5SN2WBDRKPWMQLFLRCQBPSUVCSETC3YCIB55EHYCLQ, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.288803800 (573h6m24.159886835s) [noise=true , address=45.91.116.13:4153 , piece_id=IMCJGAJ2ODHJP2FZOSDPHMBEGIG6GJ37L6U5Q7U3IC6PDXBTUCWQ, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.289091979 (573h6m24.160175024s) [noise=true , address=145.79.227.5:4222 , piece_id=DDKILONXB4HFKHCWL26DP6GBBD4A4I3ONZXUARL3RL3L3CFV2NRQ, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.289542066 (573h6m24.160625101s) [noise=true , address=145.79.227.26:4655 , piece_id=65YEWVC7SHYKE2P3PGPZYJDSN7FWAJT56IYA7IFYAGZ5EBCZJU7Q, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.290075622 (573h6m24.161158677s) [noise=true , address=45.91.116.14:4190 , piece_id=B4BO3I4FPLZ7FTVHJFSVGT6SDG3IUJYP5QIGXZZCAFOI6DNBHPYA, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.290435267 (573h6m24.161518312s) [noise=true , address=145.79.227.3:4149 , piece_id=DZIBDRTWGBOXZYMCVMNITJHFZSG27TYDYT4HQTGNBLPKHKL5GJPQ, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.291668425 (573h6m24.16275148s) [noise=true , address=145.79.227.4:4185 , piece_id=E36ZVT2O4M65WONJZRKQRSU7HBAZQNRSDHONXMVEM3L62YGTW3PA, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.292660214 (573h6m24.163743259s) [noise=true , address=145.79.227.42:4121 , piece_id=5WIFOHDUBGAARBHJ2XKG2ETNB45YQ6VEBMAMHM7EKDAAB3PNV4MQ, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.294086660 (573h6m24.165169695s) [noise=true , address=145.79.227.21:4473 , piece_id=WVW7EH3IZGQBQLVCHOHENLZRZP4QGO3ZWHDPNYNAAOYONI44J7CA, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.295434696 (573h6m24.166517731s) [noise=true , address=145.79.227.50:4374 , piece_id=QTNBBE7B3KOVMZG4CFQZWTYBLI4HFAJIZEMLKV7JSG74WNTW7IOQ, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.296744150 (573h6m24.167827195s) [noise=true , address=145.79.227.28:4725 , piece_id=OVRZCIYCFACNX4XQKLOHMTJKP6GHLTELIZCA6JF5Z7RKPXABQCDQ, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.298077228 (573h6m24.169160263s) [noise=true , address=77.90.31.11:4553 , piece_id=M5UP67N6S5A4QAZ7ESCASC3RYZYXKIUMX3CFJKKTOPMSU6ZDUQBA, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.299128450 (573h6m24.170211485s) [noise=true , address=145.79.227.38:4978 , piece_id=P3EBFDHECSKYXUPXMTQA54A7GH3EBM26SMKCU7MJR2MVDAECWVSA, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:25.918110760 (573h6m25.789193806s) [upload=502429]: Next returned nil segment Dec 29 12:35:25.918124516 (573h6m25.789207561s) [upload=502429]: Waiting for error group managing segments... Dec 29 12:35:26.274366734 (573h6m26.145449769s) [noise=true , address=145.79.227.5:4222 , piece_id=DDKILONXB4HFKHCWL26DP6GBBD4A4I3ONZXUARL3RL3L3CFV2NRQ, seg_pos={0 0}, upload=502429]: Done uploading piece. err: <nil> Dec 29 12:35:26.274616219 (573h6m26.145699264s) [noise=true , address=145.79.227.4:4185 , piece_id=E36ZVT2O4M65WONJZRKQRSU7HBAZQNRSDHONXMVEM3L62YGTW3PA, seg_pos={0 0}, upload=502429]: Done uploading piece. err: <nil> Dec 29 12:35:26.278366591 (573h6m26.149449636s) [noise=true , address=145.79.227.3:4149 , piece_id=DZIBDRTWGBOXZYMCVMNITJHFZSG27TYDYT4HQTGNBLPKHKL5GJPQ, seg_pos={0 0}, upload=502429]: Done uploading piece. err: <nil> Dec 29 12:35:26.279706392 (573h6m26.150789427s) [noise=true , address=145.79.227.50:4374 , piece_id=QTNBBE7B3KOVMZG4CFQZWTYBLI4HFAJIZEMLKV7JSG74WNTW7IOQ, seg_pos={0 0}, upload=502429]: Done uploading piece. err: <nil> Dec 29 12:35:26.284638052 (573h6m26.155721087s) [noise=true , address=145.79.227.32:4834 , piece_id=WCIVXR772TG6LYLFTHENKG46ZPMTW6IUOGVISZOJ4F7L6RFRRTQA, seg_pos={0 0}, upload=502429]: Done uploading piece. err: <nil> Dec 29 12:35:26.285181236 (573h6m26.156264291s) [noise=true , address=145.79.227.28:4725 , piece_id=OVRZCIYCFACNX4XQKLOHMTJKP6GHLTELIZCA6JF5Z7RKPXABQCDQ, seg_pos={0 0}, upload=502429]: Done uploading piece. err: <nil> Dec 29 12:35:26.301041737 (573h6m26.172124772s) [noise=true , address=145.79.227.21:4473 , piece_id=WVW7EH3IZGQBQLVCHOHENLZRZP4QGO3ZWHDPNYNAAOYONI44J7CA, seg_pos={0 0}, upload=502429]: Done uploading piece. err: <nil> Dec 29 12:35:26.301536389 (573h6m26.172619434s) [noise=true , address=145.79.227.38:4978 , piece_id=P3EBFDHECSKYXUPXMTQA54A7GH3EBM26SMKCU7MJR2MVDAECWVSA, seg_pos={0 0}, upload=502429]: Done uploading piece. err: <nil> Dec 29 12:35:26.306014897 (573h6m26.177097942s) [noise=true , address=145.79.227.42:4121 , piece_id=5WIFOHDUBGAARBHJ2XKG2ETNB45YQ6VEBMAMHM7EKDAAB3PNV4MQ, seg_pos={0 0}, upload=502429]: Done uploading piece. err: <nil> Dec 29 12:35:26.306836561 (573h6m26.177919596s) [noise=true , address=145.79.227.26:4655 , piece_id=65YEWVC7SHYKE2P3PGPZYJDSN7FWAJT56IYA7IFYAGZ5EBCZJU7Q, seg_pos={0 0}, upload=502429]: Done uploading piece. err: <nil> Dec 29 12:35:26.556961686 (573h6m26.428044721s) [noise=true , address=45.91.116.13:4153 , piece_id=IMCJGAJ2ODHJP2FZOSDPHMBEGIG6GJ37L6U5Q7U3IC6PDXBTUCWQ, seg_pos={0 0}, upload=502429]: Done uploading piece. err: <nil> Dec 29 12:35:26.565925003 (573h6m26.437008038s) [noise=true , address=45.91.116.14:4190 , piece_id=B4BO3I4FPLZ7FTVHJFSVGT6SDG3IUJYP5QIGXZZCAFOI6DNBHPYA, seg_pos={0 0}, upload=502429]: Done uploading piece. err: <nil> Dec 29 12:35:26.610410009 (573h6m26.481493044s) [noise=true , address=77.90.31.13:4627 , piece_id=N7BDRLSYN6TTRHFMTGXUOWIZTILU6YYF5GZEAQILASDFCGB3CXDQ, seg_pos={0 0}, upload=502429]: Done uploading piece. err: <nil> Dec 29 12:35:26.610427303 (573h6m26.481510338s) [seg_pos={0 0}, upload=502429]: Segment reached optimal threshold of 13 pieces. Dec 29 12:35:26.610672970 (573h6m26.481756016s) [noise=true , address=145.79.227.49:4338 , piece_id=M3FAA24AAD5SN2WBDRKPWMQLFLRCQBPSUVCSETC3YCIB55EHYCLQ, seg_pos={0 0}, upload=502429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 29 12:35:26.610683330 (573h6m26.481766365s) [noise=true , address=145.79.227.49:4338 , piece_id=M3FAA24AAD5SN2WBDRKPWMQLFLRCQBPSUVCSETC3YCIB55EHYCLQ, seg_pos={0 0}, upload=502429]: Long tail cancellation due to optimal threshold Dec 29 12:35:26.610952302 (573h6m26.482035337s) [noise=true , address=77.90.31.11:4553 , piece_id=M5UP67N6S5A4QAZ7ESCASC3RYZYXKIUMX3CFJKKTOPMSU6ZDUQBA, seg_pos={0 0}, upload=502429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:35696->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:35696->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35696->77.90.31.11:4553: use of closed network connection Dec 29 12:35:26.610962682 (573h6m26.482045737s) [noise=true , address=77.90.31.11:4553 , piece_id=M5UP67N6S5A4QAZ7ESCASC3RYZYXKIUMX3CFJKKTOPMSU6ZDUQBA, seg_pos={0 0}, upload=502429]: Long tail cancellation due to optimal threshold Dec 29 12:35:26.610982750 (573h6m26.482065785s) [noise=true , address=145.79.227.35:4482 , piece_id=FLFKAN6YIDLTZAAH6J4A4AJFDPJMJIUKRM566ZNWWRDGREDLXMMQ, seg_pos={0 0}, upload=502429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 29 12:35:26.611001165 (573h6m26.4820842s) [noise=true , address=145.79.227.35:4482 , piece_id=FLFKAN6YIDLTZAAH6J4A4AJFDPJMJIUKRM566ZNWWRDGREDLXMMQ, seg_pos={0 0}, upload=502429]: Long tail cancellation due to optimal threshold Dec 29 12:35:26.611060909 (573h6m26.482143944s) [seg_pos={0 0}, upload=502429]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:35:26.611127947 (573h6m26.482210982s) [upload=502429]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:35:26.611155329 (573h6m26.482238364s) [upload=502429]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:35:26.918139471 (573h6m26.789222516s) [upload=502429]: Done uploading segments... Dec 29 12:35:26.918164068 (573h6m26.789247103s) [upload=502429]: Upload finished. err: <nil> Duration: 30.002956148s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................X.......X..........X........] duration: 42.088652307s trace: https://jaeger.dev.storj.io/trace/68764c6cb4286314 FAIL (2025-12-29 12:35:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528204662 (572h59m20.399287707s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528497489 (572h59m20.399580524s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528793153 (572h59m20.399876198s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529083555 (572h59m20.40016659s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529391081 (572h59m20.400474126s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529735096 (572h59m20.400818131s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530109970 (572h59m20.401193015s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530449556 (572h59m20.401532591s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530831554 (572h59m20.401914599s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531229512 (572h59m20.402312557s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531698945 (572h59m20.402781991s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532204238 (572h59m20.403287283s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532744366 (572h59m20.403827401s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533231203 (572h59m20.404314248s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533740113 (572h59m20.404823148s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:21.876539279 (572h59m21.747622324s) [upload=502322]: Next returned nil segment Dec 29 12:28:21.876614272 (572h59m21.747697307s) [upload=502322]: Waiting for error group managing segments... Dec 29 12:28:22.383674429 (572h59m22.254757464s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389083709 (572h59m22.260166754s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389450968 (572h59m22.260534003s) [noise=true , address=145.79.227.46:4625 , piece_id=6NEND2CKGIZLK5Y6IMM4XB6SXPSXUYSZIVZ7CZ25IKAHFFW33C5Q, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.418373151 (572h59m22.289456186s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.619432477 (572h59m22.490515552s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.741876388 (572h59m22.612959423s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.133731187 (572h59m23.004814222s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.146041929 (572h59m23.017124964s) [noise=true , address=193.149.164.143:4697 , piece_id=HZZY3SUOEC7AIHSW6MKZRNZVSOGXSYBYSUKN7STBFTXWAOMDRYBA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.261354601 (572h59m23.132437636s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.291387197 (572h59m23.162470243s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.318370928 (572h59m23.189453973s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.321236023 (572h59m23.192319058s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.408408673 (572h59m23.279491718s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419570137 (572h59m23.290653172s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419592470 (572h59m23.290675515s) [seg_pos={0 0}, upload=502322]: Segment reached optimal threshold of 14 pieces. Dec 29 12:28:23.419837406 (572h59m23.290920441s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123jUFZTVjq6adDuvt3cco7dTvwNTQkszZX7CeajMfbeCUmEmuJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419845241 (572h59m23.290928276s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.419873174 (572h59m23.290956209s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419882021 (572h59m23.290965076s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420073716 (572h59m23.291156751s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection Dec 29 12:28:23.420082262 (572h59m23.291165307s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420257506 (572h59m23.291340551s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: read tcp 188.166.225.170:41270->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:28:23.420264649 (572h59m23.291347684s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420281822 (572h59m23.291364857s) [seg_pos={0 0}, upload=502322]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:28:23.420364349 (572h59m23.291447394s) [upload=502322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:28:23.420370270 (572h59m23.291453315s) [upload=502322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:28:23.725225770 (572h59m23.596308815s) [upload=502322]: Done uploading segments... Dec 29 12:28:23.725255426 (572h59m23.596338481s) [upload=502322]: Upload finished. err: <nil> Duration: 42.088669319s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..XXXXXX..X.X.......X.XX...X.......X.X..XXX....X...X..X....X] duration: 30.002823037s trace: https://jaeger.dev.storj.io/trace/29290dcdebfdabb FAIL (2025-12-29 12:35:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: OAXXRPE2HCSYT6ECJJN3QCTW7RX6RQZGMDAMZDYAR7PDQ4IBPYJA) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: CXJEXDSXFCZBBBF447GL6INCXX5VX6M2EXGMFW64XQBJWDTGGUHQ) context deadline exceeded ======LOGS====== m42.937291868s) [upload=502419]: Uploading segments... Dec 29 12:34:43.066323461 (573h5m42.937406496s) [upload=502419]: Got next segment. Inline: false Dec 29 12:34:43.066369318 (573h5m42.937452353s) [upload=502419]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:34:43.066373927 (573h5m42.937456962s) [upload=502419]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:34:43.325012811 (573h5m43.196095856s) [seg_pos={0 0}, upload=502419]: Begin upload segment... Dec 29 12:34:43.325067394 (573h5m43.196150429s) [seg_pos={0 0}, upload=502419]: Done begin upload segment. Dec 29 12:34:43.325084688 (573h5m43.196167723s) [noise=true , address=145.79.227.20:4437 , piece_id=P4ORQRTHDIZT3MKDLIKJJNJKNTZBOIKN4BGB46B5F2WI6GR4XIUA, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.325097221 (573h5m43.196180256s) [noise=true , address=77.90.31.14:4661 , piece_id=OAXXRPE2HCSYT6ECJJN3QCTW7RX6RQZGMDAMZDYAR7PDQ4IBPYJA, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.325255443 (573h5m43.196338478s) [noise=true , address=145.79.227.47:4267 , piece_id=IDV6KZCBTCKY2JM2U5B2GYRFSDGQDIK33JJFAMWCKUFYBPZIRTSA, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.325549322 (573h5m43.196632357s) [noise=true , address=145.79.227.21:4473 , piece_id=UYGLGKEX7MXSPSVZ5SQU42WLSHIU2FPI4HHJUK63EYTYRI2KTZ6A, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.326029567 (573h5m43.197112602s) [noise=true , address=45.91.116.5:4297 , piece_id=5FADDZAHAYDWDNYGXH6IS2VHLLBE4UVWUD4DCCAIJDMNTAIVSM7Q, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.326298168 (573h5m43.197381204s) [noise=true , address=77.90.31.21:4878 , piece_id=5NZHBG4LBZU74LGWGEAT5IOF37EMVDQAE7ZVPZSWD2E3IRMV4G7A, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.326545079 (573h5m43.197628114s) [noise=true , address=45.91.116.12:4226 , piece_id=JMU7N3SBOBGTKSP35UQF3EUC3OXGN4F4CZHUBBFLJ4HPER7BYSDQ, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.326827136 (573h5m43.197910171s) [noise=true , address=145.79.227.41:4086 , piece_id=BBE4OMYOUHHRA5LKZ4ACYELN3AUVPDIA7PKBSTESTWN3YO2GDUFQ, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.327137436 (573h5m43.198220502s) [noise=true , address=145.79.227.5:4221 , piece_id=S43XISH5LCQDSKRSRHNBX4I7WM2IRYYSPWXLKPJVAFQKOEL2776A, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.327510146 (573h5m43.198593182s) [noise=true , address=145.79.227.3:4149 , piece_id=4PCA6YHFKOTB5P5X4YWFELQ4W3OUHWCSDYDB5FBO2NKBZIIWLDEQ, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.327834744 (573h5m43.198917779s) [noise=true , address=145.79.227.38:4977 , piece_id=7KLAXLYXGYNXBMTVHAGRP2XZS7MF2UJ42367L6CHWZIAXBQANNZA, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.328725660 (573h5m43.199808695s) [noise=true , address=77.90.31.16:4735 , piece_id=CXJEXDSXFCZBBBF447GL6INCXX5VX6M2EXGMFW64XQBJWDTGGUHQ, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.328838596 (573h5m43.199921641s) [noise=true , address=145.79.227.16:4909 , piece_id=U37QA2QKVMAE4P55CVYHXL2XEM3P3IFM2X24J25PL3WDDLVRXCWA, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.329920797 (573h5m43.201003832s) [noise=true , address=145.79.227.39:4013 , piece_id=DXWQD2L57XBL2QNPN2BV3YXEAY7JQJJMFQXCLKPMB2RKZ74JJYTQ, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.331052321 (573h5m43.202135356s) [noise=true , address=45.91.116.4:4940 , piece_id=TWLYKO7OVPB56J5QBOSDFXHCVIWBD7FLILLYN5XX35BUV5F2HUQA, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.332046695 (573h5m43.20312973s) [noise=true , address=145.79.227.26:4654 , piece_id=A7BOQWYJQRIMLU7UAQ46NAWRWQ3SIYAV3YQIHK5DYAD4VGRDBY2A, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:44.936900204 (573h5m44.80798324s) [upload=502419]: Next returned nil segment Dec 29 12:34:44.936914050 (573h5m44.807997086s) [upload=502419]: Waiting for error group managing segments... Dec 29 12:34:45.313615295 (573h5m45.18469833s) [noise=true , address=145.79.227.21:4473 , piece_id=UYGLGKEX7MXSPSVZ5SQU42WLSHIU2FPI4HHJUK63EYTYRI2KTZ6A, seg_pos={0 0}, upload=502419]: Done uploading piece. err: <nil> Dec 29 12:34:45.314226138 (573h5m45.185309183s) [noise=true , address=145.79.227.20:4437 , piece_id=P4ORQRTHDIZT3MKDLIKJJNJKNTZBOIKN4BGB46B5F2WI6GR4XIUA, seg_pos={0 0}, upload=502419]: Done uploading piece. err: <nil> Dec 29 12:34:45.315203449 (573h5m45.186286484s) [noise=true , address=145.79.227.41:4086 , piece_id=BBE4OMYOUHHRA5LKZ4ACYELN3AUVPDIA7PKBSTESTWN3YO2GDUFQ, seg_pos={0 0}, upload=502419]: Done uploading piece. err: <nil> Dec 29 12:34:45.317324057 (573h5m45.188407102s) [noise=true , address=145.79.227.5:4221 , piece_id=S43XISH5LCQDSKRSRHNBX4I7WM2IRYYSPWXLKPJVAFQKOEL2776A, seg_pos={0 0}, upload=502419]: Done uploading piece. err: <nil> Dec 29 12:34:45.318854060 (573h5m45.189937105s) [noise=true , address=145.79.227.16:4909 , piece_id=U37QA2QKVMAE4P55CVYHXL2XEM3P3IFM2X24J25PL3WDDLVRXCWA, seg_pos={0 0}, upload=502419]: Done uploading piece. err: <nil> Dec 29 12:34:45.319441118 (573h5m45.190524153s) [noise=true , address=145.79.227.3:4149 , piece_id=4PCA6YHFKOTB5P5X4YWFELQ4W3OUHWCSDYDB5FBO2NKBZIIWLDEQ, seg_pos={0 0}, upload=502419]: Done uploading piece. err: <nil> Dec 29 12:34:45.320131413 (573h5m45.191214458s) [noise=true , address=145.79.227.38:4977 , piece_id=7KLAXLYXGYNXBMTVHAGRP2XZS7MF2UJ42367L6CHWZIAXBQANNZA, seg_pos={0 0}, upload=502419]: Done uploading piece. err: <nil> Dec 29 12:34:45.325024109 (573h5m45.196107144s) [noise=true , address=145.79.227.26:4654 , piece_id=A7BOQWYJQRIMLU7UAQ46NAWRWQ3SIYAV3YQIHK5DYAD4VGRDBY2A, seg_pos={0 0}, upload=502419]: Done uploading piece. err: <nil> Dec 29 12:34:45.327497859 (573h5m45.198580894s) [noise=true , address=145.79.227.47:4267 , piece_id=IDV6KZCBTCKY2JM2U5B2GYRFSDGQDIK33JJFAMWCKUFYBPZIRTSA, seg_pos={0 0}, upload=502419]: Done uploading piece. err: <nil> Dec 29 12:34:45.346052220 (573h5m45.217135255s) [noise=true , address=145.79.227.39:4013 , piece_id=DXWQD2L57XBL2QNPN2BV3YXEAY7JQJJMFQXCLKPMB2RKZ74JJYTQ, seg_pos={0 0}, upload=502419]: Done uploading piece. err: <nil> Dec 29 12:34:45.521418984 (573h5m45.392502029s) [noise=true , address=77.90.31.21:4878 , piece_id=5NZHBG4LBZU74LGWGEAT5IOF37EMVDQAE7ZVPZSWD2E3IRMV4G7A, seg_pos={0 0}, upload=502419]: Done uploading piece. err: <nil> Dec 29 12:34:45.620408424 (573h5m45.491491469s) [noise=true , address=77.90.31.16:4735 , piece_id=CXJEXDSXFCZBBBF447GL6INCXX5VX6M2EXGMFW64XQBJWDTGGUHQ, seg_pos={0 0}, upload=502419]: Done uploading piece. err: <nil> Dec 29 12:34:45.645376457 (573h5m45.516459502s) [noise=true , address=77.90.31.14:4661 , piece_id=OAXXRPE2HCSYT6ECJJN3QCTW7RX6RQZGMDAMZDYAR7PDQ4IBPYJA, seg_pos={0 0}, upload=502419]: Done uploading piece. err: <nil> Dec 29 12:34:45.645398809 (573h5m45.516481854s) [seg_pos={0 0}, upload=502419]: Segment reached optimal threshold of 13 pieces. Dec 29 12:34:45.645677029 (573h5m45.516760064s) [noise=true , address=45.91.116.12:4226 , piece_id=JMU7N3SBOBGTKSP35UQF3EUC3OXGN4F4CZHUBBFLJ4HPER7BYSDQ, seg_pos={0 0}, upload=502419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:41234->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:41234->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41234->45.91.116.12:4226: use of closed network connection Dec 29 12:34:45.645704241 (573h5m45.516787276s) [noise=true , address=45.91.116.12:4226 , piece_id=JMU7N3SBOBGTKSP35UQF3EUC3OXGN4F4CZHUBBFLJ4HPER7BYSDQ, seg_pos={0 0}, upload=502419]: Long tail cancellation due to optimal threshold Dec 29 12:34:45.645895356 (573h5m45.516978391s) [noise=true , address=45.91.116.4:4940 , piece_id=TWLYKO7OVPB56J5QBOSDFXHCVIWBD7FLILLYN5XX35BUV5F2HUQA, seg_pos={0 0}, upload=502419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:42122->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:42122->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42122->45.91.116.4:4940: use of closed network connection Dec 29 12:34:45.645924140 (573h5m45.517007185s) [noise=true , address=45.91.116.4:4940 , piece_id=TWLYKO7OVPB56J5QBOSDFXHCVIWBD7FLILLYN5XX35BUV5F2HUQA, seg_pos={0 0}, upload=502419]: Long tail cancellation due to optimal threshold Dec 29 12:34:45.645956301 (573h5m45.517039336s) [noise=true , address=45.91.116.5:4297 , piece_id=5FADDZAHAYDWDNYGXH6IS2VHLLBE4UVWUD4DCCAIJDMNTAIVSM7Q, seg_pos={0 0}, upload=502419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 29 12:34:45.645986929 (573h5m45.517069964s) [noise=true , address=45.91.116.5:4297 , piece_id=5FADDZAHAYDWDNYGXH6IS2VHLLBE4UVWUD4DCCAIJDMNTAIVSM7Q, seg_pos={0 0}, upload=502419]: Long tail cancellation due to optimal threshold Dec 29 12:34:45.646010334 (573h5m45.517093379s) [seg_pos={0 0}, upload=502419]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:34:45.646113591 (573h5m45.517196626s) [upload=502419]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:34:45.646118430 (573h5m45.517201465s) [upload=502419]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:34:45.935447816 (573h5m45.806530861s) [upload=502419]: Done uploading segments... Dec 29 12:34:45.935475378 (573h5m45.806558413s) [upload=502419]: Upload finished. err: <nil> Duration: 30.00284547s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 39 out of 60 is failed (threshold=6) [X...XXX..XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXXXXXXXXX] duration: 5.001603875s trace: https://jaeger.dev.storj.io/trace/4fd775f7e1cc83f9 FAIL (2025-12-29 12:35:02 UTC) orbiter warning: context deadline exceeded Duration: 5.001630946s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................X.......X..........X.......] duration: 44.805290832s trace: https://jaeger.dev.storj.io/trace/68764c6cb4286314 FAIL (2025-12-29 12:34:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528204662 (572h59m20.399287707s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528497489 (572h59m20.399580524s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528793153 (572h59m20.399876198s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529083555 (572h59m20.40016659s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529391081 (572h59m20.400474126s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529735096 (572h59m20.400818131s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530109970 (572h59m20.401193015s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530449556 (572h59m20.401532591s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530831554 (572h59m20.401914599s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531229512 (572h59m20.402312557s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531698945 (572h59m20.402781991s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532204238 (572h59m20.403287283s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532744366 (572h59m20.403827401s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533231203 (572h59m20.404314248s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533740113 (572h59m20.404823148s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:21.876539279 (572h59m21.747622324s) [upload=502322]: Next returned nil segment Dec 29 12:28:21.876614272 (572h59m21.747697307s) [upload=502322]: Waiting for error group managing segments... Dec 29 12:28:22.383674429 (572h59m22.254757464s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389083709 (572h59m22.260166754s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389450968 (572h59m22.260534003s) [noise=true , address=145.79.227.46:4625 , piece_id=6NEND2CKGIZLK5Y6IMM4XB6SXPSXUYSZIVZ7CZ25IKAHFFW33C5Q, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.418373151 (572h59m22.289456186s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.619432477 (572h59m22.490515552s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.741876388 (572h59m22.612959423s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.133731187 (572h59m23.004814222s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.146041929 (572h59m23.017124964s) [noise=true , address=193.149.164.143:4697 , piece_id=HZZY3SUOEC7AIHSW6MKZRNZVSOGXSYBYSUKN7STBFTXWAOMDRYBA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.261354601 (572h59m23.132437636s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.291387197 (572h59m23.162470243s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.318370928 (572h59m23.189453973s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.321236023 (572h59m23.192319058s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.408408673 (572h59m23.279491718s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419570137 (572h59m23.290653172s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419592470 (572h59m23.290675515s) [seg_pos={0 0}, upload=502322]: Segment reached optimal threshold of 14 pieces. Dec 29 12:28:23.419837406 (572h59m23.290920441s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123jUFZTVjq6adDuvt3cco7dTvwNTQkszZX7CeajMfbeCUmEmuJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419845241 (572h59m23.290928276s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.419873174 (572h59m23.290956209s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419882021 (572h59m23.290965076s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420073716 (572h59m23.291156751s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection Dec 29 12:28:23.420082262 (572h59m23.291165307s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420257506 (572h59m23.291340551s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: read tcp 188.166.225.170:41270->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:28:23.420264649 (572h59m23.291347684s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420281822 (572h59m23.291364857s) [seg_pos={0 0}, upload=502322]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:28:23.420364349 (572h59m23.291447394s) [upload=502322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:28:23.420370270 (572h59m23.291453315s) [upload=502322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:28:23.725225770 (572h59m23.596308815s) [upload=502322]: Done uploading segments... Dec 29 12:28:23.725255426 (572h59m23.596338481s) [upload=502322]: Upload finished. err: <nil> Duration: 44.805307845s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [...XXXXXX..X.X.......X.XX...X.......X.X..XXX....X...X..X....] duration: 6.012061916s trace: https://jaeger.dev.storj.io/trace/55af87306b44a0fb FAIL (2025-12-29 12:34:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 12:30:27.976557996 (573h1m27.847641031s) [upload=502356]: Starting upload Dec 29 12:30:27.976643769 (573h1m27.847726804s) [upload=502356]: Done starting upload Dec 29 12:30:27.976805467 (573h1m27.847888512s) [upload=502356]: Uploading segments... Dec 29 12:30:27.976959821 (573h1m27.848042856s) [upload=502356]: Got next segment. Inline: false Dec 29 12:30:27.977030205 (573h1m27.84811324s) [upload=502356]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:30:27.977037499 (573h1m27.848120544s) [upload=502356]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:30:28.227612220 (573h1m28.098695265s) [seg_pos={0 0}, upload=502356]: Begin upload segment... Dec 29 12:30:28.227680119 (573h1m28.098763154s) [seg_pos={0 0}, upload=502356]: Done begin upload segment. Dec 29 12:30:28.227731227 (573h1m28.098814272s) [noise=true , address=145.79.227.2:4115 , piece_id=FYK6H2CTV3YRNPU3TW5ICDUUHNJLUGYB7YVL242N7EKXA3OFMUHQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.227777565 (573h1m28.0988606s) [noise=true , address=45.91.116.14:4190 , piece_id=SVSZXQDV6WXOB5BID3HVGTAPTKSOJ2PXSGGW3OIQ5HEZRGVRG6DA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.228331690 (573h1m28.099414725s) [noise=true , address=45.91.116.12:4227 , piece_id=34PI32NCVIDSE2W6UAATADRFY2ZKVIS5DOXTHGL2TWDNUYZK5I4A, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.228647301 (573h1m28.099730336s) [noise=true , address=145.79.227.10:4401 , piece_id=TCQXBURRVQY5VJBL6CVM56AVQ23IALWXMJYCXI2FQNK5VEA7ZP6A, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.228978120 (573h1m28.100061155s) [noise=true , address=145.79.227.41:4085 , piece_id=MZYJNEXIKOQVVEZPIEZ33W6XJWKMY3LLYKVUW5PNSRHCD7Y6AZQA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.229164997 (573h1m28.100248032s) [noise=true , address=145.79.227.46:4601 , piece_id=C3VY24KTEOXUNLPAVDT45J2LRQPQ6AOR2OKTEJGLARZOUGOS64YQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.229770089 (573h1m28.100853124s) [noise=true , address=145.79.227.32:4835 , piece_id=GI4T5K7YGHDDF75UUH722QEZYSYU3DRXS4HPYWLLADM6J2EHNR5A, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.230196019 (573h1m28.101279054s) [noise=true , address=145.79.227.48:4301 , piece_id=VEJVLLQU2KDKYM3Y5XZB6TNUUXSX2MDC6Y2KXN6RUEWASB7G6TAQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.230229163 (573h1m28.101312198s) [noise=true , address=145.79.227.45:4231 , piece_id=AI4FTKKS7SWUMAQ35E4ML7TT3O2CGOZHAA5FQPIM4L2TSDOV3TSQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.230721130 (573h1m28.101804165s) [noise=true , address=77.90.31.16:4735 , piece_id=LED3MXVURDX5ITUWGZGJHXM2LXAX3ISSJKAECUJUOABOLUHEAKKA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.230975274 (573h1m28.102058309s) [noise=true , address=145.79.227.13:4802 , piece_id=MR6H5CX267HPWCETWENONX5QMETKXN3SL5LKWYNBNDEMBWR46ZAA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.231314189 (573h1m28.102397235s) [noise=true , address=45.91.116.10:4083 , piece_id=AYQXAYPZMWZN2YSVCNUENMW4KCE6BFBEBV2N57BRRDLNYL5ZCLTA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.231509651 (573h1m28.102592686s) [noise=true , address=145.79.227.26:4654 , piece_id=CJJW6NM2HL572SDZSTGZID6S6PRCFPS6DEJMQU4WN3DX2GMMLEHQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.231966481 (573h1m28.103049526s) [noise=true , address=45.91.116.6:4974 , piece_id=NCAE2UNBQ7IAI2B2EE5QKMIJQZ5NPAMBGEYODLFMERSD5IVYUUHA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.232489376 (573h1m28.103572411s) [noise=true , address=145.79.227.7:4293 , piece_id=6FYYAD6K52LDUNKOGFFAGNLPFHKA7V3WVBRPKHOZBZXMHRHIACYA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.232858590 (573h1m28.103941625s) [noise=true , address=45.91.116.5:4297 , piece_id=63ABPGW4YSCEPCPMQHFZ6NZVWNYJAELGJNXJJXWFTHVTW6YIVAQA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:29.934302048 (573h1m29.805385083s) [upload=502356]: Next returned nil segment Dec 29 12:30:29.934324049 (573h1m29.805407094s) [upload=502356]: Waiting for error group managing segments... Dec 29 12:30:30.219330104 (573h1m30.090413149s) [noise=true , address=145.79.227.2:4115 , piece_id=FYK6H2CTV3YRNPU3TW5ICDUUHNJLUGYB7YVL242N7EKXA3OFMUHQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.222585522 (573h1m30.093668557s) [noise=true , address=145.79.227.26:4654 , piece_id=CJJW6NM2HL572SDZSTGZID6S6PRCFPS6DEJMQU4WN3DX2GMMLEHQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.223473883 (573h1m30.094556928s) [noise=true , address=145.79.227.7:4293 , piece_id=6FYYAD6K52LDUNKOGFFAGNLPFHKA7V3WVBRPKHOZBZXMHRHIACYA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.225048181 (573h1m30.096131226s) [noise=true , address=145.79.227.32:4835 , piece_id=GI4T5K7YGHDDF75UUH722QEZYSYU3DRXS4HPYWLLADM6J2EHNR5A, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.227668892 (573h1m30.098751937s) [noise=true , address=145.79.227.10:4401 , piece_id=TCQXBURRVQY5VJBL6CVM56AVQ23IALWXMJYCXI2FQNK5VEA7ZP6A, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.228574516 (573h1m30.099657561s) [noise=true , address=145.79.227.13:4802 , piece_id=MR6H5CX267HPWCETWENONX5QMETKXN3SL5LKWYNBNDEMBWR46ZAA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.234122320 (573h1m30.105205355s) [noise=true , address=145.79.227.45:4231 , piece_id=AI4FTKKS7SWUMAQ35E4ML7TT3O2CGOZHAA5FQPIM4L2TSDOV3TSQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.240891529 (573h1m30.111974574s) [noise=true , address=145.79.227.48:4301 , piece_id=VEJVLLQU2KDKYM3Y5XZB6TNUUXSX2MDC6Y2KXN6RUEWASB7G6TAQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.241789950 (573h1m30.112872985s) [noise=true , address=145.79.227.46:4601 , piece_id=C3VY24KTEOXUNLPAVDT45J2LRQPQ6AOR2OKTEJGLARZOUGOS64YQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.247350498 (573h1m30.118433534s) [noise=true , address=145.79.227.41:4085 , piece_id=MZYJNEXIKOQVVEZPIEZ33W6XJWKMY3LLYKVUW5PNSRHCD7Y6AZQA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.417471593 (573h1m30.288554628s) [noise=true , address=77.90.31.16:4735 , piece_id=LED3MXVURDX5ITUWGZGJHXM2LXAX3ISSJKAECUJUOABOLUHEAKKA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.432824027 (573h1m30.303907062s) [noise=true , address=45.91.116.10:4083 , piece_id=AYQXAYPZMWZN2YSVCNUENMW4KCE6BFBEBV2N57BRRDLNYL5ZCLTA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.583724624 (573h1m30.454807659s) [noise=true , address=45.91.116.14:4190 , piece_id=SVSZXQDV6WXOB5BID3HVGTAPTKSOJ2PXSGGW3OIQ5HEZRGVRG6DA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.583743831 (573h1m30.454826876s) [seg_pos={0 0}, upload=502356]: Segment reached optimal threshold of 13 pieces. Dec 29 12:30:30.583922962 (573h1m30.455005997s) [noise=true , address=45.91.116.5:4297 , piece_id=63ABPGW4YSCEPCPMQHFZ6NZVWNYJAELGJNXJJXWFTHVTW6YIVAQA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 29 12:30:30.583929113 (573h1m30.455012149s) [noise=true , address=45.91.116.5:4297 , piece_id=63ABPGW4YSCEPCPMQHFZ6NZVWNYJAELGJNXJJXWFTHVTW6YIVAQA, seg_pos={0 0}, upload=502356]: Long tail cancellation due to optimal threshold Dec 29 12:30:30.584091974 (573h1m30.455175009s) [noise=true , address=45.91.116.6:4974 , piece_id=NCAE2UNBQ7IAI2B2EE5QKMIJQZ5NPAMBGEYODLFMERSD5IVYUUHA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:30:30.584108805 (573h1m30.455191841s) [noise=true , address=45.91.116.6:4974 , piece_id=NCAE2UNBQ7IAI2B2EE5QKMIJQZ5NPAMBGEYODLFMERSD5IVYUUHA, seg_pos={0 0}, upload=502356]: Long tail cancellation due to optimal threshold Dec 29 12:30:30.584341749 (573h1m30.455424794s) [noise=true , address=45.91.116.12:4227 , piece_id=34PI32NCVIDSE2W6UAATADRFY2ZKVIS5DOXTHGL2TWDNUYZK5I4A, seg_pos={0 0}, upload=502356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: context canceled; context canceled Dec 29 12:30:30.584349564 (573h1m30.455432609s) [noise=true , address=45.91.116.12:4227 , piece_id=34PI32NCVIDSE2W6UAATADRFY2ZKVIS5DOXTHGL2TWDNUYZK5I4A, seg_pos={0 0}, upload=502356]: Long tail cancellation due to optimal threshold Dec 29 12:30:30.584369573 (573h1m30.455452608s) [seg_pos={0 0}, upload=502356]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:30:30.584449214 (573h1m30.455532259s) [upload=502356]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:30:30.584453732 (573h1m30.455536767s) [upload=502356]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:30:30.886256078 (573h1m30.757339123s) [upload=502356]: Done uploading segments... Dec 29 12:30:30.886281006 (573h1m30.757364041s) [upload=502356]: Upload finished. err: <nil> Duration: 6.01210074s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 39 out of 60 is failed (threshold=6) [XX...XXX..XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXXXXXXXX] duration: 5.001439456s trace: https://jaeger.dev.storj.io/trace/810ca148700c57 FAIL (2025-12-29 12:34:13 UTC) orbiter warning: context deadline exceeded Duration: 5.001461187s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................X.......X..........X......] duration: 42.908403656s trace: https://jaeger.dev.storj.io/trace/68764c6cb4286314 FAIL (2025-12-29 12:34:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528204662 (572h59m20.399287707s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528497489 (572h59m20.399580524s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528793153 (572h59m20.399876198s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529083555 (572h59m20.40016659s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529391081 (572h59m20.400474126s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529735096 (572h59m20.400818131s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530109970 (572h59m20.401193015s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530449556 (572h59m20.401532591s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530831554 (572h59m20.401914599s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531229512 (572h59m20.402312557s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531698945 (572h59m20.402781991s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532204238 (572h59m20.403287283s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532744366 (572h59m20.403827401s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533231203 (572h59m20.404314248s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533740113 (572h59m20.404823148s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:21.876539279 (572h59m21.747622324s) [upload=502322]: Next returned nil segment Dec 29 12:28:21.876614272 (572h59m21.747697307s) [upload=502322]: Waiting for error group managing segments... Dec 29 12:28:22.383674429 (572h59m22.254757464s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389083709 (572h59m22.260166754s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389450968 (572h59m22.260534003s) [noise=true , address=145.79.227.46:4625 , piece_id=6NEND2CKGIZLK5Y6IMM4XB6SXPSXUYSZIVZ7CZ25IKAHFFW33C5Q, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.418373151 (572h59m22.289456186s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.619432477 (572h59m22.490515552s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.741876388 (572h59m22.612959423s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.133731187 (572h59m23.004814222s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.146041929 (572h59m23.017124964s) [noise=true , address=193.149.164.143:4697 , piece_id=HZZY3SUOEC7AIHSW6MKZRNZVSOGXSYBYSUKN7STBFTXWAOMDRYBA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.261354601 (572h59m23.132437636s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.291387197 (572h59m23.162470243s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.318370928 (572h59m23.189453973s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.321236023 (572h59m23.192319058s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.408408673 (572h59m23.279491718s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419570137 (572h59m23.290653172s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419592470 (572h59m23.290675515s) [seg_pos={0 0}, upload=502322]: Segment reached optimal threshold of 14 pieces. Dec 29 12:28:23.419837406 (572h59m23.290920441s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123jUFZTVjq6adDuvt3cco7dTvwNTQkszZX7CeajMfbeCUmEmuJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419845241 (572h59m23.290928276s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.419873174 (572h59m23.290956209s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419882021 (572h59m23.290965076s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420073716 (572h59m23.291156751s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection Dec 29 12:28:23.420082262 (572h59m23.291165307s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420257506 (572h59m23.291340551s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: read tcp 188.166.225.170:41270->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:28:23.420264649 (572h59m23.291347684s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420281822 (572h59m23.291364857s) [seg_pos={0 0}, upload=502322]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:28:23.420364349 (572h59m23.291447394s) [upload=502322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:28:23.420370270 (572h59m23.291453315s) [upload=502322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:28:23.725225770 (572h59m23.596308815s) [upload=502322]: Done uploading segments... Dec 29 12:28:23.725255426 (572h59m23.596338481s) [upload=502322]: Upload finished. err: <nil> Duration: 42.908433502s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 39 out of 60 is failed (threshold=6) [XXX...XXX..XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXXXXXXX] duration: 5.00182055s trace: https://jaeger.dev.storj.io/trace/1adb64ac4b59522 FAIL (2025-12-29 12:33:47 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001850756s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................X.......X..........X.....] duration: 42.12777906s trace: https://jaeger.dev.storj.io/trace/68764c6cb4286314 FAIL (2025-12-29 12:33:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528204662 (572h59m20.399287707s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528497489 (572h59m20.399580524s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528793153 (572h59m20.399876198s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529083555 (572h59m20.40016659s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529391081 (572h59m20.400474126s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529735096 (572h59m20.400818131s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530109970 (572h59m20.401193015s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530449556 (572h59m20.401532591s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530831554 (572h59m20.401914599s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531229512 (572h59m20.402312557s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531698945 (572h59m20.402781991s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532204238 (572h59m20.403287283s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532744366 (572h59m20.403827401s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533231203 (572h59m20.404314248s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533740113 (572h59m20.404823148s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:21.876539279 (572h59m21.747622324s) [upload=502322]: Next returned nil segment Dec 29 12:28:21.876614272 (572h59m21.747697307s) [upload=502322]: Waiting for error group managing segments... Dec 29 12:28:22.383674429 (572h59m22.254757464s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389083709 (572h59m22.260166754s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389450968 (572h59m22.260534003s) [noise=true , address=145.79.227.46:4625 , piece_id=6NEND2CKGIZLK5Y6IMM4XB6SXPSXUYSZIVZ7CZ25IKAHFFW33C5Q, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.418373151 (572h59m22.289456186s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.619432477 (572h59m22.490515552s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.741876388 (572h59m22.612959423s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.133731187 (572h59m23.004814222s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.146041929 (572h59m23.017124964s) [noise=true , address=193.149.164.143:4697 , piece_id=HZZY3SUOEC7AIHSW6MKZRNZVSOGXSYBYSUKN7STBFTXWAOMDRYBA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.261354601 (572h59m23.132437636s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.291387197 (572h59m23.162470243s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.318370928 (572h59m23.189453973s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.321236023 (572h59m23.192319058s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.408408673 (572h59m23.279491718s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419570137 (572h59m23.290653172s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419592470 (572h59m23.290675515s) [seg_pos={0 0}, upload=502322]: Segment reached optimal threshold of 14 pieces. Dec 29 12:28:23.419837406 (572h59m23.290920441s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123jUFZTVjq6adDuvt3cco7dTvwNTQkszZX7CeajMfbeCUmEmuJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419845241 (572h59m23.290928276s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.419873174 (572h59m23.290956209s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419882021 (572h59m23.290965076s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420073716 (572h59m23.291156751s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection Dec 29 12:28:23.420082262 (572h59m23.291165307s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420257506 (572h59m23.291340551s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: read tcp 188.166.225.170:41270->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:28:23.420264649 (572h59m23.291347684s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420281822 (572h59m23.291364857s) [seg_pos={0 0}, upload=502322]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:28:23.420364349 (572h59m23.291447394s) [upload=502322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:28:23.420370270 (572h59m23.291453315s) [upload=502322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:28:23.725225770 (572h59m23.596308815s) [upload=502322]: Done uploading segments... Dec 29 12:28:23.725255426 (572h59m23.596338481s) [upload=502322]: Upload finished. err: <nil> Duration: 42.127792847s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [....XXXXXX..X.X.......X.XX...X.......X.X..XXX....X...X..X...] duration: 5.949110859s trace: https://jaeger.dev.storj.io/trace/55af87306b44a0fb FAIL (2025-12-29 12:33:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 12:30:27.976557996 (573h1m27.847641031s) [upload=502356]: Starting upload Dec 29 12:30:27.976643769 (573h1m27.847726804s) [upload=502356]: Done starting upload Dec 29 12:30:27.976805467 (573h1m27.847888512s) [upload=502356]: Uploading segments... Dec 29 12:30:27.976959821 (573h1m27.848042856s) [upload=502356]: Got next segment. Inline: false Dec 29 12:30:27.977030205 (573h1m27.84811324s) [upload=502356]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:30:27.977037499 (573h1m27.848120544s) [upload=502356]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:30:28.227612220 (573h1m28.098695265s) [seg_pos={0 0}, upload=502356]: Begin upload segment... Dec 29 12:30:28.227680119 (573h1m28.098763154s) [seg_pos={0 0}, upload=502356]: Done begin upload segment. Dec 29 12:30:28.227731227 (573h1m28.098814272s) [noise=true , address=145.79.227.2:4115 , piece_id=FYK6H2CTV3YRNPU3TW5ICDUUHNJLUGYB7YVL242N7EKXA3OFMUHQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.227777565 (573h1m28.0988606s) [noise=true , address=45.91.116.14:4190 , piece_id=SVSZXQDV6WXOB5BID3HVGTAPTKSOJ2PXSGGW3OIQ5HEZRGVRG6DA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.228331690 (573h1m28.099414725s) [noise=true , address=45.91.116.12:4227 , piece_id=34PI32NCVIDSE2W6UAATADRFY2ZKVIS5DOXTHGL2TWDNUYZK5I4A, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.228647301 (573h1m28.099730336s) [noise=true , address=145.79.227.10:4401 , piece_id=TCQXBURRVQY5VJBL6CVM56AVQ23IALWXMJYCXI2FQNK5VEA7ZP6A, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.228978120 (573h1m28.100061155s) [noise=true , address=145.79.227.41:4085 , piece_id=MZYJNEXIKOQVVEZPIEZ33W6XJWKMY3LLYKVUW5PNSRHCD7Y6AZQA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.229164997 (573h1m28.100248032s) [noise=true , address=145.79.227.46:4601 , piece_id=C3VY24KTEOXUNLPAVDT45J2LRQPQ6AOR2OKTEJGLARZOUGOS64YQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.229770089 (573h1m28.100853124s) [noise=true , address=145.79.227.32:4835 , piece_id=GI4T5K7YGHDDF75UUH722QEZYSYU3DRXS4HPYWLLADM6J2EHNR5A, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.230196019 (573h1m28.101279054s) [noise=true , address=145.79.227.48:4301 , piece_id=VEJVLLQU2KDKYM3Y5XZB6TNUUXSX2MDC6Y2KXN6RUEWASB7G6TAQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.230229163 (573h1m28.101312198s) [noise=true , address=145.79.227.45:4231 , piece_id=AI4FTKKS7SWUMAQ35E4ML7TT3O2CGOZHAA5FQPIM4L2TSDOV3TSQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.230721130 (573h1m28.101804165s) [noise=true , address=77.90.31.16:4735 , piece_id=LED3MXVURDX5ITUWGZGJHXM2LXAX3ISSJKAECUJUOABOLUHEAKKA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.230975274 (573h1m28.102058309s) [noise=true , address=145.79.227.13:4802 , piece_id=MR6H5CX267HPWCETWENONX5QMETKXN3SL5LKWYNBNDEMBWR46ZAA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.231314189 (573h1m28.102397235s) [noise=true , address=45.91.116.10:4083 , piece_id=AYQXAYPZMWZN2YSVCNUENMW4KCE6BFBEBV2N57BRRDLNYL5ZCLTA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.231509651 (573h1m28.102592686s) [noise=true , address=145.79.227.26:4654 , piece_id=CJJW6NM2HL572SDZSTGZID6S6PRCFPS6DEJMQU4WN3DX2GMMLEHQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.231966481 (573h1m28.103049526s) [noise=true , address=45.91.116.6:4974 , piece_id=NCAE2UNBQ7IAI2B2EE5QKMIJQZ5NPAMBGEYODLFMERSD5IVYUUHA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.232489376 (573h1m28.103572411s) [noise=true , address=145.79.227.7:4293 , piece_id=6FYYAD6K52LDUNKOGFFAGNLPFHKA7V3WVBRPKHOZBZXMHRHIACYA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.232858590 (573h1m28.103941625s) [noise=true , address=45.91.116.5:4297 , piece_id=63ABPGW4YSCEPCPMQHFZ6NZVWNYJAELGJNXJJXWFTHVTW6YIVAQA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:29.934302048 (573h1m29.805385083s) [upload=502356]: Next returned nil segment Dec 29 12:30:29.934324049 (573h1m29.805407094s) [upload=502356]: Waiting for error group managing segments... Dec 29 12:30:30.219330104 (573h1m30.090413149s) [noise=true , address=145.79.227.2:4115 , piece_id=FYK6H2CTV3YRNPU3TW5ICDUUHNJLUGYB7YVL242N7EKXA3OFMUHQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.222585522 (573h1m30.093668557s) [noise=true , address=145.79.227.26:4654 , piece_id=CJJW6NM2HL572SDZSTGZID6S6PRCFPS6DEJMQU4WN3DX2GMMLEHQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.223473883 (573h1m30.094556928s) [noise=true , address=145.79.227.7:4293 , piece_id=6FYYAD6K52LDUNKOGFFAGNLPFHKA7V3WVBRPKHOZBZXMHRHIACYA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.225048181 (573h1m30.096131226s) [noise=true , address=145.79.227.32:4835 , piece_id=GI4T5K7YGHDDF75UUH722QEZYSYU3DRXS4HPYWLLADM6J2EHNR5A, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.227668892 (573h1m30.098751937s) [noise=true , address=145.79.227.10:4401 , piece_id=TCQXBURRVQY5VJBL6CVM56AVQ23IALWXMJYCXI2FQNK5VEA7ZP6A, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.228574516 (573h1m30.099657561s) [noise=true , address=145.79.227.13:4802 , piece_id=MR6H5CX267HPWCETWENONX5QMETKXN3SL5LKWYNBNDEMBWR46ZAA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.234122320 (573h1m30.105205355s) [noise=true , address=145.79.227.45:4231 , piece_id=AI4FTKKS7SWUMAQ35E4ML7TT3O2CGOZHAA5FQPIM4L2TSDOV3TSQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.240891529 (573h1m30.111974574s) [noise=true , address=145.79.227.48:4301 , piece_id=VEJVLLQU2KDKYM3Y5XZB6TNUUXSX2MDC6Y2KXN6RUEWASB7G6TAQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.241789950 (573h1m30.112872985s) [noise=true , address=145.79.227.46:4601 , piece_id=C3VY24KTEOXUNLPAVDT45J2LRQPQ6AOR2OKTEJGLARZOUGOS64YQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.247350498 (573h1m30.118433534s) [noise=true , address=145.79.227.41:4085 , piece_id=MZYJNEXIKOQVVEZPIEZ33W6XJWKMY3LLYKVUW5PNSRHCD7Y6AZQA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.417471593 (573h1m30.288554628s) [noise=true , address=77.90.31.16:4735 , piece_id=LED3MXVURDX5ITUWGZGJHXM2LXAX3ISSJKAECUJUOABOLUHEAKKA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.432824027 (573h1m30.303907062s) [noise=true , address=45.91.116.10:4083 , piece_id=AYQXAYPZMWZN2YSVCNUENMW4KCE6BFBEBV2N57BRRDLNYL5ZCLTA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.583724624 (573h1m30.454807659s) [noise=true , address=45.91.116.14:4190 , piece_id=SVSZXQDV6WXOB5BID3HVGTAPTKSOJ2PXSGGW3OIQ5HEZRGVRG6DA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.583743831 (573h1m30.454826876s) [seg_pos={0 0}, upload=502356]: Segment reached optimal threshold of 13 pieces. Dec 29 12:30:30.583922962 (573h1m30.455005997s) [noise=true , address=45.91.116.5:4297 , piece_id=63ABPGW4YSCEPCPMQHFZ6NZVWNYJAELGJNXJJXWFTHVTW6YIVAQA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 29 12:30:30.583929113 (573h1m30.455012149s) [noise=true , address=45.91.116.5:4297 , piece_id=63ABPGW4YSCEPCPMQHFZ6NZVWNYJAELGJNXJJXWFTHVTW6YIVAQA, seg_pos={0 0}, upload=502356]: Long tail cancellation due to optimal threshold Dec 29 12:30:30.584091974 (573h1m30.455175009s) [noise=true , address=45.91.116.6:4974 , piece_id=NCAE2UNBQ7IAI2B2EE5QKMIJQZ5NPAMBGEYODLFMERSD5IVYUUHA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:30:30.584108805 (573h1m30.455191841s) [noise=true , address=45.91.116.6:4974 , piece_id=NCAE2UNBQ7IAI2B2EE5QKMIJQZ5NPAMBGEYODLFMERSD5IVYUUHA, seg_pos={0 0}, upload=502356]: Long tail cancellation due to optimal threshold Dec 29 12:30:30.584341749 (573h1m30.455424794s) [noise=true , address=45.91.116.12:4227 , piece_id=34PI32NCVIDSE2W6UAATADRFY2ZKVIS5DOXTHGL2TWDNUYZK5I4A, seg_pos={0 0}, upload=502356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: context canceled; context canceled Dec 29 12:30:30.584349564 (573h1m30.455432609s) [noise=true , address=45.91.116.12:4227 , piece_id=34PI32NCVIDSE2W6UAATADRFY2ZKVIS5DOXTHGL2TWDNUYZK5I4A, seg_pos={0 0}, upload=502356]: Long tail cancellation due to optimal threshold Dec 29 12:30:30.584369573 (573h1m30.455452608s) [seg_pos={0 0}, upload=502356]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:30:30.584449214 (573h1m30.455532259s) [upload=502356]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:30:30.584453732 (573h1m30.455536767s) [upload=502356]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:30:30.886256078 (573h1m30.757339123s) [upload=502356]: Done uploading segments... Dec 29 12:30:30.886281006 (573h1m30.757364041s) [upload=502356]: Upload finished. err: <nil> Duration: 5.949145395s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 38 out of 60 is failed (threshold=6) [.XXX...XXX..XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXXXXXX] duration: 5.001698468s trace: https://jaeger.dev.storj.io/trace/efa624aa50d309a FAIL (2025-12-29 12:33:02 UTC) orbiter warning: context deadline exceeded Duration: 5.001724739s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................X.......X..........X....] duration: 33.756777151s trace: https://jaeger.dev.storj.io/trace/68764c6cb4286314 FAIL (2025-12-29 12:32:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528204662 (572h59m20.399287707s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528497489 (572h59m20.399580524s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528793153 (572h59m20.399876198s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529083555 (572h59m20.40016659s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529391081 (572h59m20.400474126s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529735096 (572h59m20.400818131s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530109970 (572h59m20.401193015s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530449556 (572h59m20.401532591s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530831554 (572h59m20.401914599s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531229512 (572h59m20.402312557s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531698945 (572h59m20.402781991s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532204238 (572h59m20.403287283s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532744366 (572h59m20.403827401s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533231203 (572h59m20.404314248s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533740113 (572h59m20.404823148s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:21.876539279 (572h59m21.747622324s) [upload=502322]: Next returned nil segment Dec 29 12:28:21.876614272 (572h59m21.747697307s) [upload=502322]: Waiting for error group managing segments... Dec 29 12:28:22.383674429 (572h59m22.254757464s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389083709 (572h59m22.260166754s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389450968 (572h59m22.260534003s) [noise=true , address=145.79.227.46:4625 , piece_id=6NEND2CKGIZLK5Y6IMM4XB6SXPSXUYSZIVZ7CZ25IKAHFFW33C5Q, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.418373151 (572h59m22.289456186s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.619432477 (572h59m22.490515552s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.741876388 (572h59m22.612959423s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.133731187 (572h59m23.004814222s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.146041929 (572h59m23.017124964s) [noise=true , address=193.149.164.143:4697 , piece_id=HZZY3SUOEC7AIHSW6MKZRNZVSOGXSYBYSUKN7STBFTXWAOMDRYBA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.261354601 (572h59m23.132437636s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.291387197 (572h59m23.162470243s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.318370928 (572h59m23.189453973s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.321236023 (572h59m23.192319058s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.408408673 (572h59m23.279491718s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419570137 (572h59m23.290653172s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419592470 (572h59m23.290675515s) [seg_pos={0 0}, upload=502322]: Segment reached optimal threshold of 14 pieces. Dec 29 12:28:23.419837406 (572h59m23.290920441s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123jUFZTVjq6adDuvt3cco7dTvwNTQkszZX7CeajMfbeCUmEmuJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419845241 (572h59m23.290928276s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.419873174 (572h59m23.290956209s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419882021 (572h59m23.290965076s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420073716 (572h59m23.291156751s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection Dec 29 12:28:23.420082262 (572h59m23.291165307s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420257506 (572h59m23.291340551s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: read tcp 188.166.225.170:41270->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:28:23.420264649 (572h59m23.291347684s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420281822 (572h59m23.291364857s) [seg_pos={0 0}, upload=502322]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:28:23.420364349 (572h59m23.291447394s) [upload=502322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:28:23.420370270 (572h59m23.291453315s) [upload=502322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:28:23.725225770 (572h59m23.596308815s) [upload=502322]: Done uploading segments... Dec 29 12:28:23.725255426 (572h59m23.596338481s) [upload=502322]: Upload finished. err: <nil> Duration: 33.756803942s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X....XXXXXX..X.X.......X.XX...X.......X.X..XXX....X...X..X..] duration: 5.94581261s trace: https://jaeger.dev.storj.io/trace/55af87306b44a0fb FAIL (2025-12-29 12:32:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 12:30:27.976557996 (573h1m27.847641031s) [upload=502356]: Starting upload Dec 29 12:30:27.976643769 (573h1m27.847726804s) [upload=502356]: Done starting upload Dec 29 12:30:27.976805467 (573h1m27.847888512s) [upload=502356]: Uploading segments... Dec 29 12:30:27.976959821 (573h1m27.848042856s) [upload=502356]: Got next segment. Inline: false Dec 29 12:30:27.977030205 (573h1m27.84811324s) [upload=502356]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:30:27.977037499 (573h1m27.848120544s) [upload=502356]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:30:28.227612220 (573h1m28.098695265s) [seg_pos={0 0}, upload=502356]: Begin upload segment... Dec 29 12:30:28.227680119 (573h1m28.098763154s) [seg_pos={0 0}, upload=502356]: Done begin upload segment. Dec 29 12:30:28.227731227 (573h1m28.098814272s) [noise=true , address=145.79.227.2:4115 , piece_id=FYK6H2CTV3YRNPU3TW5ICDUUHNJLUGYB7YVL242N7EKXA3OFMUHQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.227777565 (573h1m28.0988606s) [noise=true , address=45.91.116.14:4190 , piece_id=SVSZXQDV6WXOB5BID3HVGTAPTKSOJ2PXSGGW3OIQ5HEZRGVRG6DA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.228331690 (573h1m28.099414725s) [noise=true , address=45.91.116.12:4227 , piece_id=34PI32NCVIDSE2W6UAATADRFY2ZKVIS5DOXTHGL2TWDNUYZK5I4A, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.228647301 (573h1m28.099730336s) [noise=true , address=145.79.227.10:4401 , piece_id=TCQXBURRVQY5VJBL6CVM56AVQ23IALWXMJYCXI2FQNK5VEA7ZP6A, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.228978120 (573h1m28.100061155s) [noise=true , address=145.79.227.41:4085 , piece_id=MZYJNEXIKOQVVEZPIEZ33W6XJWKMY3LLYKVUW5PNSRHCD7Y6AZQA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.229164997 (573h1m28.100248032s) [noise=true , address=145.79.227.46:4601 , piece_id=C3VY24KTEOXUNLPAVDT45J2LRQPQ6AOR2OKTEJGLARZOUGOS64YQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.229770089 (573h1m28.100853124s) [noise=true , address=145.79.227.32:4835 , piece_id=GI4T5K7YGHDDF75UUH722QEZYSYU3DRXS4HPYWLLADM6J2EHNR5A, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.230196019 (573h1m28.101279054s) [noise=true , address=145.79.227.48:4301 , piece_id=VEJVLLQU2KDKYM3Y5XZB6TNUUXSX2MDC6Y2KXN6RUEWASB7G6TAQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.230229163 (573h1m28.101312198s) [noise=true , address=145.79.227.45:4231 , piece_id=AI4FTKKS7SWUMAQ35E4ML7TT3O2CGOZHAA5FQPIM4L2TSDOV3TSQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.230721130 (573h1m28.101804165s) [noise=true , address=77.90.31.16:4735 , piece_id=LED3MXVURDX5ITUWGZGJHXM2LXAX3ISSJKAECUJUOABOLUHEAKKA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.230975274 (573h1m28.102058309s) [noise=true , address=145.79.227.13:4802 , piece_id=MR6H5CX267HPWCETWENONX5QMETKXN3SL5LKWYNBNDEMBWR46ZAA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.231314189 (573h1m28.102397235s) [noise=true , address=45.91.116.10:4083 , piece_id=AYQXAYPZMWZN2YSVCNUENMW4KCE6BFBEBV2N57BRRDLNYL5ZCLTA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.231509651 (573h1m28.102592686s) [noise=true , address=145.79.227.26:4654 , piece_id=CJJW6NM2HL572SDZSTGZID6S6PRCFPS6DEJMQU4WN3DX2GMMLEHQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.231966481 (573h1m28.103049526s) [noise=true , address=45.91.116.6:4974 , piece_id=NCAE2UNBQ7IAI2B2EE5QKMIJQZ5NPAMBGEYODLFMERSD5IVYUUHA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.232489376 (573h1m28.103572411s) [noise=true , address=145.79.227.7:4293 , piece_id=6FYYAD6K52LDUNKOGFFAGNLPFHKA7V3WVBRPKHOZBZXMHRHIACYA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.232858590 (573h1m28.103941625s) [noise=true , address=45.91.116.5:4297 , piece_id=63ABPGW4YSCEPCPMQHFZ6NZVWNYJAELGJNXJJXWFTHVTW6YIVAQA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:29.934302048 (573h1m29.805385083s) [upload=502356]: Next returned nil segment Dec 29 12:30:29.934324049 (573h1m29.805407094s) [upload=502356]: Waiting for error group managing segments... Dec 29 12:30:30.219330104 (573h1m30.090413149s) [noise=true , address=145.79.227.2:4115 , piece_id=FYK6H2CTV3YRNPU3TW5ICDUUHNJLUGYB7YVL242N7EKXA3OFMUHQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.222585522 (573h1m30.093668557s) [noise=true , address=145.79.227.26:4654 , piece_id=CJJW6NM2HL572SDZSTGZID6S6PRCFPS6DEJMQU4WN3DX2GMMLEHQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.223473883 (573h1m30.094556928s) [noise=true , address=145.79.227.7:4293 , piece_id=6FYYAD6K52LDUNKOGFFAGNLPFHKA7V3WVBRPKHOZBZXMHRHIACYA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.225048181 (573h1m30.096131226s) [noise=true , address=145.79.227.32:4835 , piece_id=GI4T5K7YGHDDF75UUH722QEZYSYU3DRXS4HPYWLLADM6J2EHNR5A, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.227668892 (573h1m30.098751937s) [noise=true , address=145.79.227.10:4401 , piece_id=TCQXBURRVQY5VJBL6CVM56AVQ23IALWXMJYCXI2FQNK5VEA7ZP6A, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.228574516 (573h1m30.099657561s) [noise=true , address=145.79.227.13:4802 , piece_id=MR6H5CX267HPWCETWENONX5QMETKXN3SL5LKWYNBNDEMBWR46ZAA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.234122320 (573h1m30.105205355s) [noise=true , address=145.79.227.45:4231 , piece_id=AI4FTKKS7SWUMAQ35E4ML7TT3O2CGOZHAA5FQPIM4L2TSDOV3TSQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.240891529 (573h1m30.111974574s) [noise=true , address=145.79.227.48:4301 , piece_id=VEJVLLQU2KDKYM3Y5XZB6TNUUXSX2MDC6Y2KXN6RUEWASB7G6TAQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.241789950 (573h1m30.112872985s) [noise=true , address=145.79.227.46:4601 , piece_id=C3VY24KTEOXUNLPAVDT45J2LRQPQ6AOR2OKTEJGLARZOUGOS64YQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.247350498 (573h1m30.118433534s) [noise=true , address=145.79.227.41:4085 , piece_id=MZYJNEXIKOQVVEZPIEZ33W6XJWKMY3LLYKVUW5PNSRHCD7Y6AZQA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.417471593 (573h1m30.288554628s) [noise=true , address=77.90.31.16:4735 , piece_id=LED3MXVURDX5ITUWGZGJHXM2LXAX3ISSJKAECUJUOABOLUHEAKKA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.432824027 (573h1m30.303907062s) [noise=true , address=45.91.116.10:4083 , piece_id=AYQXAYPZMWZN2YSVCNUENMW4KCE6BFBEBV2N57BRRDLNYL5ZCLTA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.583724624 (573h1m30.454807659s) [noise=true , address=45.91.116.14:4190 , piece_id=SVSZXQDV6WXOB5BID3HVGTAPTKSOJ2PXSGGW3OIQ5HEZRGVRG6DA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.583743831 (573h1m30.454826876s) [seg_pos={0 0}, upload=502356]: Segment reached optimal threshold of 13 pieces. Dec 29 12:30:30.583922962 (573h1m30.455005997s) [noise=true , address=45.91.116.5:4297 , piece_id=63ABPGW4YSCEPCPMQHFZ6NZVWNYJAELGJNXJJXWFTHVTW6YIVAQA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 29 12:30:30.583929113 (573h1m30.455012149s) [noise=true , address=45.91.116.5:4297 , piece_id=63ABPGW4YSCEPCPMQHFZ6NZVWNYJAELGJNXJJXWFTHVTW6YIVAQA, seg_pos={0 0}, upload=502356]: Long tail cancellation due to optimal threshold Dec 29 12:30:30.584091974 (573h1m30.455175009s) [noise=true , address=45.91.116.6:4974 , piece_id=NCAE2UNBQ7IAI2B2EE5QKMIJQZ5NPAMBGEYODLFMERSD5IVYUUHA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:30:30.584108805 (573h1m30.455191841s) [noise=true , address=45.91.116.6:4974 , piece_id=NCAE2UNBQ7IAI2B2EE5QKMIJQZ5NPAMBGEYODLFMERSD5IVYUUHA, seg_pos={0 0}, upload=502356]: Long tail cancellation due to optimal threshold Dec 29 12:30:30.584341749 (573h1m30.455424794s) [noise=true , address=45.91.116.12:4227 , piece_id=34PI32NCVIDSE2W6UAATADRFY2ZKVIS5DOXTHGL2TWDNUYZK5I4A, seg_pos={0 0}, upload=502356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: context canceled; context canceled Dec 29 12:30:30.584349564 (573h1m30.455432609s) [noise=true , address=45.91.116.12:4227 , piece_id=34PI32NCVIDSE2W6UAATADRFY2ZKVIS5DOXTHGL2TWDNUYZK5I4A, seg_pos={0 0}, upload=502356]: Long tail cancellation due to optimal threshold Dec 29 12:30:30.584369573 (573h1m30.455452608s) [seg_pos={0 0}, upload=502356]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:30:30.584449214 (573h1m30.455532259s) [upload=502356]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:30:30.584453732 (573h1m30.455536767s) [upload=502356]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:30:30.886256078 (573h1m30.757339123s) [upload=502356]: Done uploading segments... Dec 29 12:30:30.886281006 (573h1m30.757364041s) [upload=502356]: Upload finished. err: <nil> Duration: 5.94583946s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 37 out of 60 is failed (threshold=6) [..XXX...XXX..XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXXXXX] duration: 5.00083816s trace: https://jaeger.dev.storj.io/trace/aee26b416e7073d FAIL (2025-12-29 12:32:16 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000864159s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................X.......X..........X...] duration: 49.45006109s trace: https://jaeger.dev.storj.io/trace/68764c6cb4286314 FAIL (2025-12-29 12:31:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528204662 (572h59m20.399287707s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528497489 (572h59m20.399580524s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528793153 (572h59m20.399876198s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529083555 (572h59m20.40016659s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529391081 (572h59m20.400474126s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529735096 (572h59m20.400818131s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530109970 (572h59m20.401193015s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530449556 (572h59m20.401532591s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530831554 (572h59m20.401914599s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531229512 (572h59m20.402312557s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531698945 (572h59m20.402781991s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532204238 (572h59m20.403287283s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532744366 (572h59m20.403827401s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533231203 (572h59m20.404314248s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533740113 (572h59m20.404823148s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:21.876539279 (572h59m21.747622324s) [upload=502322]: Next returned nil segment Dec 29 12:28:21.876614272 (572h59m21.747697307s) [upload=502322]: Waiting for error group managing segments... Dec 29 12:28:22.383674429 (572h59m22.254757464s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389083709 (572h59m22.260166754s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389450968 (572h59m22.260534003s) [noise=true , address=145.79.227.46:4625 , piece_id=6NEND2CKGIZLK5Y6IMM4XB6SXPSXUYSZIVZ7CZ25IKAHFFW33C5Q, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.418373151 (572h59m22.289456186s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.619432477 (572h59m22.490515552s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.741876388 (572h59m22.612959423s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.133731187 (572h59m23.004814222s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.146041929 (572h59m23.017124964s) [noise=true , address=193.149.164.143:4697 , piece_id=HZZY3SUOEC7AIHSW6MKZRNZVSOGXSYBYSUKN7STBFTXWAOMDRYBA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.261354601 (572h59m23.132437636s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.291387197 (572h59m23.162470243s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.318370928 (572h59m23.189453973s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.321236023 (572h59m23.192319058s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.408408673 (572h59m23.279491718s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419570137 (572h59m23.290653172s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419592470 (572h59m23.290675515s) [seg_pos={0 0}, upload=502322]: Segment reached optimal threshold of 14 pieces. Dec 29 12:28:23.419837406 (572h59m23.290920441s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123jUFZTVjq6adDuvt3cco7dTvwNTQkszZX7CeajMfbeCUmEmuJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419845241 (572h59m23.290928276s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.419873174 (572h59m23.290956209s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419882021 (572h59m23.290965076s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420073716 (572h59m23.291156751s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection Dec 29 12:28:23.420082262 (572h59m23.291165307s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420257506 (572h59m23.291340551s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: read tcp 188.166.225.170:41270->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:28:23.420264649 (572h59m23.291347684s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420281822 (572h59m23.291364857s) [seg_pos={0 0}, upload=502322]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:28:23.420364349 (572h59m23.291447394s) [upload=502322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:28:23.420370270 (572h59m23.291453315s) [upload=502322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:28:23.725225770 (572h59m23.596308815s) [upload=502322]: Done uploading segments... Dec 29 12:28:23.725255426 (572h59m23.596338481s) [upload=502322]: Upload finished. err: <nil> Duration: 49.450080016s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [...XXX...XXX..XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXXXX] duration: 5.001242529s trace: https://jaeger.dev.storj.io/trace/39f499f98da83ad6 FAIL (2025-12-29 12:31:31 UTC) orbiter warning: context deadline exceeded Duration: 5.001259361s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X....XXXXXX..X.X.......X.XX...X.......X.X..XXX....X...X..X.] duration: 5.770530615s trace: https://jaeger.dev.storj.io/trace/55af87306b44a0fb FAIL (2025-12-29 12:31:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 12:30:27.976557996 (573h1m27.847641031s) [upload=502356]: Starting upload Dec 29 12:30:27.976643769 (573h1m27.847726804s) [upload=502356]: Done starting upload Dec 29 12:30:27.976805467 (573h1m27.847888512s) [upload=502356]: Uploading segments... Dec 29 12:30:27.976959821 (573h1m27.848042856s) [upload=502356]: Got next segment. Inline: false Dec 29 12:30:27.977030205 (573h1m27.84811324s) [upload=502356]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:30:27.977037499 (573h1m27.848120544s) [upload=502356]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:30:28.227612220 (573h1m28.098695265s) [seg_pos={0 0}, upload=502356]: Begin upload segment... Dec 29 12:30:28.227680119 (573h1m28.098763154s) [seg_pos={0 0}, upload=502356]: Done begin upload segment. Dec 29 12:30:28.227731227 (573h1m28.098814272s) [noise=true , address=145.79.227.2:4115 , piece_id=FYK6H2CTV3YRNPU3TW5ICDUUHNJLUGYB7YVL242N7EKXA3OFMUHQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.227777565 (573h1m28.0988606s) [noise=true , address=45.91.116.14:4190 , piece_id=SVSZXQDV6WXOB5BID3HVGTAPTKSOJ2PXSGGW3OIQ5HEZRGVRG6DA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.228331690 (573h1m28.099414725s) [noise=true , address=45.91.116.12:4227 , piece_id=34PI32NCVIDSE2W6UAATADRFY2ZKVIS5DOXTHGL2TWDNUYZK5I4A, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.228647301 (573h1m28.099730336s) [noise=true , address=145.79.227.10:4401 , piece_id=TCQXBURRVQY5VJBL6CVM56AVQ23IALWXMJYCXI2FQNK5VEA7ZP6A, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.228978120 (573h1m28.100061155s) [noise=true , address=145.79.227.41:4085 , piece_id=MZYJNEXIKOQVVEZPIEZ33W6XJWKMY3LLYKVUW5PNSRHCD7Y6AZQA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.229164997 (573h1m28.100248032s) [noise=true , address=145.79.227.46:4601 , piece_id=C3VY24KTEOXUNLPAVDT45J2LRQPQ6AOR2OKTEJGLARZOUGOS64YQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.229770089 (573h1m28.100853124s) [noise=true , address=145.79.227.32:4835 , piece_id=GI4T5K7YGHDDF75UUH722QEZYSYU3DRXS4HPYWLLADM6J2EHNR5A, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.230196019 (573h1m28.101279054s) [noise=true , address=145.79.227.48:4301 , piece_id=VEJVLLQU2KDKYM3Y5XZB6TNUUXSX2MDC6Y2KXN6RUEWASB7G6TAQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.230229163 (573h1m28.101312198s) [noise=true , address=145.79.227.45:4231 , piece_id=AI4FTKKS7SWUMAQ35E4ML7TT3O2CGOZHAA5FQPIM4L2TSDOV3TSQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.230721130 (573h1m28.101804165s) [noise=true , address=77.90.31.16:4735 , piece_id=LED3MXVURDX5ITUWGZGJHXM2LXAX3ISSJKAECUJUOABOLUHEAKKA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.230975274 (573h1m28.102058309s) [noise=true , address=145.79.227.13:4802 , piece_id=MR6H5CX267HPWCETWENONX5QMETKXN3SL5LKWYNBNDEMBWR46ZAA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.231314189 (573h1m28.102397235s) [noise=true , address=45.91.116.10:4083 , piece_id=AYQXAYPZMWZN2YSVCNUENMW4KCE6BFBEBV2N57BRRDLNYL5ZCLTA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.231509651 (573h1m28.102592686s) [noise=true , address=145.79.227.26:4654 , piece_id=CJJW6NM2HL572SDZSTGZID6S6PRCFPS6DEJMQU4WN3DX2GMMLEHQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.231966481 (573h1m28.103049526s) [noise=true , address=45.91.116.6:4974 , piece_id=NCAE2UNBQ7IAI2B2EE5QKMIJQZ5NPAMBGEYODLFMERSD5IVYUUHA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.232489376 (573h1m28.103572411s) [noise=true , address=145.79.227.7:4293 , piece_id=6FYYAD6K52LDUNKOGFFAGNLPFHKA7V3WVBRPKHOZBZXMHRHIACYA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.232858590 (573h1m28.103941625s) [noise=true , address=45.91.116.5:4297 , piece_id=63ABPGW4YSCEPCPMQHFZ6NZVWNYJAELGJNXJJXWFTHVTW6YIVAQA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:29.934302048 (573h1m29.805385083s) [upload=502356]: Next returned nil segment Dec 29 12:30:29.934324049 (573h1m29.805407094s) [upload=502356]: Waiting for error group managing segments... Dec 29 12:30:30.219330104 (573h1m30.090413149s) [noise=true , address=145.79.227.2:4115 , piece_id=FYK6H2CTV3YRNPU3TW5ICDUUHNJLUGYB7YVL242N7EKXA3OFMUHQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.222585522 (573h1m30.093668557s) [noise=true , address=145.79.227.26:4654 , piece_id=CJJW6NM2HL572SDZSTGZID6S6PRCFPS6DEJMQU4WN3DX2GMMLEHQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.223473883 (573h1m30.094556928s) [noise=true , address=145.79.227.7:4293 , piece_id=6FYYAD6K52LDUNKOGFFAGNLPFHKA7V3WVBRPKHOZBZXMHRHIACYA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.225048181 (573h1m30.096131226s) [noise=true , address=145.79.227.32:4835 , piece_id=GI4T5K7YGHDDF75UUH722QEZYSYU3DRXS4HPYWLLADM6J2EHNR5A, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.227668892 (573h1m30.098751937s) [noise=true , address=145.79.227.10:4401 , piece_id=TCQXBURRVQY5VJBL6CVM56AVQ23IALWXMJYCXI2FQNK5VEA7ZP6A, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.228574516 (573h1m30.099657561s) [noise=true , address=145.79.227.13:4802 , piece_id=MR6H5CX267HPWCETWENONX5QMETKXN3SL5LKWYNBNDEMBWR46ZAA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.234122320 (573h1m30.105205355s) [noise=true , address=145.79.227.45:4231 , piece_id=AI4FTKKS7SWUMAQ35E4ML7TT3O2CGOZHAA5FQPIM4L2TSDOV3TSQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.240891529 (573h1m30.111974574s) [noise=true , address=145.79.227.48:4301 , piece_id=VEJVLLQU2KDKYM3Y5XZB6TNUUXSX2MDC6Y2KXN6RUEWASB7G6TAQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.241789950 (573h1m30.112872985s) [noise=true , address=145.79.227.46:4601 , piece_id=C3VY24KTEOXUNLPAVDT45J2LRQPQ6AOR2OKTEJGLARZOUGOS64YQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.247350498 (573h1m30.118433534s) [noise=true , address=145.79.227.41:4085 , piece_id=MZYJNEXIKOQVVEZPIEZ33W6XJWKMY3LLYKVUW5PNSRHCD7Y6AZQA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.417471593 (573h1m30.288554628s) [noise=true , address=77.90.31.16:4735 , piece_id=LED3MXVURDX5ITUWGZGJHXM2LXAX3ISSJKAECUJUOABOLUHEAKKA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.432824027 (573h1m30.303907062s) [noise=true , address=45.91.116.10:4083 , piece_id=AYQXAYPZMWZN2YSVCNUENMW4KCE6BFBEBV2N57BRRDLNYL5ZCLTA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.583724624 (573h1m30.454807659s) [noise=true , address=45.91.116.14:4190 , piece_id=SVSZXQDV6WXOB5BID3HVGTAPTKSOJ2PXSGGW3OIQ5HEZRGVRG6DA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.583743831 (573h1m30.454826876s) [seg_pos={0 0}, upload=502356]: Segment reached optimal threshold of 13 pieces. Dec 29 12:30:30.583922962 (573h1m30.455005997s) [noise=true , address=45.91.116.5:4297 , piece_id=63ABPGW4YSCEPCPMQHFZ6NZVWNYJAELGJNXJJXWFTHVTW6YIVAQA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 29 12:30:30.583929113 (573h1m30.455012149s) [noise=true , address=45.91.116.5:4297 , piece_id=63ABPGW4YSCEPCPMQHFZ6NZVWNYJAELGJNXJJXWFTHVTW6YIVAQA, seg_pos={0 0}, upload=502356]: Long tail cancellation due to optimal threshold Dec 29 12:30:30.584091974 (573h1m30.455175009s) [noise=true , address=45.91.116.6:4974 , piece_id=NCAE2UNBQ7IAI2B2EE5QKMIJQZ5NPAMBGEYODLFMERSD5IVYUUHA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:30:30.584108805 (573h1m30.455191841s) [noise=true , address=45.91.116.6:4974 , piece_id=NCAE2UNBQ7IAI2B2EE5QKMIJQZ5NPAMBGEYODLFMERSD5IVYUUHA, seg_pos={0 0}, upload=502356]: Long tail cancellation due to optimal threshold Dec 29 12:30:30.584341749 (573h1m30.455424794s) [noise=true , address=45.91.116.12:4227 , piece_id=34PI32NCVIDSE2W6UAATADRFY2ZKVIS5DOXTHGL2TWDNUYZK5I4A, seg_pos={0 0}, upload=502356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: context canceled; context canceled Dec 29 12:30:30.584349564 (573h1m30.455432609s) [noise=true , address=45.91.116.12:4227 , piece_id=34PI32NCVIDSE2W6UAATADRFY2ZKVIS5DOXTHGL2TWDNUYZK5I4A, seg_pos={0 0}, upload=502356]: Long tail cancellation due to optimal threshold Dec 29 12:30:30.584369573 (573h1m30.455452608s) [seg_pos={0 0}, upload=502356]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:30:30.584449214 (573h1m30.455532259s) [upload=502356]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:30:30.584453732 (573h1m30.455536767s) [upload=502356]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:30:30.886256078 (573h1m30.757339123s) [upload=502356]: Done uploading segments... Dec 29 12:30:30.886281006 (573h1m30.757364041s) [upload=502356]: Upload finished. err: <nil> Duration: 5.770558809s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X.X....XXXXXX..X.X.......X.XX...X.......X.X..XXX....X...X..X] duration: 30.005533785s trace: https://jaeger.dev.storj.io/trace/55af87306b44a0fb FAIL (2025-12-29 12:30:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 12:30:27.976557996 (573h1m27.847641031s) [upload=502356]: Starting upload Dec 29 12:30:27.976643769 (573h1m27.847726804s) [upload=502356]: Done starting upload Dec 29 12:30:27.976805467 (573h1m27.847888512s) [upload=502356]: Uploading segments... Dec 29 12:30:27.976959821 (573h1m27.848042856s) [upload=502356]: Got next segment. Inline: false Dec 29 12:30:27.977030205 (573h1m27.84811324s) [upload=502356]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:30:27.977037499 (573h1m27.848120544s) [upload=502356]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:30:28.227612220 (573h1m28.098695265s) [seg_pos={0 0}, upload=502356]: Begin upload segment... Dec 29 12:30:28.227680119 (573h1m28.098763154s) [seg_pos={0 0}, upload=502356]: Done begin upload segment. Dec 29 12:30:28.227731227 (573h1m28.098814272s) [noise=true , address=145.79.227.2:4115 , piece_id=FYK6H2CTV3YRNPU3TW5ICDUUHNJLUGYB7YVL242N7EKXA3OFMUHQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.227777565 (573h1m28.0988606s) [noise=true , address=45.91.116.14:4190 , piece_id=SVSZXQDV6WXOB5BID3HVGTAPTKSOJ2PXSGGW3OIQ5HEZRGVRG6DA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.228331690 (573h1m28.099414725s) [noise=true , address=45.91.116.12:4227 , piece_id=34PI32NCVIDSE2W6UAATADRFY2ZKVIS5DOXTHGL2TWDNUYZK5I4A, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.228647301 (573h1m28.099730336s) [noise=true , address=145.79.227.10:4401 , piece_id=TCQXBURRVQY5VJBL6CVM56AVQ23IALWXMJYCXI2FQNK5VEA7ZP6A, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.228978120 (573h1m28.100061155s) [noise=true , address=145.79.227.41:4085 , piece_id=MZYJNEXIKOQVVEZPIEZ33W6XJWKMY3LLYKVUW5PNSRHCD7Y6AZQA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.229164997 (573h1m28.100248032s) [noise=true , address=145.79.227.46:4601 , piece_id=C3VY24KTEOXUNLPAVDT45J2LRQPQ6AOR2OKTEJGLARZOUGOS64YQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.229770089 (573h1m28.100853124s) [noise=true , address=145.79.227.32:4835 , piece_id=GI4T5K7YGHDDF75UUH722QEZYSYU3DRXS4HPYWLLADM6J2EHNR5A, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.230196019 (573h1m28.101279054s) [noise=true , address=145.79.227.48:4301 , piece_id=VEJVLLQU2KDKYM3Y5XZB6TNUUXSX2MDC6Y2KXN6RUEWASB7G6TAQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.230229163 (573h1m28.101312198s) [noise=true , address=145.79.227.45:4231 , piece_id=AI4FTKKS7SWUMAQ35E4ML7TT3O2CGOZHAA5FQPIM4L2TSDOV3TSQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.230721130 (573h1m28.101804165s) [noise=true , address=77.90.31.16:4735 , piece_id=LED3MXVURDX5ITUWGZGJHXM2LXAX3ISSJKAECUJUOABOLUHEAKKA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.230975274 (573h1m28.102058309s) [noise=true , address=145.79.227.13:4802 , piece_id=MR6H5CX267HPWCETWENONX5QMETKXN3SL5LKWYNBNDEMBWR46ZAA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.231314189 (573h1m28.102397235s) [noise=true , address=45.91.116.10:4083 , piece_id=AYQXAYPZMWZN2YSVCNUENMW4KCE6BFBEBV2N57BRRDLNYL5ZCLTA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.231509651 (573h1m28.102592686s) [noise=true , address=145.79.227.26:4654 , piece_id=CJJW6NM2HL572SDZSTGZID6S6PRCFPS6DEJMQU4WN3DX2GMMLEHQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.231966481 (573h1m28.103049526s) [noise=true , address=45.91.116.6:4974 , piece_id=NCAE2UNBQ7IAI2B2EE5QKMIJQZ5NPAMBGEYODLFMERSD5IVYUUHA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.232489376 (573h1m28.103572411s) [noise=true , address=145.79.227.7:4293 , piece_id=6FYYAD6K52LDUNKOGFFAGNLPFHKA7V3WVBRPKHOZBZXMHRHIACYA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.232858590 (573h1m28.103941625s) [noise=true , address=45.91.116.5:4297 , piece_id=63ABPGW4YSCEPCPMQHFZ6NZVWNYJAELGJNXJJXWFTHVTW6YIVAQA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:29.934302048 (573h1m29.805385083s) [upload=502356]: Next returned nil segment Dec 29 12:30:29.934324049 (573h1m29.805407094s) [upload=502356]: Waiting for error group managing segments... Dec 29 12:30:30.219330104 (573h1m30.090413149s) [noise=true , address=145.79.227.2:4115 , piece_id=FYK6H2CTV3YRNPU3TW5ICDUUHNJLUGYB7YVL242N7EKXA3OFMUHQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.222585522 (573h1m30.093668557s) [noise=true , address=145.79.227.26:4654 , piece_id=CJJW6NM2HL572SDZSTGZID6S6PRCFPS6DEJMQU4WN3DX2GMMLEHQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.223473883 (573h1m30.094556928s) [noise=true , address=145.79.227.7:4293 , piece_id=6FYYAD6K52LDUNKOGFFAGNLPFHKA7V3WVBRPKHOZBZXMHRHIACYA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.225048181 (573h1m30.096131226s) [noise=true , address=145.79.227.32:4835 , piece_id=GI4T5K7YGHDDF75UUH722QEZYSYU3DRXS4HPYWLLADM6J2EHNR5A, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.227668892 (573h1m30.098751937s) [noise=true , address=145.79.227.10:4401 , piece_id=TCQXBURRVQY5VJBL6CVM56AVQ23IALWXMJYCXI2FQNK5VEA7ZP6A, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.228574516 (573h1m30.099657561s) [noise=true , address=145.79.227.13:4802 , piece_id=MR6H5CX267HPWCETWENONX5QMETKXN3SL5LKWYNBNDEMBWR46ZAA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.234122320 (573h1m30.105205355s) [noise=true , address=145.79.227.45:4231 , piece_id=AI4FTKKS7SWUMAQ35E4ML7TT3O2CGOZHAA5FQPIM4L2TSDOV3TSQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.240891529 (573h1m30.111974574s) [noise=true , address=145.79.227.48:4301 , piece_id=VEJVLLQU2KDKYM3Y5XZB6TNUUXSX2MDC6Y2KXN6RUEWASB7G6TAQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.241789950 (573h1m30.112872985s) [noise=true , address=145.79.227.46:4601 , piece_id=C3VY24KTEOXUNLPAVDT45J2LRQPQ6AOR2OKTEJGLARZOUGOS64YQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.247350498 (573h1m30.118433534s) [noise=true , address=145.79.227.41:4085 , piece_id=MZYJNEXIKOQVVEZPIEZ33W6XJWKMY3LLYKVUW5PNSRHCD7Y6AZQA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.417471593 (573h1m30.288554628s) [noise=true , address=77.90.31.16:4735 , piece_id=LED3MXVURDX5ITUWGZGJHXM2LXAX3ISSJKAECUJUOABOLUHEAKKA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.432824027 (573h1m30.303907062s) [noise=true , address=45.91.116.10:4083 , piece_id=AYQXAYPZMWZN2YSVCNUENMW4KCE6BFBEBV2N57BRRDLNYL5ZCLTA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.583724624 (573h1m30.454807659s) [noise=true , address=45.91.116.14:4190 , piece_id=SVSZXQDV6WXOB5BID3HVGTAPTKSOJ2PXSGGW3OIQ5HEZRGVRG6DA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.583743831 (573h1m30.454826876s) [seg_pos={0 0}, upload=502356]: Segment reached optimal threshold of 13 pieces. Dec 29 12:30:30.583922962 (573h1m30.455005997s) [noise=true , address=45.91.116.5:4297 , piece_id=63ABPGW4YSCEPCPMQHFZ6NZVWNYJAELGJNXJJXWFTHVTW6YIVAQA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 29 12:30:30.583929113 (573h1m30.455012149s) [noise=true , address=45.91.116.5:4297 , piece_id=63ABPGW4YSCEPCPMQHFZ6NZVWNYJAELGJNXJJXWFTHVTW6YIVAQA, seg_pos={0 0}, upload=502356]: Long tail cancellation due to optimal threshold Dec 29 12:30:30.584091974 (573h1m30.455175009s) [noise=true , address=45.91.116.6:4974 , piece_id=NCAE2UNBQ7IAI2B2EE5QKMIJQZ5NPAMBGEYODLFMERSD5IVYUUHA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:30:30.584108805 (573h1m30.455191841s) [noise=true , address=45.91.116.6:4974 , piece_id=NCAE2UNBQ7IAI2B2EE5QKMIJQZ5NPAMBGEYODLFMERSD5IVYUUHA, seg_pos={0 0}, upload=502356]: Long tail cancellation due to optimal threshold Dec 29 12:30:30.584341749 (573h1m30.455424794s) [noise=true , address=45.91.116.12:4227 , piece_id=34PI32NCVIDSE2W6UAATADRFY2ZKVIS5DOXTHGL2TWDNUYZK5I4A, seg_pos={0 0}, upload=502356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: context canceled; context canceled Dec 29 12:30:30.584349564 (573h1m30.455432609s) [noise=true , address=45.91.116.12:4227 , piece_id=34PI32NCVIDSE2W6UAATADRFY2ZKVIS5DOXTHGL2TWDNUYZK5I4A, seg_pos={0 0}, upload=502356]: Long tail cancellation due to optimal threshold Dec 29 12:30:30.584369573 (573h1m30.455452608s) [seg_pos={0 0}, upload=502356]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:30:30.584449214 (573h1m30.455532259s) [upload=502356]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:30:30.584453732 (573h1m30.455536767s) [upload=502356]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:30:30.886256078 (573h1m30.757339123s) [upload=502356]: Done uploading segments... Dec 29 12:30:30.886281006 (573h1m30.757364041s) [upload=502356]: Upload finished. err: <nil> Duration: 30.005558082s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................X.......X..........X..] duration: 52.625956373s trace: https://jaeger.dev.storj.io/trace/68764c6cb4286314 FAIL (2025-12-29 12:30:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528204662 (572h59m20.399287707s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528497489 (572h59m20.399580524s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528793153 (572h59m20.399876198s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529083555 (572h59m20.40016659s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529391081 (572h59m20.400474126s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529735096 (572h59m20.400818131s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530109970 (572h59m20.401193015s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530449556 (572h59m20.401532591s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530831554 (572h59m20.401914599s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531229512 (572h59m20.402312557s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531698945 (572h59m20.402781991s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532204238 (572h59m20.403287283s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532744366 (572h59m20.403827401s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533231203 (572h59m20.404314248s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533740113 (572h59m20.404823148s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:21.876539279 (572h59m21.747622324s) [upload=502322]: Next returned nil segment Dec 29 12:28:21.876614272 (572h59m21.747697307s) [upload=502322]: Waiting for error group managing segments... Dec 29 12:28:22.383674429 (572h59m22.254757464s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389083709 (572h59m22.260166754s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389450968 (572h59m22.260534003s) [noise=true , address=145.79.227.46:4625 , piece_id=6NEND2CKGIZLK5Y6IMM4XB6SXPSXUYSZIVZ7CZ25IKAHFFW33C5Q, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.418373151 (572h59m22.289456186s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.619432477 (572h59m22.490515552s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.741876388 (572h59m22.612959423s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.133731187 (572h59m23.004814222s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.146041929 (572h59m23.017124964s) [noise=true , address=193.149.164.143:4697 , piece_id=HZZY3SUOEC7AIHSW6MKZRNZVSOGXSYBYSUKN7STBFTXWAOMDRYBA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.261354601 (572h59m23.132437636s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.291387197 (572h59m23.162470243s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.318370928 (572h59m23.189453973s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.321236023 (572h59m23.192319058s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.408408673 (572h59m23.279491718s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419570137 (572h59m23.290653172s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419592470 (572h59m23.290675515s) [seg_pos={0 0}, upload=502322]: Segment reached optimal threshold of 14 pieces. Dec 29 12:28:23.419837406 (572h59m23.290920441s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123jUFZTVjq6adDuvt3cco7dTvwNTQkszZX7CeajMfbeCUmEmuJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419845241 (572h59m23.290928276s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.419873174 (572h59m23.290956209s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419882021 (572h59m23.290965076s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420073716 (572h59m23.291156751s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection Dec 29 12:28:23.420082262 (572h59m23.291165307s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420257506 (572h59m23.291340551s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: read tcp 188.166.225.170:41270->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:28:23.420264649 (572h59m23.291347684s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420281822 (572h59m23.291364857s) [seg_pos={0 0}, upload=502322]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:28:23.420364349 (572h59m23.291447394s) [upload=502322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:28:23.420370270 (572h59m23.291453315s) [upload=502322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:28:23.725225770 (572h59m23.596308815s) [upload=502322]: Done uploading segments... Dec 29 12:28:23.725255426 (572h59m23.596338481s) [upload=502322]: Upload finished. err: <nil> Duration: 52.625973786s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [X...XXX...XXX..XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXXX] duration: 5.001366684s trace: https://jaeger.dev.storj.io/trace/29679e50f1de0451 FAIL (2025-12-29 12:30:17 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001391011s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X.X....XXXXXX..X.X.......X.XX...X.......X.X..XXX....X...X..] duration: 6.911981917s trace: https://jaeger.dev.storj.io/trace/dce2d9c07ebeda FAIL (2025-12-29 12:30:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 12:29:13.346940948 (573h0m13.218023983s) [upload=502335]: Starting upload Dec 29 12:29:13.347011723 (573h0m13.218094768s) [upload=502335]: Done starting upload Dec 29 12:29:13.347109640 (573h0m13.218192675s) [upload=502335]: Uploading segments... Dec 29 12:29:13.347238846 (573h0m13.218321881s) [upload=502335]: Got next segment. Inline: false Dec 29 12:29:13.347286016 (573h0m13.218369051s) [upload=502335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:29:13.347292307 (573h0m13.218375342s) [upload=502335]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:29:13.615185145 (573h0m13.48626821s) [seg_pos={0 0}, upload=502335]: Begin upload segment... Dec 29 12:29:13.615238265 (573h0m13.486321301s) [seg_pos={0 0}, upload=502335]: Done begin upload segment. Dec 29 12:29:13.615269796 (573h0m13.486352831s) [noise=true , address=145.79.227.28:4725 , piece_id=X65WZ5QAWLMOJMU7OZOF2T7LX45MRB4AMRGQMHH43LW55M4B2WFQ, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.615443436 (573h0m13.486526481s) [noise=true , address=45.91.116.12:4225 , piece_id=5QF5DK7QVJ67HT2GVFBIIXUMTUC6FVVD3RBVYHIVAUIH4VGVYOQA, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.616491041 (573h0m13.487574086s) [noise=true , address=145.79.227.48:4303 , piece_id=QIONIBQIO325OCUSNOVVOI3OOVACS35JDNXEZPSYLWMHG376UD7A, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.616837881 (573h0m13.487920917s) [noise=true , address=145.79.227.7:4293 , piece_id=GDJDYEOFO4FGQ3SLJH7JK5DUJQD2OHL2IOKBZPPNEJB63YTM7F4Q, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.617296364 (573h0m13.488379409s) [noise=true , address=145.79.227.17:4970 , piece_id=RU54FB4U3LR3DLMBFMPPCS6FXDU5I7QN5GF5TIFNUQK2LIRF7R7Q, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.617643375 (573h0m13.48872641s) [noise=true , address=145.79.227.37:4567 , piece_id=LS6QLDBRF22CSPBG42E4IC6T6T4GRPNQOBRDSKQBXIDGW5NA4XLA, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.617839659 (573h0m13.488922704s) [noise=true , address=145.79.227.16:4912 , piece_id=55RKZVL4OXEOP3S7K7GTKKINENABN5V73LO34ODYIUWFLH6Q7EPA, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.618093743 (573h0m13.489176778s) [noise=true , address=77.90.31.17:4770 , piece_id=IPQXLKPDNNHH6W4PWEMJ5LFWB4GY6M25IRUHP72QXTCRVYX2SHEA, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.618256753 (573h0m13.489339799s) [noise=true , address=145.79.227.52:4446 , piece_id=NSFB7ENJNRKUGHTYJRONCFUUG4PE7ULTBZ6LWMAPZKTM6JJBFTHQ, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.618478455 (573h0m13.4895615s) [noise=true , address=145.79.227.3:4151 , piece_id=2XZFZKIEVPR6PWICIMWRKSEQ3WSW5RPHSEAW4BX7FR2GDDN5TGQA, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.618713713 (573h0m13.489796759s) [noise=true , address=45.91.116.8:4046 , piece_id=GWIETJORTBYROVXTIVVXIPIQT3M5UKC2CYTCKPN6BT4UDEPOM3YQ, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.618933772 (573h0m13.490016817s) [noise=true , address=145.79.227.50:4374 , piece_id=Z3W7JTR65Y6N7MSITROEMJJ6NJ77IK2GKS44WMPEDP6J2TOE7HNA, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.619144123 (573h0m13.490227158s) [noise=true , address=145.79.227.45:4229 , piece_id=T43GEEDD5GABJYXB64TISMBX6TSX6YEOFKZ2S2ZQRZBMKH47SX4A, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.619427203 (573h0m13.490510248s) [noise=true , address=145.79.227.29:4942 , piece_id=4GY7GGGZX2F43BFA7AY5IIGTR7KTDDQZYVWTQNXONJKRVHAK7ZEQ, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.619559173 (573h0m13.490642208s) [noise=true , address=77.90.31.13:4629 , piece_id=WOM7R2YMBAT3FMJ7VCXERRX2KRGKDA2K5KHBYGZLQHWDTYAUKB2Q, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.619965547 (573h0m13.491048592s) [noise=true , address=145.79.227.18:4007 , piece_id=FYQRHTXX6KRYN6FQTJ6GQVUJZQKL2ZAOX2JL6ELFEX67X6CNPEJA, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:15.216249353 (573h0m15.087332388s) [upload=502335]: Next returned nil segment Dec 29 12:29:15.216283378 (573h0m15.087366413s) [upload=502335]: Waiting for error group managing segments... Dec 29 12:29:15.602828155 (573h0m15.4739112s) [noise=true , address=145.79.227.16:4912 , piece_id=55RKZVL4OXEOP3S7K7GTKKINENABN5V73LO34ODYIUWFLH6Q7EPA, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.603399572 (573h0m15.474482608s) [noise=true , address=145.79.227.7:4293 , piece_id=GDJDYEOFO4FGQ3SLJH7JK5DUJQD2OHL2IOKBZPPNEJB63YTM7F4Q, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.612424137 (573h0m15.483507172s) [noise=true , address=145.79.227.18:4007 , piece_id=FYQRHTXX6KRYN6FQTJ6GQVUJZQKL2ZAOX2JL6ELFEX67X6CNPEJA, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.617153132 (573h0m15.488236167s) [noise=true , address=145.79.227.50:4374 , piece_id=Z3W7JTR65Y6N7MSITROEMJJ6NJ77IK2GKS44WMPEDP6J2TOE7HNA, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.619789702 (573h0m15.490872747s) [noise=true , address=145.79.227.45:4229 , piece_id=T43GEEDD5GABJYXB64TISMBX6TSX6YEOFKZ2S2ZQRZBMKH47SX4A, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.620193140 (573h0m15.491276185s) [noise=true , address=145.79.227.48:4303 , piece_id=QIONIBQIO325OCUSNOVVOI3OOVACS35JDNXEZPSYLWMHG376UD7A, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.625756193 (573h0m15.496839238s) [noise=true , address=145.79.227.3:4151 , piece_id=2XZFZKIEVPR6PWICIMWRKSEQ3WSW5RPHSEAW4BX7FR2GDDN5TGQA, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.627063232 (573h0m15.498146267s) [noise=true , address=145.79.227.52:4446 , piece_id=NSFB7ENJNRKUGHTYJRONCFUUG4PE7ULTBZ6LWMAPZKTM6JJBFTHQ, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.627976201 (573h0m15.499059246s) [noise=true , address=145.79.227.28:4725 , piece_id=X65WZ5QAWLMOJMU7OZOF2T7LX45MRB4AMRGQMHH43LW55M4B2WFQ, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.628514746 (573h0m15.499597781s) [noise=true , address=145.79.227.37:4567 , piece_id=LS6QLDBRF22CSPBG42E4IC6T6T4GRPNQOBRDSKQBXIDGW5NA4XLA, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.919130091 (573h0m15.790213136s) [noise=true , address=77.90.31.13:4629 , piece_id=WOM7R2YMBAT3FMJ7VCXERRX2KRGKDA2K5KHBYGZLQHWDTYAUKB2Q, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:16.012809920 (573h0m15.883892966s) [noise=true , address=45.91.116.12:4225 , piece_id=5QF5DK7QVJ67HT2GVFBIIXUMTUC6FVVD3RBVYHIVAUIH4VGVYOQA, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:16.272699189 (573h0m16.143782234s) [noise=true , address=145.79.227.17:4970 , piece_id=RU54FB4U3LR3DLMBFMPPCS6FXDU5I7QN5GF5TIFNUQK2LIRF7R7Q, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:16.272717483 (573h0m16.143800518s) [seg_pos={0 0}, upload=502335]: Segment reached optimal threshold of 13 pieces. Dec 29 12:29:16.272868170 (573h0m16.143951205s) [noise=true , address=77.90.31.17:4770 , piece_id=IPQXLKPDNNHH6W4PWEMJ5LFWB4GY6M25IRUHP72QXTCRVYX2SHEA, seg_pos={0 0}, upload=502335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 29 12:29:16.272874262 (573h0m16.143957297s) [noise=true , address=77.90.31.17:4770 , piece_id=IPQXLKPDNNHH6W4PWEMJ5LFWB4GY6M25IRUHP72QXTCRVYX2SHEA, seg_pos={0 0}, upload=502335]: Long tail cancellation due to optimal threshold Dec 29 12:29:16.272899089 (573h0m16.143982124s) [noise=true , address=45.91.116.8:4046 , piece_id=GWIETJORTBYROVXTIVVXIPIQT3M5UKC2CYTCKPN6BT4UDEPOM3YQ, seg_pos={0 0}, upload=502335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 29 12:29:16.272907444 (573h0m16.143990479s) [noise=true , address=45.91.116.8:4046 , piece_id=GWIETJORTBYROVXTIVVXIPIQT3M5UKC2CYTCKPN6BT4UDEPOM3YQ, seg_pos={0 0}, upload=502335]: Long tail cancellation due to optimal threshold Dec 29 12:29:16.273088450 (573h0m16.144171485s) [noise=true , address=145.79.227.29:4942 , piece_id=4GY7GGGZX2F43BFA7AY5IIGTR7KTDDQZYVWTQNXONJKRVHAK7ZEQ, seg_pos={0 0}, upload=502335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 29 12:29:16.273094651 (573h0m16.144177686s) [noise=true , address=145.79.227.29:4942 , piece_id=4GY7GGGZX2F43BFA7AY5IIGTR7KTDDQZYVWTQNXONJKRVHAK7ZEQ, seg_pos={0 0}, upload=502335]: Long tail cancellation due to optimal threshold Dec 29 12:29:16.273114569 (573h0m16.144197604s) [seg_pos={0 0}, upload=502335]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:29:16.273190043 (573h0m16.144273078s) [upload=502335]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:29:16.273194472 (573h0m16.144277517s) [upload=502335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:29:16.582350199 (573h0m16.453433235s) [upload=502335]: Done uploading segments... Dec 29 12:29:16.582376960 (573h0m16.453460035s) [upload=502335]: Upload finished. err: <nil> Duration: 6.912012405s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................X.......X..........X.] duration: 41.798414901s trace: https://jaeger.dev.storj.io/trace/68764c6cb4286314 FAIL (2025-12-29 12:30:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528204662 (572h59m20.399287707s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528497489 (572h59m20.399580524s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528793153 (572h59m20.399876198s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529083555 (572h59m20.40016659s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529391081 (572h59m20.400474126s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529735096 (572h59m20.400818131s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530109970 (572h59m20.401193015s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530449556 (572h59m20.401532591s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530831554 (572h59m20.401914599s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531229512 (572h59m20.402312557s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531698945 (572h59m20.402781991s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532204238 (572h59m20.403287283s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532744366 (572h59m20.403827401s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533231203 (572h59m20.404314248s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533740113 (572h59m20.404823148s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:21.876539279 (572h59m21.747622324s) [upload=502322]: Next returned nil segment Dec 29 12:28:21.876614272 (572h59m21.747697307s) [upload=502322]: Waiting for error group managing segments... Dec 29 12:28:22.383674429 (572h59m22.254757464s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389083709 (572h59m22.260166754s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389450968 (572h59m22.260534003s) [noise=true , address=145.79.227.46:4625 , piece_id=6NEND2CKGIZLK5Y6IMM4XB6SXPSXUYSZIVZ7CZ25IKAHFFW33C5Q, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.418373151 (572h59m22.289456186s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.619432477 (572h59m22.490515552s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.741876388 (572h59m22.612959423s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.133731187 (572h59m23.004814222s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.146041929 (572h59m23.017124964s) [noise=true , address=193.149.164.143:4697 , piece_id=HZZY3SUOEC7AIHSW6MKZRNZVSOGXSYBYSUKN7STBFTXWAOMDRYBA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.261354601 (572h59m23.132437636s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.291387197 (572h59m23.162470243s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.318370928 (572h59m23.189453973s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.321236023 (572h59m23.192319058s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.408408673 (572h59m23.279491718s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419570137 (572h59m23.290653172s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419592470 (572h59m23.290675515s) [seg_pos={0 0}, upload=502322]: Segment reached optimal threshold of 14 pieces. Dec 29 12:28:23.419837406 (572h59m23.290920441s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123jUFZTVjq6adDuvt3cco7dTvwNTQkszZX7CeajMfbeCUmEmuJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419845241 (572h59m23.290928276s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.419873174 (572h59m23.290956209s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419882021 (572h59m23.290965076s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420073716 (572h59m23.291156751s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection Dec 29 12:28:23.420082262 (572h59m23.291165307s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420257506 (572h59m23.291340551s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: read tcp 188.166.225.170:41270->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:28:23.420264649 (572h59m23.291347684s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420281822 (572h59m23.291364857s) [seg_pos={0 0}, upload=502322]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:28:23.420364349 (572h59m23.291447394s) [upload=502322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:28:23.420370270 (572h59m23.291453315s) [upload=502322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:28:23.725225770 (572h59m23.596308815s) [upload=502322]: Done uploading segments... Dec 29 12:28:23.725255426 (572h59m23.596338481s) [upload=502322]: Upload finished. err: <nil> Duration: 41.798438395s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [XX...XXX...XXX..XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXX] duration: 5.00164341s trace: https://jaeger.dev.storj.io/trace/67bd20ee19569f03 FAIL (2025-12-29 12:29:55 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001665703s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X.X.X....XXXXXX..X.X.......X.XX...X.......X.X..XXX....X...X.] duration: 6.00128071s trace: https://jaeger.dev.storj.io/trace/dce2d9c07ebeda FAIL (2025-12-29 12:29:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 12:29:13.346940948 (573h0m13.218023983s) [upload=502335]: Starting upload Dec 29 12:29:13.347011723 (573h0m13.218094768s) [upload=502335]: Done starting upload Dec 29 12:29:13.347109640 (573h0m13.218192675s) [upload=502335]: Uploading segments... Dec 29 12:29:13.347238846 (573h0m13.218321881s) [upload=502335]: Got next segment. Inline: false Dec 29 12:29:13.347286016 (573h0m13.218369051s) [upload=502335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:29:13.347292307 (573h0m13.218375342s) [upload=502335]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:29:13.615185145 (573h0m13.48626821s) [seg_pos={0 0}, upload=502335]: Begin upload segment... Dec 29 12:29:13.615238265 (573h0m13.486321301s) [seg_pos={0 0}, upload=502335]: Done begin upload segment. Dec 29 12:29:13.615269796 (573h0m13.486352831s) [noise=true , address=145.79.227.28:4725 , piece_id=X65WZ5QAWLMOJMU7OZOF2T7LX45MRB4AMRGQMHH43LW55M4B2WFQ, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.615443436 (573h0m13.486526481s) [noise=true , address=45.91.116.12:4225 , piece_id=5QF5DK7QVJ67HT2GVFBIIXUMTUC6FVVD3RBVYHIVAUIH4VGVYOQA, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.616491041 (573h0m13.487574086s) [noise=true , address=145.79.227.48:4303 , piece_id=QIONIBQIO325OCUSNOVVOI3OOVACS35JDNXEZPSYLWMHG376UD7A, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.616837881 (573h0m13.487920917s) [noise=true , address=145.79.227.7:4293 , piece_id=GDJDYEOFO4FGQ3SLJH7JK5DUJQD2OHL2IOKBZPPNEJB63YTM7F4Q, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.617296364 (573h0m13.488379409s) [noise=true , address=145.79.227.17:4970 , piece_id=RU54FB4U3LR3DLMBFMPPCS6FXDU5I7QN5GF5TIFNUQK2LIRF7R7Q, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.617643375 (573h0m13.48872641s) [noise=true , address=145.79.227.37:4567 , piece_id=LS6QLDBRF22CSPBG42E4IC6T6T4GRPNQOBRDSKQBXIDGW5NA4XLA, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.617839659 (573h0m13.488922704s) [noise=true , address=145.79.227.16:4912 , piece_id=55RKZVL4OXEOP3S7K7GTKKINENABN5V73LO34ODYIUWFLH6Q7EPA, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.618093743 (573h0m13.489176778s) [noise=true , address=77.90.31.17:4770 , piece_id=IPQXLKPDNNHH6W4PWEMJ5LFWB4GY6M25IRUHP72QXTCRVYX2SHEA, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.618256753 (573h0m13.489339799s) [noise=true , address=145.79.227.52:4446 , piece_id=NSFB7ENJNRKUGHTYJRONCFUUG4PE7ULTBZ6LWMAPZKTM6JJBFTHQ, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.618478455 (573h0m13.4895615s) [noise=true , address=145.79.227.3:4151 , piece_id=2XZFZKIEVPR6PWICIMWRKSEQ3WSW5RPHSEAW4BX7FR2GDDN5TGQA, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.618713713 (573h0m13.489796759s) [noise=true , address=45.91.116.8:4046 , piece_id=GWIETJORTBYROVXTIVVXIPIQT3M5UKC2CYTCKPN6BT4UDEPOM3YQ, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.618933772 (573h0m13.490016817s) [noise=true , address=145.79.227.50:4374 , piece_id=Z3W7JTR65Y6N7MSITROEMJJ6NJ77IK2GKS44WMPEDP6J2TOE7HNA, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.619144123 (573h0m13.490227158s) [noise=true , address=145.79.227.45:4229 , piece_id=T43GEEDD5GABJYXB64TISMBX6TSX6YEOFKZ2S2ZQRZBMKH47SX4A, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.619427203 (573h0m13.490510248s) [noise=true , address=145.79.227.29:4942 , piece_id=4GY7GGGZX2F43BFA7AY5IIGTR7KTDDQZYVWTQNXONJKRVHAK7ZEQ, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.619559173 (573h0m13.490642208s) [noise=true , address=77.90.31.13:4629 , piece_id=WOM7R2YMBAT3FMJ7VCXERRX2KRGKDA2K5KHBYGZLQHWDTYAUKB2Q, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.619965547 (573h0m13.491048592s) [noise=true , address=145.79.227.18:4007 , piece_id=FYQRHTXX6KRYN6FQTJ6GQVUJZQKL2ZAOX2JL6ELFEX67X6CNPEJA, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:15.216249353 (573h0m15.087332388s) [upload=502335]: Next returned nil segment Dec 29 12:29:15.216283378 (573h0m15.087366413s) [upload=502335]: Waiting for error group managing segments... Dec 29 12:29:15.602828155 (573h0m15.4739112s) [noise=true , address=145.79.227.16:4912 , piece_id=55RKZVL4OXEOP3S7K7GTKKINENABN5V73LO34ODYIUWFLH6Q7EPA, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.603399572 (573h0m15.474482608s) [noise=true , address=145.79.227.7:4293 , piece_id=GDJDYEOFO4FGQ3SLJH7JK5DUJQD2OHL2IOKBZPPNEJB63YTM7F4Q, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.612424137 (573h0m15.483507172s) [noise=true , address=145.79.227.18:4007 , piece_id=FYQRHTXX6KRYN6FQTJ6GQVUJZQKL2ZAOX2JL6ELFEX67X6CNPEJA, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.617153132 (573h0m15.488236167s) [noise=true , address=145.79.227.50:4374 , piece_id=Z3W7JTR65Y6N7MSITROEMJJ6NJ77IK2GKS44WMPEDP6J2TOE7HNA, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.619789702 (573h0m15.490872747s) [noise=true , address=145.79.227.45:4229 , piece_id=T43GEEDD5GABJYXB64TISMBX6TSX6YEOFKZ2S2ZQRZBMKH47SX4A, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.620193140 (573h0m15.491276185s) [noise=true , address=145.79.227.48:4303 , piece_id=QIONIBQIO325OCUSNOVVOI3OOVACS35JDNXEZPSYLWMHG376UD7A, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.625756193 (573h0m15.496839238s) [noise=true , address=145.79.227.3:4151 , piece_id=2XZFZKIEVPR6PWICIMWRKSEQ3WSW5RPHSEAW4BX7FR2GDDN5TGQA, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.627063232 (573h0m15.498146267s) [noise=true , address=145.79.227.52:4446 , piece_id=NSFB7ENJNRKUGHTYJRONCFUUG4PE7ULTBZ6LWMAPZKTM6JJBFTHQ, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.627976201 (573h0m15.499059246s) [noise=true , address=145.79.227.28:4725 , piece_id=X65WZ5QAWLMOJMU7OZOF2T7LX45MRB4AMRGQMHH43LW55M4B2WFQ, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.628514746 (573h0m15.499597781s) [noise=true , address=145.79.227.37:4567 , piece_id=LS6QLDBRF22CSPBG42E4IC6T6T4GRPNQOBRDSKQBXIDGW5NA4XLA, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.919130091 (573h0m15.790213136s) [noise=true , address=77.90.31.13:4629 , piece_id=WOM7R2YMBAT3FMJ7VCXERRX2KRGKDA2K5KHBYGZLQHWDTYAUKB2Q, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:16.012809920 (573h0m15.883892966s) [noise=true , address=45.91.116.12:4225 , piece_id=5QF5DK7QVJ67HT2GVFBIIXUMTUC6FVVD3RBVYHIVAUIH4VGVYOQA, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:16.272699189 (573h0m16.143782234s) [noise=true , address=145.79.227.17:4970 , piece_id=RU54FB4U3LR3DLMBFMPPCS6FXDU5I7QN5GF5TIFNUQK2LIRF7R7Q, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:16.272717483 (573h0m16.143800518s) [seg_pos={0 0}, upload=502335]: Segment reached optimal threshold of 13 pieces. Dec 29 12:29:16.272868170 (573h0m16.143951205s) [noise=true , address=77.90.31.17:4770 , piece_id=IPQXLKPDNNHH6W4PWEMJ5LFWB4GY6M25IRUHP72QXTCRVYX2SHEA, seg_pos={0 0}, upload=502335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 29 12:29:16.272874262 (573h0m16.143957297s) [noise=true , address=77.90.31.17:4770 , piece_id=IPQXLKPDNNHH6W4PWEMJ5LFWB4GY6M25IRUHP72QXTCRVYX2SHEA, seg_pos={0 0}, upload=502335]: Long tail cancellation due to optimal threshold Dec 29 12:29:16.272899089 (573h0m16.143982124s) [noise=true , address=45.91.116.8:4046 , piece_id=GWIETJORTBYROVXTIVVXIPIQT3M5UKC2CYTCKPN6BT4UDEPOM3YQ, seg_pos={0 0}, upload=502335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 29 12:29:16.272907444 (573h0m16.143990479s) [noise=true , address=45.91.116.8:4046 , piece_id=GWIETJORTBYROVXTIVVXIPIQT3M5UKC2CYTCKPN6BT4UDEPOM3YQ, seg_pos={0 0}, upload=502335]: Long tail cancellation due to optimal threshold Dec 29 12:29:16.273088450 (573h0m16.144171485s) [noise=true , address=145.79.227.29:4942 , piece_id=4GY7GGGZX2F43BFA7AY5IIGTR7KTDDQZYVWTQNXONJKRVHAK7ZEQ, seg_pos={0 0}, upload=502335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 29 12:29:16.273094651 (573h0m16.144177686s) [noise=true , address=145.79.227.29:4942 , piece_id=4GY7GGGZX2F43BFA7AY5IIGTR7KTDDQZYVWTQNXONJKRVHAK7ZEQ, seg_pos={0 0}, upload=502335]: Long tail cancellation due to optimal threshold Dec 29 12:29:16.273114569 (573h0m16.144197604s) [seg_pos={0 0}, upload=502335]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:29:16.273190043 (573h0m16.144273078s) [upload=502335]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:29:16.273194472 (573h0m16.144277517s) [upload=502335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:29:16.582350199 (573h0m16.453433235s) [upload=502335]: Done uploading segments... Dec 29 12:29:16.582376960 (573h0m16.453460035s) [upload=502335]: Upload finished. err: <nil> Duration: 6.001294135s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XX.X.X....XXXXXX..X.X.......X.XX...X.......X.X..XXX....X...X] duration: 30.004059393s trace: https://jaeger.dev.storj.io/trace/dce2d9c07ebeda FAIL (2025-12-29 12:29:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 12:29:13.346940948 (573h0m13.218023983s) [upload=502335]: Starting upload Dec 29 12:29:13.347011723 (573h0m13.218094768s) [upload=502335]: Done starting upload Dec 29 12:29:13.347109640 (573h0m13.218192675s) [upload=502335]: Uploading segments... Dec 29 12:29:13.347238846 (573h0m13.218321881s) [upload=502335]: Got next segment. Inline: false Dec 29 12:29:13.347286016 (573h0m13.218369051s) [upload=502335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:29:13.347292307 (573h0m13.218375342s) [upload=502335]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:29:13.615185145 (573h0m13.48626821s) [seg_pos={0 0}, upload=502335]: Begin upload segment... Dec 29 12:29:13.615238265 (573h0m13.486321301s) [seg_pos={0 0}, upload=502335]: Done begin upload segment. Dec 29 12:29:13.615269796 (573h0m13.486352831s) [noise=true , address=145.79.227.28:4725 , piece_id=X65WZ5QAWLMOJMU7OZOF2T7LX45MRB4AMRGQMHH43LW55M4B2WFQ, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.615443436 (573h0m13.486526481s) [noise=true , address=45.91.116.12:4225 , piece_id=5QF5DK7QVJ67HT2GVFBIIXUMTUC6FVVD3RBVYHIVAUIH4VGVYOQA, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.616491041 (573h0m13.487574086s) [noise=true , address=145.79.227.48:4303 , piece_id=QIONIBQIO325OCUSNOVVOI3OOVACS35JDNXEZPSYLWMHG376UD7A, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.616837881 (573h0m13.487920917s) [noise=true , address=145.79.227.7:4293 , piece_id=GDJDYEOFO4FGQ3SLJH7JK5DUJQD2OHL2IOKBZPPNEJB63YTM7F4Q, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.617296364 (573h0m13.488379409s) [noise=true , address=145.79.227.17:4970 , piece_id=RU54FB4U3LR3DLMBFMPPCS6FXDU5I7QN5GF5TIFNUQK2LIRF7R7Q, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.617643375 (573h0m13.48872641s) [noise=true , address=145.79.227.37:4567 , piece_id=LS6QLDBRF22CSPBG42E4IC6T6T4GRPNQOBRDSKQBXIDGW5NA4XLA, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.617839659 (573h0m13.488922704s) [noise=true , address=145.79.227.16:4912 , piece_id=55RKZVL4OXEOP3S7K7GTKKINENABN5V73LO34ODYIUWFLH6Q7EPA, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.618093743 (573h0m13.489176778s) [noise=true , address=77.90.31.17:4770 , piece_id=IPQXLKPDNNHH6W4PWEMJ5LFWB4GY6M25IRUHP72QXTCRVYX2SHEA, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.618256753 (573h0m13.489339799s) [noise=true , address=145.79.227.52:4446 , piece_id=NSFB7ENJNRKUGHTYJRONCFUUG4PE7ULTBZ6LWMAPZKTM6JJBFTHQ, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.618478455 (573h0m13.4895615s) [noise=true , address=145.79.227.3:4151 , piece_id=2XZFZKIEVPR6PWICIMWRKSEQ3WSW5RPHSEAW4BX7FR2GDDN5TGQA, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.618713713 (573h0m13.489796759s) [noise=true , address=45.91.116.8:4046 , piece_id=GWIETJORTBYROVXTIVVXIPIQT3M5UKC2CYTCKPN6BT4UDEPOM3YQ, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.618933772 (573h0m13.490016817s) [noise=true , address=145.79.227.50:4374 , piece_id=Z3W7JTR65Y6N7MSITROEMJJ6NJ77IK2GKS44WMPEDP6J2TOE7HNA, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.619144123 (573h0m13.490227158s) [noise=true , address=145.79.227.45:4229 , piece_id=T43GEEDD5GABJYXB64TISMBX6TSX6YEOFKZ2S2ZQRZBMKH47SX4A, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.619427203 (573h0m13.490510248s) [noise=true , address=145.79.227.29:4942 , piece_id=4GY7GGGZX2F43BFA7AY5IIGTR7KTDDQZYVWTQNXONJKRVHAK7ZEQ, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.619559173 (573h0m13.490642208s) [noise=true , address=77.90.31.13:4629 , piece_id=WOM7R2YMBAT3FMJ7VCXERRX2KRGKDA2K5KHBYGZLQHWDTYAUKB2Q, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.619965547 (573h0m13.491048592s) [noise=true , address=145.79.227.18:4007 , piece_id=FYQRHTXX6KRYN6FQTJ6GQVUJZQKL2ZAOX2JL6ELFEX67X6CNPEJA, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:15.216249353 (573h0m15.087332388s) [upload=502335]: Next returned nil segment Dec 29 12:29:15.216283378 (573h0m15.087366413s) [upload=502335]: Waiting for error group managing segments... Dec 29 12:29:15.602828155 (573h0m15.4739112s) [noise=true , address=145.79.227.16:4912 , piece_id=55RKZVL4OXEOP3S7K7GTKKINENABN5V73LO34ODYIUWFLH6Q7EPA, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.603399572 (573h0m15.474482608s) [noise=true , address=145.79.227.7:4293 , piece_id=GDJDYEOFO4FGQ3SLJH7JK5DUJQD2OHL2IOKBZPPNEJB63YTM7F4Q, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.612424137 (573h0m15.483507172s) [noise=true , address=145.79.227.18:4007 , piece_id=FYQRHTXX6KRYN6FQTJ6GQVUJZQKL2ZAOX2JL6ELFEX67X6CNPEJA, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.617153132 (573h0m15.488236167s) [noise=true , address=145.79.227.50:4374 , piece_id=Z3W7JTR65Y6N7MSITROEMJJ6NJ77IK2GKS44WMPEDP6J2TOE7HNA, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.619789702 (573h0m15.490872747s) [noise=true , address=145.79.227.45:4229 , piece_id=T43GEEDD5GABJYXB64TISMBX6TSX6YEOFKZ2S2ZQRZBMKH47SX4A, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.620193140 (573h0m15.491276185s) [noise=true , address=145.79.227.48:4303 , piece_id=QIONIBQIO325OCUSNOVVOI3OOVACS35JDNXEZPSYLWMHG376UD7A, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.625756193 (573h0m15.496839238s) [noise=true , address=145.79.227.3:4151 , piece_id=2XZFZKIEVPR6PWICIMWRKSEQ3WSW5RPHSEAW4BX7FR2GDDN5TGQA, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.627063232 (573h0m15.498146267s) [noise=true , address=145.79.227.52:4446 , piece_id=NSFB7ENJNRKUGHTYJRONCFUUG4PE7ULTBZ6LWMAPZKTM6JJBFTHQ, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.627976201 (573h0m15.499059246s) [noise=true , address=145.79.227.28:4725 , piece_id=X65WZ5QAWLMOJMU7OZOF2T7LX45MRB4AMRGQMHH43LW55M4B2WFQ, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.628514746 (573h0m15.499597781s) [noise=true , address=145.79.227.37:4567 , piece_id=LS6QLDBRF22CSPBG42E4IC6T6T4GRPNQOBRDSKQBXIDGW5NA4XLA, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.919130091 (573h0m15.790213136s) [noise=true , address=77.90.31.13:4629 , piece_id=WOM7R2YMBAT3FMJ7VCXERRX2KRGKDA2K5KHBYGZLQHWDTYAUKB2Q, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:16.012809920 (573h0m15.883892966s) [noise=true , address=45.91.116.12:4225 , piece_id=5QF5DK7QVJ67HT2GVFBIIXUMTUC6FVVD3RBVYHIVAUIH4VGVYOQA, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:16.272699189 (573h0m16.143782234s) [noise=true , address=145.79.227.17:4970 , piece_id=RU54FB4U3LR3DLMBFMPPCS6FXDU5I7QN5GF5TIFNUQK2LIRF7R7Q, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:16.272717483 (573h0m16.143800518s) [seg_pos={0 0}, upload=502335]: Segment reached optimal threshold of 13 pieces. Dec 29 12:29:16.272868170 (573h0m16.143951205s) [noise=true , address=77.90.31.17:4770 , piece_id=IPQXLKPDNNHH6W4PWEMJ5LFWB4GY6M25IRUHP72QXTCRVYX2SHEA, seg_pos={0 0}, upload=502335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 29 12:29:16.272874262 (573h0m16.143957297s) [noise=true , address=77.90.31.17:4770 , piece_id=IPQXLKPDNNHH6W4PWEMJ5LFWB4GY6M25IRUHP72QXTCRVYX2SHEA, seg_pos={0 0}, upload=502335]: Long tail cancellation due to optimal threshold Dec 29 12:29:16.272899089 (573h0m16.143982124s) [noise=true , address=45.91.116.8:4046 , piece_id=GWIETJORTBYROVXTIVVXIPIQT3M5UKC2CYTCKPN6BT4UDEPOM3YQ, seg_pos={0 0}, upload=502335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 29 12:29:16.272907444 (573h0m16.143990479s) [noise=true , address=45.91.116.8:4046 , piece_id=GWIETJORTBYROVXTIVVXIPIQT3M5UKC2CYTCKPN6BT4UDEPOM3YQ, seg_pos={0 0}, upload=502335]: Long tail cancellation due to optimal threshold Dec 29 12:29:16.273088450 (573h0m16.144171485s) [noise=true , address=145.79.227.29:4942 , piece_id=4GY7GGGZX2F43BFA7AY5IIGTR7KTDDQZYVWTQNXONJKRVHAK7ZEQ, seg_pos={0 0}, upload=502335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 29 12:29:16.273094651 (573h0m16.144177686s) [noise=true , address=145.79.227.29:4942 , piece_id=4GY7GGGZX2F43BFA7AY5IIGTR7KTDDQZYVWTQNXONJKRVHAK7ZEQ, seg_pos={0 0}, upload=502335]: Long tail cancellation due to optimal threshold Dec 29 12:29:16.273114569 (573h0m16.144197604s) [seg_pos={0 0}, upload=502335]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:29:16.273190043 (573h0m16.144273078s) [upload=502335]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:29:16.273194472 (573h0m16.144277517s) [upload=502335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:29:16.582350199 (573h0m16.453433235s) [upload=502335]: Done uploading segments... Dec 29 12:29:16.582376960 (573h0m16.453460035s) [upload=502335]: Upload finished. err: <nil> Duration: 30.004080995s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................X.......X..........X] duration: 1m0.002316081s trace: https://jaeger.dev.storj.io/trace/68764c6cb4286314 FAIL (2025-12-29 12:29:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528204662 (572h59m20.399287707s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528497489 (572h59m20.399580524s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528793153 (572h59m20.399876198s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529083555 (572h59m20.40016659s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529391081 (572h59m20.400474126s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529735096 (572h59m20.400818131s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530109970 (572h59m20.401193015s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530449556 (572h59m20.401532591s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530831554 (572h59m20.401914599s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531229512 (572h59m20.402312557s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531698945 (572h59m20.402781991s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532204238 (572h59m20.403287283s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532744366 (572h59m20.403827401s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533231203 (572h59m20.404314248s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533740113 (572h59m20.404823148s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:21.876539279 (572h59m21.747622324s) [upload=502322]: Next returned nil segment Dec 29 12:28:21.876614272 (572h59m21.747697307s) [upload=502322]: Waiting for error group managing segments... Dec 29 12:28:22.383674429 (572h59m22.254757464s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389083709 (572h59m22.260166754s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389450968 (572h59m22.260534003s) [noise=true , address=145.79.227.46:4625 , piece_id=6NEND2CKGIZLK5Y6IMM4XB6SXPSXUYSZIVZ7CZ25IKAHFFW33C5Q, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.418373151 (572h59m22.289456186s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.619432477 (572h59m22.490515552s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.741876388 (572h59m22.612959423s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.133731187 (572h59m23.004814222s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.146041929 (572h59m23.017124964s) [noise=true , address=193.149.164.143:4697 , piece_id=HZZY3SUOEC7AIHSW6MKZRNZVSOGXSYBYSUKN7STBFTXWAOMDRYBA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.261354601 (572h59m23.132437636s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.291387197 (572h59m23.162470243s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.318370928 (572h59m23.189453973s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.321236023 (572h59m23.192319058s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.408408673 (572h59m23.279491718s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419570137 (572h59m23.290653172s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419592470 (572h59m23.290675515s) [seg_pos={0 0}, upload=502322]: Segment reached optimal threshold of 14 pieces. Dec 29 12:28:23.419837406 (572h59m23.290920441s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123jUFZTVjq6adDuvt3cco7dTvwNTQkszZX7CeajMfbeCUmEmuJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419845241 (572h59m23.290928276s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.419873174 (572h59m23.290956209s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419882021 (572h59m23.290965076s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420073716 (572h59m23.291156751s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection Dec 29 12:28:23.420082262 (572h59m23.291165307s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420257506 (572h59m23.291340551s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: read tcp 188.166.225.170:41270->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:28:23.420264649 (572h59m23.291347684s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420281822 (572h59m23.291364857s) [seg_pos={0 0}, upload=502322]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:28:23.420364349 (572h59m23.291447394s) [upload=502322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:28:23.420370270 (572h59m23.291453315s) [upload=502322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:28:23.725225770 (572h59m23.596308815s) [upload=502322]: Done uploading segments... Dec 29 12:28:23.725255426 (572h59m23.596338481s) [upload=502322]: Upload finished. err: <nil> Duration: 1m0.002339796s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [XXX...XXX...XXX..XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXX] duration: 5.000788065s trace: https://jaeger.dev.storj.io/trace/20c6fddbfe81bd4d FAIL (2025-12-29 12:28:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000809857s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XXX.X.X....XXXXXX..X.X.......X.XX...X.......X.X..XXX....X...] duration: 5.697337436s trace: https://jaeger.dev.storj.io/trace/6047085ef9629cee FAIL (2025-12-29 12:28:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng segments... Dec 29 12:26:26.381455222 (572h57m26.252538267s) [upload=502294]: Got next segment. Inline: false Dec 29 12:26:26.381569239 (572h57m26.252652274s) [upload=502294]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:26:26.381575021 (572h57m26.252658066s) [upload=502294]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:26:26.625888218 (572h57m26.496971263s) [seg_pos={0 0}, upload=502294]: Begin upload segment... Dec 29 12:26:26.625973239 (572h57m26.497056284s) [seg_pos={0 0}, upload=502294]: Done begin upload segment. Dec 29 12:26:26.626040047 (572h57m26.497123102s) [noise=true , address=77.90.31.10:4520 , piece_id=IHPHB6JKOOWSS5ELL4HXBIYJ3O5UIK4IDKIYVEWAIXH57KFBW6NQ, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.626816195 (572h57m26.49789924s) [noise=true , address=145.79.227.15:4874 , piece_id=E3AAB4JEHERJ2R6HGP4MLRNXNMK7W4ICRPWIFWUMQ3OJNRHYOP5Q, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.627162455 (572h57m26.4982455s) [noise=true , address=145.79.227.40:4049 , piece_id=CNLK7WROCZCPC2PKAQDKYQXUL6ZUBGD4E4E3E6PPVYEIS5ITLE7Q, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.627543349 (572h57m26.498626384s) [noise=true , address=45.91.116.9:4262 , piece_id=JFLGJIBVJXSJE54ECBS4U33LARKUHPBAQB6NUTONBHMBWBM6H6XA, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.627886933 (572h57m26.498969988s) [noise=true , address=145.79.227.38:4977 , piece_id=V3H2S43AEUMFLLDTXSGCA6BFDJUPZLKAUDB3RWJVG3PK6AEPXY6A, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.628205891 (572h57m26.499288937s) [noise=true , address=145.79.227.47:4266 , piece_id=Q2FMIE5XX4LZD4GI6SSZESJZVMZBE4KGPC2H36EY7KY65EO5G3QA, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.628648774 (572h57m26.499731809s) [noise=true , address=145.79.227.42:4123 , piece_id=7EHC46NCH7XN7ZPPPM2ROQ2PSHOYAHWR4EGUB76X3Y62OFOPAPTA, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.628954356 (572h57m26.500037401s) [noise=true , address=145.79.227.45:4229 , piece_id=P5OCXGBPPBKMYJTGYCGYNO5WGIMZLFVZLUCB3GUQCJIRG637633Q, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.630379450 (572h57m26.501462495s) [noise=true , address=45.91.116.6:4976 , piece_id=SBTB7O3HFTFSG6CMUDHRDSNFT4HRGM522FNUOZAUOKV2KPGOKGTQ, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.630707424 (572h57m26.501790459s) [noise=true , address=145.79.227.14:4837 , piece_id=SQCFNKFPX7KK4LKL52NVS2KN32SWSEVPGWWUTTVW33DKCN4JILNA, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.631005512 (572h57m26.502088557s) [noise=true , address=145.79.227.2:4115 , piece_id=BXBFY2EP67QEZAX56ASJINVOZGAO6H2XAD3SBIR7MVA6VUFECNZA, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.631272210 (572h57m26.502355255s) [noise=true , address=77.90.31.14:4662 , piece_id=G3L5FXFH2HS3AR5XLPGHL6XL4FJ7EUBHYU7AF3L3FIJKOPBBDQTA, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.632025514 (572h57m26.503108559s) [noise=true , address=45.91.116.13:4156 , piece_id=O2FEYLUXWNE6O4WFP3PHDBA5DMLV5X7Q35N6O7ADD5OHV6XRVC4A, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.632463408 (572h57m26.503546453s) [noise=true , address=45.91.116.7:4009 , piece_id=CTN2ZM4XBB5XHQFHOAPW6IQ5EB773ZPBW7TOBV74V2F3SM25AZWA, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.633007343 (572h57m26.504090388s) [noise=true , address=45.91.116.4:4940 , piece_id=3676OHD74HKXC6AB5LPJEE3SAVVZZ36BGLWG56QIVFETU4C44DJQ, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.633424848 (572h57m26.504507883s) [noise=true , address=145.79.227.12:4767 , piece_id=35QG5CVAG33YCHPXLEUMCVGW23UFPGAPX6S4TEEASSCNPVO4HYAA, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:28.330358171 (572h57m28.201441206s) [upload=502294]: Next returned nil segment Dec 29 12:26:28.330375173 (572h57m28.201458218s) [upload=502294]: Waiting for error group managing segments... Dec 29 12:26:28.603285514 (572h57m28.47436855s) [noise=true , address=145.79.227.12:4767 , piece_id=35QG5CVAG33YCHPXLEUMCVGW23UFPGAPX6S4TEEASSCNPVO4HYAA, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:28.618055339 (572h57m28.489138374s) [noise=true , address=145.79.227.42:4123 , piece_id=7EHC46NCH7XN7ZPPPM2ROQ2PSHOYAHWR4EGUB76X3Y62OFOPAPTA, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:28.618182089 (572h57m28.489265124s) [noise=true , address=145.79.227.45:4229 , piece_id=P5OCXGBPPBKMYJTGYCGYNO5WGIMZLFVZLUCB3GUQCJIRG637633Q, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:28.618845883 (572h57m28.489928928s) [noise=true , address=145.79.227.38:4977 , piece_id=V3H2S43AEUMFLLDTXSGCA6BFDJUPZLKAUDB3RWJVG3PK6AEPXY6A, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:28.620502508 (572h57m28.491585543s) [noise=true , address=145.79.227.15:4874 , piece_id=E3AAB4JEHERJ2R6HGP4MLRNXNMK7W4ICRPWIFWUMQ3OJNRHYOP5Q, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:28.634073068 (572h57m28.505156103s) [noise=true , address=145.79.227.14:4837 , piece_id=SQCFNKFPX7KK4LKL52NVS2KN32SWSEVPGWWUTTVW33DKCN4JILNA, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:28.635391348 (572h57m28.506474383s) [noise=true , address=145.79.227.2:4115 , piece_id=BXBFY2EP67QEZAX56ASJINVOZGAO6H2XAD3SBIR7MVA6VUFECNZA, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:28.645652678 (572h57m28.516735713s) [noise=true , address=145.79.227.47:4266 , piece_id=Q2FMIE5XX4LZD4GI6SSZESJZVMZBE4KGPC2H36EY7KY65EO5G3QA, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:28.804359379 (572h57m28.675442424s) [noise=true , address=77.90.31.14:4662 , piece_id=G3L5FXFH2HS3AR5XLPGHL6XL4FJ7EUBHYU7AF3L3FIJKOPBBDQTA, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:28.815560217 (572h57m28.686643262s) [noise=true , address=77.90.31.10:4520 , piece_id=IHPHB6JKOOWSS5ELL4HXBIYJ3O5UIK4IDKIYVEWAIXH57KFBW6NQ, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:28.944227930 (572h57m28.815310965s) [noise=true , address=45.91.116.13:4156 , piece_id=O2FEYLUXWNE6O4WFP3PHDBA5DMLV5X7Q35N6O7ADD5OHV6XRVC4A, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:29.018960361 (572h57m28.890043406s) [noise=true , address=45.91.116.9:4262 , piece_id=JFLGJIBVJXSJE54ECBS4U33LARKUHPBAQB6NUTONBHMBWBM6H6XA, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:29.040090536 (572h57m28.911173581s) [noise=true , address=45.91.116.6:4976 , piece_id=SBTB7O3HFTFSG6CMUDHRDSNFT4HRGM522FNUOZAUOKV2KPGOKGTQ, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:29.040113139 (572h57m28.911196184s) [seg_pos={0 0}, upload=502294]: Segment reached optimal threshold of 13 pieces. Dec 29 12:26:29.040314813 (572h57m28.911397858s) [noise=true , address=45.91.116.7:4009 , piece_id=CTN2ZM4XBB5XHQFHOAPW6IQ5EB773ZPBW7TOBV74V2F3SM25AZWA, seg_pos={0 0}, upload=502294]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:43924->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:43924->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43924->45.91.116.7:4009: use of closed network connection Dec 29 12:26:29.040322277 (572h57m28.911405312s) [noise=true , address=45.91.116.7:4009 , piece_id=CTN2ZM4XBB5XHQFHOAPW6IQ5EB773ZPBW7TOBV74V2F3SM25AZWA, seg_pos={0 0}, upload=502294]: Long tail cancellation due to optimal threshold Dec 29 12:26:29.040594947 (572h57m28.911677992s) [noise=true , address=45.91.116.4:4940 , piece_id=3676OHD74HKXC6AB5LPJEE3SAVVZZ36BGLWG56QIVFETU4C44DJQ, seg_pos={0 0}, upload=502294]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:53636->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:53636->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53636->45.91.116.4:4940: use of closed network connection Dec 29 12:26:29.040603643 (572h57m28.911686688s) [noise=true , address=45.91.116.4:4940 , piece_id=3676OHD74HKXC6AB5LPJEE3SAVVZZ36BGLWG56QIVFETU4C44DJQ, seg_pos={0 0}, upload=502294]: Long tail cancellation due to optimal threshold Dec 29 12:26:29.040766493 (572h57m28.911849538s) [noise=true , address=145.79.227.40:4049 , piece_id=CNLK7WROCZCPC2PKAQDKYQXUL6ZUBGD4E4E3E6PPVYEIS5ITLE7Q, seg_pos={0 0}, upload=502294]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 29 12:26:29.040774939 (572h57m28.911857984s) [noise=true , address=145.79.227.40:4049 , piece_id=CNLK7WROCZCPC2PKAQDKYQXUL6ZUBGD4E4E3E6PPVYEIS5ITLE7Q, seg_pos={0 0}, upload=502294]: Long tail cancellation due to optimal threshold Dec 29 12:26:29.040793515 (572h57m28.91187656s) [seg_pos={0 0}, upload=502294]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:26:29.040867666 (572h57m28.911950711s) [upload=502294]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:26:29.040873437 (572h57m28.911956482s) [upload=502294]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:26:29.335733775 (572h57m29.20681682s) [upload=502294]: Done uploading segments... Dec 29 12:26:29.335758983 (572h57m29.206842018s) [upload=502294]: Upload finished. err: <nil> Duration: 5.697359218s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X.......X..........] duration: 59.380462028s trace: https://jaeger.dev.storj.io/trace/54cc96062d54e547 FAIL (2025-12-29 12:28:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 7.148417227 (572h49m17.019500273s) [noise=true , address=45.91.116.13:4162 , piece_id=UPESSP5ITLGEQCHYNL4VTSXLBB24O7QXZ5UB6FA4PC5JWHGUVCAQ, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.148925174 (572h49m17.020008209s) [noise=true , address=145.79.227.43:4169 , piece_id=O7II3XWLMOJGXSOVGJ6ITQWOW2OIL7PPYWXNQTVWNYBNU3E4FFKA, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.149222871 (572h49m17.020305906s) [noise=true , address=145.79.227.34:4924 , piece_id=LDEYPDBM6HDZKO2YHAAUGXHFXFPCQ74IH5JIRFYBPUMIS5HFXRBQ, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.149466225 (572h49m17.020549801s) [noise=true , address=145.79.227.44:4197 , piece_id=4D4YHRYNXGE5YCZH7AL3IIN6Q6FI6TLGXECM3SAZQDCKOGN4F2OQ, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.149621069 (572h49m17.020704114s) [noise=true , address=64.49.13.169:4710 , piece_id=WVVMYAG6FYQDJIWNKAAH4GREERSAJN2SOQDRMD3WFK7NMLONL4MQ, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.149918967 (572h49m17.021002012s) [noise=true , address=145.79.227.4:4195 , piece_id=6UKJB3VYO7UL4TW5IVTDA4KTXTJ2YFDN2ADALMTYQVMTSLKYGVDA, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.149948984 (572h49m17.021032029s) [noise=true , address=77.90.31.18:4839 , piece_id=26G6IXW2YT7AUPAM4R4SQQFXXT57HSS2IGWMIXJNN5NEAHSEZCIA, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.150229729 (572h49m17.021312764s) [noise=true , address=77.90.31.13:4650 , piece_id=2G6KWTAPYS6FXIZK3SL5RD4OZ6DAOSSHUL5ZZ75PB4ETPNF7MH7A, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.150278251 (572h49m17.021361296s) [noise=true , address=145.79.227.5:4231 , piece_id=D6EJVCDCFO2GP4PE32ISCUSWCQGMY6NX4EYWVHG2T3DF3LM7DW3Q, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.150629960 (572h49m17.021712995s) [noise=true , address=145.79.227.40:4052 , piece_id=JFC3UQOZSXCYUGIKJKRI73RLJ2QGIQTYN37PSECCEDMJPK2UYSDA, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.150991189 (572h49m17.022074224s) [noise=true , address=77.90.31.12:4617 , piece_id=2GNINSXEB5KKEDT2WCIK3UWDICDUS4MBMIQQ7VKTQ5JGNY6O55JQ, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.151215005 (572h49m17.02229804s) [noise=true , address=193.149.164.143:4702 , piece_id=6UADW5KZGEO4EHPJP2TLDDZZBGR2A2J2LSLNN6WR4HV5MVO5UK6A, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.151269889 (572h49m17.022352925s) [noise=true , address=145.79.227.30:4780 , piece_id=MNJQ3WRD3J6TBKAEVMRHV467KM2OGUVT2YVDBXIN32HZJ55X73IQ, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.151578597 (572h49m17.022661633s) [noise=true , address=145.79.227.35:4501 , piece_id=S335GIDTNGOW2JXIX5OZ7HHBQI3K254ZR2TPCU27ZN2K6ZS3CVOA, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.151863319 (572h49m17.022946354s) [noise=true , address=77.90.31.10:4531 , piece_id=T344ZC45IB4573ARJASYIANXG24GH26VI6VBPUR3CPYK77K7Z7TQ, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.151922863 (572h49m17.023005908s) [noise=true , address=145.79.227.29:4974 , piece_id=Q4PXHIZSLPKYURXKZ6B6HJ7YANRN5QHTZWUWZ3MVI3G3O7JFKIJA, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.152129156 (572h49m17.023212201s) [noise=true , address=145.79.227.13:4807 , piece_id=Y67QBVSHT5WUO4DG7LW2DM67ZILNFVZFUFSQSBF4GGVYJ24SMGOA, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:18.050161080 (572h49m17.921244125s) [noise=true , address=193.149.164.143:4702 , piece_id=6UADW5KZGEO4EHPJP2TLDDZZBGR2A2J2LSLNN6WR4HV5MVO5UK6A, seg_pos={0 0}, upload=502178]: Done uploading piece. err: ecclient: upload failed (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1, address:193.149.164.143:4702): protocol: storage node overloaded, request limit: 2000 Dec 29 12:18:18.609488842 (572h49m18.480571877s) [upload=502178]: Next returned nil segment Dec 29 12:18:18.609505194 (572h49m18.480588259s) [upload=502178]: Waiting for error group managing segments... Dec 29 12:18:18.996027579 (572h49m18.867110624s) [noise=true , address=145.79.227.5:4231 , piece_id=D6EJVCDCFO2GP4PE32ISCUSWCQGMY6NX4EYWVHG2T3DF3LM7DW3Q, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.015970613 (572h49m18.887053658s) [noise=true , address=145.79.227.4:4195 , piece_id=6UKJB3VYO7UL4TW5IVTDA4KTXTJ2YFDN2ADALMTYQVMTSLKYGVDA, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.024599794 (572h49m18.895682829s) [noise=true , address=145.79.227.29:4974 , piece_id=Q4PXHIZSLPKYURXKZ6B6HJ7YANRN5QHTZWUWZ3MVI3G3O7JFKIJA, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.025693498 (572h49m18.896776533s) [noise=true , address=145.79.227.13:4807 , piece_id=Y67QBVSHT5WUO4DG7LW2DM67ZILNFVZFUFSQSBF4GGVYJ24SMGOA, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.220567296 (572h49m19.091650341s) [noise=true , address=77.90.31.12:4617 , piece_id=2GNINSXEB5KKEDT2WCIK3UWDICDUS4MBMIQQ7VKTQ5JGNY6O55JQ, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.308488310 (572h49m19.179571345s) [noise=true , address=77.90.31.13:4650 , piece_id=2G6KWTAPYS6FXIZK3SL5RD4OZ6DAOSSHUL5ZZ75PB4ETPNF7MH7A, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.413574289 (572h49m19.284657324s) [noise=true , address=77.90.31.18:4839 , piece_id=26G6IXW2YT7AUPAM4R4SQQFXXT57HSS2IGWMIXJNN5NEAHSEZCIA, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.871377594 (572h49m19.742460639s) [noise=true , address=145.79.227.30:4780 , piece_id=MNJQ3WRD3J6TBKAEVMRHV467KM2OGUVT2YVDBXIN32HZJ55X73IQ, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.939637331 (572h49m19.810720366s) [noise=true , address=145.79.227.35:4501 , piece_id=S335GIDTNGOW2JXIX5OZ7HHBQI3K254ZR2TPCU27ZN2K6ZS3CVOA, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.958323732 (572h49m19.829406767s) [noise=true , address=145.79.227.44:4197 , piece_id=4D4YHRYNXGE5YCZH7AL3IIN6Q6FI6TLGXECM3SAZQDCKOGN4F2OQ, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:20.001969170 (572h49m19.873052215s) [noise=true , address=145.79.227.40:4052 , piece_id=JFC3UQOZSXCYUGIKJKRI73RLJ2QGIQTYN37PSECCEDMJPK2UYSDA, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:20.015729351 (572h49m19.886812386s) [noise=true , address=145.79.227.34:4924 , piece_id=LDEYPDBM6HDZKO2YHAAUGXHFXFPCQ74IH5JIRFYBPUMIS5HFXRBQ, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:20.114412729 (572h49m19.985495764s) [noise=true , address=145.79.227.43:4169 , piece_id=O7II3XWLMOJGXSOVGJ6ITQWOW2OIL7PPYWXNQTVWNYBNU3E4FFKA, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:20.177148507 (572h49m20.048231542s) [noise=true , address=77.90.31.10:4531 , piece_id=T344ZC45IB4573ARJASYIANXG24GH26VI6VBPUR3CPYK77K7Z7TQ, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:20.177170508 (572h49m20.048253544s) [seg_pos={0 0}, upload=502178]: Segment reached optimal threshold of 14 pieces. Dec 29 12:18:20.177339190 (572h49m20.048422225s) [noise=true , address=64.49.13.169:4710 , piece_id=WVVMYAG6FYQDJIWNKAAH4GREERSAJN2SOQDRMD3WFK7NMLONL4MQ, seg_pos={0 0}, upload=502178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57032->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57032->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57032->64.49.13.169:4710: use of closed network connection Dec 29 12:18:20.177420124 (572h49m20.048503179s) [noise=true , address=64.49.13.169:4710 , piece_id=WVVMYAG6FYQDJIWNKAAH4GREERSAJN2SOQDRMD3WFK7NMLONL4MQ, seg_pos={0 0}, upload=502178]: Long tail cancellation due to optimal threshold Dec 29 12:18:20.177586831 (572h49m20.048669866s) [noise=true , address=145.79.227.49:4343 , piece_id=JPXHILNUTSHS4RUP5CRSUFWEVASM55PBY4MQCHK47AURG5XZDS7Q, seg_pos={0 0}, upload=502178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf): protocol: expected piece hash; context canceled; EOF Dec 29 12:18:20.177593945 (572h49m20.04867698s) [noise=true , address=145.79.227.49:4343 , piece_id=JPXHILNUTSHS4RUP5CRSUFWEVASM55PBY4MQCHK47AURG5XZDS7Q, seg_pos={0 0}, upload=502178]: Long tail cancellation due to optimal threshold Dec 29 12:18:20.177724033 (572h49m20.048807068s) [noise=true , address=45.91.116.13:4162 , piece_id=UPESSP5ITLGEQCHYNL4VTSXLBB24O7QXZ5UB6FA4PC5JWHGUVCAQ, seg_pos={0 0}, upload=502178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4): protocol: expected piece hash; context canceled; EOF Dec 29 12:18:20.177729393 (572h49m20.048812428s) [noise=true , address=45.91.116.13:4162 , piece_id=UPESSP5ITLGEQCHYNL4VTSXLBB24O7QXZ5UB6FA4PC5JWHGUVCAQ, seg_pos={0 0}, upload=502178]: Long tail cancellation due to optimal threshold Dec 29 12:18:20.177744652 (572h49m20.048827687s) [seg_pos={0 0}, upload=502178]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:18:20.177850734 (572h49m20.048933769s) [upload=502178]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:18:20.177856495 (572h49m20.04893953s) [upload=502178]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:18:20.468750169 (572h49m20.339833204s) [upload=502178]: Done uploading segments... Dec 29 12:18:20.468821605 (572h49m20.33990465s) [upload=502178]: Upload finished. err: <nil> Duration: 59.380489258s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [XXXX...XXX...XXX..XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXX] duration: 5.001435268s trace: https://jaeger.dev.storj.io/trace/6f3b88300ffcf695 FAIL (2025-12-29 12:28:14 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001459434s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.XXX.X.X....XXXXXX..X.X.......X.XX...X.......X.X..XXX....X..] duration: 5.966699041s trace: https://jaeger.dev.storj.io/trace/6047085ef9629cee FAIL (2025-12-29 12:27:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng segments... Dec 29 12:26:26.381455222 (572h57m26.252538267s) [upload=502294]: Got next segment. Inline: false Dec 29 12:26:26.381569239 (572h57m26.252652274s) [upload=502294]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:26:26.381575021 (572h57m26.252658066s) [upload=502294]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:26:26.625888218 (572h57m26.496971263s) [seg_pos={0 0}, upload=502294]: Begin upload segment... Dec 29 12:26:26.625973239 (572h57m26.497056284s) [seg_pos={0 0}, upload=502294]: Done begin upload segment. Dec 29 12:26:26.626040047 (572h57m26.497123102s) [noise=true , address=77.90.31.10:4520 , piece_id=IHPHB6JKOOWSS5ELL4HXBIYJ3O5UIK4IDKIYVEWAIXH57KFBW6NQ, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.626816195 (572h57m26.49789924s) [noise=true , address=145.79.227.15:4874 , piece_id=E3AAB4JEHERJ2R6HGP4MLRNXNMK7W4ICRPWIFWUMQ3OJNRHYOP5Q, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.627162455 (572h57m26.4982455s) [noise=true , address=145.79.227.40:4049 , piece_id=CNLK7WROCZCPC2PKAQDKYQXUL6ZUBGD4E4E3E6PPVYEIS5ITLE7Q, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.627543349 (572h57m26.498626384s) [noise=true , address=45.91.116.9:4262 , piece_id=JFLGJIBVJXSJE54ECBS4U33LARKUHPBAQB6NUTONBHMBWBM6H6XA, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.627886933 (572h57m26.498969988s) [noise=true , address=145.79.227.38:4977 , piece_id=V3H2S43AEUMFLLDTXSGCA6BFDJUPZLKAUDB3RWJVG3PK6AEPXY6A, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.628205891 (572h57m26.499288937s) [noise=true , address=145.79.227.47:4266 , piece_id=Q2FMIE5XX4LZD4GI6SSZESJZVMZBE4KGPC2H36EY7KY65EO5G3QA, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.628648774 (572h57m26.499731809s) [noise=true , address=145.79.227.42:4123 , piece_id=7EHC46NCH7XN7ZPPPM2ROQ2PSHOYAHWR4EGUB76X3Y62OFOPAPTA, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.628954356 (572h57m26.500037401s) [noise=true , address=145.79.227.45:4229 , piece_id=P5OCXGBPPBKMYJTGYCGYNO5WGIMZLFVZLUCB3GUQCJIRG637633Q, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.630379450 (572h57m26.501462495s) [noise=true , address=45.91.116.6:4976 , piece_id=SBTB7O3HFTFSG6CMUDHRDSNFT4HRGM522FNUOZAUOKV2KPGOKGTQ, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.630707424 (572h57m26.501790459s) [noise=true , address=145.79.227.14:4837 , piece_id=SQCFNKFPX7KK4LKL52NVS2KN32SWSEVPGWWUTTVW33DKCN4JILNA, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.631005512 (572h57m26.502088557s) [noise=true , address=145.79.227.2:4115 , piece_id=BXBFY2EP67QEZAX56ASJINVOZGAO6H2XAD3SBIR7MVA6VUFECNZA, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.631272210 (572h57m26.502355255s) [noise=true , address=77.90.31.14:4662 , piece_id=G3L5FXFH2HS3AR5XLPGHL6XL4FJ7EUBHYU7AF3L3FIJKOPBBDQTA, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.632025514 (572h57m26.503108559s) [noise=true , address=45.91.116.13:4156 , piece_id=O2FEYLUXWNE6O4WFP3PHDBA5DMLV5X7Q35N6O7ADD5OHV6XRVC4A, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.632463408 (572h57m26.503546453s) [noise=true , address=45.91.116.7:4009 , piece_id=CTN2ZM4XBB5XHQFHOAPW6IQ5EB773ZPBW7TOBV74V2F3SM25AZWA, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.633007343 (572h57m26.504090388s) [noise=true , address=45.91.116.4:4940 , piece_id=3676OHD74HKXC6AB5LPJEE3SAVVZZ36BGLWG56QIVFETU4C44DJQ, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.633424848 (572h57m26.504507883s) [noise=true , address=145.79.227.12:4767 , piece_id=35QG5CVAG33YCHPXLEUMCVGW23UFPGAPX6S4TEEASSCNPVO4HYAA, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:28.330358171 (572h57m28.201441206s) [upload=502294]: Next returned nil segment Dec 29 12:26:28.330375173 (572h57m28.201458218s) [upload=502294]: Waiting for error group managing segments... Dec 29 12:26:28.603285514 (572h57m28.47436855s) [noise=true , address=145.79.227.12:4767 , piece_id=35QG5CVAG33YCHPXLEUMCVGW23UFPGAPX6S4TEEASSCNPVO4HYAA, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:28.618055339 (572h57m28.489138374s) [noise=true , address=145.79.227.42:4123 , piece_id=7EHC46NCH7XN7ZPPPM2ROQ2PSHOYAHWR4EGUB76X3Y62OFOPAPTA, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:28.618182089 (572h57m28.489265124s) [noise=true , address=145.79.227.45:4229 , piece_id=P5OCXGBPPBKMYJTGYCGYNO5WGIMZLFVZLUCB3GUQCJIRG637633Q, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:28.618845883 (572h57m28.489928928s) [noise=true , address=145.79.227.38:4977 , piece_id=V3H2S43AEUMFLLDTXSGCA6BFDJUPZLKAUDB3RWJVG3PK6AEPXY6A, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:28.620502508 (572h57m28.491585543s) [noise=true , address=145.79.227.15:4874 , piece_id=E3AAB4JEHERJ2R6HGP4MLRNXNMK7W4ICRPWIFWUMQ3OJNRHYOP5Q, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:28.634073068 (572h57m28.505156103s) [noise=true , address=145.79.227.14:4837 , piece_id=SQCFNKFPX7KK4LKL52NVS2KN32SWSEVPGWWUTTVW33DKCN4JILNA, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:28.635391348 (572h57m28.506474383s) [noise=true , address=145.79.227.2:4115 , piece_id=BXBFY2EP67QEZAX56ASJINVOZGAO6H2XAD3SBIR7MVA6VUFECNZA, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:28.645652678 (572h57m28.516735713s) [noise=true , address=145.79.227.47:4266 , piece_id=Q2FMIE5XX4LZD4GI6SSZESJZVMZBE4KGPC2H36EY7KY65EO5G3QA, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:28.804359379 (572h57m28.675442424s) [noise=true , address=77.90.31.14:4662 , piece_id=G3L5FXFH2HS3AR5XLPGHL6XL4FJ7EUBHYU7AF3L3FIJKOPBBDQTA, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:28.815560217 (572h57m28.686643262s) [noise=true , address=77.90.31.10:4520 , piece_id=IHPHB6JKOOWSS5ELL4HXBIYJ3O5UIK4IDKIYVEWAIXH57KFBW6NQ, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:28.944227930 (572h57m28.815310965s) [noise=true , address=45.91.116.13:4156 , piece_id=O2FEYLUXWNE6O4WFP3PHDBA5DMLV5X7Q35N6O7ADD5OHV6XRVC4A, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:29.018960361 (572h57m28.890043406s) [noise=true , address=45.91.116.9:4262 , piece_id=JFLGJIBVJXSJE54ECBS4U33LARKUHPBAQB6NUTONBHMBWBM6H6XA, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:29.040090536 (572h57m28.911173581s) [noise=true , address=45.91.116.6:4976 , piece_id=SBTB7O3HFTFSG6CMUDHRDSNFT4HRGM522FNUOZAUOKV2KPGOKGTQ, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:29.040113139 (572h57m28.911196184s) [seg_pos={0 0}, upload=502294]: Segment reached optimal threshold of 13 pieces. Dec 29 12:26:29.040314813 (572h57m28.911397858s) [noise=true , address=45.91.116.7:4009 , piece_id=CTN2ZM4XBB5XHQFHOAPW6IQ5EB773ZPBW7TOBV74V2F3SM25AZWA, seg_pos={0 0}, upload=502294]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:43924->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:43924->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43924->45.91.116.7:4009: use of closed network connection Dec 29 12:26:29.040322277 (572h57m28.911405312s) [noise=true , address=45.91.116.7:4009 , piece_id=CTN2ZM4XBB5XHQFHOAPW6IQ5EB773ZPBW7TOBV74V2F3SM25AZWA, seg_pos={0 0}, upload=502294]: Long tail cancellation due to optimal threshold Dec 29 12:26:29.040594947 (572h57m28.911677992s) [noise=true , address=45.91.116.4:4940 , piece_id=3676OHD74HKXC6AB5LPJEE3SAVVZZ36BGLWG56QIVFETU4C44DJQ, seg_pos={0 0}, upload=502294]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:53636->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:53636->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53636->45.91.116.4:4940: use of closed network connection Dec 29 12:26:29.040603643 (572h57m28.911686688s) [noise=true , address=45.91.116.4:4940 , piece_id=3676OHD74HKXC6AB5LPJEE3SAVVZZ36BGLWG56QIVFETU4C44DJQ, seg_pos={0 0}, upload=502294]: Long tail cancellation due to optimal threshold Dec 29 12:26:29.040766493 (572h57m28.911849538s) [noise=true , address=145.79.227.40:4049 , piece_id=CNLK7WROCZCPC2PKAQDKYQXUL6ZUBGD4E4E3E6PPVYEIS5ITLE7Q, seg_pos={0 0}, upload=502294]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 29 12:26:29.040774939 (572h57m28.911857984s) [noise=true , address=145.79.227.40:4049 , piece_id=CNLK7WROCZCPC2PKAQDKYQXUL6ZUBGD4E4E3E6PPVYEIS5ITLE7Q, seg_pos={0 0}, upload=502294]: Long tail cancellation due to optimal threshold Dec 29 12:26:29.040793515 (572h57m28.91187656s) [seg_pos={0 0}, upload=502294]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:26:29.040867666 (572h57m28.911950711s) [upload=502294]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:26:29.040873437 (572h57m28.911956482s) [upload=502294]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:26:29.335733775 (572h57m29.20681682s) [upload=502294]: Done uploading segments... Dec 29 12:26:29.335758983 (572h57m29.206842018s) [upload=502294]: Upload finished. err: <nil> Duration: 5.966732224s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [.XXXX...XXX...XXX..XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXX] duration: 5.001233428s trace: https://jaeger.dev.storj.io/trace/385958627fd259dd FAIL (2025-12-29 12:27:23 UTC) orbiter warning: context deadline exceeded Duration: 5.001254749s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X.XXX.X.X....XXXXXX..X.X.......X.XX...X.......X.X..XXX....X.] duration: 5.686283587s trace: https://jaeger.dev.storj.io/trace/6047085ef9629cee FAIL (2025-12-29 12:27:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng segments... Dec 29 12:26:26.381455222 (572h57m26.252538267s) [upload=502294]: Got next segment. Inline: false Dec 29 12:26:26.381569239 (572h57m26.252652274s) [upload=502294]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:26:26.381575021 (572h57m26.252658066s) [upload=502294]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:26:26.625888218 (572h57m26.496971263s) [seg_pos={0 0}, upload=502294]: Begin upload segment... Dec 29 12:26:26.625973239 (572h57m26.497056284s) [seg_pos={0 0}, upload=502294]: Done begin upload segment. Dec 29 12:26:26.626040047 (572h57m26.497123102s) [noise=true , address=77.90.31.10:4520 , piece_id=IHPHB6JKOOWSS5ELL4HXBIYJ3O5UIK4IDKIYVEWAIXH57KFBW6NQ, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.626816195 (572h57m26.49789924s) [noise=true , address=145.79.227.15:4874 , piece_id=E3AAB4JEHERJ2R6HGP4MLRNXNMK7W4ICRPWIFWUMQ3OJNRHYOP5Q, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.627162455 (572h57m26.4982455s) [noise=true , address=145.79.227.40:4049 , piece_id=CNLK7WROCZCPC2PKAQDKYQXUL6ZUBGD4E4E3E6PPVYEIS5ITLE7Q, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.627543349 (572h57m26.498626384s) [noise=true , address=45.91.116.9:4262 , piece_id=JFLGJIBVJXSJE54ECBS4U33LARKUHPBAQB6NUTONBHMBWBM6H6XA, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.627886933 (572h57m26.498969988s) [noise=true , address=145.79.227.38:4977 , piece_id=V3H2S43AEUMFLLDTXSGCA6BFDJUPZLKAUDB3RWJVG3PK6AEPXY6A, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.628205891 (572h57m26.499288937s) [noise=true , address=145.79.227.47:4266 , piece_id=Q2FMIE5XX4LZD4GI6SSZESJZVMZBE4KGPC2H36EY7KY65EO5G3QA, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.628648774 (572h57m26.499731809s) [noise=true , address=145.79.227.42:4123 , piece_id=7EHC46NCH7XN7ZPPPM2ROQ2PSHOYAHWR4EGUB76X3Y62OFOPAPTA, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.628954356 (572h57m26.500037401s) [noise=true , address=145.79.227.45:4229 , piece_id=P5OCXGBPPBKMYJTGYCGYNO5WGIMZLFVZLUCB3GUQCJIRG637633Q, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.630379450 (572h57m26.501462495s) [noise=true , address=45.91.116.6:4976 , piece_id=SBTB7O3HFTFSG6CMUDHRDSNFT4HRGM522FNUOZAUOKV2KPGOKGTQ, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.630707424 (572h57m26.501790459s) [noise=true , address=145.79.227.14:4837 , piece_id=SQCFNKFPX7KK4LKL52NVS2KN32SWSEVPGWWUTTVW33DKCN4JILNA, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.631005512 (572h57m26.502088557s) [noise=true , address=145.79.227.2:4115 , piece_id=BXBFY2EP67QEZAX56ASJINVOZGAO6H2XAD3SBIR7MVA6VUFECNZA, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.631272210 (572h57m26.502355255s) [noise=true , address=77.90.31.14:4662 , piece_id=G3L5FXFH2HS3AR5XLPGHL6XL4FJ7EUBHYU7AF3L3FIJKOPBBDQTA, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.632025514 (572h57m26.503108559s) [noise=true , address=45.91.116.13:4156 , piece_id=O2FEYLUXWNE6O4WFP3PHDBA5DMLV5X7Q35N6O7ADD5OHV6XRVC4A, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.632463408 (572h57m26.503546453s) [noise=true , address=45.91.116.7:4009 , piece_id=CTN2ZM4XBB5XHQFHOAPW6IQ5EB773ZPBW7TOBV74V2F3SM25AZWA, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.633007343 (572h57m26.504090388s) [noise=true , address=45.91.116.4:4940 , piece_id=3676OHD74HKXC6AB5LPJEE3SAVVZZ36BGLWG56QIVFETU4C44DJQ, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.633424848 (572h57m26.504507883s) [noise=true , address=145.79.227.12:4767 , piece_id=35QG5CVAG33YCHPXLEUMCVGW23UFPGAPX6S4TEEASSCNPVO4HYAA, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:28.330358171 (572h57m28.201441206s) [upload=502294]: Next returned nil segment Dec 29 12:26:28.330375173 (572h57m28.201458218s) [upload=502294]: Waiting for error group managing segments... Dec 29 12:26:28.603285514 (572h57m28.47436855s) [noise=true , address=145.79.227.12:4767 , piece_id=35QG5CVAG33YCHPXLEUMCVGW23UFPGAPX6S4TEEASSCNPVO4HYAA, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:28.618055339 (572h57m28.489138374s) [noise=true , address=145.79.227.42:4123 , piece_id=7EHC46NCH7XN7ZPPPM2ROQ2PSHOYAHWR4EGUB76X3Y62OFOPAPTA, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:28.618182089 (572h57m28.489265124s) [noise=true , address=145.79.227.45:4229 , piece_id=P5OCXGBPPBKMYJTGYCGYNO5WGIMZLFVZLUCB3GUQCJIRG637633Q, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:28.618845883 (572h57m28.489928928s) [noise=true , address=145.79.227.38:4977 , piece_id=V3H2S43AEUMFLLDTXSGCA6BFDJUPZLKAUDB3RWJVG3PK6AEPXY6A, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:28.620502508 (572h57m28.491585543s) [noise=true , address=145.79.227.15:4874 , piece_id=E3AAB4JEHERJ2R6HGP4MLRNXNMK7W4ICRPWIFWUMQ3OJNRHYOP5Q, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:28.634073068 (572h57m28.505156103s) [noise=true , address=145.79.227.14:4837 , piece_id=SQCFNKFPX7KK4LKL52NVS2KN32SWSEVPGWWUTTVW33DKCN4JILNA, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:28.635391348 (572h57m28.506474383s) [noise=true , address=145.79.227.2:4115 , piece_id=BXBFY2EP67QEZAX56ASJINVOZGAO6H2XAD3SBIR7MVA6VUFECNZA, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:28.645652678 (572h57m28.516735713s) [noise=true , address=145.79.227.47:4266 , piece_id=Q2FMIE5XX4LZD4GI6SSZESJZVMZBE4KGPC2H36EY7KY65EO5G3QA, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:28.804359379 (572h57m28.675442424s) [noise=true , address=77.90.31.14:4662 , piece_id=G3L5FXFH2HS3AR5XLPGHL6XL4FJ7EUBHYU7AF3L3FIJKOPBBDQTA, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:28.815560217 (572h57m28.686643262s) [noise=true , address=77.90.31.10:4520 , piece_id=IHPHB6JKOOWSS5ELL4HXBIYJ3O5UIK4IDKIYVEWAIXH57KFBW6NQ, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:28.944227930 (572h57m28.815310965s) [noise=true , address=45.91.116.13:4156 , piece_id=O2FEYLUXWNE6O4WFP3PHDBA5DMLV5X7Q35N6O7ADD5OHV6XRVC4A, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:29.018960361 (572h57m28.890043406s) [noise=true , address=45.91.116.9:4262 , piece_id=JFLGJIBVJXSJE54ECBS4U33LARKUHPBAQB6NUTONBHMBWBM6H6XA, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:29.040090536 (572h57m28.911173581s) [noise=true , address=45.91.116.6:4976 , piece_id=SBTB7O3HFTFSG6CMUDHRDSNFT4HRGM522FNUOZAUOKV2KPGOKGTQ, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:29.040113139 (572h57m28.911196184s) [seg_pos={0 0}, upload=502294]: Segment reached optimal threshold of 13 pieces. Dec 29 12:26:29.040314813 (572h57m28.911397858s) [noise=true , address=45.91.116.7:4009 , piece_id=CTN2ZM4XBB5XHQFHOAPW6IQ5EB773ZPBW7TOBV74V2F3SM25AZWA, seg_pos={0 0}, upload=502294]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:43924->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:43924->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43924->45.91.116.7:4009: use of closed network connection Dec 29 12:26:29.040322277 (572h57m28.911405312s) [noise=true , address=45.91.116.7:4009 , piece_id=CTN2ZM4XBB5XHQFHOAPW6IQ5EB773ZPBW7TOBV74V2F3SM25AZWA, seg_pos={0 0}, upload=502294]: Long tail cancellation due to optimal threshold Dec 29 12:26:29.040594947 (572h57m28.911677992s) [noise=true , address=45.91.116.4:4940 , piece_id=3676OHD74HKXC6AB5LPJEE3SAVVZZ36BGLWG56QIVFETU4C44DJQ, seg_pos={0 0}, upload=502294]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:53636->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:53636->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53636->45.91.116.4:4940: use of closed network connection Dec 29 12:26:29.040603643 (572h57m28.911686688s) [noise=true , address=45.91.116.4:4940 , piece_id=3676OHD74HKXC6AB5LPJEE3SAVVZZ36BGLWG56QIVFETU4C44DJQ, seg_pos={0 0}, upload=502294]: Long tail cancellation due to optimal threshold Dec 29 12:26:29.040766493 (572h57m28.911849538s) [noise=true , address=145.79.227.40:4049 , piece_id=CNLK7WROCZCPC2PKAQDKYQXUL6ZUBGD4E4E3E6PPVYEIS5ITLE7Q, seg_pos={0 0}, upload=502294]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 29 12:26:29.040774939 (572h57m28.911857984s) [noise=true , address=145.79.227.40:4049 , piece_id=CNLK7WROCZCPC2PKAQDKYQXUL6ZUBGD4E4E3E6PPVYEIS5ITLE7Q, seg_pos={0 0}, upload=502294]: Long tail cancellation due to optimal threshold Dec 29 12:26:29.040793515 (572h57m28.91187656s) [seg_pos={0 0}, upload=502294]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:26:29.040867666 (572h57m28.911950711s) [upload=502294]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:26:29.040873437 (572h57m28.911956482s) [upload=502294]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:26:29.335733775 (572h57m29.20681682s) [upload=502294]: Done uploading segments... Dec 29 12:26:29.335758983 (572h57m29.206842018s) [upload=502294]: Upload finished. err: <nil> Duration: 5.686353661s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X.......X.........] duration: 38.914453937s trace: https://jaeger.dev.storj.io/trace/54cc96062d54e547 FAIL (2025-12-29 12:27:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 7.148417227 (572h49m17.019500273s) [noise=true , address=45.91.116.13:4162 , piece_id=UPESSP5ITLGEQCHYNL4VTSXLBB24O7QXZ5UB6FA4PC5JWHGUVCAQ, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.148925174 (572h49m17.020008209s) [noise=true , address=145.79.227.43:4169 , piece_id=O7II3XWLMOJGXSOVGJ6ITQWOW2OIL7PPYWXNQTVWNYBNU3E4FFKA, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.149222871 (572h49m17.020305906s) [noise=true , address=145.79.227.34:4924 , piece_id=LDEYPDBM6HDZKO2YHAAUGXHFXFPCQ74IH5JIRFYBPUMIS5HFXRBQ, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.149466225 (572h49m17.020549801s) [noise=true , address=145.79.227.44:4197 , piece_id=4D4YHRYNXGE5YCZH7AL3IIN6Q6FI6TLGXECM3SAZQDCKOGN4F2OQ, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.149621069 (572h49m17.020704114s) [noise=true , address=64.49.13.169:4710 , piece_id=WVVMYAG6FYQDJIWNKAAH4GREERSAJN2SOQDRMD3WFK7NMLONL4MQ, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.149918967 (572h49m17.021002012s) [noise=true , address=145.79.227.4:4195 , piece_id=6UKJB3VYO7UL4TW5IVTDA4KTXTJ2YFDN2ADALMTYQVMTSLKYGVDA, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.149948984 (572h49m17.021032029s) [noise=true , address=77.90.31.18:4839 , piece_id=26G6IXW2YT7AUPAM4R4SQQFXXT57HSS2IGWMIXJNN5NEAHSEZCIA, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.150229729 (572h49m17.021312764s) [noise=true , address=77.90.31.13:4650 , piece_id=2G6KWTAPYS6FXIZK3SL5RD4OZ6DAOSSHUL5ZZ75PB4ETPNF7MH7A, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.150278251 (572h49m17.021361296s) [noise=true , address=145.79.227.5:4231 , piece_id=D6EJVCDCFO2GP4PE32ISCUSWCQGMY6NX4EYWVHG2T3DF3LM7DW3Q, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.150629960 (572h49m17.021712995s) [noise=true , address=145.79.227.40:4052 , piece_id=JFC3UQOZSXCYUGIKJKRI73RLJ2QGIQTYN37PSECCEDMJPK2UYSDA, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.150991189 (572h49m17.022074224s) [noise=true , address=77.90.31.12:4617 , piece_id=2GNINSXEB5KKEDT2WCIK3UWDICDUS4MBMIQQ7VKTQ5JGNY6O55JQ, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.151215005 (572h49m17.02229804s) [noise=true , address=193.149.164.143:4702 , piece_id=6UADW5KZGEO4EHPJP2TLDDZZBGR2A2J2LSLNN6WR4HV5MVO5UK6A, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.151269889 (572h49m17.022352925s) [noise=true , address=145.79.227.30:4780 , piece_id=MNJQ3WRD3J6TBKAEVMRHV467KM2OGUVT2YVDBXIN32HZJ55X73IQ, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.151578597 (572h49m17.022661633s) [noise=true , address=145.79.227.35:4501 , piece_id=S335GIDTNGOW2JXIX5OZ7HHBQI3K254ZR2TPCU27ZN2K6ZS3CVOA, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.151863319 (572h49m17.022946354s) [noise=true , address=77.90.31.10:4531 , piece_id=T344ZC45IB4573ARJASYIANXG24GH26VI6VBPUR3CPYK77K7Z7TQ, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.151922863 (572h49m17.023005908s) [noise=true , address=145.79.227.29:4974 , piece_id=Q4PXHIZSLPKYURXKZ6B6HJ7YANRN5QHTZWUWZ3MVI3G3O7JFKIJA, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.152129156 (572h49m17.023212201s) [noise=true , address=145.79.227.13:4807 , piece_id=Y67QBVSHT5WUO4DG7LW2DM67ZILNFVZFUFSQSBF4GGVYJ24SMGOA, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:18.050161080 (572h49m17.921244125s) [noise=true , address=193.149.164.143:4702 , piece_id=6UADW5KZGEO4EHPJP2TLDDZZBGR2A2J2LSLNN6WR4HV5MVO5UK6A, seg_pos={0 0}, upload=502178]: Done uploading piece. err: ecclient: upload failed (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1, address:193.149.164.143:4702): protocol: storage node overloaded, request limit: 2000 Dec 29 12:18:18.609488842 (572h49m18.480571877s) [upload=502178]: Next returned nil segment Dec 29 12:18:18.609505194 (572h49m18.480588259s) [upload=502178]: Waiting for error group managing segments... Dec 29 12:18:18.996027579 (572h49m18.867110624s) [noise=true , address=145.79.227.5:4231 , piece_id=D6EJVCDCFO2GP4PE32ISCUSWCQGMY6NX4EYWVHG2T3DF3LM7DW3Q, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.015970613 (572h49m18.887053658s) [noise=true , address=145.79.227.4:4195 , piece_id=6UKJB3VYO7UL4TW5IVTDA4KTXTJ2YFDN2ADALMTYQVMTSLKYGVDA, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.024599794 (572h49m18.895682829s) [noise=true , address=145.79.227.29:4974 , piece_id=Q4PXHIZSLPKYURXKZ6B6HJ7YANRN5QHTZWUWZ3MVI3G3O7JFKIJA, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.025693498 (572h49m18.896776533s) [noise=true , address=145.79.227.13:4807 , piece_id=Y67QBVSHT5WUO4DG7LW2DM67ZILNFVZFUFSQSBF4GGVYJ24SMGOA, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.220567296 (572h49m19.091650341s) [noise=true , address=77.90.31.12:4617 , piece_id=2GNINSXEB5KKEDT2WCIK3UWDICDUS4MBMIQQ7VKTQ5JGNY6O55JQ, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.308488310 (572h49m19.179571345s) [noise=true , address=77.90.31.13:4650 , piece_id=2G6KWTAPYS6FXIZK3SL5RD4OZ6DAOSSHUL5ZZ75PB4ETPNF7MH7A, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.413574289 (572h49m19.284657324s) [noise=true , address=77.90.31.18:4839 , piece_id=26G6IXW2YT7AUPAM4R4SQQFXXT57HSS2IGWMIXJNN5NEAHSEZCIA, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.871377594 (572h49m19.742460639s) [noise=true , address=145.79.227.30:4780 , piece_id=MNJQ3WRD3J6TBKAEVMRHV467KM2OGUVT2YVDBXIN32HZJ55X73IQ, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.939637331 (572h49m19.810720366s) [noise=true , address=145.79.227.35:4501 , piece_id=S335GIDTNGOW2JXIX5OZ7HHBQI3K254ZR2TPCU27ZN2K6ZS3CVOA, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.958323732 (572h49m19.829406767s) [noise=true , address=145.79.227.44:4197 , piece_id=4D4YHRYNXGE5YCZH7AL3IIN6Q6FI6TLGXECM3SAZQDCKOGN4F2OQ, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:20.001969170 (572h49m19.873052215s) [noise=true , address=145.79.227.40:4052 , piece_id=JFC3UQOZSXCYUGIKJKRI73RLJ2QGIQTYN37PSECCEDMJPK2UYSDA, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:20.015729351 (572h49m19.886812386s) [noise=true , address=145.79.227.34:4924 , piece_id=LDEYPDBM6HDZKO2YHAAUGXHFXFPCQ74IH5JIRFYBPUMIS5HFXRBQ, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:20.114412729 (572h49m19.985495764s) [noise=true , address=145.79.227.43:4169 , piece_id=O7II3XWLMOJGXSOVGJ6ITQWOW2OIL7PPYWXNQTVWNYBNU3E4FFKA, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:20.177148507 (572h49m20.048231542s) [noise=true , address=77.90.31.10:4531 , piece_id=T344ZC45IB4573ARJASYIANXG24GH26VI6VBPUR3CPYK77K7Z7TQ, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:20.177170508 (572h49m20.048253544s) [seg_pos={0 0}, upload=502178]: Segment reached optimal threshold of 14 pieces. Dec 29 12:18:20.177339190 (572h49m20.048422225s) [noise=true , address=64.49.13.169:4710 , piece_id=WVVMYAG6FYQDJIWNKAAH4GREERSAJN2SOQDRMD3WFK7NMLONL4MQ, seg_pos={0 0}, upload=502178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57032->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57032->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57032->64.49.13.169:4710: use of closed network connection Dec 29 12:18:20.177420124 (572h49m20.048503179s) [noise=true , address=64.49.13.169:4710 , piece_id=WVVMYAG6FYQDJIWNKAAH4GREERSAJN2SOQDRMD3WFK7NMLONL4MQ, seg_pos={0 0}, upload=502178]: Long tail cancellation due to optimal threshold Dec 29 12:18:20.177586831 (572h49m20.048669866s) [noise=true , address=145.79.227.49:4343 , piece_id=JPXHILNUTSHS4RUP5CRSUFWEVASM55PBY4MQCHK47AURG5XZDS7Q, seg_pos={0 0}, upload=502178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf): protocol: expected piece hash; context canceled; EOF Dec 29 12:18:20.177593945 (572h49m20.04867698s) [noise=true , address=145.79.227.49:4343 , piece_id=JPXHILNUTSHS4RUP5CRSUFWEVASM55PBY4MQCHK47AURG5XZDS7Q, seg_pos={0 0}, upload=502178]: Long tail cancellation due to optimal threshold Dec 29 12:18:20.177724033 (572h49m20.048807068s) [noise=true , address=45.91.116.13:4162 , piece_id=UPESSP5ITLGEQCHYNL4VTSXLBB24O7QXZ5UB6FA4PC5JWHGUVCAQ, seg_pos={0 0}, upload=502178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4): protocol: expected piece hash; context canceled; EOF Dec 29 12:18:20.177729393 (572h49m20.048812428s) [noise=true , address=45.91.116.13:4162 , piece_id=UPESSP5ITLGEQCHYNL4VTSXLBB24O7QXZ5UB6FA4PC5JWHGUVCAQ, seg_pos={0 0}, upload=502178]: Long tail cancellation due to optimal threshold Dec 29 12:18:20.177744652 (572h49m20.048827687s) [seg_pos={0 0}, upload=502178]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:18:20.177850734 (572h49m20.048933769s) [upload=502178]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:18:20.177856495 (572h49m20.04893953s) [upload=502178]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:18:20.468750169 (572h49m20.339833204s) [upload=502178]: Done uploading segments... Dec 29 12:18:20.468821605 (572h49m20.33990465s) [upload=502178]: Upload finished. err: <nil> Duration: 38.914480508s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XX.XXX.X.X....XXXXXX..X.X.......X.XX...X.......X.X..XXX....X] duration: 30.004071447s trace: https://jaeger.dev.storj.io/trace/6047085ef9629cee FAIL (2025-12-29 12:26:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng segments... Dec 29 12:26:26.381455222 (572h57m26.252538267s) [upload=502294]: Got next segment. Inline: false Dec 29 12:26:26.381569239 (572h57m26.252652274s) [upload=502294]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:26:26.381575021 (572h57m26.252658066s) [upload=502294]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:26:26.625888218 (572h57m26.496971263s) [seg_pos={0 0}, upload=502294]: Begin upload segment... Dec 29 12:26:26.625973239 (572h57m26.497056284s) [seg_pos={0 0}, upload=502294]: Done begin upload segment. Dec 29 12:26:26.626040047 (572h57m26.497123102s) [noise=true , address=77.90.31.10:4520 , piece_id=IHPHB6JKOOWSS5ELL4HXBIYJ3O5UIK4IDKIYVEWAIXH57KFBW6NQ, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.626816195 (572h57m26.49789924s) [noise=true , address=145.79.227.15:4874 , piece_id=E3AAB4JEHERJ2R6HGP4MLRNXNMK7W4ICRPWIFWUMQ3OJNRHYOP5Q, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.627162455 (572h57m26.4982455s) [noise=true , address=145.79.227.40:4049 , piece_id=CNLK7WROCZCPC2PKAQDKYQXUL6ZUBGD4E4E3E6PPVYEIS5ITLE7Q, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.627543349 (572h57m26.498626384s) [noise=true , address=45.91.116.9:4262 , piece_id=JFLGJIBVJXSJE54ECBS4U33LARKUHPBAQB6NUTONBHMBWBM6H6XA, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.627886933 (572h57m26.498969988s) [noise=true , address=145.79.227.38:4977 , piece_id=V3H2S43AEUMFLLDTXSGCA6BFDJUPZLKAUDB3RWJVG3PK6AEPXY6A, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.628205891 (572h57m26.499288937s) [noise=true , address=145.79.227.47:4266 , piece_id=Q2FMIE5XX4LZD4GI6SSZESJZVMZBE4KGPC2H36EY7KY65EO5G3QA, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.628648774 (572h57m26.499731809s) [noise=true , address=145.79.227.42:4123 , piece_id=7EHC46NCH7XN7ZPPPM2ROQ2PSHOYAHWR4EGUB76X3Y62OFOPAPTA, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.628954356 (572h57m26.500037401s) [noise=true , address=145.79.227.45:4229 , piece_id=P5OCXGBPPBKMYJTGYCGYNO5WGIMZLFVZLUCB3GUQCJIRG637633Q, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.630379450 (572h57m26.501462495s) [noise=true , address=45.91.116.6:4976 , piece_id=SBTB7O3HFTFSG6CMUDHRDSNFT4HRGM522FNUOZAUOKV2KPGOKGTQ, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.630707424 (572h57m26.501790459s) [noise=true , address=145.79.227.14:4837 , piece_id=SQCFNKFPX7KK4LKL52NVS2KN32SWSEVPGWWUTTVW33DKCN4JILNA, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.631005512 (572h57m26.502088557s) [noise=true , address=145.79.227.2:4115 , piece_id=BXBFY2EP67QEZAX56ASJINVOZGAO6H2XAD3SBIR7MVA6VUFECNZA, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.631272210 (572h57m26.502355255s) [noise=true , address=77.90.31.14:4662 , piece_id=G3L5FXFH2HS3AR5XLPGHL6XL4FJ7EUBHYU7AF3L3FIJKOPBBDQTA, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.632025514 (572h57m26.503108559s) [noise=true , address=45.91.116.13:4156 , piece_id=O2FEYLUXWNE6O4WFP3PHDBA5DMLV5X7Q35N6O7ADD5OHV6XRVC4A, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.632463408 (572h57m26.503546453s) [noise=true , address=45.91.116.7:4009 , piece_id=CTN2ZM4XBB5XHQFHOAPW6IQ5EB773ZPBW7TOBV74V2F3SM25AZWA, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.633007343 (572h57m26.504090388s) [noise=true , address=45.91.116.4:4940 , piece_id=3676OHD74HKXC6AB5LPJEE3SAVVZZ36BGLWG56QIVFETU4C44DJQ, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:26.633424848 (572h57m26.504507883s) [noise=true , address=145.79.227.12:4767 , piece_id=35QG5CVAG33YCHPXLEUMCVGW23UFPGAPX6S4TEEASSCNPVO4HYAA, seg_pos={0 0}, upload=502294]: Uploading piece... Dec 29 12:26:28.330358171 (572h57m28.201441206s) [upload=502294]: Next returned nil segment Dec 29 12:26:28.330375173 (572h57m28.201458218s) [upload=502294]: Waiting for error group managing segments... Dec 29 12:26:28.603285514 (572h57m28.47436855s) [noise=true , address=145.79.227.12:4767 , piece_id=35QG5CVAG33YCHPXLEUMCVGW23UFPGAPX6S4TEEASSCNPVO4HYAA, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:28.618055339 (572h57m28.489138374s) [noise=true , address=145.79.227.42:4123 , piece_id=7EHC46NCH7XN7ZPPPM2ROQ2PSHOYAHWR4EGUB76X3Y62OFOPAPTA, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:28.618182089 (572h57m28.489265124s) [noise=true , address=145.79.227.45:4229 , piece_id=P5OCXGBPPBKMYJTGYCGYNO5WGIMZLFVZLUCB3GUQCJIRG637633Q, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:28.618845883 (572h57m28.489928928s) [noise=true , address=145.79.227.38:4977 , piece_id=V3H2S43AEUMFLLDTXSGCA6BFDJUPZLKAUDB3RWJVG3PK6AEPXY6A, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:28.620502508 (572h57m28.491585543s) [noise=true , address=145.79.227.15:4874 , piece_id=E3AAB4JEHERJ2R6HGP4MLRNXNMK7W4ICRPWIFWUMQ3OJNRHYOP5Q, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:28.634073068 (572h57m28.505156103s) [noise=true , address=145.79.227.14:4837 , piece_id=SQCFNKFPX7KK4LKL52NVS2KN32SWSEVPGWWUTTVW33DKCN4JILNA, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:28.635391348 (572h57m28.506474383s) [noise=true , address=145.79.227.2:4115 , piece_id=BXBFY2EP67QEZAX56ASJINVOZGAO6H2XAD3SBIR7MVA6VUFECNZA, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:28.645652678 (572h57m28.516735713s) [noise=true , address=145.79.227.47:4266 , piece_id=Q2FMIE5XX4LZD4GI6SSZESJZVMZBE4KGPC2H36EY7KY65EO5G3QA, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:28.804359379 (572h57m28.675442424s) [noise=true , address=77.90.31.14:4662 , piece_id=G3L5FXFH2HS3AR5XLPGHL6XL4FJ7EUBHYU7AF3L3FIJKOPBBDQTA, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:28.815560217 (572h57m28.686643262s) [noise=true , address=77.90.31.10:4520 , piece_id=IHPHB6JKOOWSS5ELL4HXBIYJ3O5UIK4IDKIYVEWAIXH57KFBW6NQ, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:28.944227930 (572h57m28.815310965s) [noise=true , address=45.91.116.13:4156 , piece_id=O2FEYLUXWNE6O4WFP3PHDBA5DMLV5X7Q35N6O7ADD5OHV6XRVC4A, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:29.018960361 (572h57m28.890043406s) [noise=true , address=45.91.116.9:4262 , piece_id=JFLGJIBVJXSJE54ECBS4U33LARKUHPBAQB6NUTONBHMBWBM6H6XA, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:29.040090536 (572h57m28.911173581s) [noise=true , address=45.91.116.6:4976 , piece_id=SBTB7O3HFTFSG6CMUDHRDSNFT4HRGM522FNUOZAUOKV2KPGOKGTQ, seg_pos={0 0}, upload=502294]: Done uploading piece. err: <nil> Dec 29 12:26:29.040113139 (572h57m28.911196184s) [seg_pos={0 0}, upload=502294]: Segment reached optimal threshold of 13 pieces. Dec 29 12:26:29.040314813 (572h57m28.911397858s) [noise=true , address=45.91.116.7:4009 , piece_id=CTN2ZM4XBB5XHQFHOAPW6IQ5EB773ZPBW7TOBV74V2F3SM25AZWA, seg_pos={0 0}, upload=502294]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:43924->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:43924->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43924->45.91.116.7:4009: use of closed network connection Dec 29 12:26:29.040322277 (572h57m28.911405312s) [noise=true , address=45.91.116.7:4009 , piece_id=CTN2ZM4XBB5XHQFHOAPW6IQ5EB773ZPBW7TOBV74V2F3SM25AZWA, seg_pos={0 0}, upload=502294]: Long tail cancellation due to optimal threshold Dec 29 12:26:29.040594947 (572h57m28.911677992s) [noise=true , address=45.91.116.4:4940 , piece_id=3676OHD74HKXC6AB5LPJEE3SAVVZZ36BGLWG56QIVFETU4C44DJQ, seg_pos={0 0}, upload=502294]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:53636->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:53636->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53636->45.91.116.4:4940: use of closed network connection Dec 29 12:26:29.040603643 (572h57m28.911686688s) [noise=true , address=45.91.116.4:4940 , piece_id=3676OHD74HKXC6AB5LPJEE3SAVVZZ36BGLWG56QIVFETU4C44DJQ, seg_pos={0 0}, upload=502294]: Long tail cancellation due to optimal threshold Dec 29 12:26:29.040766493 (572h57m28.911849538s) [noise=true , address=145.79.227.40:4049 , piece_id=CNLK7WROCZCPC2PKAQDKYQXUL6ZUBGD4E4E3E6PPVYEIS5ITLE7Q, seg_pos={0 0}, upload=502294]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 29 12:26:29.040774939 (572h57m28.911857984s) [noise=true , address=145.79.227.40:4049 , piece_id=CNLK7WROCZCPC2PKAQDKYQXUL6ZUBGD4E4E3E6PPVYEIS5ITLE7Q, seg_pos={0 0}, upload=502294]: Long tail cancellation due to optimal threshold Dec 29 12:26:29.040793515 (572h57m28.91187656s) [seg_pos={0 0}, upload=502294]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:26:29.040867666 (572h57m28.911950711s) [upload=502294]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:26:29.040873437 (572h57m28.911956482s) [upload=502294]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:26:29.335733775 (572h57m29.20681682s) [upload=502294]: Done uploading segments... Dec 29 12:26:29.335758983 (572h57m29.206842018s) [upload=502294]: Upload finished. err: <nil> Duration: 30.004103479s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X.......X........] duration: 38.201167038s trace: https://jaeger.dev.storj.io/trace/54cc96062d54e547 FAIL (2025-12-29 12:26:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 7.148417227 (572h49m17.019500273s) [noise=true , address=45.91.116.13:4162 , piece_id=UPESSP5ITLGEQCHYNL4VTSXLBB24O7QXZ5UB6FA4PC5JWHGUVCAQ, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.148925174 (572h49m17.020008209s) [noise=true , address=145.79.227.43:4169 , piece_id=O7II3XWLMOJGXSOVGJ6ITQWOW2OIL7PPYWXNQTVWNYBNU3E4FFKA, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.149222871 (572h49m17.020305906s) [noise=true , address=145.79.227.34:4924 , piece_id=LDEYPDBM6HDZKO2YHAAUGXHFXFPCQ74IH5JIRFYBPUMIS5HFXRBQ, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.149466225 (572h49m17.020549801s) [noise=true , address=145.79.227.44:4197 , piece_id=4D4YHRYNXGE5YCZH7AL3IIN6Q6FI6TLGXECM3SAZQDCKOGN4F2OQ, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.149621069 (572h49m17.020704114s) [noise=true , address=64.49.13.169:4710 , piece_id=WVVMYAG6FYQDJIWNKAAH4GREERSAJN2SOQDRMD3WFK7NMLONL4MQ, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.149918967 (572h49m17.021002012s) [noise=true , address=145.79.227.4:4195 , piece_id=6UKJB3VYO7UL4TW5IVTDA4KTXTJ2YFDN2ADALMTYQVMTSLKYGVDA, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.149948984 (572h49m17.021032029s) [noise=true , address=77.90.31.18:4839 , piece_id=26G6IXW2YT7AUPAM4R4SQQFXXT57HSS2IGWMIXJNN5NEAHSEZCIA, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.150229729 (572h49m17.021312764s) [noise=true , address=77.90.31.13:4650 , piece_id=2G6KWTAPYS6FXIZK3SL5RD4OZ6DAOSSHUL5ZZ75PB4ETPNF7MH7A, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.150278251 (572h49m17.021361296s) [noise=true , address=145.79.227.5:4231 , piece_id=D6EJVCDCFO2GP4PE32ISCUSWCQGMY6NX4EYWVHG2T3DF3LM7DW3Q, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.150629960 (572h49m17.021712995s) [noise=true , address=145.79.227.40:4052 , piece_id=JFC3UQOZSXCYUGIKJKRI73RLJ2QGIQTYN37PSECCEDMJPK2UYSDA, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.150991189 (572h49m17.022074224s) [noise=true , address=77.90.31.12:4617 , piece_id=2GNINSXEB5KKEDT2WCIK3UWDICDUS4MBMIQQ7VKTQ5JGNY6O55JQ, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.151215005 (572h49m17.02229804s) [noise=true , address=193.149.164.143:4702 , piece_id=6UADW5KZGEO4EHPJP2TLDDZZBGR2A2J2LSLNN6WR4HV5MVO5UK6A, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.151269889 (572h49m17.022352925s) [noise=true , address=145.79.227.30:4780 , piece_id=MNJQ3WRD3J6TBKAEVMRHV467KM2OGUVT2YVDBXIN32HZJ55X73IQ, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.151578597 (572h49m17.022661633s) [noise=true , address=145.79.227.35:4501 , piece_id=S335GIDTNGOW2JXIX5OZ7HHBQI3K254ZR2TPCU27ZN2K6ZS3CVOA, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.151863319 (572h49m17.022946354s) [noise=true , address=77.90.31.10:4531 , piece_id=T344ZC45IB4573ARJASYIANXG24GH26VI6VBPUR3CPYK77K7Z7TQ, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.151922863 (572h49m17.023005908s) [noise=true , address=145.79.227.29:4974 , piece_id=Q4PXHIZSLPKYURXKZ6B6HJ7YANRN5QHTZWUWZ3MVI3G3O7JFKIJA, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.152129156 (572h49m17.023212201s) [noise=true , address=145.79.227.13:4807 , piece_id=Y67QBVSHT5WUO4DG7LW2DM67ZILNFVZFUFSQSBF4GGVYJ24SMGOA, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:18.050161080 (572h49m17.921244125s) [noise=true , address=193.149.164.143:4702 , piece_id=6UADW5KZGEO4EHPJP2TLDDZZBGR2A2J2LSLNN6WR4HV5MVO5UK6A, seg_pos={0 0}, upload=502178]: Done uploading piece. err: ecclient: upload failed (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1, address:193.149.164.143:4702): protocol: storage node overloaded, request limit: 2000 Dec 29 12:18:18.609488842 (572h49m18.480571877s) [upload=502178]: Next returned nil segment Dec 29 12:18:18.609505194 (572h49m18.480588259s) [upload=502178]: Waiting for error group managing segments... Dec 29 12:18:18.996027579 (572h49m18.867110624s) [noise=true , address=145.79.227.5:4231 , piece_id=D6EJVCDCFO2GP4PE32ISCUSWCQGMY6NX4EYWVHG2T3DF3LM7DW3Q, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.015970613 (572h49m18.887053658s) [noise=true , address=145.79.227.4:4195 , piece_id=6UKJB3VYO7UL4TW5IVTDA4KTXTJ2YFDN2ADALMTYQVMTSLKYGVDA, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.024599794 (572h49m18.895682829s) [noise=true , address=145.79.227.29:4974 , piece_id=Q4PXHIZSLPKYURXKZ6B6HJ7YANRN5QHTZWUWZ3MVI3G3O7JFKIJA, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.025693498 (572h49m18.896776533s) [noise=true , address=145.79.227.13:4807 , piece_id=Y67QBVSHT5WUO4DG7LW2DM67ZILNFVZFUFSQSBF4GGVYJ24SMGOA, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.220567296 (572h49m19.091650341s) [noise=true , address=77.90.31.12:4617 , piece_id=2GNINSXEB5KKEDT2WCIK3UWDICDUS4MBMIQQ7VKTQ5JGNY6O55JQ, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.308488310 (572h49m19.179571345s) [noise=true , address=77.90.31.13:4650 , piece_id=2G6KWTAPYS6FXIZK3SL5RD4OZ6DAOSSHUL5ZZ75PB4ETPNF7MH7A, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.413574289 (572h49m19.284657324s) [noise=true , address=77.90.31.18:4839 , piece_id=26G6IXW2YT7AUPAM4R4SQQFXXT57HSS2IGWMIXJNN5NEAHSEZCIA, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.871377594 (572h49m19.742460639s) [noise=true , address=145.79.227.30:4780 , piece_id=MNJQ3WRD3J6TBKAEVMRHV467KM2OGUVT2YVDBXIN32HZJ55X73IQ, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.939637331 (572h49m19.810720366s) [noise=true , address=145.79.227.35:4501 , piece_id=S335GIDTNGOW2JXIX5OZ7HHBQI3K254ZR2TPCU27ZN2K6ZS3CVOA, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.958323732 (572h49m19.829406767s) [noise=true , address=145.79.227.44:4197 , piece_id=4D4YHRYNXGE5YCZH7AL3IIN6Q6FI6TLGXECM3SAZQDCKOGN4F2OQ, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:20.001969170 (572h49m19.873052215s) [noise=true , address=145.79.227.40:4052 , piece_id=JFC3UQOZSXCYUGIKJKRI73RLJ2QGIQTYN37PSECCEDMJPK2UYSDA, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:20.015729351 (572h49m19.886812386s) [noise=true , address=145.79.227.34:4924 , piece_id=LDEYPDBM6HDZKO2YHAAUGXHFXFPCQ74IH5JIRFYBPUMIS5HFXRBQ, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:20.114412729 (572h49m19.985495764s) [noise=true , address=145.79.227.43:4169 , piece_id=O7II3XWLMOJGXSOVGJ6ITQWOW2OIL7PPYWXNQTVWNYBNU3E4FFKA, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:20.177148507 (572h49m20.048231542s) [noise=true , address=77.90.31.10:4531 , piece_id=T344ZC45IB4573ARJASYIANXG24GH26VI6VBPUR3CPYK77K7Z7TQ, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:20.177170508 (572h49m20.048253544s) [seg_pos={0 0}, upload=502178]: Segment reached optimal threshold of 14 pieces. Dec 29 12:18:20.177339190 (572h49m20.048422225s) [noise=true , address=64.49.13.169:4710 , piece_id=WVVMYAG6FYQDJIWNKAAH4GREERSAJN2SOQDRMD3WFK7NMLONL4MQ, seg_pos={0 0}, upload=502178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57032->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57032->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57032->64.49.13.169:4710: use of closed network connection Dec 29 12:18:20.177420124 (572h49m20.048503179s) [noise=true , address=64.49.13.169:4710 , piece_id=WVVMYAG6FYQDJIWNKAAH4GREERSAJN2SOQDRMD3WFK7NMLONL4MQ, seg_pos={0 0}, upload=502178]: Long tail cancellation due to optimal threshold Dec 29 12:18:20.177586831 (572h49m20.048669866s) [noise=true , address=145.79.227.49:4343 , piece_id=JPXHILNUTSHS4RUP5CRSUFWEVASM55PBY4MQCHK47AURG5XZDS7Q, seg_pos={0 0}, upload=502178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf): protocol: expected piece hash; context canceled; EOF Dec 29 12:18:20.177593945 (572h49m20.04867698s) [noise=true , address=145.79.227.49:4343 , piece_id=JPXHILNUTSHS4RUP5CRSUFWEVASM55PBY4MQCHK47AURG5XZDS7Q, seg_pos={0 0}, upload=502178]: Long tail cancellation due to optimal threshold Dec 29 12:18:20.177724033 (572h49m20.048807068s) [noise=true , address=45.91.116.13:4162 , piece_id=UPESSP5ITLGEQCHYNL4VTSXLBB24O7QXZ5UB6FA4PC5JWHGUVCAQ, seg_pos={0 0}, upload=502178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4): protocol: expected piece hash; context canceled; EOF Dec 29 12:18:20.177729393 (572h49m20.048812428s) [noise=true , address=45.91.116.13:4162 , piece_id=UPESSP5ITLGEQCHYNL4VTSXLBB24O7QXZ5UB6FA4PC5JWHGUVCAQ, seg_pos={0 0}, upload=502178]: Long tail cancellation due to optimal threshold Dec 29 12:18:20.177744652 (572h49m20.048827687s) [seg_pos={0 0}, upload=502178]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:18:20.177850734 (572h49m20.048933769s) [upload=502178]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:18:20.177856495 (572h49m20.04893953s) [upload=502178]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:18:20.468750169 (572h49m20.339833204s) [upload=502178]: Done uploading segments... Dec 29 12:18:20.468821605 (572h49m20.33990465s) [upload=502178]: Upload finished. err: <nil> Duration: 38.201192175s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [X.XXXX...XXX...XXX..XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXX] duration: 5.001088624s trace: https://jaeger.dev.storj.io/trace/188a83983b8659b3 FAIL (2025-12-29 12:26:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001109725s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.XX.XXX.X.X....XXXXXX..X.X.......X.XX...X.......X.X..XXX....] duration: 5.641404685s trace: https://jaeger.dev.storj.io/trace/31561a0261895c45 FAIL (2025-12-29 12:25:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 29 12:22:43.293612504 (572h53m43.164695559s) [upload=502242]: Starting upload Dec 29 12:22:43.293674391 (572h53m43.164757436s) [upload=502242]: Done starting upload Dec 29 12:22:43.293739686 (572h53m43.164822741s) [upload=502242]: Uploading segments... Dec 29 12:22:43.294290385 (572h53m43.16537343s) [upload=502242]: Got next segment. Inline: false Dec 29 12:22:43.294340521 (572h53m43.165423556s) [upload=502242]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:22:43.294347023 (572h53m43.165430058s) [upload=502242]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:22:43.539446638 (572h53m43.410529673s) [seg_pos={0 0}, upload=502242]: Begin upload segment... Dec 29 12:22:43.539525900 (572h53m43.410608935s) [seg_pos={0 0}, upload=502242]: Done begin upload segment. Dec 29 12:22:43.539551748 (572h53m43.410634793s) [noise=true , address=145.79.227.12:4767 , piece_id=5DUFEZOFHYRSIOVAJRH4VTTW52TX5NLIT4RCK6RXT26WDFUBVDJQ, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.539653281 (572h53m43.410736316s) [noise=true , address=45.91.116.5:4297 , piece_id=F54IX7KAJOJQ5UTM5REBB7EQFATMDPEHEZC6GQKQJDMPAOZDWPVA, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.540172510 (572h53m43.411255545s) [noise=true , address=45.91.116.10:4082 , piece_id=SHVHPT3LK4XZL3SET6F2BVQDNWYCIA4ULQNEMLTKZINE5POW3QWQ, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.540535903 (572h53m43.411618938s) [noise=true , address=45.91.116.13:4156 , piece_id=2G3DG2JEEJ7OWFKR22DFR3RTL2FO5LLRIGRPHXBH5AXH57UQMJYA, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.540739299 (572h53m43.411822344s) [noise=true , address=45.91.116.6:4976 , piece_id=VYLHBBOOTYLMFLZHKW4QVQRVXEDNQ7GD5APY32PVJ4YWNBYUOMMQ, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.541067405 (572h53m43.41215045s) [noise=true , address=145.79.227.51:4411 , piece_id=HKNFUIKRZEMHR4XW4YGPVEIBGP3NL6EKCF72UHF4TDTK3BESHDKA, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.541090047 (572h53m43.412173082s) [noise=true , address=145.79.227.6:4257 , piece_id=AR5F4DN4GPBPTSSYPDIRJYHQOOUCYCK5FTUFMXGSFIUOZAMCCRSA, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.541534954 (572h53m43.412617989s) [noise=true , address=145.79.227.5:4223 , piece_id=4X44MXO4F7O7HJBN6G6C4SFO2N224C7JCYCVVZ22I6PBVZYH6LNA, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.541605409 (572h53m43.412688444s) [noise=true , address=77.90.31.17:4773 , piece_id=D3VSGPEUFI2D6FWL23QOPKYEAA7ZS5EUXOJIB7XX7FSHKFKMZB3Q, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.541897444 (572h53m43.412980489s) [noise=true , address=145.79.227.2:4113 , piece_id=4NEWBO5BBPRGXVL6N4D3C43UWZBXTUYUJDGGWXMFFT2UM5776ZJQ, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.542018926 (572h53m43.413101971s) [noise=true , address=77.90.31.11:4556 , piece_id=XY5XPKVVX6BWLNYS2HDY5B2LM2HOCRHCY2LKJGAORACI4O46JVUQ, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.542239386 (572h53m43.413322421s) [noise=true , address=145.79.227.15:4873 , piece_id=NFX7XMHZFZH5NHIQGFIK3SKVHPYIMEBCI2BVTXFHF55OGMZVYRNA, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.542352682 (572h53m43.413435717s) [noise=true , address=145.79.227.32:4834 , piece_id=U4FL74Q3OUT5TU3EUOBIOBXTCWAOEUEMWAJYPVVODDHEUGMB3M2Q, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.542769426 (572h53m43.413852461s) [noise=true , address=45.91.116.14:4190 , piece_id=E7FJ4IE56DOHGFGMCIT7GKCARYTLIKBR34UZ6KWJSOEM7NC7TVBQ, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.543162534 (572h53m43.414245569s) [noise=true , address=145.79.227.24:4582 , piece_id=3664UNGQ3IXMQQZWVXN7CGXCC7A3CLLEJCLURNNZTF64UOLWJEEQ, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.544555176 (572h53m43.415638211s) [noise=true , address=45.91.116.12:4226 , piece_id=7MSH7VL4GFYLYNLNLYQ2X2KZHG33CYXW4PUYR6EX5A6FO7NZPYGA, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:45.240733568 (572h53m45.111816604s) [upload=502242]: Next returned nil segment Dec 29 12:22:45.240745761 (572h53m45.111828796s) [upload=502242]: Waiting for error group managing segments... Dec 29 12:22:45.513367814 (572h53m45.384450849s) [noise=true , address=145.79.227.12:4767 , piece_id=5DUFEZOFHYRSIOVAJRH4VTTW52TX5NLIT4RCK6RXT26WDFUBVDJQ, seg_pos={0 0}, upload=502242]: Done uploading piece. err: <nil> Dec 29 12:22:45.522976540 (572h53m45.394059585s) [noise=true , address=145.79.227.32:4834 , piece_id=U4FL74Q3OUT5TU3EUOBIOBXTCWAOEUEMWAJYPVVODDHEUGMB3M2Q, seg_pos={0 0}, upload=502242]: Done uploading piece. err: <nil> Dec 29 12:22:45.526549694 (572h53m45.397632729s) [noise=true , address=145.79.227.15:4873 , piece_id=NFX7XMHZFZH5NHIQGFIK3SKVHPYIMEBCI2BVTXFHF55OGMZVYRNA, seg_pos={0 0}, upload=502242]: Done uploading piece. err: <nil> Dec 29 12:22:45.527483763 (572h53m45.398566798s) [noise=true , address=145.79.227.6:4257 , piece_id=AR5F4DN4GPBPTSSYPDIRJYHQOOUCYCK5FTUFMXGSFIUOZAMCCRSA, seg_pos={0 0}, upload=502242]: Done uploading piece. err: <nil> Dec 29 12:22:45.528179628 (572h53m45.399262663s) [noise=true , address=145.79.227.51:4411 , piece_id=HKNFUIKRZEMHR4XW4YGPVEIBGP3NL6EKCF72UHF4TDTK3BESHDKA, seg_pos={0 0}, upload=502242]: Done uploading piece. err: <nil> Dec 29 12:22:45.532282561 (572h53m45.403365596s) [noise=true , address=145.79.227.24:4582 , piece_id=3664UNGQ3IXMQQZWVXN7CGXCC7A3CLLEJCLURNNZTF64UOLWJEEQ, seg_pos={0 0}, upload=502242]: Done uploading piece. err: <nil> Dec 29 12:22:45.547407200 (572h53m45.418490235s) [noise=true , address=145.79.227.5:4223 , piece_id=4X44MXO4F7O7HJBN6G6C4SFO2N224C7JCYCVVZ22I6PBVZYH6LNA, seg_pos={0 0}, upload=502242]: Done uploading piece. err: <nil> Dec 29 12:22:45.551402257 (572h53m45.422485302s) [noise=true , address=145.79.227.2:4113 , piece_id=4NEWBO5BBPRGXVL6N4D3C43UWZBXTUYUJDGGWXMFFT2UM5776ZJQ, seg_pos={0 0}, upload=502242]: Done uploading piece. err: <nil> Dec 29 12:22:45.823994002 (572h53m45.695077037s) [noise=true , address=45.91.116.10:4082 , piece_id=SHVHPT3LK4XZL3SET6F2BVQDNWYCIA4ULQNEMLTKZINE5POW3QWQ, seg_pos={0 0}, upload=502242]: Done uploading piece. err: <nil> Dec 29 12:22:45.824741105 (572h53m45.69582415s) [noise=true , address=77.90.31.17:4773 , piece_id=D3VSGPEUFI2D6FWL23QOPKYEAA7ZS5EUXOJIB7XX7FSHKFKMZB3Q, seg_pos={0 0}, upload=502242]: Done uploading piece. err: <nil> Dec 29 12:22:45.838851143 (572h53m45.709934178s) [noise=true , address=77.90.31.11:4556 , piece_id=XY5XPKVVX6BWLNYS2HDY5B2LM2HOCRHCY2LKJGAORACI4O46JVUQ, seg_pos={0 0}, upload=502242]: Done uploading piece. err: <nil> Dec 29 12:22:45.929331229 (572h53m45.800414264s) [noise=true , address=45.91.116.14:4190 , piece_id=E7FJ4IE56DOHGFGMCIT7GKCARYTLIKBR34UZ6KWJSOEM7NC7TVBQ, seg_pos={0 0}, upload=502242]: Done uploading piece. err: <nil> Dec 29 12:22:45.944728408 (572h53m45.815811443s) [noise=true , address=45.91.116.12:4226 , piece_id=7MSH7VL4GFYLYNLNLYQ2X2KZHG33CYXW4PUYR6EX5A6FO7NZPYGA, seg_pos={0 0}, upload=502242]: Done uploading piece. err: <nil> Dec 29 12:22:45.944753446 (572h53m45.815836491s) [seg_pos={0 0}, upload=502242]: Segment reached optimal threshold of 13 pieces. Dec 29 12:22:45.944930703 (572h53m45.816013748s) [noise=true , address=45.91.116.5:4297 , piece_id=F54IX7KAJOJQ5UTM5REBB7EQFATMDPEHEZC6GQKQJDMPAOZDWPVA, seg_pos={0 0}, upload=502242]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:39826->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:39826->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39826->45.91.116.5:4297: use of closed network connection Dec 29 12:22:45.944936564 (572h53m45.816019599s) [noise=true , address=45.91.116.5:4297 , piece_id=F54IX7KAJOJQ5UTM5REBB7EQFATMDPEHEZC6GQKQJDMPAOZDWPVA, seg_pos={0 0}, upload=502242]: Long tail cancellation due to optimal threshold Dec 29 12:22:45.945048759 (572h53m45.816131804s) [noise=true , address=45.91.116.6:4976 , piece_id=VYLHBBOOTYLMFLZHKW4QVQRVXEDNQ7GD5APY32PVJ4YWNBYUOMMQ, seg_pos={0 0}, upload=502242]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 29 12:22:45.945058196 (572h53m45.816141231s) [noise=true , address=45.91.116.6:4976 , piece_id=VYLHBBOOTYLMFLZHKW4QVQRVXEDNQ7GD5APY32PVJ4YWNBYUOMMQ, seg_pos={0 0}, upload=502242]: Long tail cancellation due to optimal threshold Dec 29 12:22:45.945238850 (572h53m45.816321885s) [noise=true , address=45.91.116.13:4156 , piece_id=2G3DG2JEEJ7OWFKR22DFR3RTL2FO5LLRIGRPHXBH5AXH57UQMJYA, seg_pos={0 0}, upload=502242]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 29 12:22:45.945248989 (572h53m45.816332024s) [noise=true , address=45.91.116.13:4156 , piece_id=2G3DG2JEEJ7OWFKR22DFR3RTL2FO5LLRIGRPHXBH5AXH57UQMJYA, seg_pos={0 0}, upload=502242]: Long tail cancellation due to optimal threshold Dec 29 12:22:45.945270711 (572h53m45.816353746s) [seg_pos={0 0}, upload=502242]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:22:45.945349410 (572h53m45.816432455s) [upload=502242]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:22:45.945356714 (572h53m45.816439769s) [upload=502242]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:22:46.238508068 (572h53m46.109591103s) [upload=502242]: Done uploading segments... Dec 29 12:22:46.238527545 (572h53m46.10961058s) [upload=502242]: Upload finished. err: <nil> Duration: 5.641423109s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [XX.XXXX...XXX...XXX..XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXX] duration: 5.001327842s trace: https://jaeger.dev.storj.io/trace/2e86f0f0d52a3ef3 FAIL (2025-12-29 12:25:44 UTC) orbiter warning: context deadline exceeded Duration: 5.001349573s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X.......X.......] duration: 51.911692383s trace: https://jaeger.dev.storj.io/trace/54cc96062d54e547 FAIL (2025-12-29 12:25:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 7.148417227 (572h49m17.019500273s) [noise=true , address=45.91.116.13:4162 , piece_id=UPESSP5ITLGEQCHYNL4VTSXLBB24O7QXZ5UB6FA4PC5JWHGUVCAQ, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.148925174 (572h49m17.020008209s) [noise=true , address=145.79.227.43:4169 , piece_id=O7II3XWLMOJGXSOVGJ6ITQWOW2OIL7PPYWXNQTVWNYBNU3E4FFKA, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.149222871 (572h49m17.020305906s) [noise=true , address=145.79.227.34:4924 , piece_id=LDEYPDBM6HDZKO2YHAAUGXHFXFPCQ74IH5JIRFYBPUMIS5HFXRBQ, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.149466225 (572h49m17.020549801s) [noise=true , address=145.79.227.44:4197 , piece_id=4D4YHRYNXGE5YCZH7AL3IIN6Q6FI6TLGXECM3SAZQDCKOGN4F2OQ, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.149621069 (572h49m17.020704114s) [noise=true , address=64.49.13.169:4710 , piece_id=WVVMYAG6FYQDJIWNKAAH4GREERSAJN2SOQDRMD3WFK7NMLONL4MQ, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.149918967 (572h49m17.021002012s) [noise=true , address=145.79.227.4:4195 , piece_id=6UKJB3VYO7UL4TW5IVTDA4KTXTJ2YFDN2ADALMTYQVMTSLKYGVDA, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.149948984 (572h49m17.021032029s) [noise=true , address=77.90.31.18:4839 , piece_id=26G6IXW2YT7AUPAM4R4SQQFXXT57HSS2IGWMIXJNN5NEAHSEZCIA, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.150229729 (572h49m17.021312764s) [noise=true , address=77.90.31.13:4650 , piece_id=2G6KWTAPYS6FXIZK3SL5RD4OZ6DAOSSHUL5ZZ75PB4ETPNF7MH7A, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.150278251 (572h49m17.021361296s) [noise=true , address=145.79.227.5:4231 , piece_id=D6EJVCDCFO2GP4PE32ISCUSWCQGMY6NX4EYWVHG2T3DF3LM7DW3Q, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.150629960 (572h49m17.021712995s) [noise=true , address=145.79.227.40:4052 , piece_id=JFC3UQOZSXCYUGIKJKRI73RLJ2QGIQTYN37PSECCEDMJPK2UYSDA, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.150991189 (572h49m17.022074224s) [noise=true , address=77.90.31.12:4617 , piece_id=2GNINSXEB5KKEDT2WCIK3UWDICDUS4MBMIQQ7VKTQ5JGNY6O55JQ, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.151215005 (572h49m17.02229804s) [noise=true , address=193.149.164.143:4702 , piece_id=6UADW5KZGEO4EHPJP2TLDDZZBGR2A2J2LSLNN6WR4HV5MVO5UK6A, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.151269889 (572h49m17.022352925s) [noise=true , address=145.79.227.30:4780 , piece_id=MNJQ3WRD3J6TBKAEVMRHV467KM2OGUVT2YVDBXIN32HZJ55X73IQ, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.151578597 (572h49m17.022661633s) [noise=true , address=145.79.227.35:4501 , piece_id=S335GIDTNGOW2JXIX5OZ7HHBQI3K254ZR2TPCU27ZN2K6ZS3CVOA, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.151863319 (572h49m17.022946354s) [noise=true , address=77.90.31.10:4531 , piece_id=T344ZC45IB4573ARJASYIANXG24GH26VI6VBPUR3CPYK77K7Z7TQ, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.151922863 (572h49m17.023005908s) [noise=true , address=145.79.227.29:4974 , piece_id=Q4PXHIZSLPKYURXKZ6B6HJ7YANRN5QHTZWUWZ3MVI3G3O7JFKIJA, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.152129156 (572h49m17.023212201s) [noise=true , address=145.79.227.13:4807 , piece_id=Y67QBVSHT5WUO4DG7LW2DM67ZILNFVZFUFSQSBF4GGVYJ24SMGOA, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:18.050161080 (572h49m17.921244125s) [noise=true , address=193.149.164.143:4702 , piece_id=6UADW5KZGEO4EHPJP2TLDDZZBGR2A2J2LSLNN6WR4HV5MVO5UK6A, seg_pos={0 0}, upload=502178]: Done uploading piece. err: ecclient: upload failed (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1, address:193.149.164.143:4702): protocol: storage node overloaded, request limit: 2000 Dec 29 12:18:18.609488842 (572h49m18.480571877s) [upload=502178]: Next returned nil segment Dec 29 12:18:18.609505194 (572h49m18.480588259s) [upload=502178]: Waiting for error group managing segments... Dec 29 12:18:18.996027579 (572h49m18.867110624s) [noise=true , address=145.79.227.5:4231 , piece_id=D6EJVCDCFO2GP4PE32ISCUSWCQGMY6NX4EYWVHG2T3DF3LM7DW3Q, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.015970613 (572h49m18.887053658s) [noise=true , address=145.79.227.4:4195 , piece_id=6UKJB3VYO7UL4TW5IVTDA4KTXTJ2YFDN2ADALMTYQVMTSLKYGVDA, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.024599794 (572h49m18.895682829s) [noise=true , address=145.79.227.29:4974 , piece_id=Q4PXHIZSLPKYURXKZ6B6HJ7YANRN5QHTZWUWZ3MVI3G3O7JFKIJA, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.025693498 (572h49m18.896776533s) [noise=true , address=145.79.227.13:4807 , piece_id=Y67QBVSHT5WUO4DG7LW2DM67ZILNFVZFUFSQSBF4GGVYJ24SMGOA, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.220567296 (572h49m19.091650341s) [noise=true , address=77.90.31.12:4617 , piece_id=2GNINSXEB5KKEDT2WCIK3UWDICDUS4MBMIQQ7VKTQ5JGNY6O55JQ, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.308488310 (572h49m19.179571345s) [noise=true , address=77.90.31.13:4650 , piece_id=2G6KWTAPYS6FXIZK3SL5RD4OZ6DAOSSHUL5ZZ75PB4ETPNF7MH7A, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.413574289 (572h49m19.284657324s) [noise=true , address=77.90.31.18:4839 , piece_id=26G6IXW2YT7AUPAM4R4SQQFXXT57HSS2IGWMIXJNN5NEAHSEZCIA, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.871377594 (572h49m19.742460639s) [noise=true , address=145.79.227.30:4780 , piece_id=MNJQ3WRD3J6TBKAEVMRHV467KM2OGUVT2YVDBXIN32HZJ55X73IQ, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.939637331 (572h49m19.810720366s) [noise=true , address=145.79.227.35:4501 , piece_id=S335GIDTNGOW2JXIX5OZ7HHBQI3K254ZR2TPCU27ZN2K6ZS3CVOA, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.958323732 (572h49m19.829406767s) [noise=true , address=145.79.227.44:4197 , piece_id=4D4YHRYNXGE5YCZH7AL3IIN6Q6FI6TLGXECM3SAZQDCKOGN4F2OQ, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:20.001969170 (572h49m19.873052215s) [noise=true , address=145.79.227.40:4052 , piece_id=JFC3UQOZSXCYUGIKJKRI73RLJ2QGIQTYN37PSECCEDMJPK2UYSDA, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:20.015729351 (572h49m19.886812386s) [noise=true , address=145.79.227.34:4924 , piece_id=LDEYPDBM6HDZKO2YHAAUGXHFXFPCQ74IH5JIRFYBPUMIS5HFXRBQ, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:20.114412729 (572h49m19.985495764s) [noise=true , address=145.79.227.43:4169 , piece_id=O7II3XWLMOJGXSOVGJ6ITQWOW2OIL7PPYWXNQTVWNYBNU3E4FFKA, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:20.177148507 (572h49m20.048231542s) [noise=true , address=77.90.31.10:4531 , piece_id=T344ZC45IB4573ARJASYIANXG24GH26VI6VBPUR3CPYK77K7Z7TQ, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:20.177170508 (572h49m20.048253544s) [seg_pos={0 0}, upload=502178]: Segment reached optimal threshold of 14 pieces. Dec 29 12:18:20.177339190 (572h49m20.048422225s) [noise=true , address=64.49.13.169:4710 , piece_id=WVVMYAG6FYQDJIWNKAAH4GREERSAJN2SOQDRMD3WFK7NMLONL4MQ, seg_pos={0 0}, upload=502178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57032->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57032->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57032->64.49.13.169:4710: use of closed network connection Dec 29 12:18:20.177420124 (572h49m20.048503179s) [noise=true , address=64.49.13.169:4710 , piece_id=WVVMYAG6FYQDJIWNKAAH4GREERSAJN2SOQDRMD3WFK7NMLONL4MQ, seg_pos={0 0}, upload=502178]: Long tail cancellation due to optimal threshold Dec 29 12:18:20.177586831 (572h49m20.048669866s) [noise=true , address=145.79.227.49:4343 , piece_id=JPXHILNUTSHS4RUP5CRSUFWEVASM55PBY4MQCHK47AURG5XZDS7Q, seg_pos={0 0}, upload=502178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf): protocol: expected piece hash; context canceled; EOF Dec 29 12:18:20.177593945 (572h49m20.04867698s) [noise=true , address=145.79.227.49:4343 , piece_id=JPXHILNUTSHS4RUP5CRSUFWEVASM55PBY4MQCHK47AURG5XZDS7Q, seg_pos={0 0}, upload=502178]: Long tail cancellation due to optimal threshold Dec 29 12:18:20.177724033 (572h49m20.048807068s) [noise=true , address=45.91.116.13:4162 , piece_id=UPESSP5ITLGEQCHYNL4VTSXLBB24O7QXZ5UB6FA4PC5JWHGUVCAQ, seg_pos={0 0}, upload=502178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4): protocol: expected piece hash; context canceled; EOF Dec 29 12:18:20.177729393 (572h49m20.048812428s) [noise=true , address=45.91.116.13:4162 , piece_id=UPESSP5ITLGEQCHYNL4VTSXLBB24O7QXZ5UB6FA4PC5JWHGUVCAQ, seg_pos={0 0}, upload=502178]: Long tail cancellation due to optimal threshold Dec 29 12:18:20.177744652 (572h49m20.048827687s) [seg_pos={0 0}, upload=502178]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:18:20.177850734 (572h49m20.048933769s) [upload=502178]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:18:20.177856495 (572h49m20.04893953s) [upload=502178]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:18:20.468750169 (572h49m20.339833204s) [upload=502178]: Done uploading segments... Dec 29 12:18:20.468821605 (572h49m20.33990465s) [upload=502178]: Upload finished. err: <nil> Duration: 51.911706109s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [..XX.XXX.X.X....XXXXXX..X.X.......X.XX...X.......X.X..XXX...] duration: 6.103083664s trace: https://jaeger.dev.storj.io/trace/31561a0261895c45 FAIL (2025-12-29 12:25:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 29 12:22:43.293612504 (572h53m43.164695559s) [upload=502242]: Starting upload Dec 29 12:22:43.293674391 (572h53m43.164757436s) [upload=502242]: Done starting upload Dec 29 12:22:43.293739686 (572h53m43.164822741s) [upload=502242]: Uploading segments... Dec 29 12:22:43.294290385 (572h53m43.16537343s) [upload=502242]: Got next segment. Inline: false Dec 29 12:22:43.294340521 (572h53m43.165423556s) [upload=502242]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:22:43.294347023 (572h53m43.165430058s) [upload=502242]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:22:43.539446638 (572h53m43.410529673s) [seg_pos={0 0}, upload=502242]: Begin upload segment... Dec 29 12:22:43.539525900 (572h53m43.410608935s) [seg_pos={0 0}, upload=502242]: Done begin upload segment. Dec 29 12:22:43.539551748 (572h53m43.410634793s) [noise=true , address=145.79.227.12:4767 , piece_id=5DUFEZOFHYRSIOVAJRH4VTTW52TX5NLIT4RCK6RXT26WDFUBVDJQ, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.539653281 (572h53m43.410736316s) [noise=true , address=45.91.116.5:4297 , piece_id=F54IX7KAJOJQ5UTM5REBB7EQFATMDPEHEZC6GQKQJDMPAOZDWPVA, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.540172510 (572h53m43.411255545s) [noise=true , address=45.91.116.10:4082 , piece_id=SHVHPT3LK4XZL3SET6F2BVQDNWYCIA4ULQNEMLTKZINE5POW3QWQ, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.540535903 (572h53m43.411618938s) [noise=true , address=45.91.116.13:4156 , piece_id=2G3DG2JEEJ7OWFKR22DFR3RTL2FO5LLRIGRPHXBH5AXH57UQMJYA, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.540739299 (572h53m43.411822344s) [noise=true , address=45.91.116.6:4976 , piece_id=VYLHBBOOTYLMFLZHKW4QVQRVXEDNQ7GD5APY32PVJ4YWNBYUOMMQ, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.541067405 (572h53m43.41215045s) [noise=true , address=145.79.227.51:4411 , piece_id=HKNFUIKRZEMHR4XW4YGPVEIBGP3NL6EKCF72UHF4TDTK3BESHDKA, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.541090047 (572h53m43.412173082s) [noise=true , address=145.79.227.6:4257 , piece_id=AR5F4DN4GPBPTSSYPDIRJYHQOOUCYCK5FTUFMXGSFIUOZAMCCRSA, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.541534954 (572h53m43.412617989s) [noise=true , address=145.79.227.5:4223 , piece_id=4X44MXO4F7O7HJBN6G6C4SFO2N224C7JCYCVVZ22I6PBVZYH6LNA, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.541605409 (572h53m43.412688444s) [noise=true , address=77.90.31.17:4773 , piece_id=D3VSGPEUFI2D6FWL23QOPKYEAA7ZS5EUXOJIB7XX7FSHKFKMZB3Q, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.541897444 (572h53m43.412980489s) [noise=true , address=145.79.227.2:4113 , piece_id=4NEWBO5BBPRGXVL6N4D3C43UWZBXTUYUJDGGWXMFFT2UM5776ZJQ, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.542018926 (572h53m43.413101971s) [noise=true , address=77.90.31.11:4556 , piece_id=XY5XPKVVX6BWLNYS2HDY5B2LM2HOCRHCY2LKJGAORACI4O46JVUQ, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.542239386 (572h53m43.413322421s) [noise=true , address=145.79.227.15:4873 , piece_id=NFX7XMHZFZH5NHIQGFIK3SKVHPYIMEBCI2BVTXFHF55OGMZVYRNA, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.542352682 (572h53m43.413435717s) [noise=true , address=145.79.227.32:4834 , piece_id=U4FL74Q3OUT5TU3EUOBIOBXTCWAOEUEMWAJYPVVODDHEUGMB3M2Q, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.542769426 (572h53m43.413852461s) [noise=true , address=45.91.116.14:4190 , piece_id=E7FJ4IE56DOHGFGMCIT7GKCARYTLIKBR34UZ6KWJSOEM7NC7TVBQ, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.543162534 (572h53m43.414245569s) [noise=true , address=145.79.227.24:4582 , piece_id=3664UNGQ3IXMQQZWVXN7CGXCC7A3CLLEJCLURNNZTF64UOLWJEEQ, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.544555176 (572h53m43.415638211s) [noise=true , address=45.91.116.12:4226 , piece_id=7MSH7VL4GFYLYNLNLYQ2X2KZHG33CYXW4PUYR6EX5A6FO7NZPYGA, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:45.240733568 (572h53m45.111816604s) [upload=502242]: Next returned nil segment Dec 29 12:22:45.240745761 (572h53m45.111828796s) [upload=502242]: Waiting for error group managing segments... Dec 29 12:22:45.513367814 (572h53m45.384450849s) [noise=true , address=145.79.227.12:4767 , piece_id=5DUFEZOFHYRSIOVAJRH4VTTW52TX5NLIT4RCK6RXT26WDFUBVDJQ, seg_pos={0 0}, upload=502242]: Done uploading piece. err: <nil> Dec 29 12:22:45.522976540 (572h53m45.394059585s) [noise=true , address=145.79.227.32:4834 , piece_id=U4FL74Q3OUT5TU3EUOBIOBXTCWAOEUEMWAJYPVVODDHEUGMB3M2Q, seg_pos={0 0}, upload=502242]: Done uploading piece. err: <nil> Dec 29 12:22:45.526549694 (572h53m45.397632729s) [noise=true , address=145.79.227.15:4873 , piece_id=NFX7XMHZFZH5NHIQGFIK3SKVHPYIMEBCI2BVTXFHF55OGMZVYRNA, seg_pos={0 0}, upload=502242]: Done uploading piece. err: <nil> Dec 29 12:22:45.527483763 (572h53m45.398566798s) [noise=true , address=145.79.227.6:4257 , piece_id=AR5F4DN4GPBPTSSYPDIRJYHQOOUCYCK5FTUFMXGSFIUOZAMCCRSA, seg_pos={0 0}, upload=502242]: Done uploading piece. err: <nil> Dec 29 12:22:45.528179628 (572h53m45.399262663s) [noise=true , address=145.79.227.51:4411 , piece_id=HKNFUIKRZEMHR4XW4YGPVEIBGP3NL6EKCF72UHF4TDTK3BESHDKA, seg_pos={0 0}, upload=502242]: Done uploading piece. err: <nil> Dec 29 12:22:45.532282561 (572h53m45.403365596s) [noise=true , address=145.79.227.24:4582 , piece_id=3664UNGQ3IXMQQZWVXN7CGXCC7A3CLLEJCLURNNZTF64UOLWJEEQ, seg_pos={0 0}, upload=502242]: Done uploading piece. err: <nil> Dec 29 12:22:45.547407200 (572h53m45.418490235s) [noise=true , address=145.79.227.5:4223 , piece_id=4X44MXO4F7O7HJBN6G6C4SFO2N224C7JCYCVVZ22I6PBVZYH6LNA, seg_pos={0 0}, upload=502242]: Done uploading piece. err: <nil> Dec 29 12:22:45.551402257 (572h53m45.422485302s) [noise=true , address=145.79.227.2:4113 , piece_id=4NEWBO5BBPRGXVL6N4D3C43UWZBXTUYUJDGGWXMFFT2UM5776ZJQ, seg_pos={0 0}, upload=502242]: Done uploading piece. err: <nil> Dec 29 12:22:45.823994002 (572h53m45.695077037s) [noise=true , address=45.91.116.10:4082 , piece_id=SHVHPT3LK4XZL3SET6F2BVQDNWYCIA4ULQNEMLTKZINE5POW3QWQ, seg_pos={0 0}, upload=502242]: Done uploading piece. err: <nil> Dec 29 12:22:45.824741105 (572h53m45.69582415s) [noise=true , address=77.90.31.17:4773 , piece_id=D3VSGPEUFI2D6FWL23QOPKYEAA7ZS5EUXOJIB7XX7FSHKFKMZB3Q, seg_pos={0 0}, upload=502242]: Done uploading piece. err: <nil> Dec 29 12:22:45.838851143 (572h53m45.709934178s) [noise=true , address=77.90.31.11:4556 , piece_id=XY5XPKVVX6BWLNYS2HDY5B2LM2HOCRHCY2LKJGAORACI4O46JVUQ, seg_pos={0 0}, upload=502242]: Done uploading piece. err: <nil> Dec 29 12:22:45.929331229 (572h53m45.800414264s) [noise=true , address=45.91.116.14:4190 , piece_id=E7FJ4IE56DOHGFGMCIT7GKCARYTLIKBR34UZ6KWJSOEM7NC7TVBQ, seg_pos={0 0}, upload=502242]: Done uploading piece. err: <nil> Dec 29 12:22:45.944728408 (572h53m45.815811443s) [noise=true , address=45.91.116.12:4226 , piece_id=7MSH7VL4GFYLYNLNLYQ2X2KZHG33CYXW4PUYR6EX5A6FO7NZPYGA, seg_pos={0 0}, upload=502242]: Done uploading piece. err: <nil> Dec 29 12:22:45.944753446 (572h53m45.815836491s) [seg_pos={0 0}, upload=502242]: Segment reached optimal threshold of 13 pieces. Dec 29 12:22:45.944930703 (572h53m45.816013748s) [noise=true , address=45.91.116.5:4297 , piece_id=F54IX7KAJOJQ5UTM5REBB7EQFATMDPEHEZC6GQKQJDMPAOZDWPVA, seg_pos={0 0}, upload=502242]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:39826->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:39826->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39826->45.91.116.5:4297: use of closed network connection Dec 29 12:22:45.944936564 (572h53m45.816019599s) [noise=true , address=45.91.116.5:4297 , piece_id=F54IX7KAJOJQ5UTM5REBB7EQFATMDPEHEZC6GQKQJDMPAOZDWPVA, seg_pos={0 0}, upload=502242]: Long tail cancellation due to optimal threshold Dec 29 12:22:45.945048759 (572h53m45.816131804s) [noise=true , address=45.91.116.6:4976 , piece_id=VYLHBBOOTYLMFLZHKW4QVQRVXEDNQ7GD5APY32PVJ4YWNBYUOMMQ, seg_pos={0 0}, upload=502242]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 29 12:22:45.945058196 (572h53m45.816141231s) [noise=true , address=45.91.116.6:4976 , piece_id=VYLHBBOOTYLMFLZHKW4QVQRVXEDNQ7GD5APY32PVJ4YWNBYUOMMQ, seg_pos={0 0}, upload=502242]: Long tail cancellation due to optimal threshold Dec 29 12:22:45.945238850 (572h53m45.816321885s) [noise=true , address=45.91.116.13:4156 , piece_id=2G3DG2JEEJ7OWFKR22DFR3RTL2FO5LLRIGRPHXBH5AXH57UQMJYA, seg_pos={0 0}, upload=502242]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 29 12:22:45.945248989 (572h53m45.816332024s) [noise=true , address=45.91.116.13:4156 , piece_id=2G3DG2JEEJ7OWFKR22DFR3RTL2FO5LLRIGRPHXBH5AXH57UQMJYA, seg_pos={0 0}, upload=502242]: Long tail cancellation due to optimal threshold Dec 29 12:22:45.945270711 (572h53m45.816353746s) [seg_pos={0 0}, upload=502242]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:22:45.945349410 (572h53m45.816432455s) [upload=502242]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:22:45.945356714 (572h53m45.816439769s) [upload=502242]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:22:46.238508068 (572h53m46.109591103s) [upload=502242]: Done uploading segments... Dec 29 12:22:46.238527545 (572h53m46.10961058s) [upload=502242]: Upload finished. err: <nil> Duration: 6.103114943s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [XXX.XXXX...XXX...XXX..XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXX] duration: 5.000764606s trace: https://jaeger.dev.storj.io/trace/4bd4cfc6084a552a FAIL (2025-12-29 12:24:38 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000786718s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X.......X......] duration: 43.149757594s trace: https://jaeger.dev.storj.io/trace/54cc96062d54e547 FAIL (2025-12-29 12:24:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 7.148417227 (572h49m17.019500273s) [noise=true , address=45.91.116.13:4162 , piece_id=UPESSP5ITLGEQCHYNL4VTSXLBB24O7QXZ5UB6FA4PC5JWHGUVCAQ, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.148925174 (572h49m17.020008209s) [noise=true , address=145.79.227.43:4169 , piece_id=O7II3XWLMOJGXSOVGJ6ITQWOW2OIL7PPYWXNQTVWNYBNU3E4FFKA, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.149222871 (572h49m17.020305906s) [noise=true , address=145.79.227.34:4924 , piece_id=LDEYPDBM6HDZKO2YHAAUGXHFXFPCQ74IH5JIRFYBPUMIS5HFXRBQ, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.149466225 (572h49m17.020549801s) [noise=true , address=145.79.227.44:4197 , piece_id=4D4YHRYNXGE5YCZH7AL3IIN6Q6FI6TLGXECM3SAZQDCKOGN4F2OQ, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.149621069 (572h49m17.020704114s) [noise=true , address=64.49.13.169:4710 , piece_id=WVVMYAG6FYQDJIWNKAAH4GREERSAJN2SOQDRMD3WFK7NMLONL4MQ, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.149918967 (572h49m17.021002012s) [noise=true , address=145.79.227.4:4195 , piece_id=6UKJB3VYO7UL4TW5IVTDA4KTXTJ2YFDN2ADALMTYQVMTSLKYGVDA, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.149948984 (572h49m17.021032029s) [noise=true , address=77.90.31.18:4839 , piece_id=26G6IXW2YT7AUPAM4R4SQQFXXT57HSS2IGWMIXJNN5NEAHSEZCIA, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.150229729 (572h49m17.021312764s) [noise=true , address=77.90.31.13:4650 , piece_id=2G6KWTAPYS6FXIZK3SL5RD4OZ6DAOSSHUL5ZZ75PB4ETPNF7MH7A, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.150278251 (572h49m17.021361296s) [noise=true , address=145.79.227.5:4231 , piece_id=D6EJVCDCFO2GP4PE32ISCUSWCQGMY6NX4EYWVHG2T3DF3LM7DW3Q, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.150629960 (572h49m17.021712995s) [noise=true , address=145.79.227.40:4052 , piece_id=JFC3UQOZSXCYUGIKJKRI73RLJ2QGIQTYN37PSECCEDMJPK2UYSDA, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.150991189 (572h49m17.022074224s) [noise=true , address=77.90.31.12:4617 , piece_id=2GNINSXEB5KKEDT2WCIK3UWDICDUS4MBMIQQ7VKTQ5JGNY6O55JQ, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.151215005 (572h49m17.02229804s) [noise=true , address=193.149.164.143:4702 , piece_id=6UADW5KZGEO4EHPJP2TLDDZZBGR2A2J2LSLNN6WR4HV5MVO5UK6A, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.151269889 (572h49m17.022352925s) [noise=true , address=145.79.227.30:4780 , piece_id=MNJQ3WRD3J6TBKAEVMRHV467KM2OGUVT2YVDBXIN32HZJ55X73IQ, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.151578597 (572h49m17.022661633s) [noise=true , address=145.79.227.35:4501 , piece_id=S335GIDTNGOW2JXIX5OZ7HHBQI3K254ZR2TPCU27ZN2K6ZS3CVOA, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.151863319 (572h49m17.022946354s) [noise=true , address=77.90.31.10:4531 , piece_id=T344ZC45IB4573ARJASYIANXG24GH26VI6VBPUR3CPYK77K7Z7TQ, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.151922863 (572h49m17.023005908s) [noise=true , address=145.79.227.29:4974 , piece_id=Q4PXHIZSLPKYURXKZ6B6HJ7YANRN5QHTZWUWZ3MVI3G3O7JFKIJA, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:17.152129156 (572h49m17.023212201s) [noise=true , address=145.79.227.13:4807 , piece_id=Y67QBVSHT5WUO4DG7LW2DM67ZILNFVZFUFSQSBF4GGVYJ24SMGOA, seg_pos={0 0}, upload=502178]: Uploading piece... Dec 29 12:18:18.050161080 (572h49m17.921244125s) [noise=true , address=193.149.164.143:4702 , piece_id=6UADW5KZGEO4EHPJP2TLDDZZBGR2A2J2LSLNN6WR4HV5MVO5UK6A, seg_pos={0 0}, upload=502178]: Done uploading piece. err: ecclient: upload failed (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1, address:193.149.164.143:4702): protocol: storage node overloaded, request limit: 2000 Dec 29 12:18:18.609488842 (572h49m18.480571877s) [upload=502178]: Next returned nil segment Dec 29 12:18:18.609505194 (572h49m18.480588259s) [upload=502178]: Waiting for error group managing segments... Dec 29 12:18:18.996027579 (572h49m18.867110624s) [noise=true , address=145.79.227.5:4231 , piece_id=D6EJVCDCFO2GP4PE32ISCUSWCQGMY6NX4EYWVHG2T3DF3LM7DW3Q, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.015970613 (572h49m18.887053658s) [noise=true , address=145.79.227.4:4195 , piece_id=6UKJB3VYO7UL4TW5IVTDA4KTXTJ2YFDN2ADALMTYQVMTSLKYGVDA, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.024599794 (572h49m18.895682829s) [noise=true , address=145.79.227.29:4974 , piece_id=Q4PXHIZSLPKYURXKZ6B6HJ7YANRN5QHTZWUWZ3MVI3G3O7JFKIJA, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.025693498 (572h49m18.896776533s) [noise=true , address=145.79.227.13:4807 , piece_id=Y67QBVSHT5WUO4DG7LW2DM67ZILNFVZFUFSQSBF4GGVYJ24SMGOA, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.220567296 (572h49m19.091650341s) [noise=true , address=77.90.31.12:4617 , piece_id=2GNINSXEB5KKEDT2WCIK3UWDICDUS4MBMIQQ7VKTQ5JGNY6O55JQ, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.308488310 (572h49m19.179571345s) [noise=true , address=77.90.31.13:4650 , piece_id=2G6KWTAPYS6FXIZK3SL5RD4OZ6DAOSSHUL5ZZ75PB4ETPNF7MH7A, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.413574289 (572h49m19.284657324s) [noise=true , address=77.90.31.18:4839 , piece_id=26G6IXW2YT7AUPAM4R4SQQFXXT57HSS2IGWMIXJNN5NEAHSEZCIA, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.871377594 (572h49m19.742460639s) [noise=true , address=145.79.227.30:4780 , piece_id=MNJQ3WRD3J6TBKAEVMRHV467KM2OGUVT2YVDBXIN32HZJ55X73IQ, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.939637331 (572h49m19.810720366s) [noise=true , address=145.79.227.35:4501 , piece_id=S335GIDTNGOW2JXIX5OZ7HHBQI3K254ZR2TPCU27ZN2K6ZS3CVOA, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:19.958323732 (572h49m19.829406767s) [noise=true , address=145.79.227.44:4197 , piece_id=4D4YHRYNXGE5YCZH7AL3IIN6Q6FI6TLGXECM3SAZQDCKOGN4F2OQ, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:20.001969170 (572h49m19.873052215s) [noise=true , address=145.79.227.40:4052 , piece_id=JFC3UQOZSXCYUGIKJKRI73RLJ2QGIQTYN37PSECCEDMJPK2UYSDA, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:20.015729351 (572h49m19.886812386s) [noise=true , address=145.79.227.34:4924 , piece_id=LDEYPDBM6HDZKO2YHAAUGXHFXFPCQ74IH5JIRFYBPUMIS5HFXRBQ, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:20.114412729 (572h49m19.985495764s) [noise=true , address=145.79.227.43:4169 , piece_id=O7II3XWLMOJGXSOVGJ6ITQWOW2OIL7PPYWXNQTVWNYBNU3E4FFKA, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:20.177148507 (572h49m20.048231542s) [noise=true , address=77.90.31.10:4531 , piece_id=T344ZC45IB4573ARJASYIANXG24GH26VI6VBPUR3CPYK77K7Z7TQ, seg_pos={0 0}, upload=502178]: Done uploading piece. err: <nil> Dec 29 12:18:20.177170508 (572h49m20.048253544s) [seg_pos={0 0}, upload=502178]: Segment reached optimal threshold of 14 pieces. Dec 29 12:18:20.177339190 (572h49m20.048422225s) [noise=true , address=64.49.13.169:4710 , piece_id=WVVMYAG6FYQDJIWNKAAH4GREERSAJN2SOQDRMD3WFK7NMLONL4MQ, seg_pos={0 0}, upload=502178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57032->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57032->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57032->64.49.13.169:4710: use of closed network connection Dec 29 12:18:20.177420124 (572h49m20.048503179s) [noise=true , address=64.49.13.169:4710 , piece_id=WVVMYAG6FYQDJIWNKAAH4GREERSAJN2SOQDRMD3WFK7NMLONL4MQ, seg_pos={0 0}, upload=502178]: Long tail cancellation due to optimal threshold Dec 29 12:18:20.177586831 (572h49m20.048669866s) [noise=true , address=145.79.227.49:4343 , piece_id=JPXHILNUTSHS4RUP5CRSUFWEVASM55PBY4MQCHK47AURG5XZDS7Q, seg_pos={0 0}, upload=502178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf): protocol: expected piece hash; context canceled; EOF Dec 29 12:18:20.177593945 (572h49m20.04867698s) [noise=true , address=145.79.227.49:4343 , piece_id=JPXHILNUTSHS4RUP5CRSUFWEVASM55PBY4MQCHK47AURG5XZDS7Q, seg_pos={0 0}, upload=502178]: Long tail cancellation due to optimal threshold Dec 29 12:18:20.177724033 (572h49m20.048807068s) [noise=true , address=45.91.116.13:4162 , piece_id=UPESSP5ITLGEQCHYNL4VTSXLBB24O7QXZ5UB6FA4PC5JWHGUVCAQ, seg_pos={0 0}, upload=502178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4): protocol: expected piece hash; context canceled; EOF Dec 29 12:18:20.177729393 (572h49m20.048812428s) [noise=true , address=45.91.116.13:4162 , piece_id=UPESSP5ITLGEQCHYNL4VTSXLBB24O7QXZ5UB6FA4PC5JWHGUVCAQ, seg_pos={0 0}, upload=502178]: Long tail cancellation due to optimal threshold Dec 29 12:18:20.177744652 (572h49m20.048827687s) [seg_pos={0 0}, upload=502178]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:18:20.177850734 (572h49m20.048933769s) [upload=502178]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:18:20.177856495 (572h49m20.04893953s) [upload=502178]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:18:20.468750169 (572h49m20.339833204s) [upload=502178]: Done uploading segments... Dec 29 12:18:20.468821605 (572h49m20.33990465s) [upload=502178]: Upload finished. err: <nil> Duration: 43.149773676s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [...XX.XXX.X.X....XXXXXX..X.X.......X.XX...X.......X.X..XXX..] duration: 6.026654668s trace: https://jaeger.dev.storj.io/trace/31561a0261895c45 FAIL (2025-12-29 12:24:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 29 12:22:43.293612504 (572h53m43.164695559s) [upload=502242]: Starting upload Dec 29 12:22:43.293674391 (572h53m43.164757436s) [upload=502242]: Done starting upload Dec 29 12:22:43.293739686 (572h53m43.164822741s) [upload=502242]: Uploading segments... Dec 29 12:22:43.294290385 (572h53m43.16537343s) [upload=502242]: Got next segment. Inline: false Dec 29 12:22:43.294340521 (572h53m43.165423556s) [upload=502242]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:22:43.294347023 (572h53m43.165430058s) [upload=502242]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:22:43.539446638 (572h53m43.410529673s) [seg_pos={0 0}, upload=502242]: Begin upload segment... Dec 29 12:22:43.539525900 (572h53m43.410608935s) [seg_pos={0 0}, upload=502242]: Done begin upload segment. Dec 29 12:22:43.539551748 (572h53m43.410634793s) [noise=true , address=145.79.227.12:4767 , piece_id=5DUFEZOFHYRSIOVAJRH4VTTW52TX5NLIT4RCK6RXT26WDFUBVDJQ, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.539653281 (572h53m43.410736316s) [noise=true , address=45.91.116.5:4297 , piece_id=F54IX7KAJOJQ5UTM5REBB7EQFATMDPEHEZC6GQKQJDMPAOZDWPVA, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.540172510 (572h53m43.411255545s) [noise=true , address=45.91.116.10:4082 , piece_id=SHVHPT3LK4XZL3SET6F2BVQDNWYCIA4ULQNEMLTKZINE5POW3QWQ, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.540535903 (572h53m43.411618938s) [noise=true , address=45.91.116.13:4156 , piece_id=2G3DG2JEEJ7OWFKR22DFR3RTL2FO5LLRIGRPHXBH5AXH57UQMJYA, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.540739299 (572h53m43.411822344s) [noise=true , address=45.91.116.6:4976 , piece_id=VYLHBBOOTYLMFLZHKW4QVQRVXEDNQ7GD5APY32PVJ4YWNBYUOMMQ, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.541067405 (572h53m43.41215045s) [noise=true , address=145.79.227.51:4411 , piece_id=HKNFUIKRZEMHR4XW4YGPVEIBGP3NL6EKCF72UHF4TDTK3BESHDKA, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.541090047 (572h53m43.412173082s) [noise=true , address=145.79.227.6:4257 , piece_id=AR5F4DN4GPBPTSSYPDIRJYHQOOUCYCK5FTUFMXGSFIUOZAMCCRSA, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.541534954 (572h53m43.412617989s) [noise=true , address=145.79.227.5:4223 , piece_id=4X44MXO4F7O7HJBN6G6C4SFO2N224C7JCYCVVZ22I6PBVZYH6LNA, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.541605409 (572h53m43.412688444s) [noise=true , address=77.90.31.17:4773 , piece_id=D3VSGPEUFI2D6FWL23QOPKYEAA7ZS5EUXOJIB7XX7FSHKFKMZB3Q, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.541897444 (572h53m43.412980489s) [noise=true , address=145.79.227.2:4113 , piece_id=4NEWBO5BBPRGXVL6N4D3C43UWZBXTUYUJDGGWXMFFT2UM5776ZJQ, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.542018926 (572h53m43.413101971s) [noise=true , address=77.90.31.11:4556 , piece_id=XY5XPKVVX6BWLNYS2HDY5B2LM2HOCRHCY2LKJGAORACI4O46JVUQ, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.542239386 (572h53m43.413322421s) [noise=true , address=145.79.227.15:4873 , piece_id=NFX7XMHZFZH5NHIQGFIK3SKVHPYIMEBCI2BVTXFHF55OGMZVYRNA, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.542352682 (572h53m43.413435717s) [noise=true , address=145.79.227.32:4834 , piece_id=U4FL74Q3OUT5TU3EUOBIOBXTCWAOEUEMWAJYPVVODDHEUGMB3M2Q, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.542769426 (572h53m43.413852461s) [noise=true , address=45.91.116.14:4190 , piece_id=E7FJ4IE56DOHGFGMCIT7GKCARYTLIKBR34UZ6KWJSOEM7NC7TVBQ, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.543162534 (572h53m43.414245569s) [noise=true , address=145.79.227.24:4582 , piece_id=3664UNGQ3IXMQQZWVXN7CGXCC7A3CLLEJCLURNNZTF64UOLWJEEQ, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:43.544555176 (572h53m43.415638211s) [noise=true , address=45.91.116.12:4226 , piece_id=7MSH7VL4GFYLYNLNLYQ2X2KZHG33CYXW4PUYR6EX5A6FO7NZPYGA, seg_pos={0 0}, upload=502242]: Uploading piece... Dec 29 12:22:45.240733568 (572h53m45.111816604s) [upload=502242]: Next returned nil segment Dec 29 12:22:45.240745761 (572h53m45.111828796s) [upload=502242]: Waiting for error group managing segments... Dec 29 12:22:45.513367814 (572h53m45.384450849s) [noise=true , address=145.79.227.12:4767 , piece_id=5DUFEZOFHYRSIOVAJRH4VTTW52TX5NLIT4RCK6RXT26WDFUBVDJQ, seg_pos={0 0}, upload=502242]: Done uploading piece. err: <nil> Dec 29 12:22:45.522976540 (572h53m45.394059585s) [noise=true , address=145.79.227.32:4834 , piece_id=U4FL74Q3OUT5TU3EUOBIOBXTCWAOEUEMWAJYPVVODDHEUGMB3M2Q, seg_pos={0 0}, upload=502242]: Done uploading piece. err: <nil> Dec 29 12:22:45.526549694 (572h53m45.397632729s) [noise=true , address=145.79.227.15:4873 , piece_id=NFX7XMHZFZH5NHIQGFIK3SKVHPYIMEBCI2BVTXFHF55OGMZVYRNA, seg_pos={0 0}, upload=502242]: Done uploading piece. err: <nil> Dec 29 12:22:45.527483763 (572h53m45.398566798s) [noise=true , address=145.79.227.6:4257 , piece_id=AR5F4DN4GPBPTSSYPDIRJYHQOOUCYCK5FTUFMXGSFIUOZAMCCRSA, seg_pos={0 0}, upload=502242]: Done uploading piece. err: <nil> Dec 29 12:22:45.528179628 (572h53m45.399262663s) [noise=true , address=145.79.227.51:4411 , piece_id=HKNFUIKRZEMHR4XW4YGPVEIBGP3NL6EKCF72UHF4TDTK3BESHDKA, seg_pos={0 0}, upload=502242]: Done uploading piece. err: <nil> Dec 29 12:22:45.532282561 (572h53m45.403365596s) [noise=true , address=145.79.227.24:4582 , piece_id=3664UNGQ3IXMQQZWVXN7CGXCC7A3CLLEJCLURNNZTF64UOLWJEEQ, seg_pos={0 0}, upload=502242]: Done uploading piece. err: <nil> Dec 29 12:22:45.547407200 (572h53m45.418490235s) [noise=true , address=145.79.227.5:4223 , piece_id=4X44MXO4F7O7HJBN6G6C4SFO2N224C7JCYCVVZ22I6PBVZYH6LNA, seg_pos={0 0}, upload=502242]: Done uploading piece. err: <nil> Dec 29 12:22:45.551402257 (572h53m45.422485302s) [noise=true , address=145.79.227.2:4113 , piece_id=4NEWBO5BBPRGXVL6N4D3C43UWZBXTUYUJDGGWXMFFT2UM5776ZJQ, seg_pos={0 0}, upload=502242]: Done uploading piece. err: <nil> Dec 29 12:22:45.823994002 (572h53m45.695077037s) [noise=true , address=45.91.116.10:4082 , piece_id=SHVHPT3LK4XZL3SET6F2BVQDNWYCIA4ULQNEMLTKZINE5POW3QWQ, seg_pos={0 0}, upload=502242]: Done uploading piece. err: <nil> Dec 29 12:22:45.824741105 (572h53m45.69582415s) [noise=true , address=77.90.31.17:4773 , piece_id=D3VSGPEUFI2D6FWL23QOPKYEAA7ZS5EUXOJIB7XX7FSHKFKMZB3Q, seg_pos={0 0}, upload=502242]: Done uploading piece. err: <nil> Dec 29 12:22:45.838851143 (572h53m45.709934178s) [noise=true , address=77.90.31.11:4556 , piece_id=XY5XPKVVX6BWLNYS2HDY5B2LM2HOCRHCY2LKJGAORACI4O46JVUQ, seg_pos={0 0}, upload=502242]: Done uploading piece. err: <nil> Dec 29 12:22:45.929331229 (572h53m45.800414264s) [noise=true , address=45.91.116.14:4190 , piece_id=E7FJ4IE56DOHGFGMCIT7GKCARYTLIKBR34UZ6KWJSOEM7NC7TVBQ, seg_pos={0 0}, upload=502242]: Done uploading piece. err: <nil> Dec 29 12:22:45.944728408 (572h53m45.815811443s) [noise=true , address=45.91.116.12:4226 , piece_id=7MSH7VL4GFYLYNLNLYQ2X2KZHG33CYXW4PUYR6EX5A6FO7NZPYGA, seg_pos={0 0}, upload=502242]: Done uploading piece. err: <nil> Dec 29 12:22:45.944753446 (572h53m45.815836491s) [seg_pos={0 0}, upload=502242]: Segment reached optimal threshold of 13 pieces. Dec 29 12:22:45.944930703 (572h53m45.816013748s) [noise=true , address=45.91.116.5:4297 , piece_id=F54IX7KAJOJQ5UTM5REBB7EQFATMDPEHEZC6GQKQJDMPAOZDWPVA, seg_pos={0 0}, upload=502242]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:39826->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:39826->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39826->45.91.116.5:4297: use of closed network connection Dec 29 12:22:45.944936564 (572h53m45.816019599s) [noise=true , address=45.91.116.5:4297 , piece_id=F54IX7KAJOJQ5UTM5REBB7EQFATMDPEHEZC6GQKQJDMPAOZDWPVA, seg_pos={0 0}, upload=502242]: Long tail cancellation due to optimal threshold Dec 29 12:22:45.945048759 (572h53m45.816131804s) [noise=true , address=45.91.116.6:4976 , piece_id=VYLHBBOOTYLMFLZHKW4QVQRVXEDNQ7GD5APY32PVJ4YWNBYUOMMQ, seg_pos={0 0}, upload=502242]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 29 12:22:45.945058196 (572h53m45.816141231s) [noise=true , address=45.91.116.6:4976 , piece_id=VYLHBBOOTYLMFLZHKW4QVQRVXEDNQ7GD5APY32PVJ4YWNBYUOMMQ, seg_pos={0 0}, upload=502242]: Long tail cancellation due to optimal threshold Dec 29 12:22:45.945238850 (572h53m45.816321885s) [noise=true , address=45.91.116.13:4156 , piece_id=2G3DG2JEEJ7OWFKR22DFR3RTL2FO5LLRIGRPHXBH5AXH57UQMJYA, seg_pos={0 0}, upload=502242]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 29 12:22:45.945248989 (572h53m45.816332024s) [noise=true , address=45.91.116.13:4156 , piece_id=2G3DG2JEEJ7OWFKR22DFR3RTL2FO5LLRIGRPHXBH5AXH57UQMJYA, seg_pos={0 0}, upload=502242]: Long tail cancellation due to optimal threshold Dec 29 12:22:45.945270711 (572h53m45.816353746s) [seg_pos={0 0}, upload=502242]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:22:45.945349410 (572h53m45.816432455s) [upload=502242]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:22:45.945356714 (572h53m45.816439769s) [upload=502242]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:22:46.238508068 (572h53m46.109591103s) [upload=502242]: Done uploading segments... Dec 29 12:22:46.238527545 (572h53m46.10961058s) [upload=502242]: Upload finished. err: <nil> Duration: 6.026688431s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)