Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [X.....X.....XXX.XX...XX.XX..X...XXX..XX.X...X.X....XXXXX...X] duration: 5.000959154s trace: https://jaeger.dev.storj.io/trace/7043e8c353c2d7b FAIL (2025-12-28 09:52:34 UTC) orbiter warning: context deadline exceeded Duration: 5.000975605s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X..............X....X...X......XX...X..X.........X.........] duration: 5.972920925s trace: https://jaeger.dev.storj.io/trace/3519e7bb67af2319 FAIL (2025-12-28 09:51:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =479018]: Starting upload Dec 28 09:45:28.272345405 (546h16m28.14342845s) [upload=479018]: Done starting upload Dec 28 09:45:28.272542140 (546h16m28.143625175s) [upload=479018]: Uploading segments... Dec 28 09:45:28.272601182 (546h16m28.143684227s) [upload=479018]: Got next segment. Inline: false Dec 28 09:45:28.272633553 (546h16m28.143716598s) [upload=479018]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:45:28.272638573 (546h16m28.143721618s) [upload=479018]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:45:28.517108586 (546h16m28.388191621s) [seg_pos={0 0}, upload=479018]: Begin upload segment... Dec 28 09:45:28.517151057 (546h16m28.388234092s) [seg_pos={0 0}, upload=479018]: Done begin upload segment. Dec 28 09:45:28.517196494 (546h16m28.388279529s) [noise=true , address=145.79.227.5:4223 , piece_id=R44JRSLYRBLUYS4SXMWQVYA5FKKO42JFFYCBUFR53FE3OLMAH2HQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.517205591 (546h16m28.388288636s) [noise=true , address=145.79.227.37:4567 , piece_id=OH3FDZCBZPH26ST3BTBUVK6PDT7TBUYGVINGJ535CV762EY7UUTQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.517645489 (546h16m28.388728524s) [noise=true , address=145.79.227.26:4655 , piece_id=XKEPME47XXIQQTBYHFJLM3ACRFBD3NL62HUSMFIJQDP2BAT3DJDA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.518064396 (546h16m28.389147442s) [noise=true , address=145.79.227.51:4411 , piece_id=LDSCU6FJJVECH2ZALLLKHT44QVEUT3VVQWCO4QXV4JKSHA4RJNFA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.518486981 (546h16m28.389570016s) [noise=true , address=45.91.116.7:4012 , piece_id=H7ES6CP7VA3KG5YAK3JOBUHOY5EMQW2FKEJPPIHEFTRSWQRIOBNA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.519070302 (546h16m28.390153337s) [noise=true , address=45.91.116.9:4261 , piece_id=6FDSNDNMZAFTISKN7FEJXBXJE2GERGVS3FQA4CHME47USEJ5DQUQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.519522914 (546h16m28.390605959s) [noise=true , address=45.91.116.12:4228 , piece_id=3LUQHNGM6QTUQDUOW37UXWUGE5WDPSTPQXHY4KP6TJ7DSNUE33VA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.520088731 (546h16m28.391171766s) [noise=true , address=145.79.227.11:4729 , piece_id=5MGX65D76ZGRV4YTDBFYR5LR436B45AO332N5S4XOVCYQQSRUFNQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.520469777 (546h16m28.391552812s) [noise=true , address=45.91.116.8:4047 , piece_id=JBFUI5HGPHGUYIZPM5SMHGHVN6FRJEACIWMYIJFG5DEG72N4IXDA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.520791429 (546h16m28.391874464s) [noise=true , address=145.79.227.4:4185 , piece_id=NKASFGR5SW4L43BFYJBX6LG667UUKD7QTWRZ3ZDD2ZWR6WXUVTFA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.521099125 (546h16m28.39218216s) [noise=true , address=45.91.116.4:4939 , piece_id=MBMUBN6V44OXK2VSOGDNIE2NPMTHNDBN2DWODT5DAMW2R2PMTY3A, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.521452418 (546h16m28.392535453s) [noise=true , address=145.79.227.29:4942 , piece_id=IDK63X74PG7UOAOWZIDMBQESETUZCMIVOAIQKCV2PJYIUND7R4LA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.523396439 (546h16m28.394479485s) [noise=true , address=145.79.227.9:4367 , piece_id=PMZIP647X4MH4W7SQGCANR63ZEUPGMMUPF2W5R2N6ZZTQ3SULZ3A, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.523464770 (546h16m28.394547815s) [noise=true , address=145.79.227.15:4876 , piece_id=7DJ6S25DS2QKYSMOJZVA5NRJATEYQGKICF7PRAZI7L7FBPK3OTEQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.523963379 (546h16m28.395046424s) [noise=true , address=145.79.227.45:4231 , piece_id=YUDXJSY5GZYFCYEDV2UN35WEUSRVSZOA7PST533Q4WYJO477OQGQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.525670970 (546h16m28.396754005s) [noise=true , address=45.91.116.14:4192 , piece_id=2TDWHC5ALNSZIZ5UY5LVHMTE736M3ANWPZOKXR3AWVONWCK6PG6A, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:30.237256159 (546h16m30.108339194s) [upload=479018]: Next returned nil segment Dec 28 09:45:30.237270817 (546h16m30.108353852s) [upload=479018]: Waiting for error group managing segments... Dec 28 09:45:30.525556952 (546h16m30.396639997s) [noise=true , address=145.79.227.15:4876 , piece_id=7DJ6S25DS2QKYSMOJZVA5NRJATEYQGKICF7PRAZI7L7FBPK3OTEQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.526594779 (546h16m30.397677824s) [noise=true , address=145.79.227.51:4411 , piece_id=LDSCU6FJJVECH2ZALLLKHT44QVEUT3VVQWCO4QXV4JKSHA4RJNFA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.526730968 (546h16m30.397814013s) [noise=true , address=145.79.227.9:4367 , piece_id=PMZIP647X4MH4W7SQGCANR63ZEUPGMMUPF2W5R2N6ZZTQ3SULZ3A, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.531182765 (546h16m30.4022658s) [noise=true , address=145.79.227.45:4231 , piece_id=YUDXJSY5GZYFCYEDV2UN35WEUSRVSZOA7PST533Q4WYJO477OQGQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.538899700 (546h16m30.409982745s) [noise=true , address=145.79.227.5:4223 , piece_id=R44JRSLYRBLUYS4SXMWQVYA5FKKO42JFFYCBUFR53FE3OLMAH2HQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.540774660 (546h16m30.411857715s) [noise=true , address=145.79.227.4:4185 , piece_id=NKASFGR5SW4L43BFYJBX6LG667UUKD7QTWRZ3ZDD2ZWR6WXUVTFA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.541541320 (546h16m30.412624355s) [noise=true , address=145.79.227.11:4729 , piece_id=5MGX65D76ZGRV4YTDBFYR5LR436B45AO332N5S4XOVCYQQSRUFNQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.545531758 (546h16m30.416614793s) [noise=true , address=145.79.227.37:4567 , piece_id=OH3FDZCBZPH26ST3BTBUVK6PDT7TBUYGVINGJ535CV762EY7UUTQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.547314974 (546h16m30.418398019s) [noise=true , address=145.79.227.26:4655 , piece_id=XKEPME47XXIQQTBYHFJLM3ACRFBD3NL62HUSMFIJQDP2BAT3DJDA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.560249253 (546h16m30.431332288s) [noise=true , address=145.79.227.29:4942 , piece_id=IDK63X74PG7UOAOWZIDMBQESETUZCMIVOAIQKCV2PJYIUND7R4LA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.741263574 (546h16m30.612346609s) [noise=true , address=45.91.116.7:4012 , piece_id=H7ES6CP7VA3KG5YAK3JOBUHOY5EMQW2FKEJPPIHEFTRSWQRIOBNA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.871268365 (546h16m30.74235141s) [noise=true , address=45.91.116.12:4228 , piece_id=3LUQHNGM6QTUQDUOW37UXWUGE5WDPSTPQXHY4KP6TJ7DSNUE33VA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:31.129984185 (546h16m31.00106724s) [noise=true , address=45.91.116.14:4192 , piece_id=2TDWHC5ALNSZIZ5UY5LVHMTE736M3ANWPZOKXR3AWVONWCK6PG6A, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:31.130007730 (546h16m31.001090775s) [seg_pos={0 0}, upload=479018]: Segment reached optimal threshold of 13 pieces. Dec 28 09:45:31.130221627 (546h16m31.001304662s) [noise=true , address=45.91.116.4:4939 , piece_id=MBMUBN6V44OXK2VSOGDNIE2NPMTHNDBN2DWODT5DAMW2R2PMTY3A, seg_pos={0 0}, upload=479018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 09:45:31.130340102 (546h16m31.001423177s) [noise=true , address=45.91.116.4:4939 , piece_id=MBMUBN6V44OXK2VSOGDNIE2NPMTHNDBN2DWODT5DAMW2R2PMTY3A, seg_pos={0 0}, upload=479018]: Long tail cancellation due to optimal threshold Dec 28 09:45:31.130379537 (546h16m31.001462572s) [noise=true , address=45.91.116.9:4261 , piece_id=6FDSNDNMZAFTISKN7FEJXBXJE2GERGVS3FQA4CHME47USEJ5DQUQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:49258->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:49258->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49258->45.91.116.9:4261: use of closed network connection Dec 28 09:45:31.130389626 (546h16m31.001472682s) [noise=true , address=45.91.116.9:4261 , piece_id=6FDSNDNMZAFTISKN7FEJXBXJE2GERGVS3FQA4CHME47USEJ5DQUQ, seg_pos={0 0}, upload=479018]: Long tail cancellation due to optimal threshold Dec 28 09:45:31.130664179 (546h16m31.001747224s) [noise=true , address=45.91.116.8:4047 , piece_id=JBFUI5HGPHGUYIZPM5SMHGHVN6FRJEACIWMYIJFG5DEG72N4IXDA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: read tcp 188.166.225.170:47210->45.91.116.8:4047: use of closed network connection; context canceled Dec 28 09:45:31.130671965 (546h16m31.001755s) [noise=true , address=45.91.116.8:4047 , piece_id=JBFUI5HGPHGUYIZPM5SMHGHVN6FRJEACIWMYIJFG5DEG72N4IXDA, seg_pos={0 0}, upload=479018]: Long tail cancellation due to optimal threshold Dec 28 09:45:31.130689287 (546h16m31.001772322s) [seg_pos={0 0}, upload=479018]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:45:31.130754111 (546h16m31.001837146s) [upload=479018]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:45:31.130758349 (546h16m31.001841384s) [upload=479018]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:45:31.421778310 (546h16m31.292861345s) [upload=479018]: Done uploading segments... Dec 28 09:45:31.421849585 (546h16m31.29293262s) [upload=479018]: Upload finished. err: <nil> Duration: 5.97294997s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [.X.....X.....XXX.XX...XX.XX..X...XXX..XX.X...X.X....XXXXX...] duration: 4.303911545s trace: https://jaeger.dev.storj.io/trace/55fc41bad766f1a2 FAIL (2025-12-28 09:51:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.30394086s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..X..............X....X...X......XX...X..X.........X........] duration: 5.661173637s trace: https://jaeger.dev.storj.io/trace/3519e7bb67af2319 FAIL (2025-12-28 09:51:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =479018]: Starting upload Dec 28 09:45:28.272345405 (546h16m28.14342845s) [upload=479018]: Done starting upload Dec 28 09:45:28.272542140 (546h16m28.143625175s) [upload=479018]: Uploading segments... Dec 28 09:45:28.272601182 (546h16m28.143684227s) [upload=479018]: Got next segment. Inline: false Dec 28 09:45:28.272633553 (546h16m28.143716598s) [upload=479018]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:45:28.272638573 (546h16m28.143721618s) [upload=479018]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:45:28.517108586 (546h16m28.388191621s) [seg_pos={0 0}, upload=479018]: Begin upload segment... Dec 28 09:45:28.517151057 (546h16m28.388234092s) [seg_pos={0 0}, upload=479018]: Done begin upload segment. Dec 28 09:45:28.517196494 (546h16m28.388279529s) [noise=true , address=145.79.227.5:4223 , piece_id=R44JRSLYRBLUYS4SXMWQVYA5FKKO42JFFYCBUFR53FE3OLMAH2HQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.517205591 (546h16m28.388288636s) [noise=true , address=145.79.227.37:4567 , piece_id=OH3FDZCBZPH26ST3BTBUVK6PDT7TBUYGVINGJ535CV762EY7UUTQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.517645489 (546h16m28.388728524s) [noise=true , address=145.79.227.26:4655 , piece_id=XKEPME47XXIQQTBYHFJLM3ACRFBD3NL62HUSMFIJQDP2BAT3DJDA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.518064396 (546h16m28.389147442s) [noise=true , address=145.79.227.51:4411 , piece_id=LDSCU6FJJVECH2ZALLLKHT44QVEUT3VVQWCO4QXV4JKSHA4RJNFA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.518486981 (546h16m28.389570016s) [noise=true , address=45.91.116.7:4012 , piece_id=H7ES6CP7VA3KG5YAK3JOBUHOY5EMQW2FKEJPPIHEFTRSWQRIOBNA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.519070302 (546h16m28.390153337s) [noise=true , address=45.91.116.9:4261 , piece_id=6FDSNDNMZAFTISKN7FEJXBXJE2GERGVS3FQA4CHME47USEJ5DQUQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.519522914 (546h16m28.390605959s) [noise=true , address=45.91.116.12:4228 , piece_id=3LUQHNGM6QTUQDUOW37UXWUGE5WDPSTPQXHY4KP6TJ7DSNUE33VA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.520088731 (546h16m28.391171766s) [noise=true , address=145.79.227.11:4729 , piece_id=5MGX65D76ZGRV4YTDBFYR5LR436B45AO332N5S4XOVCYQQSRUFNQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.520469777 (546h16m28.391552812s) [noise=true , address=45.91.116.8:4047 , piece_id=JBFUI5HGPHGUYIZPM5SMHGHVN6FRJEACIWMYIJFG5DEG72N4IXDA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.520791429 (546h16m28.391874464s) [noise=true , address=145.79.227.4:4185 , piece_id=NKASFGR5SW4L43BFYJBX6LG667UUKD7QTWRZ3ZDD2ZWR6WXUVTFA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.521099125 (546h16m28.39218216s) [noise=true , address=45.91.116.4:4939 , piece_id=MBMUBN6V44OXK2VSOGDNIE2NPMTHNDBN2DWODT5DAMW2R2PMTY3A, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.521452418 (546h16m28.392535453s) [noise=true , address=145.79.227.29:4942 , piece_id=IDK63X74PG7UOAOWZIDMBQESETUZCMIVOAIQKCV2PJYIUND7R4LA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.523396439 (546h16m28.394479485s) [noise=true , address=145.79.227.9:4367 , piece_id=PMZIP647X4MH4W7SQGCANR63ZEUPGMMUPF2W5R2N6ZZTQ3SULZ3A, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.523464770 (546h16m28.394547815s) [noise=true , address=145.79.227.15:4876 , piece_id=7DJ6S25DS2QKYSMOJZVA5NRJATEYQGKICF7PRAZI7L7FBPK3OTEQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.523963379 (546h16m28.395046424s) [noise=true , address=145.79.227.45:4231 , piece_id=YUDXJSY5GZYFCYEDV2UN35WEUSRVSZOA7PST533Q4WYJO477OQGQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.525670970 (546h16m28.396754005s) [noise=true , address=45.91.116.14:4192 , piece_id=2TDWHC5ALNSZIZ5UY5LVHMTE736M3ANWPZOKXR3AWVONWCK6PG6A, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:30.237256159 (546h16m30.108339194s) [upload=479018]: Next returned nil segment Dec 28 09:45:30.237270817 (546h16m30.108353852s) [upload=479018]: Waiting for error group managing segments... Dec 28 09:45:30.525556952 (546h16m30.396639997s) [noise=true , address=145.79.227.15:4876 , piece_id=7DJ6S25DS2QKYSMOJZVA5NRJATEYQGKICF7PRAZI7L7FBPK3OTEQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.526594779 (546h16m30.397677824s) [noise=true , address=145.79.227.51:4411 , piece_id=LDSCU6FJJVECH2ZALLLKHT44QVEUT3VVQWCO4QXV4JKSHA4RJNFA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.526730968 (546h16m30.397814013s) [noise=true , address=145.79.227.9:4367 , piece_id=PMZIP647X4MH4W7SQGCANR63ZEUPGMMUPF2W5R2N6ZZTQ3SULZ3A, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.531182765 (546h16m30.4022658s) [noise=true , address=145.79.227.45:4231 , piece_id=YUDXJSY5GZYFCYEDV2UN35WEUSRVSZOA7PST533Q4WYJO477OQGQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.538899700 (546h16m30.409982745s) [noise=true , address=145.79.227.5:4223 , piece_id=R44JRSLYRBLUYS4SXMWQVYA5FKKO42JFFYCBUFR53FE3OLMAH2HQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.540774660 (546h16m30.411857715s) [noise=true , address=145.79.227.4:4185 , piece_id=NKASFGR5SW4L43BFYJBX6LG667UUKD7QTWRZ3ZDD2ZWR6WXUVTFA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.541541320 (546h16m30.412624355s) [noise=true , address=145.79.227.11:4729 , piece_id=5MGX65D76ZGRV4YTDBFYR5LR436B45AO332N5S4XOVCYQQSRUFNQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.545531758 (546h16m30.416614793s) [noise=true , address=145.79.227.37:4567 , piece_id=OH3FDZCBZPH26ST3BTBUVK6PDT7TBUYGVINGJ535CV762EY7UUTQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.547314974 (546h16m30.418398019s) [noise=true , address=145.79.227.26:4655 , piece_id=XKEPME47XXIQQTBYHFJLM3ACRFBD3NL62HUSMFIJQDP2BAT3DJDA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.560249253 (546h16m30.431332288s) [noise=true , address=145.79.227.29:4942 , piece_id=IDK63X74PG7UOAOWZIDMBQESETUZCMIVOAIQKCV2PJYIUND7R4LA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.741263574 (546h16m30.612346609s) [noise=true , address=45.91.116.7:4012 , piece_id=H7ES6CP7VA3KG5YAK3JOBUHOY5EMQW2FKEJPPIHEFTRSWQRIOBNA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.871268365 (546h16m30.74235141s) [noise=true , address=45.91.116.12:4228 , piece_id=3LUQHNGM6QTUQDUOW37UXWUGE5WDPSTPQXHY4KP6TJ7DSNUE33VA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:31.129984185 (546h16m31.00106724s) [noise=true , address=45.91.116.14:4192 , piece_id=2TDWHC5ALNSZIZ5UY5LVHMTE736M3ANWPZOKXR3AWVONWCK6PG6A, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:31.130007730 (546h16m31.001090775s) [seg_pos={0 0}, upload=479018]: Segment reached optimal threshold of 13 pieces. Dec 28 09:45:31.130221627 (546h16m31.001304662s) [noise=true , address=45.91.116.4:4939 , piece_id=MBMUBN6V44OXK2VSOGDNIE2NPMTHNDBN2DWODT5DAMW2R2PMTY3A, seg_pos={0 0}, upload=479018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 09:45:31.130340102 (546h16m31.001423177s) [noise=true , address=45.91.116.4:4939 , piece_id=MBMUBN6V44OXK2VSOGDNIE2NPMTHNDBN2DWODT5DAMW2R2PMTY3A, seg_pos={0 0}, upload=479018]: Long tail cancellation due to optimal threshold Dec 28 09:45:31.130379537 (546h16m31.001462572s) [noise=true , address=45.91.116.9:4261 , piece_id=6FDSNDNMZAFTISKN7FEJXBXJE2GERGVS3FQA4CHME47USEJ5DQUQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:49258->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:49258->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49258->45.91.116.9:4261: use of closed network connection Dec 28 09:45:31.130389626 (546h16m31.001472682s) [noise=true , address=45.91.116.9:4261 , piece_id=6FDSNDNMZAFTISKN7FEJXBXJE2GERGVS3FQA4CHME47USEJ5DQUQ, seg_pos={0 0}, upload=479018]: Long tail cancellation due to optimal threshold Dec 28 09:45:31.130664179 (546h16m31.001747224s) [noise=true , address=45.91.116.8:4047 , piece_id=JBFUI5HGPHGUYIZPM5SMHGHVN6FRJEACIWMYIJFG5DEG72N4IXDA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: read tcp 188.166.225.170:47210->45.91.116.8:4047: use of closed network connection; context canceled Dec 28 09:45:31.130671965 (546h16m31.001755s) [noise=true , address=45.91.116.8:4047 , piece_id=JBFUI5HGPHGUYIZPM5SMHGHVN6FRJEACIWMYIJFG5DEG72N4IXDA, seg_pos={0 0}, upload=479018]: Long tail cancellation due to optimal threshold Dec 28 09:45:31.130689287 (546h16m31.001772322s) [seg_pos={0 0}, upload=479018]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:45:31.130754111 (546h16m31.001837146s) [upload=479018]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:45:31.130758349 (546h16m31.001841384s) [upload=479018]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:45:31.421778310 (546h16m31.292861345s) [upload=479018]: Done uploading segments... Dec 28 09:45:31.421849585 (546h16m31.29293262s) [upload=479018]: Upload finished. err: <nil> Duration: 5.661209617s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [X.X.....X.....XXX.XX...XX.XX..X...XXX..XX.X...X.X....XXXXX..] duration: 4.661379657s trace: https://jaeger.dev.storj.io/trace/55fc41bad766f1a2 FAIL (2025-12-28 09:50:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.661410806s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [.X.X.....X.....XXX.XX...XX.XX..X...XXX..XX.X...X.X....XXXXX.] duration: 4.223301373s trace: https://jaeger.dev.storj.io/trace/55fc41bad766f1a2 FAIL (2025-12-28 09:50:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.223324638s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...X..............X....X...X......XX...X..X.........X.......] duration: 6.507631444s trace: https://jaeger.dev.storj.io/trace/3519e7bb67af2319 FAIL (2025-12-28 09:50:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =479018]: Starting upload Dec 28 09:45:28.272345405 (546h16m28.14342845s) [upload=479018]: Done starting upload Dec 28 09:45:28.272542140 (546h16m28.143625175s) [upload=479018]: Uploading segments... Dec 28 09:45:28.272601182 (546h16m28.143684227s) [upload=479018]: Got next segment. Inline: false Dec 28 09:45:28.272633553 (546h16m28.143716598s) [upload=479018]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:45:28.272638573 (546h16m28.143721618s) [upload=479018]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:45:28.517108586 (546h16m28.388191621s) [seg_pos={0 0}, upload=479018]: Begin upload segment... Dec 28 09:45:28.517151057 (546h16m28.388234092s) [seg_pos={0 0}, upload=479018]: Done begin upload segment. Dec 28 09:45:28.517196494 (546h16m28.388279529s) [noise=true , address=145.79.227.5:4223 , piece_id=R44JRSLYRBLUYS4SXMWQVYA5FKKO42JFFYCBUFR53FE3OLMAH2HQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.517205591 (546h16m28.388288636s) [noise=true , address=145.79.227.37:4567 , piece_id=OH3FDZCBZPH26ST3BTBUVK6PDT7TBUYGVINGJ535CV762EY7UUTQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.517645489 (546h16m28.388728524s) [noise=true , address=145.79.227.26:4655 , piece_id=XKEPME47XXIQQTBYHFJLM3ACRFBD3NL62HUSMFIJQDP2BAT3DJDA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.518064396 (546h16m28.389147442s) [noise=true , address=145.79.227.51:4411 , piece_id=LDSCU6FJJVECH2ZALLLKHT44QVEUT3VVQWCO4QXV4JKSHA4RJNFA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.518486981 (546h16m28.389570016s) [noise=true , address=45.91.116.7:4012 , piece_id=H7ES6CP7VA3KG5YAK3JOBUHOY5EMQW2FKEJPPIHEFTRSWQRIOBNA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.519070302 (546h16m28.390153337s) [noise=true , address=45.91.116.9:4261 , piece_id=6FDSNDNMZAFTISKN7FEJXBXJE2GERGVS3FQA4CHME47USEJ5DQUQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.519522914 (546h16m28.390605959s) [noise=true , address=45.91.116.12:4228 , piece_id=3LUQHNGM6QTUQDUOW37UXWUGE5WDPSTPQXHY4KP6TJ7DSNUE33VA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.520088731 (546h16m28.391171766s) [noise=true , address=145.79.227.11:4729 , piece_id=5MGX65D76ZGRV4YTDBFYR5LR436B45AO332N5S4XOVCYQQSRUFNQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.520469777 (546h16m28.391552812s) [noise=true , address=45.91.116.8:4047 , piece_id=JBFUI5HGPHGUYIZPM5SMHGHVN6FRJEACIWMYIJFG5DEG72N4IXDA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.520791429 (546h16m28.391874464s) [noise=true , address=145.79.227.4:4185 , piece_id=NKASFGR5SW4L43BFYJBX6LG667UUKD7QTWRZ3ZDD2ZWR6WXUVTFA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.521099125 (546h16m28.39218216s) [noise=true , address=45.91.116.4:4939 , piece_id=MBMUBN6V44OXK2VSOGDNIE2NPMTHNDBN2DWODT5DAMW2R2PMTY3A, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.521452418 (546h16m28.392535453s) [noise=true , address=145.79.227.29:4942 , piece_id=IDK63X74PG7UOAOWZIDMBQESETUZCMIVOAIQKCV2PJYIUND7R4LA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.523396439 (546h16m28.394479485s) [noise=true , address=145.79.227.9:4367 , piece_id=PMZIP647X4MH4W7SQGCANR63ZEUPGMMUPF2W5R2N6ZZTQ3SULZ3A, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.523464770 (546h16m28.394547815s) [noise=true , address=145.79.227.15:4876 , piece_id=7DJ6S25DS2QKYSMOJZVA5NRJATEYQGKICF7PRAZI7L7FBPK3OTEQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.523963379 (546h16m28.395046424s) [noise=true , address=145.79.227.45:4231 , piece_id=YUDXJSY5GZYFCYEDV2UN35WEUSRVSZOA7PST533Q4WYJO477OQGQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.525670970 (546h16m28.396754005s) [noise=true , address=45.91.116.14:4192 , piece_id=2TDWHC5ALNSZIZ5UY5LVHMTE736M3ANWPZOKXR3AWVONWCK6PG6A, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:30.237256159 (546h16m30.108339194s) [upload=479018]: Next returned nil segment Dec 28 09:45:30.237270817 (546h16m30.108353852s) [upload=479018]: Waiting for error group managing segments... Dec 28 09:45:30.525556952 (546h16m30.396639997s) [noise=true , address=145.79.227.15:4876 , piece_id=7DJ6S25DS2QKYSMOJZVA5NRJATEYQGKICF7PRAZI7L7FBPK3OTEQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.526594779 (546h16m30.397677824s) [noise=true , address=145.79.227.51:4411 , piece_id=LDSCU6FJJVECH2ZALLLKHT44QVEUT3VVQWCO4QXV4JKSHA4RJNFA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.526730968 (546h16m30.397814013s) [noise=true , address=145.79.227.9:4367 , piece_id=PMZIP647X4MH4W7SQGCANR63ZEUPGMMUPF2W5R2N6ZZTQ3SULZ3A, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.531182765 (546h16m30.4022658s) [noise=true , address=145.79.227.45:4231 , piece_id=YUDXJSY5GZYFCYEDV2UN35WEUSRVSZOA7PST533Q4WYJO477OQGQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.538899700 (546h16m30.409982745s) [noise=true , address=145.79.227.5:4223 , piece_id=R44JRSLYRBLUYS4SXMWQVYA5FKKO42JFFYCBUFR53FE3OLMAH2HQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.540774660 (546h16m30.411857715s) [noise=true , address=145.79.227.4:4185 , piece_id=NKASFGR5SW4L43BFYJBX6LG667UUKD7QTWRZ3ZDD2ZWR6WXUVTFA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.541541320 (546h16m30.412624355s) [noise=true , address=145.79.227.11:4729 , piece_id=5MGX65D76ZGRV4YTDBFYR5LR436B45AO332N5S4XOVCYQQSRUFNQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.545531758 (546h16m30.416614793s) [noise=true , address=145.79.227.37:4567 , piece_id=OH3FDZCBZPH26ST3BTBUVK6PDT7TBUYGVINGJ535CV762EY7UUTQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.547314974 (546h16m30.418398019s) [noise=true , address=145.79.227.26:4655 , piece_id=XKEPME47XXIQQTBYHFJLM3ACRFBD3NL62HUSMFIJQDP2BAT3DJDA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.560249253 (546h16m30.431332288s) [noise=true , address=145.79.227.29:4942 , piece_id=IDK63X74PG7UOAOWZIDMBQESETUZCMIVOAIQKCV2PJYIUND7R4LA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.741263574 (546h16m30.612346609s) [noise=true , address=45.91.116.7:4012 , piece_id=H7ES6CP7VA3KG5YAK3JOBUHOY5EMQW2FKEJPPIHEFTRSWQRIOBNA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.871268365 (546h16m30.74235141s) [noise=true , address=45.91.116.12:4228 , piece_id=3LUQHNGM6QTUQDUOW37UXWUGE5WDPSTPQXHY4KP6TJ7DSNUE33VA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:31.129984185 (546h16m31.00106724s) [noise=true , address=45.91.116.14:4192 , piece_id=2TDWHC5ALNSZIZ5UY5LVHMTE736M3ANWPZOKXR3AWVONWCK6PG6A, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:31.130007730 (546h16m31.001090775s) [seg_pos={0 0}, upload=479018]: Segment reached optimal threshold of 13 pieces. Dec 28 09:45:31.130221627 (546h16m31.001304662s) [noise=true , address=45.91.116.4:4939 , piece_id=MBMUBN6V44OXK2VSOGDNIE2NPMTHNDBN2DWODT5DAMW2R2PMTY3A, seg_pos={0 0}, upload=479018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 09:45:31.130340102 (546h16m31.001423177s) [noise=true , address=45.91.116.4:4939 , piece_id=MBMUBN6V44OXK2VSOGDNIE2NPMTHNDBN2DWODT5DAMW2R2PMTY3A, seg_pos={0 0}, upload=479018]: Long tail cancellation due to optimal threshold Dec 28 09:45:31.130379537 (546h16m31.001462572s) [noise=true , address=45.91.116.9:4261 , piece_id=6FDSNDNMZAFTISKN7FEJXBXJE2GERGVS3FQA4CHME47USEJ5DQUQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:49258->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:49258->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49258->45.91.116.9:4261: use of closed network connection Dec 28 09:45:31.130389626 (546h16m31.001472682s) [noise=true , address=45.91.116.9:4261 , piece_id=6FDSNDNMZAFTISKN7FEJXBXJE2GERGVS3FQA4CHME47USEJ5DQUQ, seg_pos={0 0}, upload=479018]: Long tail cancellation due to optimal threshold Dec 28 09:45:31.130664179 (546h16m31.001747224s) [noise=true , address=45.91.116.8:4047 , piece_id=JBFUI5HGPHGUYIZPM5SMHGHVN6FRJEACIWMYIJFG5DEG72N4IXDA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: read tcp 188.166.225.170:47210->45.91.116.8:4047: use of closed network connection; context canceled Dec 28 09:45:31.130671965 (546h16m31.001755s) [noise=true , address=45.91.116.8:4047 , piece_id=JBFUI5HGPHGUYIZPM5SMHGHVN6FRJEACIWMYIJFG5DEG72N4IXDA, seg_pos={0 0}, upload=479018]: Long tail cancellation due to optimal threshold Dec 28 09:45:31.130689287 (546h16m31.001772322s) [seg_pos={0 0}, upload=479018]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:45:31.130754111 (546h16m31.001837146s) [upload=479018]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:45:31.130758349 (546h16m31.001841384s) [upload=479018]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:45:31.421778310 (546h16m31.292861345s) [upload=479018]: Done uploading segments... Dec 28 09:45:31.421849585 (546h16m31.29293262s) [upload=479018]: Upload finished. err: <nil> Duration: 6.507668575s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [..X.X.....X.....XXX.XX...XX.XX..X...XXX..XX.X...X.X....XXXXX] duration: 5.002352357s trace: https://jaeger.dev.storj.io/trace/55fc41bad766f1a2 FAIL (2025-12-28 09:49:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00239589s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [....X..............X....X...X......XX...X..X.........X......] duration: 5.829447817s trace: https://jaeger.dev.storj.io/trace/3519e7bb67af2319 FAIL (2025-12-28 09:49:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =479018]: Starting upload Dec 28 09:45:28.272345405 (546h16m28.14342845s) [upload=479018]: Done starting upload Dec 28 09:45:28.272542140 (546h16m28.143625175s) [upload=479018]: Uploading segments... Dec 28 09:45:28.272601182 (546h16m28.143684227s) [upload=479018]: Got next segment. Inline: false Dec 28 09:45:28.272633553 (546h16m28.143716598s) [upload=479018]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:45:28.272638573 (546h16m28.143721618s) [upload=479018]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:45:28.517108586 (546h16m28.388191621s) [seg_pos={0 0}, upload=479018]: Begin upload segment... Dec 28 09:45:28.517151057 (546h16m28.388234092s) [seg_pos={0 0}, upload=479018]: Done begin upload segment. Dec 28 09:45:28.517196494 (546h16m28.388279529s) [noise=true , address=145.79.227.5:4223 , piece_id=R44JRSLYRBLUYS4SXMWQVYA5FKKO42JFFYCBUFR53FE3OLMAH2HQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.517205591 (546h16m28.388288636s) [noise=true , address=145.79.227.37:4567 , piece_id=OH3FDZCBZPH26ST3BTBUVK6PDT7TBUYGVINGJ535CV762EY7UUTQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.517645489 (546h16m28.388728524s) [noise=true , address=145.79.227.26:4655 , piece_id=XKEPME47XXIQQTBYHFJLM3ACRFBD3NL62HUSMFIJQDP2BAT3DJDA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.518064396 (546h16m28.389147442s) [noise=true , address=145.79.227.51:4411 , piece_id=LDSCU6FJJVECH2ZALLLKHT44QVEUT3VVQWCO4QXV4JKSHA4RJNFA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.518486981 (546h16m28.389570016s) [noise=true , address=45.91.116.7:4012 , piece_id=H7ES6CP7VA3KG5YAK3JOBUHOY5EMQW2FKEJPPIHEFTRSWQRIOBNA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.519070302 (546h16m28.390153337s) [noise=true , address=45.91.116.9:4261 , piece_id=6FDSNDNMZAFTISKN7FEJXBXJE2GERGVS3FQA4CHME47USEJ5DQUQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.519522914 (546h16m28.390605959s) [noise=true , address=45.91.116.12:4228 , piece_id=3LUQHNGM6QTUQDUOW37UXWUGE5WDPSTPQXHY4KP6TJ7DSNUE33VA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.520088731 (546h16m28.391171766s) [noise=true , address=145.79.227.11:4729 , piece_id=5MGX65D76ZGRV4YTDBFYR5LR436B45AO332N5S4XOVCYQQSRUFNQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.520469777 (546h16m28.391552812s) [noise=true , address=45.91.116.8:4047 , piece_id=JBFUI5HGPHGUYIZPM5SMHGHVN6FRJEACIWMYIJFG5DEG72N4IXDA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.520791429 (546h16m28.391874464s) [noise=true , address=145.79.227.4:4185 , piece_id=NKASFGR5SW4L43BFYJBX6LG667UUKD7QTWRZ3ZDD2ZWR6WXUVTFA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.521099125 (546h16m28.39218216s) [noise=true , address=45.91.116.4:4939 , piece_id=MBMUBN6V44OXK2VSOGDNIE2NPMTHNDBN2DWODT5DAMW2R2PMTY3A, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.521452418 (546h16m28.392535453s) [noise=true , address=145.79.227.29:4942 , piece_id=IDK63X74PG7UOAOWZIDMBQESETUZCMIVOAIQKCV2PJYIUND7R4LA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.523396439 (546h16m28.394479485s) [noise=true , address=145.79.227.9:4367 , piece_id=PMZIP647X4MH4W7SQGCANR63ZEUPGMMUPF2W5R2N6ZZTQ3SULZ3A, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.523464770 (546h16m28.394547815s) [noise=true , address=145.79.227.15:4876 , piece_id=7DJ6S25DS2QKYSMOJZVA5NRJATEYQGKICF7PRAZI7L7FBPK3OTEQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.523963379 (546h16m28.395046424s) [noise=true , address=145.79.227.45:4231 , piece_id=YUDXJSY5GZYFCYEDV2UN35WEUSRVSZOA7PST533Q4WYJO477OQGQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.525670970 (546h16m28.396754005s) [noise=true , address=45.91.116.14:4192 , piece_id=2TDWHC5ALNSZIZ5UY5LVHMTE736M3ANWPZOKXR3AWVONWCK6PG6A, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:30.237256159 (546h16m30.108339194s) [upload=479018]: Next returned nil segment Dec 28 09:45:30.237270817 (546h16m30.108353852s) [upload=479018]: Waiting for error group managing segments... Dec 28 09:45:30.525556952 (546h16m30.396639997s) [noise=true , address=145.79.227.15:4876 , piece_id=7DJ6S25DS2QKYSMOJZVA5NRJATEYQGKICF7PRAZI7L7FBPK3OTEQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.526594779 (546h16m30.397677824s) [noise=true , address=145.79.227.51:4411 , piece_id=LDSCU6FJJVECH2ZALLLKHT44QVEUT3VVQWCO4QXV4JKSHA4RJNFA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.526730968 (546h16m30.397814013s) [noise=true , address=145.79.227.9:4367 , piece_id=PMZIP647X4MH4W7SQGCANR63ZEUPGMMUPF2W5R2N6ZZTQ3SULZ3A, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.531182765 (546h16m30.4022658s) [noise=true , address=145.79.227.45:4231 , piece_id=YUDXJSY5GZYFCYEDV2UN35WEUSRVSZOA7PST533Q4WYJO477OQGQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.538899700 (546h16m30.409982745s) [noise=true , address=145.79.227.5:4223 , piece_id=R44JRSLYRBLUYS4SXMWQVYA5FKKO42JFFYCBUFR53FE3OLMAH2HQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.540774660 (546h16m30.411857715s) [noise=true , address=145.79.227.4:4185 , piece_id=NKASFGR5SW4L43BFYJBX6LG667UUKD7QTWRZ3ZDD2ZWR6WXUVTFA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.541541320 (546h16m30.412624355s) [noise=true , address=145.79.227.11:4729 , piece_id=5MGX65D76ZGRV4YTDBFYR5LR436B45AO332N5S4XOVCYQQSRUFNQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.545531758 (546h16m30.416614793s) [noise=true , address=145.79.227.37:4567 , piece_id=OH3FDZCBZPH26ST3BTBUVK6PDT7TBUYGVINGJ535CV762EY7UUTQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.547314974 (546h16m30.418398019s) [noise=true , address=145.79.227.26:4655 , piece_id=XKEPME47XXIQQTBYHFJLM3ACRFBD3NL62HUSMFIJQDP2BAT3DJDA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.560249253 (546h16m30.431332288s) [noise=true , address=145.79.227.29:4942 , piece_id=IDK63X74PG7UOAOWZIDMBQESETUZCMIVOAIQKCV2PJYIUND7R4LA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.741263574 (546h16m30.612346609s) [noise=true , address=45.91.116.7:4012 , piece_id=H7ES6CP7VA3KG5YAK3JOBUHOY5EMQW2FKEJPPIHEFTRSWQRIOBNA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.871268365 (546h16m30.74235141s) [noise=true , address=45.91.116.12:4228 , piece_id=3LUQHNGM6QTUQDUOW37UXWUGE5WDPSTPQXHY4KP6TJ7DSNUE33VA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:31.129984185 (546h16m31.00106724s) [noise=true , address=45.91.116.14:4192 , piece_id=2TDWHC5ALNSZIZ5UY5LVHMTE736M3ANWPZOKXR3AWVONWCK6PG6A, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:31.130007730 (546h16m31.001090775s) [seg_pos={0 0}, upload=479018]: Segment reached optimal threshold of 13 pieces. Dec 28 09:45:31.130221627 (546h16m31.001304662s) [noise=true , address=45.91.116.4:4939 , piece_id=MBMUBN6V44OXK2VSOGDNIE2NPMTHNDBN2DWODT5DAMW2R2PMTY3A, seg_pos={0 0}, upload=479018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 09:45:31.130340102 (546h16m31.001423177s) [noise=true , address=45.91.116.4:4939 , piece_id=MBMUBN6V44OXK2VSOGDNIE2NPMTHNDBN2DWODT5DAMW2R2PMTY3A, seg_pos={0 0}, upload=479018]: Long tail cancellation due to optimal threshold Dec 28 09:45:31.130379537 (546h16m31.001462572s) [noise=true , address=45.91.116.9:4261 , piece_id=6FDSNDNMZAFTISKN7FEJXBXJE2GERGVS3FQA4CHME47USEJ5DQUQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:49258->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:49258->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49258->45.91.116.9:4261: use of closed network connection Dec 28 09:45:31.130389626 (546h16m31.001472682s) [noise=true , address=45.91.116.9:4261 , piece_id=6FDSNDNMZAFTISKN7FEJXBXJE2GERGVS3FQA4CHME47USEJ5DQUQ, seg_pos={0 0}, upload=479018]: Long tail cancellation due to optimal threshold Dec 28 09:45:31.130664179 (546h16m31.001747224s) [noise=true , address=45.91.116.8:4047 , piece_id=JBFUI5HGPHGUYIZPM5SMHGHVN6FRJEACIWMYIJFG5DEG72N4IXDA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: read tcp 188.166.225.170:47210->45.91.116.8:4047: use of closed network connection; context canceled Dec 28 09:45:31.130671965 (546h16m31.001755s) [noise=true , address=45.91.116.8:4047 , piece_id=JBFUI5HGPHGUYIZPM5SMHGHVN6FRJEACIWMYIJFG5DEG72N4IXDA, seg_pos={0 0}, upload=479018]: Long tail cancellation due to optimal threshold Dec 28 09:45:31.130689287 (546h16m31.001772322s) [seg_pos={0 0}, upload=479018]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:45:31.130754111 (546h16m31.001837146s) [upload=479018]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:45:31.130758349 (546h16m31.001841384s) [upload=479018]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:45:31.421778310 (546h16m31.292861345s) [upload=479018]: Done uploading segments... Dec 28 09:45:31.421849585 (546h16m31.29293262s) [upload=479018]: Upload finished. err: <nil> Duration: 5.829477402s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [...X.X.....X.....XXX.XX...XX.XX..X...XXX..XX.X...X.X....XXXX] duration: 5.001036802s trace: https://jaeger.dev.storj.io/trace/44658e9d9197a143 FAIL (2025-12-28 09:48:36 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001058404s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.....X..............X....X...X......XX...X..X.........X.....] duration: 5.819207108s trace: https://jaeger.dev.storj.io/trace/3519e7bb67af2319 FAIL (2025-12-28 09:48:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =479018]: Starting upload Dec 28 09:45:28.272345405 (546h16m28.14342845s) [upload=479018]: Done starting upload Dec 28 09:45:28.272542140 (546h16m28.143625175s) [upload=479018]: Uploading segments... Dec 28 09:45:28.272601182 (546h16m28.143684227s) [upload=479018]: Got next segment. Inline: false Dec 28 09:45:28.272633553 (546h16m28.143716598s) [upload=479018]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:45:28.272638573 (546h16m28.143721618s) [upload=479018]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:45:28.517108586 (546h16m28.388191621s) [seg_pos={0 0}, upload=479018]: Begin upload segment... Dec 28 09:45:28.517151057 (546h16m28.388234092s) [seg_pos={0 0}, upload=479018]: Done begin upload segment. Dec 28 09:45:28.517196494 (546h16m28.388279529s) [noise=true , address=145.79.227.5:4223 , piece_id=R44JRSLYRBLUYS4SXMWQVYA5FKKO42JFFYCBUFR53FE3OLMAH2HQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.517205591 (546h16m28.388288636s) [noise=true , address=145.79.227.37:4567 , piece_id=OH3FDZCBZPH26ST3BTBUVK6PDT7TBUYGVINGJ535CV762EY7UUTQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.517645489 (546h16m28.388728524s) [noise=true , address=145.79.227.26:4655 , piece_id=XKEPME47XXIQQTBYHFJLM3ACRFBD3NL62HUSMFIJQDP2BAT3DJDA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.518064396 (546h16m28.389147442s) [noise=true , address=145.79.227.51:4411 , piece_id=LDSCU6FJJVECH2ZALLLKHT44QVEUT3VVQWCO4QXV4JKSHA4RJNFA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.518486981 (546h16m28.389570016s) [noise=true , address=45.91.116.7:4012 , piece_id=H7ES6CP7VA3KG5YAK3JOBUHOY5EMQW2FKEJPPIHEFTRSWQRIOBNA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.519070302 (546h16m28.390153337s) [noise=true , address=45.91.116.9:4261 , piece_id=6FDSNDNMZAFTISKN7FEJXBXJE2GERGVS3FQA4CHME47USEJ5DQUQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.519522914 (546h16m28.390605959s) [noise=true , address=45.91.116.12:4228 , piece_id=3LUQHNGM6QTUQDUOW37UXWUGE5WDPSTPQXHY4KP6TJ7DSNUE33VA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.520088731 (546h16m28.391171766s) [noise=true , address=145.79.227.11:4729 , piece_id=5MGX65D76ZGRV4YTDBFYR5LR436B45AO332N5S4XOVCYQQSRUFNQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.520469777 (546h16m28.391552812s) [noise=true , address=45.91.116.8:4047 , piece_id=JBFUI5HGPHGUYIZPM5SMHGHVN6FRJEACIWMYIJFG5DEG72N4IXDA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.520791429 (546h16m28.391874464s) [noise=true , address=145.79.227.4:4185 , piece_id=NKASFGR5SW4L43BFYJBX6LG667UUKD7QTWRZ3ZDD2ZWR6WXUVTFA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.521099125 (546h16m28.39218216s) [noise=true , address=45.91.116.4:4939 , piece_id=MBMUBN6V44OXK2VSOGDNIE2NPMTHNDBN2DWODT5DAMW2R2PMTY3A, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.521452418 (546h16m28.392535453s) [noise=true , address=145.79.227.29:4942 , piece_id=IDK63X74PG7UOAOWZIDMBQESETUZCMIVOAIQKCV2PJYIUND7R4LA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.523396439 (546h16m28.394479485s) [noise=true , address=145.79.227.9:4367 , piece_id=PMZIP647X4MH4W7SQGCANR63ZEUPGMMUPF2W5R2N6ZZTQ3SULZ3A, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.523464770 (546h16m28.394547815s) [noise=true , address=145.79.227.15:4876 , piece_id=7DJ6S25DS2QKYSMOJZVA5NRJATEYQGKICF7PRAZI7L7FBPK3OTEQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.523963379 (546h16m28.395046424s) [noise=true , address=145.79.227.45:4231 , piece_id=YUDXJSY5GZYFCYEDV2UN35WEUSRVSZOA7PST533Q4WYJO477OQGQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.525670970 (546h16m28.396754005s) [noise=true , address=45.91.116.14:4192 , piece_id=2TDWHC5ALNSZIZ5UY5LVHMTE736M3ANWPZOKXR3AWVONWCK6PG6A, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:30.237256159 (546h16m30.108339194s) [upload=479018]: Next returned nil segment Dec 28 09:45:30.237270817 (546h16m30.108353852s) [upload=479018]: Waiting for error group managing segments... Dec 28 09:45:30.525556952 (546h16m30.396639997s) [noise=true , address=145.79.227.15:4876 , piece_id=7DJ6S25DS2QKYSMOJZVA5NRJATEYQGKICF7PRAZI7L7FBPK3OTEQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.526594779 (546h16m30.397677824s) [noise=true , address=145.79.227.51:4411 , piece_id=LDSCU6FJJVECH2ZALLLKHT44QVEUT3VVQWCO4QXV4JKSHA4RJNFA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.526730968 (546h16m30.397814013s) [noise=true , address=145.79.227.9:4367 , piece_id=PMZIP647X4MH4W7SQGCANR63ZEUPGMMUPF2W5R2N6ZZTQ3SULZ3A, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.531182765 (546h16m30.4022658s) [noise=true , address=145.79.227.45:4231 , piece_id=YUDXJSY5GZYFCYEDV2UN35WEUSRVSZOA7PST533Q4WYJO477OQGQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.538899700 (546h16m30.409982745s) [noise=true , address=145.79.227.5:4223 , piece_id=R44JRSLYRBLUYS4SXMWQVYA5FKKO42JFFYCBUFR53FE3OLMAH2HQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.540774660 (546h16m30.411857715s) [noise=true , address=145.79.227.4:4185 , piece_id=NKASFGR5SW4L43BFYJBX6LG667UUKD7QTWRZ3ZDD2ZWR6WXUVTFA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.541541320 (546h16m30.412624355s) [noise=true , address=145.79.227.11:4729 , piece_id=5MGX65D76ZGRV4YTDBFYR5LR436B45AO332N5S4XOVCYQQSRUFNQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.545531758 (546h16m30.416614793s) [noise=true , address=145.79.227.37:4567 , piece_id=OH3FDZCBZPH26ST3BTBUVK6PDT7TBUYGVINGJ535CV762EY7UUTQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.547314974 (546h16m30.418398019s) [noise=true , address=145.79.227.26:4655 , piece_id=XKEPME47XXIQQTBYHFJLM3ACRFBD3NL62HUSMFIJQDP2BAT3DJDA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.560249253 (546h16m30.431332288s) [noise=true , address=145.79.227.29:4942 , piece_id=IDK63X74PG7UOAOWZIDMBQESETUZCMIVOAIQKCV2PJYIUND7R4LA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.741263574 (546h16m30.612346609s) [noise=true , address=45.91.116.7:4012 , piece_id=H7ES6CP7VA3KG5YAK3JOBUHOY5EMQW2FKEJPPIHEFTRSWQRIOBNA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.871268365 (546h16m30.74235141s) [noise=true , address=45.91.116.12:4228 , piece_id=3LUQHNGM6QTUQDUOW37UXWUGE5WDPSTPQXHY4KP6TJ7DSNUE33VA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:31.129984185 (546h16m31.00106724s) [noise=true , address=45.91.116.14:4192 , piece_id=2TDWHC5ALNSZIZ5UY5LVHMTE736M3ANWPZOKXR3AWVONWCK6PG6A, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:31.130007730 (546h16m31.001090775s) [seg_pos={0 0}, upload=479018]: Segment reached optimal threshold of 13 pieces. Dec 28 09:45:31.130221627 (546h16m31.001304662s) [noise=true , address=45.91.116.4:4939 , piece_id=MBMUBN6V44OXK2VSOGDNIE2NPMTHNDBN2DWODT5DAMW2R2PMTY3A, seg_pos={0 0}, upload=479018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 09:45:31.130340102 (546h16m31.001423177s) [noise=true , address=45.91.116.4:4939 , piece_id=MBMUBN6V44OXK2VSOGDNIE2NPMTHNDBN2DWODT5DAMW2R2PMTY3A, seg_pos={0 0}, upload=479018]: Long tail cancellation due to optimal threshold Dec 28 09:45:31.130379537 (546h16m31.001462572s) [noise=true , address=45.91.116.9:4261 , piece_id=6FDSNDNMZAFTISKN7FEJXBXJE2GERGVS3FQA4CHME47USEJ5DQUQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:49258->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:49258->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49258->45.91.116.9:4261: use of closed network connection Dec 28 09:45:31.130389626 (546h16m31.001472682s) [noise=true , address=45.91.116.9:4261 , piece_id=6FDSNDNMZAFTISKN7FEJXBXJE2GERGVS3FQA4CHME47USEJ5DQUQ, seg_pos={0 0}, upload=479018]: Long tail cancellation due to optimal threshold Dec 28 09:45:31.130664179 (546h16m31.001747224s) [noise=true , address=45.91.116.8:4047 , piece_id=JBFUI5HGPHGUYIZPM5SMHGHVN6FRJEACIWMYIJFG5DEG72N4IXDA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: read tcp 188.166.225.170:47210->45.91.116.8:4047: use of closed network connection; context canceled Dec 28 09:45:31.130671965 (546h16m31.001755s) [noise=true , address=45.91.116.8:4047 , piece_id=JBFUI5HGPHGUYIZPM5SMHGHVN6FRJEACIWMYIJFG5DEG72N4IXDA, seg_pos={0 0}, upload=479018]: Long tail cancellation due to optimal threshold Dec 28 09:45:31.130689287 (546h16m31.001772322s) [seg_pos={0 0}, upload=479018]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:45:31.130754111 (546h16m31.001837146s) [upload=479018]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:45:31.130758349 (546h16m31.001841384s) [upload=479018]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:45:31.421778310 (546h16m31.292861345s) [upload=479018]: Done uploading segments... Dec 28 09:45:31.421849585 (546h16m31.29293262s) [upload=479018]: Upload finished. err: <nil> Duration: 5.819235573s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [....X.X.....X.....XXX.XX...XX.XX..X...XXX..XX.X...X.X....XXX] duration: 5.000621502s trace: https://jaeger.dev.storj.io/trace/45aa82ef756b211a FAIL (2025-12-28 09:47:56 UTC) orbiter warning: context deadline exceeded Duration: 5.000642721s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [......X..............X....X...X......XX...X..X.........X....] duration: 6.047348748s trace: https://jaeger.dev.storj.io/trace/3519e7bb67af2319 FAIL (2025-12-28 09:47:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =479018]: Starting upload Dec 28 09:45:28.272345405 (546h16m28.14342845s) [upload=479018]: Done starting upload Dec 28 09:45:28.272542140 (546h16m28.143625175s) [upload=479018]: Uploading segments... Dec 28 09:45:28.272601182 (546h16m28.143684227s) [upload=479018]: Got next segment. Inline: false Dec 28 09:45:28.272633553 (546h16m28.143716598s) [upload=479018]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:45:28.272638573 (546h16m28.143721618s) [upload=479018]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:45:28.517108586 (546h16m28.388191621s) [seg_pos={0 0}, upload=479018]: Begin upload segment... Dec 28 09:45:28.517151057 (546h16m28.388234092s) [seg_pos={0 0}, upload=479018]: Done begin upload segment. Dec 28 09:45:28.517196494 (546h16m28.388279529s) [noise=true , address=145.79.227.5:4223 , piece_id=R44JRSLYRBLUYS4SXMWQVYA5FKKO42JFFYCBUFR53FE3OLMAH2HQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.517205591 (546h16m28.388288636s) [noise=true , address=145.79.227.37:4567 , piece_id=OH3FDZCBZPH26ST3BTBUVK6PDT7TBUYGVINGJ535CV762EY7UUTQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.517645489 (546h16m28.388728524s) [noise=true , address=145.79.227.26:4655 , piece_id=XKEPME47XXIQQTBYHFJLM3ACRFBD3NL62HUSMFIJQDP2BAT3DJDA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.518064396 (546h16m28.389147442s) [noise=true , address=145.79.227.51:4411 , piece_id=LDSCU6FJJVECH2ZALLLKHT44QVEUT3VVQWCO4QXV4JKSHA4RJNFA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.518486981 (546h16m28.389570016s) [noise=true , address=45.91.116.7:4012 , piece_id=H7ES6CP7VA3KG5YAK3JOBUHOY5EMQW2FKEJPPIHEFTRSWQRIOBNA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.519070302 (546h16m28.390153337s) [noise=true , address=45.91.116.9:4261 , piece_id=6FDSNDNMZAFTISKN7FEJXBXJE2GERGVS3FQA4CHME47USEJ5DQUQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.519522914 (546h16m28.390605959s) [noise=true , address=45.91.116.12:4228 , piece_id=3LUQHNGM6QTUQDUOW37UXWUGE5WDPSTPQXHY4KP6TJ7DSNUE33VA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.520088731 (546h16m28.391171766s) [noise=true , address=145.79.227.11:4729 , piece_id=5MGX65D76ZGRV4YTDBFYR5LR436B45AO332N5S4XOVCYQQSRUFNQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.520469777 (546h16m28.391552812s) [noise=true , address=45.91.116.8:4047 , piece_id=JBFUI5HGPHGUYIZPM5SMHGHVN6FRJEACIWMYIJFG5DEG72N4IXDA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.520791429 (546h16m28.391874464s) [noise=true , address=145.79.227.4:4185 , piece_id=NKASFGR5SW4L43BFYJBX6LG667UUKD7QTWRZ3ZDD2ZWR6WXUVTFA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.521099125 (546h16m28.39218216s) [noise=true , address=45.91.116.4:4939 , piece_id=MBMUBN6V44OXK2VSOGDNIE2NPMTHNDBN2DWODT5DAMW2R2PMTY3A, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.521452418 (546h16m28.392535453s) [noise=true , address=145.79.227.29:4942 , piece_id=IDK63X74PG7UOAOWZIDMBQESETUZCMIVOAIQKCV2PJYIUND7R4LA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.523396439 (546h16m28.394479485s) [noise=true , address=145.79.227.9:4367 , piece_id=PMZIP647X4MH4W7SQGCANR63ZEUPGMMUPF2W5R2N6ZZTQ3SULZ3A, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.523464770 (546h16m28.394547815s) [noise=true , address=145.79.227.15:4876 , piece_id=7DJ6S25DS2QKYSMOJZVA5NRJATEYQGKICF7PRAZI7L7FBPK3OTEQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.523963379 (546h16m28.395046424s) [noise=true , address=145.79.227.45:4231 , piece_id=YUDXJSY5GZYFCYEDV2UN35WEUSRVSZOA7PST533Q4WYJO477OQGQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.525670970 (546h16m28.396754005s) [noise=true , address=45.91.116.14:4192 , piece_id=2TDWHC5ALNSZIZ5UY5LVHMTE736M3ANWPZOKXR3AWVONWCK6PG6A, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:30.237256159 (546h16m30.108339194s) [upload=479018]: Next returned nil segment Dec 28 09:45:30.237270817 (546h16m30.108353852s) [upload=479018]: Waiting for error group managing segments... Dec 28 09:45:30.525556952 (546h16m30.396639997s) [noise=true , address=145.79.227.15:4876 , piece_id=7DJ6S25DS2QKYSMOJZVA5NRJATEYQGKICF7PRAZI7L7FBPK3OTEQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.526594779 (546h16m30.397677824s) [noise=true , address=145.79.227.51:4411 , piece_id=LDSCU6FJJVECH2ZALLLKHT44QVEUT3VVQWCO4QXV4JKSHA4RJNFA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.526730968 (546h16m30.397814013s) [noise=true , address=145.79.227.9:4367 , piece_id=PMZIP647X4MH4W7SQGCANR63ZEUPGMMUPF2W5R2N6ZZTQ3SULZ3A, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.531182765 (546h16m30.4022658s) [noise=true , address=145.79.227.45:4231 , piece_id=YUDXJSY5GZYFCYEDV2UN35WEUSRVSZOA7PST533Q4WYJO477OQGQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.538899700 (546h16m30.409982745s) [noise=true , address=145.79.227.5:4223 , piece_id=R44JRSLYRBLUYS4SXMWQVYA5FKKO42JFFYCBUFR53FE3OLMAH2HQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.540774660 (546h16m30.411857715s) [noise=true , address=145.79.227.4:4185 , piece_id=NKASFGR5SW4L43BFYJBX6LG667UUKD7QTWRZ3ZDD2ZWR6WXUVTFA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.541541320 (546h16m30.412624355s) [noise=true , address=145.79.227.11:4729 , piece_id=5MGX65D76ZGRV4YTDBFYR5LR436B45AO332N5S4XOVCYQQSRUFNQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.545531758 (546h16m30.416614793s) [noise=true , address=145.79.227.37:4567 , piece_id=OH3FDZCBZPH26ST3BTBUVK6PDT7TBUYGVINGJ535CV762EY7UUTQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.547314974 (546h16m30.418398019s) [noise=true , address=145.79.227.26:4655 , piece_id=XKEPME47XXIQQTBYHFJLM3ACRFBD3NL62HUSMFIJQDP2BAT3DJDA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.560249253 (546h16m30.431332288s) [noise=true , address=145.79.227.29:4942 , piece_id=IDK63X74PG7UOAOWZIDMBQESETUZCMIVOAIQKCV2PJYIUND7R4LA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.741263574 (546h16m30.612346609s) [noise=true , address=45.91.116.7:4012 , piece_id=H7ES6CP7VA3KG5YAK3JOBUHOY5EMQW2FKEJPPIHEFTRSWQRIOBNA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.871268365 (546h16m30.74235141s) [noise=true , address=45.91.116.12:4228 , piece_id=3LUQHNGM6QTUQDUOW37UXWUGE5WDPSTPQXHY4KP6TJ7DSNUE33VA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:31.129984185 (546h16m31.00106724s) [noise=true , address=45.91.116.14:4192 , piece_id=2TDWHC5ALNSZIZ5UY5LVHMTE736M3ANWPZOKXR3AWVONWCK6PG6A, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:31.130007730 (546h16m31.001090775s) [seg_pos={0 0}, upload=479018]: Segment reached optimal threshold of 13 pieces. Dec 28 09:45:31.130221627 (546h16m31.001304662s) [noise=true , address=45.91.116.4:4939 , piece_id=MBMUBN6V44OXK2VSOGDNIE2NPMTHNDBN2DWODT5DAMW2R2PMTY3A, seg_pos={0 0}, upload=479018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 09:45:31.130340102 (546h16m31.001423177s) [noise=true , address=45.91.116.4:4939 , piece_id=MBMUBN6V44OXK2VSOGDNIE2NPMTHNDBN2DWODT5DAMW2R2PMTY3A, seg_pos={0 0}, upload=479018]: Long tail cancellation due to optimal threshold Dec 28 09:45:31.130379537 (546h16m31.001462572s) [noise=true , address=45.91.116.9:4261 , piece_id=6FDSNDNMZAFTISKN7FEJXBXJE2GERGVS3FQA4CHME47USEJ5DQUQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:49258->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:49258->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49258->45.91.116.9:4261: use of closed network connection Dec 28 09:45:31.130389626 (546h16m31.001472682s) [noise=true , address=45.91.116.9:4261 , piece_id=6FDSNDNMZAFTISKN7FEJXBXJE2GERGVS3FQA4CHME47USEJ5DQUQ, seg_pos={0 0}, upload=479018]: Long tail cancellation due to optimal threshold Dec 28 09:45:31.130664179 (546h16m31.001747224s) [noise=true , address=45.91.116.8:4047 , piece_id=JBFUI5HGPHGUYIZPM5SMHGHVN6FRJEACIWMYIJFG5DEG72N4IXDA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: read tcp 188.166.225.170:47210->45.91.116.8:4047: use of closed network connection; context canceled Dec 28 09:45:31.130671965 (546h16m31.001755s) [noise=true , address=45.91.116.8:4047 , piece_id=JBFUI5HGPHGUYIZPM5SMHGHVN6FRJEACIWMYIJFG5DEG72N4IXDA, seg_pos={0 0}, upload=479018]: Long tail cancellation due to optimal threshold Dec 28 09:45:31.130689287 (546h16m31.001772322s) [seg_pos={0 0}, upload=479018]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:45:31.130754111 (546h16m31.001837146s) [upload=479018]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:45:31.130758349 (546h16m31.001841384s) [upload=479018]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:45:31.421778310 (546h16m31.292861345s) [upload=479018]: Done uploading segments... Dec 28 09:45:31.421849585 (546h16m31.29293262s) [upload=479018]: Upload finished. err: <nil> Duration: 6.047373696s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [X....X.X.....X.....XXX.XX...XX.XX..X...XXX..XX.X...X.X....XX] duration: 5.001839039s trace: https://jaeger.dev.storj.io/trace/64d6c205d23bf212 FAIL (2025-12-28 09:47:22 UTC) orbiter warning: context deadline exceeded Duration: 5.001851832s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X......X..............X....X...X......XX...X..X.........X...] duration: 5.596792323s trace: https://jaeger.dev.storj.io/trace/3519e7bb67af2319 FAIL (2025-12-28 09:47:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =479018]: Starting upload Dec 28 09:45:28.272345405 (546h16m28.14342845s) [upload=479018]: Done starting upload Dec 28 09:45:28.272542140 (546h16m28.143625175s) [upload=479018]: Uploading segments... Dec 28 09:45:28.272601182 (546h16m28.143684227s) [upload=479018]: Got next segment. Inline: false Dec 28 09:45:28.272633553 (546h16m28.143716598s) [upload=479018]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:45:28.272638573 (546h16m28.143721618s) [upload=479018]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:45:28.517108586 (546h16m28.388191621s) [seg_pos={0 0}, upload=479018]: Begin upload segment... Dec 28 09:45:28.517151057 (546h16m28.388234092s) [seg_pos={0 0}, upload=479018]: Done begin upload segment. Dec 28 09:45:28.517196494 (546h16m28.388279529s) [noise=true , address=145.79.227.5:4223 , piece_id=R44JRSLYRBLUYS4SXMWQVYA5FKKO42JFFYCBUFR53FE3OLMAH2HQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.517205591 (546h16m28.388288636s) [noise=true , address=145.79.227.37:4567 , piece_id=OH3FDZCBZPH26ST3BTBUVK6PDT7TBUYGVINGJ535CV762EY7UUTQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.517645489 (546h16m28.388728524s) [noise=true , address=145.79.227.26:4655 , piece_id=XKEPME47XXIQQTBYHFJLM3ACRFBD3NL62HUSMFIJQDP2BAT3DJDA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.518064396 (546h16m28.389147442s) [noise=true , address=145.79.227.51:4411 , piece_id=LDSCU6FJJVECH2ZALLLKHT44QVEUT3VVQWCO4QXV4JKSHA4RJNFA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.518486981 (546h16m28.389570016s) [noise=true , address=45.91.116.7:4012 , piece_id=H7ES6CP7VA3KG5YAK3JOBUHOY5EMQW2FKEJPPIHEFTRSWQRIOBNA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.519070302 (546h16m28.390153337s) [noise=true , address=45.91.116.9:4261 , piece_id=6FDSNDNMZAFTISKN7FEJXBXJE2GERGVS3FQA4CHME47USEJ5DQUQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.519522914 (546h16m28.390605959s) [noise=true , address=45.91.116.12:4228 , piece_id=3LUQHNGM6QTUQDUOW37UXWUGE5WDPSTPQXHY4KP6TJ7DSNUE33VA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.520088731 (546h16m28.391171766s) [noise=true , address=145.79.227.11:4729 , piece_id=5MGX65D76ZGRV4YTDBFYR5LR436B45AO332N5S4XOVCYQQSRUFNQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.520469777 (546h16m28.391552812s) [noise=true , address=45.91.116.8:4047 , piece_id=JBFUI5HGPHGUYIZPM5SMHGHVN6FRJEACIWMYIJFG5DEG72N4IXDA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.520791429 (546h16m28.391874464s) [noise=true , address=145.79.227.4:4185 , piece_id=NKASFGR5SW4L43BFYJBX6LG667UUKD7QTWRZ3ZDD2ZWR6WXUVTFA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.521099125 (546h16m28.39218216s) [noise=true , address=45.91.116.4:4939 , piece_id=MBMUBN6V44OXK2VSOGDNIE2NPMTHNDBN2DWODT5DAMW2R2PMTY3A, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.521452418 (546h16m28.392535453s) [noise=true , address=145.79.227.29:4942 , piece_id=IDK63X74PG7UOAOWZIDMBQESETUZCMIVOAIQKCV2PJYIUND7R4LA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.523396439 (546h16m28.394479485s) [noise=true , address=145.79.227.9:4367 , piece_id=PMZIP647X4MH4W7SQGCANR63ZEUPGMMUPF2W5R2N6ZZTQ3SULZ3A, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.523464770 (546h16m28.394547815s) [noise=true , address=145.79.227.15:4876 , piece_id=7DJ6S25DS2QKYSMOJZVA5NRJATEYQGKICF7PRAZI7L7FBPK3OTEQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.523963379 (546h16m28.395046424s) [noise=true , address=145.79.227.45:4231 , piece_id=YUDXJSY5GZYFCYEDV2UN35WEUSRVSZOA7PST533Q4WYJO477OQGQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.525670970 (546h16m28.396754005s) [noise=true , address=45.91.116.14:4192 , piece_id=2TDWHC5ALNSZIZ5UY5LVHMTE736M3ANWPZOKXR3AWVONWCK6PG6A, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:30.237256159 (546h16m30.108339194s) [upload=479018]: Next returned nil segment Dec 28 09:45:30.237270817 (546h16m30.108353852s) [upload=479018]: Waiting for error group managing segments... Dec 28 09:45:30.525556952 (546h16m30.396639997s) [noise=true , address=145.79.227.15:4876 , piece_id=7DJ6S25DS2QKYSMOJZVA5NRJATEYQGKICF7PRAZI7L7FBPK3OTEQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.526594779 (546h16m30.397677824s) [noise=true , address=145.79.227.51:4411 , piece_id=LDSCU6FJJVECH2ZALLLKHT44QVEUT3VVQWCO4QXV4JKSHA4RJNFA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.526730968 (546h16m30.397814013s) [noise=true , address=145.79.227.9:4367 , piece_id=PMZIP647X4MH4W7SQGCANR63ZEUPGMMUPF2W5R2N6ZZTQ3SULZ3A, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.531182765 (546h16m30.4022658s) [noise=true , address=145.79.227.45:4231 , piece_id=YUDXJSY5GZYFCYEDV2UN35WEUSRVSZOA7PST533Q4WYJO477OQGQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.538899700 (546h16m30.409982745s) [noise=true , address=145.79.227.5:4223 , piece_id=R44JRSLYRBLUYS4SXMWQVYA5FKKO42JFFYCBUFR53FE3OLMAH2HQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.540774660 (546h16m30.411857715s) [noise=true , address=145.79.227.4:4185 , piece_id=NKASFGR5SW4L43BFYJBX6LG667UUKD7QTWRZ3ZDD2ZWR6WXUVTFA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.541541320 (546h16m30.412624355s) [noise=true , address=145.79.227.11:4729 , piece_id=5MGX65D76ZGRV4YTDBFYR5LR436B45AO332N5S4XOVCYQQSRUFNQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.545531758 (546h16m30.416614793s) [noise=true , address=145.79.227.37:4567 , piece_id=OH3FDZCBZPH26ST3BTBUVK6PDT7TBUYGVINGJ535CV762EY7UUTQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.547314974 (546h16m30.418398019s) [noise=true , address=145.79.227.26:4655 , piece_id=XKEPME47XXIQQTBYHFJLM3ACRFBD3NL62HUSMFIJQDP2BAT3DJDA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.560249253 (546h16m30.431332288s) [noise=true , address=145.79.227.29:4942 , piece_id=IDK63X74PG7UOAOWZIDMBQESETUZCMIVOAIQKCV2PJYIUND7R4LA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.741263574 (546h16m30.612346609s) [noise=true , address=45.91.116.7:4012 , piece_id=H7ES6CP7VA3KG5YAK3JOBUHOY5EMQW2FKEJPPIHEFTRSWQRIOBNA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.871268365 (546h16m30.74235141s) [noise=true , address=45.91.116.12:4228 , piece_id=3LUQHNGM6QTUQDUOW37UXWUGE5WDPSTPQXHY4KP6TJ7DSNUE33VA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:31.129984185 (546h16m31.00106724s) [noise=true , address=45.91.116.14:4192 , piece_id=2TDWHC5ALNSZIZ5UY5LVHMTE736M3ANWPZOKXR3AWVONWCK6PG6A, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:31.130007730 (546h16m31.001090775s) [seg_pos={0 0}, upload=479018]: Segment reached optimal threshold of 13 pieces. Dec 28 09:45:31.130221627 (546h16m31.001304662s) [noise=true , address=45.91.116.4:4939 , piece_id=MBMUBN6V44OXK2VSOGDNIE2NPMTHNDBN2DWODT5DAMW2R2PMTY3A, seg_pos={0 0}, upload=479018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 09:45:31.130340102 (546h16m31.001423177s) [noise=true , address=45.91.116.4:4939 , piece_id=MBMUBN6V44OXK2VSOGDNIE2NPMTHNDBN2DWODT5DAMW2R2PMTY3A, seg_pos={0 0}, upload=479018]: Long tail cancellation due to optimal threshold Dec 28 09:45:31.130379537 (546h16m31.001462572s) [noise=true , address=45.91.116.9:4261 , piece_id=6FDSNDNMZAFTISKN7FEJXBXJE2GERGVS3FQA4CHME47USEJ5DQUQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:49258->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:49258->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49258->45.91.116.9:4261: use of closed network connection Dec 28 09:45:31.130389626 (546h16m31.001472682s) [noise=true , address=45.91.116.9:4261 , piece_id=6FDSNDNMZAFTISKN7FEJXBXJE2GERGVS3FQA4CHME47USEJ5DQUQ, seg_pos={0 0}, upload=479018]: Long tail cancellation due to optimal threshold Dec 28 09:45:31.130664179 (546h16m31.001747224s) [noise=true , address=45.91.116.8:4047 , piece_id=JBFUI5HGPHGUYIZPM5SMHGHVN6FRJEACIWMYIJFG5DEG72N4IXDA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: read tcp 188.166.225.170:47210->45.91.116.8:4047: use of closed network connection; context canceled Dec 28 09:45:31.130671965 (546h16m31.001755s) [noise=true , address=45.91.116.8:4047 , piece_id=JBFUI5HGPHGUYIZPM5SMHGHVN6FRJEACIWMYIJFG5DEG72N4IXDA, seg_pos={0 0}, upload=479018]: Long tail cancellation due to optimal threshold Dec 28 09:45:31.130689287 (546h16m31.001772322s) [seg_pos={0 0}, upload=479018]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:45:31.130754111 (546h16m31.001837146s) [upload=479018]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:45:31.130758349 (546h16m31.001841384s) [upload=479018]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:45:31.421778310 (546h16m31.292861345s) [upload=479018]: Done uploading segments... Dec 28 09:45:31.421849585 (546h16m31.29293262s) [upload=479018]: Upload finished. err: <nil> Duration: 5.596822079s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.X......X..............X....X...X......XX...X..X.........X..] duration: 5.63454614s trace: https://jaeger.dev.storj.io/trace/3519e7bb67af2319 FAIL (2025-12-28 09:46:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =479018]: Starting upload Dec 28 09:45:28.272345405 (546h16m28.14342845s) [upload=479018]: Done starting upload Dec 28 09:45:28.272542140 (546h16m28.143625175s) [upload=479018]: Uploading segments... Dec 28 09:45:28.272601182 (546h16m28.143684227s) [upload=479018]: Got next segment. Inline: false Dec 28 09:45:28.272633553 (546h16m28.143716598s) [upload=479018]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:45:28.272638573 (546h16m28.143721618s) [upload=479018]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:45:28.517108586 (546h16m28.388191621s) [seg_pos={0 0}, upload=479018]: Begin upload segment... Dec 28 09:45:28.517151057 (546h16m28.388234092s) [seg_pos={0 0}, upload=479018]: Done begin upload segment. Dec 28 09:45:28.517196494 (546h16m28.388279529s) [noise=true , address=145.79.227.5:4223 , piece_id=R44JRSLYRBLUYS4SXMWQVYA5FKKO42JFFYCBUFR53FE3OLMAH2HQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.517205591 (546h16m28.388288636s) [noise=true , address=145.79.227.37:4567 , piece_id=OH3FDZCBZPH26ST3BTBUVK6PDT7TBUYGVINGJ535CV762EY7UUTQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.517645489 (546h16m28.388728524s) [noise=true , address=145.79.227.26:4655 , piece_id=XKEPME47XXIQQTBYHFJLM3ACRFBD3NL62HUSMFIJQDP2BAT3DJDA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.518064396 (546h16m28.389147442s) [noise=true , address=145.79.227.51:4411 , piece_id=LDSCU6FJJVECH2ZALLLKHT44QVEUT3VVQWCO4QXV4JKSHA4RJNFA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.518486981 (546h16m28.389570016s) [noise=true , address=45.91.116.7:4012 , piece_id=H7ES6CP7VA3KG5YAK3JOBUHOY5EMQW2FKEJPPIHEFTRSWQRIOBNA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.519070302 (546h16m28.390153337s) [noise=true , address=45.91.116.9:4261 , piece_id=6FDSNDNMZAFTISKN7FEJXBXJE2GERGVS3FQA4CHME47USEJ5DQUQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.519522914 (546h16m28.390605959s) [noise=true , address=45.91.116.12:4228 , piece_id=3LUQHNGM6QTUQDUOW37UXWUGE5WDPSTPQXHY4KP6TJ7DSNUE33VA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.520088731 (546h16m28.391171766s) [noise=true , address=145.79.227.11:4729 , piece_id=5MGX65D76ZGRV4YTDBFYR5LR436B45AO332N5S4XOVCYQQSRUFNQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.520469777 (546h16m28.391552812s) [noise=true , address=45.91.116.8:4047 , piece_id=JBFUI5HGPHGUYIZPM5SMHGHVN6FRJEACIWMYIJFG5DEG72N4IXDA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.520791429 (546h16m28.391874464s) [noise=true , address=145.79.227.4:4185 , piece_id=NKASFGR5SW4L43BFYJBX6LG667UUKD7QTWRZ3ZDD2ZWR6WXUVTFA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.521099125 (546h16m28.39218216s) [noise=true , address=45.91.116.4:4939 , piece_id=MBMUBN6V44OXK2VSOGDNIE2NPMTHNDBN2DWODT5DAMW2R2PMTY3A, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.521452418 (546h16m28.392535453s) [noise=true , address=145.79.227.29:4942 , piece_id=IDK63X74PG7UOAOWZIDMBQESETUZCMIVOAIQKCV2PJYIUND7R4LA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.523396439 (546h16m28.394479485s) [noise=true , address=145.79.227.9:4367 , piece_id=PMZIP647X4MH4W7SQGCANR63ZEUPGMMUPF2W5R2N6ZZTQ3SULZ3A, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.523464770 (546h16m28.394547815s) [noise=true , address=145.79.227.15:4876 , piece_id=7DJ6S25DS2QKYSMOJZVA5NRJATEYQGKICF7PRAZI7L7FBPK3OTEQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.523963379 (546h16m28.395046424s) [noise=true , address=145.79.227.45:4231 , piece_id=YUDXJSY5GZYFCYEDV2UN35WEUSRVSZOA7PST533Q4WYJO477OQGQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.525670970 (546h16m28.396754005s) [noise=true , address=45.91.116.14:4192 , piece_id=2TDWHC5ALNSZIZ5UY5LVHMTE736M3ANWPZOKXR3AWVONWCK6PG6A, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:30.237256159 (546h16m30.108339194s) [upload=479018]: Next returned nil segment Dec 28 09:45:30.237270817 (546h16m30.108353852s) [upload=479018]: Waiting for error group managing segments... Dec 28 09:45:30.525556952 (546h16m30.396639997s) [noise=true , address=145.79.227.15:4876 , piece_id=7DJ6S25DS2QKYSMOJZVA5NRJATEYQGKICF7PRAZI7L7FBPK3OTEQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.526594779 (546h16m30.397677824s) [noise=true , address=145.79.227.51:4411 , piece_id=LDSCU6FJJVECH2ZALLLKHT44QVEUT3VVQWCO4QXV4JKSHA4RJNFA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.526730968 (546h16m30.397814013s) [noise=true , address=145.79.227.9:4367 , piece_id=PMZIP647X4MH4W7SQGCANR63ZEUPGMMUPF2W5R2N6ZZTQ3SULZ3A, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.531182765 (546h16m30.4022658s) [noise=true , address=145.79.227.45:4231 , piece_id=YUDXJSY5GZYFCYEDV2UN35WEUSRVSZOA7PST533Q4WYJO477OQGQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.538899700 (546h16m30.409982745s) [noise=true , address=145.79.227.5:4223 , piece_id=R44JRSLYRBLUYS4SXMWQVYA5FKKO42JFFYCBUFR53FE3OLMAH2HQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.540774660 (546h16m30.411857715s) [noise=true , address=145.79.227.4:4185 , piece_id=NKASFGR5SW4L43BFYJBX6LG667UUKD7QTWRZ3ZDD2ZWR6WXUVTFA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.541541320 (546h16m30.412624355s) [noise=true , address=145.79.227.11:4729 , piece_id=5MGX65D76ZGRV4YTDBFYR5LR436B45AO332N5S4XOVCYQQSRUFNQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.545531758 (546h16m30.416614793s) [noise=true , address=145.79.227.37:4567 , piece_id=OH3FDZCBZPH26ST3BTBUVK6PDT7TBUYGVINGJ535CV762EY7UUTQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.547314974 (546h16m30.418398019s) [noise=true , address=145.79.227.26:4655 , piece_id=XKEPME47XXIQQTBYHFJLM3ACRFBD3NL62HUSMFIJQDP2BAT3DJDA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.560249253 (546h16m30.431332288s) [noise=true , address=145.79.227.29:4942 , piece_id=IDK63X74PG7UOAOWZIDMBQESETUZCMIVOAIQKCV2PJYIUND7R4LA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.741263574 (546h16m30.612346609s) [noise=true , address=45.91.116.7:4012 , piece_id=H7ES6CP7VA3KG5YAK3JOBUHOY5EMQW2FKEJPPIHEFTRSWQRIOBNA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.871268365 (546h16m30.74235141s) [noise=true , address=45.91.116.12:4228 , piece_id=3LUQHNGM6QTUQDUOW37UXWUGE5WDPSTPQXHY4KP6TJ7DSNUE33VA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:31.129984185 (546h16m31.00106724s) [noise=true , address=45.91.116.14:4192 , piece_id=2TDWHC5ALNSZIZ5UY5LVHMTE736M3ANWPZOKXR3AWVONWCK6PG6A, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:31.130007730 (546h16m31.001090775s) [seg_pos={0 0}, upload=479018]: Segment reached optimal threshold of 13 pieces. Dec 28 09:45:31.130221627 (546h16m31.001304662s) [noise=true , address=45.91.116.4:4939 , piece_id=MBMUBN6V44OXK2VSOGDNIE2NPMTHNDBN2DWODT5DAMW2R2PMTY3A, seg_pos={0 0}, upload=479018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 09:45:31.130340102 (546h16m31.001423177s) [noise=true , address=45.91.116.4:4939 , piece_id=MBMUBN6V44OXK2VSOGDNIE2NPMTHNDBN2DWODT5DAMW2R2PMTY3A, seg_pos={0 0}, upload=479018]: Long tail cancellation due to optimal threshold Dec 28 09:45:31.130379537 (546h16m31.001462572s) [noise=true , address=45.91.116.9:4261 , piece_id=6FDSNDNMZAFTISKN7FEJXBXJE2GERGVS3FQA4CHME47USEJ5DQUQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:49258->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:49258->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49258->45.91.116.9:4261: use of closed network connection Dec 28 09:45:31.130389626 (546h16m31.001472682s) [noise=true , address=45.91.116.9:4261 , piece_id=6FDSNDNMZAFTISKN7FEJXBXJE2GERGVS3FQA4CHME47USEJ5DQUQ, seg_pos={0 0}, upload=479018]: Long tail cancellation due to optimal threshold Dec 28 09:45:31.130664179 (546h16m31.001747224s) [noise=true , address=45.91.116.8:4047 , piece_id=JBFUI5HGPHGUYIZPM5SMHGHVN6FRJEACIWMYIJFG5DEG72N4IXDA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: read tcp 188.166.225.170:47210->45.91.116.8:4047: use of closed network connection; context canceled Dec 28 09:45:31.130671965 (546h16m31.001755s) [noise=true , address=45.91.116.8:4047 , piece_id=JBFUI5HGPHGUYIZPM5SMHGHVN6FRJEACIWMYIJFG5DEG72N4IXDA, seg_pos={0 0}, upload=479018]: Long tail cancellation due to optimal threshold Dec 28 09:45:31.130689287 (546h16m31.001772322s) [seg_pos={0 0}, upload=479018]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:45:31.130754111 (546h16m31.001837146s) [upload=479018]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:45:31.130758349 (546h16m31.001841384s) [upload=479018]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:45:31.421778310 (546h16m31.292861345s) [upload=479018]: Done uploading segments... Dec 28 09:45:31.421849585 (546h16m31.29293262s) [upload=479018]: Upload finished. err: <nil> Duration: 5.634573492s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [.X....X.X.....X.....XXX.XX...XX.XX..X...XXX..XX.X...X.X....X] duration: 5.001302879s trace: https://jaeger.dev.storj.io/trace/75df1da63186611d FAIL (2025-12-28 09:46:17 UTC) orbiter warning: context deadline exceeded Duration: 5.001321664s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..X......X..............X....X...X......XX...X..X.........X.] duration: 5.957796134s trace: https://jaeger.dev.storj.io/trace/3519e7bb67af2319 FAIL (2025-12-28 09:46:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =479018]: Starting upload Dec 28 09:45:28.272345405 (546h16m28.14342845s) [upload=479018]: Done starting upload Dec 28 09:45:28.272542140 (546h16m28.143625175s) [upload=479018]: Uploading segments... Dec 28 09:45:28.272601182 (546h16m28.143684227s) [upload=479018]: Got next segment. Inline: false Dec 28 09:45:28.272633553 (546h16m28.143716598s) [upload=479018]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:45:28.272638573 (546h16m28.143721618s) [upload=479018]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:45:28.517108586 (546h16m28.388191621s) [seg_pos={0 0}, upload=479018]: Begin upload segment... Dec 28 09:45:28.517151057 (546h16m28.388234092s) [seg_pos={0 0}, upload=479018]: Done begin upload segment. Dec 28 09:45:28.517196494 (546h16m28.388279529s) [noise=true , address=145.79.227.5:4223 , piece_id=R44JRSLYRBLUYS4SXMWQVYA5FKKO42JFFYCBUFR53FE3OLMAH2HQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.517205591 (546h16m28.388288636s) [noise=true , address=145.79.227.37:4567 , piece_id=OH3FDZCBZPH26ST3BTBUVK6PDT7TBUYGVINGJ535CV762EY7UUTQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.517645489 (546h16m28.388728524s) [noise=true , address=145.79.227.26:4655 , piece_id=XKEPME47XXIQQTBYHFJLM3ACRFBD3NL62HUSMFIJQDP2BAT3DJDA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.518064396 (546h16m28.389147442s) [noise=true , address=145.79.227.51:4411 , piece_id=LDSCU6FJJVECH2ZALLLKHT44QVEUT3VVQWCO4QXV4JKSHA4RJNFA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.518486981 (546h16m28.389570016s) [noise=true , address=45.91.116.7:4012 , piece_id=H7ES6CP7VA3KG5YAK3JOBUHOY5EMQW2FKEJPPIHEFTRSWQRIOBNA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.519070302 (546h16m28.390153337s) [noise=true , address=45.91.116.9:4261 , piece_id=6FDSNDNMZAFTISKN7FEJXBXJE2GERGVS3FQA4CHME47USEJ5DQUQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.519522914 (546h16m28.390605959s) [noise=true , address=45.91.116.12:4228 , piece_id=3LUQHNGM6QTUQDUOW37UXWUGE5WDPSTPQXHY4KP6TJ7DSNUE33VA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.520088731 (546h16m28.391171766s) [noise=true , address=145.79.227.11:4729 , piece_id=5MGX65D76ZGRV4YTDBFYR5LR436B45AO332N5S4XOVCYQQSRUFNQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.520469777 (546h16m28.391552812s) [noise=true , address=45.91.116.8:4047 , piece_id=JBFUI5HGPHGUYIZPM5SMHGHVN6FRJEACIWMYIJFG5DEG72N4IXDA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.520791429 (546h16m28.391874464s) [noise=true , address=145.79.227.4:4185 , piece_id=NKASFGR5SW4L43BFYJBX6LG667UUKD7QTWRZ3ZDD2ZWR6WXUVTFA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.521099125 (546h16m28.39218216s) [noise=true , address=45.91.116.4:4939 , piece_id=MBMUBN6V44OXK2VSOGDNIE2NPMTHNDBN2DWODT5DAMW2R2PMTY3A, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.521452418 (546h16m28.392535453s) [noise=true , address=145.79.227.29:4942 , piece_id=IDK63X74PG7UOAOWZIDMBQESETUZCMIVOAIQKCV2PJYIUND7R4LA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.523396439 (546h16m28.394479485s) [noise=true , address=145.79.227.9:4367 , piece_id=PMZIP647X4MH4W7SQGCANR63ZEUPGMMUPF2W5R2N6ZZTQ3SULZ3A, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.523464770 (546h16m28.394547815s) [noise=true , address=145.79.227.15:4876 , piece_id=7DJ6S25DS2QKYSMOJZVA5NRJATEYQGKICF7PRAZI7L7FBPK3OTEQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.523963379 (546h16m28.395046424s) [noise=true , address=145.79.227.45:4231 , piece_id=YUDXJSY5GZYFCYEDV2UN35WEUSRVSZOA7PST533Q4WYJO477OQGQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.525670970 (546h16m28.396754005s) [noise=true , address=45.91.116.14:4192 , piece_id=2TDWHC5ALNSZIZ5UY5LVHMTE736M3ANWPZOKXR3AWVONWCK6PG6A, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:30.237256159 (546h16m30.108339194s) [upload=479018]: Next returned nil segment Dec 28 09:45:30.237270817 (546h16m30.108353852s) [upload=479018]: Waiting for error group managing segments... Dec 28 09:45:30.525556952 (546h16m30.396639997s) [noise=true , address=145.79.227.15:4876 , piece_id=7DJ6S25DS2QKYSMOJZVA5NRJATEYQGKICF7PRAZI7L7FBPK3OTEQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.526594779 (546h16m30.397677824s) [noise=true , address=145.79.227.51:4411 , piece_id=LDSCU6FJJVECH2ZALLLKHT44QVEUT3VVQWCO4QXV4JKSHA4RJNFA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.526730968 (546h16m30.397814013s) [noise=true , address=145.79.227.9:4367 , piece_id=PMZIP647X4MH4W7SQGCANR63ZEUPGMMUPF2W5R2N6ZZTQ3SULZ3A, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.531182765 (546h16m30.4022658s) [noise=true , address=145.79.227.45:4231 , piece_id=YUDXJSY5GZYFCYEDV2UN35WEUSRVSZOA7PST533Q4WYJO477OQGQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.538899700 (546h16m30.409982745s) [noise=true , address=145.79.227.5:4223 , piece_id=R44JRSLYRBLUYS4SXMWQVYA5FKKO42JFFYCBUFR53FE3OLMAH2HQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.540774660 (546h16m30.411857715s) [noise=true , address=145.79.227.4:4185 , piece_id=NKASFGR5SW4L43BFYJBX6LG667UUKD7QTWRZ3ZDD2ZWR6WXUVTFA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.541541320 (546h16m30.412624355s) [noise=true , address=145.79.227.11:4729 , piece_id=5MGX65D76ZGRV4YTDBFYR5LR436B45AO332N5S4XOVCYQQSRUFNQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.545531758 (546h16m30.416614793s) [noise=true , address=145.79.227.37:4567 , piece_id=OH3FDZCBZPH26ST3BTBUVK6PDT7TBUYGVINGJ535CV762EY7UUTQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.547314974 (546h16m30.418398019s) [noise=true , address=145.79.227.26:4655 , piece_id=XKEPME47XXIQQTBYHFJLM3ACRFBD3NL62HUSMFIJQDP2BAT3DJDA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.560249253 (546h16m30.431332288s) [noise=true , address=145.79.227.29:4942 , piece_id=IDK63X74PG7UOAOWZIDMBQESETUZCMIVOAIQKCV2PJYIUND7R4LA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.741263574 (546h16m30.612346609s) [noise=true , address=45.91.116.7:4012 , piece_id=H7ES6CP7VA3KG5YAK3JOBUHOY5EMQW2FKEJPPIHEFTRSWQRIOBNA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.871268365 (546h16m30.74235141s) [noise=true , address=45.91.116.12:4228 , piece_id=3LUQHNGM6QTUQDUOW37UXWUGE5WDPSTPQXHY4KP6TJ7DSNUE33VA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:31.129984185 (546h16m31.00106724s) [noise=true , address=45.91.116.14:4192 , piece_id=2TDWHC5ALNSZIZ5UY5LVHMTE736M3ANWPZOKXR3AWVONWCK6PG6A, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:31.130007730 (546h16m31.001090775s) [seg_pos={0 0}, upload=479018]: Segment reached optimal threshold of 13 pieces. Dec 28 09:45:31.130221627 (546h16m31.001304662s) [noise=true , address=45.91.116.4:4939 , piece_id=MBMUBN6V44OXK2VSOGDNIE2NPMTHNDBN2DWODT5DAMW2R2PMTY3A, seg_pos={0 0}, upload=479018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 09:45:31.130340102 (546h16m31.001423177s) [noise=true , address=45.91.116.4:4939 , piece_id=MBMUBN6V44OXK2VSOGDNIE2NPMTHNDBN2DWODT5DAMW2R2PMTY3A, seg_pos={0 0}, upload=479018]: Long tail cancellation due to optimal threshold Dec 28 09:45:31.130379537 (546h16m31.001462572s) [noise=true , address=45.91.116.9:4261 , piece_id=6FDSNDNMZAFTISKN7FEJXBXJE2GERGVS3FQA4CHME47USEJ5DQUQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:49258->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:49258->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49258->45.91.116.9:4261: use of closed network connection Dec 28 09:45:31.130389626 (546h16m31.001472682s) [noise=true , address=45.91.116.9:4261 , piece_id=6FDSNDNMZAFTISKN7FEJXBXJE2GERGVS3FQA4CHME47USEJ5DQUQ, seg_pos={0 0}, upload=479018]: Long tail cancellation due to optimal threshold Dec 28 09:45:31.130664179 (546h16m31.001747224s) [noise=true , address=45.91.116.8:4047 , piece_id=JBFUI5HGPHGUYIZPM5SMHGHVN6FRJEACIWMYIJFG5DEG72N4IXDA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: read tcp 188.166.225.170:47210->45.91.116.8:4047: use of closed network connection; context canceled Dec 28 09:45:31.130671965 (546h16m31.001755s) [noise=true , address=45.91.116.8:4047 , piece_id=JBFUI5HGPHGUYIZPM5SMHGHVN6FRJEACIWMYIJFG5DEG72N4IXDA, seg_pos={0 0}, upload=479018]: Long tail cancellation due to optimal threshold Dec 28 09:45:31.130689287 (546h16m31.001772322s) [seg_pos={0 0}, upload=479018]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:45:31.130754111 (546h16m31.001837146s) [upload=479018]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:45:31.130758349 (546h16m31.001841384s) [upload=479018]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:45:31.421778310 (546h16m31.292861345s) [upload=479018]: Done uploading segments... Dec 28 09:45:31.421849585 (546h16m31.29293262s) [upload=479018]: Upload finished. err: <nil> Duration: 5.957813047s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...X......X..............X....X...X......XX...X..X.........X] duration: 30.00361883s trace: https://jaeger.dev.storj.io/trace/3519e7bb67af2319 FAIL (2025-12-28 09:45:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =479018]: Starting upload Dec 28 09:45:28.272345405 (546h16m28.14342845s) [upload=479018]: Done starting upload Dec 28 09:45:28.272542140 (546h16m28.143625175s) [upload=479018]: Uploading segments... Dec 28 09:45:28.272601182 (546h16m28.143684227s) [upload=479018]: Got next segment. Inline: false Dec 28 09:45:28.272633553 (546h16m28.143716598s) [upload=479018]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:45:28.272638573 (546h16m28.143721618s) [upload=479018]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:45:28.517108586 (546h16m28.388191621s) [seg_pos={0 0}, upload=479018]: Begin upload segment... Dec 28 09:45:28.517151057 (546h16m28.388234092s) [seg_pos={0 0}, upload=479018]: Done begin upload segment. Dec 28 09:45:28.517196494 (546h16m28.388279529s) [noise=true , address=145.79.227.5:4223 , piece_id=R44JRSLYRBLUYS4SXMWQVYA5FKKO42JFFYCBUFR53FE3OLMAH2HQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.517205591 (546h16m28.388288636s) [noise=true , address=145.79.227.37:4567 , piece_id=OH3FDZCBZPH26ST3BTBUVK6PDT7TBUYGVINGJ535CV762EY7UUTQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.517645489 (546h16m28.388728524s) [noise=true , address=145.79.227.26:4655 , piece_id=XKEPME47XXIQQTBYHFJLM3ACRFBD3NL62HUSMFIJQDP2BAT3DJDA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.518064396 (546h16m28.389147442s) [noise=true , address=145.79.227.51:4411 , piece_id=LDSCU6FJJVECH2ZALLLKHT44QVEUT3VVQWCO4QXV4JKSHA4RJNFA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.518486981 (546h16m28.389570016s) [noise=true , address=45.91.116.7:4012 , piece_id=H7ES6CP7VA3KG5YAK3JOBUHOY5EMQW2FKEJPPIHEFTRSWQRIOBNA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.519070302 (546h16m28.390153337s) [noise=true , address=45.91.116.9:4261 , piece_id=6FDSNDNMZAFTISKN7FEJXBXJE2GERGVS3FQA4CHME47USEJ5DQUQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.519522914 (546h16m28.390605959s) [noise=true , address=45.91.116.12:4228 , piece_id=3LUQHNGM6QTUQDUOW37UXWUGE5WDPSTPQXHY4KP6TJ7DSNUE33VA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.520088731 (546h16m28.391171766s) [noise=true , address=145.79.227.11:4729 , piece_id=5MGX65D76ZGRV4YTDBFYR5LR436B45AO332N5S4XOVCYQQSRUFNQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.520469777 (546h16m28.391552812s) [noise=true , address=45.91.116.8:4047 , piece_id=JBFUI5HGPHGUYIZPM5SMHGHVN6FRJEACIWMYIJFG5DEG72N4IXDA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.520791429 (546h16m28.391874464s) [noise=true , address=145.79.227.4:4185 , piece_id=NKASFGR5SW4L43BFYJBX6LG667UUKD7QTWRZ3ZDD2ZWR6WXUVTFA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.521099125 (546h16m28.39218216s) [noise=true , address=45.91.116.4:4939 , piece_id=MBMUBN6V44OXK2VSOGDNIE2NPMTHNDBN2DWODT5DAMW2R2PMTY3A, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.521452418 (546h16m28.392535453s) [noise=true , address=145.79.227.29:4942 , piece_id=IDK63X74PG7UOAOWZIDMBQESETUZCMIVOAIQKCV2PJYIUND7R4LA, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.523396439 (546h16m28.394479485s) [noise=true , address=145.79.227.9:4367 , piece_id=PMZIP647X4MH4W7SQGCANR63ZEUPGMMUPF2W5R2N6ZZTQ3SULZ3A, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.523464770 (546h16m28.394547815s) [noise=true , address=145.79.227.15:4876 , piece_id=7DJ6S25DS2QKYSMOJZVA5NRJATEYQGKICF7PRAZI7L7FBPK3OTEQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.523963379 (546h16m28.395046424s) [noise=true , address=145.79.227.45:4231 , piece_id=YUDXJSY5GZYFCYEDV2UN35WEUSRVSZOA7PST533Q4WYJO477OQGQ, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:28.525670970 (546h16m28.396754005s) [noise=true , address=45.91.116.14:4192 , piece_id=2TDWHC5ALNSZIZ5UY5LVHMTE736M3ANWPZOKXR3AWVONWCK6PG6A, seg_pos={0 0}, upload=479018]: Uploading piece... Dec 28 09:45:30.237256159 (546h16m30.108339194s) [upload=479018]: Next returned nil segment Dec 28 09:45:30.237270817 (546h16m30.108353852s) [upload=479018]: Waiting for error group managing segments... Dec 28 09:45:30.525556952 (546h16m30.396639997s) [noise=true , address=145.79.227.15:4876 , piece_id=7DJ6S25DS2QKYSMOJZVA5NRJATEYQGKICF7PRAZI7L7FBPK3OTEQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.526594779 (546h16m30.397677824s) [noise=true , address=145.79.227.51:4411 , piece_id=LDSCU6FJJVECH2ZALLLKHT44QVEUT3VVQWCO4QXV4JKSHA4RJNFA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.526730968 (546h16m30.397814013s) [noise=true , address=145.79.227.9:4367 , piece_id=PMZIP647X4MH4W7SQGCANR63ZEUPGMMUPF2W5R2N6ZZTQ3SULZ3A, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.531182765 (546h16m30.4022658s) [noise=true , address=145.79.227.45:4231 , piece_id=YUDXJSY5GZYFCYEDV2UN35WEUSRVSZOA7PST533Q4WYJO477OQGQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.538899700 (546h16m30.409982745s) [noise=true , address=145.79.227.5:4223 , piece_id=R44JRSLYRBLUYS4SXMWQVYA5FKKO42JFFYCBUFR53FE3OLMAH2HQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.540774660 (546h16m30.411857715s) [noise=true , address=145.79.227.4:4185 , piece_id=NKASFGR5SW4L43BFYJBX6LG667UUKD7QTWRZ3ZDD2ZWR6WXUVTFA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.541541320 (546h16m30.412624355s) [noise=true , address=145.79.227.11:4729 , piece_id=5MGX65D76ZGRV4YTDBFYR5LR436B45AO332N5S4XOVCYQQSRUFNQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.545531758 (546h16m30.416614793s) [noise=true , address=145.79.227.37:4567 , piece_id=OH3FDZCBZPH26ST3BTBUVK6PDT7TBUYGVINGJ535CV762EY7UUTQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.547314974 (546h16m30.418398019s) [noise=true , address=145.79.227.26:4655 , piece_id=XKEPME47XXIQQTBYHFJLM3ACRFBD3NL62HUSMFIJQDP2BAT3DJDA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.560249253 (546h16m30.431332288s) [noise=true , address=145.79.227.29:4942 , piece_id=IDK63X74PG7UOAOWZIDMBQESETUZCMIVOAIQKCV2PJYIUND7R4LA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.741263574 (546h16m30.612346609s) [noise=true , address=45.91.116.7:4012 , piece_id=H7ES6CP7VA3KG5YAK3JOBUHOY5EMQW2FKEJPPIHEFTRSWQRIOBNA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:30.871268365 (546h16m30.74235141s) [noise=true , address=45.91.116.12:4228 , piece_id=3LUQHNGM6QTUQDUOW37UXWUGE5WDPSTPQXHY4KP6TJ7DSNUE33VA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:31.129984185 (546h16m31.00106724s) [noise=true , address=45.91.116.14:4192 , piece_id=2TDWHC5ALNSZIZ5UY5LVHMTE736M3ANWPZOKXR3AWVONWCK6PG6A, seg_pos={0 0}, upload=479018]: Done uploading piece. err: <nil> Dec 28 09:45:31.130007730 (546h16m31.001090775s) [seg_pos={0 0}, upload=479018]: Segment reached optimal threshold of 13 pieces. Dec 28 09:45:31.130221627 (546h16m31.001304662s) [noise=true , address=45.91.116.4:4939 , piece_id=MBMUBN6V44OXK2VSOGDNIE2NPMTHNDBN2DWODT5DAMW2R2PMTY3A, seg_pos={0 0}, upload=479018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 09:45:31.130340102 (546h16m31.001423177s) [noise=true , address=45.91.116.4:4939 , piece_id=MBMUBN6V44OXK2VSOGDNIE2NPMTHNDBN2DWODT5DAMW2R2PMTY3A, seg_pos={0 0}, upload=479018]: Long tail cancellation due to optimal threshold Dec 28 09:45:31.130379537 (546h16m31.001462572s) [noise=true , address=45.91.116.9:4261 , piece_id=6FDSNDNMZAFTISKN7FEJXBXJE2GERGVS3FQA4CHME47USEJ5DQUQ, seg_pos={0 0}, upload=479018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:49258->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:49258->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49258->45.91.116.9:4261: use of closed network connection Dec 28 09:45:31.130389626 (546h16m31.001472682s) [noise=true , address=45.91.116.9:4261 , piece_id=6FDSNDNMZAFTISKN7FEJXBXJE2GERGVS3FQA4CHME47USEJ5DQUQ, seg_pos={0 0}, upload=479018]: Long tail cancellation due to optimal threshold Dec 28 09:45:31.130664179 (546h16m31.001747224s) [noise=true , address=45.91.116.8:4047 , piece_id=JBFUI5HGPHGUYIZPM5SMHGHVN6FRJEACIWMYIJFG5DEG72N4IXDA, seg_pos={0 0}, upload=479018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: read tcp 188.166.225.170:47210->45.91.116.8:4047: use of closed network connection; context canceled Dec 28 09:45:31.130671965 (546h16m31.001755s) [noise=true , address=45.91.116.8:4047 , piece_id=JBFUI5HGPHGUYIZPM5SMHGHVN6FRJEACIWMYIJFG5DEG72N4IXDA, seg_pos={0 0}, upload=479018]: Long tail cancellation due to optimal threshold Dec 28 09:45:31.130689287 (546h16m31.001772322s) [seg_pos={0 0}, upload=479018]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:45:31.130754111 (546h16m31.001837146s) [upload=479018]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:45:31.130758349 (546h16m31.001841384s) [upload=479018]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:45:31.421778310 (546h16m31.292861345s) [upload=479018]: Done uploading segments... Dec 28 09:45:31.421849585 (546h16m31.29293262s) [upload=479018]: Upload finished. err: <nil> Duration: 30.003643989s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [..X....X.X.....X.....XXX.XX...XX.XX..X...XXX..XX.X...X.X....] duration: 3.414040121s trace: https://jaeger.dev.storj.io/trace/58b8734740c16627 FAIL (2025-12-28 09:45:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.414073175s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [...X....X.X.....X.....XXX.XX...XX.XX..X...XXX..XX.X...X.X...] duration: 4.904801745s trace: https://jaeger.dev.storj.io/trace/58b8734740c16627 FAIL (2025-12-28 09:45:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.904831031s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [....X......X..............X....X...X......XX...X..X.........] duration: 5.85348597s trace: https://jaeger.dev.storj.io/trace/29090502e7ed37c5 FAIL (2025-12-28 09:44:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 4EJIFP2QGBDXAFAX22XGTMLKZZ7MRARIHSIIXO77SCN436ZVGWFA) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: ALZ4O3NJMIWBZPVLQ3BAFL7KYQVECYUA3JEOLIOJVPRWQ326FLFQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 4MB6SKKVKJNFWWWE24U2YJI25VAF7RLRPRIZQ2QTZXGLWPRSKI4Q) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: N75N4G6QH4PAHZPQYFBE5LKCYNLTBS25V6CN2NCRM4FPSSEB75VA) context deadline exceeded ======LOGS====== Dec 28 09:35:27.505217329 (546h6m27.376300374s) [upload=478880]: Starting upload Dec 28 09:35:27.505340463 (546h6m27.376423508s) [upload=478880]: Done starting upload Dec 28 09:35:27.505687754 (546h6m27.376770789s) [upload=478880]: Uploading segments... Dec 28 09:35:27.506519929 (546h6m27.377602964s) [upload=478880]: Got next segment. Inline: false Dec 28 09:35:27.506630178 (546h6m27.377713213s) [upload=478880]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:35:27.506638113 (546h6m27.377721158s) [upload=478880]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:35:27.751540519 (546h6m27.622623554s) [seg_pos={0 0}, upload=478880]: Begin upload segment... Dec 28 09:35:27.751587490 (546h6m27.622670525s) [seg_pos={0 0}, upload=478880]: Done begin upload segment. Dec 28 09:35:27.751618938 (546h6m27.622701985s) [noise=true , address=145.79.227.11:4730 , piece_id=T4GWZTVADA55XZLK6UQ2KP6ITZVMIHYKNA4KXQ2PUKUKLORBI3XQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.751629229 (546h6m27.622712264s) [noise=true , address=77.90.31.14:4662 , piece_id=ALZ4O3NJMIWBZPVLQ3BAFL7KYQVECYUA3JEOLIOJVPRWQ326FLFQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.752546295 (546h6m27.62362934s) [noise=true , address=77.90.31.18:4805 , piece_id=2OFQMO45JG3NG5QAXLM3MJ6UUPZS67LXTZNSTZ2NABRU2V5U346Q, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.753149643 (546h6m27.624232688s) [noise=true , address=45.91.116.8:4046 , piece_id=3JVV2WAVRHM5MA7M6GYBDYSTAL6LAMLYZIZD46BE2IQ3OHXV45OQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.753703628 (546h6m27.624786683s) [noise=true , address=45.91.116.9:4263 , piece_id=4MB6SKKVKJNFWWWE24U2YJI25VAF7RLRPRIZQ2QTZXGLWPRSKI4Q, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754042864 (546h6m27.62512593s) [noise=true , address=145.79.227.45:4230 , piece_id=NONHN4EKNARBD2YIAQ756DW4NAPOS3UMFCEWN4KJ5UEAJEHH4QIQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754144698 (546h6m27.625227733s) [noise=true , address=145.79.227.37:4565 , piece_id=25KGPQZXGUI2ZDEXOYAETSUI6YTFT65UJFAAAIYRWYUWEXSUCNHQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754566932 (546h6m27.625649977s) [noise=true , address=45.91.116.10:4083 , piece_id=VD32C5TT6OWORW3U6767NYXRFDULZSOA2PU6CSBTLLC3AQZR2HQA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754764529 (546h6m27.625847574s) [noise=true , address=45.91.116.14:4191 , piece_id=N75N4G6QH4PAHZPQYFBE5LKCYNLTBS25V6CN2NCRM4FPSSEB75VA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754905227 (546h6m27.625988272s) [noise=true , address=145.79.227.20:4438 , piece_id=4SZKGU2LEZ5WK6UTZVQJVYDO7J65K4OT2GKGERVSSTT57W7EECTA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.755227731 (546h6m27.626310766s) [noise=true , address=45.91.116.7:4010 , piece_id=P47MG5Q5GNZKA5U544TUKNK7PTARAHQAN2CHDMVXEZT42TBNXCJQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.755637491 (546h6m27.626720536s) [noise=true , address=145.79.227.44:4195 , piece_id=YNBFGVEO34DUAXVGJGHY3IO6KC6WNET72UONMMPOGNK5DQCIRVSA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.756059785 (546h6m27.62714282s) [noise=true , address=145.79.227.51:4411 , piece_id=WLJDZHCZIN2YWRBUHWP3WLELR5CBSVBZJBYPTP5MR7VZD4W5EPFA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.756485676 (546h6m27.627568711s) [noise=true , address=45.91.116.13:4156 , piece_id=4EJIFP2QGBDXAFAX22XGTMLKZZ7MRARIHSIIXO77SCN436ZVGWFA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.757098633 (546h6m27.628181668s) [noise=true , address=145.79.227.9:4367 , piece_id=KEPVDQAHLLDT4WCNQTHRQYDC3VKE3M5VDHGCUS6MCR6Z6ZILMXGQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.765253682 (546h6m27.636336727s) [noise=true , address=145.79.227.39:4013 , piece_id=KFXOW2DVPX4LHEHMUCZJBI7KED3SKWZIM53LGK32NZXTZOJKGDGQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:29.480652613 (546h6m29.351735648s) [upload=478880]: Next returned nil segment Dec 28 09:35:29.480666730 (546h6m29.351749765s) [upload=478880]: Waiting for error group managing segments... Dec 28 09:35:29.766845633 (546h6m29.637928678s) [noise=true , address=145.79.227.45:4230 , piece_id=NONHN4EKNARBD2YIAQ756DW4NAPOS3UMFCEWN4KJ5UEAJEHH4QIQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.767303565 (546h6m29.6383866s) [noise=true , address=145.79.227.11:4730 , piece_id=T4GWZTVADA55XZLK6UQ2KP6ITZVMIHYKNA4KXQ2PUKUKLORBI3XQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.779454122 (546h6m29.650537167s) [noise=true , address=145.79.227.39:4013 , piece_id=KFXOW2DVPX4LHEHMUCZJBI7KED3SKWZIM53LGK32NZXTZOJKGDGQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.780555569 (546h6m29.651638614s) [noise=true , address=145.79.227.37:4565 , piece_id=25KGPQZXGUI2ZDEXOYAETSUI6YTFT65UJFAAAIYRWYUWEXSUCNHQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.786442138 (546h6m29.657525183s) [noise=true , address=145.79.227.20:4438 , piece_id=4SZKGU2LEZ5WK6UTZVQJVYDO7J65K4OT2GKGERVSSTT57W7EECTA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.787226852 (546h6m29.658309887s) [noise=true , address=145.79.227.51:4411 , piece_id=WLJDZHCZIN2YWRBUHWP3WLELR5CBSVBZJBYPTP5MR7VZD4W5EPFA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.794995575 (546h6m29.66607862s) [noise=true , address=145.79.227.9:4367 , piece_id=KEPVDQAHLLDT4WCNQTHRQYDC3VKE3M5VDHGCUS6MCR6Z6ZILMXGQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.997843773 (546h6m29.868926808s) [noise=true , address=45.91.116.9:4263 , piece_id=4MB6SKKVKJNFWWWE24U2YJI25VAF7RLRPRIZQ2QTZXGLWPRSKI4Q, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.018887895 (546h6m29.88997093s) [noise=true , address=45.91.116.7:4010 , piece_id=P47MG5Q5GNZKA5U544TUKNK7PTARAHQAN2CHDMVXEZT42TBNXCJQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.048991647 (546h6m29.920074692s) [noise=true , address=45.91.116.13:4156 , piece_id=4EJIFP2QGBDXAFAX22XGTMLKZZ7MRARIHSIIXO77SCN436ZVGWFA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.057505478 (546h6m29.928588513s) [noise=true , address=77.90.31.14:4662 , piece_id=ALZ4O3NJMIWBZPVLQ3BAFL7KYQVECYUA3JEOLIOJVPRWQ326FLFQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.691813288 (546h6m30.562896333s) [noise=true , address=45.91.116.8:4046 , piece_id=3JVV2WAVRHM5MA7M6GYBDYSTAL6LAMLYZIZD46BE2IQ3OHXV45OQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.749635081 (546h6m30.620718116s) [noise=true , address=45.91.116.14:4191 , piece_id=N75N4G6QH4PAHZPQYFBE5LKCYNLTBS25V6CN2NCRM4FPSSEB75VA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.749656672 (546h6m30.620739717s) [seg_pos={0 0}, upload=478880]: Segment reached optimal threshold of 13 pieces. Dec 28 09:35:30.749916657 (546h6m30.620999702s) [noise=true , address=77.90.31.18:4805 , piece_id=2OFQMO45JG3NG5QAXLM3MJ6UUPZS67LXTZNSTZ2NABRU2V5U346Q, seg_pos={0 0}, upload=478880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 28 09:35:30.749925453 (546h6m30.621008498s) [noise=true , address=77.90.31.18:4805 , piece_id=2OFQMO45JG3NG5QAXLM3MJ6UUPZS67LXTZNSTZ2NABRU2V5U346Q, seg_pos={0 0}, upload=478880]: Long tail cancellation due to optimal threshold Dec 28 09:35:30.750339852 (546h6m30.621422897s) [noise=true , address=45.91.116.10:4083 , piece_id=VD32C5TT6OWORW3U6767NYXRFDULZSOA2PU6CSBTLLC3AQZR2HQA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:36368->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:36368->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36368->45.91.116.10:4083: use of closed network connection Dec 28 09:35:30.750348650 (546h6m30.621431695s) [noise=true , address=45.91.116.10:4083 , piece_id=VD32C5TT6OWORW3U6767NYXRFDULZSOA2PU6CSBTLLC3AQZR2HQA, seg_pos={0 0}, upload=478880]: Long tail cancellation due to optimal threshold Dec 28 09:35:30.750494658 (546h6m30.621577703s) [noise=true , address=145.79.227.44:4195 , piece_id=YNBFGVEO34DUAXVGJGHY3IO6KC6WNET72UONMMPOGNK5DQCIRVSA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 09:35:30.750502983 (546h6m30.621586018s) [noise=true , address=145.79.227.44:4195 , piece_id=YNBFGVEO34DUAXVGJGHY3IO6KC6WNET72UONMMPOGNK5DQCIRVSA, seg_pos={0 0}, upload=478880]: Long tail cancellation due to optimal threshold Dec 28 09:35:30.750523643 (546h6m30.621606688s) [seg_pos={0 0}, upload=478880]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:35:30.750604236 (546h6m30.621687281s) [upload=478880]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:35:30.750610137 (546h6m30.621693172s) [upload=478880]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:35:31.045721700 (546h6m30.916804735s) [upload=478880]: Done uploading segments... Dec 28 09:35:31.045746788 (546h6m30.916829833s) [upload=478880]: Upload finished. err: <nil> Duration: 5.853513571s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [X...X....X.X.....X.....XXX.XX...XX.XX..X...XXX..XX.X...X.X..] duration: 3.46815775s trace: https://jaeger.dev.storj.io/trace/58b8734740c16627 FAIL (2025-12-28 09:44:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.468187528s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.....X......X..............X....X...X......XX...X..X........] duration: 6.00802568s trace: https://jaeger.dev.storj.io/trace/29090502e7ed37c5 FAIL (2025-12-28 09:43:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 4EJIFP2QGBDXAFAX22XGTMLKZZ7MRARIHSIIXO77SCN436ZVGWFA) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: ALZ4O3NJMIWBZPVLQ3BAFL7KYQVECYUA3JEOLIOJVPRWQ326FLFQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 4MB6SKKVKJNFWWWE24U2YJI25VAF7RLRPRIZQ2QTZXGLWPRSKI4Q) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: N75N4G6QH4PAHZPQYFBE5LKCYNLTBS25V6CN2NCRM4FPSSEB75VA) context deadline exceeded ======LOGS====== Dec 28 09:35:27.505217329 (546h6m27.376300374s) [upload=478880]: Starting upload Dec 28 09:35:27.505340463 (546h6m27.376423508s) [upload=478880]: Done starting upload Dec 28 09:35:27.505687754 (546h6m27.376770789s) [upload=478880]: Uploading segments... Dec 28 09:35:27.506519929 (546h6m27.377602964s) [upload=478880]: Got next segment. Inline: false Dec 28 09:35:27.506630178 (546h6m27.377713213s) [upload=478880]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:35:27.506638113 (546h6m27.377721158s) [upload=478880]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:35:27.751540519 (546h6m27.622623554s) [seg_pos={0 0}, upload=478880]: Begin upload segment... Dec 28 09:35:27.751587490 (546h6m27.622670525s) [seg_pos={0 0}, upload=478880]: Done begin upload segment. Dec 28 09:35:27.751618938 (546h6m27.622701985s) [noise=true , address=145.79.227.11:4730 , piece_id=T4GWZTVADA55XZLK6UQ2KP6ITZVMIHYKNA4KXQ2PUKUKLORBI3XQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.751629229 (546h6m27.622712264s) [noise=true , address=77.90.31.14:4662 , piece_id=ALZ4O3NJMIWBZPVLQ3BAFL7KYQVECYUA3JEOLIOJVPRWQ326FLFQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.752546295 (546h6m27.62362934s) [noise=true , address=77.90.31.18:4805 , piece_id=2OFQMO45JG3NG5QAXLM3MJ6UUPZS67LXTZNSTZ2NABRU2V5U346Q, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.753149643 (546h6m27.624232688s) [noise=true , address=45.91.116.8:4046 , piece_id=3JVV2WAVRHM5MA7M6GYBDYSTAL6LAMLYZIZD46BE2IQ3OHXV45OQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.753703628 (546h6m27.624786683s) [noise=true , address=45.91.116.9:4263 , piece_id=4MB6SKKVKJNFWWWE24U2YJI25VAF7RLRPRIZQ2QTZXGLWPRSKI4Q, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754042864 (546h6m27.62512593s) [noise=true , address=145.79.227.45:4230 , piece_id=NONHN4EKNARBD2YIAQ756DW4NAPOS3UMFCEWN4KJ5UEAJEHH4QIQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754144698 (546h6m27.625227733s) [noise=true , address=145.79.227.37:4565 , piece_id=25KGPQZXGUI2ZDEXOYAETSUI6YTFT65UJFAAAIYRWYUWEXSUCNHQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754566932 (546h6m27.625649977s) [noise=true , address=45.91.116.10:4083 , piece_id=VD32C5TT6OWORW3U6767NYXRFDULZSOA2PU6CSBTLLC3AQZR2HQA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754764529 (546h6m27.625847574s) [noise=true , address=45.91.116.14:4191 , piece_id=N75N4G6QH4PAHZPQYFBE5LKCYNLTBS25V6CN2NCRM4FPSSEB75VA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754905227 (546h6m27.625988272s) [noise=true , address=145.79.227.20:4438 , piece_id=4SZKGU2LEZ5WK6UTZVQJVYDO7J65K4OT2GKGERVSSTT57W7EECTA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.755227731 (546h6m27.626310766s) [noise=true , address=45.91.116.7:4010 , piece_id=P47MG5Q5GNZKA5U544TUKNK7PTARAHQAN2CHDMVXEZT42TBNXCJQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.755637491 (546h6m27.626720536s) [noise=true , address=145.79.227.44:4195 , piece_id=YNBFGVEO34DUAXVGJGHY3IO6KC6WNET72UONMMPOGNK5DQCIRVSA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.756059785 (546h6m27.62714282s) [noise=true , address=145.79.227.51:4411 , piece_id=WLJDZHCZIN2YWRBUHWP3WLELR5CBSVBZJBYPTP5MR7VZD4W5EPFA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.756485676 (546h6m27.627568711s) [noise=true , address=45.91.116.13:4156 , piece_id=4EJIFP2QGBDXAFAX22XGTMLKZZ7MRARIHSIIXO77SCN436ZVGWFA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.757098633 (546h6m27.628181668s) [noise=true , address=145.79.227.9:4367 , piece_id=KEPVDQAHLLDT4WCNQTHRQYDC3VKE3M5VDHGCUS6MCR6Z6ZILMXGQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.765253682 (546h6m27.636336727s) [noise=true , address=145.79.227.39:4013 , piece_id=KFXOW2DVPX4LHEHMUCZJBI7KED3SKWZIM53LGK32NZXTZOJKGDGQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:29.480652613 (546h6m29.351735648s) [upload=478880]: Next returned nil segment Dec 28 09:35:29.480666730 (546h6m29.351749765s) [upload=478880]: Waiting for error group managing segments... Dec 28 09:35:29.766845633 (546h6m29.637928678s) [noise=true , address=145.79.227.45:4230 , piece_id=NONHN4EKNARBD2YIAQ756DW4NAPOS3UMFCEWN4KJ5UEAJEHH4QIQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.767303565 (546h6m29.6383866s) [noise=true , address=145.79.227.11:4730 , piece_id=T4GWZTVADA55XZLK6UQ2KP6ITZVMIHYKNA4KXQ2PUKUKLORBI3XQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.779454122 (546h6m29.650537167s) [noise=true , address=145.79.227.39:4013 , piece_id=KFXOW2DVPX4LHEHMUCZJBI7KED3SKWZIM53LGK32NZXTZOJKGDGQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.780555569 (546h6m29.651638614s) [noise=true , address=145.79.227.37:4565 , piece_id=25KGPQZXGUI2ZDEXOYAETSUI6YTFT65UJFAAAIYRWYUWEXSUCNHQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.786442138 (546h6m29.657525183s) [noise=true , address=145.79.227.20:4438 , piece_id=4SZKGU2LEZ5WK6UTZVQJVYDO7J65K4OT2GKGERVSSTT57W7EECTA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.787226852 (546h6m29.658309887s) [noise=true , address=145.79.227.51:4411 , piece_id=WLJDZHCZIN2YWRBUHWP3WLELR5CBSVBZJBYPTP5MR7VZD4W5EPFA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.794995575 (546h6m29.66607862s) [noise=true , address=145.79.227.9:4367 , piece_id=KEPVDQAHLLDT4WCNQTHRQYDC3VKE3M5VDHGCUS6MCR6Z6ZILMXGQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.997843773 (546h6m29.868926808s) [noise=true , address=45.91.116.9:4263 , piece_id=4MB6SKKVKJNFWWWE24U2YJI25VAF7RLRPRIZQ2QTZXGLWPRSKI4Q, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.018887895 (546h6m29.88997093s) [noise=true , address=45.91.116.7:4010 , piece_id=P47MG5Q5GNZKA5U544TUKNK7PTARAHQAN2CHDMVXEZT42TBNXCJQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.048991647 (546h6m29.920074692s) [noise=true , address=45.91.116.13:4156 , piece_id=4EJIFP2QGBDXAFAX22XGTMLKZZ7MRARIHSIIXO77SCN436ZVGWFA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.057505478 (546h6m29.928588513s) [noise=true , address=77.90.31.14:4662 , piece_id=ALZ4O3NJMIWBZPVLQ3BAFL7KYQVECYUA3JEOLIOJVPRWQ326FLFQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.691813288 (546h6m30.562896333s) [noise=true , address=45.91.116.8:4046 , piece_id=3JVV2WAVRHM5MA7M6GYBDYSTAL6LAMLYZIZD46BE2IQ3OHXV45OQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.749635081 (546h6m30.620718116s) [noise=true , address=45.91.116.14:4191 , piece_id=N75N4G6QH4PAHZPQYFBE5LKCYNLTBS25V6CN2NCRM4FPSSEB75VA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.749656672 (546h6m30.620739717s) [seg_pos={0 0}, upload=478880]: Segment reached optimal threshold of 13 pieces. Dec 28 09:35:30.749916657 (546h6m30.620999702s) [noise=true , address=77.90.31.18:4805 , piece_id=2OFQMO45JG3NG5QAXLM3MJ6UUPZS67LXTZNSTZ2NABRU2V5U346Q, seg_pos={0 0}, upload=478880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 28 09:35:30.749925453 (546h6m30.621008498s) [noise=true , address=77.90.31.18:4805 , piece_id=2OFQMO45JG3NG5QAXLM3MJ6UUPZS67LXTZNSTZ2NABRU2V5U346Q, seg_pos={0 0}, upload=478880]: Long tail cancellation due to optimal threshold Dec 28 09:35:30.750339852 (546h6m30.621422897s) [noise=true , address=45.91.116.10:4083 , piece_id=VD32C5TT6OWORW3U6767NYXRFDULZSOA2PU6CSBTLLC3AQZR2HQA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:36368->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:36368->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36368->45.91.116.10:4083: use of closed network connection Dec 28 09:35:30.750348650 (546h6m30.621431695s) [noise=true , address=45.91.116.10:4083 , piece_id=VD32C5TT6OWORW3U6767NYXRFDULZSOA2PU6CSBTLLC3AQZR2HQA, seg_pos={0 0}, upload=478880]: Long tail cancellation due to optimal threshold Dec 28 09:35:30.750494658 (546h6m30.621577703s) [noise=true , address=145.79.227.44:4195 , piece_id=YNBFGVEO34DUAXVGJGHY3IO6KC6WNET72UONMMPOGNK5DQCIRVSA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 09:35:30.750502983 (546h6m30.621586018s) [noise=true , address=145.79.227.44:4195 , piece_id=YNBFGVEO34DUAXVGJGHY3IO6KC6WNET72UONMMPOGNK5DQCIRVSA, seg_pos={0 0}, upload=478880]: Long tail cancellation due to optimal threshold Dec 28 09:35:30.750523643 (546h6m30.621606688s) [seg_pos={0 0}, upload=478880]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:35:30.750604236 (546h6m30.621687281s) [upload=478880]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:35:30.750610137 (546h6m30.621693172s) [upload=478880]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:35:31.045721700 (546h6m30.916804735s) [upload=478880]: Done uploading segments... Dec 28 09:35:31.045746788 (546h6m30.916829833s) [upload=478880]: Upload finished. err: <nil> Duration: 6.008053744s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [.X...X....X.X.....X.....XXX.XX...XX.XX..X...XXX..XX.X...X.X.] duration: 3.534893357s trace: https://jaeger.dev.storj.io/trace/58b8734740c16627 FAIL (2025-12-28 09:42:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.534925298s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X.....X......X..............X....X...X......XX...X..X.......] duration: 5.943163311s trace: https://jaeger.dev.storj.io/trace/29090502e7ed37c5 FAIL (2025-12-28 09:42:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 4EJIFP2QGBDXAFAX22XGTMLKZZ7MRARIHSIIXO77SCN436ZVGWFA) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: ALZ4O3NJMIWBZPVLQ3BAFL7KYQVECYUA3JEOLIOJVPRWQ326FLFQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 4MB6SKKVKJNFWWWE24U2YJI25VAF7RLRPRIZQ2QTZXGLWPRSKI4Q) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: N75N4G6QH4PAHZPQYFBE5LKCYNLTBS25V6CN2NCRM4FPSSEB75VA) context deadline exceeded ======LOGS====== Dec 28 09:35:27.505217329 (546h6m27.376300374s) [upload=478880]: Starting upload Dec 28 09:35:27.505340463 (546h6m27.376423508s) [upload=478880]: Done starting upload Dec 28 09:35:27.505687754 (546h6m27.376770789s) [upload=478880]: Uploading segments... Dec 28 09:35:27.506519929 (546h6m27.377602964s) [upload=478880]: Got next segment. Inline: false Dec 28 09:35:27.506630178 (546h6m27.377713213s) [upload=478880]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:35:27.506638113 (546h6m27.377721158s) [upload=478880]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:35:27.751540519 (546h6m27.622623554s) [seg_pos={0 0}, upload=478880]: Begin upload segment... Dec 28 09:35:27.751587490 (546h6m27.622670525s) [seg_pos={0 0}, upload=478880]: Done begin upload segment. Dec 28 09:35:27.751618938 (546h6m27.622701985s) [noise=true , address=145.79.227.11:4730 , piece_id=T4GWZTVADA55XZLK6UQ2KP6ITZVMIHYKNA4KXQ2PUKUKLORBI3XQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.751629229 (546h6m27.622712264s) [noise=true , address=77.90.31.14:4662 , piece_id=ALZ4O3NJMIWBZPVLQ3BAFL7KYQVECYUA3JEOLIOJVPRWQ326FLFQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.752546295 (546h6m27.62362934s) [noise=true , address=77.90.31.18:4805 , piece_id=2OFQMO45JG3NG5QAXLM3MJ6UUPZS67LXTZNSTZ2NABRU2V5U346Q, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.753149643 (546h6m27.624232688s) [noise=true , address=45.91.116.8:4046 , piece_id=3JVV2WAVRHM5MA7M6GYBDYSTAL6LAMLYZIZD46BE2IQ3OHXV45OQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.753703628 (546h6m27.624786683s) [noise=true , address=45.91.116.9:4263 , piece_id=4MB6SKKVKJNFWWWE24U2YJI25VAF7RLRPRIZQ2QTZXGLWPRSKI4Q, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754042864 (546h6m27.62512593s) [noise=true , address=145.79.227.45:4230 , piece_id=NONHN4EKNARBD2YIAQ756DW4NAPOS3UMFCEWN4KJ5UEAJEHH4QIQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754144698 (546h6m27.625227733s) [noise=true , address=145.79.227.37:4565 , piece_id=25KGPQZXGUI2ZDEXOYAETSUI6YTFT65UJFAAAIYRWYUWEXSUCNHQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754566932 (546h6m27.625649977s) [noise=true , address=45.91.116.10:4083 , piece_id=VD32C5TT6OWORW3U6767NYXRFDULZSOA2PU6CSBTLLC3AQZR2HQA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754764529 (546h6m27.625847574s) [noise=true , address=45.91.116.14:4191 , piece_id=N75N4G6QH4PAHZPQYFBE5LKCYNLTBS25V6CN2NCRM4FPSSEB75VA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754905227 (546h6m27.625988272s) [noise=true , address=145.79.227.20:4438 , piece_id=4SZKGU2LEZ5WK6UTZVQJVYDO7J65K4OT2GKGERVSSTT57W7EECTA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.755227731 (546h6m27.626310766s) [noise=true , address=45.91.116.7:4010 , piece_id=P47MG5Q5GNZKA5U544TUKNK7PTARAHQAN2CHDMVXEZT42TBNXCJQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.755637491 (546h6m27.626720536s) [noise=true , address=145.79.227.44:4195 , piece_id=YNBFGVEO34DUAXVGJGHY3IO6KC6WNET72UONMMPOGNK5DQCIRVSA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.756059785 (546h6m27.62714282s) [noise=true , address=145.79.227.51:4411 , piece_id=WLJDZHCZIN2YWRBUHWP3WLELR5CBSVBZJBYPTP5MR7VZD4W5EPFA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.756485676 (546h6m27.627568711s) [noise=true , address=45.91.116.13:4156 , piece_id=4EJIFP2QGBDXAFAX22XGTMLKZZ7MRARIHSIIXO77SCN436ZVGWFA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.757098633 (546h6m27.628181668s) [noise=true , address=145.79.227.9:4367 , piece_id=KEPVDQAHLLDT4WCNQTHRQYDC3VKE3M5VDHGCUS6MCR6Z6ZILMXGQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.765253682 (546h6m27.636336727s) [noise=true , address=145.79.227.39:4013 , piece_id=KFXOW2DVPX4LHEHMUCZJBI7KED3SKWZIM53LGK32NZXTZOJKGDGQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:29.480652613 (546h6m29.351735648s) [upload=478880]: Next returned nil segment Dec 28 09:35:29.480666730 (546h6m29.351749765s) [upload=478880]: Waiting for error group managing segments... Dec 28 09:35:29.766845633 (546h6m29.637928678s) [noise=true , address=145.79.227.45:4230 , piece_id=NONHN4EKNARBD2YIAQ756DW4NAPOS3UMFCEWN4KJ5UEAJEHH4QIQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.767303565 (546h6m29.6383866s) [noise=true , address=145.79.227.11:4730 , piece_id=T4GWZTVADA55XZLK6UQ2KP6ITZVMIHYKNA4KXQ2PUKUKLORBI3XQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.779454122 (546h6m29.650537167s) [noise=true , address=145.79.227.39:4013 , piece_id=KFXOW2DVPX4LHEHMUCZJBI7KED3SKWZIM53LGK32NZXTZOJKGDGQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.780555569 (546h6m29.651638614s) [noise=true , address=145.79.227.37:4565 , piece_id=25KGPQZXGUI2ZDEXOYAETSUI6YTFT65UJFAAAIYRWYUWEXSUCNHQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.786442138 (546h6m29.657525183s) [noise=true , address=145.79.227.20:4438 , piece_id=4SZKGU2LEZ5WK6UTZVQJVYDO7J65K4OT2GKGERVSSTT57W7EECTA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.787226852 (546h6m29.658309887s) [noise=true , address=145.79.227.51:4411 , piece_id=WLJDZHCZIN2YWRBUHWP3WLELR5CBSVBZJBYPTP5MR7VZD4W5EPFA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.794995575 (546h6m29.66607862s) [noise=true , address=145.79.227.9:4367 , piece_id=KEPVDQAHLLDT4WCNQTHRQYDC3VKE3M5VDHGCUS6MCR6Z6ZILMXGQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.997843773 (546h6m29.868926808s) [noise=true , address=45.91.116.9:4263 , piece_id=4MB6SKKVKJNFWWWE24U2YJI25VAF7RLRPRIZQ2QTZXGLWPRSKI4Q, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.018887895 (546h6m29.88997093s) [noise=true , address=45.91.116.7:4010 , piece_id=P47MG5Q5GNZKA5U544TUKNK7PTARAHQAN2CHDMVXEZT42TBNXCJQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.048991647 (546h6m29.920074692s) [noise=true , address=45.91.116.13:4156 , piece_id=4EJIFP2QGBDXAFAX22XGTMLKZZ7MRARIHSIIXO77SCN436ZVGWFA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.057505478 (546h6m29.928588513s) [noise=true , address=77.90.31.14:4662 , piece_id=ALZ4O3NJMIWBZPVLQ3BAFL7KYQVECYUA3JEOLIOJVPRWQ326FLFQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.691813288 (546h6m30.562896333s) [noise=true , address=45.91.116.8:4046 , piece_id=3JVV2WAVRHM5MA7M6GYBDYSTAL6LAMLYZIZD46BE2IQ3OHXV45OQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.749635081 (546h6m30.620718116s) [noise=true , address=45.91.116.14:4191 , piece_id=N75N4G6QH4PAHZPQYFBE5LKCYNLTBS25V6CN2NCRM4FPSSEB75VA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.749656672 (546h6m30.620739717s) [seg_pos={0 0}, upload=478880]: Segment reached optimal threshold of 13 pieces. Dec 28 09:35:30.749916657 (546h6m30.620999702s) [noise=true , address=77.90.31.18:4805 , piece_id=2OFQMO45JG3NG5QAXLM3MJ6UUPZS67LXTZNSTZ2NABRU2V5U346Q, seg_pos={0 0}, upload=478880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 28 09:35:30.749925453 (546h6m30.621008498s) [noise=true , address=77.90.31.18:4805 , piece_id=2OFQMO45JG3NG5QAXLM3MJ6UUPZS67LXTZNSTZ2NABRU2V5U346Q, seg_pos={0 0}, upload=478880]: Long tail cancellation due to optimal threshold Dec 28 09:35:30.750339852 (546h6m30.621422897s) [noise=true , address=45.91.116.10:4083 , piece_id=VD32C5TT6OWORW3U6767NYXRFDULZSOA2PU6CSBTLLC3AQZR2HQA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:36368->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:36368->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36368->45.91.116.10:4083: use of closed network connection Dec 28 09:35:30.750348650 (546h6m30.621431695s) [noise=true , address=45.91.116.10:4083 , piece_id=VD32C5TT6OWORW3U6767NYXRFDULZSOA2PU6CSBTLLC3AQZR2HQA, seg_pos={0 0}, upload=478880]: Long tail cancellation due to optimal threshold Dec 28 09:35:30.750494658 (546h6m30.621577703s) [noise=true , address=145.79.227.44:4195 , piece_id=YNBFGVEO34DUAXVGJGHY3IO6KC6WNET72UONMMPOGNK5DQCIRVSA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 09:35:30.750502983 (546h6m30.621586018s) [noise=true , address=145.79.227.44:4195 , piece_id=YNBFGVEO34DUAXVGJGHY3IO6KC6WNET72UONMMPOGNK5DQCIRVSA, seg_pos={0 0}, upload=478880]: Long tail cancellation due to optimal threshold Dec 28 09:35:30.750523643 (546h6m30.621606688s) [seg_pos={0 0}, upload=478880]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:35:30.750604236 (546h6m30.621687281s) [upload=478880]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:35:30.750610137 (546h6m30.621693172s) [upload=478880]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:35:31.045721700 (546h6m30.916804735s) [upload=478880]: Done uploading segments... Dec 28 09:35:31.045746788 (546h6m30.916829833s) [upload=478880]: Upload finished. err: <nil> Duration: 5.943191705s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [X.X...X....X.X.....X.....XXX.XX...XX.XX..X...XXX..XX.X...X.X] duration: 5.001678746s trace: https://jaeger.dev.storj.io/trace/58b8734740c16627 FAIL (2025-12-28 09:41:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001696338s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.X.....X......X..............X....X...X......XX...X..X......] duration: 5.941751866s trace: https://jaeger.dev.storj.io/trace/29090502e7ed37c5 FAIL (2025-12-28 09:41:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 4EJIFP2QGBDXAFAX22XGTMLKZZ7MRARIHSIIXO77SCN436ZVGWFA) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: ALZ4O3NJMIWBZPVLQ3BAFL7KYQVECYUA3JEOLIOJVPRWQ326FLFQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 4MB6SKKVKJNFWWWE24U2YJI25VAF7RLRPRIZQ2QTZXGLWPRSKI4Q) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: N75N4G6QH4PAHZPQYFBE5LKCYNLTBS25V6CN2NCRM4FPSSEB75VA) context deadline exceeded ======LOGS====== Dec 28 09:35:27.505217329 (546h6m27.376300374s) [upload=478880]: Starting upload Dec 28 09:35:27.505340463 (546h6m27.376423508s) [upload=478880]: Done starting upload Dec 28 09:35:27.505687754 (546h6m27.376770789s) [upload=478880]: Uploading segments... Dec 28 09:35:27.506519929 (546h6m27.377602964s) [upload=478880]: Got next segment. Inline: false Dec 28 09:35:27.506630178 (546h6m27.377713213s) [upload=478880]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:35:27.506638113 (546h6m27.377721158s) [upload=478880]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:35:27.751540519 (546h6m27.622623554s) [seg_pos={0 0}, upload=478880]: Begin upload segment... Dec 28 09:35:27.751587490 (546h6m27.622670525s) [seg_pos={0 0}, upload=478880]: Done begin upload segment. Dec 28 09:35:27.751618938 (546h6m27.622701985s) [noise=true , address=145.79.227.11:4730 , piece_id=T4GWZTVADA55XZLK6UQ2KP6ITZVMIHYKNA4KXQ2PUKUKLORBI3XQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.751629229 (546h6m27.622712264s) [noise=true , address=77.90.31.14:4662 , piece_id=ALZ4O3NJMIWBZPVLQ3BAFL7KYQVECYUA3JEOLIOJVPRWQ326FLFQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.752546295 (546h6m27.62362934s) [noise=true , address=77.90.31.18:4805 , piece_id=2OFQMO45JG3NG5QAXLM3MJ6UUPZS67LXTZNSTZ2NABRU2V5U346Q, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.753149643 (546h6m27.624232688s) [noise=true , address=45.91.116.8:4046 , piece_id=3JVV2WAVRHM5MA7M6GYBDYSTAL6LAMLYZIZD46BE2IQ3OHXV45OQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.753703628 (546h6m27.624786683s) [noise=true , address=45.91.116.9:4263 , piece_id=4MB6SKKVKJNFWWWE24U2YJI25VAF7RLRPRIZQ2QTZXGLWPRSKI4Q, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754042864 (546h6m27.62512593s) [noise=true , address=145.79.227.45:4230 , piece_id=NONHN4EKNARBD2YIAQ756DW4NAPOS3UMFCEWN4KJ5UEAJEHH4QIQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754144698 (546h6m27.625227733s) [noise=true , address=145.79.227.37:4565 , piece_id=25KGPQZXGUI2ZDEXOYAETSUI6YTFT65UJFAAAIYRWYUWEXSUCNHQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754566932 (546h6m27.625649977s) [noise=true , address=45.91.116.10:4083 , piece_id=VD32C5TT6OWORW3U6767NYXRFDULZSOA2PU6CSBTLLC3AQZR2HQA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754764529 (546h6m27.625847574s) [noise=true , address=45.91.116.14:4191 , piece_id=N75N4G6QH4PAHZPQYFBE5LKCYNLTBS25V6CN2NCRM4FPSSEB75VA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754905227 (546h6m27.625988272s) [noise=true , address=145.79.227.20:4438 , piece_id=4SZKGU2LEZ5WK6UTZVQJVYDO7J65K4OT2GKGERVSSTT57W7EECTA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.755227731 (546h6m27.626310766s) [noise=true , address=45.91.116.7:4010 , piece_id=P47MG5Q5GNZKA5U544TUKNK7PTARAHQAN2CHDMVXEZT42TBNXCJQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.755637491 (546h6m27.626720536s) [noise=true , address=145.79.227.44:4195 , piece_id=YNBFGVEO34DUAXVGJGHY3IO6KC6WNET72UONMMPOGNK5DQCIRVSA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.756059785 (546h6m27.62714282s) [noise=true , address=145.79.227.51:4411 , piece_id=WLJDZHCZIN2YWRBUHWP3WLELR5CBSVBZJBYPTP5MR7VZD4W5EPFA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.756485676 (546h6m27.627568711s) [noise=true , address=45.91.116.13:4156 , piece_id=4EJIFP2QGBDXAFAX22XGTMLKZZ7MRARIHSIIXO77SCN436ZVGWFA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.757098633 (546h6m27.628181668s) [noise=true , address=145.79.227.9:4367 , piece_id=KEPVDQAHLLDT4WCNQTHRQYDC3VKE3M5VDHGCUS6MCR6Z6ZILMXGQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.765253682 (546h6m27.636336727s) [noise=true , address=145.79.227.39:4013 , piece_id=KFXOW2DVPX4LHEHMUCZJBI7KED3SKWZIM53LGK32NZXTZOJKGDGQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:29.480652613 (546h6m29.351735648s) [upload=478880]: Next returned nil segment Dec 28 09:35:29.480666730 (546h6m29.351749765s) [upload=478880]: Waiting for error group managing segments... Dec 28 09:35:29.766845633 (546h6m29.637928678s) [noise=true , address=145.79.227.45:4230 , piece_id=NONHN4EKNARBD2YIAQ756DW4NAPOS3UMFCEWN4KJ5UEAJEHH4QIQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.767303565 (546h6m29.6383866s) [noise=true , address=145.79.227.11:4730 , piece_id=T4GWZTVADA55XZLK6UQ2KP6ITZVMIHYKNA4KXQ2PUKUKLORBI3XQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.779454122 (546h6m29.650537167s) [noise=true , address=145.79.227.39:4013 , piece_id=KFXOW2DVPX4LHEHMUCZJBI7KED3SKWZIM53LGK32NZXTZOJKGDGQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.780555569 (546h6m29.651638614s) [noise=true , address=145.79.227.37:4565 , piece_id=25KGPQZXGUI2ZDEXOYAETSUI6YTFT65UJFAAAIYRWYUWEXSUCNHQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.786442138 (546h6m29.657525183s) [noise=true , address=145.79.227.20:4438 , piece_id=4SZKGU2LEZ5WK6UTZVQJVYDO7J65K4OT2GKGERVSSTT57W7EECTA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.787226852 (546h6m29.658309887s) [noise=true , address=145.79.227.51:4411 , piece_id=WLJDZHCZIN2YWRBUHWP3WLELR5CBSVBZJBYPTP5MR7VZD4W5EPFA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.794995575 (546h6m29.66607862s) [noise=true , address=145.79.227.9:4367 , piece_id=KEPVDQAHLLDT4WCNQTHRQYDC3VKE3M5VDHGCUS6MCR6Z6ZILMXGQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.997843773 (546h6m29.868926808s) [noise=true , address=45.91.116.9:4263 , piece_id=4MB6SKKVKJNFWWWE24U2YJI25VAF7RLRPRIZQ2QTZXGLWPRSKI4Q, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.018887895 (546h6m29.88997093s) [noise=true , address=45.91.116.7:4010 , piece_id=P47MG5Q5GNZKA5U544TUKNK7PTARAHQAN2CHDMVXEZT42TBNXCJQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.048991647 (546h6m29.920074692s) [noise=true , address=45.91.116.13:4156 , piece_id=4EJIFP2QGBDXAFAX22XGTMLKZZ7MRARIHSIIXO77SCN436ZVGWFA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.057505478 (546h6m29.928588513s) [noise=true , address=77.90.31.14:4662 , piece_id=ALZ4O3NJMIWBZPVLQ3BAFL7KYQVECYUA3JEOLIOJVPRWQ326FLFQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.691813288 (546h6m30.562896333s) [noise=true , address=45.91.116.8:4046 , piece_id=3JVV2WAVRHM5MA7M6GYBDYSTAL6LAMLYZIZD46BE2IQ3OHXV45OQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.749635081 (546h6m30.620718116s) [noise=true , address=45.91.116.14:4191 , piece_id=N75N4G6QH4PAHZPQYFBE5LKCYNLTBS25V6CN2NCRM4FPSSEB75VA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.749656672 (546h6m30.620739717s) [seg_pos={0 0}, upload=478880]: Segment reached optimal threshold of 13 pieces. Dec 28 09:35:30.749916657 (546h6m30.620999702s) [noise=true , address=77.90.31.18:4805 , piece_id=2OFQMO45JG3NG5QAXLM3MJ6UUPZS67LXTZNSTZ2NABRU2V5U346Q, seg_pos={0 0}, upload=478880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 28 09:35:30.749925453 (546h6m30.621008498s) [noise=true , address=77.90.31.18:4805 , piece_id=2OFQMO45JG3NG5QAXLM3MJ6UUPZS67LXTZNSTZ2NABRU2V5U346Q, seg_pos={0 0}, upload=478880]: Long tail cancellation due to optimal threshold Dec 28 09:35:30.750339852 (546h6m30.621422897s) [noise=true , address=45.91.116.10:4083 , piece_id=VD32C5TT6OWORW3U6767NYXRFDULZSOA2PU6CSBTLLC3AQZR2HQA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:36368->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:36368->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36368->45.91.116.10:4083: use of closed network connection Dec 28 09:35:30.750348650 (546h6m30.621431695s) [noise=true , address=45.91.116.10:4083 , piece_id=VD32C5TT6OWORW3U6767NYXRFDULZSOA2PU6CSBTLLC3AQZR2HQA, seg_pos={0 0}, upload=478880]: Long tail cancellation due to optimal threshold Dec 28 09:35:30.750494658 (546h6m30.621577703s) [noise=true , address=145.79.227.44:4195 , piece_id=YNBFGVEO34DUAXVGJGHY3IO6KC6WNET72UONMMPOGNK5DQCIRVSA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 09:35:30.750502983 (546h6m30.621586018s) [noise=true , address=145.79.227.44:4195 , piece_id=YNBFGVEO34DUAXVGJGHY3IO6KC6WNET72UONMMPOGNK5DQCIRVSA, seg_pos={0 0}, upload=478880]: Long tail cancellation due to optimal threshold Dec 28 09:35:30.750523643 (546h6m30.621606688s) [seg_pos={0 0}, upload=478880]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:35:30.750604236 (546h6m30.621687281s) [upload=478880]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:35:30.750610137 (546h6m30.621693172s) [upload=478880]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:35:31.045721700 (546h6m30.916804735s) [upload=478880]: Done uploading segments... Dec 28 09:35:31.045746788 (546h6m30.916829833s) [upload=478880]: Upload finished. err: <nil> Duration: 5.941779228s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [.X.X...X....X.X.....X.....XXX.XX...XX.XX..X...XXX..XX.X...X.] duration: 5.001016255s trace: https://jaeger.dev.storj.io/trace/6e14d995906d41c2 FAIL (2025-12-28 09:40:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001075607s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X.X.....X......X..............X....X...X......XX...X..X.....] duration: 6.041015177s trace: https://jaeger.dev.storj.io/trace/29090502e7ed37c5 FAIL (2025-12-28 09:40:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 4EJIFP2QGBDXAFAX22XGTMLKZZ7MRARIHSIIXO77SCN436ZVGWFA) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: ALZ4O3NJMIWBZPVLQ3BAFL7KYQVECYUA3JEOLIOJVPRWQ326FLFQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 4MB6SKKVKJNFWWWE24U2YJI25VAF7RLRPRIZQ2QTZXGLWPRSKI4Q) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: N75N4G6QH4PAHZPQYFBE5LKCYNLTBS25V6CN2NCRM4FPSSEB75VA) context deadline exceeded ======LOGS====== Dec 28 09:35:27.505217329 (546h6m27.376300374s) [upload=478880]: Starting upload Dec 28 09:35:27.505340463 (546h6m27.376423508s) [upload=478880]: Done starting upload Dec 28 09:35:27.505687754 (546h6m27.376770789s) [upload=478880]: Uploading segments... Dec 28 09:35:27.506519929 (546h6m27.377602964s) [upload=478880]: Got next segment. Inline: false Dec 28 09:35:27.506630178 (546h6m27.377713213s) [upload=478880]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:35:27.506638113 (546h6m27.377721158s) [upload=478880]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:35:27.751540519 (546h6m27.622623554s) [seg_pos={0 0}, upload=478880]: Begin upload segment... Dec 28 09:35:27.751587490 (546h6m27.622670525s) [seg_pos={0 0}, upload=478880]: Done begin upload segment. Dec 28 09:35:27.751618938 (546h6m27.622701985s) [noise=true , address=145.79.227.11:4730 , piece_id=T4GWZTVADA55XZLK6UQ2KP6ITZVMIHYKNA4KXQ2PUKUKLORBI3XQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.751629229 (546h6m27.622712264s) [noise=true , address=77.90.31.14:4662 , piece_id=ALZ4O3NJMIWBZPVLQ3BAFL7KYQVECYUA3JEOLIOJVPRWQ326FLFQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.752546295 (546h6m27.62362934s) [noise=true , address=77.90.31.18:4805 , piece_id=2OFQMO45JG3NG5QAXLM3MJ6UUPZS67LXTZNSTZ2NABRU2V5U346Q, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.753149643 (546h6m27.624232688s) [noise=true , address=45.91.116.8:4046 , piece_id=3JVV2WAVRHM5MA7M6GYBDYSTAL6LAMLYZIZD46BE2IQ3OHXV45OQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.753703628 (546h6m27.624786683s) [noise=true , address=45.91.116.9:4263 , piece_id=4MB6SKKVKJNFWWWE24U2YJI25VAF7RLRPRIZQ2QTZXGLWPRSKI4Q, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754042864 (546h6m27.62512593s) [noise=true , address=145.79.227.45:4230 , piece_id=NONHN4EKNARBD2YIAQ756DW4NAPOS3UMFCEWN4KJ5UEAJEHH4QIQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754144698 (546h6m27.625227733s) [noise=true , address=145.79.227.37:4565 , piece_id=25KGPQZXGUI2ZDEXOYAETSUI6YTFT65UJFAAAIYRWYUWEXSUCNHQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754566932 (546h6m27.625649977s) [noise=true , address=45.91.116.10:4083 , piece_id=VD32C5TT6OWORW3U6767NYXRFDULZSOA2PU6CSBTLLC3AQZR2HQA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754764529 (546h6m27.625847574s) [noise=true , address=45.91.116.14:4191 , piece_id=N75N4G6QH4PAHZPQYFBE5LKCYNLTBS25V6CN2NCRM4FPSSEB75VA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754905227 (546h6m27.625988272s) [noise=true , address=145.79.227.20:4438 , piece_id=4SZKGU2LEZ5WK6UTZVQJVYDO7J65K4OT2GKGERVSSTT57W7EECTA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.755227731 (546h6m27.626310766s) [noise=true , address=45.91.116.7:4010 , piece_id=P47MG5Q5GNZKA5U544TUKNK7PTARAHQAN2CHDMVXEZT42TBNXCJQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.755637491 (546h6m27.626720536s) [noise=true , address=145.79.227.44:4195 , piece_id=YNBFGVEO34DUAXVGJGHY3IO6KC6WNET72UONMMPOGNK5DQCIRVSA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.756059785 (546h6m27.62714282s) [noise=true , address=145.79.227.51:4411 , piece_id=WLJDZHCZIN2YWRBUHWP3WLELR5CBSVBZJBYPTP5MR7VZD4W5EPFA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.756485676 (546h6m27.627568711s) [noise=true , address=45.91.116.13:4156 , piece_id=4EJIFP2QGBDXAFAX22XGTMLKZZ7MRARIHSIIXO77SCN436ZVGWFA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.757098633 (546h6m27.628181668s) [noise=true , address=145.79.227.9:4367 , piece_id=KEPVDQAHLLDT4WCNQTHRQYDC3VKE3M5VDHGCUS6MCR6Z6ZILMXGQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.765253682 (546h6m27.636336727s) [noise=true , address=145.79.227.39:4013 , piece_id=KFXOW2DVPX4LHEHMUCZJBI7KED3SKWZIM53LGK32NZXTZOJKGDGQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:29.480652613 (546h6m29.351735648s) [upload=478880]: Next returned nil segment Dec 28 09:35:29.480666730 (546h6m29.351749765s) [upload=478880]: Waiting for error group managing segments... Dec 28 09:35:29.766845633 (546h6m29.637928678s) [noise=true , address=145.79.227.45:4230 , piece_id=NONHN4EKNARBD2YIAQ756DW4NAPOS3UMFCEWN4KJ5UEAJEHH4QIQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.767303565 (546h6m29.6383866s) [noise=true , address=145.79.227.11:4730 , piece_id=T4GWZTVADA55XZLK6UQ2KP6ITZVMIHYKNA4KXQ2PUKUKLORBI3XQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.779454122 (546h6m29.650537167s) [noise=true , address=145.79.227.39:4013 , piece_id=KFXOW2DVPX4LHEHMUCZJBI7KED3SKWZIM53LGK32NZXTZOJKGDGQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.780555569 (546h6m29.651638614s) [noise=true , address=145.79.227.37:4565 , piece_id=25KGPQZXGUI2ZDEXOYAETSUI6YTFT65UJFAAAIYRWYUWEXSUCNHQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.786442138 (546h6m29.657525183s) [noise=true , address=145.79.227.20:4438 , piece_id=4SZKGU2LEZ5WK6UTZVQJVYDO7J65K4OT2GKGERVSSTT57W7EECTA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.787226852 (546h6m29.658309887s) [noise=true , address=145.79.227.51:4411 , piece_id=WLJDZHCZIN2YWRBUHWP3WLELR5CBSVBZJBYPTP5MR7VZD4W5EPFA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.794995575 (546h6m29.66607862s) [noise=true , address=145.79.227.9:4367 , piece_id=KEPVDQAHLLDT4WCNQTHRQYDC3VKE3M5VDHGCUS6MCR6Z6ZILMXGQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.997843773 (546h6m29.868926808s) [noise=true , address=45.91.116.9:4263 , piece_id=4MB6SKKVKJNFWWWE24U2YJI25VAF7RLRPRIZQ2QTZXGLWPRSKI4Q, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.018887895 (546h6m29.88997093s) [noise=true , address=45.91.116.7:4010 , piece_id=P47MG5Q5GNZKA5U544TUKNK7PTARAHQAN2CHDMVXEZT42TBNXCJQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.048991647 (546h6m29.920074692s) [noise=true , address=45.91.116.13:4156 , piece_id=4EJIFP2QGBDXAFAX22XGTMLKZZ7MRARIHSIIXO77SCN436ZVGWFA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.057505478 (546h6m29.928588513s) [noise=true , address=77.90.31.14:4662 , piece_id=ALZ4O3NJMIWBZPVLQ3BAFL7KYQVECYUA3JEOLIOJVPRWQ326FLFQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.691813288 (546h6m30.562896333s) [noise=true , address=45.91.116.8:4046 , piece_id=3JVV2WAVRHM5MA7M6GYBDYSTAL6LAMLYZIZD46BE2IQ3OHXV45OQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.749635081 (546h6m30.620718116s) [noise=true , address=45.91.116.14:4191 , piece_id=N75N4G6QH4PAHZPQYFBE5LKCYNLTBS25V6CN2NCRM4FPSSEB75VA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.749656672 (546h6m30.620739717s) [seg_pos={0 0}, upload=478880]: Segment reached optimal threshold of 13 pieces. Dec 28 09:35:30.749916657 (546h6m30.620999702s) [noise=true , address=77.90.31.18:4805 , piece_id=2OFQMO45JG3NG5QAXLM3MJ6UUPZS67LXTZNSTZ2NABRU2V5U346Q, seg_pos={0 0}, upload=478880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 28 09:35:30.749925453 (546h6m30.621008498s) [noise=true , address=77.90.31.18:4805 , piece_id=2OFQMO45JG3NG5QAXLM3MJ6UUPZS67LXTZNSTZ2NABRU2V5U346Q, seg_pos={0 0}, upload=478880]: Long tail cancellation due to optimal threshold Dec 28 09:35:30.750339852 (546h6m30.621422897s) [noise=true , address=45.91.116.10:4083 , piece_id=VD32C5TT6OWORW3U6767NYXRFDULZSOA2PU6CSBTLLC3AQZR2HQA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:36368->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:36368->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36368->45.91.116.10:4083: use of closed network connection Dec 28 09:35:30.750348650 (546h6m30.621431695s) [noise=true , address=45.91.116.10:4083 , piece_id=VD32C5TT6OWORW3U6767NYXRFDULZSOA2PU6CSBTLLC3AQZR2HQA, seg_pos={0 0}, upload=478880]: Long tail cancellation due to optimal threshold Dec 28 09:35:30.750494658 (546h6m30.621577703s) [noise=true , address=145.79.227.44:4195 , piece_id=YNBFGVEO34DUAXVGJGHY3IO6KC6WNET72UONMMPOGNK5DQCIRVSA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 09:35:30.750502983 (546h6m30.621586018s) [noise=true , address=145.79.227.44:4195 , piece_id=YNBFGVEO34DUAXVGJGHY3IO6KC6WNET72UONMMPOGNK5DQCIRVSA, seg_pos={0 0}, upload=478880]: Long tail cancellation due to optimal threshold Dec 28 09:35:30.750523643 (546h6m30.621606688s) [seg_pos={0 0}, upload=478880]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:35:30.750604236 (546h6m30.621687281s) [upload=478880]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:35:30.750610137 (546h6m30.621693172s) [upload=478880]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:35:31.045721700 (546h6m30.916804735s) [upload=478880]: Done uploading segments... Dec 28 09:35:31.045746788 (546h6m30.916829833s) [upload=478880]: Upload finished. err: <nil> Duration: 6.041043492s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [..X.X...X....X.X.....X.....XXX.XX...XX.XX..X...XXX..XX.X...X] duration: 5.000982691s trace: https://jaeger.dev.storj.io/trace/6e14d995906d41c2 FAIL (2025-12-28 09:40:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001022626s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [XX.X.....X......X..............X....X...X......XX...X..X....] duration: 6.021389618s trace: https://jaeger.dev.storj.io/trace/29090502e7ed37c5 FAIL (2025-12-28 09:39:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 4EJIFP2QGBDXAFAX22XGTMLKZZ7MRARIHSIIXO77SCN436ZVGWFA) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: ALZ4O3NJMIWBZPVLQ3BAFL7KYQVECYUA3JEOLIOJVPRWQ326FLFQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 4MB6SKKVKJNFWWWE24U2YJI25VAF7RLRPRIZQ2QTZXGLWPRSKI4Q) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: N75N4G6QH4PAHZPQYFBE5LKCYNLTBS25V6CN2NCRM4FPSSEB75VA) context deadline exceeded ======LOGS====== Dec 28 09:35:27.505217329 (546h6m27.376300374s) [upload=478880]: Starting upload Dec 28 09:35:27.505340463 (546h6m27.376423508s) [upload=478880]: Done starting upload Dec 28 09:35:27.505687754 (546h6m27.376770789s) [upload=478880]: Uploading segments... Dec 28 09:35:27.506519929 (546h6m27.377602964s) [upload=478880]: Got next segment. Inline: false Dec 28 09:35:27.506630178 (546h6m27.377713213s) [upload=478880]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:35:27.506638113 (546h6m27.377721158s) [upload=478880]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:35:27.751540519 (546h6m27.622623554s) [seg_pos={0 0}, upload=478880]: Begin upload segment... Dec 28 09:35:27.751587490 (546h6m27.622670525s) [seg_pos={0 0}, upload=478880]: Done begin upload segment. Dec 28 09:35:27.751618938 (546h6m27.622701985s) [noise=true , address=145.79.227.11:4730 , piece_id=T4GWZTVADA55XZLK6UQ2KP6ITZVMIHYKNA4KXQ2PUKUKLORBI3XQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.751629229 (546h6m27.622712264s) [noise=true , address=77.90.31.14:4662 , piece_id=ALZ4O3NJMIWBZPVLQ3BAFL7KYQVECYUA3JEOLIOJVPRWQ326FLFQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.752546295 (546h6m27.62362934s) [noise=true , address=77.90.31.18:4805 , piece_id=2OFQMO45JG3NG5QAXLM3MJ6UUPZS67LXTZNSTZ2NABRU2V5U346Q, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.753149643 (546h6m27.624232688s) [noise=true , address=45.91.116.8:4046 , piece_id=3JVV2WAVRHM5MA7M6GYBDYSTAL6LAMLYZIZD46BE2IQ3OHXV45OQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.753703628 (546h6m27.624786683s) [noise=true , address=45.91.116.9:4263 , piece_id=4MB6SKKVKJNFWWWE24U2YJI25VAF7RLRPRIZQ2QTZXGLWPRSKI4Q, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754042864 (546h6m27.62512593s) [noise=true , address=145.79.227.45:4230 , piece_id=NONHN4EKNARBD2YIAQ756DW4NAPOS3UMFCEWN4KJ5UEAJEHH4QIQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754144698 (546h6m27.625227733s) [noise=true , address=145.79.227.37:4565 , piece_id=25KGPQZXGUI2ZDEXOYAETSUI6YTFT65UJFAAAIYRWYUWEXSUCNHQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754566932 (546h6m27.625649977s) [noise=true , address=45.91.116.10:4083 , piece_id=VD32C5TT6OWORW3U6767NYXRFDULZSOA2PU6CSBTLLC3AQZR2HQA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754764529 (546h6m27.625847574s) [noise=true , address=45.91.116.14:4191 , piece_id=N75N4G6QH4PAHZPQYFBE5LKCYNLTBS25V6CN2NCRM4FPSSEB75VA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754905227 (546h6m27.625988272s) [noise=true , address=145.79.227.20:4438 , piece_id=4SZKGU2LEZ5WK6UTZVQJVYDO7J65K4OT2GKGERVSSTT57W7EECTA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.755227731 (546h6m27.626310766s) [noise=true , address=45.91.116.7:4010 , piece_id=P47MG5Q5GNZKA5U544TUKNK7PTARAHQAN2CHDMVXEZT42TBNXCJQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.755637491 (546h6m27.626720536s) [noise=true , address=145.79.227.44:4195 , piece_id=YNBFGVEO34DUAXVGJGHY3IO6KC6WNET72UONMMPOGNK5DQCIRVSA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.756059785 (546h6m27.62714282s) [noise=true , address=145.79.227.51:4411 , piece_id=WLJDZHCZIN2YWRBUHWP3WLELR5CBSVBZJBYPTP5MR7VZD4W5EPFA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.756485676 (546h6m27.627568711s) [noise=true , address=45.91.116.13:4156 , piece_id=4EJIFP2QGBDXAFAX22XGTMLKZZ7MRARIHSIIXO77SCN436ZVGWFA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.757098633 (546h6m27.628181668s) [noise=true , address=145.79.227.9:4367 , piece_id=KEPVDQAHLLDT4WCNQTHRQYDC3VKE3M5VDHGCUS6MCR6Z6ZILMXGQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.765253682 (546h6m27.636336727s) [noise=true , address=145.79.227.39:4013 , piece_id=KFXOW2DVPX4LHEHMUCZJBI7KED3SKWZIM53LGK32NZXTZOJKGDGQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:29.480652613 (546h6m29.351735648s) [upload=478880]: Next returned nil segment Dec 28 09:35:29.480666730 (546h6m29.351749765s) [upload=478880]: Waiting for error group managing segments... Dec 28 09:35:29.766845633 (546h6m29.637928678s) [noise=true , address=145.79.227.45:4230 , piece_id=NONHN4EKNARBD2YIAQ756DW4NAPOS3UMFCEWN4KJ5UEAJEHH4QIQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.767303565 (546h6m29.6383866s) [noise=true , address=145.79.227.11:4730 , piece_id=T4GWZTVADA55XZLK6UQ2KP6ITZVMIHYKNA4KXQ2PUKUKLORBI3XQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.779454122 (546h6m29.650537167s) [noise=true , address=145.79.227.39:4013 , piece_id=KFXOW2DVPX4LHEHMUCZJBI7KED3SKWZIM53LGK32NZXTZOJKGDGQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.780555569 (546h6m29.651638614s) [noise=true , address=145.79.227.37:4565 , piece_id=25KGPQZXGUI2ZDEXOYAETSUI6YTFT65UJFAAAIYRWYUWEXSUCNHQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.786442138 (546h6m29.657525183s) [noise=true , address=145.79.227.20:4438 , piece_id=4SZKGU2LEZ5WK6UTZVQJVYDO7J65K4OT2GKGERVSSTT57W7EECTA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.787226852 (546h6m29.658309887s) [noise=true , address=145.79.227.51:4411 , piece_id=WLJDZHCZIN2YWRBUHWP3WLELR5CBSVBZJBYPTP5MR7VZD4W5EPFA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.794995575 (546h6m29.66607862s) [noise=true , address=145.79.227.9:4367 , piece_id=KEPVDQAHLLDT4WCNQTHRQYDC3VKE3M5VDHGCUS6MCR6Z6ZILMXGQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.997843773 (546h6m29.868926808s) [noise=true , address=45.91.116.9:4263 , piece_id=4MB6SKKVKJNFWWWE24U2YJI25VAF7RLRPRIZQ2QTZXGLWPRSKI4Q, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.018887895 (546h6m29.88997093s) [noise=true , address=45.91.116.7:4010 , piece_id=P47MG5Q5GNZKA5U544TUKNK7PTARAHQAN2CHDMVXEZT42TBNXCJQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.048991647 (546h6m29.920074692s) [noise=true , address=45.91.116.13:4156 , piece_id=4EJIFP2QGBDXAFAX22XGTMLKZZ7MRARIHSIIXO77SCN436ZVGWFA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.057505478 (546h6m29.928588513s) [noise=true , address=77.90.31.14:4662 , piece_id=ALZ4O3NJMIWBZPVLQ3BAFL7KYQVECYUA3JEOLIOJVPRWQ326FLFQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.691813288 (546h6m30.562896333s) [noise=true , address=45.91.116.8:4046 , piece_id=3JVV2WAVRHM5MA7M6GYBDYSTAL6LAMLYZIZD46BE2IQ3OHXV45OQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.749635081 (546h6m30.620718116s) [noise=true , address=45.91.116.14:4191 , piece_id=N75N4G6QH4PAHZPQYFBE5LKCYNLTBS25V6CN2NCRM4FPSSEB75VA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.749656672 (546h6m30.620739717s) [seg_pos={0 0}, upload=478880]: Segment reached optimal threshold of 13 pieces. Dec 28 09:35:30.749916657 (546h6m30.620999702s) [noise=true , address=77.90.31.18:4805 , piece_id=2OFQMO45JG3NG5QAXLM3MJ6UUPZS67LXTZNSTZ2NABRU2V5U346Q, seg_pos={0 0}, upload=478880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 28 09:35:30.749925453 (546h6m30.621008498s) [noise=true , address=77.90.31.18:4805 , piece_id=2OFQMO45JG3NG5QAXLM3MJ6UUPZS67LXTZNSTZ2NABRU2V5U346Q, seg_pos={0 0}, upload=478880]: Long tail cancellation due to optimal threshold Dec 28 09:35:30.750339852 (546h6m30.621422897s) [noise=true , address=45.91.116.10:4083 , piece_id=VD32C5TT6OWORW3U6767NYXRFDULZSOA2PU6CSBTLLC3AQZR2HQA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:36368->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:36368->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36368->45.91.116.10:4083: use of closed network connection Dec 28 09:35:30.750348650 (546h6m30.621431695s) [noise=true , address=45.91.116.10:4083 , piece_id=VD32C5TT6OWORW3U6767NYXRFDULZSOA2PU6CSBTLLC3AQZR2HQA, seg_pos={0 0}, upload=478880]: Long tail cancellation due to optimal threshold Dec 28 09:35:30.750494658 (546h6m30.621577703s) [noise=true , address=145.79.227.44:4195 , piece_id=YNBFGVEO34DUAXVGJGHY3IO6KC6WNET72UONMMPOGNK5DQCIRVSA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 09:35:30.750502983 (546h6m30.621586018s) [noise=true , address=145.79.227.44:4195 , piece_id=YNBFGVEO34DUAXVGJGHY3IO6KC6WNET72UONMMPOGNK5DQCIRVSA, seg_pos={0 0}, upload=478880]: Long tail cancellation due to optimal threshold Dec 28 09:35:30.750523643 (546h6m30.621606688s) [seg_pos={0 0}, upload=478880]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:35:30.750604236 (546h6m30.621687281s) [upload=478880]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:35:30.750610137 (546h6m30.621693172s) [upload=478880]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:35:31.045721700 (546h6m30.916804735s) [upload=478880]: Done uploading segments... Dec 28 09:35:31.045746788 (546h6m30.916829833s) [upload=478880]: Upload finished. err: <nil> Duration: 6.021423263s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [...X.X...X....X.X.....X.....XXX.XX...XX.XX..X...XXX..XX.X...] duration: 4.076496068s trace: https://jaeger.dev.storj.io/trace/629a316cbadc86f8 FAIL (2025-12-28 09:39:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.076527519s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.XX.X.....X......X..............X....X...X......XX...X..X...] duration: 5.820129895s trace: https://jaeger.dev.storj.io/trace/29090502e7ed37c5 FAIL (2025-12-28 09:38:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 4EJIFP2QGBDXAFAX22XGTMLKZZ7MRARIHSIIXO77SCN436ZVGWFA) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: ALZ4O3NJMIWBZPVLQ3BAFL7KYQVECYUA3JEOLIOJVPRWQ326FLFQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 4MB6SKKVKJNFWWWE24U2YJI25VAF7RLRPRIZQ2QTZXGLWPRSKI4Q) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: N75N4G6QH4PAHZPQYFBE5LKCYNLTBS25V6CN2NCRM4FPSSEB75VA) context deadline exceeded ======LOGS====== Dec 28 09:35:27.505217329 (546h6m27.376300374s) [upload=478880]: Starting upload Dec 28 09:35:27.505340463 (546h6m27.376423508s) [upload=478880]: Done starting upload Dec 28 09:35:27.505687754 (546h6m27.376770789s) [upload=478880]: Uploading segments... Dec 28 09:35:27.506519929 (546h6m27.377602964s) [upload=478880]: Got next segment. Inline: false Dec 28 09:35:27.506630178 (546h6m27.377713213s) [upload=478880]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:35:27.506638113 (546h6m27.377721158s) [upload=478880]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:35:27.751540519 (546h6m27.622623554s) [seg_pos={0 0}, upload=478880]: Begin upload segment... Dec 28 09:35:27.751587490 (546h6m27.622670525s) [seg_pos={0 0}, upload=478880]: Done begin upload segment. Dec 28 09:35:27.751618938 (546h6m27.622701985s) [noise=true , address=145.79.227.11:4730 , piece_id=T4GWZTVADA55XZLK6UQ2KP6ITZVMIHYKNA4KXQ2PUKUKLORBI3XQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.751629229 (546h6m27.622712264s) [noise=true , address=77.90.31.14:4662 , piece_id=ALZ4O3NJMIWBZPVLQ3BAFL7KYQVECYUA3JEOLIOJVPRWQ326FLFQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.752546295 (546h6m27.62362934s) [noise=true , address=77.90.31.18:4805 , piece_id=2OFQMO45JG3NG5QAXLM3MJ6UUPZS67LXTZNSTZ2NABRU2V5U346Q, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.753149643 (546h6m27.624232688s) [noise=true , address=45.91.116.8:4046 , piece_id=3JVV2WAVRHM5MA7M6GYBDYSTAL6LAMLYZIZD46BE2IQ3OHXV45OQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.753703628 (546h6m27.624786683s) [noise=true , address=45.91.116.9:4263 , piece_id=4MB6SKKVKJNFWWWE24U2YJI25VAF7RLRPRIZQ2QTZXGLWPRSKI4Q, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754042864 (546h6m27.62512593s) [noise=true , address=145.79.227.45:4230 , piece_id=NONHN4EKNARBD2YIAQ756DW4NAPOS3UMFCEWN4KJ5UEAJEHH4QIQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754144698 (546h6m27.625227733s) [noise=true , address=145.79.227.37:4565 , piece_id=25KGPQZXGUI2ZDEXOYAETSUI6YTFT65UJFAAAIYRWYUWEXSUCNHQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754566932 (546h6m27.625649977s) [noise=true , address=45.91.116.10:4083 , piece_id=VD32C5TT6OWORW3U6767NYXRFDULZSOA2PU6CSBTLLC3AQZR2HQA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754764529 (546h6m27.625847574s) [noise=true , address=45.91.116.14:4191 , piece_id=N75N4G6QH4PAHZPQYFBE5LKCYNLTBS25V6CN2NCRM4FPSSEB75VA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754905227 (546h6m27.625988272s) [noise=true , address=145.79.227.20:4438 , piece_id=4SZKGU2LEZ5WK6UTZVQJVYDO7J65K4OT2GKGERVSSTT57W7EECTA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.755227731 (546h6m27.626310766s) [noise=true , address=45.91.116.7:4010 , piece_id=P47MG5Q5GNZKA5U544TUKNK7PTARAHQAN2CHDMVXEZT42TBNXCJQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.755637491 (546h6m27.626720536s) [noise=true , address=145.79.227.44:4195 , piece_id=YNBFGVEO34DUAXVGJGHY3IO6KC6WNET72UONMMPOGNK5DQCIRVSA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.756059785 (546h6m27.62714282s) [noise=true , address=145.79.227.51:4411 , piece_id=WLJDZHCZIN2YWRBUHWP3WLELR5CBSVBZJBYPTP5MR7VZD4W5EPFA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.756485676 (546h6m27.627568711s) [noise=true , address=45.91.116.13:4156 , piece_id=4EJIFP2QGBDXAFAX22XGTMLKZZ7MRARIHSIIXO77SCN436ZVGWFA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.757098633 (546h6m27.628181668s) [noise=true , address=145.79.227.9:4367 , piece_id=KEPVDQAHLLDT4WCNQTHRQYDC3VKE3M5VDHGCUS6MCR6Z6ZILMXGQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.765253682 (546h6m27.636336727s) [noise=true , address=145.79.227.39:4013 , piece_id=KFXOW2DVPX4LHEHMUCZJBI7KED3SKWZIM53LGK32NZXTZOJKGDGQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:29.480652613 (546h6m29.351735648s) [upload=478880]: Next returned nil segment Dec 28 09:35:29.480666730 (546h6m29.351749765s) [upload=478880]: Waiting for error group managing segments... Dec 28 09:35:29.766845633 (546h6m29.637928678s) [noise=true , address=145.79.227.45:4230 , piece_id=NONHN4EKNARBD2YIAQ756DW4NAPOS3UMFCEWN4KJ5UEAJEHH4QIQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.767303565 (546h6m29.6383866s) [noise=true , address=145.79.227.11:4730 , piece_id=T4GWZTVADA55XZLK6UQ2KP6ITZVMIHYKNA4KXQ2PUKUKLORBI3XQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.779454122 (546h6m29.650537167s) [noise=true , address=145.79.227.39:4013 , piece_id=KFXOW2DVPX4LHEHMUCZJBI7KED3SKWZIM53LGK32NZXTZOJKGDGQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.780555569 (546h6m29.651638614s) [noise=true , address=145.79.227.37:4565 , piece_id=25KGPQZXGUI2ZDEXOYAETSUI6YTFT65UJFAAAIYRWYUWEXSUCNHQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.786442138 (546h6m29.657525183s) [noise=true , address=145.79.227.20:4438 , piece_id=4SZKGU2LEZ5WK6UTZVQJVYDO7J65K4OT2GKGERVSSTT57W7EECTA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.787226852 (546h6m29.658309887s) [noise=true , address=145.79.227.51:4411 , piece_id=WLJDZHCZIN2YWRBUHWP3WLELR5CBSVBZJBYPTP5MR7VZD4W5EPFA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.794995575 (546h6m29.66607862s) [noise=true , address=145.79.227.9:4367 , piece_id=KEPVDQAHLLDT4WCNQTHRQYDC3VKE3M5VDHGCUS6MCR6Z6ZILMXGQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.997843773 (546h6m29.868926808s) [noise=true , address=45.91.116.9:4263 , piece_id=4MB6SKKVKJNFWWWE24U2YJI25VAF7RLRPRIZQ2QTZXGLWPRSKI4Q, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.018887895 (546h6m29.88997093s) [noise=true , address=45.91.116.7:4010 , piece_id=P47MG5Q5GNZKA5U544TUKNK7PTARAHQAN2CHDMVXEZT42TBNXCJQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.048991647 (546h6m29.920074692s) [noise=true , address=45.91.116.13:4156 , piece_id=4EJIFP2QGBDXAFAX22XGTMLKZZ7MRARIHSIIXO77SCN436ZVGWFA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.057505478 (546h6m29.928588513s) [noise=true , address=77.90.31.14:4662 , piece_id=ALZ4O3NJMIWBZPVLQ3BAFL7KYQVECYUA3JEOLIOJVPRWQ326FLFQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.691813288 (546h6m30.562896333s) [noise=true , address=45.91.116.8:4046 , piece_id=3JVV2WAVRHM5MA7M6GYBDYSTAL6LAMLYZIZD46BE2IQ3OHXV45OQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.749635081 (546h6m30.620718116s) [noise=true , address=45.91.116.14:4191 , piece_id=N75N4G6QH4PAHZPQYFBE5LKCYNLTBS25V6CN2NCRM4FPSSEB75VA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.749656672 (546h6m30.620739717s) [seg_pos={0 0}, upload=478880]: Segment reached optimal threshold of 13 pieces. Dec 28 09:35:30.749916657 (546h6m30.620999702s) [noise=true , address=77.90.31.18:4805 , piece_id=2OFQMO45JG3NG5QAXLM3MJ6UUPZS67LXTZNSTZ2NABRU2V5U346Q, seg_pos={0 0}, upload=478880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 28 09:35:30.749925453 (546h6m30.621008498s) [noise=true , address=77.90.31.18:4805 , piece_id=2OFQMO45JG3NG5QAXLM3MJ6UUPZS67LXTZNSTZ2NABRU2V5U346Q, seg_pos={0 0}, upload=478880]: Long tail cancellation due to optimal threshold Dec 28 09:35:30.750339852 (546h6m30.621422897s) [noise=true , address=45.91.116.10:4083 , piece_id=VD32C5TT6OWORW3U6767NYXRFDULZSOA2PU6CSBTLLC3AQZR2HQA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:36368->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:36368->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36368->45.91.116.10:4083: use of closed network connection Dec 28 09:35:30.750348650 (546h6m30.621431695s) [noise=true , address=45.91.116.10:4083 , piece_id=VD32C5TT6OWORW3U6767NYXRFDULZSOA2PU6CSBTLLC3AQZR2HQA, seg_pos={0 0}, upload=478880]: Long tail cancellation due to optimal threshold Dec 28 09:35:30.750494658 (546h6m30.621577703s) [noise=true , address=145.79.227.44:4195 , piece_id=YNBFGVEO34DUAXVGJGHY3IO6KC6WNET72UONMMPOGNK5DQCIRVSA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 09:35:30.750502983 (546h6m30.621586018s) [noise=true , address=145.79.227.44:4195 , piece_id=YNBFGVEO34DUAXVGJGHY3IO6KC6WNET72UONMMPOGNK5DQCIRVSA, seg_pos={0 0}, upload=478880]: Long tail cancellation due to optimal threshold Dec 28 09:35:30.750523643 (546h6m30.621606688s) [seg_pos={0 0}, upload=478880]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:35:30.750604236 (546h6m30.621687281s) [upload=478880]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:35:30.750610137 (546h6m30.621693172s) [upload=478880]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:35:31.045721700 (546h6m30.916804735s) [upload=478880]: Done uploading segments... Dec 28 09:35:31.045746788 (546h6m30.916829833s) [upload=478880]: Upload finished. err: <nil> Duration: 5.820149363s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [....X.X...X....X.X.....X.....XXX.XX...XX.XX..X...XXX..XX.X..] duration: 3.417903469s trace: https://jaeger.dev.storj.io/trace/629a316cbadc86f8 FAIL (2025-12-28 09:38:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.417928626s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [.....X.X...X....X.X.....X.....XXX.XX...XX.XX..X...XXX..XX.X.] duration: 3.498994499s trace: https://jaeger.dev.storj.io/trace/629a316cbadc86f8 FAIL (2025-12-28 09:37:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.499027132s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..XX.X.....X......X..............X....X...X......XX...X..X..] duration: 5.64964303s trace: https://jaeger.dev.storj.io/trace/29090502e7ed37c5 FAIL (2025-12-28 09:37:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 4EJIFP2QGBDXAFAX22XGTMLKZZ7MRARIHSIIXO77SCN436ZVGWFA) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: ALZ4O3NJMIWBZPVLQ3BAFL7KYQVECYUA3JEOLIOJVPRWQ326FLFQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 4MB6SKKVKJNFWWWE24U2YJI25VAF7RLRPRIZQ2QTZXGLWPRSKI4Q) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: N75N4G6QH4PAHZPQYFBE5LKCYNLTBS25V6CN2NCRM4FPSSEB75VA) context deadline exceeded ======LOGS====== Dec 28 09:35:27.505217329 (546h6m27.376300374s) [upload=478880]: Starting upload Dec 28 09:35:27.505340463 (546h6m27.376423508s) [upload=478880]: Done starting upload Dec 28 09:35:27.505687754 (546h6m27.376770789s) [upload=478880]: Uploading segments... Dec 28 09:35:27.506519929 (546h6m27.377602964s) [upload=478880]: Got next segment. Inline: false Dec 28 09:35:27.506630178 (546h6m27.377713213s) [upload=478880]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:35:27.506638113 (546h6m27.377721158s) [upload=478880]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:35:27.751540519 (546h6m27.622623554s) [seg_pos={0 0}, upload=478880]: Begin upload segment... Dec 28 09:35:27.751587490 (546h6m27.622670525s) [seg_pos={0 0}, upload=478880]: Done begin upload segment. Dec 28 09:35:27.751618938 (546h6m27.622701985s) [noise=true , address=145.79.227.11:4730 , piece_id=T4GWZTVADA55XZLK6UQ2KP6ITZVMIHYKNA4KXQ2PUKUKLORBI3XQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.751629229 (546h6m27.622712264s) [noise=true , address=77.90.31.14:4662 , piece_id=ALZ4O3NJMIWBZPVLQ3BAFL7KYQVECYUA3JEOLIOJVPRWQ326FLFQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.752546295 (546h6m27.62362934s) [noise=true , address=77.90.31.18:4805 , piece_id=2OFQMO45JG3NG5QAXLM3MJ6UUPZS67LXTZNSTZ2NABRU2V5U346Q, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.753149643 (546h6m27.624232688s) [noise=true , address=45.91.116.8:4046 , piece_id=3JVV2WAVRHM5MA7M6GYBDYSTAL6LAMLYZIZD46BE2IQ3OHXV45OQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.753703628 (546h6m27.624786683s) [noise=true , address=45.91.116.9:4263 , piece_id=4MB6SKKVKJNFWWWE24U2YJI25VAF7RLRPRIZQ2QTZXGLWPRSKI4Q, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754042864 (546h6m27.62512593s) [noise=true , address=145.79.227.45:4230 , piece_id=NONHN4EKNARBD2YIAQ756DW4NAPOS3UMFCEWN4KJ5UEAJEHH4QIQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754144698 (546h6m27.625227733s) [noise=true , address=145.79.227.37:4565 , piece_id=25KGPQZXGUI2ZDEXOYAETSUI6YTFT65UJFAAAIYRWYUWEXSUCNHQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754566932 (546h6m27.625649977s) [noise=true , address=45.91.116.10:4083 , piece_id=VD32C5TT6OWORW3U6767NYXRFDULZSOA2PU6CSBTLLC3AQZR2HQA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754764529 (546h6m27.625847574s) [noise=true , address=45.91.116.14:4191 , piece_id=N75N4G6QH4PAHZPQYFBE5LKCYNLTBS25V6CN2NCRM4FPSSEB75VA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754905227 (546h6m27.625988272s) [noise=true , address=145.79.227.20:4438 , piece_id=4SZKGU2LEZ5WK6UTZVQJVYDO7J65K4OT2GKGERVSSTT57W7EECTA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.755227731 (546h6m27.626310766s) [noise=true , address=45.91.116.7:4010 , piece_id=P47MG5Q5GNZKA5U544TUKNK7PTARAHQAN2CHDMVXEZT42TBNXCJQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.755637491 (546h6m27.626720536s) [noise=true , address=145.79.227.44:4195 , piece_id=YNBFGVEO34DUAXVGJGHY3IO6KC6WNET72UONMMPOGNK5DQCIRVSA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.756059785 (546h6m27.62714282s) [noise=true , address=145.79.227.51:4411 , piece_id=WLJDZHCZIN2YWRBUHWP3WLELR5CBSVBZJBYPTP5MR7VZD4W5EPFA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.756485676 (546h6m27.627568711s) [noise=true , address=45.91.116.13:4156 , piece_id=4EJIFP2QGBDXAFAX22XGTMLKZZ7MRARIHSIIXO77SCN436ZVGWFA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.757098633 (546h6m27.628181668s) [noise=true , address=145.79.227.9:4367 , piece_id=KEPVDQAHLLDT4WCNQTHRQYDC3VKE3M5VDHGCUS6MCR6Z6ZILMXGQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.765253682 (546h6m27.636336727s) [noise=true , address=145.79.227.39:4013 , piece_id=KFXOW2DVPX4LHEHMUCZJBI7KED3SKWZIM53LGK32NZXTZOJKGDGQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:29.480652613 (546h6m29.351735648s) [upload=478880]: Next returned nil segment Dec 28 09:35:29.480666730 (546h6m29.351749765s) [upload=478880]: Waiting for error group managing segments... Dec 28 09:35:29.766845633 (546h6m29.637928678s) [noise=true , address=145.79.227.45:4230 , piece_id=NONHN4EKNARBD2YIAQ756DW4NAPOS3UMFCEWN4KJ5UEAJEHH4QIQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.767303565 (546h6m29.6383866s) [noise=true , address=145.79.227.11:4730 , piece_id=T4GWZTVADA55XZLK6UQ2KP6ITZVMIHYKNA4KXQ2PUKUKLORBI3XQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.779454122 (546h6m29.650537167s) [noise=true , address=145.79.227.39:4013 , piece_id=KFXOW2DVPX4LHEHMUCZJBI7KED3SKWZIM53LGK32NZXTZOJKGDGQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.780555569 (546h6m29.651638614s) [noise=true , address=145.79.227.37:4565 , piece_id=25KGPQZXGUI2ZDEXOYAETSUI6YTFT65UJFAAAIYRWYUWEXSUCNHQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.786442138 (546h6m29.657525183s) [noise=true , address=145.79.227.20:4438 , piece_id=4SZKGU2LEZ5WK6UTZVQJVYDO7J65K4OT2GKGERVSSTT57W7EECTA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.787226852 (546h6m29.658309887s) [noise=true , address=145.79.227.51:4411 , piece_id=WLJDZHCZIN2YWRBUHWP3WLELR5CBSVBZJBYPTP5MR7VZD4W5EPFA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.794995575 (546h6m29.66607862s) [noise=true , address=145.79.227.9:4367 , piece_id=KEPVDQAHLLDT4WCNQTHRQYDC3VKE3M5VDHGCUS6MCR6Z6ZILMXGQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.997843773 (546h6m29.868926808s) [noise=true , address=45.91.116.9:4263 , piece_id=4MB6SKKVKJNFWWWE24U2YJI25VAF7RLRPRIZQ2QTZXGLWPRSKI4Q, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.018887895 (546h6m29.88997093s) [noise=true , address=45.91.116.7:4010 , piece_id=P47MG5Q5GNZKA5U544TUKNK7PTARAHQAN2CHDMVXEZT42TBNXCJQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.048991647 (546h6m29.920074692s) [noise=true , address=45.91.116.13:4156 , piece_id=4EJIFP2QGBDXAFAX22XGTMLKZZ7MRARIHSIIXO77SCN436ZVGWFA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.057505478 (546h6m29.928588513s) [noise=true , address=77.90.31.14:4662 , piece_id=ALZ4O3NJMIWBZPVLQ3BAFL7KYQVECYUA3JEOLIOJVPRWQ326FLFQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.691813288 (546h6m30.562896333s) [noise=true , address=45.91.116.8:4046 , piece_id=3JVV2WAVRHM5MA7M6GYBDYSTAL6LAMLYZIZD46BE2IQ3OHXV45OQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.749635081 (546h6m30.620718116s) [noise=true , address=45.91.116.14:4191 , piece_id=N75N4G6QH4PAHZPQYFBE5LKCYNLTBS25V6CN2NCRM4FPSSEB75VA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.749656672 (546h6m30.620739717s) [seg_pos={0 0}, upload=478880]: Segment reached optimal threshold of 13 pieces. Dec 28 09:35:30.749916657 (546h6m30.620999702s) [noise=true , address=77.90.31.18:4805 , piece_id=2OFQMO45JG3NG5QAXLM3MJ6UUPZS67LXTZNSTZ2NABRU2V5U346Q, seg_pos={0 0}, upload=478880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 28 09:35:30.749925453 (546h6m30.621008498s) [noise=true , address=77.90.31.18:4805 , piece_id=2OFQMO45JG3NG5QAXLM3MJ6UUPZS67LXTZNSTZ2NABRU2V5U346Q, seg_pos={0 0}, upload=478880]: Long tail cancellation due to optimal threshold Dec 28 09:35:30.750339852 (546h6m30.621422897s) [noise=true , address=45.91.116.10:4083 , piece_id=VD32C5TT6OWORW3U6767NYXRFDULZSOA2PU6CSBTLLC3AQZR2HQA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:36368->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:36368->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36368->45.91.116.10:4083: use of closed network connection Dec 28 09:35:30.750348650 (546h6m30.621431695s) [noise=true , address=45.91.116.10:4083 , piece_id=VD32C5TT6OWORW3U6767NYXRFDULZSOA2PU6CSBTLLC3AQZR2HQA, seg_pos={0 0}, upload=478880]: Long tail cancellation due to optimal threshold Dec 28 09:35:30.750494658 (546h6m30.621577703s) [noise=true , address=145.79.227.44:4195 , piece_id=YNBFGVEO34DUAXVGJGHY3IO6KC6WNET72UONMMPOGNK5DQCIRVSA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 09:35:30.750502983 (546h6m30.621586018s) [noise=true , address=145.79.227.44:4195 , piece_id=YNBFGVEO34DUAXVGJGHY3IO6KC6WNET72UONMMPOGNK5DQCIRVSA, seg_pos={0 0}, upload=478880]: Long tail cancellation due to optimal threshold Dec 28 09:35:30.750523643 (546h6m30.621606688s) [seg_pos={0 0}, upload=478880]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:35:30.750604236 (546h6m30.621687281s) [upload=478880]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:35:30.750610137 (546h6m30.621693172s) [upload=478880]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:35:31.045721700 (546h6m30.916804735s) [upload=478880]: Done uploading segments... Dec 28 09:35:31.045746788 (546h6m30.916829833s) [upload=478880]: Upload finished. err: <nil> Duration: 5.64966931s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [......X.X...X....X.X.....X.....XXX.XX...XX.XX..X...XXX..XX.X] duration: 5.001554671s trace: https://jaeger.dev.storj.io/trace/629a316cbadc86f8 FAIL (2025-12-28 09:37:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001579257s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...XX.X.....X......X..............X....X...X......XX...X..X.] duration: 5.725748287s trace: https://jaeger.dev.storj.io/trace/29090502e7ed37c5 FAIL (2025-12-28 09:36:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 4EJIFP2QGBDXAFAX22XGTMLKZZ7MRARIHSIIXO77SCN436ZVGWFA) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: ALZ4O3NJMIWBZPVLQ3BAFL7KYQVECYUA3JEOLIOJVPRWQ326FLFQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 4MB6SKKVKJNFWWWE24U2YJI25VAF7RLRPRIZQ2QTZXGLWPRSKI4Q) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: N75N4G6QH4PAHZPQYFBE5LKCYNLTBS25V6CN2NCRM4FPSSEB75VA) context deadline exceeded ======LOGS====== Dec 28 09:35:27.505217329 (546h6m27.376300374s) [upload=478880]: Starting upload Dec 28 09:35:27.505340463 (546h6m27.376423508s) [upload=478880]: Done starting upload Dec 28 09:35:27.505687754 (546h6m27.376770789s) [upload=478880]: Uploading segments... Dec 28 09:35:27.506519929 (546h6m27.377602964s) [upload=478880]: Got next segment. Inline: false Dec 28 09:35:27.506630178 (546h6m27.377713213s) [upload=478880]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:35:27.506638113 (546h6m27.377721158s) [upload=478880]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:35:27.751540519 (546h6m27.622623554s) [seg_pos={0 0}, upload=478880]: Begin upload segment... Dec 28 09:35:27.751587490 (546h6m27.622670525s) [seg_pos={0 0}, upload=478880]: Done begin upload segment. Dec 28 09:35:27.751618938 (546h6m27.622701985s) [noise=true , address=145.79.227.11:4730 , piece_id=T4GWZTVADA55XZLK6UQ2KP6ITZVMIHYKNA4KXQ2PUKUKLORBI3XQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.751629229 (546h6m27.622712264s) [noise=true , address=77.90.31.14:4662 , piece_id=ALZ4O3NJMIWBZPVLQ3BAFL7KYQVECYUA3JEOLIOJVPRWQ326FLFQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.752546295 (546h6m27.62362934s) [noise=true , address=77.90.31.18:4805 , piece_id=2OFQMO45JG3NG5QAXLM3MJ6UUPZS67LXTZNSTZ2NABRU2V5U346Q, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.753149643 (546h6m27.624232688s) [noise=true , address=45.91.116.8:4046 , piece_id=3JVV2WAVRHM5MA7M6GYBDYSTAL6LAMLYZIZD46BE2IQ3OHXV45OQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.753703628 (546h6m27.624786683s) [noise=true , address=45.91.116.9:4263 , piece_id=4MB6SKKVKJNFWWWE24U2YJI25VAF7RLRPRIZQ2QTZXGLWPRSKI4Q, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754042864 (546h6m27.62512593s) [noise=true , address=145.79.227.45:4230 , piece_id=NONHN4EKNARBD2YIAQ756DW4NAPOS3UMFCEWN4KJ5UEAJEHH4QIQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754144698 (546h6m27.625227733s) [noise=true , address=145.79.227.37:4565 , piece_id=25KGPQZXGUI2ZDEXOYAETSUI6YTFT65UJFAAAIYRWYUWEXSUCNHQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754566932 (546h6m27.625649977s) [noise=true , address=45.91.116.10:4083 , piece_id=VD32C5TT6OWORW3U6767NYXRFDULZSOA2PU6CSBTLLC3AQZR2HQA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754764529 (546h6m27.625847574s) [noise=true , address=45.91.116.14:4191 , piece_id=N75N4G6QH4PAHZPQYFBE5LKCYNLTBS25V6CN2NCRM4FPSSEB75VA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754905227 (546h6m27.625988272s) [noise=true , address=145.79.227.20:4438 , piece_id=4SZKGU2LEZ5WK6UTZVQJVYDO7J65K4OT2GKGERVSSTT57W7EECTA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.755227731 (546h6m27.626310766s) [noise=true , address=45.91.116.7:4010 , piece_id=P47MG5Q5GNZKA5U544TUKNK7PTARAHQAN2CHDMVXEZT42TBNXCJQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.755637491 (546h6m27.626720536s) [noise=true , address=145.79.227.44:4195 , piece_id=YNBFGVEO34DUAXVGJGHY3IO6KC6WNET72UONMMPOGNK5DQCIRVSA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.756059785 (546h6m27.62714282s) [noise=true , address=145.79.227.51:4411 , piece_id=WLJDZHCZIN2YWRBUHWP3WLELR5CBSVBZJBYPTP5MR7VZD4W5EPFA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.756485676 (546h6m27.627568711s) [noise=true , address=45.91.116.13:4156 , piece_id=4EJIFP2QGBDXAFAX22XGTMLKZZ7MRARIHSIIXO77SCN436ZVGWFA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.757098633 (546h6m27.628181668s) [noise=true , address=145.79.227.9:4367 , piece_id=KEPVDQAHLLDT4WCNQTHRQYDC3VKE3M5VDHGCUS6MCR6Z6ZILMXGQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.765253682 (546h6m27.636336727s) [noise=true , address=145.79.227.39:4013 , piece_id=KFXOW2DVPX4LHEHMUCZJBI7KED3SKWZIM53LGK32NZXTZOJKGDGQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:29.480652613 (546h6m29.351735648s) [upload=478880]: Next returned nil segment Dec 28 09:35:29.480666730 (546h6m29.351749765s) [upload=478880]: Waiting for error group managing segments... Dec 28 09:35:29.766845633 (546h6m29.637928678s) [noise=true , address=145.79.227.45:4230 , piece_id=NONHN4EKNARBD2YIAQ756DW4NAPOS3UMFCEWN4KJ5UEAJEHH4QIQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.767303565 (546h6m29.6383866s) [noise=true , address=145.79.227.11:4730 , piece_id=T4GWZTVADA55XZLK6UQ2KP6ITZVMIHYKNA4KXQ2PUKUKLORBI3XQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.779454122 (546h6m29.650537167s) [noise=true , address=145.79.227.39:4013 , piece_id=KFXOW2DVPX4LHEHMUCZJBI7KED3SKWZIM53LGK32NZXTZOJKGDGQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.780555569 (546h6m29.651638614s) [noise=true , address=145.79.227.37:4565 , piece_id=25KGPQZXGUI2ZDEXOYAETSUI6YTFT65UJFAAAIYRWYUWEXSUCNHQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.786442138 (546h6m29.657525183s) [noise=true , address=145.79.227.20:4438 , piece_id=4SZKGU2LEZ5WK6UTZVQJVYDO7J65K4OT2GKGERVSSTT57W7EECTA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.787226852 (546h6m29.658309887s) [noise=true , address=145.79.227.51:4411 , piece_id=WLJDZHCZIN2YWRBUHWP3WLELR5CBSVBZJBYPTP5MR7VZD4W5EPFA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.794995575 (546h6m29.66607862s) [noise=true , address=145.79.227.9:4367 , piece_id=KEPVDQAHLLDT4WCNQTHRQYDC3VKE3M5VDHGCUS6MCR6Z6ZILMXGQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.997843773 (546h6m29.868926808s) [noise=true , address=45.91.116.9:4263 , piece_id=4MB6SKKVKJNFWWWE24U2YJI25VAF7RLRPRIZQ2QTZXGLWPRSKI4Q, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.018887895 (546h6m29.88997093s) [noise=true , address=45.91.116.7:4010 , piece_id=P47MG5Q5GNZKA5U544TUKNK7PTARAHQAN2CHDMVXEZT42TBNXCJQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.048991647 (546h6m29.920074692s) [noise=true , address=45.91.116.13:4156 , piece_id=4EJIFP2QGBDXAFAX22XGTMLKZZ7MRARIHSIIXO77SCN436ZVGWFA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.057505478 (546h6m29.928588513s) [noise=true , address=77.90.31.14:4662 , piece_id=ALZ4O3NJMIWBZPVLQ3BAFL7KYQVECYUA3JEOLIOJVPRWQ326FLFQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.691813288 (546h6m30.562896333s) [noise=true , address=45.91.116.8:4046 , piece_id=3JVV2WAVRHM5MA7M6GYBDYSTAL6LAMLYZIZD46BE2IQ3OHXV45OQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.749635081 (546h6m30.620718116s) [noise=true , address=45.91.116.14:4191 , piece_id=N75N4G6QH4PAHZPQYFBE5LKCYNLTBS25V6CN2NCRM4FPSSEB75VA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.749656672 (546h6m30.620739717s) [seg_pos={0 0}, upload=478880]: Segment reached optimal threshold of 13 pieces. Dec 28 09:35:30.749916657 (546h6m30.620999702s) [noise=true , address=77.90.31.18:4805 , piece_id=2OFQMO45JG3NG5QAXLM3MJ6UUPZS67LXTZNSTZ2NABRU2V5U346Q, seg_pos={0 0}, upload=478880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 28 09:35:30.749925453 (546h6m30.621008498s) [noise=true , address=77.90.31.18:4805 , piece_id=2OFQMO45JG3NG5QAXLM3MJ6UUPZS67LXTZNSTZ2NABRU2V5U346Q, seg_pos={0 0}, upload=478880]: Long tail cancellation due to optimal threshold Dec 28 09:35:30.750339852 (546h6m30.621422897s) [noise=true , address=45.91.116.10:4083 , piece_id=VD32C5TT6OWORW3U6767NYXRFDULZSOA2PU6CSBTLLC3AQZR2HQA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:36368->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:36368->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36368->45.91.116.10:4083: use of closed network connection Dec 28 09:35:30.750348650 (546h6m30.621431695s) [noise=true , address=45.91.116.10:4083 , piece_id=VD32C5TT6OWORW3U6767NYXRFDULZSOA2PU6CSBTLLC3AQZR2HQA, seg_pos={0 0}, upload=478880]: Long tail cancellation due to optimal threshold Dec 28 09:35:30.750494658 (546h6m30.621577703s) [noise=true , address=145.79.227.44:4195 , piece_id=YNBFGVEO34DUAXVGJGHY3IO6KC6WNET72UONMMPOGNK5DQCIRVSA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 09:35:30.750502983 (546h6m30.621586018s) [noise=true , address=145.79.227.44:4195 , piece_id=YNBFGVEO34DUAXVGJGHY3IO6KC6WNET72UONMMPOGNK5DQCIRVSA, seg_pos={0 0}, upload=478880]: Long tail cancellation due to optimal threshold Dec 28 09:35:30.750523643 (546h6m30.621606688s) [seg_pos={0 0}, upload=478880]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:35:30.750604236 (546h6m30.621687281s) [upload=478880]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:35:30.750610137 (546h6m30.621693172s) [upload=478880]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:35:31.045721700 (546h6m30.916804735s) [upload=478880]: Done uploading segments... Dec 28 09:35:31.045746788 (546h6m30.916829833s) [upload=478880]: Upload finished. err: <nil> Duration: 5.725773013s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [.......X.X...X....X.X.....X.....XXX.XX...XX.XX..X...XXX..XX.] duration: 5.001301549s trace: https://jaeger.dev.storj.io/trace/6c2381cf43b7de34 FAIL (2025-12-28 09:36:28 UTC) orbiter warning: context deadline exceeded Duration: 5.00137078s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [....XX.X.....X......X..............X....X...X......XX...X..X] duration: 30.002988112s trace: https://jaeger.dev.storj.io/trace/29090502e7ed37c5 FAIL (2025-12-28 09:35:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 4EJIFP2QGBDXAFAX22XGTMLKZZ7MRARIHSIIXO77SCN436ZVGWFA) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: ALZ4O3NJMIWBZPVLQ3BAFL7KYQVECYUA3JEOLIOJVPRWQ326FLFQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 4MB6SKKVKJNFWWWE24U2YJI25VAF7RLRPRIZQ2QTZXGLWPRSKI4Q) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: N75N4G6QH4PAHZPQYFBE5LKCYNLTBS25V6CN2NCRM4FPSSEB75VA) context deadline exceeded ======LOGS====== Dec 28 09:35:27.505217329 (546h6m27.376300374s) [upload=478880]: Starting upload Dec 28 09:35:27.505340463 (546h6m27.376423508s) [upload=478880]: Done starting upload Dec 28 09:35:27.505687754 (546h6m27.376770789s) [upload=478880]: Uploading segments... Dec 28 09:35:27.506519929 (546h6m27.377602964s) [upload=478880]: Got next segment. Inline: false Dec 28 09:35:27.506630178 (546h6m27.377713213s) [upload=478880]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:35:27.506638113 (546h6m27.377721158s) [upload=478880]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:35:27.751540519 (546h6m27.622623554s) [seg_pos={0 0}, upload=478880]: Begin upload segment... Dec 28 09:35:27.751587490 (546h6m27.622670525s) [seg_pos={0 0}, upload=478880]: Done begin upload segment. Dec 28 09:35:27.751618938 (546h6m27.622701985s) [noise=true , address=145.79.227.11:4730 , piece_id=T4GWZTVADA55XZLK6UQ2KP6ITZVMIHYKNA4KXQ2PUKUKLORBI3XQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.751629229 (546h6m27.622712264s) [noise=true , address=77.90.31.14:4662 , piece_id=ALZ4O3NJMIWBZPVLQ3BAFL7KYQVECYUA3JEOLIOJVPRWQ326FLFQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.752546295 (546h6m27.62362934s) [noise=true , address=77.90.31.18:4805 , piece_id=2OFQMO45JG3NG5QAXLM3MJ6UUPZS67LXTZNSTZ2NABRU2V5U346Q, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.753149643 (546h6m27.624232688s) [noise=true , address=45.91.116.8:4046 , piece_id=3JVV2WAVRHM5MA7M6GYBDYSTAL6LAMLYZIZD46BE2IQ3OHXV45OQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.753703628 (546h6m27.624786683s) [noise=true , address=45.91.116.9:4263 , piece_id=4MB6SKKVKJNFWWWE24U2YJI25VAF7RLRPRIZQ2QTZXGLWPRSKI4Q, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754042864 (546h6m27.62512593s) [noise=true , address=145.79.227.45:4230 , piece_id=NONHN4EKNARBD2YIAQ756DW4NAPOS3UMFCEWN4KJ5UEAJEHH4QIQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754144698 (546h6m27.625227733s) [noise=true , address=145.79.227.37:4565 , piece_id=25KGPQZXGUI2ZDEXOYAETSUI6YTFT65UJFAAAIYRWYUWEXSUCNHQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754566932 (546h6m27.625649977s) [noise=true , address=45.91.116.10:4083 , piece_id=VD32C5TT6OWORW3U6767NYXRFDULZSOA2PU6CSBTLLC3AQZR2HQA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754764529 (546h6m27.625847574s) [noise=true , address=45.91.116.14:4191 , piece_id=N75N4G6QH4PAHZPQYFBE5LKCYNLTBS25V6CN2NCRM4FPSSEB75VA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.754905227 (546h6m27.625988272s) [noise=true , address=145.79.227.20:4438 , piece_id=4SZKGU2LEZ5WK6UTZVQJVYDO7J65K4OT2GKGERVSSTT57W7EECTA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.755227731 (546h6m27.626310766s) [noise=true , address=45.91.116.7:4010 , piece_id=P47MG5Q5GNZKA5U544TUKNK7PTARAHQAN2CHDMVXEZT42TBNXCJQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.755637491 (546h6m27.626720536s) [noise=true , address=145.79.227.44:4195 , piece_id=YNBFGVEO34DUAXVGJGHY3IO6KC6WNET72UONMMPOGNK5DQCIRVSA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.756059785 (546h6m27.62714282s) [noise=true , address=145.79.227.51:4411 , piece_id=WLJDZHCZIN2YWRBUHWP3WLELR5CBSVBZJBYPTP5MR7VZD4W5EPFA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.756485676 (546h6m27.627568711s) [noise=true , address=45.91.116.13:4156 , piece_id=4EJIFP2QGBDXAFAX22XGTMLKZZ7MRARIHSIIXO77SCN436ZVGWFA, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.757098633 (546h6m27.628181668s) [noise=true , address=145.79.227.9:4367 , piece_id=KEPVDQAHLLDT4WCNQTHRQYDC3VKE3M5VDHGCUS6MCR6Z6ZILMXGQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:27.765253682 (546h6m27.636336727s) [noise=true , address=145.79.227.39:4013 , piece_id=KFXOW2DVPX4LHEHMUCZJBI7KED3SKWZIM53LGK32NZXTZOJKGDGQ, seg_pos={0 0}, upload=478880]: Uploading piece... Dec 28 09:35:29.480652613 (546h6m29.351735648s) [upload=478880]: Next returned nil segment Dec 28 09:35:29.480666730 (546h6m29.351749765s) [upload=478880]: Waiting for error group managing segments... Dec 28 09:35:29.766845633 (546h6m29.637928678s) [noise=true , address=145.79.227.45:4230 , piece_id=NONHN4EKNARBD2YIAQ756DW4NAPOS3UMFCEWN4KJ5UEAJEHH4QIQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.767303565 (546h6m29.6383866s) [noise=true , address=145.79.227.11:4730 , piece_id=T4GWZTVADA55XZLK6UQ2KP6ITZVMIHYKNA4KXQ2PUKUKLORBI3XQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.779454122 (546h6m29.650537167s) [noise=true , address=145.79.227.39:4013 , piece_id=KFXOW2DVPX4LHEHMUCZJBI7KED3SKWZIM53LGK32NZXTZOJKGDGQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.780555569 (546h6m29.651638614s) [noise=true , address=145.79.227.37:4565 , piece_id=25KGPQZXGUI2ZDEXOYAETSUI6YTFT65UJFAAAIYRWYUWEXSUCNHQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.786442138 (546h6m29.657525183s) [noise=true , address=145.79.227.20:4438 , piece_id=4SZKGU2LEZ5WK6UTZVQJVYDO7J65K4OT2GKGERVSSTT57W7EECTA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.787226852 (546h6m29.658309887s) [noise=true , address=145.79.227.51:4411 , piece_id=WLJDZHCZIN2YWRBUHWP3WLELR5CBSVBZJBYPTP5MR7VZD4W5EPFA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.794995575 (546h6m29.66607862s) [noise=true , address=145.79.227.9:4367 , piece_id=KEPVDQAHLLDT4WCNQTHRQYDC3VKE3M5VDHGCUS6MCR6Z6ZILMXGQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:29.997843773 (546h6m29.868926808s) [noise=true , address=45.91.116.9:4263 , piece_id=4MB6SKKVKJNFWWWE24U2YJI25VAF7RLRPRIZQ2QTZXGLWPRSKI4Q, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.018887895 (546h6m29.88997093s) [noise=true , address=45.91.116.7:4010 , piece_id=P47MG5Q5GNZKA5U544TUKNK7PTARAHQAN2CHDMVXEZT42TBNXCJQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.048991647 (546h6m29.920074692s) [noise=true , address=45.91.116.13:4156 , piece_id=4EJIFP2QGBDXAFAX22XGTMLKZZ7MRARIHSIIXO77SCN436ZVGWFA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.057505478 (546h6m29.928588513s) [noise=true , address=77.90.31.14:4662 , piece_id=ALZ4O3NJMIWBZPVLQ3BAFL7KYQVECYUA3JEOLIOJVPRWQ326FLFQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.691813288 (546h6m30.562896333s) [noise=true , address=45.91.116.8:4046 , piece_id=3JVV2WAVRHM5MA7M6GYBDYSTAL6LAMLYZIZD46BE2IQ3OHXV45OQ, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.749635081 (546h6m30.620718116s) [noise=true , address=45.91.116.14:4191 , piece_id=N75N4G6QH4PAHZPQYFBE5LKCYNLTBS25V6CN2NCRM4FPSSEB75VA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: <nil> Dec 28 09:35:30.749656672 (546h6m30.620739717s) [seg_pos={0 0}, upload=478880]: Segment reached optimal threshold of 13 pieces. Dec 28 09:35:30.749916657 (546h6m30.620999702s) [noise=true , address=77.90.31.18:4805 , piece_id=2OFQMO45JG3NG5QAXLM3MJ6UUPZS67LXTZNSTZ2NABRU2V5U346Q, seg_pos={0 0}, upload=478880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 28 09:35:30.749925453 (546h6m30.621008498s) [noise=true , address=77.90.31.18:4805 , piece_id=2OFQMO45JG3NG5QAXLM3MJ6UUPZS67LXTZNSTZ2NABRU2V5U346Q, seg_pos={0 0}, upload=478880]: Long tail cancellation due to optimal threshold Dec 28 09:35:30.750339852 (546h6m30.621422897s) [noise=true , address=45.91.116.10:4083 , piece_id=VD32C5TT6OWORW3U6767NYXRFDULZSOA2PU6CSBTLLC3AQZR2HQA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:36368->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:36368->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36368->45.91.116.10:4083: use of closed network connection Dec 28 09:35:30.750348650 (546h6m30.621431695s) [noise=true , address=45.91.116.10:4083 , piece_id=VD32C5TT6OWORW3U6767NYXRFDULZSOA2PU6CSBTLLC3AQZR2HQA, seg_pos={0 0}, upload=478880]: Long tail cancellation due to optimal threshold Dec 28 09:35:30.750494658 (546h6m30.621577703s) [noise=true , address=145.79.227.44:4195 , piece_id=YNBFGVEO34DUAXVGJGHY3IO6KC6WNET72UONMMPOGNK5DQCIRVSA, seg_pos={0 0}, upload=478880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 09:35:30.750502983 (546h6m30.621586018s) [noise=true , address=145.79.227.44:4195 , piece_id=YNBFGVEO34DUAXVGJGHY3IO6KC6WNET72UONMMPOGNK5DQCIRVSA, seg_pos={0 0}, upload=478880]: Long tail cancellation due to optimal threshold Dec 28 09:35:30.750523643 (546h6m30.621606688s) [seg_pos={0 0}, upload=478880]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:35:30.750604236 (546h6m30.621687281s) [upload=478880]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:35:30.750610137 (546h6m30.621693172s) [upload=478880]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:35:31.045721700 (546h6m30.916804735s) [upload=478880]: Done uploading segments... Dec 28 09:35:31.045746788 (546h6m30.916829833s) [upload=478880]: Upload finished. err: <nil> Duration: 30.003011807s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [........X.X...X....X.X.....X.....XXX.XX...XX.XX..X...XXX..XX] duration: 5.001766934s trace: https://jaeger.dev.storj.io/trace/6c2381cf43b7de34 FAIL (2025-12-28 09:35:33 UTC) orbiter warning: context deadline exceeded Duration: 5.001786512s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [X........X.X...X....X.X.....X.....XXX.XX...XX.XX..X...XXX..X] duration: 5.001305033s trace: https://jaeger.dev.storj.io/trace/619aea6e90670677 FAIL (2025-12-28 09:34:38 UTC) orbiter warning: context deadline exceeded Duration: 5.001328509s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.....XX.X.....X......X..............X....X...X......XX...X..] duration: 5.986022343s trace: https://jaeger.dev.storj.io/trace/641e37f753f3376a FAIL (2025-12-28 09:34:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: H2HDJ7N2XSHT3VFPHTLPJ7HXCM2WAFNG74CAZURGBZANYR3T6UWA) context deadline exceeded; piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: ZAWG7JOFQYNXXA6XIRYBGVNWYZSRVBBXUB2RYKAGMN5F4BXHZE5Q) context deadline exceeded ======LOGS====== upload Dec 28 09:32:48.344299721 (546h3m48.215382766s) [upload=478840]: Uploading segments... Dec 28 09:32:48.344423687 (546h3m48.215506732s) [upload=478840]: Got next segment. Inline: false Dec 28 09:32:48.344474133 (546h3m48.215557168s) [upload=478840]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:32:48.344478801 (546h3m48.215561846s) [upload=478840]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:32:48.589211504 (546h3m48.460294559s) [seg_pos={0 0}, upload=478840]: Begin upload segment... Dec 28 09:32:48.589281548 (546h3m48.460364583s) [seg_pos={0 0}, upload=478840]: Done begin upload segment. Dec 28 09:32:48.589317646 (546h3m48.460400691s) [noise=true , address=45.91.116.11:4117 , piece_id=T7EEWP5C3PV3TJMRIDNCUNXDI4TLNKGYCW2SSVJPGU7AOXSUD6QA, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.589337825 (546h3m48.46042086s) [noise=true , address=145.79.227.29:4943 , piece_id=WLECIP4WAJKHQUT2UMP36VOKVC6WNBITVOUA7H3S5TLKQRBII5TQ, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.589952275 (546h3m48.46103532s) [noise=true , address=145.79.227.14:4838 , piece_id=PAIRS7AXMJAMBQADBEVPM5P5IT4EUG5VW4EVKSMPDHCEHSGIVKUA, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.590387844 (546h3m48.46147088s) [noise=true , address=77.90.31.14:4661 , piece_id=ZAWG7JOFQYNXXA6XIRYBGVNWYZSRVBBXUB2RYKAGMN5F4BXHZE5Q, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.590695621 (546h3m48.461778656s) [noise=true , address=145.79.227.42:4123 , piece_id=NNZVDJNHQIEHSHYLZIWBOUBSG5UGAZV7KRPFW5O633IXNNMKTBFA, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.591066556 (546h3m48.462149601s) [noise=true , address=145.79.227.41:4086 , piece_id=WBSHPU26MOBWO3WRAMBUMRCCZ4CW7FCLMWC3AR2FL5S7LIRMESCA, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.591268541 (546h3m48.462351586s) [noise=true , address=145.79.227.51:4409 , piece_id=AS2D7LBL4ZG6F7XTVI4FUAMJ3CMHXTPTZE7DU6Y76B6SL73SZ33Q, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.591811185 (546h3m48.46289423s) [noise=true , address=45.91.116.4:4937 , piece_id=QPAJWKZAS7JLYYK4H6Z3R5FG2VO7D4U2I6ALRCO5T2PTGIVHIAAA, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.592352164 (546h3m48.463435199s) [noise=true , address=145.79.227.19:4079 , piece_id=KFS2MLOLFAGUEYUY4B3XBAUHXAVANGK2RYWAZTO3MAMUXLNBISRA, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.593020206 (546h3m48.464103241s) [noise=true , address=145.79.227.20:4438 , piece_id=ZP4ZERSBOEXUVUKZBD5LTH3UWFWVAAA4GZUYGMBAXRFW65GUB7RA, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.593342541 (546h3m48.464425576s) [noise=true , address=77.90.31.16:4735 , piece_id=OORLTYOX2VRZSWLAFTDICVJ22ZYANR4475X7XIXQXGM5TTE3Q2AQ, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.593880665 (546h3m48.46496371s) [noise=true , address=145.79.227.4:4185 , piece_id=3D3MT3UFBCWJVAJFMDDIF46WG7HOKL5Y6J2XESDBUC4QK2EPT7VQ, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.594478052 (546h3m48.465561087s) [noise=true , address=145.79.227.47:4266 , piece_id=DW3U5UWPSJNPKDEKUF2Z4SIWDJZQSQMDVBG4S4C7XEG34OIOLECQ, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.594868717 (546h3m48.465952313s) [noise=true , address=145.79.227.32:4834 , piece_id=D3UOF4LBHHFTVEQLS2ALXDNI3JSF42BJ7TX3VYI5SRORFPD5DQIQ, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.595397724 (546h3m48.466480759s) [noise=true , address=145.79.227.36:4518 , piece_id=GR4P64FEXICCBGSB33D76KGJ65QDFOLFE5KEPFJIPRCCHW57OJLA, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.595456726 (546h3m48.466539761s) [noise=true , address=45.91.116.12:4226 , piece_id=H2HDJ7N2XSHT3VFPHTLPJ7HXCM2WAFNG74CAZURGBZANYR3T6UWA, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:50.303908198 (546h3m50.174991233s) [upload=478840]: Next returned nil segment Dec 28 09:32:50.303922435 (546h3m50.17500548s) [upload=478840]: Waiting for error group managing segments... Dec 28 09:32:50.585333951 (546h3m50.456416996s) [noise=true , address=145.79.227.32:4834 , piece_id=D3UOF4LBHHFTVEQLS2ALXDNI3JSF42BJ7TX3VYI5SRORFPD5DQIQ, seg_pos={0 0}, upload=478840]: Done uploading piece. err: <nil> Dec 28 09:32:50.591814791 (546h3m50.462897836s) [noise=true , address=145.79.227.29:4943 , piece_id=WLECIP4WAJKHQUT2UMP36VOKVC6WNBITVOUA7H3S5TLKQRBII5TQ, seg_pos={0 0}, upload=478840]: Done uploading piece. err: <nil> Dec 28 09:32:50.598522353 (546h3m50.469605388s) [noise=true , address=145.79.227.4:4185 , piece_id=3D3MT3UFBCWJVAJFMDDIF46WG7HOKL5Y6J2XESDBUC4QK2EPT7VQ, seg_pos={0 0}, upload=478840]: Done uploading piece. err: <nil> Dec 28 09:32:50.599218238 (546h3m50.470301273s) [noise=true , address=145.79.227.20:4438 , piece_id=ZP4ZERSBOEXUVUKZBD5LTH3UWFWVAAA4GZUYGMBAXRFW65GUB7RA, seg_pos={0 0}, upload=478840]: Done uploading piece. err: <nil> Dec 28 09:32:50.599424351 (546h3m50.470507386s) [noise=true , address=145.79.227.42:4123 , piece_id=NNZVDJNHQIEHSHYLZIWBOUBSG5UGAZV7KRPFW5O633IXNNMKTBFA, seg_pos={0 0}, upload=478840]: Done uploading piece. err: <nil> Dec 28 09:32:50.600993699 (546h3m50.472076744s) [noise=true , address=145.79.227.41:4086 , piece_id=WBSHPU26MOBWO3WRAMBUMRCCZ4CW7FCLMWC3AR2FL5S7LIRMESCA, seg_pos={0 0}, upload=478840]: Done uploading piece. err: <nil> Dec 28 09:32:50.608841283 (546h3m50.479924338s) [noise=true , address=145.79.227.47:4266 , piece_id=DW3U5UWPSJNPKDEKUF2Z4SIWDJZQSQMDVBG4S4C7XEG34OIOLECQ, seg_pos={0 0}, upload=478840]: Done uploading piece. err: <nil> Dec 28 09:32:50.614748762 (546h3m50.485831797s) [noise=true , address=145.79.227.14:4838 , piece_id=PAIRS7AXMJAMBQADBEVPM5P5IT4EUG5VW4EVKSMPDHCEHSGIVKUA, seg_pos={0 0}, upload=478840]: Done uploading piece. err: <nil> Dec 28 09:32:50.614892856 (546h3m50.485975921s) [noise=true , address=145.79.227.51:4409 , piece_id=AS2D7LBL4ZG6F7XTVI4FUAMJ3CMHXTPTZE7DU6Y76B6SL73SZ33Q, seg_pos={0 0}, upload=478840]: Done uploading piece. err: <nil> Dec 28 09:32:50.639871699 (546h3m50.510954745s) [noise=true , address=145.79.227.36:4518 , piece_id=GR4P64FEXICCBGSB33D76KGJ65QDFOLFE5KEPFJIPRCCHW57OJLA, seg_pos={0 0}, upload=478840]: Done uploading piece. err: <nil> Dec 28 09:32:50.655384319 (546h3m50.526467364s) [noise=true , address=145.79.227.19:4079 , piece_id=KFS2MLOLFAGUEYUY4B3XBAUHXAVANGK2RYWAZTO3MAMUXLNBISRA, seg_pos={0 0}, upload=478840]: Done uploading piece. err: <nil> Dec 28 09:32:50.829258795 (546h3m50.70034184s) [noise=true , address=45.91.116.12:4226 , piece_id=H2HDJ7N2XSHT3VFPHTLPJ7HXCM2WAFNG74CAZURGBZANYR3T6UWA, seg_pos={0 0}, upload=478840]: Done uploading piece. err: <nil> Dec 28 09:32:50.857304619 (546h3m50.728387664s) [noise=true , address=77.90.31.14:4661 , piece_id=ZAWG7JOFQYNXXA6XIRYBGVNWYZSRVBBXUB2RYKAGMN5F4BXHZE5Q, seg_pos={0 0}, upload=478840]: Done uploading piece. err: <nil> Dec 28 09:32:50.857329938 (546h3m50.728412993s) [seg_pos={0 0}, upload=478840]: Segment reached optimal threshold of 13 pieces. Dec 28 09:32:50.857636971 (546h3m50.728720017s) [noise=true , address=45.91.116.11:4117 , piece_id=T7EEWP5C3PV3TJMRIDNCUNXDI4TLNKGYCW2SSVJPGU7AOXSUD6QA, seg_pos={0 0}, upload=478840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52688->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52688->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52688->45.91.116.11:4117: use of closed network connection Dec 28 09:32:50.857659745 (546h3m50.72874278s) [noise=true , address=45.91.116.11:4117 , piece_id=T7EEWP5C3PV3TJMRIDNCUNXDI4TLNKGYCW2SSVJPGU7AOXSUD6QA, seg_pos={0 0}, upload=478840]: Long tail cancellation due to optimal threshold Dec 28 09:32:50.857704961 (546h3m50.728787996s) [noise=true , address=45.91.116.4:4937 , piece_id=QPAJWKZAS7JLYYK4H6Z3R5FG2VO7D4U2I6ALRCO5T2PTGIVHIAAA, seg_pos={0 0}, upload=478840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: read tcp 188.166.225.170:58902->45.91.116.4:4937: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 09:32:50.857714199 (546h3m50.728797244s) [noise=true , address=45.91.116.4:4937 , piece_id=QPAJWKZAS7JLYYK4H6Z3R5FG2VO7D4U2I6ALRCO5T2PTGIVHIAAA, seg_pos={0 0}, upload=478840]: Long tail cancellation due to optimal threshold Dec 28 09:32:50.857924570 (546h3m50.729007605s) [noise=true , address=77.90.31.16:4735 , piece_id=OORLTYOX2VRZSWLAFTDICVJ22ZYANR4475X7XIXQXGM5TTE3Q2AQ, seg_pos={0 0}, upload=478840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: read tcp 188.166.225.170:45320->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 09:32:50.857931403 (546h3m50.729014438s) [noise=true , address=77.90.31.16:4735 , piece_id=OORLTYOX2VRZSWLAFTDICVJ22ZYANR4475X7XIXQXGM5TTE3Q2AQ, seg_pos={0 0}, upload=478840]: Long tail cancellation due to optimal threshold Dec 28 09:32:50.857947463 (546h3m50.729030508s) [seg_pos={0 0}, upload=478840]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:32:50.858033507 (546h3m50.729116542s) [upload=478840]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:32:50.858041823 (546h3m50.729124858s) [upload=478840]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:32:51.142168850 (546h3m51.013251885s) [upload=478840]: Done uploading segments... Dec 28 09:32:51.142188176 (546h3m51.013271211s) [upload=478840]: Upload finished. err: <nil> Duration: 5.98604711s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X.....XX.X.....X......X..............X....X...X......XX...X.] duration: 5.813955938s trace: https://jaeger.dev.storj.io/trace/641e37f753f3376a FAIL (2025-12-28 09:33:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: H2HDJ7N2XSHT3VFPHTLPJ7HXCM2WAFNG74CAZURGBZANYR3T6UWA) context deadline exceeded; piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: ZAWG7JOFQYNXXA6XIRYBGVNWYZSRVBBXUB2RYKAGMN5F4BXHZE5Q) context deadline exceeded ======LOGS====== upload Dec 28 09:32:48.344299721 (546h3m48.215382766s) [upload=478840]: Uploading segments... Dec 28 09:32:48.344423687 (546h3m48.215506732s) [upload=478840]: Got next segment. Inline: false Dec 28 09:32:48.344474133 (546h3m48.215557168s) [upload=478840]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:32:48.344478801 (546h3m48.215561846s) [upload=478840]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:32:48.589211504 (546h3m48.460294559s) [seg_pos={0 0}, upload=478840]: Begin upload segment... Dec 28 09:32:48.589281548 (546h3m48.460364583s) [seg_pos={0 0}, upload=478840]: Done begin upload segment. Dec 28 09:32:48.589317646 (546h3m48.460400691s) [noise=true , address=45.91.116.11:4117 , piece_id=T7EEWP5C3PV3TJMRIDNCUNXDI4TLNKGYCW2SSVJPGU7AOXSUD6QA, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.589337825 (546h3m48.46042086s) [noise=true , address=145.79.227.29:4943 , piece_id=WLECIP4WAJKHQUT2UMP36VOKVC6WNBITVOUA7H3S5TLKQRBII5TQ, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.589952275 (546h3m48.46103532s) [noise=true , address=145.79.227.14:4838 , piece_id=PAIRS7AXMJAMBQADBEVPM5P5IT4EUG5VW4EVKSMPDHCEHSGIVKUA, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.590387844 (546h3m48.46147088s) [noise=true , address=77.90.31.14:4661 , piece_id=ZAWG7JOFQYNXXA6XIRYBGVNWYZSRVBBXUB2RYKAGMN5F4BXHZE5Q, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.590695621 (546h3m48.461778656s) [noise=true , address=145.79.227.42:4123 , piece_id=NNZVDJNHQIEHSHYLZIWBOUBSG5UGAZV7KRPFW5O633IXNNMKTBFA, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.591066556 (546h3m48.462149601s) [noise=true , address=145.79.227.41:4086 , piece_id=WBSHPU26MOBWO3WRAMBUMRCCZ4CW7FCLMWC3AR2FL5S7LIRMESCA, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.591268541 (546h3m48.462351586s) [noise=true , address=145.79.227.51:4409 , piece_id=AS2D7LBL4ZG6F7XTVI4FUAMJ3CMHXTPTZE7DU6Y76B6SL73SZ33Q, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.591811185 (546h3m48.46289423s) [noise=true , address=45.91.116.4:4937 , piece_id=QPAJWKZAS7JLYYK4H6Z3R5FG2VO7D4U2I6ALRCO5T2PTGIVHIAAA, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.592352164 (546h3m48.463435199s) [noise=true , address=145.79.227.19:4079 , piece_id=KFS2MLOLFAGUEYUY4B3XBAUHXAVANGK2RYWAZTO3MAMUXLNBISRA, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.593020206 (546h3m48.464103241s) [noise=true , address=145.79.227.20:4438 , piece_id=ZP4ZERSBOEXUVUKZBD5LTH3UWFWVAAA4GZUYGMBAXRFW65GUB7RA, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.593342541 (546h3m48.464425576s) [noise=true , address=77.90.31.16:4735 , piece_id=OORLTYOX2VRZSWLAFTDICVJ22ZYANR4475X7XIXQXGM5TTE3Q2AQ, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.593880665 (546h3m48.46496371s) [noise=true , address=145.79.227.4:4185 , piece_id=3D3MT3UFBCWJVAJFMDDIF46WG7HOKL5Y6J2XESDBUC4QK2EPT7VQ, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.594478052 (546h3m48.465561087s) [noise=true , address=145.79.227.47:4266 , piece_id=DW3U5UWPSJNPKDEKUF2Z4SIWDJZQSQMDVBG4S4C7XEG34OIOLECQ, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.594868717 (546h3m48.465952313s) [noise=true , address=145.79.227.32:4834 , piece_id=D3UOF4LBHHFTVEQLS2ALXDNI3JSF42BJ7TX3VYI5SRORFPD5DQIQ, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.595397724 (546h3m48.466480759s) [noise=true , address=145.79.227.36:4518 , piece_id=GR4P64FEXICCBGSB33D76KGJ65QDFOLFE5KEPFJIPRCCHW57OJLA, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.595456726 (546h3m48.466539761s) [noise=true , address=45.91.116.12:4226 , piece_id=H2HDJ7N2XSHT3VFPHTLPJ7HXCM2WAFNG74CAZURGBZANYR3T6UWA, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:50.303908198 (546h3m50.174991233s) [upload=478840]: Next returned nil segment Dec 28 09:32:50.303922435 (546h3m50.17500548s) [upload=478840]: Waiting for error group managing segments... Dec 28 09:32:50.585333951 (546h3m50.456416996s) [noise=true , address=145.79.227.32:4834 , piece_id=D3UOF4LBHHFTVEQLS2ALXDNI3JSF42BJ7TX3VYI5SRORFPD5DQIQ, seg_pos={0 0}, upload=478840]: Done uploading piece. err: <nil> Dec 28 09:32:50.591814791 (546h3m50.462897836s) [noise=true , address=145.79.227.29:4943 , piece_id=WLECIP4WAJKHQUT2UMP36VOKVC6WNBITVOUA7H3S5TLKQRBII5TQ, seg_pos={0 0}, upload=478840]: Done uploading piece. err: <nil> Dec 28 09:32:50.598522353 (546h3m50.469605388s) [noise=true , address=145.79.227.4:4185 , piece_id=3D3MT3UFBCWJVAJFMDDIF46WG7HOKL5Y6J2XESDBUC4QK2EPT7VQ, seg_pos={0 0}, upload=478840]: Done uploading piece. err: <nil> Dec 28 09:32:50.599218238 (546h3m50.470301273s) [noise=true , address=145.79.227.20:4438 , piece_id=ZP4ZERSBOEXUVUKZBD5LTH3UWFWVAAA4GZUYGMBAXRFW65GUB7RA, seg_pos={0 0}, upload=478840]: Done uploading piece. err: <nil> Dec 28 09:32:50.599424351 (546h3m50.470507386s) [noise=true , address=145.79.227.42:4123 , piece_id=NNZVDJNHQIEHSHYLZIWBOUBSG5UGAZV7KRPFW5O633IXNNMKTBFA, seg_pos={0 0}, upload=478840]: Done uploading piece. err: <nil> Dec 28 09:32:50.600993699 (546h3m50.472076744s) [noise=true , address=145.79.227.41:4086 , piece_id=WBSHPU26MOBWO3WRAMBUMRCCZ4CW7FCLMWC3AR2FL5S7LIRMESCA, seg_pos={0 0}, upload=478840]: Done uploading piece. err: <nil> Dec 28 09:32:50.608841283 (546h3m50.479924338s) [noise=true , address=145.79.227.47:4266 , piece_id=DW3U5UWPSJNPKDEKUF2Z4SIWDJZQSQMDVBG4S4C7XEG34OIOLECQ, seg_pos={0 0}, upload=478840]: Done uploading piece. err: <nil> Dec 28 09:32:50.614748762 (546h3m50.485831797s) [noise=true , address=145.79.227.14:4838 , piece_id=PAIRS7AXMJAMBQADBEVPM5P5IT4EUG5VW4EVKSMPDHCEHSGIVKUA, seg_pos={0 0}, upload=478840]: Done uploading piece. err: <nil> Dec 28 09:32:50.614892856 (546h3m50.485975921s) [noise=true , address=145.79.227.51:4409 , piece_id=AS2D7LBL4ZG6F7XTVI4FUAMJ3CMHXTPTZE7DU6Y76B6SL73SZ33Q, seg_pos={0 0}, upload=478840]: Done uploading piece. err: <nil> Dec 28 09:32:50.639871699 (546h3m50.510954745s) [noise=true , address=145.79.227.36:4518 , piece_id=GR4P64FEXICCBGSB33D76KGJ65QDFOLFE5KEPFJIPRCCHW57OJLA, seg_pos={0 0}, upload=478840]: Done uploading piece. err: <nil> Dec 28 09:32:50.655384319 (546h3m50.526467364s) [noise=true , address=145.79.227.19:4079 , piece_id=KFS2MLOLFAGUEYUY4B3XBAUHXAVANGK2RYWAZTO3MAMUXLNBISRA, seg_pos={0 0}, upload=478840]: Done uploading piece. err: <nil> Dec 28 09:32:50.829258795 (546h3m50.70034184s) [noise=true , address=45.91.116.12:4226 , piece_id=H2HDJ7N2XSHT3VFPHTLPJ7HXCM2WAFNG74CAZURGBZANYR3T6UWA, seg_pos={0 0}, upload=478840]: Done uploading piece. err: <nil> Dec 28 09:32:50.857304619 (546h3m50.728387664s) [noise=true , address=77.90.31.14:4661 , piece_id=ZAWG7JOFQYNXXA6XIRYBGVNWYZSRVBBXUB2RYKAGMN5F4BXHZE5Q, seg_pos={0 0}, upload=478840]: Done uploading piece. err: <nil> Dec 28 09:32:50.857329938 (546h3m50.728412993s) [seg_pos={0 0}, upload=478840]: Segment reached optimal threshold of 13 pieces. Dec 28 09:32:50.857636971 (546h3m50.728720017s) [noise=true , address=45.91.116.11:4117 , piece_id=T7EEWP5C3PV3TJMRIDNCUNXDI4TLNKGYCW2SSVJPGU7AOXSUD6QA, seg_pos={0 0}, upload=478840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52688->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52688->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52688->45.91.116.11:4117: use of closed network connection Dec 28 09:32:50.857659745 (546h3m50.72874278s) [noise=true , address=45.91.116.11:4117 , piece_id=T7EEWP5C3PV3TJMRIDNCUNXDI4TLNKGYCW2SSVJPGU7AOXSUD6QA, seg_pos={0 0}, upload=478840]: Long tail cancellation due to optimal threshold Dec 28 09:32:50.857704961 (546h3m50.728787996s) [noise=true , address=45.91.116.4:4937 , piece_id=QPAJWKZAS7JLYYK4H6Z3R5FG2VO7D4U2I6ALRCO5T2PTGIVHIAAA, seg_pos={0 0}, upload=478840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: read tcp 188.166.225.170:58902->45.91.116.4:4937: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 09:32:50.857714199 (546h3m50.728797244s) [noise=true , address=45.91.116.4:4937 , piece_id=QPAJWKZAS7JLYYK4H6Z3R5FG2VO7D4U2I6ALRCO5T2PTGIVHIAAA, seg_pos={0 0}, upload=478840]: Long tail cancellation due to optimal threshold Dec 28 09:32:50.857924570 (546h3m50.729007605s) [noise=true , address=77.90.31.16:4735 , piece_id=OORLTYOX2VRZSWLAFTDICVJ22ZYANR4475X7XIXQXGM5TTE3Q2AQ, seg_pos={0 0}, upload=478840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: read tcp 188.166.225.170:45320->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 09:32:50.857931403 (546h3m50.729014438s) [noise=true , address=77.90.31.16:4735 , piece_id=OORLTYOX2VRZSWLAFTDICVJ22ZYANR4475X7XIXQXGM5TTE3Q2AQ, seg_pos={0 0}, upload=478840]: Long tail cancellation due to optimal threshold Dec 28 09:32:50.857947463 (546h3m50.729030508s) [seg_pos={0 0}, upload=478840]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:32:50.858033507 (546h3m50.729116542s) [upload=478840]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:32:50.858041823 (546h3m50.729124858s) [upload=478840]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:32:51.142168850 (546h3m51.013251885s) [upload=478840]: Done uploading segments... Dec 28 09:32:51.142188176 (546h3m51.013271211s) [upload=478840]: Upload finished. err: <nil> Duration: 5.813987198s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [.X........X.X...X....X.X.....X.....XXX.XX...XX.XX..X...XXX..] duration: 4.675047004s trace: https://jaeger.dev.storj.io/trace/5a881e44b788246b FAIL (2025-12-28 09:33:44 UTC) orbiter warning: context deadline exceeded Duration: 4.675069726s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [XX.....XX.X.....X......X..............X....X...X......XX...X] duration: 30.002776448s trace: https://jaeger.dev.storj.io/trace/641e37f753f3376a FAIL (2025-12-28 09:33:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: H2HDJ7N2XSHT3VFPHTLPJ7HXCM2WAFNG74CAZURGBZANYR3T6UWA) context deadline exceeded; piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: ZAWG7JOFQYNXXA6XIRYBGVNWYZSRVBBXUB2RYKAGMN5F4BXHZE5Q) context deadline exceeded ======LOGS====== upload Dec 28 09:32:48.344299721 (546h3m48.215382766s) [upload=478840]: Uploading segments... Dec 28 09:32:48.344423687 (546h3m48.215506732s) [upload=478840]: Got next segment. Inline: false Dec 28 09:32:48.344474133 (546h3m48.215557168s) [upload=478840]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:32:48.344478801 (546h3m48.215561846s) [upload=478840]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:32:48.589211504 (546h3m48.460294559s) [seg_pos={0 0}, upload=478840]: Begin upload segment... Dec 28 09:32:48.589281548 (546h3m48.460364583s) [seg_pos={0 0}, upload=478840]: Done begin upload segment. Dec 28 09:32:48.589317646 (546h3m48.460400691s) [noise=true , address=45.91.116.11:4117 , piece_id=T7EEWP5C3PV3TJMRIDNCUNXDI4TLNKGYCW2SSVJPGU7AOXSUD6QA, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.589337825 (546h3m48.46042086s) [noise=true , address=145.79.227.29:4943 , piece_id=WLECIP4WAJKHQUT2UMP36VOKVC6WNBITVOUA7H3S5TLKQRBII5TQ, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.589952275 (546h3m48.46103532s) [noise=true , address=145.79.227.14:4838 , piece_id=PAIRS7AXMJAMBQADBEVPM5P5IT4EUG5VW4EVKSMPDHCEHSGIVKUA, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.590387844 (546h3m48.46147088s) [noise=true , address=77.90.31.14:4661 , piece_id=ZAWG7JOFQYNXXA6XIRYBGVNWYZSRVBBXUB2RYKAGMN5F4BXHZE5Q, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.590695621 (546h3m48.461778656s) [noise=true , address=145.79.227.42:4123 , piece_id=NNZVDJNHQIEHSHYLZIWBOUBSG5UGAZV7KRPFW5O633IXNNMKTBFA, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.591066556 (546h3m48.462149601s) [noise=true , address=145.79.227.41:4086 , piece_id=WBSHPU26MOBWO3WRAMBUMRCCZ4CW7FCLMWC3AR2FL5S7LIRMESCA, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.591268541 (546h3m48.462351586s) [noise=true , address=145.79.227.51:4409 , piece_id=AS2D7LBL4ZG6F7XTVI4FUAMJ3CMHXTPTZE7DU6Y76B6SL73SZ33Q, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.591811185 (546h3m48.46289423s) [noise=true , address=45.91.116.4:4937 , piece_id=QPAJWKZAS7JLYYK4H6Z3R5FG2VO7D4U2I6ALRCO5T2PTGIVHIAAA, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.592352164 (546h3m48.463435199s) [noise=true , address=145.79.227.19:4079 , piece_id=KFS2MLOLFAGUEYUY4B3XBAUHXAVANGK2RYWAZTO3MAMUXLNBISRA, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.593020206 (546h3m48.464103241s) [noise=true , address=145.79.227.20:4438 , piece_id=ZP4ZERSBOEXUVUKZBD5LTH3UWFWVAAA4GZUYGMBAXRFW65GUB7RA, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.593342541 (546h3m48.464425576s) [noise=true , address=77.90.31.16:4735 , piece_id=OORLTYOX2VRZSWLAFTDICVJ22ZYANR4475X7XIXQXGM5TTE3Q2AQ, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.593880665 (546h3m48.46496371s) [noise=true , address=145.79.227.4:4185 , piece_id=3D3MT3UFBCWJVAJFMDDIF46WG7HOKL5Y6J2XESDBUC4QK2EPT7VQ, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.594478052 (546h3m48.465561087s) [noise=true , address=145.79.227.47:4266 , piece_id=DW3U5UWPSJNPKDEKUF2Z4SIWDJZQSQMDVBG4S4C7XEG34OIOLECQ, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.594868717 (546h3m48.465952313s) [noise=true , address=145.79.227.32:4834 , piece_id=D3UOF4LBHHFTVEQLS2ALXDNI3JSF42BJ7TX3VYI5SRORFPD5DQIQ, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.595397724 (546h3m48.466480759s) [noise=true , address=145.79.227.36:4518 , piece_id=GR4P64FEXICCBGSB33D76KGJ65QDFOLFE5KEPFJIPRCCHW57OJLA, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:48.595456726 (546h3m48.466539761s) [noise=true , address=45.91.116.12:4226 , piece_id=H2HDJ7N2XSHT3VFPHTLPJ7HXCM2WAFNG74CAZURGBZANYR3T6UWA, seg_pos={0 0}, upload=478840]: Uploading piece... Dec 28 09:32:50.303908198 (546h3m50.174991233s) [upload=478840]: Next returned nil segment Dec 28 09:32:50.303922435 (546h3m50.17500548s) [upload=478840]: Waiting for error group managing segments... Dec 28 09:32:50.585333951 (546h3m50.456416996s) [noise=true , address=145.79.227.32:4834 , piece_id=D3UOF4LBHHFTVEQLS2ALXDNI3JSF42BJ7TX3VYI5SRORFPD5DQIQ, seg_pos={0 0}, upload=478840]: Done uploading piece. err: <nil> Dec 28 09:32:50.591814791 (546h3m50.462897836s) [noise=true , address=145.79.227.29:4943 , piece_id=WLECIP4WAJKHQUT2UMP36VOKVC6WNBITVOUA7H3S5TLKQRBII5TQ, seg_pos={0 0}, upload=478840]: Done uploading piece. err: <nil> Dec 28 09:32:50.598522353 (546h3m50.469605388s) [noise=true , address=145.79.227.4:4185 , piece_id=3D3MT3UFBCWJVAJFMDDIF46WG7HOKL5Y6J2XESDBUC4QK2EPT7VQ, seg_pos={0 0}, upload=478840]: Done uploading piece. err: <nil> Dec 28 09:32:50.599218238 (546h3m50.470301273s) [noise=true , address=145.79.227.20:4438 , piece_id=ZP4ZERSBOEXUVUKZBD5LTH3UWFWVAAA4GZUYGMBAXRFW65GUB7RA, seg_pos={0 0}, upload=478840]: Done uploading piece. err: <nil> Dec 28 09:32:50.599424351 (546h3m50.470507386s) [noise=true , address=145.79.227.42:4123 , piece_id=NNZVDJNHQIEHSHYLZIWBOUBSG5UGAZV7KRPFW5O633IXNNMKTBFA, seg_pos={0 0}, upload=478840]: Done uploading piece. err: <nil> Dec 28 09:32:50.600993699 (546h3m50.472076744s) [noise=true , address=145.79.227.41:4086 , piece_id=WBSHPU26MOBWO3WRAMBUMRCCZ4CW7FCLMWC3AR2FL5S7LIRMESCA, seg_pos={0 0}, upload=478840]: Done uploading piece. err: <nil> Dec 28 09:32:50.608841283 (546h3m50.479924338s) [noise=true , address=145.79.227.47:4266 , piece_id=DW3U5UWPSJNPKDEKUF2Z4SIWDJZQSQMDVBG4S4C7XEG34OIOLECQ, seg_pos={0 0}, upload=478840]: Done uploading piece. err: <nil> Dec 28 09:32:50.614748762 (546h3m50.485831797s) [noise=true , address=145.79.227.14:4838 , piece_id=PAIRS7AXMJAMBQADBEVPM5P5IT4EUG5VW4EVKSMPDHCEHSGIVKUA, seg_pos={0 0}, upload=478840]: Done uploading piece. err: <nil> Dec 28 09:32:50.614892856 (546h3m50.485975921s) [noise=true , address=145.79.227.51:4409 , piece_id=AS2D7LBL4ZG6F7XTVI4FUAMJ3CMHXTPTZE7DU6Y76B6SL73SZ33Q, seg_pos={0 0}, upload=478840]: Done uploading piece. err: <nil> Dec 28 09:32:50.639871699 (546h3m50.510954745s) [noise=true , address=145.79.227.36:4518 , piece_id=GR4P64FEXICCBGSB33D76KGJ65QDFOLFE5KEPFJIPRCCHW57OJLA, seg_pos={0 0}, upload=478840]: Done uploading piece. err: <nil> Dec 28 09:32:50.655384319 (546h3m50.526467364s) [noise=true , address=145.79.227.19:4079 , piece_id=KFS2MLOLFAGUEYUY4B3XBAUHXAVANGK2RYWAZTO3MAMUXLNBISRA, seg_pos={0 0}, upload=478840]: Done uploading piece. err: <nil> Dec 28 09:32:50.829258795 (546h3m50.70034184s) [noise=true , address=45.91.116.12:4226 , piece_id=H2HDJ7N2XSHT3VFPHTLPJ7HXCM2WAFNG74CAZURGBZANYR3T6UWA, seg_pos={0 0}, upload=478840]: Done uploading piece. err: <nil> Dec 28 09:32:50.857304619 (546h3m50.728387664s) [noise=true , address=77.90.31.14:4661 , piece_id=ZAWG7JOFQYNXXA6XIRYBGVNWYZSRVBBXUB2RYKAGMN5F4BXHZE5Q, seg_pos={0 0}, upload=478840]: Done uploading piece. err: <nil> Dec 28 09:32:50.857329938 (546h3m50.728412993s) [seg_pos={0 0}, upload=478840]: Segment reached optimal threshold of 13 pieces. Dec 28 09:32:50.857636971 (546h3m50.728720017s) [noise=true , address=45.91.116.11:4117 , piece_id=T7EEWP5C3PV3TJMRIDNCUNXDI4TLNKGYCW2SSVJPGU7AOXSUD6QA, seg_pos={0 0}, upload=478840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52688->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52688->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52688->45.91.116.11:4117: use of closed network connection Dec 28 09:32:50.857659745 (546h3m50.72874278s) [noise=true , address=45.91.116.11:4117 , piece_id=T7EEWP5C3PV3TJMRIDNCUNXDI4TLNKGYCW2SSVJPGU7AOXSUD6QA, seg_pos={0 0}, upload=478840]: Long tail cancellation due to optimal threshold Dec 28 09:32:50.857704961 (546h3m50.728787996s) [noise=true , address=45.91.116.4:4937 , piece_id=QPAJWKZAS7JLYYK4H6Z3R5FG2VO7D4U2I6ALRCO5T2PTGIVHIAAA, seg_pos={0 0}, upload=478840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: read tcp 188.166.225.170:58902->45.91.116.4:4937: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 09:32:50.857714199 (546h3m50.728797244s) [noise=true , address=45.91.116.4:4937 , piece_id=QPAJWKZAS7JLYYK4H6Z3R5FG2VO7D4U2I6ALRCO5T2PTGIVHIAAA, seg_pos={0 0}, upload=478840]: Long tail cancellation due to optimal threshold Dec 28 09:32:50.857924570 (546h3m50.729007605s) [noise=true , address=77.90.31.16:4735 , piece_id=OORLTYOX2VRZSWLAFTDICVJ22ZYANR4475X7XIXQXGM5TTE3Q2AQ, seg_pos={0 0}, upload=478840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: read tcp 188.166.225.170:45320->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 09:32:50.857931403 (546h3m50.729014438s) [noise=true , address=77.90.31.16:4735 , piece_id=OORLTYOX2VRZSWLAFTDICVJ22ZYANR4475X7XIXQXGM5TTE3Q2AQ, seg_pos={0 0}, upload=478840]: Long tail cancellation due to optimal threshold Dec 28 09:32:50.857947463 (546h3m50.729030508s) [seg_pos={0 0}, upload=478840]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:32:50.858033507 (546h3m50.729116542s) [upload=478840]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:32:50.858041823 (546h3m50.729124858s) [upload=478840]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:32:51.142168850 (546h3m51.013251885s) [upload=478840]: Done uploading segments... Dec 28 09:32:51.142188176 (546h3m51.013271211s) [upload=478840]: Upload finished. err: <nil> Duration: 30.002800683s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [..X........X.X...X....X.X.....X.....XXX.XX...XX.XX..X...XXX.] duration: 5.001101688s trace: https://jaeger.dev.storj.io/trace/5a881e44b788246b FAIL (2025-12-28 09:32:37 UTC) orbiter warning: context deadline exceeded Duration: 5.001128319s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [XXX.....XX.X.....X......X..............X....X...X......XX...] duration: 5.697254109s trace: https://jaeger.dev.storj.io/trace/6fbec07b9aa45c0 FAIL (2025-12-28 09:32:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: SJ7DJZ7EGTXQA6QTYN7GYCT3JBHPPL4OGJ7KWZEKPIGPYF723TJQ) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: FZKNOUH4VDAKCDWXU2X2RZQS33JJOQEG52BQNCWCKZYODDSTLNHQ) context deadline exceeded ======LOGS====== Dec 28 09:29:32.522899569 (546h0m32.393982604s) [upload=478797]: Starting upload Dec 28 09:29:32.522980202 (546h0m32.394063237s) [upload=478797]: Done starting upload Dec 28 09:29:32.523031148 (546h0m32.394114194s) [upload=478797]: Uploading segments... Dec 28 09:29:32.523306413 (546h0m32.394389468s) [upload=478797]: Got next segment. Inline: false Dec 28 09:29:32.523363752 (546h0m32.394446797s) [upload=478797]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:29:32.523383901 (546h0m32.394466956s) [upload=478797]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:29:32.774133750 (546h0m32.645216795s) [seg_pos={0 0}, upload=478797]: Begin upload segment... Dec 28 09:29:32.774178084 (546h0m32.645261119s) [seg_pos={0 0}, upload=478797]: Done begin upload segment. Dec 28 09:29:32.774265410 (546h0m32.645348455s) [noise=true , address=45.91.116.4:4937 , piece_id=TMV2AJVIMTJBHSW5TIW6LTLKVJGBB6CIEBN3NQ4LO64BWA6FTP7A, seg_pos={0 0}, upload=478797]: Uploading piece... Dec 28 09:29:32.774342888 (546h0m32.645425933s) [noise=true , address=45.91.116.8:4048 , piece_id=FZKNOUH4VDAKCDWXU2X2RZQS33JJOQEG52BQNCWCKZYODDSTLNHQ, seg_pos={0 0}, upload=478797]: Uploading piece... Dec 28 09:29:32.774754853 (546h0m32.645837888s) [noise=true , address=145.79.227.34:4905 , piece_id=34F5FPLHRD3ZCDVWRULQLGPUETEFSF5GP3JSH2TYNOMDPDFNPJ3A, seg_pos={0 0}, upload=478797]: Uploading piece... Dec 28 09:29:32.775198487 (546h0m32.646281552s) [noise=true , address=45.91.116.10:4082 , piece_id=SCFNGIN4JFA3WZFVVBJ7VPBY5XS55LCAUNO7R4MOBGPU27RLRZUQ, seg_pos={0 0}, upload=478797]: Uploading piece... Dec 28 09:29:32.775426792 (546h0m32.646509827s) [noise=true , address=145.79.227.31:4797 , piece_id=KKZPOYVEGETX5KAVKRJXUEVMVXEHASEBK5YDRF3RI2AGX4WA6Y2Q, seg_pos={0 0}, upload=478797]: Uploading piece... Dec 28 09:29:32.775769775 (546h0m32.64685282s) [noise=true , address=45.91.116.5:4298 , piece_id=TCJUHKRWNC45EDJ7Z7BZJPJIHXGMCJXQI24NYUPKQ2WVF5W4U4JQ, seg_pos={0 0}, upload=478797]: Uploading piece... Dec 28 09:29:32.776079184 (546h0m32.647162219s) [noise=true , address=145.79.227.39:4013 , piece_id=JGXXLSXWKGZLJXTRYJBMJIB27B3N5RCZD6UDLEO6UFRKIN2YUHIA, seg_pos={0 0}, upload=478797]: Uploading piece... Dec 28 09:29:32.776140110 (546h0m32.647223145s) [noise=true , address=145.79.227.51:4409 , piece_id=RPIL4OPZAZPPEV6JTG74NM3RT464ULDHXKIJONKQDBT4UITVQGRQ, seg_pos={0 0}, upload=478797]: Uploading piece... Dec 28 09:29:32.776392832 (546h0m32.647475867s) [noise=true , address=145.79.227.5:4221 , piece_id=2NTZGZ2VSXNUIKHYEKCLFMMIN52ZPPMRH54FUZDEZGF3X54FLTFQ, seg_pos={0 0}, upload=478797]: Uploading piece... Dec 28 09:29:32.776513551 (546h0m32.647596586s) [noise=true , address=145.79.227.29:4943 , piece_id=MUR3D4Y4X2OKYUE7XSZ4SCECGH5MJ6AYFHEYEOA26F3PLNOHJ6EA, seg_pos={0 0}, upload=478797]: Uploading piece... Dec 28 09:29:32.776664638 (546h0m32.647747673s) [noise=true , address=145.79.227.3:4150 , piece_id=UIO4FL4LZCW3XWNHO3I2Z7KVLOQUJPZM22Y575ZZGP2Y6SU5KYIQ, seg_pos={0 0}, upload=478797]: Uploading piece... Dec 28 09:29:32.776842338 (546h0m32.647925373s) [noise=true , address=145.79.227.44:4193 , piece_id=YSU7PDSNU2B3MQDD7PX3KVUZEBVMDFQJSJFNF77N4RYKTWZ2X7LA, seg_pos={0 0}, upload=478797]: Uploading piece... Dec 28 09:29:32.776938720 (546h0m32.648021755s) [noise=true , address=145.79.227.37:4567 , piece_id=NROYE66QYS4D4AKHNTZYNZTBXMY466EKLZHDVL3JXZEFTTF7RRZQ, seg_pos={0 0}, upload=478797]: Uploading piece... Dec 28 09:29:32.777174289 (546h0m32.648257324s) [noise=true , address=45.91.116.7:4010 , piece_id=SJ7DJZ7EGTXQA6QTYN7GYCT3JBHPPL4OGJ7KWZEKPIGPYF723TJQ, seg_pos={0 0}, upload=478797]: Uploading piece... Dec 28 09:29:32.777201682 (546h0m32.648284717s) [noise=true , address=145.79.227.16:4912 , piece_id=UVEBV5XKTDAKARJ3W6RTS6ZBHLT2SHEFZSPH3HMDEX23IAHEOHNA, seg_pos={0 0}, upload=478797]: Uploading piece... Dec 28 09:29:32.777503757 (546h0m32.648586792s) [noise=true , address=45.91.116.6:4976 , piece_id=4T4YMJN6H5XFRASSVODYPCSNBKL5EQL74IKEYILVA6O2LL3MGPEQ, seg_pos={0 0}, upload=478797]: Uploading piece... Dec 28 09:29:34.485770338 (546h0m34.356853383s) [upload=478797]: Next returned nil segment Dec 28 09:29:34.485790767 (546h0m34.356873812s) [upload=478797]: Waiting for error group managing segments... Dec 28 09:29:34.770506965 (546h0m34.64159001s) [noise=true , address=145.79.227.3:4150 , piece_id=UIO4FL4LZCW3XWNHO3I2Z7KVLOQUJPZM22Y575ZZGP2Y6SU5KYIQ, seg_pos={0 0}, upload=478797]: Done uploading piece. err: <nil> Dec 28 09:29:34.779564151 (546h0m34.650647196s) [noise=true , address=145.79.227.29:4943 , piece_id=MUR3D4Y4X2OKYUE7XSZ4SCECGH5MJ6AYFHEYEOA26F3PLNOHJ6EA, seg_pos={0 0}, upload=478797]: Done uploading piece. err: <nil> Dec 28 09:29:34.785081707 (546h0m34.656164752s) [noise=true , address=145.79.227.51:4409 , piece_id=RPIL4OPZAZPPEV6JTG74NM3RT464ULDHXKIJONKQDBT4UITVQGRQ, seg_pos={0 0}, upload=478797]: Done uploading piece. err: <nil> Dec 28 09:29:34.785356420 (546h0m34.656439465s) [noise=true , address=145.79.227.31:4797 , piece_id=KKZPOYVEGETX5KAVKRJXUEVMVXEHASEBK5YDRF3RI2AGX4WA6Y2Q, seg_pos={0 0}, upload=478797]: Done uploading piece. err: <nil> Dec 28 09:29:34.786010325 (546h0m34.65709336s) [noise=true , address=145.79.227.5:4221 , piece_id=2NTZGZ2VSXNUIKHYEKCLFMMIN52ZPPMRH54FUZDEZGF3X54FLTFQ, seg_pos={0 0}, upload=478797]: Done uploading piece. err: <nil> Dec 28 09:29:34.800774789 (546h0m34.671857834s) [noise=true , address=145.79.227.39:4013 , piece_id=JGXXLSXWKGZLJXTRYJBMJIB27B3N5RCZD6UDLEO6UFRKIN2YUHIA, seg_pos={0 0}, upload=478797]: Done uploading piece. err: <nil> Dec 28 09:29:34.802604032 (546h0m34.673687077s) [noise=true , address=145.79.227.37:4567 , piece_id=NROYE66QYS4D4AKHNTZYNZTBXMY466EKLZHDVL3JXZEFTTF7RRZQ, seg_pos={0 0}, upload=478797]: Done uploading piece. err: <nil> Dec 28 09:29:34.807270047 (546h0m34.678353092s) [noise=true , address=145.79.227.16:4912 , piece_id=UVEBV5XKTDAKARJ3W6RTS6ZBHLT2SHEFZSPH3HMDEX23IAHEOHNA, seg_pos={0 0}, upload=478797]: Done uploading piece. err: <nil> Dec 28 09:29:35.029392366 (546h0m34.900475401s) [noise=true , address=45.91.116.10:4082 , piece_id=SCFNGIN4JFA3WZFVVBJ7VPBY5XS55LCAUNO7R4MOBGPU27RLRZUQ, seg_pos={0 0}, upload=478797]: Done uploading piece. err: <nil> Dec 28 09:29:35.092194653 (546h0m34.963277688s) [noise=true , address=45.91.116.4:4937 , piece_id=TMV2AJVIMTJBHSW5TIW6LTLKVJGBB6CIEBN3NQ4LO64BWA6FTP7A, seg_pos={0 0}, upload=478797]: Done uploading piece. err: <nil> Dec 28 09:29:35.135389393 (546h0m35.006472458s) [noise=true , address=45.91.116.6:4976 , piece_id=4T4YMJN6H5XFRASSVODYPCSNBKL5EQL74IKEYILVA6O2LL3MGPEQ, seg_pos={0 0}, upload=478797]: Done uploading piece. err: <nil> Dec 28 09:29:35.149608017 (546h0m35.020691052s) [noise=true , address=45.91.116.8:4048 , piece_id=FZKNOUH4VDAKCDWXU2X2RZQS33JJOQEG52BQNCWCKZYODDSTLNHQ, seg_pos={0 0}, upload=478797]: Done uploading piece. err: <nil> Dec 28 09:29:35.371309144 (546h0m35.242392189s) [noise=true , address=45.91.116.7:4010 , piece_id=SJ7DJZ7EGTXQA6QTYN7GYCT3JBHPPL4OGJ7KWZEKPIGPYF723TJQ, seg_pos={0 0}, upload=478797]: Done uploading piece. err: <nil> Dec 28 09:29:35.371335364 (546h0m35.242418409s) [seg_pos={0 0}, upload=478797]: Segment reached optimal threshold of 13 pieces. Dec 28 09:29:35.371599117 (546h0m35.242682152s) [noise=true , address=145.79.227.44:4193 , piece_id=YSU7PDSNU2B3MQDD7PX3KVUZEBVMDFQJSJFNF77N4RYKTWZ2X7LA, seg_pos={0 0}, upload=478797]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 28 09:29:35.371607011 (546h0m35.242690046s) [noise=true , address=145.79.227.44:4193 , piece_id=YSU7PDSNU2B3MQDD7PX3KVUZEBVMDFQJSJFNF77N4RYKTWZ2X7LA, seg_pos={0 0}, upload=478797]: Long tail cancellation due to optimal threshold Dec 28 09:29:35.371886804 (546h0m35.242969859s) [noise=true , address=45.91.116.5:4298 , piece_id=TCJUHKRWNC45EDJ7Z7BZJPJIHXGMCJXQI24NYUPKQ2WVF5W4U4JQ, seg_pos={0 0}, upload=478797]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:47040->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:47040->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47040->45.91.116.5:4298: use of closed network connection Dec 28 09:29:35.371908776 (546h0m35.242991821s) [noise=true , address=45.91.116.5:4298 , piece_id=TCJUHKRWNC45EDJ7Z7BZJPJIHXGMCJXQI24NYUPKQ2WVF5W4U4JQ, seg_pos={0 0}, upload=478797]: Long tail cancellation due to optimal threshold Dec 28 09:29:35.371974391 (546h0m35.243057426s) [noise=true , address=145.79.227.34:4905 , piece_id=34F5FPLHRD3ZCDVWRULQLGPUETEFSF5GP3JSH2TYNOMDPDFNPJ3A, seg_pos={0 0}, upload=478797]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 28 09:29:35.372029175 (546h0m35.24311224s) [noise=true , address=145.79.227.34:4905 , piece_id=34F5FPLHRD3ZCDVWRULQLGPUETEFSF5GP3JSH2TYNOMDPDFNPJ3A, seg_pos={0 0}, upload=478797]: Long tail cancellation due to optimal threshold Dec 28 09:29:35.372094389 (546h0m35.243177434s) [seg_pos={0 0}, upload=478797]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:29:35.372233895 (546h0m35.24331694s) [upload=478797]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:29:35.372241539 (546h0m35.243324574s) [upload=478797]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:29:35.665426965 (546h0m35.53651s) [upload=478797]: Done uploading segments... Dec 28 09:29:35.665451350 (546h0m35.536534396s) [upload=478797]: Upload finished. err: <nil> Duration: 5.697283115s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [X..X........X.X...X....X.X.....X.....XXX.XX...XX.XX..X...XXX] duration: 5.001653037s trace: https://jaeger.dev.storj.io/trace/5a881e44b788246b FAIL (2025-12-28 09:32:09 UTC) orbiter warning: context deadline exceeded Duration: 5.001674527s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)