Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [......X.........X...................X..XX.....X..X..........] duration: 5.962314664s trace: https://jaeger.dev.storj.io/trace/37d347f90dc836a FAIL (2025-12-28 10:29:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 28 10:20:52.978430940 (546h51m52.849513975s) [upload=479529]: Got next segment. Inline: false Dec 28 10:20:52.978465887 (546h51m52.849548922s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:20:52.978471527 (546h51m52.849554562s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:20:53.224037597 (546h51m53.095120632s) [seg_pos={0 0}, upload=479529]: Begin upload segment... Dec 28 10:20:53.224084497 (546h51m53.095167532s) [seg_pos={0 0}, upload=479529]: Done begin upload segment. Dec 28 10:20:53.224140062 (546h51m53.095223117s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224148639 (546h51m53.095231685s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224718384 (546h51m53.095801419s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225234898 (546h51m53.096317933s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225504882 (546h51m53.096587917s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226212750 (546h51m53.097295785s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226608573 (546h51m53.097691618s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226767135 (546h51m53.09785018s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227078098 (546h51m53.098161143s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227238392 (546h51m53.098321427s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227466998 (546h51m53.098550033s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227646309 (546h51m53.098729354s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228003639 (546h51m53.099086684s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228329440 (546h51m53.099412475s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228632276 (546h51m53.099715311s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228942016 (546h51m53.100025061s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:54.851414651 (546h51m54.722497686s) [upload=479529]: Next returned nil segment Dec 28 10:20:54.851429859 (546h51m54.722512894s) [upload=479529]: Waiting for error group managing segments... Dec 28 10:20:55.232471268 (546h51m55.103554303s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.232529519 (546h51m55.103612554s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.234914169 (546h51m55.105997204s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235465209 (546h51m55.106548244s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235615435 (546h51m55.10669847s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.236221559 (546h51m55.107304594s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246296964 (546h51m55.117379999s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246947282 (546h51m55.118030317s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.251425650 (546h51m55.122508695s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.256324468 (546h51m55.127407513s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.259608812 (546h51m55.130691847s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.483932663 (546h51m55.355015698s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551135548 (546h51m55.422218594s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551161338 (546h51m55.422244383s) [seg_pos={0 0}, upload=479529]: Segment reached optimal threshold of 13 pieces. Dec 28 10:20:55.551510112 (546h51m55.422593147s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection Dec 28 10:20:55.551527014 (546h51m55.422610059s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551578462 (546h51m55.422661507s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 28 10:20:55.551591066 (546h51m55.422674121s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551835822 (546h51m55.422918867s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection Dec 28 10:20:55.551845641 (546h51m55.422928676s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551895075 (546h51m55.42297812s) [seg_pos={0 0}, upload=479529]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:20:55.552135894 (546h51m55.423218929s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:20:55.552144850 (546h51m55.423227885s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:20:55.854005394 (546h51m55.72508845s) [upload=479529]: Done uploading segments... Dec 28 10:20:55.854081920 (546h51m55.725164965s) [upload=479529]: Upload finished. err: <nil> Duration: 5.962338047s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.X....XXXXX...X........X.....X..................X...........] duration: 3.694809968s trace: https://jaeger.dev.storj.io/trace/49dbd650238e8f85 FAIL (2025-12-28 10:28:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.694837891s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [X.X....XXXXX...X........X.....X..................X..........] duration: 4.060766814s trace: https://jaeger.dev.storj.io/trace/49dbd650238e8f85 FAIL (2025-12-28 10:28:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.060810607s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......X.........X...................X..XX.....X..X.........] duration: 5.732544972s trace: https://jaeger.dev.storj.io/trace/37d347f90dc836a FAIL (2025-12-28 10:28:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 28 10:20:52.978430940 (546h51m52.849513975s) [upload=479529]: Got next segment. Inline: false Dec 28 10:20:52.978465887 (546h51m52.849548922s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:20:52.978471527 (546h51m52.849554562s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:20:53.224037597 (546h51m53.095120632s) [seg_pos={0 0}, upload=479529]: Begin upload segment... Dec 28 10:20:53.224084497 (546h51m53.095167532s) [seg_pos={0 0}, upload=479529]: Done begin upload segment. Dec 28 10:20:53.224140062 (546h51m53.095223117s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224148639 (546h51m53.095231685s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224718384 (546h51m53.095801419s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225234898 (546h51m53.096317933s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225504882 (546h51m53.096587917s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226212750 (546h51m53.097295785s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226608573 (546h51m53.097691618s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226767135 (546h51m53.09785018s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227078098 (546h51m53.098161143s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227238392 (546h51m53.098321427s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227466998 (546h51m53.098550033s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227646309 (546h51m53.098729354s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228003639 (546h51m53.099086684s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228329440 (546h51m53.099412475s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228632276 (546h51m53.099715311s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228942016 (546h51m53.100025061s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:54.851414651 (546h51m54.722497686s) [upload=479529]: Next returned nil segment Dec 28 10:20:54.851429859 (546h51m54.722512894s) [upload=479529]: Waiting for error group managing segments... Dec 28 10:20:55.232471268 (546h51m55.103554303s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.232529519 (546h51m55.103612554s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.234914169 (546h51m55.105997204s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235465209 (546h51m55.106548244s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235615435 (546h51m55.10669847s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.236221559 (546h51m55.107304594s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246296964 (546h51m55.117379999s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246947282 (546h51m55.118030317s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.251425650 (546h51m55.122508695s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.256324468 (546h51m55.127407513s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.259608812 (546h51m55.130691847s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.483932663 (546h51m55.355015698s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551135548 (546h51m55.422218594s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551161338 (546h51m55.422244383s) [seg_pos={0 0}, upload=479529]: Segment reached optimal threshold of 13 pieces. Dec 28 10:20:55.551510112 (546h51m55.422593147s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection Dec 28 10:20:55.551527014 (546h51m55.422610059s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551578462 (546h51m55.422661507s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 28 10:20:55.551591066 (546h51m55.422674121s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551835822 (546h51m55.422918867s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection Dec 28 10:20:55.551845641 (546h51m55.422928676s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551895075 (546h51m55.42297812s) [seg_pos={0 0}, upload=479529]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:20:55.552135894 (546h51m55.423218929s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:20:55.552144850 (546h51m55.423227885s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:20:55.854005394 (546h51m55.72508845s) [upload=479529]: Done uploading segments... Dec 28 10:20:55.854081920 (546h51m55.725164965s) [upload=479529]: Upload finished. err: <nil> Duration: 5.732582404s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [.X.X....XXXXX...X........X.....X..................X.........] duration: 3.642689203s trace: https://jaeger.dev.storj.io/trace/49dbd650238e8f85 FAIL (2025-12-28 10:27:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.642717126s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [........X.........X...................X..XX.....X..X........] duration: 6.042706984s trace: https://jaeger.dev.storj.io/trace/37d347f90dc836a FAIL (2025-12-28 10:27:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 28 10:20:52.978430940 (546h51m52.849513975s) [upload=479529]: Got next segment. Inline: false Dec 28 10:20:52.978465887 (546h51m52.849548922s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:20:52.978471527 (546h51m52.849554562s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:20:53.224037597 (546h51m53.095120632s) [seg_pos={0 0}, upload=479529]: Begin upload segment... Dec 28 10:20:53.224084497 (546h51m53.095167532s) [seg_pos={0 0}, upload=479529]: Done begin upload segment. Dec 28 10:20:53.224140062 (546h51m53.095223117s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224148639 (546h51m53.095231685s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224718384 (546h51m53.095801419s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225234898 (546h51m53.096317933s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225504882 (546h51m53.096587917s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226212750 (546h51m53.097295785s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226608573 (546h51m53.097691618s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226767135 (546h51m53.09785018s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227078098 (546h51m53.098161143s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227238392 (546h51m53.098321427s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227466998 (546h51m53.098550033s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227646309 (546h51m53.098729354s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228003639 (546h51m53.099086684s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228329440 (546h51m53.099412475s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228632276 (546h51m53.099715311s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228942016 (546h51m53.100025061s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:54.851414651 (546h51m54.722497686s) [upload=479529]: Next returned nil segment Dec 28 10:20:54.851429859 (546h51m54.722512894s) [upload=479529]: Waiting for error group managing segments... Dec 28 10:20:55.232471268 (546h51m55.103554303s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.232529519 (546h51m55.103612554s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.234914169 (546h51m55.105997204s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235465209 (546h51m55.106548244s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235615435 (546h51m55.10669847s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.236221559 (546h51m55.107304594s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246296964 (546h51m55.117379999s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246947282 (546h51m55.118030317s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.251425650 (546h51m55.122508695s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.256324468 (546h51m55.127407513s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.259608812 (546h51m55.130691847s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.483932663 (546h51m55.355015698s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551135548 (546h51m55.422218594s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551161338 (546h51m55.422244383s) [seg_pos={0 0}, upload=479529]: Segment reached optimal threshold of 13 pieces. Dec 28 10:20:55.551510112 (546h51m55.422593147s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection Dec 28 10:20:55.551527014 (546h51m55.422610059s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551578462 (546h51m55.422661507s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 28 10:20:55.551591066 (546h51m55.422674121s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551835822 (546h51m55.422918867s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection Dec 28 10:20:55.551845641 (546h51m55.422928676s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551895075 (546h51m55.42297812s) [seg_pos={0 0}, upload=479529]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:20:55.552135894 (546h51m55.423218929s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:20:55.552144850 (546h51m55.423227885s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:20:55.854005394 (546h51m55.72508845s) [upload=479529]: Done uploading segments... Dec 28 10:20:55.854081920 (546h51m55.725164965s) [upload=479529]: Upload finished. err: <nil> Duration: 6.042734827s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [..X.X....XXXXX...X........X.....X..................X........] duration: 3.954679955s trace: https://jaeger.dev.storj.io/trace/49dbd650238e8f85 FAIL (2025-12-28 10:26:59 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.954705954s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........X.........X...................X..XX.....X..X.......] duration: 5.807320389s trace: https://jaeger.dev.storj.io/trace/37d347f90dc836a FAIL (2025-12-28 10:26:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 28 10:20:52.978430940 (546h51m52.849513975s) [upload=479529]: Got next segment. Inline: false Dec 28 10:20:52.978465887 (546h51m52.849548922s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:20:52.978471527 (546h51m52.849554562s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:20:53.224037597 (546h51m53.095120632s) [seg_pos={0 0}, upload=479529]: Begin upload segment... Dec 28 10:20:53.224084497 (546h51m53.095167532s) [seg_pos={0 0}, upload=479529]: Done begin upload segment. Dec 28 10:20:53.224140062 (546h51m53.095223117s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224148639 (546h51m53.095231685s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224718384 (546h51m53.095801419s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225234898 (546h51m53.096317933s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225504882 (546h51m53.096587917s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226212750 (546h51m53.097295785s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226608573 (546h51m53.097691618s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226767135 (546h51m53.09785018s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227078098 (546h51m53.098161143s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227238392 (546h51m53.098321427s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227466998 (546h51m53.098550033s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227646309 (546h51m53.098729354s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228003639 (546h51m53.099086684s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228329440 (546h51m53.099412475s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228632276 (546h51m53.099715311s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228942016 (546h51m53.100025061s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:54.851414651 (546h51m54.722497686s) [upload=479529]: Next returned nil segment Dec 28 10:20:54.851429859 (546h51m54.722512894s) [upload=479529]: Waiting for error group managing segments... Dec 28 10:20:55.232471268 (546h51m55.103554303s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.232529519 (546h51m55.103612554s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.234914169 (546h51m55.105997204s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235465209 (546h51m55.106548244s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235615435 (546h51m55.10669847s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.236221559 (546h51m55.107304594s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246296964 (546h51m55.117379999s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246947282 (546h51m55.118030317s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.251425650 (546h51m55.122508695s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.256324468 (546h51m55.127407513s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.259608812 (546h51m55.130691847s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.483932663 (546h51m55.355015698s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551135548 (546h51m55.422218594s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551161338 (546h51m55.422244383s) [seg_pos={0 0}, upload=479529]: Segment reached optimal threshold of 13 pieces. Dec 28 10:20:55.551510112 (546h51m55.422593147s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection Dec 28 10:20:55.551527014 (546h51m55.422610059s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551578462 (546h51m55.422661507s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 28 10:20:55.551591066 (546h51m55.422674121s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551835822 (546h51m55.422918867s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection Dec 28 10:20:55.551845641 (546h51m55.422928676s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551895075 (546h51m55.42297812s) [seg_pos={0 0}, upload=479529]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:20:55.552135894 (546h51m55.423218929s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:20:55.552144850 (546h51m55.423227885s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:20:55.854005394 (546h51m55.72508845s) [upload=479529]: Done uploading segments... Dec 28 10:20:55.854081920 (546h51m55.725164965s) [upload=479529]: Upload finished. err: <nil> Duration: 5.807352078s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [...X.X....XXXXX...X........X.....X..................X.......] duration: 3.577011523s trace: https://jaeger.dev.storj.io/trace/49dbd650238e8f85 FAIL (2025-12-28 10:26:01 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.577033875s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X.........X.........X...................X..XX.....X..X......] duration: 5.693173504s trace: https://jaeger.dev.storj.io/trace/37d347f90dc836a FAIL (2025-12-28 10:25:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 28 10:20:52.978430940 (546h51m52.849513975s) [upload=479529]: Got next segment. Inline: false Dec 28 10:20:52.978465887 (546h51m52.849548922s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:20:52.978471527 (546h51m52.849554562s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:20:53.224037597 (546h51m53.095120632s) [seg_pos={0 0}, upload=479529]: Begin upload segment... Dec 28 10:20:53.224084497 (546h51m53.095167532s) [seg_pos={0 0}, upload=479529]: Done begin upload segment. Dec 28 10:20:53.224140062 (546h51m53.095223117s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224148639 (546h51m53.095231685s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224718384 (546h51m53.095801419s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225234898 (546h51m53.096317933s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225504882 (546h51m53.096587917s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226212750 (546h51m53.097295785s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226608573 (546h51m53.097691618s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226767135 (546h51m53.09785018s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227078098 (546h51m53.098161143s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227238392 (546h51m53.098321427s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227466998 (546h51m53.098550033s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227646309 (546h51m53.098729354s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228003639 (546h51m53.099086684s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228329440 (546h51m53.099412475s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228632276 (546h51m53.099715311s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228942016 (546h51m53.100025061s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:54.851414651 (546h51m54.722497686s) [upload=479529]: Next returned nil segment Dec 28 10:20:54.851429859 (546h51m54.722512894s) [upload=479529]: Waiting for error group managing segments... Dec 28 10:20:55.232471268 (546h51m55.103554303s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.232529519 (546h51m55.103612554s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.234914169 (546h51m55.105997204s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235465209 (546h51m55.106548244s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235615435 (546h51m55.10669847s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.236221559 (546h51m55.107304594s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246296964 (546h51m55.117379999s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246947282 (546h51m55.118030317s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.251425650 (546h51m55.122508695s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.256324468 (546h51m55.127407513s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.259608812 (546h51m55.130691847s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.483932663 (546h51m55.355015698s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551135548 (546h51m55.422218594s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551161338 (546h51m55.422244383s) [seg_pos={0 0}, upload=479529]: Segment reached optimal threshold of 13 pieces. Dec 28 10:20:55.551510112 (546h51m55.422593147s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection Dec 28 10:20:55.551527014 (546h51m55.422610059s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551578462 (546h51m55.422661507s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 28 10:20:55.551591066 (546h51m55.422674121s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551835822 (546h51m55.422918867s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection Dec 28 10:20:55.551845641 (546h51m55.422928676s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551895075 (546h51m55.42297812s) [seg_pos={0 0}, upload=479529]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:20:55.552135894 (546h51m55.423218929s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:20:55.552144850 (546h51m55.423227885s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:20:55.854005394 (546h51m55.72508845s) [upload=479529]: Done uploading segments... Dec 28 10:20:55.854081920 (546h51m55.725164965s) [upload=479529]: Upload finished. err: <nil> Duration: 5.693200675s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [X...X.X....XXXXX...X........X.....X..................X......] duration: 5.000566401s trace: https://jaeger.dev.storj.io/trace/49dbd650238e8f85 FAIL (2025-12-28 10:25:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.0005979s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X.........X.........X...................X..XX.....X..X.....] duration: 6.064192947s trace: https://jaeger.dev.storj.io/trace/37d347f90dc836a FAIL (2025-12-28 10:24:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 28 10:20:52.978430940 (546h51m52.849513975s) [upload=479529]: Got next segment. Inline: false Dec 28 10:20:52.978465887 (546h51m52.849548922s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:20:52.978471527 (546h51m52.849554562s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:20:53.224037597 (546h51m53.095120632s) [seg_pos={0 0}, upload=479529]: Begin upload segment... Dec 28 10:20:53.224084497 (546h51m53.095167532s) [seg_pos={0 0}, upload=479529]: Done begin upload segment. Dec 28 10:20:53.224140062 (546h51m53.095223117s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224148639 (546h51m53.095231685s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224718384 (546h51m53.095801419s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225234898 (546h51m53.096317933s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225504882 (546h51m53.096587917s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226212750 (546h51m53.097295785s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226608573 (546h51m53.097691618s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226767135 (546h51m53.09785018s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227078098 (546h51m53.098161143s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227238392 (546h51m53.098321427s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227466998 (546h51m53.098550033s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227646309 (546h51m53.098729354s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228003639 (546h51m53.099086684s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228329440 (546h51m53.099412475s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228632276 (546h51m53.099715311s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228942016 (546h51m53.100025061s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:54.851414651 (546h51m54.722497686s) [upload=479529]: Next returned nil segment Dec 28 10:20:54.851429859 (546h51m54.722512894s) [upload=479529]: Waiting for error group managing segments... Dec 28 10:20:55.232471268 (546h51m55.103554303s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.232529519 (546h51m55.103612554s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.234914169 (546h51m55.105997204s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235465209 (546h51m55.106548244s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235615435 (546h51m55.10669847s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.236221559 (546h51m55.107304594s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246296964 (546h51m55.117379999s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246947282 (546h51m55.118030317s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.251425650 (546h51m55.122508695s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.256324468 (546h51m55.127407513s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.259608812 (546h51m55.130691847s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.483932663 (546h51m55.355015698s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551135548 (546h51m55.422218594s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551161338 (546h51m55.422244383s) [seg_pos={0 0}, upload=479529]: Segment reached optimal threshold of 13 pieces. Dec 28 10:20:55.551510112 (546h51m55.422593147s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection Dec 28 10:20:55.551527014 (546h51m55.422610059s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551578462 (546h51m55.422661507s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 28 10:20:55.551591066 (546h51m55.422674121s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551835822 (546h51m55.422918867s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection Dec 28 10:20:55.551845641 (546h51m55.422928676s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551895075 (546h51m55.42297812s) [seg_pos={0 0}, upload=479529]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:20:55.552135894 (546h51m55.423218929s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:20:55.552144850 (546h51m55.423227885s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:20:55.854005394 (546h51m55.72508845s) [upload=479529]: Done uploading segments... Dec 28 10:20:55.854081920 (546h51m55.725164965s) [upload=479529]: Upload finished. err: <nil> Duration: 6.06422018s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [.X...X.X....XXXXX...X........X.....X..................X.....] duration: 3.758628108s trace: https://jaeger.dev.storj.io/trace/49dbd650238e8f85 FAIL (2025-12-28 10:24:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.758660692s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..X.........X.........X...................X..XX.....X..X....] duration: 6.291490529s trace: https://jaeger.dev.storj.io/trace/37d347f90dc836a FAIL (2025-12-28 10:23:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 28 10:20:52.978430940 (546h51m52.849513975s) [upload=479529]: Got next segment. Inline: false Dec 28 10:20:52.978465887 (546h51m52.849548922s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:20:52.978471527 (546h51m52.849554562s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:20:53.224037597 (546h51m53.095120632s) [seg_pos={0 0}, upload=479529]: Begin upload segment... Dec 28 10:20:53.224084497 (546h51m53.095167532s) [seg_pos={0 0}, upload=479529]: Done begin upload segment. Dec 28 10:20:53.224140062 (546h51m53.095223117s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224148639 (546h51m53.095231685s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224718384 (546h51m53.095801419s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225234898 (546h51m53.096317933s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225504882 (546h51m53.096587917s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226212750 (546h51m53.097295785s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226608573 (546h51m53.097691618s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226767135 (546h51m53.09785018s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227078098 (546h51m53.098161143s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227238392 (546h51m53.098321427s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227466998 (546h51m53.098550033s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227646309 (546h51m53.098729354s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228003639 (546h51m53.099086684s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228329440 (546h51m53.099412475s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228632276 (546h51m53.099715311s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228942016 (546h51m53.100025061s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:54.851414651 (546h51m54.722497686s) [upload=479529]: Next returned nil segment Dec 28 10:20:54.851429859 (546h51m54.722512894s) [upload=479529]: Waiting for error group managing segments... Dec 28 10:20:55.232471268 (546h51m55.103554303s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.232529519 (546h51m55.103612554s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.234914169 (546h51m55.105997204s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235465209 (546h51m55.106548244s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235615435 (546h51m55.10669847s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.236221559 (546h51m55.107304594s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246296964 (546h51m55.117379999s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246947282 (546h51m55.118030317s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.251425650 (546h51m55.122508695s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.256324468 (546h51m55.127407513s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.259608812 (546h51m55.130691847s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.483932663 (546h51m55.355015698s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551135548 (546h51m55.422218594s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551161338 (546h51m55.422244383s) [seg_pos={0 0}, upload=479529]: Segment reached optimal threshold of 13 pieces. Dec 28 10:20:55.551510112 (546h51m55.422593147s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection Dec 28 10:20:55.551527014 (546h51m55.422610059s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551578462 (546h51m55.422661507s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 28 10:20:55.551591066 (546h51m55.422674121s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551835822 (546h51m55.422918867s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection Dec 28 10:20:55.551845641 (546h51m55.422928676s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551895075 (546h51m55.42297812s) [seg_pos={0 0}, upload=479529]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:20:55.552135894 (546h51m55.423218929s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:20:55.552144850 (546h51m55.423227885s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:20:55.854005394 (546h51m55.72508845s) [upload=479529]: Done uploading segments... Dec 28 10:20:55.854081920 (546h51m55.725164965s) [upload=479529]: Upload finished. err: <nil> Duration: 6.291517692s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X..X.........X.........X...................X..XX.....X..X...] duration: 5.659040658s trace: https://jaeger.dev.storj.io/trace/37d347f90dc836a FAIL (2025-12-28 10:22:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 28 10:20:52.978430940 (546h51m52.849513975s) [upload=479529]: Got next segment. Inline: false Dec 28 10:20:52.978465887 (546h51m52.849548922s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:20:52.978471527 (546h51m52.849554562s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:20:53.224037597 (546h51m53.095120632s) [seg_pos={0 0}, upload=479529]: Begin upload segment... Dec 28 10:20:53.224084497 (546h51m53.095167532s) [seg_pos={0 0}, upload=479529]: Done begin upload segment. Dec 28 10:20:53.224140062 (546h51m53.095223117s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224148639 (546h51m53.095231685s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224718384 (546h51m53.095801419s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225234898 (546h51m53.096317933s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225504882 (546h51m53.096587917s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226212750 (546h51m53.097295785s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226608573 (546h51m53.097691618s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226767135 (546h51m53.09785018s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227078098 (546h51m53.098161143s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227238392 (546h51m53.098321427s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227466998 (546h51m53.098550033s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227646309 (546h51m53.098729354s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228003639 (546h51m53.099086684s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228329440 (546h51m53.099412475s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228632276 (546h51m53.099715311s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228942016 (546h51m53.100025061s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:54.851414651 (546h51m54.722497686s) [upload=479529]: Next returned nil segment Dec 28 10:20:54.851429859 (546h51m54.722512894s) [upload=479529]: Waiting for error group managing segments... Dec 28 10:20:55.232471268 (546h51m55.103554303s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.232529519 (546h51m55.103612554s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.234914169 (546h51m55.105997204s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235465209 (546h51m55.106548244s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235615435 (546h51m55.10669847s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.236221559 (546h51m55.107304594s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246296964 (546h51m55.117379999s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246947282 (546h51m55.118030317s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.251425650 (546h51m55.122508695s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.256324468 (546h51m55.127407513s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.259608812 (546h51m55.130691847s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.483932663 (546h51m55.355015698s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551135548 (546h51m55.422218594s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551161338 (546h51m55.422244383s) [seg_pos={0 0}, upload=479529]: Segment reached optimal threshold of 13 pieces. Dec 28 10:20:55.551510112 (546h51m55.422593147s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection Dec 28 10:20:55.551527014 (546h51m55.422610059s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551578462 (546h51m55.422661507s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 28 10:20:55.551591066 (546h51m55.422674121s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551835822 (546h51m55.422918867s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection Dec 28 10:20:55.551845641 (546h51m55.422928676s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551895075 (546h51m55.42297812s) [seg_pos={0 0}, upload=479529]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:20:55.552135894 (546h51m55.423218929s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:20:55.552144850 (546h51m55.423227885s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:20:55.854005394 (546h51m55.72508845s) [upload=479529]: Done uploading segments... Dec 28 10:20:55.854081920 (546h51m55.725164965s) [upload=479529]: Upload finished. err: <nil> Duration: 5.65907854s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [X.X...X.X....XXXXX...X........X.....X..................X....] duration: 3.383932415s trace: https://jaeger.dev.storj.io/trace/49dbd650238e8f85 FAIL (2025-12-28 10:22:59 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.383955088s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [XX.X...X.X....XXXXX...X........X.....X..................X...] duration: 3.96605189s trace: https://jaeger.dev.storj.io/trace/49dbd650238e8f85 FAIL (2025-12-28 10:21:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.96607902s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X..X.........X.........X...................X..XX.....X..X..] duration: 5.6355302s trace: https://jaeger.dev.storj.io/trace/37d347f90dc836a FAIL (2025-12-28 10:21:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 28 10:20:52.978430940 (546h51m52.849513975s) [upload=479529]: Got next segment. Inline: false Dec 28 10:20:52.978465887 (546h51m52.849548922s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:20:52.978471527 (546h51m52.849554562s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:20:53.224037597 (546h51m53.095120632s) [seg_pos={0 0}, upload=479529]: Begin upload segment... Dec 28 10:20:53.224084497 (546h51m53.095167532s) [seg_pos={0 0}, upload=479529]: Done begin upload segment. Dec 28 10:20:53.224140062 (546h51m53.095223117s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224148639 (546h51m53.095231685s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224718384 (546h51m53.095801419s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225234898 (546h51m53.096317933s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225504882 (546h51m53.096587917s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226212750 (546h51m53.097295785s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226608573 (546h51m53.097691618s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226767135 (546h51m53.09785018s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227078098 (546h51m53.098161143s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227238392 (546h51m53.098321427s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227466998 (546h51m53.098550033s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227646309 (546h51m53.098729354s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228003639 (546h51m53.099086684s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228329440 (546h51m53.099412475s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228632276 (546h51m53.099715311s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228942016 (546h51m53.100025061s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:54.851414651 (546h51m54.722497686s) [upload=479529]: Next returned nil segment Dec 28 10:20:54.851429859 (546h51m54.722512894s) [upload=479529]: Waiting for error group managing segments... Dec 28 10:20:55.232471268 (546h51m55.103554303s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.232529519 (546h51m55.103612554s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.234914169 (546h51m55.105997204s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235465209 (546h51m55.106548244s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235615435 (546h51m55.10669847s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.236221559 (546h51m55.107304594s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246296964 (546h51m55.117379999s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246947282 (546h51m55.118030317s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.251425650 (546h51m55.122508695s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.256324468 (546h51m55.127407513s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.259608812 (546h51m55.130691847s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.483932663 (546h51m55.355015698s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551135548 (546h51m55.422218594s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551161338 (546h51m55.422244383s) [seg_pos={0 0}, upload=479529]: Segment reached optimal threshold of 13 pieces. Dec 28 10:20:55.551510112 (546h51m55.422593147s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection Dec 28 10:20:55.551527014 (546h51m55.422610059s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551578462 (546h51m55.422661507s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 28 10:20:55.551591066 (546h51m55.422674121s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551835822 (546h51m55.422918867s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection Dec 28 10:20:55.551845641 (546h51m55.422928676s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551895075 (546h51m55.42297812s) [seg_pos={0 0}, upload=479529]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:20:55.552135894 (546h51m55.423218929s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:20:55.552144850 (546h51m55.423227885s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:20:55.854005394 (546h51m55.72508845s) [upload=479529]: Done uploading segments... Dec 28 10:20:55.854081920 (546h51m55.725164965s) [upload=479529]: Upload finished. err: <nil> Duration: 5.635560497s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..X..X.........X.........X...................X..XX.....X..X.] duration: 6.018281723s trace: https://jaeger.dev.storj.io/trace/37d347f90dc836a FAIL (2025-12-28 10:21:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 28 10:20:52.978430940 (546h51m52.849513975s) [upload=479529]: Got next segment. Inline: false Dec 28 10:20:52.978465887 (546h51m52.849548922s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:20:52.978471527 (546h51m52.849554562s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:20:53.224037597 (546h51m53.095120632s) [seg_pos={0 0}, upload=479529]: Begin upload segment... Dec 28 10:20:53.224084497 (546h51m53.095167532s) [seg_pos={0 0}, upload=479529]: Done begin upload segment. Dec 28 10:20:53.224140062 (546h51m53.095223117s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224148639 (546h51m53.095231685s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224718384 (546h51m53.095801419s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225234898 (546h51m53.096317933s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225504882 (546h51m53.096587917s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226212750 (546h51m53.097295785s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226608573 (546h51m53.097691618s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226767135 (546h51m53.09785018s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227078098 (546h51m53.098161143s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227238392 (546h51m53.098321427s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227466998 (546h51m53.098550033s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227646309 (546h51m53.098729354s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228003639 (546h51m53.099086684s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228329440 (546h51m53.099412475s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228632276 (546h51m53.099715311s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228942016 (546h51m53.100025061s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:54.851414651 (546h51m54.722497686s) [upload=479529]: Next returned nil segment Dec 28 10:20:54.851429859 (546h51m54.722512894s) [upload=479529]: Waiting for error group managing segments... Dec 28 10:20:55.232471268 (546h51m55.103554303s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.232529519 (546h51m55.103612554s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.234914169 (546h51m55.105997204s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235465209 (546h51m55.106548244s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235615435 (546h51m55.10669847s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.236221559 (546h51m55.107304594s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246296964 (546h51m55.117379999s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246947282 (546h51m55.118030317s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.251425650 (546h51m55.122508695s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.256324468 (546h51m55.127407513s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.259608812 (546h51m55.130691847s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.483932663 (546h51m55.355015698s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551135548 (546h51m55.422218594s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551161338 (546h51m55.422244383s) [seg_pos={0 0}, upload=479529]: Segment reached optimal threshold of 13 pieces. Dec 28 10:20:55.551510112 (546h51m55.422593147s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection Dec 28 10:20:55.551527014 (546h51m55.422610059s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551578462 (546h51m55.422661507s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 28 10:20:55.551591066 (546h51m55.422674121s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551835822 (546h51m55.422918867s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection Dec 28 10:20:55.551845641 (546h51m55.422928676s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551895075 (546h51m55.42297812s) [seg_pos={0 0}, upload=479529]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:20:55.552135894 (546h51m55.423218929s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:20:55.552144850 (546h51m55.423227885s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:20:55.854005394 (546h51m55.72508845s) [upload=479529]: Done uploading segments... Dec 28 10:20:55.854081920 (546h51m55.725164965s) [upload=479529]: Upload finished. err: <nil> Duration: 6.018323062s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...X..X.........X.........X...................X..XX.....X..X] duration: 30.002734301s trace: https://jaeger.dev.storj.io/trace/37d347f90dc836a FAIL (2025-12-28 10:21:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 28 10:20:52.978430940 (546h51m52.849513975s) [upload=479529]: Got next segment. Inline: false Dec 28 10:20:52.978465887 (546h51m52.849548922s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:20:52.978471527 (546h51m52.849554562s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:20:53.224037597 (546h51m53.095120632s) [seg_pos={0 0}, upload=479529]: Begin upload segment... Dec 28 10:20:53.224084497 (546h51m53.095167532s) [seg_pos={0 0}, upload=479529]: Done begin upload segment. Dec 28 10:20:53.224140062 (546h51m53.095223117s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224148639 (546h51m53.095231685s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224718384 (546h51m53.095801419s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225234898 (546h51m53.096317933s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225504882 (546h51m53.096587917s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226212750 (546h51m53.097295785s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226608573 (546h51m53.097691618s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226767135 (546h51m53.09785018s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227078098 (546h51m53.098161143s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227238392 (546h51m53.098321427s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227466998 (546h51m53.098550033s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227646309 (546h51m53.098729354s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228003639 (546h51m53.099086684s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228329440 (546h51m53.099412475s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228632276 (546h51m53.099715311s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228942016 (546h51m53.100025061s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:54.851414651 (546h51m54.722497686s) [upload=479529]: Next returned nil segment Dec 28 10:20:54.851429859 (546h51m54.722512894s) [upload=479529]: Waiting for error group managing segments... Dec 28 10:20:55.232471268 (546h51m55.103554303s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.232529519 (546h51m55.103612554s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.234914169 (546h51m55.105997204s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235465209 (546h51m55.106548244s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235615435 (546h51m55.10669847s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.236221559 (546h51m55.107304594s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246296964 (546h51m55.117379999s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246947282 (546h51m55.118030317s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.251425650 (546h51m55.122508695s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.256324468 (546h51m55.127407513s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.259608812 (546h51m55.130691847s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.483932663 (546h51m55.355015698s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551135548 (546h51m55.422218594s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551161338 (546h51m55.422244383s) [seg_pos={0 0}, upload=479529]: Segment reached optimal threshold of 13 pieces. Dec 28 10:20:55.551510112 (546h51m55.422593147s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection Dec 28 10:20:55.551527014 (546h51m55.422610059s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551578462 (546h51m55.422661507s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 28 10:20:55.551591066 (546h51m55.422674121s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551835822 (546h51m55.422918867s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection Dec 28 10:20:55.551845641 (546h51m55.422928676s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551895075 (546h51m55.42297812s) [seg_pos={0 0}, upload=479529]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:20:55.552135894 (546h51m55.423218929s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:20:55.552144850 (546h51m55.423227885s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:20:55.854005394 (546h51m55.72508845s) [upload=479529]: Done uploading segments... Dec 28 10:20:55.854081920 (546h51m55.725164965s) [upload=479529]: Upload finished. err: <nil> Duration: 30.002757074s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [.XX.X...X.X....XXXXX...X........X.....X..................X..] duration: 3.560916161s trace: https://jaeger.dev.storj.io/trace/49dbd650238e8f85 FAIL (2025-12-28 10:20:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.560944324s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X...X..X.........X.........X...................X..XX.....X..] duration: 5.989482996s trace: https://jaeger.dev.storj.io/trace/67a1aca7935ff554 FAIL (2025-12-28 10:20:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: AN4NJQHGJZOFQV7ORYVBHYUYI342VNU3DP3NX7N2NM2BMYCFFOPQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: 3D7RRA5BWEZ53QEIM7STRQUDL2RSEBMXCR3EFUSGVL3MZDWMW37Q) context deadline exceeded ======LOGS====== Dec 28 10:18:15.232717407 (546h49m15.103800452s) [upload=479491]: Starting upload Dec 28 10:18:15.232895657 (546h49m15.103978702s) [upload=479491]: Done starting upload Dec 28 10:18:15.232960500 (546h49m15.104043545s) [upload=479491]: Uploading segments... Dec 28 10:18:15.233147817 (546h49m15.104230852s) [upload=479491]: Got next segment. Inline: false Dec 28 10:18:15.233190378 (546h49m15.104273433s) [upload=479491]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:18:15.233196840 (546h49m15.104279875s) [upload=479491]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:18:15.478852011 (546h49m15.349935066s) [seg_pos={0 0}, upload=479491]: Begin upload segment... Dec 28 10:18:15.478918126 (546h49m15.350001171s) [seg_pos={0 0}, upload=479491]: Done begin upload segment. Dec 28 10:18:15.478925210 (546h49m15.350008245s) [noise=true , address=45.91.116.9:4264 , piece_id=6IKCH5SFYCBIYSZIKBFEK2EP2MOILJBOQA2VQXY5TLU4S273OKDQ, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.478949356 (546h49m15.350032391s) [noise=true , address=77.90.31.10:4519 , piece_id=AVBSS2GLD7GWCVSGJU3WV3JYEZEAFD4R7B563EISYQWYRHNLW74Q, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.479090785 (546h49m15.35017383s) [noise=true , address=145.79.227.38:4977 , piece_id=UF2GVCDHOW7W2XKK43EY7I45THJC7UOMWKS6KEFGJ2TMIZYFABBA, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.479858939 (546h49m15.350941984s) [noise=true , address=145.79.227.14:4840 , piece_id=T6JJ54YDOUNNYOF2KI3ZGY25EYJTSMFH74S6B5S7MIJO2DTWP62Q, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.480268197 (546h49m15.351351242s) [noise=true , address=145.79.227.23:4547 , piece_id=SU36GD6KUH6QKK2RQRUHGWIG7VCZPTNAQG6CETI7CYP5OA3MD5VA, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.480626941 (546h49m15.351709976s) [noise=true , address=45.91.116.6:4976 , piece_id=AN4NJQHGJZOFQV7ORYVBHYUYI342VNU3DP3NX7N2NM2BMYCFFOPQ, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.480984702 (546h49m15.352067757s) [noise=true , address=45.91.116.14:4192 , piece_id=3D7RRA5BWEZ53QEIM7STRQUDL2RSEBMXCR3EFUSGVL3MZDWMW37Q, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.481363664 (546h49m15.352446709s) [noise=true , address=145.79.227.50:4373 , piece_id=3MF6H25P3PXZR6CRB6QSMMUG5G7ACGHJK5XDPMFZ7JOKYDETEIOQ, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.481740762 (546h49m15.352823797s) [noise=true , address=145.79.227.37:4565 , piece_id=IND6XBSVS3UG4N252K6JNHDEKFJCLTVMVW5YNAQ6W2THBWRVDCSA, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.481964758 (546h49m15.353047803s) [noise=true , address=77.90.31.14:4663 , piece_id=BZEMBNU6MH57HOMY5IOK6IMXRQRZ7GGBY6WXPCUEPMNKRJ4VI7BQ, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.482337077 (546h49m15.353420112s) [noise=true , address=145.79.227.48:4303 , piece_id=RMXZUKR64I2OIXGOPAQDFFPKHXJUAZN22U56HGMTS36IJRO6PPUA, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.482651846 (546h49m15.353734882s) [noise=true , address=145.79.227.36:4518 , piece_id=FL4OYXW2E6OXHJPUVHIPHRFYXKYEPP64KL4KQI6XCJJFGI2N77MA, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.482970112 (546h49m15.354053157s) [noise=true , address=145.79.227.27:4689 , piece_id=WUUCHWICUXSNRO2CRZXQD42LDZOM3PTQZ2TEDDQTDOEAFUVRKQBQ, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.483637102 (546h49m15.354720147s) [noise=true , address=145.79.227.2:4114 , piece_id=LKPJS3JNFN3UNNEWNW5MXQR533TDWQZQMFJVICCHPB6CL6DKOW6A, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.484310093 (546h49m15.355393138s) [noise=true , address=145.79.227.47:4267 , piece_id=TZJG6HGLOV2NU3NFVSYAZTRR5MQC7ZZYKOIZP2DS6JLDLN3YXE6Q, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.486412978 (546h49m15.357496013s) [noise=true , address=145.79.227.42:4123 , piece_id=G7ECBZ5D3V6C2CG5DK53J3JYYYEGW77KQVZ5RN5WN7VFLGASO3UQ, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:17.190839167 (546h49m17.061922202s) [upload=479491]: Next returned nil segment Dec 28 10:18:17.190859146 (546h49m17.061942181s) [upload=479491]: Waiting for error group managing segments... Dec 28 10:18:17.475649795 (546h49m17.34673285s) [noise=true , address=145.79.227.38:4977 , piece_id=UF2GVCDHOW7W2XKK43EY7I45THJC7UOMWKS6KEFGJ2TMIZYFABBA, seg_pos={0 0}, upload=479491]: Done uploading piece. err: <nil> Dec 28 10:18:17.481978897 (546h49m17.353061942s) [noise=true , address=145.79.227.48:4303 , piece_id=RMXZUKR64I2OIXGOPAQDFFPKHXJUAZN22U56HGMTS36IJRO6PPUA, seg_pos={0 0}, upload=479491]: Done uploading piece. err: <nil> Dec 28 10:18:17.488016143 (546h49m17.359099188s) [noise=true , address=145.79.227.36:4518 , piece_id=FL4OYXW2E6OXHJPUVHIPHRFYXKYEPP64KL4KQI6XCJJFGI2N77MA, seg_pos={0 0}, upload=479491]: Done uploading piece. err: <nil> Dec 28 10:18:17.491465933 (546h49m17.362548968s) [noise=true , address=145.79.227.23:4547 , piece_id=SU36GD6KUH6QKK2RQRUHGWIG7VCZPTNAQG6CETI7CYP5OA3MD5VA, seg_pos={0 0}, upload=479491]: Done uploading piece. err: <nil> Dec 28 10:18:17.492022262 (546h49m17.363105307s) [noise=true , address=145.79.227.50:4373 , piece_id=3MF6H25P3PXZR6CRB6QSMMUG5G7ACGHJK5XDPMFZ7JOKYDETEIOQ, seg_pos={0 0}, upload=479491]: Done uploading piece. err: <nil> Dec 28 10:18:17.492978823 (546h49m17.364061858s) [noise=true , address=145.79.227.27:4689 , piece_id=WUUCHWICUXSNRO2CRZXQD42LDZOM3PTQZ2TEDDQTDOEAFUVRKQBQ, seg_pos={0 0}, upload=479491]: Done uploading piece. err: <nil> Dec 28 10:18:17.496314375 (546h49m17.36739742s) [noise=true , address=145.79.227.37:4565 , piece_id=IND6XBSVS3UG4N252K6JNHDEKFJCLTVMVW5YNAQ6W2THBWRVDCSA, seg_pos={0 0}, upload=479491]: Done uploading piece. err: <nil> Dec 28 10:18:17.507404021 (546h49m17.378487066s) [noise=true , address=145.79.227.14:4840 , piece_id=T6JJ54YDOUNNYOF2KI3ZGY25EYJTSMFH74S6B5S7MIJO2DTWP62Q, seg_pos={0 0}, upload=479491]: Done uploading piece. err: <nil> Dec 28 10:18:17.510907242 (546h49m17.381990287s) [noise=true , address=145.79.227.47:4267 , piece_id=TZJG6HGLOV2NU3NFVSYAZTRR5MQC7ZZYKOIZP2DS6JLDLN3YXE6Q, seg_pos={0 0}, upload=479491]: Done uploading piece. err: <nil> Dec 28 10:18:17.513621480 (546h49m17.384704515s) [noise=true , address=145.79.227.2:4114 , piece_id=LKPJS3JNFN3UNNEWNW5MXQR533TDWQZQMFJVICCHPB6CL6DKOW6A, seg_pos={0 0}, upload=479491]: Done uploading piece. err: <nil> Dec 28 10:18:17.518832943 (546h49m17.389915988s) [noise=true , address=145.79.227.42:4123 , piece_id=G7ECBZ5D3V6C2CG5DK53J3JYYYEGW77KQVZ5RN5WN7VFLGASO3UQ, seg_pos={0 0}, upload=479491]: Done uploading piece. err: <nil> Dec 28 10:18:17.727901746 (546h49m17.598984781s) [noise=true , address=45.91.116.14:4192 , piece_id=3D7RRA5BWEZ53QEIM7STRQUDL2RSEBMXCR3EFUSGVL3MZDWMW37Q, seg_pos={0 0}, upload=479491]: Done uploading piece. err: <nil> Dec 28 10:18:17.733928622 (546h49m17.605011667s) [noise=true , address=45.91.116.6:4976 , piece_id=AN4NJQHGJZOFQV7ORYVBHYUYI342VNU3DP3NX7N2NM2BMYCFFOPQ, seg_pos={0 0}, upload=479491]: Done uploading piece. err: <nil> Dec 28 10:18:17.733953079 (546h49m17.605036124s) [seg_pos={0 0}, upload=479491]: Segment reached optimal threshold of 13 pieces. Dec 28 10:18:17.734173228 (546h49m17.605256273s) [noise=true , address=77.90.31.14:4663 , piece_id=BZEMBNU6MH57HOMY5IOK6IMXRQRZ7GGBY6WXPCUEPMNKRJ4VI7BQ, seg_pos={0 0}, upload=479491]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 28 10:18:17.734181373 (546h49m17.605264408s) [noise=true , address=77.90.31.14:4663 , piece_id=BZEMBNU6MH57HOMY5IOK6IMXRQRZ7GGBY6WXPCUEPMNKRJ4VI7BQ, seg_pos={0 0}, upload=479491]: Long tail cancellation due to optimal threshold Dec 28 10:18:17.734477468 (546h49m17.605560503s) [noise=true , address=45.91.116.9:4264 , piece_id=6IKCH5SFYCBIYSZIKBFEK2EP2MOILJBOQA2VQXY5TLU4S273OKDQ, seg_pos={0 0}, upload=479491]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 28 10:18:17.734484561 (546h49m17.605567596s) [noise=true , address=45.91.116.9:4264 , piece_id=6IKCH5SFYCBIYSZIKBFEK2EP2MOILJBOQA2VQXY5TLU4S273OKDQ, seg_pos={0 0}, upload=479491]: Long tail cancellation due to optimal threshold Dec 28 10:18:17.734584842 (546h49m17.605667877s) [noise=true , address=77.90.31.10:4519 , piece_id=AVBSS2GLD7GWCVSGJU3WV3JYEZEAFD4R7B563EISYQWYRHNLW74Q, seg_pos={0 0}, upload=479491]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 28 10:18:17.734602195 (546h49m17.605685241s) [noise=true , address=77.90.31.10:4519 , piece_id=AVBSS2GLD7GWCVSGJU3WV3JYEZEAFD4R7B563EISYQWYRHNLW74Q, seg_pos={0 0}, upload=479491]: Long tail cancellation due to optimal threshold Dec 28 10:18:17.734655076 (546h49m17.605738121s) [seg_pos={0 0}, upload=479491]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:18:17.734802987 (546h49m17.605886022s) [upload=479491]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:18:17.734809701 (546h49m17.605892736s) [upload=479491]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:18:18.035635943 (546h49m17.906718978s) [upload=479491]: Done uploading segments... Dec 28 10:18:18.035661372 (546h49m17.906744407s) [upload=479491]: Upload finished. err: <nil> Duration: 5.989507874s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [..XX.X...X.X....XXXXX...X........X.....X..................X.] duration: 4.834277058s trace: https://jaeger.dev.storj.io/trace/49dbd650238e8f85 FAIL (2025-12-28 10:20:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.834316162s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [XX...X..X.........X.........X...................X..XX.....X.] duration: 5.938899257s trace: https://jaeger.dev.storj.io/trace/67a1aca7935ff554 FAIL (2025-12-28 10:19:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: AN4NJQHGJZOFQV7ORYVBHYUYI342VNU3DP3NX7N2NM2BMYCFFOPQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: 3D7RRA5BWEZ53QEIM7STRQUDL2RSEBMXCR3EFUSGVL3MZDWMW37Q) context deadline exceeded ======LOGS====== Dec 28 10:18:15.232717407 (546h49m15.103800452s) [upload=479491]: Starting upload Dec 28 10:18:15.232895657 (546h49m15.103978702s) [upload=479491]: Done starting upload Dec 28 10:18:15.232960500 (546h49m15.104043545s) [upload=479491]: Uploading segments... Dec 28 10:18:15.233147817 (546h49m15.104230852s) [upload=479491]: Got next segment. Inline: false Dec 28 10:18:15.233190378 (546h49m15.104273433s) [upload=479491]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:18:15.233196840 (546h49m15.104279875s) [upload=479491]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:18:15.478852011 (546h49m15.349935066s) [seg_pos={0 0}, upload=479491]: Begin upload segment... Dec 28 10:18:15.478918126 (546h49m15.350001171s) [seg_pos={0 0}, upload=479491]: Done begin upload segment. Dec 28 10:18:15.478925210 (546h49m15.350008245s) [noise=true , address=45.91.116.9:4264 , piece_id=6IKCH5SFYCBIYSZIKBFEK2EP2MOILJBOQA2VQXY5TLU4S273OKDQ, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.478949356 (546h49m15.350032391s) [noise=true , address=77.90.31.10:4519 , piece_id=AVBSS2GLD7GWCVSGJU3WV3JYEZEAFD4R7B563EISYQWYRHNLW74Q, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.479090785 (546h49m15.35017383s) [noise=true , address=145.79.227.38:4977 , piece_id=UF2GVCDHOW7W2XKK43EY7I45THJC7UOMWKS6KEFGJ2TMIZYFABBA, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.479858939 (546h49m15.350941984s) [noise=true , address=145.79.227.14:4840 , piece_id=T6JJ54YDOUNNYOF2KI3ZGY25EYJTSMFH74S6B5S7MIJO2DTWP62Q, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.480268197 (546h49m15.351351242s) [noise=true , address=145.79.227.23:4547 , piece_id=SU36GD6KUH6QKK2RQRUHGWIG7VCZPTNAQG6CETI7CYP5OA3MD5VA, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.480626941 (546h49m15.351709976s) [noise=true , address=45.91.116.6:4976 , piece_id=AN4NJQHGJZOFQV7ORYVBHYUYI342VNU3DP3NX7N2NM2BMYCFFOPQ, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.480984702 (546h49m15.352067757s) [noise=true , address=45.91.116.14:4192 , piece_id=3D7RRA5BWEZ53QEIM7STRQUDL2RSEBMXCR3EFUSGVL3MZDWMW37Q, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.481363664 (546h49m15.352446709s) [noise=true , address=145.79.227.50:4373 , piece_id=3MF6H25P3PXZR6CRB6QSMMUG5G7ACGHJK5XDPMFZ7JOKYDETEIOQ, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.481740762 (546h49m15.352823797s) [noise=true , address=145.79.227.37:4565 , piece_id=IND6XBSVS3UG4N252K6JNHDEKFJCLTVMVW5YNAQ6W2THBWRVDCSA, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.481964758 (546h49m15.353047803s) [noise=true , address=77.90.31.14:4663 , piece_id=BZEMBNU6MH57HOMY5IOK6IMXRQRZ7GGBY6WXPCUEPMNKRJ4VI7BQ, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.482337077 (546h49m15.353420112s) [noise=true , address=145.79.227.48:4303 , piece_id=RMXZUKR64I2OIXGOPAQDFFPKHXJUAZN22U56HGMTS36IJRO6PPUA, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.482651846 (546h49m15.353734882s) [noise=true , address=145.79.227.36:4518 , piece_id=FL4OYXW2E6OXHJPUVHIPHRFYXKYEPP64KL4KQI6XCJJFGI2N77MA, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.482970112 (546h49m15.354053157s) [noise=true , address=145.79.227.27:4689 , piece_id=WUUCHWICUXSNRO2CRZXQD42LDZOM3PTQZ2TEDDQTDOEAFUVRKQBQ, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.483637102 (546h49m15.354720147s) [noise=true , address=145.79.227.2:4114 , piece_id=LKPJS3JNFN3UNNEWNW5MXQR533TDWQZQMFJVICCHPB6CL6DKOW6A, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.484310093 (546h49m15.355393138s) [noise=true , address=145.79.227.47:4267 , piece_id=TZJG6HGLOV2NU3NFVSYAZTRR5MQC7ZZYKOIZP2DS6JLDLN3YXE6Q, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.486412978 (546h49m15.357496013s) [noise=true , address=145.79.227.42:4123 , piece_id=G7ECBZ5D3V6C2CG5DK53J3JYYYEGW77KQVZ5RN5WN7VFLGASO3UQ, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:17.190839167 (546h49m17.061922202s) [upload=479491]: Next returned nil segment Dec 28 10:18:17.190859146 (546h49m17.061942181s) [upload=479491]: Waiting for error group managing segments... Dec 28 10:18:17.475649795 (546h49m17.34673285s) [noise=true , address=145.79.227.38:4977 , piece_id=UF2GVCDHOW7W2XKK43EY7I45THJC7UOMWKS6KEFGJ2TMIZYFABBA, seg_pos={0 0}, upload=479491]: Done uploading piece. err: <nil> Dec 28 10:18:17.481978897 (546h49m17.353061942s) [noise=true , address=145.79.227.48:4303 , piece_id=RMXZUKR64I2OIXGOPAQDFFPKHXJUAZN22U56HGMTS36IJRO6PPUA, seg_pos={0 0}, upload=479491]: Done uploading piece. err: <nil> Dec 28 10:18:17.488016143 (546h49m17.359099188s) [noise=true , address=145.79.227.36:4518 , piece_id=FL4OYXW2E6OXHJPUVHIPHRFYXKYEPP64KL4KQI6XCJJFGI2N77MA, seg_pos={0 0}, upload=479491]: Done uploading piece. err: <nil> Dec 28 10:18:17.491465933 (546h49m17.362548968s) [noise=true , address=145.79.227.23:4547 , piece_id=SU36GD6KUH6QKK2RQRUHGWIG7VCZPTNAQG6CETI7CYP5OA3MD5VA, seg_pos={0 0}, upload=479491]: Done uploading piece. err: <nil> Dec 28 10:18:17.492022262 (546h49m17.363105307s) [noise=true , address=145.79.227.50:4373 , piece_id=3MF6H25P3PXZR6CRB6QSMMUG5G7ACGHJK5XDPMFZ7JOKYDETEIOQ, seg_pos={0 0}, upload=479491]: Done uploading piece. err: <nil> Dec 28 10:18:17.492978823 (546h49m17.364061858s) [noise=true , address=145.79.227.27:4689 , piece_id=WUUCHWICUXSNRO2CRZXQD42LDZOM3PTQZ2TEDDQTDOEAFUVRKQBQ, seg_pos={0 0}, upload=479491]: Done uploading piece. err: <nil> Dec 28 10:18:17.496314375 (546h49m17.36739742s) [noise=true , address=145.79.227.37:4565 , piece_id=IND6XBSVS3UG4N252K6JNHDEKFJCLTVMVW5YNAQ6W2THBWRVDCSA, seg_pos={0 0}, upload=479491]: Done uploading piece. err: <nil> Dec 28 10:18:17.507404021 (546h49m17.378487066s) [noise=true , address=145.79.227.14:4840 , piece_id=T6JJ54YDOUNNYOF2KI3ZGY25EYJTSMFH74S6B5S7MIJO2DTWP62Q, seg_pos={0 0}, upload=479491]: Done uploading piece. err: <nil> Dec 28 10:18:17.510907242 (546h49m17.381990287s) [noise=true , address=145.79.227.47:4267 , piece_id=TZJG6HGLOV2NU3NFVSYAZTRR5MQC7ZZYKOIZP2DS6JLDLN3YXE6Q, seg_pos={0 0}, upload=479491]: Done uploading piece. err: <nil> Dec 28 10:18:17.513621480 (546h49m17.384704515s) [noise=true , address=145.79.227.2:4114 , piece_id=LKPJS3JNFN3UNNEWNW5MXQR533TDWQZQMFJVICCHPB6CL6DKOW6A, seg_pos={0 0}, upload=479491]: Done uploading piece. err: <nil> Dec 28 10:18:17.518832943 (546h49m17.389915988s) [noise=true , address=145.79.227.42:4123 , piece_id=G7ECBZ5D3V6C2CG5DK53J3JYYYEGW77KQVZ5RN5WN7VFLGASO3UQ, seg_pos={0 0}, upload=479491]: Done uploading piece. err: <nil> Dec 28 10:18:17.727901746 (546h49m17.598984781s) [noise=true , address=45.91.116.14:4192 , piece_id=3D7RRA5BWEZ53QEIM7STRQUDL2RSEBMXCR3EFUSGVL3MZDWMW37Q, seg_pos={0 0}, upload=479491]: Done uploading piece. err: <nil> Dec 28 10:18:17.733928622 (546h49m17.605011667s) [noise=true , address=45.91.116.6:4976 , piece_id=AN4NJQHGJZOFQV7ORYVBHYUYI342VNU3DP3NX7N2NM2BMYCFFOPQ, seg_pos={0 0}, upload=479491]: Done uploading piece. err: <nil> Dec 28 10:18:17.733953079 (546h49m17.605036124s) [seg_pos={0 0}, upload=479491]: Segment reached optimal threshold of 13 pieces. Dec 28 10:18:17.734173228 (546h49m17.605256273s) [noise=true , address=77.90.31.14:4663 , piece_id=BZEMBNU6MH57HOMY5IOK6IMXRQRZ7GGBY6WXPCUEPMNKRJ4VI7BQ, seg_pos={0 0}, upload=479491]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 28 10:18:17.734181373 (546h49m17.605264408s) [noise=true , address=77.90.31.14:4663 , piece_id=BZEMBNU6MH57HOMY5IOK6IMXRQRZ7GGBY6WXPCUEPMNKRJ4VI7BQ, seg_pos={0 0}, upload=479491]: Long tail cancellation due to optimal threshold Dec 28 10:18:17.734477468 (546h49m17.605560503s) [noise=true , address=45.91.116.9:4264 , piece_id=6IKCH5SFYCBIYSZIKBFEK2EP2MOILJBOQA2VQXY5TLU4S273OKDQ, seg_pos={0 0}, upload=479491]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 28 10:18:17.734484561 (546h49m17.605567596s) [noise=true , address=45.91.116.9:4264 , piece_id=6IKCH5SFYCBIYSZIKBFEK2EP2MOILJBOQA2VQXY5TLU4S273OKDQ, seg_pos={0 0}, upload=479491]: Long tail cancellation due to optimal threshold Dec 28 10:18:17.734584842 (546h49m17.605667877s) [noise=true , address=77.90.31.10:4519 , piece_id=AVBSS2GLD7GWCVSGJU3WV3JYEZEAFD4R7B563EISYQWYRHNLW74Q, seg_pos={0 0}, upload=479491]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 28 10:18:17.734602195 (546h49m17.605685241s) [noise=true , address=77.90.31.10:4519 , piece_id=AVBSS2GLD7GWCVSGJU3WV3JYEZEAFD4R7B563EISYQWYRHNLW74Q, seg_pos={0 0}, upload=479491]: Long tail cancellation due to optimal threshold Dec 28 10:18:17.734655076 (546h49m17.605738121s) [seg_pos={0 0}, upload=479491]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:18:17.734802987 (546h49m17.605886022s) [upload=479491]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:18:17.734809701 (546h49m17.605892736s) [upload=479491]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:18:18.035635943 (546h49m17.906718978s) [upload=479491]: Done uploading segments... Dec 28 10:18:18.035661372 (546h49m17.906744407s) [upload=479491]: Upload finished. err: <nil> Duration: 5.938930407s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [X..XX.X...X.X....XXXXX...X........X.....X..................X] duration: 5.002092157s trace: https://jaeger.dev.storj.io/trace/49dbd650238e8f85 FAIL (2025-12-28 10:19:05 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002118408s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.XX...X..X.........X.........X...................X..XX.....X] duration: 30.003430266s trace: https://jaeger.dev.storj.io/trace/67a1aca7935ff554 FAIL (2025-12-28 10:18:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: AN4NJQHGJZOFQV7ORYVBHYUYI342VNU3DP3NX7N2NM2BMYCFFOPQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: 3D7RRA5BWEZ53QEIM7STRQUDL2RSEBMXCR3EFUSGVL3MZDWMW37Q) context deadline exceeded ======LOGS====== Dec 28 10:18:15.232717407 (546h49m15.103800452s) [upload=479491]: Starting upload Dec 28 10:18:15.232895657 (546h49m15.103978702s) [upload=479491]: Done starting upload Dec 28 10:18:15.232960500 (546h49m15.104043545s) [upload=479491]: Uploading segments... Dec 28 10:18:15.233147817 (546h49m15.104230852s) [upload=479491]: Got next segment. Inline: false Dec 28 10:18:15.233190378 (546h49m15.104273433s) [upload=479491]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:18:15.233196840 (546h49m15.104279875s) [upload=479491]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:18:15.478852011 (546h49m15.349935066s) [seg_pos={0 0}, upload=479491]: Begin upload segment... Dec 28 10:18:15.478918126 (546h49m15.350001171s) [seg_pos={0 0}, upload=479491]: Done begin upload segment. Dec 28 10:18:15.478925210 (546h49m15.350008245s) [noise=true , address=45.91.116.9:4264 , piece_id=6IKCH5SFYCBIYSZIKBFEK2EP2MOILJBOQA2VQXY5TLU4S273OKDQ, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.478949356 (546h49m15.350032391s) [noise=true , address=77.90.31.10:4519 , piece_id=AVBSS2GLD7GWCVSGJU3WV3JYEZEAFD4R7B563EISYQWYRHNLW74Q, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.479090785 (546h49m15.35017383s) [noise=true , address=145.79.227.38:4977 , piece_id=UF2GVCDHOW7W2XKK43EY7I45THJC7UOMWKS6KEFGJ2TMIZYFABBA, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.479858939 (546h49m15.350941984s) [noise=true , address=145.79.227.14:4840 , piece_id=T6JJ54YDOUNNYOF2KI3ZGY25EYJTSMFH74S6B5S7MIJO2DTWP62Q, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.480268197 (546h49m15.351351242s) [noise=true , address=145.79.227.23:4547 , piece_id=SU36GD6KUH6QKK2RQRUHGWIG7VCZPTNAQG6CETI7CYP5OA3MD5VA, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.480626941 (546h49m15.351709976s) [noise=true , address=45.91.116.6:4976 , piece_id=AN4NJQHGJZOFQV7ORYVBHYUYI342VNU3DP3NX7N2NM2BMYCFFOPQ, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.480984702 (546h49m15.352067757s) [noise=true , address=45.91.116.14:4192 , piece_id=3D7RRA5BWEZ53QEIM7STRQUDL2RSEBMXCR3EFUSGVL3MZDWMW37Q, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.481363664 (546h49m15.352446709s) [noise=true , address=145.79.227.50:4373 , piece_id=3MF6H25P3PXZR6CRB6QSMMUG5G7ACGHJK5XDPMFZ7JOKYDETEIOQ, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.481740762 (546h49m15.352823797s) [noise=true , address=145.79.227.37:4565 , piece_id=IND6XBSVS3UG4N252K6JNHDEKFJCLTVMVW5YNAQ6W2THBWRVDCSA, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.481964758 (546h49m15.353047803s) [noise=true , address=77.90.31.14:4663 , piece_id=BZEMBNU6MH57HOMY5IOK6IMXRQRZ7GGBY6WXPCUEPMNKRJ4VI7BQ, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.482337077 (546h49m15.353420112s) [noise=true , address=145.79.227.48:4303 , piece_id=RMXZUKR64I2OIXGOPAQDFFPKHXJUAZN22U56HGMTS36IJRO6PPUA, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.482651846 (546h49m15.353734882s) [noise=true , address=145.79.227.36:4518 , piece_id=FL4OYXW2E6OXHJPUVHIPHRFYXKYEPP64KL4KQI6XCJJFGI2N77MA, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.482970112 (546h49m15.354053157s) [noise=true , address=145.79.227.27:4689 , piece_id=WUUCHWICUXSNRO2CRZXQD42LDZOM3PTQZ2TEDDQTDOEAFUVRKQBQ, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.483637102 (546h49m15.354720147s) [noise=true , address=145.79.227.2:4114 , piece_id=LKPJS3JNFN3UNNEWNW5MXQR533TDWQZQMFJVICCHPB6CL6DKOW6A, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.484310093 (546h49m15.355393138s) [noise=true , address=145.79.227.47:4267 , piece_id=TZJG6HGLOV2NU3NFVSYAZTRR5MQC7ZZYKOIZP2DS6JLDLN3YXE6Q, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:15.486412978 (546h49m15.357496013s) [noise=true , address=145.79.227.42:4123 , piece_id=G7ECBZ5D3V6C2CG5DK53J3JYYYEGW77KQVZ5RN5WN7VFLGASO3UQ, seg_pos={0 0}, upload=479491]: Uploading piece... Dec 28 10:18:17.190839167 (546h49m17.061922202s) [upload=479491]: Next returned nil segment Dec 28 10:18:17.190859146 (546h49m17.061942181s) [upload=479491]: Waiting for error group managing segments... Dec 28 10:18:17.475649795 (546h49m17.34673285s) [noise=true , address=145.79.227.38:4977 , piece_id=UF2GVCDHOW7W2XKK43EY7I45THJC7UOMWKS6KEFGJ2TMIZYFABBA, seg_pos={0 0}, upload=479491]: Done uploading piece. err: <nil> Dec 28 10:18:17.481978897 (546h49m17.353061942s) [noise=true , address=145.79.227.48:4303 , piece_id=RMXZUKR64I2OIXGOPAQDFFPKHXJUAZN22U56HGMTS36IJRO6PPUA, seg_pos={0 0}, upload=479491]: Done uploading piece. err: <nil> Dec 28 10:18:17.488016143 (546h49m17.359099188s) [noise=true , address=145.79.227.36:4518 , piece_id=FL4OYXW2E6OXHJPUVHIPHRFYXKYEPP64KL4KQI6XCJJFGI2N77MA, seg_pos={0 0}, upload=479491]: Done uploading piece. err: <nil> Dec 28 10:18:17.491465933 (546h49m17.362548968s) [noise=true , address=145.79.227.23:4547 , piece_id=SU36GD6KUH6QKK2RQRUHGWIG7VCZPTNAQG6CETI7CYP5OA3MD5VA, seg_pos={0 0}, upload=479491]: Done uploading piece. err: <nil> Dec 28 10:18:17.492022262 (546h49m17.363105307s) [noise=true , address=145.79.227.50:4373 , piece_id=3MF6H25P3PXZR6CRB6QSMMUG5G7ACGHJK5XDPMFZ7JOKYDETEIOQ, seg_pos={0 0}, upload=479491]: Done uploading piece. err: <nil> Dec 28 10:18:17.492978823 (546h49m17.364061858s) [noise=true , address=145.79.227.27:4689 , piece_id=WUUCHWICUXSNRO2CRZXQD42LDZOM3PTQZ2TEDDQTDOEAFUVRKQBQ, seg_pos={0 0}, upload=479491]: Done uploading piece. err: <nil> Dec 28 10:18:17.496314375 (546h49m17.36739742s) [noise=true , address=145.79.227.37:4565 , piece_id=IND6XBSVS3UG4N252K6JNHDEKFJCLTVMVW5YNAQ6W2THBWRVDCSA, seg_pos={0 0}, upload=479491]: Done uploading piece. err: <nil> Dec 28 10:18:17.507404021 (546h49m17.378487066s) [noise=true , address=145.79.227.14:4840 , piece_id=T6JJ54YDOUNNYOF2KI3ZGY25EYJTSMFH74S6B5S7MIJO2DTWP62Q, seg_pos={0 0}, upload=479491]: Done uploading piece. err: <nil> Dec 28 10:18:17.510907242 (546h49m17.381990287s) [noise=true , address=145.79.227.47:4267 , piece_id=TZJG6HGLOV2NU3NFVSYAZTRR5MQC7ZZYKOIZP2DS6JLDLN3YXE6Q, seg_pos={0 0}, upload=479491]: Done uploading piece. err: <nil> Dec 28 10:18:17.513621480 (546h49m17.384704515s) [noise=true , address=145.79.227.2:4114 , piece_id=LKPJS3JNFN3UNNEWNW5MXQR533TDWQZQMFJVICCHPB6CL6DKOW6A, seg_pos={0 0}, upload=479491]: Done uploading piece. err: <nil> Dec 28 10:18:17.518832943 (546h49m17.389915988s) [noise=true , address=145.79.227.42:4123 , piece_id=G7ECBZ5D3V6C2CG5DK53J3JYYYEGW77KQVZ5RN5WN7VFLGASO3UQ, seg_pos={0 0}, upload=479491]: Done uploading piece. err: <nil> Dec 28 10:18:17.727901746 (546h49m17.598984781s) [noise=true , address=45.91.116.14:4192 , piece_id=3D7RRA5BWEZ53QEIM7STRQUDL2RSEBMXCR3EFUSGVL3MZDWMW37Q, seg_pos={0 0}, upload=479491]: Done uploading piece. err: <nil> Dec 28 10:18:17.733928622 (546h49m17.605011667s) [noise=true , address=45.91.116.6:4976 , piece_id=AN4NJQHGJZOFQV7ORYVBHYUYI342VNU3DP3NX7N2NM2BMYCFFOPQ, seg_pos={0 0}, upload=479491]: Done uploading piece. err: <nil> Dec 28 10:18:17.733953079 (546h49m17.605036124s) [seg_pos={0 0}, upload=479491]: Segment reached optimal threshold of 13 pieces. Dec 28 10:18:17.734173228 (546h49m17.605256273s) [noise=true , address=77.90.31.14:4663 , piece_id=BZEMBNU6MH57HOMY5IOK6IMXRQRZ7GGBY6WXPCUEPMNKRJ4VI7BQ, seg_pos={0 0}, upload=479491]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 28 10:18:17.734181373 (546h49m17.605264408s) [noise=true , address=77.90.31.14:4663 , piece_id=BZEMBNU6MH57HOMY5IOK6IMXRQRZ7GGBY6WXPCUEPMNKRJ4VI7BQ, seg_pos={0 0}, upload=479491]: Long tail cancellation due to optimal threshold Dec 28 10:18:17.734477468 (546h49m17.605560503s) [noise=true , address=45.91.116.9:4264 , piece_id=6IKCH5SFYCBIYSZIKBFEK2EP2MOILJBOQA2VQXY5TLU4S273OKDQ, seg_pos={0 0}, upload=479491]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 28 10:18:17.734484561 (546h49m17.605567596s) [noise=true , address=45.91.116.9:4264 , piece_id=6IKCH5SFYCBIYSZIKBFEK2EP2MOILJBOQA2VQXY5TLU4S273OKDQ, seg_pos={0 0}, upload=479491]: Long tail cancellation due to optimal threshold Dec 28 10:18:17.734584842 (546h49m17.605667877s) [noise=true , address=77.90.31.10:4519 , piece_id=AVBSS2GLD7GWCVSGJU3WV3JYEZEAFD4R7B563EISYQWYRHNLW74Q, seg_pos={0 0}, upload=479491]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 28 10:18:17.734602195 (546h49m17.605685241s) [noise=true , address=77.90.31.10:4519 , piece_id=AVBSS2GLD7GWCVSGJU3WV3JYEZEAFD4R7B563EISYQWYRHNLW74Q, seg_pos={0 0}, upload=479491]: Long tail cancellation due to optimal threshold Dec 28 10:18:17.734655076 (546h49m17.605738121s) [seg_pos={0 0}, upload=479491]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:18:17.734802987 (546h49m17.605886022s) [upload=479491]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:18:17.734809701 (546h49m17.605892736s) [upload=479491]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:18:18.035635943 (546h49m17.906718978s) [upload=479491]: Done uploading segments... Dec 28 10:18:18.035661372 (546h49m17.906744407s) [upload=479491]: Upload finished. err: <nil> Duration: 30.003454231s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [XX..XX.X...X.X....XXXXX...X........X.....X..................] duration: 3.556393438s trace: https://jaeger.dev.storj.io/trace/3f41c3ec34adb72f FAIL (2025-12-28 10:18:24 UTC) orbiter warning: context deadline exceeded Duration: 3.556422553s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [XXX..XX.X...X.X....XXXXX...X........X.....X.................] duration: 3.641947678s trace: https://jaeger.dev.storj.io/trace/3f41c3ec34adb72f FAIL (2025-12-28 10:17:53 UTC) orbiter warning: context deadline exceeded Duration: 3.641973357s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..XX...X..X.........X.........X...................X..XX.....] duration: 6.104997979s trace: https://jaeger.dev.storj.io/trace/3df6ebd21e3261a FAIL (2025-12-28 10:17:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 8 10:12:25.751002002 (546h43m25.622085037s) [upload=479407]: Starting upload Dec 28 10:12:25.751076032 (546h43m25.622159088s) [upload=479407]: Done starting upload Dec 28 10:12:25.751202923 (546h43m25.622285959s) [upload=479407]: Uploading segments... Dec 28 10:12:25.751376043 (546h43m25.622459078s) [upload=479407]: Got next segment. Inline: false Dec 28 10:12:25.751435637 (546h43m25.622518672s) [upload=479407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:12:25.751440015 (546h43m25.62252305s) [upload=479407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:12:25.999029369 (546h43m25.870112404s) [seg_pos={0 0}, upload=479407]: Begin upload segment... Dec 28 10:12:25.999087810 (546h43m25.870170855s) [seg_pos={0 0}, upload=479407]: Done begin upload segment. Dec 28 10:12:25.999119601 (546h43m25.870202646s) [noise=true , address=45.91.116.11:4120 , piece_id=XLKNNWMCQ7C6DKLC3VQGKK7PD243EZELISP4NMENOLI46DGCL7TQ, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:25.999185977 (546h43m25.870269012s) [noise=true , address=145.79.227.48:4301 , piece_id=ZFOSU45PBQCFYPJGH6EAIH4VWRWLMYRC3ALRVRIGCDPM62YOY5LA, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:25.999724041 (546h43m25.870807076s) [noise=true , address=77.90.31.10:4520 , piece_id=WETUFKFYTBGHSYHUGZVBASQHNLX6ZBSHLI4IYKPTSF2GLVVO6W4A, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.000051184 (546h43m25.871134229s) [noise=true , address=77.90.31.21:4877 , piece_id=FLW7LIMJEC7YIRLKXN67BRBDCZ5M6GJ7BBQP24UD23DSXGRVNKBA, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.000393516 (546h43m25.871476561s) [noise=true , address=45.91.116.13:4156 , piece_id=Z5UHYFISTNNJG7OXZW5OCOSOI7VCVCXSFN2WNX72S34AWDLIAXTA, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.000710519 (546h43m25.871793554s) [noise=true , address=145.79.227.18:4007 , piece_id=7WTRLRG542RHHIKCUNG3GDJSD6LQQ4RARLNIBMU52QRSOPFIS7ZA, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.001059654 (546h43m25.872142689s) [noise=true , address=145.79.227.42:4122 , piece_id=XOGO3W2CB4TIH22UY64DSDSLUGJNGCF5DQBC5U5OOA3EVP56DUXA, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.001379444 (546h43m25.872462479s) [noise=true , address=77.90.31.18:4808 , piece_id=PGUSLD66RBMEJBKPCTZ7MXPCO7JRGWXYSWXNEXPNQGLFUFH3X5IQ, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.001750710 (546h43m25.872833745s) [noise=true , address=145.79.227.13:4804 , piece_id=7ZCAIPPPXT2IMUZ4XBVNCQHBGZ7QXJ4Y6KV6J4V5ZH2GJ4O6EGGQ, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.002298023 (546h43m25.873381058s) [noise=true , address=145.79.227.32:4835 , piece_id=AVLVCL2OE5OM3RBR7WKFYHTWYRMHBB6LQ5XAZWJR4SJYITANC32Q, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.002725286 (546h43m25.873808331s) [noise=true , address=145.79.227.46:4601 , piece_id=B3GNLPMUESSJEBVF2ZRI2FWKCQ5UHOJKRMTF3JVZWIVSYQ3FEO7A, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.003114979 (546h43m25.874198024s) [noise=true , address=45.91.116.12:4226 , piece_id=IYM3F5QBL7BRHYGWDSTP6LO4G3LHD7AAH4KY6OVCSWHRY4FBELGA, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.003507286 (546h43m25.874590321s) [noise=true , address=145.79.227.15:4875 , piece_id=AWJXUCJMY3SQQNHFLDPSEQKBPZNP5XGRW7NVFUTTCETUEN3ZXD3A, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.003893661 (546h43m25.874976706s) [noise=true , address=145.79.227.40:4051 , piece_id=UZ764WOMEUPWIAPXCNHKMFAFAGO3IITHQQZN4BZSGI7PLQFYBILQ, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.004306387 (546h43m25.875389432s) [noise=true , address=145.79.227.41:4086 , piece_id=UFRF3CAQ7XPEM27EGPB5QDM2Q7SCBCGM2E3OQ27GDHGATTCXPJ3Q, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.004719544 (546h43m25.875802589s) [noise=true , address=145.79.227.38:4979 , piece_id=27GUZM4NTZYC2B5CVEJHUW4HDIRCHACAMIBK5WMH6WUXNHBQE52A, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:27.734491693 (546h43m27.605574758s) [upload=479407]: Next returned nil segment Dec 28 10:12:27.734530317 (546h43m27.605613362s) [upload=479407]: Waiting for error group managing segments... Dec 28 10:12:28.018123596 (546h43m27.889206652s) [noise=true , address=145.79.227.18:4007 , piece_id=7WTRLRG542RHHIKCUNG3GDJSD6LQQ4RARLNIBMU52QRSOPFIS7ZA, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.024834725 (546h43m27.89591776s) [noise=true , address=145.79.227.38:4979 , piece_id=27GUZM4NTZYC2B5CVEJHUW4HDIRCHACAMIBK5WMH6WUXNHBQE52A, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.025075894 (546h43m27.896158939s) [noise=true , address=145.79.227.15:4875 , piece_id=AWJXUCJMY3SQQNHFLDPSEQKBPZNP5XGRW7NVFUTTCETUEN3ZXD3A, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.025588511 (546h43m27.896671556s) [noise=true , address=145.79.227.32:4835 , piece_id=AVLVCL2OE5OM3RBR7WKFYHTWYRMHBB6LQ5XAZWJR4SJYITANC32Q, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.026177292 (546h43m27.897260337s) [noise=true , address=145.79.227.41:4086 , piece_id=UFRF3CAQ7XPEM27EGPB5QDM2Q7SCBCGM2E3OQ27GDHGATTCXPJ3Q, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.026761184 (546h43m27.897844229s) [noise=true , address=145.79.227.48:4301 , piece_id=ZFOSU45PBQCFYPJGH6EAIH4VWRWLMYRC3ALRVRIGCDPM62YOY5LA, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.030818669 (546h43m27.901901704s) [noise=true , address=145.79.227.46:4601 , piece_id=B3GNLPMUESSJEBVF2ZRI2FWKCQ5UHOJKRMTF3JVZWIVSYQ3FEO7A, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.034124584 (546h43m27.90520762s) [noise=true , address=145.79.227.13:4804 , piece_id=7ZCAIPPPXT2IMUZ4XBVNCQHBGZ7QXJ4Y6KV6J4V5ZH2GJ4O6EGGQ, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.034564222 (546h43m27.905647257s) [noise=true , address=145.79.227.42:4122 , piece_id=XOGO3W2CB4TIH22UY64DSDSLUGJNGCF5DQBC5U5OOA3EVP56DUXA, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.101769883 (546h43m27.972852918s) [noise=true , address=145.79.227.40:4051 , piece_id=UZ764WOMEUPWIAPXCNHKMFAFAGO3IITHQQZN4BZSGI7PLQFYBILQ, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.240152857 (546h43m28.111235892s) [noise=true , address=45.91.116.12:4226 , piece_id=IYM3F5QBL7BRHYGWDSTP6LO4G3LHD7AAH4KY6OVCSWHRY4FBELGA, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.307608374 (546h43m28.17869141s) [noise=true , address=77.90.31.18:4808 , piece_id=PGUSLD66RBMEJBKPCTZ7MXPCO7JRGWXYSWXNEXPNQGLFUFH3X5IQ, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.359465499 (546h43m28.230548534s) [noise=true , address=77.90.31.10:4520 , piece_id=WETUFKFYTBGHSYHUGZVBASQHNLX6ZBSHLI4IYKPTSF2GLVVO6W4A, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.359492871 (546h43m28.230575916s) [seg_pos={0 0}, upload=479407]: Segment reached optimal threshold of 13 pieces. Dec 28 10:12:28.359826747 (546h43m28.230909782s) [noise=true , address=45.91.116.13:4156 , piece_id=Z5UHYFISTNNJG7OXZW5OCOSOI7VCVCXSFN2WNX72S34AWDLIAXTA, seg_pos={0 0}, upload=479407]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 28 10:12:28.359843228 (546h43m28.230926263s) [noise=true , address=45.91.116.13:4156 , piece_id=Z5UHYFISTNNJG7OXZW5OCOSOI7VCVCXSFN2WNX72S34AWDLIAXTA, seg_pos={0 0}, upload=479407]: Long tail cancellation due to optimal threshold Dec 28 10:12:28.359887582 (546h43m28.230970617s) [noise=true , address=45.91.116.11:4120 , piece_id=XLKNNWMCQ7C6DKLC3VQGKK7PD243EZELISP4NMENOLI46DGCL7TQ, seg_pos={0 0}, upload=479407]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:36868->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:36868->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36868->45.91.116.11:4120: use of closed network connection Dec 28 10:12:28.359895919 (546h43m28.230978954s) [noise=true , address=45.91.116.11:4120 , piece_id=XLKNNWMCQ7C6DKLC3VQGKK7PD243EZELISP4NMENOLI46DGCL7TQ, seg_pos={0 0}, upload=479407]: Long tail cancellation due to optimal threshold Dec 28 10:12:28.360084688 (546h43m28.231167733s) [noise=true , address=77.90.31.21:4877 , piece_id=FLW7LIMJEC7YIRLKXN67BRBDCZ5M6GJ7BBQP24UD23DSXGRVNKBA, seg_pos={0 0}, upload=479407]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 28 10:12:28.360091691 (546h43m28.231174736s) [noise=true , address=77.90.31.21:4877 , piece_id=FLW7LIMJEC7YIRLKXN67BRBDCZ5M6GJ7BBQP24UD23DSXGRVNKBA, seg_pos={0 0}, upload=479407]: Long tail cancellation due to optimal threshold Dec 28 10:12:28.360135645 (546h43m28.23121868s) [seg_pos={0 0}, upload=479407]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:12:28.360225937 (546h43m28.231308972s) [upload=479407]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:12:28.360230796 (546h43m28.231313831s) [upload=479407]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:12:28.655722823 (546h43m28.526805858s) [upload=479407]: Done uploading segments... Dec 28 10:12:28.655742540 (546h43m28.526825585s) [upload=479407]: Upload finished. err: <nil> Duration: 6.105035019s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...XX...X..X.........X.........X...................X..XX....] duration: 5.763727294s trace: https://jaeger.dev.storj.io/trace/3df6ebd21e3261a FAIL (2025-12-28 10:16:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 8 10:12:25.751002002 (546h43m25.622085037s) [upload=479407]: Starting upload Dec 28 10:12:25.751076032 (546h43m25.622159088s) [upload=479407]: Done starting upload Dec 28 10:12:25.751202923 (546h43m25.622285959s) [upload=479407]: Uploading segments... Dec 28 10:12:25.751376043 (546h43m25.622459078s) [upload=479407]: Got next segment. Inline: false Dec 28 10:12:25.751435637 (546h43m25.622518672s) [upload=479407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:12:25.751440015 (546h43m25.62252305s) [upload=479407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:12:25.999029369 (546h43m25.870112404s) [seg_pos={0 0}, upload=479407]: Begin upload segment... Dec 28 10:12:25.999087810 (546h43m25.870170855s) [seg_pos={0 0}, upload=479407]: Done begin upload segment. Dec 28 10:12:25.999119601 (546h43m25.870202646s) [noise=true , address=45.91.116.11:4120 , piece_id=XLKNNWMCQ7C6DKLC3VQGKK7PD243EZELISP4NMENOLI46DGCL7TQ, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:25.999185977 (546h43m25.870269012s) [noise=true , address=145.79.227.48:4301 , piece_id=ZFOSU45PBQCFYPJGH6EAIH4VWRWLMYRC3ALRVRIGCDPM62YOY5LA, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:25.999724041 (546h43m25.870807076s) [noise=true , address=77.90.31.10:4520 , piece_id=WETUFKFYTBGHSYHUGZVBASQHNLX6ZBSHLI4IYKPTSF2GLVVO6W4A, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.000051184 (546h43m25.871134229s) [noise=true , address=77.90.31.21:4877 , piece_id=FLW7LIMJEC7YIRLKXN67BRBDCZ5M6GJ7BBQP24UD23DSXGRVNKBA, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.000393516 (546h43m25.871476561s) [noise=true , address=45.91.116.13:4156 , piece_id=Z5UHYFISTNNJG7OXZW5OCOSOI7VCVCXSFN2WNX72S34AWDLIAXTA, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.000710519 (546h43m25.871793554s) [noise=true , address=145.79.227.18:4007 , piece_id=7WTRLRG542RHHIKCUNG3GDJSD6LQQ4RARLNIBMU52QRSOPFIS7ZA, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.001059654 (546h43m25.872142689s) [noise=true , address=145.79.227.42:4122 , piece_id=XOGO3W2CB4TIH22UY64DSDSLUGJNGCF5DQBC5U5OOA3EVP56DUXA, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.001379444 (546h43m25.872462479s) [noise=true , address=77.90.31.18:4808 , piece_id=PGUSLD66RBMEJBKPCTZ7MXPCO7JRGWXYSWXNEXPNQGLFUFH3X5IQ, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.001750710 (546h43m25.872833745s) [noise=true , address=145.79.227.13:4804 , piece_id=7ZCAIPPPXT2IMUZ4XBVNCQHBGZ7QXJ4Y6KV6J4V5ZH2GJ4O6EGGQ, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.002298023 (546h43m25.873381058s) [noise=true , address=145.79.227.32:4835 , piece_id=AVLVCL2OE5OM3RBR7WKFYHTWYRMHBB6LQ5XAZWJR4SJYITANC32Q, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.002725286 (546h43m25.873808331s) [noise=true , address=145.79.227.46:4601 , piece_id=B3GNLPMUESSJEBVF2ZRI2FWKCQ5UHOJKRMTF3JVZWIVSYQ3FEO7A, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.003114979 (546h43m25.874198024s) [noise=true , address=45.91.116.12:4226 , piece_id=IYM3F5QBL7BRHYGWDSTP6LO4G3LHD7AAH4KY6OVCSWHRY4FBELGA, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.003507286 (546h43m25.874590321s) [noise=true , address=145.79.227.15:4875 , piece_id=AWJXUCJMY3SQQNHFLDPSEQKBPZNP5XGRW7NVFUTTCETUEN3ZXD3A, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.003893661 (546h43m25.874976706s) [noise=true , address=145.79.227.40:4051 , piece_id=UZ764WOMEUPWIAPXCNHKMFAFAGO3IITHQQZN4BZSGI7PLQFYBILQ, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.004306387 (546h43m25.875389432s) [noise=true , address=145.79.227.41:4086 , piece_id=UFRF3CAQ7XPEM27EGPB5QDM2Q7SCBCGM2E3OQ27GDHGATTCXPJ3Q, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.004719544 (546h43m25.875802589s) [noise=true , address=145.79.227.38:4979 , piece_id=27GUZM4NTZYC2B5CVEJHUW4HDIRCHACAMIBK5WMH6WUXNHBQE52A, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:27.734491693 (546h43m27.605574758s) [upload=479407]: Next returned nil segment Dec 28 10:12:27.734530317 (546h43m27.605613362s) [upload=479407]: Waiting for error group managing segments... Dec 28 10:12:28.018123596 (546h43m27.889206652s) [noise=true , address=145.79.227.18:4007 , piece_id=7WTRLRG542RHHIKCUNG3GDJSD6LQQ4RARLNIBMU52QRSOPFIS7ZA, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.024834725 (546h43m27.89591776s) [noise=true , address=145.79.227.38:4979 , piece_id=27GUZM4NTZYC2B5CVEJHUW4HDIRCHACAMIBK5WMH6WUXNHBQE52A, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.025075894 (546h43m27.896158939s) [noise=true , address=145.79.227.15:4875 , piece_id=AWJXUCJMY3SQQNHFLDPSEQKBPZNP5XGRW7NVFUTTCETUEN3ZXD3A, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.025588511 (546h43m27.896671556s) [noise=true , address=145.79.227.32:4835 , piece_id=AVLVCL2OE5OM3RBR7WKFYHTWYRMHBB6LQ5XAZWJR4SJYITANC32Q, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.026177292 (546h43m27.897260337s) [noise=true , address=145.79.227.41:4086 , piece_id=UFRF3CAQ7XPEM27EGPB5QDM2Q7SCBCGM2E3OQ27GDHGATTCXPJ3Q, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.026761184 (546h43m27.897844229s) [noise=true , address=145.79.227.48:4301 , piece_id=ZFOSU45PBQCFYPJGH6EAIH4VWRWLMYRC3ALRVRIGCDPM62YOY5LA, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.030818669 (546h43m27.901901704s) [noise=true , address=145.79.227.46:4601 , piece_id=B3GNLPMUESSJEBVF2ZRI2FWKCQ5UHOJKRMTF3JVZWIVSYQ3FEO7A, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.034124584 (546h43m27.90520762s) [noise=true , address=145.79.227.13:4804 , piece_id=7ZCAIPPPXT2IMUZ4XBVNCQHBGZ7QXJ4Y6KV6J4V5ZH2GJ4O6EGGQ, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.034564222 (546h43m27.905647257s) [noise=true , address=145.79.227.42:4122 , piece_id=XOGO3W2CB4TIH22UY64DSDSLUGJNGCF5DQBC5U5OOA3EVP56DUXA, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.101769883 (546h43m27.972852918s) [noise=true , address=145.79.227.40:4051 , piece_id=UZ764WOMEUPWIAPXCNHKMFAFAGO3IITHQQZN4BZSGI7PLQFYBILQ, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.240152857 (546h43m28.111235892s) [noise=true , address=45.91.116.12:4226 , piece_id=IYM3F5QBL7BRHYGWDSTP6LO4G3LHD7AAH4KY6OVCSWHRY4FBELGA, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.307608374 (546h43m28.17869141s) [noise=true , address=77.90.31.18:4808 , piece_id=PGUSLD66RBMEJBKPCTZ7MXPCO7JRGWXYSWXNEXPNQGLFUFH3X5IQ, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.359465499 (546h43m28.230548534s) [noise=true , address=77.90.31.10:4520 , piece_id=WETUFKFYTBGHSYHUGZVBASQHNLX6ZBSHLI4IYKPTSF2GLVVO6W4A, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.359492871 (546h43m28.230575916s) [seg_pos={0 0}, upload=479407]: Segment reached optimal threshold of 13 pieces. Dec 28 10:12:28.359826747 (546h43m28.230909782s) [noise=true , address=45.91.116.13:4156 , piece_id=Z5UHYFISTNNJG7OXZW5OCOSOI7VCVCXSFN2WNX72S34AWDLIAXTA, seg_pos={0 0}, upload=479407]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 28 10:12:28.359843228 (546h43m28.230926263s) [noise=true , address=45.91.116.13:4156 , piece_id=Z5UHYFISTNNJG7OXZW5OCOSOI7VCVCXSFN2WNX72S34AWDLIAXTA, seg_pos={0 0}, upload=479407]: Long tail cancellation due to optimal threshold Dec 28 10:12:28.359887582 (546h43m28.230970617s) [noise=true , address=45.91.116.11:4120 , piece_id=XLKNNWMCQ7C6DKLC3VQGKK7PD243EZELISP4NMENOLI46DGCL7TQ, seg_pos={0 0}, upload=479407]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:36868->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:36868->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36868->45.91.116.11:4120: use of closed network connection Dec 28 10:12:28.359895919 (546h43m28.230978954s) [noise=true , address=45.91.116.11:4120 , piece_id=XLKNNWMCQ7C6DKLC3VQGKK7PD243EZELISP4NMENOLI46DGCL7TQ, seg_pos={0 0}, upload=479407]: Long tail cancellation due to optimal threshold Dec 28 10:12:28.360084688 (546h43m28.231167733s) [noise=true , address=77.90.31.21:4877 , piece_id=FLW7LIMJEC7YIRLKXN67BRBDCZ5M6GJ7BBQP24UD23DSXGRVNKBA, seg_pos={0 0}, upload=479407]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 28 10:12:28.360091691 (546h43m28.231174736s) [noise=true , address=77.90.31.21:4877 , piece_id=FLW7LIMJEC7YIRLKXN67BRBDCZ5M6GJ7BBQP24UD23DSXGRVNKBA, seg_pos={0 0}, upload=479407]: Long tail cancellation due to optimal threshold Dec 28 10:12:28.360135645 (546h43m28.23121868s) [seg_pos={0 0}, upload=479407]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:12:28.360225937 (546h43m28.231308972s) [upload=479407]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:12:28.360230796 (546h43m28.231313831s) [upload=479407]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:12:28.655722823 (546h43m28.526805858s) [upload=479407]: Done uploading segments... Dec 28 10:12:28.655742540 (546h43m28.526825585s) [upload=479407]: Upload finished. err: <nil> Duration: 5.763756268s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [.XXX..XX.X...X.X....XXXXX...X........X.....X................] duration: 3.538449557s trace: https://jaeger.dev.storj.io/trace/3f41c3ec34adb72f FAIL (2025-12-28 10:16:40 UTC) orbiter warning: context deadline exceeded Duration: 3.538476989s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [..XXX..XX.X...X.X....XXXXX...X........X.....X...............] duration: 3.525224464s trace: https://jaeger.dev.storj.io/trace/3f41c3ec34adb72f FAIL (2025-12-28 10:16:16 UTC) orbiter warning: context deadline exceeded Duration: 3.525254201s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [...XXX..XX.X...X.X....XXXXX...X........X.....X..............] duration: 4.389707516s trace: https://jaeger.dev.storj.io/trace/3f41c3ec34adb72f FAIL (2025-12-28 10:15:47 UTC) orbiter warning: context deadline exceeded Duration: 4.389735378s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [....XX...X..X.........X.........X...................X..XX...] duration: 5.877341231s trace: https://jaeger.dev.storj.io/trace/3df6ebd21e3261a FAIL (2025-12-28 10:15:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 8 10:12:25.751002002 (546h43m25.622085037s) [upload=479407]: Starting upload Dec 28 10:12:25.751076032 (546h43m25.622159088s) [upload=479407]: Done starting upload Dec 28 10:12:25.751202923 (546h43m25.622285959s) [upload=479407]: Uploading segments... Dec 28 10:12:25.751376043 (546h43m25.622459078s) [upload=479407]: Got next segment. Inline: false Dec 28 10:12:25.751435637 (546h43m25.622518672s) [upload=479407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:12:25.751440015 (546h43m25.62252305s) [upload=479407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:12:25.999029369 (546h43m25.870112404s) [seg_pos={0 0}, upload=479407]: Begin upload segment... Dec 28 10:12:25.999087810 (546h43m25.870170855s) [seg_pos={0 0}, upload=479407]: Done begin upload segment. Dec 28 10:12:25.999119601 (546h43m25.870202646s) [noise=true , address=45.91.116.11:4120 , piece_id=XLKNNWMCQ7C6DKLC3VQGKK7PD243EZELISP4NMENOLI46DGCL7TQ, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:25.999185977 (546h43m25.870269012s) [noise=true , address=145.79.227.48:4301 , piece_id=ZFOSU45PBQCFYPJGH6EAIH4VWRWLMYRC3ALRVRIGCDPM62YOY5LA, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:25.999724041 (546h43m25.870807076s) [noise=true , address=77.90.31.10:4520 , piece_id=WETUFKFYTBGHSYHUGZVBASQHNLX6ZBSHLI4IYKPTSF2GLVVO6W4A, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.000051184 (546h43m25.871134229s) [noise=true , address=77.90.31.21:4877 , piece_id=FLW7LIMJEC7YIRLKXN67BRBDCZ5M6GJ7BBQP24UD23DSXGRVNKBA, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.000393516 (546h43m25.871476561s) [noise=true , address=45.91.116.13:4156 , piece_id=Z5UHYFISTNNJG7OXZW5OCOSOI7VCVCXSFN2WNX72S34AWDLIAXTA, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.000710519 (546h43m25.871793554s) [noise=true , address=145.79.227.18:4007 , piece_id=7WTRLRG542RHHIKCUNG3GDJSD6LQQ4RARLNIBMU52QRSOPFIS7ZA, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.001059654 (546h43m25.872142689s) [noise=true , address=145.79.227.42:4122 , piece_id=XOGO3W2CB4TIH22UY64DSDSLUGJNGCF5DQBC5U5OOA3EVP56DUXA, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.001379444 (546h43m25.872462479s) [noise=true , address=77.90.31.18:4808 , piece_id=PGUSLD66RBMEJBKPCTZ7MXPCO7JRGWXYSWXNEXPNQGLFUFH3X5IQ, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.001750710 (546h43m25.872833745s) [noise=true , address=145.79.227.13:4804 , piece_id=7ZCAIPPPXT2IMUZ4XBVNCQHBGZ7QXJ4Y6KV6J4V5ZH2GJ4O6EGGQ, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.002298023 (546h43m25.873381058s) [noise=true , address=145.79.227.32:4835 , piece_id=AVLVCL2OE5OM3RBR7WKFYHTWYRMHBB6LQ5XAZWJR4SJYITANC32Q, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.002725286 (546h43m25.873808331s) [noise=true , address=145.79.227.46:4601 , piece_id=B3GNLPMUESSJEBVF2ZRI2FWKCQ5UHOJKRMTF3JVZWIVSYQ3FEO7A, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.003114979 (546h43m25.874198024s) [noise=true , address=45.91.116.12:4226 , piece_id=IYM3F5QBL7BRHYGWDSTP6LO4G3LHD7AAH4KY6OVCSWHRY4FBELGA, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.003507286 (546h43m25.874590321s) [noise=true , address=145.79.227.15:4875 , piece_id=AWJXUCJMY3SQQNHFLDPSEQKBPZNP5XGRW7NVFUTTCETUEN3ZXD3A, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.003893661 (546h43m25.874976706s) [noise=true , address=145.79.227.40:4051 , piece_id=UZ764WOMEUPWIAPXCNHKMFAFAGO3IITHQQZN4BZSGI7PLQFYBILQ, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.004306387 (546h43m25.875389432s) [noise=true , address=145.79.227.41:4086 , piece_id=UFRF3CAQ7XPEM27EGPB5QDM2Q7SCBCGM2E3OQ27GDHGATTCXPJ3Q, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.004719544 (546h43m25.875802589s) [noise=true , address=145.79.227.38:4979 , piece_id=27GUZM4NTZYC2B5CVEJHUW4HDIRCHACAMIBK5WMH6WUXNHBQE52A, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:27.734491693 (546h43m27.605574758s) [upload=479407]: Next returned nil segment Dec 28 10:12:27.734530317 (546h43m27.605613362s) [upload=479407]: Waiting for error group managing segments... Dec 28 10:12:28.018123596 (546h43m27.889206652s) [noise=true , address=145.79.227.18:4007 , piece_id=7WTRLRG542RHHIKCUNG3GDJSD6LQQ4RARLNIBMU52QRSOPFIS7ZA, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.024834725 (546h43m27.89591776s) [noise=true , address=145.79.227.38:4979 , piece_id=27GUZM4NTZYC2B5CVEJHUW4HDIRCHACAMIBK5WMH6WUXNHBQE52A, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.025075894 (546h43m27.896158939s) [noise=true , address=145.79.227.15:4875 , piece_id=AWJXUCJMY3SQQNHFLDPSEQKBPZNP5XGRW7NVFUTTCETUEN3ZXD3A, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.025588511 (546h43m27.896671556s) [noise=true , address=145.79.227.32:4835 , piece_id=AVLVCL2OE5OM3RBR7WKFYHTWYRMHBB6LQ5XAZWJR4SJYITANC32Q, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.026177292 (546h43m27.897260337s) [noise=true , address=145.79.227.41:4086 , piece_id=UFRF3CAQ7XPEM27EGPB5QDM2Q7SCBCGM2E3OQ27GDHGATTCXPJ3Q, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.026761184 (546h43m27.897844229s) [noise=true , address=145.79.227.48:4301 , piece_id=ZFOSU45PBQCFYPJGH6EAIH4VWRWLMYRC3ALRVRIGCDPM62YOY5LA, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.030818669 (546h43m27.901901704s) [noise=true , address=145.79.227.46:4601 , piece_id=B3GNLPMUESSJEBVF2ZRI2FWKCQ5UHOJKRMTF3JVZWIVSYQ3FEO7A, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.034124584 (546h43m27.90520762s) [noise=true , address=145.79.227.13:4804 , piece_id=7ZCAIPPPXT2IMUZ4XBVNCQHBGZ7QXJ4Y6KV6J4V5ZH2GJ4O6EGGQ, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.034564222 (546h43m27.905647257s) [noise=true , address=145.79.227.42:4122 , piece_id=XOGO3W2CB4TIH22UY64DSDSLUGJNGCF5DQBC5U5OOA3EVP56DUXA, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.101769883 (546h43m27.972852918s) [noise=true , address=145.79.227.40:4051 , piece_id=UZ764WOMEUPWIAPXCNHKMFAFAGO3IITHQQZN4BZSGI7PLQFYBILQ, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.240152857 (546h43m28.111235892s) [noise=true , address=45.91.116.12:4226 , piece_id=IYM3F5QBL7BRHYGWDSTP6LO4G3LHD7AAH4KY6OVCSWHRY4FBELGA, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.307608374 (546h43m28.17869141s) [noise=true , address=77.90.31.18:4808 , piece_id=PGUSLD66RBMEJBKPCTZ7MXPCO7JRGWXYSWXNEXPNQGLFUFH3X5IQ, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.359465499 (546h43m28.230548534s) [noise=true , address=77.90.31.10:4520 , piece_id=WETUFKFYTBGHSYHUGZVBASQHNLX6ZBSHLI4IYKPTSF2GLVVO6W4A, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.359492871 (546h43m28.230575916s) [seg_pos={0 0}, upload=479407]: Segment reached optimal threshold of 13 pieces. Dec 28 10:12:28.359826747 (546h43m28.230909782s) [noise=true , address=45.91.116.13:4156 , piece_id=Z5UHYFISTNNJG7OXZW5OCOSOI7VCVCXSFN2WNX72S34AWDLIAXTA, seg_pos={0 0}, upload=479407]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 28 10:12:28.359843228 (546h43m28.230926263s) [noise=true , address=45.91.116.13:4156 , piece_id=Z5UHYFISTNNJG7OXZW5OCOSOI7VCVCXSFN2WNX72S34AWDLIAXTA, seg_pos={0 0}, upload=479407]: Long tail cancellation due to optimal threshold Dec 28 10:12:28.359887582 (546h43m28.230970617s) [noise=true , address=45.91.116.11:4120 , piece_id=XLKNNWMCQ7C6DKLC3VQGKK7PD243EZELISP4NMENOLI46DGCL7TQ, seg_pos={0 0}, upload=479407]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:36868->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:36868->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36868->45.91.116.11:4120: use of closed network connection Dec 28 10:12:28.359895919 (546h43m28.230978954s) [noise=true , address=45.91.116.11:4120 , piece_id=XLKNNWMCQ7C6DKLC3VQGKK7PD243EZELISP4NMENOLI46DGCL7TQ, seg_pos={0 0}, upload=479407]: Long tail cancellation due to optimal threshold Dec 28 10:12:28.360084688 (546h43m28.231167733s) [noise=true , address=77.90.31.21:4877 , piece_id=FLW7LIMJEC7YIRLKXN67BRBDCZ5M6GJ7BBQP24UD23DSXGRVNKBA, seg_pos={0 0}, upload=479407]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 28 10:12:28.360091691 (546h43m28.231174736s) [noise=true , address=77.90.31.21:4877 , piece_id=FLW7LIMJEC7YIRLKXN67BRBDCZ5M6GJ7BBQP24UD23DSXGRVNKBA, seg_pos={0 0}, upload=479407]: Long tail cancellation due to optimal threshold Dec 28 10:12:28.360135645 (546h43m28.23121868s) [seg_pos={0 0}, upload=479407]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:12:28.360225937 (546h43m28.231308972s) [upload=479407]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:12:28.360230796 (546h43m28.231313831s) [upload=479407]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:12:28.655722823 (546h43m28.526805858s) [upload=479407]: Done uploading segments... Dec 28 10:12:28.655742540 (546h43m28.526825585s) [upload=479407]: Upload finished. err: <nil> Duration: 5.877369995s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.....XX...X..X.........X.........X...................X..XX..] duration: 5.693718222s trace: https://jaeger.dev.storj.io/trace/3df6ebd21e3261a FAIL (2025-12-28 10:14:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 8 10:12:25.751002002 (546h43m25.622085037s) [upload=479407]: Starting upload Dec 28 10:12:25.751076032 (546h43m25.622159088s) [upload=479407]: Done starting upload Dec 28 10:12:25.751202923 (546h43m25.622285959s) [upload=479407]: Uploading segments... Dec 28 10:12:25.751376043 (546h43m25.622459078s) [upload=479407]: Got next segment. Inline: false Dec 28 10:12:25.751435637 (546h43m25.622518672s) [upload=479407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:12:25.751440015 (546h43m25.62252305s) [upload=479407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:12:25.999029369 (546h43m25.870112404s) [seg_pos={0 0}, upload=479407]: Begin upload segment... Dec 28 10:12:25.999087810 (546h43m25.870170855s) [seg_pos={0 0}, upload=479407]: Done begin upload segment. Dec 28 10:12:25.999119601 (546h43m25.870202646s) [noise=true , address=45.91.116.11:4120 , piece_id=XLKNNWMCQ7C6DKLC3VQGKK7PD243EZELISP4NMENOLI46DGCL7TQ, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:25.999185977 (546h43m25.870269012s) [noise=true , address=145.79.227.48:4301 , piece_id=ZFOSU45PBQCFYPJGH6EAIH4VWRWLMYRC3ALRVRIGCDPM62YOY5LA, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:25.999724041 (546h43m25.870807076s) [noise=true , address=77.90.31.10:4520 , piece_id=WETUFKFYTBGHSYHUGZVBASQHNLX6ZBSHLI4IYKPTSF2GLVVO6W4A, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.000051184 (546h43m25.871134229s) [noise=true , address=77.90.31.21:4877 , piece_id=FLW7LIMJEC7YIRLKXN67BRBDCZ5M6GJ7BBQP24UD23DSXGRVNKBA, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.000393516 (546h43m25.871476561s) [noise=true , address=45.91.116.13:4156 , piece_id=Z5UHYFISTNNJG7OXZW5OCOSOI7VCVCXSFN2WNX72S34AWDLIAXTA, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.000710519 (546h43m25.871793554s) [noise=true , address=145.79.227.18:4007 , piece_id=7WTRLRG542RHHIKCUNG3GDJSD6LQQ4RARLNIBMU52QRSOPFIS7ZA, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.001059654 (546h43m25.872142689s) [noise=true , address=145.79.227.42:4122 , piece_id=XOGO3W2CB4TIH22UY64DSDSLUGJNGCF5DQBC5U5OOA3EVP56DUXA, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.001379444 (546h43m25.872462479s) [noise=true , address=77.90.31.18:4808 , piece_id=PGUSLD66RBMEJBKPCTZ7MXPCO7JRGWXYSWXNEXPNQGLFUFH3X5IQ, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.001750710 (546h43m25.872833745s) [noise=true , address=145.79.227.13:4804 , piece_id=7ZCAIPPPXT2IMUZ4XBVNCQHBGZ7QXJ4Y6KV6J4V5ZH2GJ4O6EGGQ, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.002298023 (546h43m25.873381058s) [noise=true , address=145.79.227.32:4835 , piece_id=AVLVCL2OE5OM3RBR7WKFYHTWYRMHBB6LQ5XAZWJR4SJYITANC32Q, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.002725286 (546h43m25.873808331s) [noise=true , address=145.79.227.46:4601 , piece_id=B3GNLPMUESSJEBVF2ZRI2FWKCQ5UHOJKRMTF3JVZWIVSYQ3FEO7A, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.003114979 (546h43m25.874198024s) [noise=true , address=45.91.116.12:4226 , piece_id=IYM3F5QBL7BRHYGWDSTP6LO4G3LHD7AAH4KY6OVCSWHRY4FBELGA, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.003507286 (546h43m25.874590321s) [noise=true , address=145.79.227.15:4875 , piece_id=AWJXUCJMY3SQQNHFLDPSEQKBPZNP5XGRW7NVFUTTCETUEN3ZXD3A, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.003893661 (546h43m25.874976706s) [noise=true , address=145.79.227.40:4051 , piece_id=UZ764WOMEUPWIAPXCNHKMFAFAGO3IITHQQZN4BZSGI7PLQFYBILQ, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.004306387 (546h43m25.875389432s) [noise=true , address=145.79.227.41:4086 , piece_id=UFRF3CAQ7XPEM27EGPB5QDM2Q7SCBCGM2E3OQ27GDHGATTCXPJ3Q, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.004719544 (546h43m25.875802589s) [noise=true , address=145.79.227.38:4979 , piece_id=27GUZM4NTZYC2B5CVEJHUW4HDIRCHACAMIBK5WMH6WUXNHBQE52A, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:27.734491693 (546h43m27.605574758s) [upload=479407]: Next returned nil segment Dec 28 10:12:27.734530317 (546h43m27.605613362s) [upload=479407]: Waiting for error group managing segments... Dec 28 10:12:28.018123596 (546h43m27.889206652s) [noise=true , address=145.79.227.18:4007 , piece_id=7WTRLRG542RHHIKCUNG3GDJSD6LQQ4RARLNIBMU52QRSOPFIS7ZA, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.024834725 (546h43m27.89591776s) [noise=true , address=145.79.227.38:4979 , piece_id=27GUZM4NTZYC2B5CVEJHUW4HDIRCHACAMIBK5WMH6WUXNHBQE52A, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.025075894 (546h43m27.896158939s) [noise=true , address=145.79.227.15:4875 , piece_id=AWJXUCJMY3SQQNHFLDPSEQKBPZNP5XGRW7NVFUTTCETUEN3ZXD3A, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.025588511 (546h43m27.896671556s) [noise=true , address=145.79.227.32:4835 , piece_id=AVLVCL2OE5OM3RBR7WKFYHTWYRMHBB6LQ5XAZWJR4SJYITANC32Q, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.026177292 (546h43m27.897260337s) [noise=true , address=145.79.227.41:4086 , piece_id=UFRF3CAQ7XPEM27EGPB5QDM2Q7SCBCGM2E3OQ27GDHGATTCXPJ3Q, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.026761184 (546h43m27.897844229s) [noise=true , address=145.79.227.48:4301 , piece_id=ZFOSU45PBQCFYPJGH6EAIH4VWRWLMYRC3ALRVRIGCDPM62YOY5LA, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.030818669 (546h43m27.901901704s) [noise=true , address=145.79.227.46:4601 , piece_id=B3GNLPMUESSJEBVF2ZRI2FWKCQ5UHOJKRMTF3JVZWIVSYQ3FEO7A, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.034124584 (546h43m27.90520762s) [noise=true , address=145.79.227.13:4804 , piece_id=7ZCAIPPPXT2IMUZ4XBVNCQHBGZ7QXJ4Y6KV6J4V5ZH2GJ4O6EGGQ, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.034564222 (546h43m27.905647257s) [noise=true , address=145.79.227.42:4122 , piece_id=XOGO3W2CB4TIH22UY64DSDSLUGJNGCF5DQBC5U5OOA3EVP56DUXA, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.101769883 (546h43m27.972852918s) [noise=true , address=145.79.227.40:4051 , piece_id=UZ764WOMEUPWIAPXCNHKMFAFAGO3IITHQQZN4BZSGI7PLQFYBILQ, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.240152857 (546h43m28.111235892s) [noise=true , address=45.91.116.12:4226 , piece_id=IYM3F5QBL7BRHYGWDSTP6LO4G3LHD7AAH4KY6OVCSWHRY4FBELGA, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.307608374 (546h43m28.17869141s) [noise=true , address=77.90.31.18:4808 , piece_id=PGUSLD66RBMEJBKPCTZ7MXPCO7JRGWXYSWXNEXPNQGLFUFH3X5IQ, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.359465499 (546h43m28.230548534s) [noise=true , address=77.90.31.10:4520 , piece_id=WETUFKFYTBGHSYHUGZVBASQHNLX6ZBSHLI4IYKPTSF2GLVVO6W4A, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.359492871 (546h43m28.230575916s) [seg_pos={0 0}, upload=479407]: Segment reached optimal threshold of 13 pieces. Dec 28 10:12:28.359826747 (546h43m28.230909782s) [noise=true , address=45.91.116.13:4156 , piece_id=Z5UHYFISTNNJG7OXZW5OCOSOI7VCVCXSFN2WNX72S34AWDLIAXTA, seg_pos={0 0}, upload=479407]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 28 10:12:28.359843228 (546h43m28.230926263s) [noise=true , address=45.91.116.13:4156 , piece_id=Z5UHYFISTNNJG7OXZW5OCOSOI7VCVCXSFN2WNX72S34AWDLIAXTA, seg_pos={0 0}, upload=479407]: Long tail cancellation due to optimal threshold Dec 28 10:12:28.359887582 (546h43m28.230970617s) [noise=true , address=45.91.116.11:4120 , piece_id=XLKNNWMCQ7C6DKLC3VQGKK7PD243EZELISP4NMENOLI46DGCL7TQ, seg_pos={0 0}, upload=479407]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:36868->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:36868->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36868->45.91.116.11:4120: use of closed network connection Dec 28 10:12:28.359895919 (546h43m28.230978954s) [noise=true , address=45.91.116.11:4120 , piece_id=XLKNNWMCQ7C6DKLC3VQGKK7PD243EZELISP4NMENOLI46DGCL7TQ, seg_pos={0 0}, upload=479407]: Long tail cancellation due to optimal threshold Dec 28 10:12:28.360084688 (546h43m28.231167733s) [noise=true , address=77.90.31.21:4877 , piece_id=FLW7LIMJEC7YIRLKXN67BRBDCZ5M6GJ7BBQP24UD23DSXGRVNKBA, seg_pos={0 0}, upload=479407]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 28 10:12:28.360091691 (546h43m28.231174736s) [noise=true , address=77.90.31.21:4877 , piece_id=FLW7LIMJEC7YIRLKXN67BRBDCZ5M6GJ7BBQP24UD23DSXGRVNKBA, seg_pos={0 0}, upload=479407]: Long tail cancellation due to optimal threshold Dec 28 10:12:28.360135645 (546h43m28.23121868s) [seg_pos={0 0}, upload=479407]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:12:28.360225937 (546h43m28.231308972s) [upload=479407]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:12:28.360230796 (546h43m28.231313831s) [upload=479407]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:12:28.655722823 (546h43m28.526805858s) [upload=479407]: Done uploading segments... Dec 28 10:12:28.655742540 (546h43m28.526825585s) [upload=479407]: Upload finished. err: <nil> Duration: 5.693746075s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [X...XXX..XX.X...X.X....XXXXX...X........X.....X.............] duration: 4.380190451s trace: https://jaeger.dev.storj.io/trace/3f41c3ec34adb72f FAIL (2025-12-28 10:14:35 UTC) orbiter warning: context deadline exceeded Duration: 4.380217492s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [.X...XXX..XX.X...X.X....XXXXX...X........X.....X............] duration: 4.753302511s trace: https://jaeger.dev.storj.io/trace/3f41c3ec34adb72f FAIL (2025-12-28 10:14:20 UTC) orbiter warning: context deadline exceeded Duration: 4.75340634s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [..X...XXX..XX.X...X.X....XXXXX...X........X.....X...........] duration: 3.442979395s trace: https://jaeger.dev.storj.io/trace/3f41c3ec34adb72f FAIL (2025-12-28 10:13:43 UTC) orbiter warning: context deadline exceeded Duration: 3.443006676s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [......XX...X..X.........X.........X...................X..XX.] duration: 6.842226295s trace: https://jaeger.dev.storj.io/trace/3df6ebd21e3261a FAIL (2025-12-28 10:13:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 8 10:12:25.751002002 (546h43m25.622085037s) [upload=479407]: Starting upload Dec 28 10:12:25.751076032 (546h43m25.622159088s) [upload=479407]: Done starting upload Dec 28 10:12:25.751202923 (546h43m25.622285959s) [upload=479407]: Uploading segments... Dec 28 10:12:25.751376043 (546h43m25.622459078s) [upload=479407]: Got next segment. Inline: false Dec 28 10:12:25.751435637 (546h43m25.622518672s) [upload=479407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:12:25.751440015 (546h43m25.62252305s) [upload=479407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:12:25.999029369 (546h43m25.870112404s) [seg_pos={0 0}, upload=479407]: Begin upload segment... Dec 28 10:12:25.999087810 (546h43m25.870170855s) [seg_pos={0 0}, upload=479407]: Done begin upload segment. Dec 28 10:12:25.999119601 (546h43m25.870202646s) [noise=true , address=45.91.116.11:4120 , piece_id=XLKNNWMCQ7C6DKLC3VQGKK7PD243EZELISP4NMENOLI46DGCL7TQ, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:25.999185977 (546h43m25.870269012s) [noise=true , address=145.79.227.48:4301 , piece_id=ZFOSU45PBQCFYPJGH6EAIH4VWRWLMYRC3ALRVRIGCDPM62YOY5LA, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:25.999724041 (546h43m25.870807076s) [noise=true , address=77.90.31.10:4520 , piece_id=WETUFKFYTBGHSYHUGZVBASQHNLX6ZBSHLI4IYKPTSF2GLVVO6W4A, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.000051184 (546h43m25.871134229s) [noise=true , address=77.90.31.21:4877 , piece_id=FLW7LIMJEC7YIRLKXN67BRBDCZ5M6GJ7BBQP24UD23DSXGRVNKBA, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.000393516 (546h43m25.871476561s) [noise=true , address=45.91.116.13:4156 , piece_id=Z5UHYFISTNNJG7OXZW5OCOSOI7VCVCXSFN2WNX72S34AWDLIAXTA, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.000710519 (546h43m25.871793554s) [noise=true , address=145.79.227.18:4007 , piece_id=7WTRLRG542RHHIKCUNG3GDJSD6LQQ4RARLNIBMU52QRSOPFIS7ZA, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.001059654 (546h43m25.872142689s) [noise=true , address=145.79.227.42:4122 , piece_id=XOGO3W2CB4TIH22UY64DSDSLUGJNGCF5DQBC5U5OOA3EVP56DUXA, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.001379444 (546h43m25.872462479s) [noise=true , address=77.90.31.18:4808 , piece_id=PGUSLD66RBMEJBKPCTZ7MXPCO7JRGWXYSWXNEXPNQGLFUFH3X5IQ, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.001750710 (546h43m25.872833745s) [noise=true , address=145.79.227.13:4804 , piece_id=7ZCAIPPPXT2IMUZ4XBVNCQHBGZ7QXJ4Y6KV6J4V5ZH2GJ4O6EGGQ, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.002298023 (546h43m25.873381058s) [noise=true , address=145.79.227.32:4835 , piece_id=AVLVCL2OE5OM3RBR7WKFYHTWYRMHBB6LQ5XAZWJR4SJYITANC32Q, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.002725286 (546h43m25.873808331s) [noise=true , address=145.79.227.46:4601 , piece_id=B3GNLPMUESSJEBVF2ZRI2FWKCQ5UHOJKRMTF3JVZWIVSYQ3FEO7A, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.003114979 (546h43m25.874198024s) [noise=true , address=45.91.116.12:4226 , piece_id=IYM3F5QBL7BRHYGWDSTP6LO4G3LHD7AAH4KY6OVCSWHRY4FBELGA, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.003507286 (546h43m25.874590321s) [noise=true , address=145.79.227.15:4875 , piece_id=AWJXUCJMY3SQQNHFLDPSEQKBPZNP5XGRW7NVFUTTCETUEN3ZXD3A, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.003893661 (546h43m25.874976706s) [noise=true , address=145.79.227.40:4051 , piece_id=UZ764WOMEUPWIAPXCNHKMFAFAGO3IITHQQZN4BZSGI7PLQFYBILQ, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.004306387 (546h43m25.875389432s) [noise=true , address=145.79.227.41:4086 , piece_id=UFRF3CAQ7XPEM27EGPB5QDM2Q7SCBCGM2E3OQ27GDHGATTCXPJ3Q, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.004719544 (546h43m25.875802589s) [noise=true , address=145.79.227.38:4979 , piece_id=27GUZM4NTZYC2B5CVEJHUW4HDIRCHACAMIBK5WMH6WUXNHBQE52A, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:27.734491693 (546h43m27.605574758s) [upload=479407]: Next returned nil segment Dec 28 10:12:27.734530317 (546h43m27.605613362s) [upload=479407]: Waiting for error group managing segments... Dec 28 10:12:28.018123596 (546h43m27.889206652s) [noise=true , address=145.79.227.18:4007 , piece_id=7WTRLRG542RHHIKCUNG3GDJSD6LQQ4RARLNIBMU52QRSOPFIS7ZA, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.024834725 (546h43m27.89591776s) [noise=true , address=145.79.227.38:4979 , piece_id=27GUZM4NTZYC2B5CVEJHUW4HDIRCHACAMIBK5WMH6WUXNHBQE52A, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.025075894 (546h43m27.896158939s) [noise=true , address=145.79.227.15:4875 , piece_id=AWJXUCJMY3SQQNHFLDPSEQKBPZNP5XGRW7NVFUTTCETUEN3ZXD3A, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.025588511 (546h43m27.896671556s) [noise=true , address=145.79.227.32:4835 , piece_id=AVLVCL2OE5OM3RBR7WKFYHTWYRMHBB6LQ5XAZWJR4SJYITANC32Q, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.026177292 (546h43m27.897260337s) [noise=true , address=145.79.227.41:4086 , piece_id=UFRF3CAQ7XPEM27EGPB5QDM2Q7SCBCGM2E3OQ27GDHGATTCXPJ3Q, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.026761184 (546h43m27.897844229s) [noise=true , address=145.79.227.48:4301 , piece_id=ZFOSU45PBQCFYPJGH6EAIH4VWRWLMYRC3ALRVRIGCDPM62YOY5LA, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.030818669 (546h43m27.901901704s) [noise=true , address=145.79.227.46:4601 , piece_id=B3GNLPMUESSJEBVF2ZRI2FWKCQ5UHOJKRMTF3JVZWIVSYQ3FEO7A, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.034124584 (546h43m27.90520762s) [noise=true , address=145.79.227.13:4804 , piece_id=7ZCAIPPPXT2IMUZ4XBVNCQHBGZ7QXJ4Y6KV6J4V5ZH2GJ4O6EGGQ, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.034564222 (546h43m27.905647257s) [noise=true , address=145.79.227.42:4122 , piece_id=XOGO3W2CB4TIH22UY64DSDSLUGJNGCF5DQBC5U5OOA3EVP56DUXA, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.101769883 (546h43m27.972852918s) [noise=true , address=145.79.227.40:4051 , piece_id=UZ764WOMEUPWIAPXCNHKMFAFAGO3IITHQQZN4BZSGI7PLQFYBILQ, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.240152857 (546h43m28.111235892s) [noise=true , address=45.91.116.12:4226 , piece_id=IYM3F5QBL7BRHYGWDSTP6LO4G3LHD7AAH4KY6OVCSWHRY4FBELGA, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.307608374 (546h43m28.17869141s) [noise=true , address=77.90.31.18:4808 , piece_id=PGUSLD66RBMEJBKPCTZ7MXPCO7JRGWXYSWXNEXPNQGLFUFH3X5IQ, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.359465499 (546h43m28.230548534s) [noise=true , address=77.90.31.10:4520 , piece_id=WETUFKFYTBGHSYHUGZVBASQHNLX6ZBSHLI4IYKPTSF2GLVVO6W4A, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.359492871 (546h43m28.230575916s) [seg_pos={0 0}, upload=479407]: Segment reached optimal threshold of 13 pieces. Dec 28 10:12:28.359826747 (546h43m28.230909782s) [noise=true , address=45.91.116.13:4156 , piece_id=Z5UHYFISTNNJG7OXZW5OCOSOI7VCVCXSFN2WNX72S34AWDLIAXTA, seg_pos={0 0}, upload=479407]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 28 10:12:28.359843228 (546h43m28.230926263s) [noise=true , address=45.91.116.13:4156 , piece_id=Z5UHYFISTNNJG7OXZW5OCOSOI7VCVCXSFN2WNX72S34AWDLIAXTA, seg_pos={0 0}, upload=479407]: Long tail cancellation due to optimal threshold Dec 28 10:12:28.359887582 (546h43m28.230970617s) [noise=true , address=45.91.116.11:4120 , piece_id=XLKNNWMCQ7C6DKLC3VQGKK7PD243EZELISP4NMENOLI46DGCL7TQ, seg_pos={0 0}, upload=479407]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:36868->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:36868->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36868->45.91.116.11:4120: use of closed network connection Dec 28 10:12:28.359895919 (546h43m28.230978954s) [noise=true , address=45.91.116.11:4120 , piece_id=XLKNNWMCQ7C6DKLC3VQGKK7PD243EZELISP4NMENOLI46DGCL7TQ, seg_pos={0 0}, upload=479407]: Long tail cancellation due to optimal threshold Dec 28 10:12:28.360084688 (546h43m28.231167733s) [noise=true , address=77.90.31.21:4877 , piece_id=FLW7LIMJEC7YIRLKXN67BRBDCZ5M6GJ7BBQP24UD23DSXGRVNKBA, seg_pos={0 0}, upload=479407]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 28 10:12:28.360091691 (546h43m28.231174736s) [noise=true , address=77.90.31.21:4877 , piece_id=FLW7LIMJEC7YIRLKXN67BRBDCZ5M6GJ7BBQP24UD23DSXGRVNKBA, seg_pos={0 0}, upload=479407]: Long tail cancellation due to optimal threshold Dec 28 10:12:28.360135645 (546h43m28.23121868s) [seg_pos={0 0}, upload=479407]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:12:28.360225937 (546h43m28.231308972s) [upload=479407]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:12:28.360230796 (546h43m28.231313831s) [upload=479407]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:12:28.655722823 (546h43m28.526805858s) [upload=479407]: Done uploading segments... Dec 28 10:12:28.655742540 (546h43m28.526825585s) [upload=479407]: Upload finished. err: <nil> Duration: 6.842255611s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X......XX...X..X.........X.........X...................X..XX] duration: 30.003484167s trace: https://jaeger.dev.storj.io/trace/3df6ebd21e3261a FAIL (2025-12-28 10:12:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 8 10:12:25.751002002 (546h43m25.622085037s) [upload=479407]: Starting upload Dec 28 10:12:25.751076032 (546h43m25.622159088s) [upload=479407]: Done starting upload Dec 28 10:12:25.751202923 (546h43m25.622285959s) [upload=479407]: Uploading segments... Dec 28 10:12:25.751376043 (546h43m25.622459078s) [upload=479407]: Got next segment. Inline: false Dec 28 10:12:25.751435637 (546h43m25.622518672s) [upload=479407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:12:25.751440015 (546h43m25.62252305s) [upload=479407]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:12:25.999029369 (546h43m25.870112404s) [seg_pos={0 0}, upload=479407]: Begin upload segment... Dec 28 10:12:25.999087810 (546h43m25.870170855s) [seg_pos={0 0}, upload=479407]: Done begin upload segment. Dec 28 10:12:25.999119601 (546h43m25.870202646s) [noise=true , address=45.91.116.11:4120 , piece_id=XLKNNWMCQ7C6DKLC3VQGKK7PD243EZELISP4NMENOLI46DGCL7TQ, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:25.999185977 (546h43m25.870269012s) [noise=true , address=145.79.227.48:4301 , piece_id=ZFOSU45PBQCFYPJGH6EAIH4VWRWLMYRC3ALRVRIGCDPM62YOY5LA, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:25.999724041 (546h43m25.870807076s) [noise=true , address=77.90.31.10:4520 , piece_id=WETUFKFYTBGHSYHUGZVBASQHNLX6ZBSHLI4IYKPTSF2GLVVO6W4A, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.000051184 (546h43m25.871134229s) [noise=true , address=77.90.31.21:4877 , piece_id=FLW7LIMJEC7YIRLKXN67BRBDCZ5M6GJ7BBQP24UD23DSXGRVNKBA, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.000393516 (546h43m25.871476561s) [noise=true , address=45.91.116.13:4156 , piece_id=Z5UHYFISTNNJG7OXZW5OCOSOI7VCVCXSFN2WNX72S34AWDLIAXTA, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.000710519 (546h43m25.871793554s) [noise=true , address=145.79.227.18:4007 , piece_id=7WTRLRG542RHHIKCUNG3GDJSD6LQQ4RARLNIBMU52QRSOPFIS7ZA, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.001059654 (546h43m25.872142689s) [noise=true , address=145.79.227.42:4122 , piece_id=XOGO3W2CB4TIH22UY64DSDSLUGJNGCF5DQBC5U5OOA3EVP56DUXA, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.001379444 (546h43m25.872462479s) [noise=true , address=77.90.31.18:4808 , piece_id=PGUSLD66RBMEJBKPCTZ7MXPCO7JRGWXYSWXNEXPNQGLFUFH3X5IQ, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.001750710 (546h43m25.872833745s) [noise=true , address=145.79.227.13:4804 , piece_id=7ZCAIPPPXT2IMUZ4XBVNCQHBGZ7QXJ4Y6KV6J4V5ZH2GJ4O6EGGQ, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.002298023 (546h43m25.873381058s) [noise=true , address=145.79.227.32:4835 , piece_id=AVLVCL2OE5OM3RBR7WKFYHTWYRMHBB6LQ5XAZWJR4SJYITANC32Q, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.002725286 (546h43m25.873808331s) [noise=true , address=145.79.227.46:4601 , piece_id=B3GNLPMUESSJEBVF2ZRI2FWKCQ5UHOJKRMTF3JVZWIVSYQ3FEO7A, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.003114979 (546h43m25.874198024s) [noise=true , address=45.91.116.12:4226 , piece_id=IYM3F5QBL7BRHYGWDSTP6LO4G3LHD7AAH4KY6OVCSWHRY4FBELGA, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.003507286 (546h43m25.874590321s) [noise=true , address=145.79.227.15:4875 , piece_id=AWJXUCJMY3SQQNHFLDPSEQKBPZNP5XGRW7NVFUTTCETUEN3ZXD3A, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.003893661 (546h43m25.874976706s) [noise=true , address=145.79.227.40:4051 , piece_id=UZ764WOMEUPWIAPXCNHKMFAFAGO3IITHQQZN4BZSGI7PLQFYBILQ, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.004306387 (546h43m25.875389432s) [noise=true , address=145.79.227.41:4086 , piece_id=UFRF3CAQ7XPEM27EGPB5QDM2Q7SCBCGM2E3OQ27GDHGATTCXPJ3Q, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:26.004719544 (546h43m25.875802589s) [noise=true , address=145.79.227.38:4979 , piece_id=27GUZM4NTZYC2B5CVEJHUW4HDIRCHACAMIBK5WMH6WUXNHBQE52A, seg_pos={0 0}, upload=479407]: Uploading piece... Dec 28 10:12:27.734491693 (546h43m27.605574758s) [upload=479407]: Next returned nil segment Dec 28 10:12:27.734530317 (546h43m27.605613362s) [upload=479407]: Waiting for error group managing segments... Dec 28 10:12:28.018123596 (546h43m27.889206652s) [noise=true , address=145.79.227.18:4007 , piece_id=7WTRLRG542RHHIKCUNG3GDJSD6LQQ4RARLNIBMU52QRSOPFIS7ZA, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.024834725 (546h43m27.89591776s) [noise=true , address=145.79.227.38:4979 , piece_id=27GUZM4NTZYC2B5CVEJHUW4HDIRCHACAMIBK5WMH6WUXNHBQE52A, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.025075894 (546h43m27.896158939s) [noise=true , address=145.79.227.15:4875 , piece_id=AWJXUCJMY3SQQNHFLDPSEQKBPZNP5XGRW7NVFUTTCETUEN3ZXD3A, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.025588511 (546h43m27.896671556s) [noise=true , address=145.79.227.32:4835 , piece_id=AVLVCL2OE5OM3RBR7WKFYHTWYRMHBB6LQ5XAZWJR4SJYITANC32Q, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.026177292 (546h43m27.897260337s) [noise=true , address=145.79.227.41:4086 , piece_id=UFRF3CAQ7XPEM27EGPB5QDM2Q7SCBCGM2E3OQ27GDHGATTCXPJ3Q, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.026761184 (546h43m27.897844229s) [noise=true , address=145.79.227.48:4301 , piece_id=ZFOSU45PBQCFYPJGH6EAIH4VWRWLMYRC3ALRVRIGCDPM62YOY5LA, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.030818669 (546h43m27.901901704s) [noise=true , address=145.79.227.46:4601 , piece_id=B3GNLPMUESSJEBVF2ZRI2FWKCQ5UHOJKRMTF3JVZWIVSYQ3FEO7A, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.034124584 (546h43m27.90520762s) [noise=true , address=145.79.227.13:4804 , piece_id=7ZCAIPPPXT2IMUZ4XBVNCQHBGZ7QXJ4Y6KV6J4V5ZH2GJ4O6EGGQ, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.034564222 (546h43m27.905647257s) [noise=true , address=145.79.227.42:4122 , piece_id=XOGO3W2CB4TIH22UY64DSDSLUGJNGCF5DQBC5U5OOA3EVP56DUXA, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.101769883 (546h43m27.972852918s) [noise=true , address=145.79.227.40:4051 , piece_id=UZ764WOMEUPWIAPXCNHKMFAFAGO3IITHQQZN4BZSGI7PLQFYBILQ, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.240152857 (546h43m28.111235892s) [noise=true , address=45.91.116.12:4226 , piece_id=IYM3F5QBL7BRHYGWDSTP6LO4G3LHD7AAH4KY6OVCSWHRY4FBELGA, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.307608374 (546h43m28.17869141s) [noise=true , address=77.90.31.18:4808 , piece_id=PGUSLD66RBMEJBKPCTZ7MXPCO7JRGWXYSWXNEXPNQGLFUFH3X5IQ, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.359465499 (546h43m28.230548534s) [noise=true , address=77.90.31.10:4520 , piece_id=WETUFKFYTBGHSYHUGZVBASQHNLX6ZBSHLI4IYKPTSF2GLVVO6W4A, seg_pos={0 0}, upload=479407]: Done uploading piece. err: <nil> Dec 28 10:12:28.359492871 (546h43m28.230575916s) [seg_pos={0 0}, upload=479407]: Segment reached optimal threshold of 13 pieces. Dec 28 10:12:28.359826747 (546h43m28.230909782s) [noise=true , address=45.91.116.13:4156 , piece_id=Z5UHYFISTNNJG7OXZW5OCOSOI7VCVCXSFN2WNX72S34AWDLIAXTA, seg_pos={0 0}, upload=479407]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 28 10:12:28.359843228 (546h43m28.230926263s) [noise=true , address=45.91.116.13:4156 , piece_id=Z5UHYFISTNNJG7OXZW5OCOSOI7VCVCXSFN2WNX72S34AWDLIAXTA, seg_pos={0 0}, upload=479407]: Long tail cancellation due to optimal threshold Dec 28 10:12:28.359887582 (546h43m28.230970617s) [noise=true , address=45.91.116.11:4120 , piece_id=XLKNNWMCQ7C6DKLC3VQGKK7PD243EZELISP4NMENOLI46DGCL7TQ, seg_pos={0 0}, upload=479407]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:36868->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:36868->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36868->45.91.116.11:4120: use of closed network connection Dec 28 10:12:28.359895919 (546h43m28.230978954s) [noise=true , address=45.91.116.11:4120 , piece_id=XLKNNWMCQ7C6DKLC3VQGKK7PD243EZELISP4NMENOLI46DGCL7TQ, seg_pos={0 0}, upload=479407]: Long tail cancellation due to optimal threshold Dec 28 10:12:28.360084688 (546h43m28.231167733s) [noise=true , address=77.90.31.21:4877 , piece_id=FLW7LIMJEC7YIRLKXN67BRBDCZ5M6GJ7BBQP24UD23DSXGRVNKBA, seg_pos={0 0}, upload=479407]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 28 10:12:28.360091691 (546h43m28.231174736s) [noise=true , address=77.90.31.21:4877 , piece_id=FLW7LIMJEC7YIRLKXN67BRBDCZ5M6GJ7BBQP24UD23DSXGRVNKBA, seg_pos={0 0}, upload=479407]: Long tail cancellation due to optimal threshold Dec 28 10:12:28.360135645 (546h43m28.23121868s) [seg_pos={0 0}, upload=479407]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:12:28.360225937 (546h43m28.231308972s) [upload=479407]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:12:28.360230796 (546h43m28.231313831s) [upload=479407]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:12:28.655722823 (546h43m28.526805858s) [upload=479407]: Done uploading segments... Dec 28 10:12:28.655742540 (546h43m28.526825585s) [upload=479407]: Upload finished. err: <nil> Duration: 30.00349586s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [X..X...XXX..XX.X...X.X....XXXXX...X........X.....X..........] duration: 3.395592504s trace: https://jaeger.dev.storj.io/trace/3f41c3ec34adb72f FAIL (2025-12-28 10:12:45 UTC) orbiter warning: context deadline exceeded Duration: 3.395614926s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X......XX...X..X.........X.........X...................X..X] duration: 30.004324624s trace: https://jaeger.dev.storj.io/trace/74796d7ff799ca6 FAIL (2025-12-28 10:12:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: LMHMF3TNFPO6VKRXWZQQLA4JO62NKIKQUVDZ2DUPFUJNVLTIVDJQ) context deadline exceeded; piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: C45D52ALAAHI4EV6HIOCMWDN3GIHV6WQYVGFZ2PEUITKFCIDXWMQ) context deadline exceeded ======LOGS====== Dec 28 10:11:55.508656938 (546h42m55.379739983s) [upload=479401]: Starting upload Dec 28 10:11:55.508752521 (546h42m55.379835566s) [upload=479401]: Done starting upload Dec 28 10:11:55.508846228 (546h42m55.379929263s) [upload=479401]: Uploading segments... Dec 28 10:11:55.509063603 (546h42m55.380146648s) [upload=479401]: Got next segment. Inline: false Dec 28 10:11:55.509108909 (546h42m55.380191954s) [upload=479401]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:11:55.509115341 (546h42m55.380198386s) [upload=479401]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:11:55.755431018 (546h42m55.626514063s) [seg_pos={0 0}, upload=479401]: Begin upload segment... Dec 28 10:11:55.755478698 (546h42m55.626561743s) [seg_pos={0 0}, upload=479401]: Done begin upload segment. Dec 28 10:11:55.755525939 (546h42m55.626608974s) [noise=true , address=77.90.31.17:4770 , piece_id=CX67R6HA2PB4MQ7BLMZMQ25W4OET556XONSPAFCW6N4BYGNMLLDA, seg_pos={0 0}, upload=479401]: Uploading piece... Dec 28 10:11:55.755560054 (546h42m55.626643089s) [noise=true , address=145.79.227.2:4115 , piece_id=OKHUL765NXRTHYADKATEB5INELKKU3SPJKZWSNGYBMQGBATIXK7A, seg_pos={0 0}, upload=479401]: Uploading piece... Dec 28 10:11:55.756194121 (546h42m55.627277156s) [noise=true , address=145.79.227.43:4158 , piece_id=S5CQZM2YWWMBEHTJREMYLSS5QBA4KY72EIQA7BVTUSEV2QERPSJQ, seg_pos={0 0}, upload=479401]: Uploading piece... Dec 28 10:11:55.756218738 (546h42m55.627301773s) [noise=true , address=145.79.227.39:4014 , piece_id=ZGYJ23AV25R2PQWD27PM2GI753RZEZBSXC6BXHUSVBV3OMRZOE7A, seg_pos={0 0}, upload=479401]: Uploading piece... Dec 28 10:11:55.756614101 (546h42m55.627697146s) [noise=true , address=145.79.227.38:4979 , piece_id=MHDSRX3KMIVGTCTF6QGKNKAD53SHH4A6LEJ3XUTPOEL2EHGEAJOQ, seg_pos={0 0}, upload=479401]: Uploading piece... Dec 28 10:11:55.756785808 (546h42m55.627868853s) [noise=true , address=145.79.227.46:4603 , piece_id=IZLLSPMEP3VFF3MIGQZE3MMNSLXLFH3OKRR2VJXFX73KOO5SKM6A, seg_pos={0 0}, upload=479401]: Uploading piece... Dec 28 10:11:55.757008451 (546h42m55.628091486s) [noise=true , address=45.91.116.12:4225 , piece_id=BAIOLNIY7AWP72LKXN4RAICHY3QMDLLPCHM2R5FC6PDJDSTAV6NA, seg_pos={0 0}, upload=479401]: Uploading piece... Dec 28 10:11:55.757130795 (546h42m55.62821384s) [noise=true , address=145.79.227.5:4221 , piece_id=WZBB3OCGC53IBVZ6PGCU6ZWSTMWH5WXNGR5AFUD2IKDMW7R2BDGA, seg_pos={0 0}, upload=479401]: Uploading piece... Dec 28 10:11:55.757401751 (546h42m55.628484786s) [noise=true , address=145.79.227.3:4150 , piece_id=TCIFHASPJXVUHQS6XVOPYURHGBHYG4SCNG3H74AHWJNOHVYTRTKA, seg_pos={0 0}, upload=479401]: Uploading piece... Dec 28 10:11:55.757510498 (546h42m55.628593533s) [noise=true , address=145.79.227.37:4565 , piece_id=NPFMTL5CXTGE3FY4AA2HKRLDBVMOHNOQZKNG2HSVRGFP737APVWQ, seg_pos={0 0}, upload=479401]: Uploading piece... Dec 28 10:11:55.758579663 (546h42m55.629662708s) [noise=true , address=45.91.116.10:4081 , piece_id=LMHMF3TNFPO6VKRXWZQQLA4JO62NKIKQUVDZ2DUPFUJNVLTIVDJQ, seg_pos={0 0}, upload=479401]: Uploading piece... Dec 28 10:11:55.758622746 (546h42m55.629705791s) [noise=true , address=77.90.31.18:4808 , piece_id=C45D52ALAAHI4EV6HIOCMWDN3GIHV6WQYVGFZ2PEUITKFCIDXWMQ, seg_pos={0 0}, upload=479401]: Uploading piece... Dec 28 10:11:55.759192841 (546h42m55.630275886s) [noise=true , address=45.91.116.6:4976 , piece_id=J6I3RXSBVEBE22KUWFL36PY6DHTQNVWUOYWPQLE2E7LVHQAMIACQ, seg_pos={0 0}, upload=479401]: Uploading piece... Dec 28 10:11:55.760347018 (546h42m55.631430054s) [noise=true , address=77.90.31.13:4629 , piece_id=Z2BWHOXCPGB76CJ5C74HTSWIXHRY3DIS47UYBOC7OYLUVY7ZK7ZA, seg_pos={0 0}, upload=479401]: Uploading piece... Dec 28 10:11:55.760372267 (546h42m55.631455322s) [noise=true , address=145.79.227.15:4876 , piece_id=6X7BDS2T5FLSMEJ2FXPC64H2JDK3IQXE77SYOMSHPUOJXEXGZIQA, seg_pos={0 0}, upload=479401]: Uploading piece... Dec 28 10:11:55.762818024 (546h42m55.633901079s) [noise=true , address=45.91.116.9:4264 , piece_id=OIM5OQTUD3I3BKUE7UUWIXDJK5SL6HZ3YBRGQZVWGX7WEHFCFJFQ, seg_pos={0 0}, upload=479401]: Uploading piece... Dec 28 10:11:57.481578765 (546h42m57.3526618s) [upload=479401]: Next returned nil segment Dec 28 10:11:57.481599003 (546h42m57.352682048s) [upload=479401]: Waiting for error group managing segments... Dec 28 10:11:57.768968814 (546h42m57.640051859s) [noise=true , address=145.79.227.39:4014 , piece_id=ZGYJ23AV25R2PQWD27PM2GI753RZEZBSXC6BXHUSVBV3OMRZOE7A, seg_pos={0 0}, upload=479401]: Done uploading piece. err: <nil> Dec 28 10:11:57.771490266 (546h42m57.642573311s) [noise=true , address=145.79.227.2:4115 , piece_id=OKHUL765NXRTHYADKATEB5INELKKU3SPJKZWSNGYBMQGBATIXK7A, seg_pos={0 0}, upload=479401]: Done uploading piece. err: <nil> Dec 28 10:11:57.772781745 (546h42m57.64386479s) [noise=true , address=145.79.227.46:4603 , piece_id=IZLLSPMEP3VFF3MIGQZE3MMNSLXLFH3OKRR2VJXFX73KOO5SKM6A, seg_pos={0 0}, upload=479401]: Done uploading piece. err: <nil> Dec 28 10:11:57.774976594 (546h42m57.646059639s) [noise=true , address=145.79.227.38:4979 , piece_id=MHDSRX3KMIVGTCTF6QGKNKAD53SHH4A6LEJ3XUTPOEL2EHGEAJOQ, seg_pos={0 0}, upload=479401]: Done uploading piece. err: <nil> Dec 28 10:11:57.775713337 (546h42m57.646796372s) [noise=true , address=145.79.227.5:4221 , piece_id=WZBB3OCGC53IBVZ6PGCU6ZWSTMWH5WXNGR5AFUD2IKDMW7R2BDGA, seg_pos={0 0}, upload=479401]: Done uploading piece. err: <nil> Dec 28 10:11:57.779020234 (546h42m57.650103269s) [noise=true , address=145.79.227.37:4565 , piece_id=NPFMTL5CXTGE3FY4AA2HKRLDBVMOHNOQZKNG2HSVRGFP737APVWQ, seg_pos={0 0}, upload=479401]: Done uploading piece. err: <nil> Dec 28 10:11:57.786723031 (546h42m57.657806066s) [noise=true , address=145.79.227.3:4150 , piece_id=TCIFHASPJXVUHQS6XVOPYURHGBHYG4SCNG3H74AHWJNOHVYTRTKA, seg_pos={0 0}, upload=479401]: Done uploading piece. err: <nil> Dec 28 10:11:57.808659452 (546h42m57.679742497s) [noise=true , address=145.79.227.15:4876 , piece_id=6X7BDS2T5FLSMEJ2FXPC64H2JDK3IQXE77SYOMSHPUOJXEXGZIQA, seg_pos={0 0}, upload=479401]: Done uploading piece. err: <nil> Dec 28 10:11:57.973920306 (546h42m57.845003351s) [noise=true , address=77.90.31.18:4808 , piece_id=C45D52ALAAHI4EV6HIOCMWDN3GIHV6WQYVGFZ2PEUITKFCIDXWMQ, seg_pos={0 0}, upload=479401]: Done uploading piece. err: <nil> Dec 28 10:11:58.035542335 (546h42m57.90662538s) [noise=true , address=77.90.31.13:4629 , piece_id=Z2BWHOXCPGB76CJ5C74HTSWIXHRY3DIS47UYBOC7OYLUVY7ZK7ZA, seg_pos={0 0}, upload=479401]: Done uploading piece. err: <nil> Dec 28 10:11:58.088197751 (546h42m57.959280786s) [noise=true , address=45.91.116.12:4225 , piece_id=BAIOLNIY7AWP72LKXN4RAICHY3QMDLLPCHM2R5FC6PDJDSTAV6NA, seg_pos={0 0}, upload=479401]: Done uploading piece. err: <nil> Dec 28 10:11:58.090672513 (546h42m57.961755558s) [noise=true , address=45.91.116.10:4081 , piece_id=LMHMF3TNFPO6VKRXWZQQLA4JO62NKIKQUVDZ2DUPFUJNVLTIVDJQ, seg_pos={0 0}, upload=479401]: Done uploading piece. err: <nil> Dec 28 10:11:58.107381740 (546h42m57.978464775s) [noise=true , address=45.91.116.6:4976 , piece_id=J6I3RXSBVEBE22KUWFL36PY6DHTQNVWUOYWPQLE2E7LVHQAMIACQ, seg_pos={0 0}, upload=479401]: Done uploading piece. err: <nil> Dec 28 10:11:58.107410525 (546h42m57.97849356s) [seg_pos={0 0}, upload=479401]: Segment reached optimal threshold of 13 pieces. Dec 28 10:11:58.107636385 (546h42m57.97871942s) [noise=true , address=145.79.227.43:4158 , piece_id=S5CQZM2YWWMBEHTJREMYLSS5QBA4KY72EIQA7BVTUSEV2QERPSJQ, seg_pos={0 0}, upload=479401]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 28 10:11:58.107644981 (546h42m57.978728016s) [noise=true , address=145.79.227.43:4158 , piece_id=S5CQZM2YWWMBEHTJREMYLSS5QBA4KY72EIQA7BVTUSEV2QERPSJQ, seg_pos={0 0}, upload=479401]: Long tail cancellation due to optimal threshold Dec 28 10:11:58.107831046 (546h42m57.978914091s) [noise=true , address=77.90.31.17:4770 , piece_id=CX67R6HA2PB4MQ7BLMZMQ25W4OET556XONSPAFCW6N4BYGNMLLDA, seg_pos={0 0}, upload=479401]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 28 10:11:58.107857135 (546h42m57.97894017s) [noise=true , address=77.90.31.17:4770 , piece_id=CX67R6HA2PB4MQ7BLMZMQ25W4OET556XONSPAFCW6N4BYGNMLLDA, seg_pos={0 0}, upload=479401]: Long tail cancellation due to optimal threshold Dec 28 10:11:58.107943800 (546h42m57.979026835s) [noise=true , address=45.91.116.9:4264 , piece_id=OIM5OQTUD3I3BKUE7UUWIXDJK5SL6HZ3YBRGQZVWGX7WEHFCFJFQ, seg_pos={0 0}, upload=479401]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 28 10:11:58.107954540 (546h42m57.979037575s) [noise=true , address=45.91.116.9:4264 , piece_id=OIM5OQTUD3I3BKUE7UUWIXDJK5SL6HZ3YBRGQZVWGX7WEHFCFJFQ, seg_pos={0 0}, upload=479401]: Long tail cancellation due to optimal threshold Dec 28 10:11:58.107992032 (546h42m57.979075087s) [seg_pos={0 0}, upload=479401]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:11:58.108123091 (546h42m57.979206136s) [upload=479401]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:11:58.108132369 (546h42m57.979215414s) [upload=479401]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:11:58.397875990 (546h42m58.268959035s) [upload=479401]: Done uploading segments... Dec 28 10:11:58.397900738 (546h42m58.268983783s) [upload=479401]: Upload finished. err: <nil> Duration: 30.004341536s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [XX..X...XXX..XX.X...X.X....XXXXX...X........X.....X.........] duration: 4.35521229s trace: https://jaeger.dev.storj.io/trace/3f41c3ec34adb72f FAIL (2025-12-28 10:11:58 UTC) orbiter warning: context deadline exceeded Duration: 4.355242298s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [.XX..X...XXX..XX.X...X.X....XXXXX...X........X.....X........] duration: 4.248661701s trace: https://jaeger.dev.storj.io/trace/3f41c3ec34adb72f FAIL (2025-12-28 10:11:15 UTC) orbiter warning: context deadline exceeded Duration: 4.248694794s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..X......XX...X..X.........X.........X...................X..] duration: 5.814962672s trace: https://jaeger.dev.storj.io/trace/298f7e849145378d FAIL (2025-12-28 10:10:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 546h40m49.686771476s) [upload=479365]: Done starting upload Dec 28 10:09:49.815901106 (546h40m49.686984141s) [upload=479365]: Uploading segments... Dec 28 10:09:49.815945662 (546h40m49.687028707s) [upload=479365]: Got next segment. Inline: false Dec 28 10:09:49.815975247 (546h40m49.687058292s) [upload=479365]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:09:49.815979797 (546h40m49.687062832s) [upload=479365]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:09:50.069821454 (546h40m49.940904489s) [seg_pos={0 0}, upload=479365]: Begin upload segment... Dec 28 10:09:50.069862241 (546h40m49.940945276s) [seg_pos={0 0}, upload=479365]: Done begin upload segment. Dec 28 10:09:50.069897478 (546h40m49.940980513s) [noise=true , address=145.79.227.4:4187 , piece_id=MWP3DU5HEPGFEWVGNNAX5QNY2OZCB4L3ZQ3GGNUPUJF5J6UHZCSA, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.070930515 (546h40m49.94201356s) [noise=true , address=77.90.31.15:4697 , piece_id=MF3WHAUG3ER5SAAC67CFC77WYSBO7TO6ZGAOTB6KUIMM6R66AJ5Q, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.071091021 (546h40m49.942174056s) [noise=true , address=145.79.227.11:4730 , piece_id=7TTT6UD33GMOKJJFFMN6AV227QDJNK3TREMSWUFNLIOA7PJIZGUQ, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.071334024 (546h40m49.942417069s) [noise=true , address=77.90.31.10:4520 , piece_id=4QKC3GYDDP3YEREIAEXW6NVKT5SPGFCK54U6K7UMWITPNEIH4HOQ, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.071567939 (546h40m49.942650974s) [noise=true , address=45.91.116.5:4298 , piece_id=GDPNW5S3Z3L3Q4ZBU6QLFY6STGHLMEHYDFDNBSCSKQX4LWFFP2FA, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.071694199 (546h40m49.942777244s) [noise=true , address=145.79.227.37:4566 , piece_id=ZZB5ADNELY6LXKBJCDRRAFU34J4ICCWJIO62FICBIPT7QCMSVOLQ, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.071916012 (546h40m49.942999047s) [noise=true , address=145.79.227.2:4114 , piece_id=OX6KOGTRDJSLP6KZYTKCC5ERQDNVLNK7LBF2TZ5DLME45LLCJ6QQ, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.072047642 (546h40m49.943130687s) [noise=true , address=145.79.227.48:4302 , piece_id=VUZQ2TDAQMD4BE3QFK57UWVBQJA3AWXLNNPUSHO4U2W7MNZW2DOA, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.072298921 (546h40m49.943381956s) [noise=true , address=145.79.227.34:4905 , piece_id=PTAKHKSPPU3K67IV6XN74OHAVWLF45X5ZHYXEVOR4KJGE7FNSIYQ, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.072401216 (546h40m49.943484261s) [noise=true , address=145.79.227.6:4258 , piece_id=QVL3KNSVPCGK3OD5LH5HLU5ZUGN5SSBYBVTL62IWZRC2VXVNLS7A, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.072623439 (546h40m49.943706474s) [noise=true , address=45.91.116.8:4048 , piece_id=ZG5KHG5OTVF2WTAMIPIR4DINKP5S4VQQCHNNUZDJ6L2TCQ465AHA, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.072767583 (546h40m49.943850628s) [noise=true , address=145.79.227.12:4765 , piece_id=YHHYUMZMXVYEOFL3X7KEM3D4EMEQ4MP63EKNKRF3Y43OL4CHDEQQ, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.072934110 (546h40m49.944017156s) [noise=true , address=145.79.227.21:4474 , piece_id=NYWNSVVEGNZWZRUXKY4WMLPUUC7W5PYZEAANP4W74OH35E55XYJQ, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.073134392 (546h40m49.944217427s) [noise=true , address=145.79.227.32:4835 , piece_id=F6U7THJV5UQU4RTCUMDKHCLGLWV7ALFNCEJUNSK3YXXHG5K3HO5A, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.073503574 (546h40m49.944586609s) [noise=true , address=145.79.227.16:4912 , piece_id=2SY5IGKX5GDCZPPTAGCVKDF5OCTIBHTMTD3W4N4PXLZD2W2JBAHQ, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.073878869 (546h40m49.944961904s) [noise=true , address=145.79.227.3:4149 , piece_id=33GQOMODSGCKTVBJQ4BTB5B7ILCBB4CCJADI4XDHQVMR547EPSIQ, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:51.795106108 (546h40m51.666189143s) [upload=479365]: Next returned nil segment Dec 28 10:09:51.795149140 (546h40m51.666232185s) [upload=479365]: Waiting for error group managing segments... Dec 28 10:09:52.082599142 (546h40m51.953682177s) [noise=true , address=145.79.227.2:4114 , piece_id=OX6KOGTRDJSLP6KZYTKCC5ERQDNVLNK7LBF2TZ5DLME45LLCJ6QQ, seg_pos={0 0}, upload=479365]: Done uploading piece. err: <nil> Dec 28 10:09:52.100437450 (546h40m51.971520485s) [noise=true , address=145.79.227.3:4149 , piece_id=33GQOMODSGCKTVBJQ4BTB5B7ILCBB4CCJADI4XDHQVMR547EPSIQ, seg_pos={0 0}, upload=479365]: Done uploading piece. err: <nil> Dec 28 10:09:52.101903622 (546h40m51.972986657s) [noise=true , address=145.79.227.37:4566 , piece_id=ZZB5ADNELY6LXKBJCDRRAFU34J4ICCWJIO62FICBIPT7QCMSVOLQ, seg_pos={0 0}, upload=479365]: Done uploading piece. err: <nil> Dec 28 10:09:52.103627946 (546h40m51.974710981s) [noise=true , address=145.79.227.32:4835 , piece_id=F6U7THJV5UQU4RTCUMDKHCLGLWV7ALFNCEJUNSK3YXXHG5K3HO5A, seg_pos={0 0}, upload=479365]: Done uploading piece. err: <nil> Dec 28 10:09:52.107846759 (546h40m51.978929794s) [noise=true , address=145.79.227.21:4474 , piece_id=NYWNSVVEGNZWZRUXKY4WMLPUUC7W5PYZEAANP4W74OH35E55XYJQ, seg_pos={0 0}, upload=479365]: Done uploading piece. err: <nil> Dec 28 10:09:52.111095154 (546h40m51.982178189s) [noise=true , address=145.79.227.48:4302 , piece_id=VUZQ2TDAQMD4BE3QFK57UWVBQJA3AWXLNNPUSHO4U2W7MNZW2DOA, seg_pos={0 0}, upload=479365]: Done uploading piece. err: <nil> Dec 28 10:09:52.113636133 (546h40m51.984719178s) [noise=true , address=145.79.227.12:4765 , piece_id=YHHYUMZMXVYEOFL3X7KEM3D4EMEQ4MP63EKNKRF3Y43OL4CHDEQQ, seg_pos={0 0}, upload=479365]: Done uploading piece. err: <nil> Dec 28 10:09:52.113768515 (546h40m51.98485155s) [noise=true , address=145.79.227.16:4912 , piece_id=2SY5IGKX5GDCZPPTAGCVKDF5OCTIBHTMTD3W4N4PXLZD2W2JBAHQ, seg_pos={0 0}, upload=479365]: Done uploading piece. err: <nil> Dec 28 10:09:52.118758556 (546h40m51.989841592s) [noise=true , address=145.79.227.4:4187 , piece_id=MWP3DU5HEPGFEWVGNNAX5QNY2OZCB4L3ZQ3GGNUPUJF5J6UHZCSA, seg_pos={0 0}, upload=479365]: Done uploading piece. err: <nil> Dec 28 10:09:52.119015597 (546h40m51.990098632s) [noise=true , address=145.79.227.11:4730 , piece_id=7TTT6UD33GMOKJJFFMN6AV227QDJNK3TREMSWUFNLIOA7PJIZGUQ, seg_pos={0 0}, upload=479365]: Done uploading piece. err: <nil> Dec 28 10:09:52.119331037 (546h40m51.990414082s) [noise=true , address=145.79.227.6:4258 , piece_id=QVL3KNSVPCGK3OD5LH5HLU5ZUGN5SSBYBVTL62IWZRC2VXVNLS7A, seg_pos={0 0}, upload=479365]: Done uploading piece. err: <nil> Dec 28 10:09:52.292892297 (546h40m52.163975342s) [noise=true , address=45.91.116.5:4298 , piece_id=GDPNW5S3Z3L3Q4ZBU6QLFY6STGHLMEHYDFDNBSCSKQX4LWFFP2FA, seg_pos={0 0}, upload=479365]: Done uploading piece. err: <nil> Dec 28 10:09:52.423498404 (546h40m52.294581449s) [noise=true , address=45.91.116.8:4048 , piece_id=ZG5KHG5OTVF2WTAMIPIR4DINKP5S4VQQCHNNUZDJ6L2TCQ465AHA, seg_pos={0 0}, upload=479365]: Done uploading piece. err: <nil> Dec 28 10:09:52.423618842 (546h40m52.294701887s) [seg_pos={0 0}, upload=479365]: Segment reached optimal threshold of 13 pieces. Dec 28 10:09:52.423873567 (546h40m52.294956602s) [noise=true , address=145.79.227.34:4905 , piece_id=PTAKHKSPPU3K67IV6XN74OHAVWLF45X5ZHYXEVOR4KJGE7FNSIYQ, seg_pos={0 0}, upload=479365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 28 10:09:52.423883487 (546h40m52.294966522s) [noise=true , address=145.79.227.34:4905 , piece_id=PTAKHKSPPU3K67IV6XN74OHAVWLF45X5ZHYXEVOR4KJGE7FNSIYQ, seg_pos={0 0}, upload=479365]: Long tail cancellation due to optimal threshold Dec 28 10:09:52.424357258 (546h40m52.295440313s) [noise=true , address=77.90.31.10:4520 , piece_id=4QKC3GYDDP3YEREIAEXW6NVKT5SPGFCK54U6K7UMWITPNEIH4HOQ, seg_pos={0 0}, upload=479365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:39386->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:39386->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39386->77.90.31.10:4520: use of closed network connection Dec 28 10:09:52.424369061 (546h40m52.295452117s) [noise=true , address=77.90.31.10:4520 , piece_id=4QKC3GYDDP3YEREIAEXW6NVKT5SPGFCK54U6K7UMWITPNEIH4HOQ, seg_pos={0 0}, upload=479365]: Long tail cancellation due to optimal threshold Dec 28 10:09:52.424762400 (546h40m52.295845465s) [noise=true , address=77.90.31.15:4697 , piece_id=MF3WHAUG3ER5SAAC67CFC77WYSBO7TO6ZGAOTB6KUIMM6R66AJ5Q, seg_pos={0 0}, upload=479365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: read tcp 188.166.225.170:40794->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 10:09:52.424771748 (546h40m52.295854783s) [noise=true , address=77.90.31.15:4697 , piece_id=MF3WHAUG3ER5SAAC67CFC77WYSBO7TO6ZGAOTB6KUIMM6R66AJ5Q, seg_pos={0 0}, upload=479365]: Long tail cancellation due to optimal threshold Dec 28 10:09:52.424821042 (546h40m52.295904087s) [seg_pos={0 0}, upload=479365]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:09:52.424905633 (546h40m52.295988688s) [upload=479365]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:09:52.424912926 (546h40m52.295995961s) [upload=479365]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:09:52.713376890 (546h40m52.584459925s) [upload=479365]: Done uploading segments... Dec 28 10:09:52.713400135 (546h40m52.58448317s) [upload=479365]: Upload finished. err: <nil> Duration: 5.814991407s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...X......XX...X..X.........X.........X...................X.] duration: 6.105679736s trace: https://jaeger.dev.storj.io/trace/298f7e849145378d FAIL (2025-12-28 10:10:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 546h40m49.686771476s) [upload=479365]: Done starting upload Dec 28 10:09:49.815901106 (546h40m49.686984141s) [upload=479365]: Uploading segments... Dec 28 10:09:49.815945662 (546h40m49.687028707s) [upload=479365]: Got next segment. Inline: false Dec 28 10:09:49.815975247 (546h40m49.687058292s) [upload=479365]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:09:49.815979797 (546h40m49.687062832s) [upload=479365]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:09:50.069821454 (546h40m49.940904489s) [seg_pos={0 0}, upload=479365]: Begin upload segment... Dec 28 10:09:50.069862241 (546h40m49.940945276s) [seg_pos={0 0}, upload=479365]: Done begin upload segment. Dec 28 10:09:50.069897478 (546h40m49.940980513s) [noise=true , address=145.79.227.4:4187 , piece_id=MWP3DU5HEPGFEWVGNNAX5QNY2OZCB4L3ZQ3GGNUPUJF5J6UHZCSA, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.070930515 (546h40m49.94201356s) [noise=true , address=77.90.31.15:4697 , piece_id=MF3WHAUG3ER5SAAC67CFC77WYSBO7TO6ZGAOTB6KUIMM6R66AJ5Q, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.071091021 (546h40m49.942174056s) [noise=true , address=145.79.227.11:4730 , piece_id=7TTT6UD33GMOKJJFFMN6AV227QDJNK3TREMSWUFNLIOA7PJIZGUQ, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.071334024 (546h40m49.942417069s) [noise=true , address=77.90.31.10:4520 , piece_id=4QKC3GYDDP3YEREIAEXW6NVKT5SPGFCK54U6K7UMWITPNEIH4HOQ, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.071567939 (546h40m49.942650974s) [noise=true , address=45.91.116.5:4298 , piece_id=GDPNW5S3Z3L3Q4ZBU6QLFY6STGHLMEHYDFDNBSCSKQX4LWFFP2FA, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.071694199 (546h40m49.942777244s) [noise=true , address=145.79.227.37:4566 , piece_id=ZZB5ADNELY6LXKBJCDRRAFU34J4ICCWJIO62FICBIPT7QCMSVOLQ, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.071916012 (546h40m49.942999047s) [noise=true , address=145.79.227.2:4114 , piece_id=OX6KOGTRDJSLP6KZYTKCC5ERQDNVLNK7LBF2TZ5DLME45LLCJ6QQ, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.072047642 (546h40m49.943130687s) [noise=true , address=145.79.227.48:4302 , piece_id=VUZQ2TDAQMD4BE3QFK57UWVBQJA3AWXLNNPUSHO4U2W7MNZW2DOA, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.072298921 (546h40m49.943381956s) [noise=true , address=145.79.227.34:4905 , piece_id=PTAKHKSPPU3K67IV6XN74OHAVWLF45X5ZHYXEVOR4KJGE7FNSIYQ, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.072401216 (546h40m49.943484261s) [noise=true , address=145.79.227.6:4258 , piece_id=QVL3KNSVPCGK3OD5LH5HLU5ZUGN5SSBYBVTL62IWZRC2VXVNLS7A, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.072623439 (546h40m49.943706474s) [noise=true , address=45.91.116.8:4048 , piece_id=ZG5KHG5OTVF2WTAMIPIR4DINKP5S4VQQCHNNUZDJ6L2TCQ465AHA, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.072767583 (546h40m49.943850628s) [noise=true , address=145.79.227.12:4765 , piece_id=YHHYUMZMXVYEOFL3X7KEM3D4EMEQ4MP63EKNKRF3Y43OL4CHDEQQ, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.072934110 (546h40m49.944017156s) [noise=true , address=145.79.227.21:4474 , piece_id=NYWNSVVEGNZWZRUXKY4WMLPUUC7W5PYZEAANP4W74OH35E55XYJQ, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.073134392 (546h40m49.944217427s) [noise=true , address=145.79.227.32:4835 , piece_id=F6U7THJV5UQU4RTCUMDKHCLGLWV7ALFNCEJUNSK3YXXHG5K3HO5A, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.073503574 (546h40m49.944586609s) [noise=true , address=145.79.227.16:4912 , piece_id=2SY5IGKX5GDCZPPTAGCVKDF5OCTIBHTMTD3W4N4PXLZD2W2JBAHQ, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.073878869 (546h40m49.944961904s) [noise=true , address=145.79.227.3:4149 , piece_id=33GQOMODSGCKTVBJQ4BTB5B7ILCBB4CCJADI4XDHQVMR547EPSIQ, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:51.795106108 (546h40m51.666189143s) [upload=479365]: Next returned nil segment Dec 28 10:09:51.795149140 (546h40m51.666232185s) [upload=479365]: Waiting for error group managing segments... Dec 28 10:09:52.082599142 (546h40m51.953682177s) [noise=true , address=145.79.227.2:4114 , piece_id=OX6KOGTRDJSLP6KZYTKCC5ERQDNVLNK7LBF2TZ5DLME45LLCJ6QQ, seg_pos={0 0}, upload=479365]: Done uploading piece. err: <nil> Dec 28 10:09:52.100437450 (546h40m51.971520485s) [noise=true , address=145.79.227.3:4149 , piece_id=33GQOMODSGCKTVBJQ4BTB5B7ILCBB4CCJADI4XDHQVMR547EPSIQ, seg_pos={0 0}, upload=479365]: Done uploading piece. err: <nil> Dec 28 10:09:52.101903622 (546h40m51.972986657s) [noise=true , address=145.79.227.37:4566 , piece_id=ZZB5ADNELY6LXKBJCDRRAFU34J4ICCWJIO62FICBIPT7QCMSVOLQ, seg_pos={0 0}, upload=479365]: Done uploading piece. err: <nil> Dec 28 10:09:52.103627946 (546h40m51.974710981s) [noise=true , address=145.79.227.32:4835 , piece_id=F6U7THJV5UQU4RTCUMDKHCLGLWV7ALFNCEJUNSK3YXXHG5K3HO5A, seg_pos={0 0}, upload=479365]: Done uploading piece. err: <nil> Dec 28 10:09:52.107846759 (546h40m51.978929794s) [noise=true , address=145.79.227.21:4474 , piece_id=NYWNSVVEGNZWZRUXKY4WMLPUUC7W5PYZEAANP4W74OH35E55XYJQ, seg_pos={0 0}, upload=479365]: Done uploading piece. err: <nil> Dec 28 10:09:52.111095154 (546h40m51.982178189s) [noise=true , address=145.79.227.48:4302 , piece_id=VUZQ2TDAQMD4BE3QFK57UWVBQJA3AWXLNNPUSHO4U2W7MNZW2DOA, seg_pos={0 0}, upload=479365]: Done uploading piece. err: <nil> Dec 28 10:09:52.113636133 (546h40m51.984719178s) [noise=true , address=145.79.227.12:4765 , piece_id=YHHYUMZMXVYEOFL3X7KEM3D4EMEQ4MP63EKNKRF3Y43OL4CHDEQQ, seg_pos={0 0}, upload=479365]: Done uploading piece. err: <nil> Dec 28 10:09:52.113768515 (546h40m51.98485155s) [noise=true , address=145.79.227.16:4912 , piece_id=2SY5IGKX5GDCZPPTAGCVKDF5OCTIBHTMTD3W4N4PXLZD2W2JBAHQ, seg_pos={0 0}, upload=479365]: Done uploading piece. err: <nil> Dec 28 10:09:52.118758556 (546h40m51.989841592s) [noise=true , address=145.79.227.4:4187 , piece_id=MWP3DU5HEPGFEWVGNNAX5QNY2OZCB4L3ZQ3GGNUPUJF5J6UHZCSA, seg_pos={0 0}, upload=479365]: Done uploading piece. err: <nil> Dec 28 10:09:52.119015597 (546h40m51.990098632s) [noise=true , address=145.79.227.11:4730 , piece_id=7TTT6UD33GMOKJJFFMN6AV227QDJNK3TREMSWUFNLIOA7PJIZGUQ, seg_pos={0 0}, upload=479365]: Done uploading piece. err: <nil> Dec 28 10:09:52.119331037 (546h40m51.990414082s) [noise=true , address=145.79.227.6:4258 , piece_id=QVL3KNSVPCGK3OD5LH5HLU5ZUGN5SSBYBVTL62IWZRC2VXVNLS7A, seg_pos={0 0}, upload=479365]: Done uploading piece. err: <nil> Dec 28 10:09:52.292892297 (546h40m52.163975342s) [noise=true , address=45.91.116.5:4298 , piece_id=GDPNW5S3Z3L3Q4ZBU6QLFY6STGHLMEHYDFDNBSCSKQX4LWFFP2FA, seg_pos={0 0}, upload=479365]: Done uploading piece. err: <nil> Dec 28 10:09:52.423498404 (546h40m52.294581449s) [noise=true , address=45.91.116.8:4048 , piece_id=ZG5KHG5OTVF2WTAMIPIR4DINKP5S4VQQCHNNUZDJ6L2TCQ465AHA, seg_pos={0 0}, upload=479365]: Done uploading piece. err: <nil> Dec 28 10:09:52.423618842 (546h40m52.294701887s) [seg_pos={0 0}, upload=479365]: Segment reached optimal threshold of 13 pieces. Dec 28 10:09:52.423873567 (546h40m52.294956602s) [noise=true , address=145.79.227.34:4905 , piece_id=PTAKHKSPPU3K67IV6XN74OHAVWLF45X5ZHYXEVOR4KJGE7FNSIYQ, seg_pos={0 0}, upload=479365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 28 10:09:52.423883487 (546h40m52.294966522s) [noise=true , address=145.79.227.34:4905 , piece_id=PTAKHKSPPU3K67IV6XN74OHAVWLF45X5ZHYXEVOR4KJGE7FNSIYQ, seg_pos={0 0}, upload=479365]: Long tail cancellation due to optimal threshold Dec 28 10:09:52.424357258 (546h40m52.295440313s) [noise=true , address=77.90.31.10:4520 , piece_id=4QKC3GYDDP3YEREIAEXW6NVKT5SPGFCK54U6K7UMWITPNEIH4HOQ, seg_pos={0 0}, upload=479365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:39386->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:39386->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39386->77.90.31.10:4520: use of closed network connection Dec 28 10:09:52.424369061 (546h40m52.295452117s) [noise=true , address=77.90.31.10:4520 , piece_id=4QKC3GYDDP3YEREIAEXW6NVKT5SPGFCK54U6K7UMWITPNEIH4HOQ, seg_pos={0 0}, upload=479365]: Long tail cancellation due to optimal threshold Dec 28 10:09:52.424762400 (546h40m52.295845465s) [noise=true , address=77.90.31.15:4697 , piece_id=MF3WHAUG3ER5SAAC67CFC77WYSBO7TO6ZGAOTB6KUIMM6R66AJ5Q, seg_pos={0 0}, upload=479365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: read tcp 188.166.225.170:40794->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 10:09:52.424771748 (546h40m52.295854783s) [noise=true , address=77.90.31.15:4697 , piece_id=MF3WHAUG3ER5SAAC67CFC77WYSBO7TO6ZGAOTB6KUIMM6R66AJ5Q, seg_pos={0 0}, upload=479365]: Long tail cancellation due to optimal threshold Dec 28 10:09:52.424821042 (546h40m52.295904087s) [seg_pos={0 0}, upload=479365]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:09:52.424905633 (546h40m52.295988688s) [upload=479365]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:09:52.424912926 (546h40m52.295995961s) [upload=479365]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:09:52.713376890 (546h40m52.584459925s) [upload=479365]: Done uploading segments... Dec 28 10:09:52.713400135 (546h40m52.58448317s) [upload=479365]: Upload finished. err: <nil> Duration: 6.105698271s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X...X......XX...X..X.........X.........X...................X] duration: 30.00343911s trace: https://jaeger.dev.storj.io/trace/298f7e849145378d FAIL (2025-12-28 10:10:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 546h40m49.686771476s) [upload=479365]: Done starting upload Dec 28 10:09:49.815901106 (546h40m49.686984141s) [upload=479365]: Uploading segments... Dec 28 10:09:49.815945662 (546h40m49.687028707s) [upload=479365]: Got next segment. Inline: false Dec 28 10:09:49.815975247 (546h40m49.687058292s) [upload=479365]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:09:49.815979797 (546h40m49.687062832s) [upload=479365]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:09:50.069821454 (546h40m49.940904489s) [seg_pos={0 0}, upload=479365]: Begin upload segment... Dec 28 10:09:50.069862241 (546h40m49.940945276s) [seg_pos={0 0}, upload=479365]: Done begin upload segment. Dec 28 10:09:50.069897478 (546h40m49.940980513s) [noise=true , address=145.79.227.4:4187 , piece_id=MWP3DU5HEPGFEWVGNNAX5QNY2OZCB4L3ZQ3GGNUPUJF5J6UHZCSA, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.070930515 (546h40m49.94201356s) [noise=true , address=77.90.31.15:4697 , piece_id=MF3WHAUG3ER5SAAC67CFC77WYSBO7TO6ZGAOTB6KUIMM6R66AJ5Q, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.071091021 (546h40m49.942174056s) [noise=true , address=145.79.227.11:4730 , piece_id=7TTT6UD33GMOKJJFFMN6AV227QDJNK3TREMSWUFNLIOA7PJIZGUQ, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.071334024 (546h40m49.942417069s) [noise=true , address=77.90.31.10:4520 , piece_id=4QKC3GYDDP3YEREIAEXW6NVKT5SPGFCK54U6K7UMWITPNEIH4HOQ, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.071567939 (546h40m49.942650974s) [noise=true , address=45.91.116.5:4298 , piece_id=GDPNW5S3Z3L3Q4ZBU6QLFY6STGHLMEHYDFDNBSCSKQX4LWFFP2FA, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.071694199 (546h40m49.942777244s) [noise=true , address=145.79.227.37:4566 , piece_id=ZZB5ADNELY6LXKBJCDRRAFU34J4ICCWJIO62FICBIPT7QCMSVOLQ, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.071916012 (546h40m49.942999047s) [noise=true , address=145.79.227.2:4114 , piece_id=OX6KOGTRDJSLP6KZYTKCC5ERQDNVLNK7LBF2TZ5DLME45LLCJ6QQ, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.072047642 (546h40m49.943130687s) [noise=true , address=145.79.227.48:4302 , piece_id=VUZQ2TDAQMD4BE3QFK57UWVBQJA3AWXLNNPUSHO4U2W7MNZW2DOA, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.072298921 (546h40m49.943381956s) [noise=true , address=145.79.227.34:4905 , piece_id=PTAKHKSPPU3K67IV6XN74OHAVWLF45X5ZHYXEVOR4KJGE7FNSIYQ, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.072401216 (546h40m49.943484261s) [noise=true , address=145.79.227.6:4258 , piece_id=QVL3KNSVPCGK3OD5LH5HLU5ZUGN5SSBYBVTL62IWZRC2VXVNLS7A, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.072623439 (546h40m49.943706474s) [noise=true , address=45.91.116.8:4048 , piece_id=ZG5KHG5OTVF2WTAMIPIR4DINKP5S4VQQCHNNUZDJ6L2TCQ465AHA, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.072767583 (546h40m49.943850628s) [noise=true , address=145.79.227.12:4765 , piece_id=YHHYUMZMXVYEOFL3X7KEM3D4EMEQ4MP63EKNKRF3Y43OL4CHDEQQ, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.072934110 (546h40m49.944017156s) [noise=true , address=145.79.227.21:4474 , piece_id=NYWNSVVEGNZWZRUXKY4WMLPUUC7W5PYZEAANP4W74OH35E55XYJQ, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.073134392 (546h40m49.944217427s) [noise=true , address=145.79.227.32:4835 , piece_id=F6U7THJV5UQU4RTCUMDKHCLGLWV7ALFNCEJUNSK3YXXHG5K3HO5A, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.073503574 (546h40m49.944586609s) [noise=true , address=145.79.227.16:4912 , piece_id=2SY5IGKX5GDCZPPTAGCVKDF5OCTIBHTMTD3W4N4PXLZD2W2JBAHQ, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:50.073878869 (546h40m49.944961904s) [noise=true , address=145.79.227.3:4149 , piece_id=33GQOMODSGCKTVBJQ4BTB5B7ILCBB4CCJADI4XDHQVMR547EPSIQ, seg_pos={0 0}, upload=479365]: Uploading piece... Dec 28 10:09:51.795106108 (546h40m51.666189143s) [upload=479365]: Next returned nil segment Dec 28 10:09:51.795149140 (546h40m51.666232185s) [upload=479365]: Waiting for error group managing segments... Dec 28 10:09:52.082599142 (546h40m51.953682177s) [noise=true , address=145.79.227.2:4114 , piece_id=OX6KOGTRDJSLP6KZYTKCC5ERQDNVLNK7LBF2TZ5DLME45LLCJ6QQ, seg_pos={0 0}, upload=479365]: Done uploading piece. err: <nil> Dec 28 10:09:52.100437450 (546h40m51.971520485s) [noise=true , address=145.79.227.3:4149 , piece_id=33GQOMODSGCKTVBJQ4BTB5B7ILCBB4CCJADI4XDHQVMR547EPSIQ, seg_pos={0 0}, upload=479365]: Done uploading piece. err: <nil> Dec 28 10:09:52.101903622 (546h40m51.972986657s) [noise=true , address=145.79.227.37:4566 , piece_id=ZZB5ADNELY6LXKBJCDRRAFU34J4ICCWJIO62FICBIPT7QCMSVOLQ, seg_pos={0 0}, upload=479365]: Done uploading piece. err: <nil> Dec 28 10:09:52.103627946 (546h40m51.974710981s) [noise=true , address=145.79.227.32:4835 , piece_id=F6U7THJV5UQU4RTCUMDKHCLGLWV7ALFNCEJUNSK3YXXHG5K3HO5A, seg_pos={0 0}, upload=479365]: Done uploading piece. err: <nil> Dec 28 10:09:52.107846759 (546h40m51.978929794s) [noise=true , address=145.79.227.21:4474 , piece_id=NYWNSVVEGNZWZRUXKY4WMLPUUC7W5PYZEAANP4W74OH35E55XYJQ, seg_pos={0 0}, upload=479365]: Done uploading piece. err: <nil> Dec 28 10:09:52.111095154 (546h40m51.982178189s) [noise=true , address=145.79.227.48:4302 , piece_id=VUZQ2TDAQMD4BE3QFK57UWVBQJA3AWXLNNPUSHO4U2W7MNZW2DOA, seg_pos={0 0}, upload=479365]: Done uploading piece. err: <nil> Dec 28 10:09:52.113636133 (546h40m51.984719178s) [noise=true , address=145.79.227.12:4765 , piece_id=YHHYUMZMXVYEOFL3X7KEM3D4EMEQ4MP63EKNKRF3Y43OL4CHDEQQ, seg_pos={0 0}, upload=479365]: Done uploading piece. err: <nil> Dec 28 10:09:52.113768515 (546h40m51.98485155s) [noise=true , address=145.79.227.16:4912 , piece_id=2SY5IGKX5GDCZPPTAGCVKDF5OCTIBHTMTD3W4N4PXLZD2W2JBAHQ, seg_pos={0 0}, upload=479365]: Done uploading piece. err: <nil> Dec 28 10:09:52.118758556 (546h40m51.989841592s) [noise=true , address=145.79.227.4:4187 , piece_id=MWP3DU5HEPGFEWVGNNAX5QNY2OZCB4L3ZQ3GGNUPUJF5J6UHZCSA, seg_pos={0 0}, upload=479365]: Done uploading piece. err: <nil> Dec 28 10:09:52.119015597 (546h40m51.990098632s) [noise=true , address=145.79.227.11:4730 , piece_id=7TTT6UD33GMOKJJFFMN6AV227QDJNK3TREMSWUFNLIOA7PJIZGUQ, seg_pos={0 0}, upload=479365]: Done uploading piece. err: <nil> Dec 28 10:09:52.119331037 (546h40m51.990414082s) [noise=true , address=145.79.227.6:4258 , piece_id=QVL3KNSVPCGK3OD5LH5HLU5ZUGN5SSBYBVTL62IWZRC2VXVNLS7A, seg_pos={0 0}, upload=479365]: Done uploading piece. err: <nil> Dec 28 10:09:52.292892297 (546h40m52.163975342s) [noise=true , address=45.91.116.5:4298 , piece_id=GDPNW5S3Z3L3Q4ZBU6QLFY6STGHLMEHYDFDNBSCSKQX4LWFFP2FA, seg_pos={0 0}, upload=479365]: Done uploading piece. err: <nil> Dec 28 10:09:52.423498404 (546h40m52.294581449s) [noise=true , address=45.91.116.8:4048 , piece_id=ZG5KHG5OTVF2WTAMIPIR4DINKP5S4VQQCHNNUZDJ6L2TCQ465AHA, seg_pos={0 0}, upload=479365]: Done uploading piece. err: <nil> Dec 28 10:09:52.423618842 (546h40m52.294701887s) [seg_pos={0 0}, upload=479365]: Segment reached optimal threshold of 13 pieces. Dec 28 10:09:52.423873567 (546h40m52.294956602s) [noise=true , address=145.79.227.34:4905 , piece_id=PTAKHKSPPU3K67IV6XN74OHAVWLF45X5ZHYXEVOR4KJGE7FNSIYQ, seg_pos={0 0}, upload=479365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 28 10:09:52.423883487 (546h40m52.294966522s) [noise=true , address=145.79.227.34:4905 , piece_id=PTAKHKSPPU3K67IV6XN74OHAVWLF45X5ZHYXEVOR4KJGE7FNSIYQ, seg_pos={0 0}, upload=479365]: Long tail cancellation due to optimal threshold Dec 28 10:09:52.424357258 (546h40m52.295440313s) [noise=true , address=77.90.31.10:4520 , piece_id=4QKC3GYDDP3YEREIAEXW6NVKT5SPGFCK54U6K7UMWITPNEIH4HOQ, seg_pos={0 0}, upload=479365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:39386->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:39386->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39386->77.90.31.10:4520: use of closed network connection Dec 28 10:09:52.424369061 (546h40m52.295452117s) [noise=true , address=77.90.31.10:4520 , piece_id=4QKC3GYDDP3YEREIAEXW6NVKT5SPGFCK54U6K7UMWITPNEIH4HOQ, seg_pos={0 0}, upload=479365]: Long tail cancellation due to optimal threshold Dec 28 10:09:52.424762400 (546h40m52.295845465s) [noise=true , address=77.90.31.15:4697 , piece_id=MF3WHAUG3ER5SAAC67CFC77WYSBO7TO6ZGAOTB6KUIMM6R66AJ5Q, seg_pos={0 0}, upload=479365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: read tcp 188.166.225.170:40794->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 10:09:52.424771748 (546h40m52.295854783s) [noise=true , address=77.90.31.15:4697 , piece_id=MF3WHAUG3ER5SAAC67CFC77WYSBO7TO6ZGAOTB6KUIMM6R66AJ5Q, seg_pos={0 0}, upload=479365]: Long tail cancellation due to optimal threshold Dec 28 10:09:52.424821042 (546h40m52.295904087s) [seg_pos={0 0}, upload=479365]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:09:52.424905633 (546h40m52.295988688s) [upload=479365]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:09:52.424912926 (546h40m52.295995961s) [upload=479365]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:09:52.713376890 (546h40m52.584459925s) [upload=479365]: Done uploading segments... Dec 28 10:09:52.713400135 (546h40m52.58448317s) [upload=479365]: Upload finished. err: <nil> Duration: 30.00346435s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [X.XX..X...XXX..XX.X...X.X....XXXXX...X........X.....X.......] duration: 3.455327096s trace: https://jaeger.dev.storj.io/trace/3f41c3ec34adb72f FAIL (2025-12-28 10:10:00 UTC) orbiter warning: context deadline exceeded Duration: 3.455394344s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X...X......XX...X..X.........X.........X...................] duration: 5.957976052s trace: https://jaeger.dev.storj.io/trace/793e8a2843c4a07d FAIL (2025-12-28 10:09:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 09:52:24.930460164 (546h23m24.801543199s) [upload=479121]: Starting upload Dec 28 09:52:24.930539404 (546h23m24.80162245s) [upload=479121]: Done starting upload Dec 28 09:52:24.930681986 (546h23m24.801765021s) [upload=479121]: Uploading segments... Dec 28 09:52:24.930770294 (546h23m24.80185333s) [upload=479121]: Got next segment. Inline: false Dec 28 09:52:24.930818035 (546h23m24.80190107s) [upload=479121]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 09:52:24.930823295 (546h23m24.80190634s) [upload=479121]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 09:52:25.174324282 (546h23m25.045407327s) [seg_pos={0 0}, upload=479121]: Begin upload segment... Dec 28 09:52:25.174405998 (546h23m25.045489043s) [seg_pos={0 0}, upload=479121]: Done begin upload segment. Dec 28 09:52:25.174442007 (546h23m25.045525052s) [noise=true , address=145.79.227.23:4547 , piece_id=BBYTYZ5B3TC2GAEKY7DB6PY6GYADFB5LMEXIDPRBA2XL43GZDL3Q, seg_pos={0 0}, upload=479121]: Uploading piece... Dec 28 09:52:25.174450212 (546h23m25.045533247s) [noise=true , address=145.79.227.19:4077 , piece_id=O64IFVO37B5P3DSRVZHSKYC3WDZYDAA47TMVR7KNQ366SM7CMGSQ, seg_pos={0 0}, upload=479121]: Uploading piece... Dec 28 09:52:25.174548720 (546h23m25.045631755s) [noise=true , address=45.91.116.9:4261 , piece_id=RORJKZHLYE5N36FNOYXWVJWRMGGURJNUBBUR7OHTZ2YOCPELFOPQ, seg_pos={0 0}, upload=479121]: Uploading piece... Dec 28 09:52:25.174864992 (546h23m25.045948037s) [noise=true , address=45.91.116.13:4155 , piece_id=I3KTF7F4SX4CKUSB5Q7A6WM5YKDUVFRIU6XAIZMKLAKVU7DTG2JQ, seg_pos={0 0}, upload=479121]: Uploading piece... Dec 28 09:52:25.175170764 (546h23m25.046253799s) [noise=true , address=145.79.227.4:4185 , piece_id=EJCPHCKREI6VTCLEJ6HIR4TMC4GMCI5YKGFPMNYKXPNO53WDI4BA, seg_pos={0 0}, upload=479121]: Uploading piece... Dec 28 09:52:25.175461357 (546h23m25.046544392s) [noise=true , address=145.79.227.47:4265 , piece_id=HQKUIZPCUZENDM3MLE2BTBHMNPA3QZYEY3O7ZJIGPDRWNDHLDZSA, seg_pos={0 0}, upload=479121]: Uploading piece... Dec 28 09:52:25.175819008 (546h23m25.046902043s) [noise=true , address=145.79.227.11:4732 , piece_id=SG7FXUEWIZHWL3SUTK7Z5DXQQBEFGSC2UUXETK4ZELNDWF3Q2PUQ, seg_pos={0 0}, upload=479121]: Uploading piece... Dec 28 09:52:25.176217868 (546h23m25.047300903s) [noise=true , address=145.79.227.3:4150 , piece_id=6DFHZ3WVUTUHUF557YW6ACRPJ5POKBNRS3DH2YWVWLL76PTENS7A, seg_pos={0 0}, upload=479121]: Uploading piece... Dec 28 09:52:25.176564178 (546h23m25.047647223s) [noise=true , address=145.79.227.34:4906 , piece_id=ZWHVJYIBK4SM6DOUQMGZXLEWNQFC4PLF3KVOQWG4ENEDWAPEXFKQ, seg_pos={0 0}, upload=479121]: Uploading piece... Dec 28 09:52:25.176913082 (546h23m25.047996117s) [noise=true , address=145.79.227.29:4943 , piece_id=7LVJFEMKPEBXWNO3D45B4D2YUDJVBYFCBVPCRPYLZLLZXOE2BT3A, seg_pos={0 0}, upload=479121]: Uploading piece... Dec 28 09:52:25.177911093 (546h23m25.048994128s) [noise=true , address=77.90.31.11:4555 , piece_id=LHVOKMLEQ2WGDGID2M5VEKOW2DEZX2TEPCFA6VQKJMKXAINQGXGA, seg_pos={0 0}, upload=479121]: Uploading piece... Dec 28 09:52:25.178060377 (546h23m25.049143412s) [noise=true , address=145.79.227.37:4565 , piece_id=MBKUVATFY3BLCGMH3BWFK6SL2YQUG4LUITKKALIX3CXH5JR7O7CA, seg_pos={0 0}, upload=479121]: Uploading piece... Dec 28 09:52:25.179229272 (546h23m25.050312307s) [noise=true , address=77.90.31.13:4626 , piece_id=IINEUJY2XAHCCQ2DOULNIKMJJXCA73477EQ3MR5WCXG5RVDQPROA, seg_pos={0 0}, upload=479121]: Uploading piece... Dec 28 09:52:25.180452943 (546h23m25.051535978s) [noise=true , address=145.79.227.52:4445 , piece_id=EWNJK7QMSKUEDSAFY4R5JELYD2FFAVNUGVHXFNY4OMMFT5GHEQ4Q, seg_pos={0 0}, upload=479121]: Uploading piece... Dec 28 09:52:25.181673807 (546h23m25.052756852s) [noise=true , address=145.79.227.27:4691 , piece_id=EUGSZIRMJZO5XHAYUVVPEX2RTGEOELBTOY7ZIYD2RYS4Z5UQA6YQ, seg_pos={0 0}, upload=479121]: Uploading piece... Dec 28 09:52:25.183712709 (546h23m25.054795744s) [noise=true , address=145.79.227.25:4619 , piece_id=NOJNAKUQPQ2GM6GYXBVQNHM6L3CX6R5M5JXCOZV3NPJMYN4BIB7Q, seg_pos={0 0}, upload=479121]: Uploading piece... Dec 28 09:52:26.898855523 (546h23m26.769938568s) [upload=479121]: Next returned nil segment Dec 28 09:52:26.898876522 (546h23m26.769959557s) [upload=479121]: Waiting for error group managing segments... Dec 28 09:52:27.184661556 (546h23m27.055744611s) [noise=true , address=145.79.227.37:4565 , piece_id=MBKUVATFY3BLCGMH3BWFK6SL2YQUG4LUITKKALIX3CXH5JR7O7CA, seg_pos={0 0}, upload=479121]: Done uploading piece. err: <nil> Dec 28 09:52:27.188900978 (546h23m27.059984013s) [noise=true , address=145.79.227.3:4150 , piece_id=6DFHZ3WVUTUHUF557YW6ACRPJ5POKBNRS3DH2YWVWLL76PTENS7A, seg_pos={0 0}, upload=479121]: Done uploading piece. err: <nil> Dec 28 09:52:27.196314926 (546h23m27.067397971s) [noise=true , address=145.79.227.29:4943 , piece_id=7LVJFEMKPEBXWNO3D45B4D2YUDJVBYFCBVPCRPYLZLLZXOE2BT3A, seg_pos={0 0}, upload=479121]: Done uploading piece. err: <nil> Dec 28 09:52:27.200795417 (546h23m27.071878482s) [noise=true , address=145.79.227.11:4732 , piece_id=SG7FXUEWIZHWL3SUTK7Z5DXQQBEFGSC2UUXETK4ZELNDWF3Q2PUQ, seg_pos={0 0}, upload=479121]: Done uploading piece. err: <nil> Dec 28 09:52:27.202473503 (546h23m27.073556548s) [noise=true , address=145.79.227.4:4185 , piece_id=EJCPHCKREI6VTCLEJ6HIR4TMC4GMCI5YKGFPMNYKXPNO53WDI4BA, seg_pos={0 0}, upload=479121]: Done uploading piece. err: <nil> Dec 28 09:52:27.205997453 (546h23m27.077080508s) [noise=true , address=145.79.227.23:4547 , piece_id=BBYTYZ5B3TC2GAEKY7DB6PY6GYADFB5LMEXIDPRBA2XL43GZDL3Q, seg_pos={0 0}, upload=479121]: Done uploading piece. err: <nil> Dec 28 09:52:27.212532948 (546h23m27.083615983s) [noise=true , address=145.79.227.27:4691 , piece_id=EUGSZIRMJZO5XHAYUVVPEX2RTGEOELBTOY7ZIYD2RYS4Z5UQA6YQ, seg_pos={0 0}, upload=479121]: Done uploading piece. err: <nil> Dec 28 09:52:27.213243360 (546h23m27.084326395s) [noise=true , address=145.79.227.25:4619 , piece_id=NOJNAKUQPQ2GM6GYXBVQNHM6L3CX6R5M5JXCOZV3NPJMYN4BIB7Q, seg_pos={0 0}, upload=479121]: Done uploading piece. err: <nil> Dec 28 09:52:27.215701541 (546h23m27.086784586s) [noise=true , address=145.79.227.52:4445 , piece_id=EWNJK7QMSKUEDSAFY4R5JELYD2FFAVNUGVHXFNY4OMMFT5GHEQ4Q, seg_pos={0 0}, upload=479121]: Done uploading piece. err: <nil> Dec 28 09:52:27.215857678 (546h23m27.086940723s) [noise=true , address=145.79.227.47:4265 , piece_id=HQKUIZPCUZENDM3MLE2BTBHMNPA3QZYEY3O7ZJIGPDRWNDHLDZSA, seg_pos={0 0}, upload=479121]: Done uploading piece. err: <nil> Dec 28 09:52:27.242052217 (546h23m27.113135262s) [noise=true , address=145.79.227.19:4077 , piece_id=O64IFVO37B5P3DSRVZHSKYC3WDZYDAA47TMVR7KNQ366SM7CMGSQ, seg_pos={0 0}, upload=479121]: Done uploading piece. err: <nil> Dec 28 09:52:27.918716361 (546h23m27.789799396s) [noise=true , address=77.90.31.13:4626 , piece_id=IINEUJY2XAHCCQ2DOULNIKMJJXCA73477EQ3MR5WCXG5RVDQPROA, seg_pos={0 0}, upload=479121]: Done uploading piece. err: <nil> Dec 28 09:52:28.254750486 (546h23m28.125833521s) [noise=true , address=77.90.31.11:4555 , piece_id=LHVOKMLEQ2WGDGID2M5VEKOW2DEZX2TEPCFA6VQKJMKXAINQGXGA, seg_pos={0 0}, upload=479121]: Done uploading piece. err: <nil> Dec 28 09:52:28.254772227 (546h23m28.125855262s) [seg_pos={0 0}, upload=479121]: Segment reached optimal threshold of 13 pieces. Dec 28 09:52:28.254960997 (546h23m28.126044032s) [noise=true , address=45.91.116.13:4155 , piece_id=I3KTF7F4SX4CKUSB5Q7A6WM5YKDUVFRIU6XAIZMKLAKVU7DTG2JQ, seg_pos={0 0}, upload=479121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 28 09:52:28.254970225 (546h23m28.12605327s) [noise=true , address=45.91.116.13:4155 , piece_id=I3KTF7F4SX4CKUSB5Q7A6WM5YKDUVFRIU6XAIZMKLAKVU7DTG2JQ, seg_pos={0 0}, upload=479121]: Long tail cancellation due to optimal threshold Dec 28 09:52:28.255070776 (546h23m28.126153811s) [noise=true , address=45.91.116.9:4261 , piece_id=RORJKZHLYE5N36FNOYXWVJWRMGGURJNUBBUR7OHTZ2YOCPELFOPQ, seg_pos={0 0}, upload=479121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 28 09:52:28.255081105 (546h23m28.126164151s) [noise=true , address=45.91.116.9:4261 , piece_id=RORJKZHLYE5N36FNOYXWVJWRMGGURJNUBBUR7OHTZ2YOCPELFOPQ, seg_pos={0 0}, upload=479121]: Long tail cancellation due to optimal threshold Dec 28 09:52:28.255244747 (546h23m28.126327802s) [noise=true , address=145.79.227.34:4906 , piece_id=ZWHVJYIBK4SM6DOUQMGZXLEWNQFC4PLF3KVOQWG4ENEDWAPEXFKQ, seg_pos={0 0}, upload=479121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 28 09:52:28.255254145 (546h23m28.12633719s) [noise=true , address=145.79.227.34:4906 , piece_id=ZWHVJYIBK4SM6DOUQMGZXLEWNQFC4PLF3KVOQWG4ENEDWAPEXFKQ, seg_pos={0 0}, upload=479121]: Long tail cancellation due to optimal threshold Dec 28 09:52:28.255280215 (546h23m28.12636325s) [seg_pos={0 0}, upload=479121]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 09:52:28.255414841 (546h23m28.126497876s) [upload=479121]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 09:52:28.255422566 (546h23m28.126505611s) [upload=479121]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 09:52:28.550316404 (546h23m28.421399439s) [upload=479121]: Done uploading segments... Dec 28 09:52:28.550339337 (546h23m28.421422382s) [upload=479121]: Upload finished. err: <nil> Duration: 5.958003485s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [XX.XX..X...XXX..XX.X...X.X....XXXXX...X........X.....X......] duration: 3.505290154s trace: https://jaeger.dev.storj.io/trace/3f41c3ec34adb72f FAIL (2025-12-28 10:08:59 UTC) orbiter warning: context deadline exceeded Duration: 3.505334597s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)